Planet Perrygeo

Some things I learned about Berlin this trip:

• Berlin isn’t really home to as large of a ‘classical German’ attitude as I would have assumed. Instead, it is (as far as I understand it) pretty much hippyville for Germany. Things like the fact that everyone ignores no-smoking laws — Berlin much more so than anywhere else in Germany. The hipster attitude: same. The silly love of 80’s music… well, I don’t know much about that. (I do know that I’ve not seen posters for Journey concerts anywhere else in the US…)
• Oranienburger is part of the red-light district of Berlin — This is the first time I’ve ever been propositioned by two prostitutes in 10 minutes!

(Oh, also, I did some work.)

БогородицаThis week, Jess pointed out to me “An Invisible Woman Taught Me German“, a story in Slate about German Language learning via the Deutsche Welle organization.

My favorite quote: “It’s basically a Teutonic Scooby-Doo, with overt sexual tension among the young mystery-solvers. They investigate weird occurrences like crop circles, Beethoven’s ghost, and a Hamburg shark. (As in Scooby-Doo, you’ll see the ending from 1.6 km away.) When they get stumped, they query their talking computer Compu, who has impossibly advanced speech recognition, yet for some reason still whirrs and clicks like a 1970s adding machine. Sometimes a spooky talking owl named Eulalia lends a hand, flapping in on a cloud of horror movie sound effects.”

How can you beat that for foreign language learning?

This fall New York City will open The Academy for Software Engineering, the city’s first public high school that will actually train kids to develop software. The project has been a long time dream of Mike Zamansky, the highly-regarded CS teacher at New York’s elite Stuyvesant public high school. It was jump started when Fred Wilson, a VC at Union Square Ventures, promised to get the tech community to help with knowledge, advice, and money.

I’m on the board of advisors of the new school, which plans to accept ninth graders for fall of 2012. Here’s why I’m excited about this new school:

1. It’s a “limited, unscreened” school.  That’s Board of Ed jargon. It means that any student who is interested can apply—their grades and attendence record are not taken into account in deciding whether or not to admit them, only their interest. I think this is the best thing about the school. A lot of kids are just not interested enough in other academic subjects to get good grades, but they would make great software engineers. A lot of immigrants (especially in New York) are not yet proficient enough in English to get good grades in all their subjects, but they’re going to make great software engineers, too. And in my humble opinion, a school that accepts a cross-section of students is bound to be more enriching than a school that only accepts academic superstars.

2. OMG do we ever need more software engineers. The US post-secondary education system is massively failing us: it’s not producing even remotely enough programmers to meet the hiring needs of the technology industry. Not even remotely enough. Starting salaries for smart programmers from top schools are flirting with the $100,000 mark. Supply isn’t even close to meeting demand. This school is going to be pretty small (in the 400-500 student range) but the Board of Ed has promised that if it’s successful it’ll be used as a template for more schools or for special programs inside larger schools. I predict that they will be overwhelmed with applicants and this will be the most popular new school in New York City in years.

3. And we need more diversity, too. One of the reasons the elite US colleges seem to turn out so few computer science majors every year is that they are only drawing from a narrow pool of mostly white and asian males. Minorities and women are embarrassingly under-represented. Hopefully an unscreened school in New York City can pump a lot more diversity into the pool.

4. It’s not a vocational school. Unlike traditional vocational schools, this new school will have a rigorous academic component and will prepare students for college. But college is not for everyone—many of the best programmers I know were just not interested enough in a general four year degree and went straight into jobs programming.

I’m pleased to be involved in this project, but it needs more help: they’re still looking for qualified computer science teachers and a principal. If you’re interested drop me an email and I’ll make sure it gets through to the right people.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Just a few months ago, we launched Trello, a super simple, web-based team coordination system. The feedback has been overwhelmingly positive and adoption has been very strong, even in its early, 1.0 state.

Trello is new kind of development project for Fog Creek. It’s 100% hosted; there will never be an “installed software” version of Trello. That allowed us to modernize many aspects of our development process; I am happy to announce that there is absolutely no Visual Basic code involved in any part of Trello. What’s next, flying cars?

The biggest difference you’ll notice (compared to our previous products pitched solely at software developers) is that Trello is a totally horizontal product.

Horizontal means that it can be used by people from all walks of life. Word processors and web browsers are horizontal. The software your dentist uses to torture you with drills is vertical.

Vertical software is much easier to pull off and make money with, and it’s a good choice for your first startup. Here are two key reasons:

• It’s easier to find customers. If you make dentist software, you know which conventions to go to and which magazines to advertise in. All you have to do is find dentists.
• The margins are better. Your users are professionals at work and it makes sense for them to give you money if you can solve their problems.

Making a major horizontal product that’s useful in any walk of life is almost impossible to pull off. You can’t charge very much, because you’re competing with other horizontal products that can amortize their development costs across a huge number of users. It’s high risk, high reward: not suitable for a young bootstrapped startup, but not a bad idea for a second or third product from a mature and stable company like Fog Creek.

Forgive me if I now divert into telling you a quick story about my time spent on the Microsoft Excel team way back in 1991. (Yes, I know you were not born yet, but I assure you that computers had been invented. Just hop up here on my knee and shut up.)

Everybody thought of Excel as a financial modeling application. It was used for creating calculation models with formulas and stuff. You would put in your assumptions and then calculate things like “if interest rates go up by 0.00001% next year, what percentage of Las Vegas homeowners will plunge into bankruptcy?” For example.

Round about 1993 a couple of us went on customer visits to see how people were using Excel.

We found a fellow whose entire job consisted of maintaining the “number of injuries this week” spreadsheet for a large, highly-regulated utility.

Once a week, he opened an Excel spreadsheet which listed ten facilities, containing the name of the facilities and the number 0, which indicated that were 0 injuries that week. (They never had injuries).

He typed the current date in the top of the spreadsheet, printed a copy, put it in a three-ring binder, and that was pretty much his whole, entire job. It was kind of sad. He took two lunch breaks a day. I would too, if that was my whole job.

Over the next two weeks we visited dozens of Excel customers, and did not see anyone using Excel to actually perform what you would call “calculations.” Almost all of them were using Excel because it was a convenient way to create a table.

(Irrelevant sidenote: the few customers we could find who were doing calculations were banks, devising explosive devices called “derivatives.” They used Excel to maximize the bankers’ bonuses on nine out of ten years, and to cause western civilization to nearly collapse every tenth year. Something about black swans. Probably just a floating point rounding error.)

What was I talking about? Oh yeah... most people just used Excel to make lists. Suddenly we understood why Lotus Improv, which was this fancy futuristic spreadsheet that was going to make Excel obsolete, had failed completely: because it was great at calculations, but terrible at creating tables, and everyone was using Excel for tables, not calculations.

Bing! A light went off in my head.

The great horizontal killer applications are actually just fancy data structures.

Spreadsheets are not just tools for doing “what-if” analysis. They provide a specific data structure: a table. Most Excel users never enter a formula. They use Excel when they need a table. The gridlines are the most important feature of Excel, not recalc.

Word processors are not just tools for writing books, reports, and letters. They provide a specific data structure: lines of text which automatically wrap and split into pages.

PowerPoint is not just a tool for making boring meetings. It provides a specific data structure: an array of full-screen images. 

Some people saw Trello and said, “oh, it’s Kanban boards. For developing software the agile way.” Yeah, it’s that, but it’s also for planning a wedding, for making a list of potential vacation spots to share with your family, for keeping track of applicants to open job positions, and for a billion other things. In fact Trello is for anything where you want to maintain a list of lists with a group of people.

There are millions of things that need that kind of data structure, and there hasn’t been a great “list-of-list” app before Trello. (There have been outliners, but outlines are, IMHO, one of the great dead ends in UI design: so appealing to programmers, yet so useless to civilians).

Once you get into Trello, you’ll use it for everything. I use about thirty Trello boards regularly, and I use them with everyone in my life, from the APs (Aged Parents), with whom I plan vacations, with every team at work, and just about every project I’m involved in.

So, ok, that was the first big difference with Trello: horizonal, not vertical. But there are a bunch of other differences:

It’s delivered continuously. Rather than having major and minor releases, we pretty much just continuously push out new features from development to customers. A feature that you built and tested, but didn’t deliver yet because you’re waiting for the next major release, becomes inventory. Inventory is dead weight: money you spent that’s just wasting away without earning you anything. Sure, 100 years ago, we had these things called “CD-ROMs” and we shipped software that way, so there was an economic reason to bunch up features before we inflict ‘em on the world. But there’s no reason to work that way any more. You already knew that, of course. I’m just saying—I stopped using Visual Basic about five minutes ago. Brave New World.

It’s not exhaustively tested before being released. We thought we could get away with this because Trello is free, so customers are more forgiving. But to tell the truth, the real reason we get away with it is because bugs are fixed in a matter of hours, not months, so the net number of “bugs experienced by the public” is low.

We work in public. The rule on the Trello team is “default public.” We have a public Trello board that shows everything that we’re working on and where it’s up to. We use this to let customers vote and comment on their favorite features. By the way, while Trello was under development, it was secret. We had a lot of beta testers who gave us customer feedback so that the development team could use lean startup principles, but the nine months we spent building version 1.0 in secret gave us a significant lead in a competitive marketplace. But now that we’re shipping, there’s no reason not to talk about our plans.

This is a “Get Big Fast” product, not a “Ben and Jerry’s”  product. See Strategy Letter I. The business goal for Trello is to ultimately get to 100 million users. That means that our highest priority is removing any obstacles to adoption. Anything that people might use as a reason not to use Trello has to be found and eliminated. For example:

Trello is free. The friction caused by charging for a product is the biggest impediment to massive growth. In the long run, we think it’s much easier to figure out how to extract a small amount of money out of a large number of users than to extract a large amount of money out of a small number of users. Once you have 100 million users, it’s easy to figure out which of those users are getting the most value out of the product you built. The ones who are getting the most value will be happy to pay you. The others don’t cost much to support.

The API and plug-in architectures are the highest priority. Another way of putting that is:  never build anything in-house if you can expose a basic API and get those high-value users (the ones who are getting the most value out of the platform) to build it for you. On the Trello team, any feature that can be provided by a plug-in must be provided by a plug-in.

(The API is currently in very rudimentary form. You can already use it to do very interesting things. It is under rapid development.)

We use cutting edge technology. Often, this means we get cut fingers. Our developers bleed all over MongoDB, WebSockets, CoffeeScript and Node. But at least they’re having fun. And in today’s tight job market, great programmers have a lot of sway on what they’re going to be working on. If you can give them an exciting product that will touch millions of people, and let them dig deep into TCP-IP internals while they try to figure out why simple things aren’t working, they’ll have fun and they’ll love their jobs. Besides, we’re creating a product that we’ll be working on for the next ten years. Technology that’s merely “state of the art” today is going to be old and creaky in five years. We tried to go a little bit beyond “state of the art.” It’s a calculated risk.

None of this is very radical. TL;DR: Fog Creek Software develops an internet product using techniques that every Y-combinator startup has been using since spez was sleeping with his laptop so he could reboot Reddit when Lisp crashed in the middle of the night. If you haven’t tried Trello yet, try it, then tell me on twitter if it worked.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Overall, I’m loving the Air.

The laptop is small — so small it took me a couple days to adjust. But overall, it has everything I want or need, and I couldn’t ask for anything else. Most importantly, it has working wireless in my office — when the wireless in the office works at all, which comes-and-goes at best.

I may hate freedom, but I sure do love nifty hardware.

Yesterday, I finished my first deployment of a real service into AWS.

Along the way, I learned some things:

- Overall, the growth of the Amazon service offering is rapid and huge. I’ve said for a long time that much of the net today runs on software that was pioneered within LiveJournal — I think that what LiveJournal did for the web at large, Amazon is really doing for people who are moving to the cloud. Having things like S3 and EC2 available really changes the entire game as far as these things go, and the rapid growth of their service offering is continuing to change the way a lot of key websites around the world do things.

- This makes it really hard to keep up with everything that Amazon is offering!

- There really isn’t a good ‘medium memory’ sized instance; your next jump after 1.7GB is 7.5GB (at 4x the price). For some people that probably doesn’t matter, but it felt a bit frustrating to me.

Overall, our transition has (so far) gone as well as I could possibly have hoped. Here’s hoping it stays that way.

"When issues arose during the course of the project, it became clear that MSU's leadership did not adequately understand the technology and the steps necessary to complete the project," [Oracle] stated. "Instead of cooperating with Oracle and resolving issues through discussions and collaboration, MSU's project leadership, motivated by their own agenda and fearful of being blamed for delays, escalated manageable differences into major disputes."

"This is a textbook example of how to file a legal action against a vendor for failure to deliver," said analyst Ray Wang, CEO of Constellation Research, who reviewed the updated complaint on Wednesday.



MSU made some smart moves to protect itself, such as documenting all conversations and interactions with Oracle, and working out an escalation procedure in the event the project ran into problems. It also was wise to use real-life use cases for the demonstrations, Wang added.

Well, not quite yet, but next week.

After using Linux for a month, and being relatively okay with it in general, I have, in the end, decided to go back to Mac — not for any reason related to what I do at home, but simply because using wireless internet in our office on Linux is a gigantic pain in the ass. (Note that using wireless internet on Mac is only *slightly* less of a pain in the ass, but it’s at least usable.)

The various solutions for wireless in the office I work in are:

• WPA login with certificate stored on secure token — This one might, in theory, be possible to get working under Linux, but it’s not trivial, and not something that I have any knowledge for. Basically, using a Windows based UI, I was able to export a personal cert, which then gets stored on a third party token (where I can’t get the cert back out); I can then use this to authenticate to the wireless. This solution is the best in terms of latency, limited login pain, etc., and breaks less often than the other solutions we have. (Only about once every 2-3 weeks instead of once every 2-3 days.) Practically speaking, this option is Mac or Windows only — and even there, the Mac support is only in a very beta trial. (I may be the only one in the company with it.)
• Juniper SSL/VPN: There’s a juniper-networks provided SSL/VPN that requires a login through the browser, and is then able to start up a Java client. However, to use it on Linux requires some magic that my particular install doesn’t seem to have, and I haven’t heard particularly good things about its Linux support in general. This option only introduces 120ms of latency to local machines, so it is the best option of the VPN based options.
• Cisco VPN Client/vpnc: This is the solution that exists in a reasonable form on Linux. This is essentially no worse on Linux than it is on Mac, but it has serious problems if you’re actually moving around an office with limited wireless connectivity in some parts of it: if I move from one conference room to another and hop between Wireless APs, the Cisco VPN/vpnc connection will usually drop, and is not reconnect in any way. (Even worse, unless I’m actively looking at the screen and notice the OSD message, I usually don’t even notice.) This is somewhat exacerbated in Linux by overall somewhat poorer Wireless reception with the particular hardware that I have (”Intel Corporation Ultimate N WiFi Link 5300″ in a Dell Precision M2400). It’s possible (even plausible) that other Linux hardware could either get better reception for any number of reasons, or be better at managing transitions between wireless APs (which this model seems to try very hard not to do), but rather than experiment with a dozen different laptops, I’m falling back to what I know.
• IPSec VPN connection built into OS X: It would be nice if this actually worked as well as the vpnc connection, but this is actually even worse, in my experience, than the vpnc connection: It requires re-passwording every hour (which no other solution that exists seems to, so I assume it’s the client doing something different), doesn’t handle reconnects any better, etc.

Of these solutions #1 and #2 do not appear to work at all on Linux, and the #4 fallback isn’t available on Linux. Given how often these services fall over - as I said, some form of VPN probably falls over on an almost-daily basis - in order to have ‘working’ wireless, I really need to have the largest set of options available to me. Other than this, the only issues I’ve run into on Linux at all are some very minor hardware issues around power management, trackpad drivers, and the size of the laptop I currently have — all of which would likely be fixed by the upgrade to an X220 that I was considering before I decided to go back to Mac.

It will be a bit of a shame to switch back to Mac after being on Linux and actually being able to work locally for a while, but overall, I think I won’t mind it as much as I expected: it seems a lot more of my work is done on remote hosts as of late anyway, since a lot of the data I work with has grown in size by 2-3 orders of magnitude over the past year. Still, I really wish that I could have stuck it out — being one of ‘those people’ using a mac in our office just feels wrong.

Relying solely on support contracts and generic solutions is a good way to self-limit the agility and performance of any business. In short, more gurus equals less hand-wringing and stress all around.

The work was done entirely by MAV employees which made it much less expensive than if external contractors would have had to have been used. Overall it is estimated that as a result of using internal resources and a GPS/GLONASS-based approach the project was 16 times more efficient than a traditional survey. And the project generated a lot of pride among MAV employees who carried out the work because it was such a remarkable achievement from a data collection, management and quality perspective.

• have a predictable cost structure
  
  
• already understand your core business
  
  
• have a feel for the historical reasoning behind business processes
  
  

* Water Pump hacking: A water pump in Illinois was alleged to have been hacked and broken by Russian hackers over the past couple weeks by various news sources, including the BBC. The real story? The tech consultant who helps to maintain the pump was, at the time the pump broke, at a conference in Russia — so when the pump broke, he got a call to look into it, and logged into the control system from… you guessed it, Russia. The pump just burnt out, and tying the ‘attack’ to a Russian IP was just because that’s where the consultant happened to be at the time. (Via On The Media’s Cyber Warfare piece)

* In ‘not news’: Teens are generally more aware of and conscious of their privacy than adults, taking care to limit their postings, limit their friend groups, etc. (Via On The Media)

* Heard an interview with the founder of “Is Anyone Up”, probably the most scummy person I have ever had the ‘pleasure’ of listening to on the radio. “I can do whatever I want with photos people send me, because the law protects me”… and because I have no decency. (Is Anyone Up is a ‘porn/revenge’ site, where anyone can submit nude photos of people, and they’ll be posted along with a Facebook link.) I … yeah. This report made me angry enough to want to turn off the radio, because the guy being interviewed clearly had no positive intentions: “This keeps me in beer money and lets me keep throwing parties, why *wouldn’t* I embarrass people this way?” (Via Revenge Porn’s Latest Frontier)

Just a few clips I thought were interesting that I heard this weekend.

I use Flickr as my primary image hosting. I like the Flickr UI, I like their tagging, I like pretty much everything about it. However, after years of using Flickr, I got sort of tired of always seeing “Photos of You (2)” on Facebook: The fact that there was some indication that there were no pictures of me always kinda peeved me for no sane/logical reason.

So, a couple years back, I wrote some software that let me copy photos from Flickr to Facebook. Over the years, I had been relatively consistent in tagging my photos with the names of people who were in them, so I was able to map tagged photos on Flickr to people tags on Facebook. There were some limitations to this, of course: flickr tags are whole-photo tags, while Facebook tags are a specific spot in an image. To get around this, I just tag the middle of the photo.

Generally speaking, this works relatively well — it’s certainly not perfect, but it works well enough that I haven’t run into serious problems… until last night, when I uploaded a few pictures of Alicia to Facebook.. and realized that because of how the pictures were taken, every picture was centered on her breasts. It’s a bit weird to visit your daughter’s Facebook profile and find that the person posting suggestive pictures of her on Facebook is… you.

I quickly retagged the photos, and now know to be more careful when uploading pictures of females — with an automated selection of tagging, it’s easy to have… unintended consequences when uploading pictures.

Today, at cjb’s for Caturday Hacking: Uploading photos from Grendel’s last week, and pushing my flickr2facebook scripts for copying photosets from Flickr to Facebook to github so that people could theoretically look at it and possibly re-use it. Though I’m not really convinced that’s very likely, given that apparently I’m using a two-generations old facebook API to do the stuff it does.

So, today I was just pondering the fact that I’ve been out of high school for almost 10 years now, and wondering if the school I went to does a 10-year reunion. From a brief search, I stumbled into the Wikipedia page about my high school.

A couple things came to my attention:

- At one point in the history of Wikipedia, I remember actively having a discussion about whether the school was notable enough for its own Wikipedia page. (Specifically, the fact that it was in a couple of non-local news articles for the mold outbreak in 2001 was brought up in discussion). I have no idea where those discussions went — they’re no longer on the talk page — but it’s interesting that the school now has a reasonably extensive Wikipedia page.

- In 2009, apparently “During the fall of 2009, many students came down with Swine Flu and other flus after attending the homecoming dance. The Monday after homecoming weekend, 611 students were sick. By Tuesday, 972 had called in sick.” This is a 2100 person school: that means that almost 1 out of every 2 students came down with a flu of some kind. I can’t really even imagine this. (Admittedly, as Jess points out, ‘Although I can’t imagine 1/2 of all the kids getting the flu, I can imagine half of them wanting to stay home, get high and play video games.’ Though in the town I grew up in, ‘drunk’ seems more likely.)

Nothing really special here, it’s just sort of … a weird feeling, to remember arguing that this school shouldn’t even have a Wikipedia page, and now to see that it has one that’s pretty long.

So, my N9 wireless hotspot didn’t work for a long time; it just returned a ‘not allowed’ error. Apparently, this ‘feature’ is because the N9 ships to… well, wherever this one happened to come from… with a disabled adhoc mode. The fix, provided by this thread, is trivial: Enable developer mode, open the terminal, and type:

devel-su

echo 1 > /sys/devices/platform/wl1271/allow_adhoc

(Note: ‘echo 1>’ is not the same as “echo 1 >”; oops.)

The password for devel-su on the N9 is apparently ‘rootme’ — simple.

But that just gives me adhoc wifi, and the Acer Iconia… not so happy with that. And unfortunately, unlike the N9, rooting the Iconia is decidedly more difficult.

So, the fix to get working Ad-hoc wireless support on the Iconia is to install a wpa_supplicant from another device that *does* support it. But that requires root. And there is no application out there that provides root for the Iconia running 3.2 — the ‘fix’ is to downgrade to an earlier version, then upgrade to 3.2 by running some different firmware that provides root out of the box.

So, I’d have to downgrade, then install a new firmware — not supported by the manufacturer, and probably losing my netflix access in the process — then install a modified wpa_supplicant which might or might not work. All just to get it so that I can use the wifi provided by my phone on my tablet.

Of course, this isn’t as bad as the Windows Phone, where I can’t even *do* that — there is simply no way to get ad-hoc wireless support on the Windows Phone that I can find, hacked or not. (mmm, closed source.)

Why must everything be so hard?

So, after 7 years of being a mac laptop user, I’m strongly considering switching back to Linux.

On Sunday, the hard drive in my (18 month old) Macbook died. Nothing super-serious, but unlike in the past, replacing the hard drive isn’t a 3-screws operation, so rather than rushing out and getting a new drive, I just waited until work on Monday and grabbed myself a Linux machine.

In using Linux for the past week, there’s relatively little that I miss from OS X. (The biggest drawback may be lack of Netflix; curse my addiction to DRMed streaming media!)

Although I’m on a bit of a beast of a machine — a Dell Precision M2400, hardly the prettiest of Linux machines — I’ve found that overall, Linux has most of the things that I use on a daily basis from OS X — in many cases easier than it would be to get them via OS X.

All of my nitpicks come down to minor differences than any switch would have — for example, I miss being able to quickly change between *applications*, instead of just windows of an application; I feel a bit slower on Linux than I did on Mac still, and for some reason my several attempts to install Flash don’t seem to have actually worked. I am missing having Exchange MAPI support in a mail client; with our Nokia mail setup, this means I can’t get my mail from outside the firewall except via webmail (ewww).

But a lot of the things that were problems in the past, just haven’t been issues. Sound and Power seem to work just as well as on OS X. Since I already use VLC, not much difference there. I miss Safari a little bit — not for any particular reason, just in the ‘getting used to things’ sense — but a full screen Firefox… version-whatever-I-Have (apparently, after checking, the answer is ‘6.0′) doesn’t feel particularly different in any significant way.

After 7 years of being on mac, and now switching back to Linux for the past week — I haven’t yet found anything that feels like a huge win for mac, and I have found many small wins for Linux. I expect that in the next couple weeks I’ll make a final decision on whether I want to make the jump and get a long-term computer that’s Linux, but right now, I’m leaning strongly towards “yes.”

I’ll just have to keep my tablet handy for watching Netflix.

Dear Member of NDP,



I would really appreciate your participation in a study we’re currently conducting amongst members of the federal NDP.



I recognize that you’re busy, so this survey is very straightforward and can easily be completed online at your convenience, in about 15 minutes. Please complete the survey before Tuesday November 15th.



All information provided by respondents will be kept strictly confidential and used only for legitimate research purposes. Study sponsors will not have access to your name, address or phone number.



To begin the survey, simply click on the link below. If your email does not support hotlinks, copy and paste the link into your browser.



http://...omitted...



If you encounter any problems, please contact me at the e-mail address below.



Thank you in advance for participating in our survey.



Regards,



Agnes Klich

Project Management Team Leader

klich@logitgroup.com

Update: Just to leave no stone unturned, I asked the researcher who commissioned the study, but the answer is not illuminating:

The Logit Group is a Gold Seal Member Agency of the Marketing Research and Intelligence Associaton (MRIA), Canada's governing body for all market research firms. As such, we conform to all regulations related to privacy and confidentiality. In this instance, the organization that provided us with member lists required that the survey be completed in a confidential, or "blind" method (whereby the survey sponsor was not identified at the outset). In return, the responses of individual members (such as yourself) would not be attributed to you specifically when reported back (survey findings would only be provided in aggregate form).

Which language would you prefer to complete this survey in? Dans quelle langue préférez-vous répondre à ce sondage?

• English / Anglais
• French / Français

We are interviewing members of the NDP. Are you a member of this party?

• Yes
• No

How long have you been a member of the NDP?

• Less than 1 year
• 1 to 2 years
• 2 to 5 years
• 5 to 10 years
• 10 to 15 years
• 15 to 20 years
• Over 20 years

And in which province do you currently reside?

• Alberta
• British Columbia
• Manitoba
• New Brunswick
• Newfoundland & Labrador
• Nova Scotia
• Ontario
• Prince Edward Island
• Quebec
• Saskatchewan
• Yukon
• Northwest Territories
• Nunavut

Are you...

• Male
• Female

Were you active for the NDP in the last federal election in May, 2011? (As a candidate, campaign worker, fundraiser, etc.)

• Yes
• No

As you may know, a leadership campaign has been scheduled for March 2012. Each member of the NDP will be eligible to cast a vote for their choice of leader.

As far as you know today, are you likely to be voting in this leadership election?

• Yes
• No
• Not Sure / Depends

Thinking about the next federal election, do you think the NDP will win more seats, will win fewer seats or will win the same number of seats as it has currently?

• Win more seats
• Win fewer seats
• Win same number of seats
• Don't know

How confident are you that with a new leader, the NDP can defeat Stephen Harper in the next federal election and take over government?

• Very confident
• Somewhat confident
• Not very confident
• Not at all confident

How much do you think the NDP's chances of defeating Stephen Harper depends on who is selected as the new leader?

• Depends a great deal
• Depends somewhat
• Depends a little
• Does not depend at all

When you think about the federal NDP as a whole, where would you describe yourself in your own political thinking relative to the party?

• Far to the right
• Moderately to the right
• In the centre
• Moderately to the left
• Far to the left

Some people say that they would like to know where each of the leadership candidates stand and want them to spell out in detail the policy direction they want to take the party in.

Others say that they don't want the candidates to present detailed policy because the party members and caucus should have a say in the party's future direction after the leadership campaign.

Which of these two points of view best represents your own thinking?

• Want to know details
• Party members should decide
• Don't know / Not sure

Each political party has an establishment: a group of people who have been important to, and actively involved with, the party for some time either provincially, federally or both. What is your impression of the establishment of the NDP? Is it very favourable? Somewhat favourable? Somewhat unfavourable? Very unfavourable?

• Very favourable
• Somewhat favourable
• Somewhat unfavourable
• Very unfavourable

Do you consider yourself to be a member of the NDP establishment?

• Yes
• No
• Don't know

There has been some discussion about whether the NDP should at some point discuss a relationship or merger with the Liberal Party and the federal level in order to unite the centre left of the political spectrum. What would be your opinion of a move to discuss such a relationship or merger at some point in time? Would you be strongly in favour? Somewhat in favour? Somewhat against? Strongly against?

• Strongly in favour
• Somewhat in favour
• Somewhat against
• Strongly against

As you may know, one of the leadership candidates is proposing that wealthier Canadians should pay more in taxes. Would this make it more or less likely that you would support a candidate who proposed this, or would it make no difference in your support?

• More likely
• Less likely
• Makes no difference

One of the policy issues being debated these days is the creation of a carbon tax for individuals and corporations who use hydrocarbon fuels and the use of rewards for those who reduce their carbon usage. Would you be strongly in favour, somewhat in favour, somewhat against or strongly against the NDP supporting the introduction of a carbon tax?

• Strongly in favour
• Somewhat in favour
• Somewhat against
• Strongly against

Different people have been telling us that they are looking for various things in the next leader.

From your own personal point of view, how important is it that the leader you choose ... (READ EACH OF THE FOLLOWING STATEMENTS) ... ? Is it extremely important, very important, somewhat important, not very important or not important at all? (REPEAT FOR EACH STATEMENT)

• Be closest ideologically to yourself
• Be able to work with, and appeal to, the members of the party and the electorate in Quebec
• Be someone who will be able to put together the best possible campaign team
• Be someone who will provide the party with much the same direction as Jack Layton
• Be someone who understands the concerns of your province
• Be someone who will provide the party with a new direction
• Be able to hold the party together
• Be a currently elected MP
• Be impeccably bilingual in English and French
• Have experience in government
• Be someone who has been a long term NDP member
• Be someone who will be able to beat Stephen Harper in a debate
• Be someone who is able to appeal to those who have traditionally voted NDP
• Be someone who will have the support of unions
• Be supported by the establishment of the party
• Be someone who runs a positive leadership campaign and avoids attacking other leadership candidates
• Be able to win the next federal election
• Be someone with a broad base of support in caucus
• Be someone who is able to appeal to those who have traditionally not voted NDP

Now we would like to ask you to think about the candidates who have declared for the NDP leadership.

What is your impression of ... (READ EACH CANDIDATE'S NAME) ... as a leadership candidate? Is it very favourable? Somewhat favourable? Somewhat unfavourable? Or very unfavourable? (REPEAT FOR EACH CANDIDATE)

• Thomas Mulcair
• Robert Chisholm
• Brian Topp
• Paul Dewar
• Nicki Ashton
• Nathan Cullen
• Romeo Saganash
• Martin Singh
• Peggy Nash

Overall, based upon what you know about these individuals or have heard, where would you put each of these candidates on the political spectrum in relation to the Federal NDP? Far to the right? Somewhat to the right? In the centre? Somewhat to the left? Far to the left?

• Robert Chisholm
• Brian Topp
• Nathan Cullen
• Thomas Mulcair
• Paul Dewar
• Peggy Nash
• Martin Singh
• Nicki Ashton
• Romeo Saganash

Thinking about some of the leadership candidates, in your view, which of them ...

• Is most likely to win the leadership
• Will be most supported by the establishment of the party
• Is running the most positive leadership campaign and not attacking other leadership candidates
• Will best understands the concerns of your province
• Is best able to appeal to those who have traditionally voted NDP
• Will be able to put together the best possible campaign team
• Will be best able to work with, and appeal to, the members of the party and the electorate in Quebec
• Will be best able to beat Stephen Harper in a debate
• Will have the broadest base of support in caucus
• Will have the support of the unions
• Is best able to hold the party together
• Is someone who will best provide the party with much the same direction as Jack Layton
• Has been the longest term NDP member
• Is someone who will best provide the party with a new direction
• Is best able to win the next federal election
• Is best able to appeal to those who have traditionally not voted NDP
• Has most experience in government
• Is closest ideologically to yourself

If the leadership vote was being held today, which of the leadership candidates would you be voting for? (Please select one only.)

• Nicki Ashton
• Robert Chisholm
• Nathan Cullen
• Paul Dewar
• Thomas Mulcair
• Peggy Nash
• Romeo Saganash
• Martin Singh
• Brian Topp
• Undecided

How likely is it that you will change your mind and support another candidate before election day?

• Very Likely
• Somewhat likely
• Somewhat unlikely
• Very unlikely

Who would be your second choice for leader?

• Nicki Ashton
• Robert Chisholm
• Nathan Cullen
• Paul Dewar
• Thomas Mulcair
• Peggy Nash
• Romeo Saganash
• Martin Singh
• Brian Topp
• Undecided

Who would be your third choice for leader?

• Nicki Ashton
• Robert Chisholm
• Nathan Cullen
• Paul Dewar
• Thomas Mulcair
• Peggy Nash
• Romeo Saganash
• Martin Singh
• Brian Topp
• Undecided

Are there any candidates for leader whom you could never vote for? (Please select all that apply.)

• Nicki Ashton
• Robert Chisholm
• Nathan Cullen
• Paul Dewar
• Thomas Mulcair
• Peggy Nash
• Romeo Saganash
• Martin Singh
• Brian Topp
• None - Could vote for any of them

Which of the candidates do you think Jack Layton would have wanted to succeed him as leader?

• Nicki Ashton
• Robert Chisholm
• Nathan Cullen
• Paul Dewar
• Thomas Mulcair
• Peggy Nash
• Romeo Saganash
• Martin Singh
• Brian Topp
• Undecided

Finally, I want to ask you about some of the personalities in the NDP.

If the following personality supported a particular leadership candidate, would that make it more or less likely that you would support that leadership candidate, or would it make no difference as to whom you supported.

• Joy MacPhail
• Adrian Dix
• Lorne Nystrom
• Daryl Dexter
• Libby Davies
• Dwain Lingenfelter
• Greg Selinger
• Ed Broadbent
• Roy Romanow

I would like to ask your reactions to the following statements.

Please tell me whether you strongly agree, somewhat agree, somewhat disagree or strongly disagree with each of the following.

• The NDP needs to move away from trade unions to win government.
• Given how well the NDP did in Quebec in the past election, the next leader of the NDP should be from Quebec.
• The next leader of the NDP needs to address inequality as a top priority
• The NDP can win government without becoming Liberals by running as responsible, principled New Democrats
• I would never vote for a leadership candidate who is not already a Member of Parliament
• I would never vote for a leadership candidate who has close ties to the labour movement
• The NDP needs to move closer to the political centre in order to be elected to government
• It is important that the next NDP leader be a woman
• I would never vote for a leadership candidate who was not perfectly bilingual

Some people say that the next NDP leader must continue with a positive approach to politics in order to motivate progressive voters to defeat Conservatives.

Others say that the next NDP leader must take a more confrontational approach to Stephen Harper because he needs to be faced down in order to be defeated in the next election.

Which of these statements is closest to your own point of view?

• Take positive approach
• Take confrontational approach
• Don't know

Some people say that the federal NDP has been basically on the right track since Jack Layton was elect ed leader and it should continue with this proven approach under a new leader.

Others say that the NDP has been on the wrong track and won't win government unless the new leader makes fundamental changes to the workings of the party.

Which of these statements is closest to your own point of view?

• On right track - continue
• On wrong track - fundamental changes needed
• Don't know

And finally, a few questions about you...

Which of the following best describes your current marital status...?

• Single, never married
• Married / cohabiting
• Separated / divorced
• Widowed

Do you, or did you, work in the private or public sector?

• Private
• Public
• Neither

Are you currently, or have you ever been, a member of a union?

• Yes
• No

Are you a full-time student?

• Yes
• No

How would you describe the community in which you live?

• Urban
• Suburban
• Rural

While I know that most of us would classify ourselves as Canadians first, can you please tell me what your specific ethnic background is?

• White / Caucasian
• Black / African Canadian / Caribbean Canadian
• Hispanic / Latino
• Asian
• South Asian
• Middle Eastern
• Aboriginal / First Nation

Which of the following categories best reflects your total annual household income before taxes?

• Less than $25,000
• $25,000 to $50,000
• $51,000 to $75,000
• $76,000 to $99,999
• $100,000 plus
• Prefer not to say

Manager, Strategic Management

The Manager, Strategic Management is accountable for leading the development, maintenance and evaluation of corporate planning, performance management, benchmarking, risk management, and reporting programs. The position reports to the Chief Executive Officer of BC Assessment and works directly with the governing Board of Directors to facilitate strategic planning and risk management sessions. The position exercises considerable latitude and independence to oversee and develop a coordinated and consultative corporate plan, risk and performance management culture across BC Assessment. In this role, the position is expected to manage the corporate planning cycle to achieve a top to bottom linking of mandate and vision of operational business activities including the annual and year-over-year alignment of budgets, resource allocation, performance and risk management programs. This position leads a small team, including senior program analysts and a research officer.

Nearly 7 years ago, I first started hacking software on phones. This blog was started in part to track my interest in topics around mobile hacking, a desire that I’ve never given up on.

Amusingly enough, when I joined Nokia as an employee, I was able to actually find references to some of these early works as being important in the space of the Symbian Python developments. It was kinda cool to see my name in some PowerPoint presentation that had been put together 6 years before by someone I’d never met or even heard of.

But you’ll notice that since then, I’ve had a serious dropping off; there’s been a dearth of development in that space for a lot of years. The reason for this is simple: on the platform that I used the most for many years, development was a pain. Shipping applications to users was not possible, and as time went on, the number of people who even used the same OS or platform as I did was shrinking drastically, due to Nokia’s tiny marketshare in the US.

Up until two weeks ago, I had never owned a non-Nokia phone. Sure, when we were acquired by Nokia, I got an opportunity to use other phones — both to test out our software on them, and to compare how they succeeded and failed — but I never owned anything but a Nokia phone, from summer of 2003 when I got my first 3650, to April of 2010 when MetaCarta was acquired and I still had my N95 (with a 6600 in the middle).

That’s not true anymore, though.

I recently attended a Microsoft Windows Phone developer event, and I’m — almost 7 years later — hacking phones again. While there still isn’t anything quite as easy as Symbian Python — it’s really hard to beat something in a programming language I speak all the time, presenting a beautiful API to the widget sets that the platform makes available with almost no need for platform specific knowledge — the Windows Phone platform combines a strong toolkit, extremely broad set of documentation and examples, and another key thing that I never had in the Symbian world: the ability to get the applications that I write to users.

Nine days ago, just three days after getting Visual Studio installed in my Windows 7 VM, I was able to not only build, but deploy to my phone and upload to the Windows Phone Marketplace — where just a few days later, my app was available for all to download (well, at least everybody on Mango).

Windows Phone development is *fun*. The tools are helpful, intuitive, and a breeze to get started with. The documentation and support community is huge — building on top of one of the things Microsoft does best. And thanks to the developer event, I now own my first ever non-Nokia phone: an LG Quantum, won in a raffle among people who were able to build and demo an app in just a day at the Microsoft developer event.

While nothing can compete with Symbian Python for pure quick-hack ability — at least, nothing I’ve seen yet — the Windows Phone platform provides a great development ecosystem, with a strong community backing, and the ability to quickly and easily get together apps that do things. Once you do — not only to put them on your phone, but you can also get them in a store so everyone else can install them… and that’s just damn cool.

Many thanks to Joe Healy, and the whole MS Events team who were able to put together the Mango Boston event a couple weeks ago. It was pretty awesome to not only be able to learn about Windows Phone — but to be able to *do* Windows Phone development, and really get my hands dirty, and ship an app less than a week after I even got the dev tools installed on my system.

Should you launch at Launch? (Or TechCrunch Disrupt? Or Demo? They’re all pretty similar).

This year I launched two major new products at conferences: Careers 2.0 and Trello, and both times, it was totally worth it.

First, a little background. There are three popular conferences where you can launch new products: Launch, TechCrunch Disrupt, and Demo. They all work the same way:

• You apply. If you have a half-decent product that is genuinely new, you’re likely to get a spot. That said, hundreds of companies apply for these conferences with unbearably awful products, so there’s always a risk that you’ll get lost in the noise.
  
  
• If you get in, you will have a chance to give a demo on stage for exactly six minutes. There will be some celebrity judges who will give you a few sentences of honest feedback about your startup. (Here’s how our demo went down).
  
  
• Even if you don’t get a slot presenting, you may have a chance to set up at a little table in the conference area where you can show off your product to passers-by.
  
  
• The official promise is that you’ll get exposure to a lot of journalists and VCs, and this will launch your startup on the way to huge success. The truth is, well, complicated, but I’ll get into that in a minute.
  
  
• At the end there is a “winner.” For example at Disrupt the winner (chosen by a panel of utterly uncorruptable, gazillionaire judges) receives a check for $50,000. There are between 30 and 50 startups presenting at each conference, and the politics behind who “wins” are murky enough that you should basically assume that the chance of winning is zero. There’s always going to be a “Netflix for Cabbage” or a “Second Life for Facebook” that the judges fall in love with. So the benefits of winning, which is vanishingly unlikely, should never factor into your decision as to whether to go or not.

So, are these conferences worth it?

Let’s look, individually, at the two big promises of the conferences: exposure to VCs and exposure to the press.

Are VCs at these conferences? Absolutely. Does going to one of these conferences get you funded? It’s complicated.

• If you have a brilliant product, a great team, and you’re eminently fundable, but you don’t know any VCs yet, and you launch at one of these conferences, you will meet a bunch of VCs—even some top notch ones—and the conference may actually get you funded. At the last TechCrunch Disrupt, the finalist judging panel consisted of some of the best investors in Silicon Valley. If you made it to the finals, these folks now know who you are and what your product does, and if your company is fundable, they’ll all take your call.
  
  
• That said, if there’s some reason your product is not fundable, all the conferences in the world can’t help you. Yeah, you may have a chance to present to a bunch of unknown VCs wandering around looking for investment ideas, but most of them won’t actually invest in you and those that will may be more trouble than they’re worth.

I’ve been tossing around the word fundable without defining it. Every entrepreneur thinks their “Mint.com for Laundry Tickets” is the most fundable idea ever, and all VCs should be dying to invest, if they would only sit still for the brief 62 minute demo!

No. Technically, whether you’re fundable has to do with things like traction, the total size of the opportunity, the quality of the team, whether you build moats (?), and a bunch of other gibberish that VCs like to tell themselves in their heads so that they don’t think they’re just spinning bottles.

But it’s too hard for an entrepreneur to evaluate their own fundability. So here’s a working definition of fundable which is all that matters for you as an entrepreneur:

• If you’re knocking on VC’s doors and they all seem to be opening, you’re fundable. If you keep getting more meetings, more introductions, and good vibes, keep going. You’ll get funded.
  
  
• If you’re knocking on VC’s doors and they all seem to be closed, you’re not fundable. These days most VCs will just tell you why. If you can’t get a second meeting with anyone, just stop. You’re beautiful, you’re smart, and you’re going to change the world, but you happen to be non-fundable, so just stop. Either change the company or the product, or find a way to make your product popular and successful without investors. 

So, that said, if you don’t know any VCs and think you might have a fundable company, a conference like Launch or Disrupt will get you your first intros.

Now, on to the other promise: Press and publicity.

It is possible, nay, common, to launch at one of these conferences and get NO press whatsoever. Zero. Nada. At Disrupt you’re guaranteed at least one mention in TechCrunch, but you’ll soon discover that TechCrunch’s tech-industry insiders may not really be the audience you need.

Yes, there are a lot of journalists at these conferences. Disrupt probably had about 200. When we launched Trello this week, you know how much press we got?

Four stories.

And every one of those stories came because I knew the reporter and emailed them before we launched, and pre-briefed them on our product under embargo.

Yep. There was not a single reporter, from the 200 that were registered, at Disrupt who saw our presentation and said, “Oh cool, I’m going to write about that.”

You know why? Because there were dozens of companies launching in two days, and reporters usually file one or two stories a day, so they all focus on one or two companies they find interesting (and at this last conference, they mostly wanted to talk about Arringtongate).

That said, you can get exactly the burst of publicity you need from launching at one of these conferences, if you do it right. You have to:

• Prebrief friendly media (under embargo)
• Get the bloggers in your area to write about you
• Have a sensational demo that gets retweeted
• And do this all at exactly the same moment when it’s newsworthy.

We did all that and leveraged 6 minutes of fame into 130,000 eyeballs.

The thing entrepreneurs often forget about news media: It’s supposed to be news. They want new things. As a startup, you are only going to have two or three new things that happen, ever:

1. Launching your product
2. Raising money from a VC
3. Reporting insane traffic or revenue (optional)

That’s it. Those are your chances to get news. Under no circumstances can you expect to be covered because you take a walk in the woods with potential employees... you’re not Mark Zuckerberg. (Unless you are, in which case, Hi Mark!) You’re not getting font changes on the home page covered, unless you used to work for Mark Zuckerberg.

In short, you only have two or at most three chances to got coverage unless there’s Mark Zuckerberg involvement.

Well, wait, there’s one more way. If you are very lucky, you will have some famous people involved in your company, and some of them will have tawdry affairs with prostitutes that are captured on video. That will get you a fourth story. Otherwise, you’re not news. Get over it.

Also important: the news cycle is 12 hours, tops. If you call journalists the day after you release your product, it’s not news. They won’t care. You have to call them two days before you launch, tell them you’re going to launch in two days, and offer to pre-brief them, so that they can run their story when it’s actually newsworthy. The bottom line is that you have to get all your coverage within a period of a few hours which means you have to plan ahead and work hard. This is not the time for incrementalism. Don’t worry about DDOSing your own server. There’s no choice: you can’t spread out the newsworthiness of your launch.

Because there are so few opportunities for a startup to get press, you have to make the most out of each one. That’s why I am still a big believer in “the big launch” even though the Lean Startup ethic today is all about trickling things out to your users bit by bit and pivoting a million times.

Here’s the story of Trello. We wrote the first line of code last January. By the time we hit 700 lines of code, the product was useful, and we immediately started dogfooding it in-house. We probably could have brought it to market after three months. That would have been ever so lean. There was a strong temptation just to dump it on the world super-early and spend the next year iterating and improving.

We didn’t do that. We worked for nine months, and then launched.

I couldn’t stop thinking that you never have a second chance to make a first impression. We got 131,000 eyeballs on 9-month-old Trello when we launched, and it was AWESOME, so 22% of them signed up. If we had launched 3-month-old Trello, it would have been NOT SO AWESOME. Maybe even MEH. I don’t want 131,000 eyeballs on MEH.

Still, I do, firmly, believe that a completely new product has to go through what Steve Blank calls customer development to find “product/customer fit.” I.e., you have to get real people really using your product and you have to watch them and listen to them and make changes to make your product better, and you have to do this very, very early.

How did we reconcile this? Through the old fashioned method of a closed beta. We got a hundred of our best friends to use Trello and tell us what they thought while we iterated and polished and improved.

So the thing we launched, nine-month-old Trello, is really kind of slick. And we got a little initial bit of publicity for it, but then that publicity became massively viral. So those four news stories caused a few people to check out the product, and they liked it, because it was AWESOME NINE-MONTH-OLD TRELLO, and they wrote amazingly nice tweets. Thousands of amazingly nice tweets.

So, the story so far: if your product is really good, launching at one of these conferences is an incredible catalyst. If your product is “meh,” it won’t help.

But wait—there’s one important, bonus reason to launch at a conference, and it’s a good enough reason to do it even if you don’t need the publicity or the VC at all.

It’s all about your team.

When you launch at a conference, you have an incredible hard deadline. This deadline forces you to ship. It forces you to make decisions about what has to be in version 1.0. It's actually an incredible team-building exercise to work your butt off, together, for the weeks leading up to the conference.

The morale boost you’ll get will be incredible. After months of toiling away, the feeling you get from seeing real-world people actually start using your product is the best feeling you will ever get as a software programmer in your professional life. These are the great moments that make it all worthwhile. We *made* something. People used it. It matters.

It's like sex, with clothes on.

The members of our team who came out to San Francisco for Disrupt (including two summer interns who skipped a week of classes to join us) had a blast. It was the best week, ever. The members of the team who stayed back in the office, watching the conference piped in over the Internet, had a blast. It was the best week, ever.

Work has to matter.

The stuff we create can’t just be bits on a hard drive.

Brett, Daniel, Bobby, Justin, Ian, and Aaron built something with their bare hands that will be a part of how the future works.

One company that just launched at Disrupt is trying to fix medical bills. Another wants to bring fresh produce from farmers direct to households. Another company built the universal translator from Star Trek. Good software developers invent the future.

This is what matters: launching products, getting them in the hands of users, and hearing them get value out of it. That’s why we stay up late, ruin our wrists and our eyesight, and drive our families crazy. It’s all about shipping.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

иконописSo, on Sunday, we released the new version of OpenLayers, with updated mobile support. This included the ability to do dragging and panning and even editing of features on mobile devices, including Android and iOS.

Last week, I finished up a quick project, called olhttp. olhttp is a demo of how to use the OpenLayers HTTP protocol to create a simple, straightforward UI for editing features — but one that is easily customizable.

This afternoon, I decided to make an attempt to put these two things together — specifically, to make it possible to demo feature editing on a mobile device like my new tablet, saving the features and then being able to display them on my Nexus S or a laptop. So, I wanted a quick and easy deployment plan for a GeoDjango app — after my experience with DotCloud, I’ve come to the realization that hosting my own shit is for chumps (unless I really need some specific high-level SLAs for uptime, which I almost never do for personal projects).

Luckily, I happened to be at a table full of FOSS4G Django Hackers, so they were able to suggest DjangoZoom, which recently added support for GeoDjango/PostGIS. I was able to apply for an invite, and later that afternoon, I had one, and was able to start playing with it.

Now, at the time, I was in a talk, so the only thing I had with me was my tablet; I figured I’d set my account up, and see how it went. Turned out, the answer was “really well”.

The way DjangoZoom works by default is that you give it a Github repository URL, and it will automatically fetch the code for you, and set up a Django database, appserver, etc., deploying your application’s code to the DjangoZoom servers. What does this mean in practice? Well, for me, it meant that I was able to continue with the setup process for DjangoZoom — all the way up through actually getting a working application deployed, without even switching from the tablet to the laptop. I provided my Django app to the platform, and it worked right out of the box.

After getting my application deployed in just minutes, I then moved onto modifying my app to specifically target touch devices. This included modifying the UI to be more touch friendly — larger editing icons, for example, since the defaults are very difficult to hit on a tablet or phone screen, with 200 ppi. This work (in a new github repo for the time being, olhttp-django) complete, I now have a simple, easy to use tablet editing UI. It works on phones like iOS and Android, it works in all browsers on the desktop, and it provides an easy to use data-input mechanism — and I never had to touch an Apache config.

That’s what I call “success”.

Around the time of Fog Creek Software's ten year anniversary, I started thinking that if we want to keep our employees excited and motivated for another ten years, we were going to need some new things to work on. It occurred to me that we could easily afford to make four little two-person teams to launch four new products. That would give our developers more chances to move around from product to product when they got bored, which would make Fog Creek Software an even better place to work.

Each team, we decided, would be guided by the spirit of lean startups. They would ship early and often. They would listen to real-world customers instead of building things in an ivory tower. And they wouldn't be afraid to pivot endlessly until they made something that people wanted.

Next, we needed some business ideas. After ten years in management I still never knew what anyone was supposed to be working on. Once in a while I would walk around asking everyone what they were doing, and half the time, my reaction was "why the hell are you working on THAT?" So one of the teams started working on finding better ways to keep track of who was working on what. It had to be super simple and friction-free so that everyone would use it, but it had to be powerful, too.

We had an early idea called FIVE THINGS. Everybody would have a list of exactly five things that they were allowed to work on. The top two were things they were actively doing right now. The other three were things that they would do as soon as they finished the first two. But nobody was ever allowed to have SIX things assigned to them. If you have too many things on your to-do list, your motivation tends to sag.

Five Things wasn't the right idea, but it led us to the idea that became Trello. Pretty soon we had four programmers and two summer interns working on it. We started dogfooding the product when it was only 700 lines of code, and even in that super-simple form, we found it incredibly useful. By the end of the summer we realized we had a hit on our hands: an incredibly simple, easy-to-understand way for teams to collaborate online.

So without further ado, I'd like to introduce you to Fog Creek's newest product: Trello.

• Read more about what Trello does
• Sign up, it's awesome!
  
  

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Of course, in addition to the good — solid OS, and overall positive experience from all sides — there is some bad that comes with the owning of my new tablet.

Lacking Apps

• No Netflix. I said about 2 months ago that I’d run into my first real pain from DRM — not being able to move files from a broken Wii to a new Wii. This Netflix disaster is another one. Apparently there’s some software issues that prevent this, but the biggest reason that things seem to be stuck is simply because ensuring playback remains DRM-safe is harder than it should be. Netflix support exists on certain Android platforms, and there are a bunch of hacks out there to make it work on the tablet I have, but it’s not currently possible without rooting it.
• No Hulu. Same as above. Hulu just gives a “No streaming support for your device” message when loading it in the browser, and there’s no App for Android. I’ve really enjoyed the fact that over the past year, more and more of my video watching can be done completely legally, and the lack of support for the tablet is forcing me to back off of that position overnight, which is a shame. (I’m happy to watch adverts, let people track me, even give feedback on how much ads appeal to me — if I can actually get my content. Since I can’t…)
• No Facebook app? Not really sure what’s up with that, since it doesn’t seem like Facebook would be that hard, but there doesn’t seem to be an official Android Tablet app. I did grab a copy of “Friend Me”, which seems like it’s providing a reasonable replacement.
• No support for DivX video playback out of the box — but easily solved with a number of other video players on the device, it seems. (Sadly, my personal favorite, VLC, isn’t yet publicly available for Android.

Hardware/Tablet Nits

The Tablet doesn’t seem to support USB mass storage. It does support the “MTP”/Media Transfer Protocol, so I was able to use an app called “XNJB” to copy files over to it, but not being able to use the built in filesystem tools in the OS is pretty unfortunate.

It’s not possible to connect to ad-hoc wireless networks, like those created on mobile devices using software like Joikuspot. It seems like there are (again) some workarounds to this — replacing wpa_supplicant on a rooted device — but nothing that you can do out of the box. (I hate turning ‘toys’ into ‘work’, so I’m trying to avoid rooting if I can.)

No charging, not even trickle charging, over USB. I really think this is unacceptable: even if you can’t increase battery power over USB, there is no reason that you can’t prevent my battery from draining as fast if I’m plugged into USB.

I’m sure I’ll run into more; this is just what I’ve run into so far as I go through. Overall, I’m still 100% happy with the purchase; it is so far looking like it will definitely be worth it for me. But I also recognize that it’s not a perfect tool, and there are improvements I wish could be made.

(As an aside: does anyone have any recommendations for a good Flickr *browsing* application, both my photos and others, on Android Tablets? “Flickr Viewer HD” is … suboptimal in the way it navigates, and browsing contacts photos seems hard to impossible.)

(Cross-posted from my LiveJournal.)

Yesterday, I bought myself an Android-based tablet.

I’ve been considering this purchase for a long time. Tablets are one of those things that I have always been wary of, because I’m not as much of a media user as a tablet really targets, I think. However, after a recent flight to Berlin, I realized that I have at least one solid use case for a tablet, in the form of a media player on flights. With that in mind, I quickly moved onto others: showing photos I’d taken to family members, an easy way to get computerized video to the TV, etc.

In addition, it would be a chance to explore Android. I’ve played with a lot of other mobile platforms over the past 18 months, but not Android — and although it’s not quite a phone, the Android tablet experience is still an Android experience.

I’ve been biding my time for a while, while the marketplace kind of leveled out. Handily, my coworkers have done my research for me: over the past 3 months, 3 other people in the office have bought the same Tablet, so I joined the crowd yesterday, and bought an Acer Iconia Tab.

So far, I love it.

Using it makes me feel like I’m in Star Trek, or Minority Report. *whoosh* goes the desktop! *Whoosh* Slicing some fruit!

The application support is obviously much broader than WP7 or Symbian’s; more quick toys, more real tools, more access to third party services that make using the thing easier. The OS really feels clean and relatively solid — I’ve used Android on phones before, but not a recent version, and the tablet experience feels so much ‘cleaner’ than any of the phones I’ve used. In the browser, things like OpenLayers support dragging, pinch-zooming, drawing in the browser, which is awesome. (I should really see if I can dig up whether/when that is happening or not happening in the 2.x series… or maybe 4.x will just fix it for everyone.) The browser really feels nice and relatively solid. The desktop, and widgets, are both nifty and useful.

The Acer was what I picked because it was essentially the same as the Xoom and Galaxy Tab 10.1 in functionality, form factor, etc. — but it was $100 less. I still need to explore sleeve/case for it to protect it from my stupidity, but for now: I have a new shiny toy, and it is Nifty.

икониFor a long time, I’ve seen complaints that the OpenLayers API is ‘too complicated’ — that there are too many options, that it’s too difficult to configure, etc. I’ve said for a long time that although OpenLayers offers a lot of power, it’s also possible to make things simpler with a relatively thin layer on top of OpenLayers — that the power doesn’t necessary have to compete with simplicity, but that maintaining a simple wrapper at the OpenLayers level is probably not ideal, for a lot of reasons.

I’m obviously not a good target for making a simpler API — I don’t consider OpenLayers difficult to use, so it’s really hard for me to be the one to implement an ‘easier’ to use API! Thankfully, only 3.5 years after I started saying someone should write a simpler API, there appears to be a significant competitor to OpenLayers which is being praised for its ease of use: Leaflet, from Cloudmade.

So, after watching people continually praise the Leaflet API, I thought I’d see what it took to make OpenLayers as easy to use as Leaflet. I started with the Quick Start example, and worked my way down the features of the page until I got to the end.

The result is the ol-leaflet-wrapper quick-start — a substantially similar page, which looks the essentially the same as the Leaflet Quick Start guide (minus popups). Total code to implement: 99 lines.

Now, I’m not saying this is a full-fledged wrapper: in fact, anyone with half a brain would be able to observe that this particular example is missing one big feature, in the form of popups. (In fact, I think this kind of demonstration is one very easy way to make it clear where the OpenLayers *library* is deficient: things that are hard to wrap between one library and another are a clear target for API refactoring and cleanup.) Any deeper look would show that I’ve only minimally wrapped Leaflet, and that a more complete wrapper would still be a significant investment.

However, I do think that it’s possible to make the kinds of changes that people desire most from OpenLayers *without* completely reimplementing the library — and even without a refactoring of the (admittedly bulky) core library; instead, approaching a new API as something built on top of the existing is a way to rapidly prototype a new approach, and integration or improvements can move into the core library as needed.

And if people who like OpenLayers really want a simpler API like Leaflet — well, I’m happy to help develop the Leaflet API wrapper into something more real, to help direct what a suitably friendly API for working with OpenLayers might actually look like.

I guess a while back I fiddled around with "Hide User Agent" and changed the user agent of my Nokia N900 phone. This had the unintended consequence of blocking me from the store.ovi.com site - saying "incompatible device". And, of course, there weren't any options on the web page to override this mistaken identity.

read more

иконописПравославни икониToday, my 13-year old daughter was flying from Manchester to Greenville/Spartanburg, transferring in Dulles. She was flying without the unaccompanied minor service. Her connection time in Dulles was short (only 1hr15m), but should have been sufficient.

At 4:50PM, her flight status was updated, making it clear that her flight was going to be an hour late in arriving, arriving at 4:54PM. (Admittedly, I should have been keeping closer track and making some phone calls earlier, but I doubt it would have helped.) She pocket-dialed me at 5:00, so I knew she was at least somewhere with cell reception; unfortunately, calling her back didn’t get any response.

At 5:05, I called United, and asked if they could confirm her status — whether she made her flight. While navigating their phone menus, I observed that the flight she was supposed to take had taken off, 5 minutes early. I gave United her name + flight information, and they then proceeded to take 5+ minutes to tell me that she had left from Manchester earlier that day. (Thanks for telling me information I gave you.) They also told me her flight was late (thanks again), and that she was rebooked for the next morning at 8:20AM.

Now, for me, cue a bit of panic; I made the decision not to purchase unaccompanied minor service for her trip based on the idea that she could make her flights, and even if she didn’t, there was another flight later in the day she could pick up (still the case here: there were two more flights out of IAD to her destination 30 minutes later). This may have been a mistake, but that was a decision. Having United rebook her on a flight the next day is an entirely different beast, obviously.

I asked the agent whether they could confirm she had *not* made her flight: I was told that she could tell me that she had been rebooked the next morning, which they would not have done if she had made her flight, but she repeated that she could not tell me if she actually made her flight. (What?)

I called a couple other numbers, including the airport traveller assistance number, who told me that they could not do anything other than page in the baggage claim area; we punted on that for the time and kept trying United, still trying to get a confirmation that she had or had not made it on her flight.

In my next call, I got the same basic story: She’s rebooked for tomorrow. Since she is rebooked for tomorrow, United will be putting her up in a hotel tonight. No, I can’t tell you what hotel. No, I can’t tell you a number you can call for that information. No, I can’t tell you whether she made it on the flight. No, I can’t tell you hotels we use for putting passengers up. I can tell you that she was rebooked for tomorrow morning.

At this point, we had her paged in the airport — which ended up producing no response, though they were attempting to be very helpful. They did clarify that they can’t page in the gate area, and we should contact the airline for that.

Again, we called a different United number, only to be shunted back to the main United phone disaster. No, we can’t contact our gate agents because of 9/11. No we can’t page in the gate are. No, we can’t tell you what hotels we use. No, we can’t give you a number to find out where this passenger has gone.

Kristan and I at this point started calling hotels — this is one hour after she has landed in Dulles and we have been told she will be staying overnight in DC and that we could get no information from anyone. We made it through about 10… at which point we got a call from Alicia. She had not answered her cell phone (which she did have on her, and charged)… because she was on her plane.

She was fine, and on her plane, the whole time. United put her on the next flight, but rebooked her anyway. She’s not in IAD, and had no problems… other than our complete lack of ability to communicate with United at all.

The first response to this will naturally be: “You should have paid for unaccompanied minor service.” While this may be true, I’m not convinced that it would have helped: specifically, the problem was not that Alicia did not make her flight — she did. The problem is that *no one could tell us this* — despite the fact that, presumably, someone scanned her boarding pass. Since we did not have a phone number — and I looked for an ‘unaccompanied minor contact number’ on United’s site — and she made her flight on time, I doubt we would have been contacted by United. (In fact, unaccompanied minor service might well have meant the difference between her making her flight and not in this case!)

Overall, I’m not upset that United didn’t know where my daughter was: that’s my fault. I’m not really that upset about most of the question they couldn’t answer in this case — though if she *was* stuck overnight, I’d be damn upset. Instead, I’m only upset about one thing: United told me she missed her flight, rebooked her on a flight tomorrow morning, and was completely unable to tell us that *she actually made her flight*. That has to be seen as a failure somewhere in the system, in my opinion, and I think the complete lack of being able to get to anyone other than reservations via the phone on short notice is completely unreasonable.

I do want to thank people who helped reassure me on twitter and elsewhere, and the IAD Travelers Aid desk, who did everything within their limited power to help us out, and the most important thing is that Alicia is at her destination + safe.

[UPDATE - September 6th - Regrettably, DevDays had to be cancelled. See the announcement on the Stack Exchange Blog for details.]

Stack Overflow DevDays, the universe's best conference series for coders, is back, and it's bigger than ever!

Here's the idea behind DevDays. You're a developer. You'd love to learn all the latest hot new technologies. Things like DVCS, HTML 5, Node.js, CSS3, Hadoop, etc. The stuff the cool kids are all talking about on the playground while you're stuck in the basement somewhere grinding away on Java Enterprise Visual Basic.

The idea behind DevDays is a fast, high-bandwidth, fire hose tutorial on at least ten interesting concepts. We'll assume that you're a developer, you know what a loop is, but each tutorial starts at the ground level and gives you a whirlwind tour through a technology by showing you actual code. Every presenter launches an editor and writes code from scratch and shows you what it does. There are almost no prepared PowerPoint slides with ten bullet items each containing 10 words explaining the ROI benefits of some new technology. There are not even any PowerPoint slides with cats and pandas doing hilarious things, such as this one:

Yes, DevDays contains precisely NO funny pictures of cats. We might have Jon Skeet with a sock puppet, though:

(That was Jon Skeet and Tony the Pony from London DevDays 2009.)

What we have instead is some great presenters from the community who will write code and compile code and explain it all while you watch, and you'll come away knowing enough about each new technology to know what it's good for, what it's not so good at, how to do the basics, and how to learn more. Bottom line: it's the best possible way to spend two days and learn as much as you would learn in two years of reading Twitter.

We have FOUR, yes FOUR different DevDays conferences coming up this fall. Each one is its own production, and they're all going to be spectacular. If you came to DevDays last time, prepare to get blown away. This time everything is DOUBLE. Two days instead of one. Better food and coffee. Better locations. Bigger screens to make it easier to follow along. Lots of social activities. And, for the first time ever, we'll be visiting one city in Australia (shown at right), for an antipodean increase of infinity percent.

Anyway, registration is now open. The schedule is:

• October 12-13 San Francisco
• October 25-26 Sydney
• November 14-15 London
• December 15-16 Washington, DC

There are two! special! bonuses! you should know about before you choose a city:

1. In San Francisco, the day after the conference (October 14), Server Fault is holding a one-day High Scalability conference. You may want to go to both for a full three days of amazing amazingness... if you think your heart can handle the excitement.
2. In Washington, the day before the conference (December 14), we're have a big open source hackathon. The entire Stack Exchange dev team will be on hand and it'll be a lot of fun.

So, go, sign up now. You can save $100 using discount code JOELONSOFTWARE.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Their top reasons for going around IT? The need to respond quickly to changes in the market, self-sufficiency of their IT-savvy workforce, and the easy availability of top-quality it services that can be bought without long implementation or testing (cloud and SAAs apps, primarily).

I've been learning Autodesk's Revit Architecture 2012 along with my kids and we are really enjoying it. We've used the open source Blender project and Sketchup a lot in the past but wanted to stretch our wings a bit more seriously.

read more

икониOne of the interesting things that I’ve noticed in exploring various cloud providers is the limited amount of support for strong geo integration. This isn’t particularly surprising — Geo on the web is still a niche market, even if it is a niche market I tend to care about a lot. Here are some things that I think that might make people more eager to adopt cloud-y services for geo, specifically in the context of DotCloud, where I’ve been investing most of my time recently.

DotCloud has the idea of ’services’ — independent units which work together to form an overall operating picture. The idea behind these services is that they are simple to set up, and perform one particular task well. So rather than having a service which combines PostGIS, GeoDjango, and MapServer, you have three separate services: one for PostGIS, one for GeoDjango, and one for MapServer, or map rendering, and you connect between them. This way, if your application needs to scale at the frontend, you can easily simply deploy additional GeoDjango services; if you need more map rendering ‘oomph’, same deal. (Deploying additional service for databases won’t magically scale them, but you do have the ability to use the software’s internal scaling across your multiple instances.)

So, again, using DotCloud, there are three types of ’services’ that I think would be interesting to users who are working in Geo:

• PostGIS — Supporting PostGIS on most debian+debian-derivatives platforms is pretty simple; the tools are all easily installable via packages. (pgRouting is another step beyond — something which might actually have some benefit simply because it’s more difficult to get started with, so having it preconfigured could make a big difference in adoption.) Nothing really special needed here other than apt-get install postgis and some wrapper tools to make setting up PostGIS databases easy. There probably isn’t any reason why this couldn’t be included in the default Postgresql service type in DotCloud.
• GeoDjango — This one is even easier than PostGIS. GeoDjango already comes with the appropriate batteries included. The primary blocker on this is to get the GEOS, proj, and GDAL libraries installed on the Django service type. I don’t think there is any need for a separate service type for GeoDjango.
• Map Rendering — This one is a big one for a lot of people, and I’m not entirely sure the best way to work it within DotCloud. Map Rendering — taking a set of raster or vector data, and making it available as rendered tiles based on requests via WMS or other protocols — is one of the things that is not pursued as often by the community right now, and I think a lot of that is in the difficulty of setup. As data grows large, coping with it all on the client side becomes more difficult; some applications simply never get built because the jump from ’small’ to ‘big’ is too expensive.
  
  There are three different ‘big’ map rendering setups that I can think of that might be worth trying to support:
  • MapServer — MapServer is a tried and true map rendering tool. It primarily exists as a C program with a set of command line tools around it; it is usually run under CGI/FastCGI. Configuration exists as a set of text-based configuration files (mapfiles); rendering uses GDAL/OGR for most data interactions, and GD or AGG (plus more esoteric output formats) for output. MapServer is often combined with TileCache, for caching purposes; TileCache is based on Python.
  • GeoServer — GeoServer is a Java program, which runs inside a servlet container like Tomcat. Like MapServer, it supports a variety of input/output formats; configuration is typically via its internal web interface. Caching is built in (via geowebcache). I think GeoServer would probably run as is under the ‘java’ service type that exists on DotCloud, assuming the appropriate PostGIS support exists for the database side.
  • OSM data rendering — This one is a bit less solid. OpenStreetMap data rendering has a number of different rendering frontend environments, but the primary one that I think people would tend to set up these days is a stack of mod_tile (Apache-side frontend) talking to tirex (renderer backend) which calls out to/depends on Mapnik, the actual software which does tile rendering. Data comes from a PostGIS database — though in the case of OSM, even that requires some additional finagling, since getting a worldwide OSM dump is… pretty big. (It’s probably safe to set that point aside as a starting point, and concentrate instead on targeting localized OSM rendering deployments — solve the little problems first, scale up later.)

One thing that all of these tools have in common is that they really like having fast access to lots of disk for quickly reading and writing small files. I’m not sure what the right way to do that within the DotCloud setup is — I don’t see an obvious service type which is designed for this — so that might be another component in the overall picture. (Things like the redis service try to solve this problem I think, but since the tools primarily intend to write to disk as is, adopting them to support other ways of storing tile data persistently would require modifying the upstream libraries.)

I think that there is room to significantly simplify deployment of some components of geographic applications by centralizing installation in cloud-based services; the above sketches out some of the components that it might make sense to consider as a first pass. These components would let someone create a relatively complex client + server side geographic data application; exploring and expanding on these — especially the OSM data rendering component — could make deploying to the cloud easier than deploying locally, with the net effect of increased adoption of cloud-based services… and more geodata in the world to consume, which is something I’m always in favor of.

My previous attempts to make GDAL work in the cloud were sort of silly; compiling directly on the instance is sort of counter to the design of DotCloud, which lets you scale primarily by assuming that a ‘push’ to the server establishes all of the configuration you’ll need. (sshing into the machine is certainly possible, but it seems it is designed more for debugging than it is for active work.)

So, with a few suggestions from the folks in #dotcloud, I started exploring how to set up my dotcloud service in a bit more of a repeatable way — such that if I theoretically needed to scale, I could do it more easily. Rather than manually installing packages on the host, I moved all of that configuration into a ‘postinstall’ file — fenceposted so it runs only once.

After a bit of experimentation (and reporting a bug in GDAL), I was able to get a repeatable installation going; this means that I no longer have any manual steps in creating and setting up a new service doing OpenAerialMap thumbnailing; in fact, the entire process — right down to setting up redis as a distributed cache — can now be completed automatically.

The postinstall is pretty simple: download and configure, then install, curl; Download and configure, then install, GDAL. In both cases, use the respective -config tool as a fencepost so we don’t install multiple times.

Once this is done, setting up a new deployment with the appropriate services is trivial: that’s outlined in setup.sh — configure a redis service, grab the config information, and deploy a wsgi service using that config and the rest of the code.

In fact, while I was writing this post, I followed that very set of instructions, running ‘./setup.sh openaerialmap’ — before I finished the post, I had a running www.openaerialmap.dotcloud.com instance, which I could then commit in place of the silly-named ‘pepperoni’ service that oam.osgeo.org was using before.

The big thing for me about these one-off deployments is that by making available better tools for doing iterative deployment, they allow me to be more disciplined in how I configure services. Normally, the act of installing GDAL or curl is a one-time event: without reformatting my system (or setting up a VM and reinstalling/rolling back every time), it’s non-trivial to test what happens on a fresh system. With DotCloud, deploying another service literally takes seconds — and when I do, it’s a green field, ready for me to test my deployment scripts all over again. No more mistakes in one-off apt-get install commands that I’ll never remember if my system goes down. Now, I’ve got the ability to script deployments of software.

This is of course possible with EC2 directly as well — building custom AMIs, redeploying them, etc. For me, though, DotCloud took the pain out of having to do such a thing. They did the hard part for me — and I got to do the more interesting and fun parts.

One of the key components of the OpenAerialMap design that I have been working on since around FOSS4G of last year is to use distributed services rather than a single service — the goal being to identify a way to avoid a single point of failure, and instead allow the data and infrastructure to be shared among many different people and organizations. This is why OpenAerialMap does not provide hosting for imagery directly — instead, it expects that you will upload images to your own servers and provide the information about them to the OpenAerialMap repository.

One of the things that many people want from OpenAerialMap is (naturally) a way to see the images in the repository more easily. Since the images themselves are quite large — in some cases, hundreds of megabytes — it’s not as easy as just browsing a directory of files.

Since OAM does not host the imagery itself, there is actually nothing about this type of service that would be better served by hosting it centrally; the meta-information about the images is small, and easily fetched by HTTP, and the images themselves are already remote from the catalog.

As a result, when I saw people were interested in trying to get OpenAerialMap image thumbnails, I decided that I would write it as a separately hosted service. Thankfully, Mike Migurski had already made a branch with the ‘interesting’ parts of the problem solved: Python code using the GDAL library to fetch an overviewed version of the image and return it as a reasonably sized thumbnail. With that in mind, I thought that I would take this as an opportunity to explore setting this code up in the ‘cloud’.

A requirement for such a cloud deployment is that it needs to have support for GDAL: this severely limits the options available. In general, most of the ‘cloud hosting’ that is out there is to host your applications, using pre-installed libraries; very few of the sites out there would let you compile and run code in them, at least that I can find. However, I remembered that during my vacation week, I happened to sign up for an account with ‘DotCloud’, a service which is designed to help you “Assemble your stack from pre-configured and heavily tested components.” — instead of tying you to some specific setup, you’re able to build your own.

At first, I wasn’t convinced that I could do what I needed, but after reading about another migration, I realized that I could get access to the instance I was running on directly via SSH, and investigated more deeply.

I followed the instructions to create a Python deployment; this got me a simple instance running a webserver under uwsgi behind nginx. I was able to quickly SSH in, and with a bit of poking about, was able to compile and run curl and GDAL via the SSH command line into my local home:



$ wget http://curl.haxx.se/download/curl-7.21.6.tar.gz; tar -zvxf curl-7.21.6.tar.gz

$ wget http://download.osgeo.org/gdal/gdal-1.8.0.tar.gz; tar -zvxf gdal-1.8.0.tar.gz

$ cd curl-7.21.6.tar.gz

$ ./configure --prefix=$HOME --bindir=$HOME/env/bin

$ make

$ make install

$ cd ../gdal-1.8.0;

$ ./configure --prefix=$HOME --bindir=$HOME/env/bin --with-curl --without-libtool --with-python

$ make

$ make install

$ cd ..

$ LD_LIBRARY_PATH=lib/ python



So this got me a running python from which I could import the GDAL Python libraries; a great first step. Then I had to figure out how to make the webserver know about that LD_LIBRARY_PATH — a much more difficult step.

After some mucking about and poking in /etc/, I realized that the way that the uwsgi process was actually being run was under ’supervsisord’; basically, whenever I deployed, my Python process running under supervisord was restarted, and was running my Python code inside it. So then all I had to do was figure out how to tell the supervisor on the DotCloud instance to run with my environment variables.

I found the ‘/etc/supervisor/conf.d/uwsgi.conf’ which was actually running my uwsgi process; after some research, I found that DotCloud will let you append to the supervisor configuration by simply creating a ’supervisord.conf’ into your application install directory; it hooks this config at the end of all your other configs. So, I copied the contents of uwsgi.conf into my own supervisord.conf, dropped it in my app directory, and added: environment=LD_LIBRARY_PATH=”/home/dotcloud/lib” to the end of it.

Pushed my app to the server, and now, instead of “libgdal.so.1: cannot open shared object file: No such file or directory”, I was able to successfully import GDAL; I then easy_installed PIL, and with a little bit more work, I got the image thumbnailing code working on my DotCloud instance.

I’ve still got some more to do to clean this up — I’m not sure I’m actually using DotCloud ‘right’ in this sense — but I was able to get GDAL 1.8.0 in a Python server running on a DotCloud instance and use it to run a service, without having to install or screw with the running GDAL libraries on any of my production servers, so I’ll consider that a win.

�One of the things that sets me apart from other OpenLayers users and contributors at times is the strong belief that it is not the responsibility of OpenLayers to be everything for everyone. I’ve often been faced with a situation where a user will say “Well, OpenLayers doesn’t have this functionality: doesn’t that mean it should?”

Just this morning, Eric made a similar argument: In response to my saying that OpenLayers is not for everyone, h responded if OpenLayers isn’t for everyone we gotta do something to change the situation, I think.

I completely disagree with this statement, and I think that anyone who thought about it would also disagree. There are some things OpenLayers is not meant to do, which other mapping tools are in a better position to do.

• OpenLayers is not a 3D spinning globe — we will probably never have support for rendering 3D globes inside of OpenLayers itself.
• OpenLayers is the ideal tool for browsing 3D panoramas: we are likely not the best tool for doing something like Google Streetview.
• OpenLayers is not a full GIS application: We will likely never ship as part of OpenLayers itself a full UI for doing GIS editing against some server.
• OpenLayers should not have as one of its primary goals creating user interface elements for complex functionality.

These choices might seem obvious, but I bring them up specifically because they are things that I have seen people state OpenLayers should do, and I think it’s important for any project to identify goals and seek to solve those goals.

There are also other things which are less obvious that I think OpenLayers is unlikely to do, which other mapping APIs have chosen to do.

• OpenLayers should not abandon support for Internet Explorer: One of the moves I’ve seen recently is to create applications which abandon support for Internet Explorer, since supporting IE takes more effort than supporting other platforms. Although that is reasonable for many platforms, I think it would not be the right path for OpenLayers. Building a library which supports more platforms will have side effects: OpenLayers has IE support baked into some of its core functionality, like event handling, so it will always have some minimal impact on things like download size, and there are some things we may choose not to do because doing so would make supporting IE more difficult. Internet Explorer may be close to becoming a non-majority browser, but it’s still going to be very important to OpenLayers users for years to come.
• OpenLayers should not remove support for commercial layers, even if supporting these types of layers requires a more complex architecture: Supporting commercial layers is one of the key components of OpenLayers, and I think that it is important to continue to support using commercial layers, even though this does, at times, make the OpenLayers code more complex. This problem is thankfully getting somewhat better with newer APIs from Bing and Google for direct tile access, but there exist other APIs out there that aren’t as advanced, and continuing to support the types of APIs we need to make that happen is something I feel is central to OpenLayers.
• OpenLayers should not remove support for direct WMS access. One of the things that some other mapping APIs have chosen to do is to limit their target audience, and as a result, they do not worry about adding WMS access or other similar functionality for accessing data through means other than laying out X/Y/Z tiles on a map. Supporting WMS and other OGC web standards is something that takes some non-trivial portion of OpenLayers developer time. If we were to abandon anything other than support for OSM or XYZ style tiled layers and vector layers, we could certainly concentrate on a smaller API — but I don’t think that is something OpenLayers should do, even if it would mean a better overall API.
• OpenLayers should not remove support for fetching via various protocols, parsing via various formats, and choosing what to do with that data via various strategies: I have seen an argument that the more recent work with formats, strategies, and protocols is confusing to users, and should simply be removed in favor of letting that be handled at the application level. Most of the APIs OpenLayers is being compared to do not have this kind of support. This is another thing I strongly disagree with.
• OpenLayers should not remove support for dealing with data in projections other than Spherical Mercator. Many other libraries simplify user experience by picking and sticking with a single projection; I think that is impractical for OpenLayers.

Now, I think it is completely true that it would be possible to rip out 80% of the functionality of OpenLayers, create a smaller, easier to maintain library, and hit a use case that could solve interesting problems. I will agree that OpenLayers is a large codebase — after all, we have more than 200 *thousand* lines of code, compared to just under 20,000 in Polymaps, 60,000 in Mapstraction, 9,000 in Leaflet. This is the ‘big daddy’ of mapping frameworks — no one is denying that — and it has a lot of technical debt built up the same way any Open Source project has happen over 6 years of development without a rewrite.

Some of that code is cruft, and should be removed. No one is denying that. In fact, there’s a fair amount of already-deprecated code; controls that have been deprecated or non-default for more than 4 years are still part of the main OpenLayers trunk release. However, I’d put the amount that is cruft at closer to 20% than 80%: the much bigger portion of the OpenLayers code is the broad support for the many different ways of interacting with remote data. In our formats alone — things which are generally designed to do only one thing, read and write from an external string to an OpenLayers resource — we have 69 files with 20,000 lines of code. That’s right, our format parsing — for everything from OSM to GeoJSON, GeoRSS to ArcXML to CQL to KML — is larger than several other entire mapping libraries.

Eric followed up by pointing out that of course he didn’t mean to suggest OpenLayers shouldn’t do everything, but that OpenLayers should have an easier API to hit simple use cases — something which is better documented, easier to use, and less confusing to users. I find it a bit amusing that this is an argument someone feels they need to make to *me*, since I feel like I’ve been pushing that argument for years now within the project. However, since it may not have been clear, let me clarify:

Some areas where this is most apparent: difficulty in working with projected coordinates for the purposes of interacting with OSM or other spherical mercator layers. Difficulty in setting up a simple request to download a file from a server and render it — even *I* can’t configure that from memory, and I’ve done it dozens of times. These are real problems that users run into all the time and simplifying them would be a huge step forward in usability for solving the simple problems. (Note that I’m saying this in a blog post rather than in code, which also shows that I don’t think this is a trivial problem: the reason these things are not done is in part because coming up with a solid way to do these things in a helpful way is not trivial.)

I just want to clarify that there will always be some things OpenLayers shouldn’t do. Some of the decisions we’ve made have increased our overall technical debt: Maintaining support for IE, even for relatively advanced features like rotated client-side graphics in VML, was a cost that we could have saved if we chose a narrower supported platform range. However, I think that some of these decisions are important: a key component of OpenLayers is its broad support for loading data of any kind, in a wide variety of browsers. That was the core idea when OpenLayers was started, and I think that it is an extremely important to maintain part of our legacy.

Some of the competing mapping frameworks target a narrower use case. As a result, they can concentrate their developer time on better examples and documentation for a subset of functionality that OpenLayers has. This is great for the users of those tools, and will likely make those tools more attractive, short term, to some users. Competition is good: it encourages innovation, and pushes the limits, especially when the competition is open source and can collaborate across teams. If another tool is better for users than OpenLayers, it is in their best interest to use it. In the end, I hope that OpenLayers can continue to expand the set of users for which it is the best tool, and there is a lot of work that can be done there — and I look forward to continuing to see competition and collaboration between the various mapping projects out there to maximize user success.

Tom MacWright has responded to my previous post about OpenLayers; in the spirit of continuing the conversation, I’ll respond in kind. (Note that this is a conversation that feels disjointed, which is unfortunate; but I’m not sure of a better way to continue the conversation; oh for the days of Usenet, where I feel like this conversation could be so much more effective.)

First, Tom points out that I compared to the wrong mapping libraries. That’s unfortunate, I guess, but I was going entirely from the list that was presented; I don’t have any beef in the argument. Since I’m an OpenLayers user, I don’t end up doing a lot of research on new hotness, so I’ll admit I had a flaw there; I apologize. (In fact, I couldn’t even *find* Leaflet until it was pointed out to me; clear evidence of my flawed knowledge in this regard.) It was suggested that any comparison should include Leaflet, OpenLayers, and Modest Maps.

I reject that statement as simply incorrect. Leaflet, I’ll gladly accept into the ring — I’m especially happy to see that Cloudmade has open sourced their new efforts, since that means that there is the possibility to collaborate and contribute across the projects, which I think is a great step forward. However, Modest Maps is not a JavaScript web mapping library — it’s based on Flash, and requires developing using the Flash toolchain. While Modest Maps may be an interesting comparison for some small percentage of the OpenLayers userbase, I would argue that I am happy to have people whose needs could be solved by Modest Maps use that library, but it’s not an interesting comparison for the vast majority of OpenLayers usage. While it may be possible for OpenLayers to learn from Modest Maps, it’s not an interesting comparison for the users of the software as an API.

Some questions which are asked:

Where does control of the map come from? The baselayer? The map object? Both?

Ah, a softball! The layer configuration for resolutions, projections, and all other information related to it should come directly from the current baseLayer of the map. The information assigned to the map may occasionally offer a shortcut for that configuration, but it should not be in any case necessary to have that configuration on the map. (Any case where that is untrue — and I’m sure there are some! — is likely a bug.) For other aspects of the overall configuration, the map may be a home, but in almost all cases, configuration options on the map are just a shortcut (and likely a confusing one, because it compounds the possible configuration issues). When configuring options for projection information, the information comes from the layer; as a courtesy, we do our best to copy configuration from the map to the layers to make it easier (since most maps have the same properties on all layers.)

OpenLayers has its own events system … which is different than the $.proxy and other apis in jQuery or anything else (or in pure Javascript) that people are used to.

Well, in actuality, the OpenLayers event system is based almost entirely on the Prototype system. As we have moved on we have incorporated shortcuts very similar to the ExtJS event registration system. But really, we’re talking about one or two functions here, which are consistent throughout the OpenLayers library: on anything with an ‘events’ object, you can call “foo.events.register(’eventname’, null, function(evt) {});”. So while it may not be particularly familiar — since it’s based on a less commonly used syntax, perhaps — it’s not like this is the most complex part of the library. I’ll also point out that we’re talking about a library which is older than jQuery, and older than ExtJS — it’s hardly a sin to not follow conventions that were invented years after you started your project

OpenLayers has its own HTML/styling system, that’s just built-out enough to be terribly limiting to anyone who wants controls to look nice, and uses anywhere from 40% to 60% hardcoded styles.

A mistake (largely created by our early development) that we have been slowly paying back ever since we got some external developers. Tim Schaub has been pushing the use of CSS and stylesheets for new controls since 2.4. We added CSS-stylable zoom and pan panels in OpenLayers 2.7, almost 3 years ago. While there are still a limited number of controls that use non-CSS based styling — specifically, the Control.PanZoomBar is probably a big one that will bite people, and the LayerSwitcher — I hardly think that these flaws are something that are the biggest problem with the library.

Now, the LayerSwitcher is a point: it’s *very* difficult to style, and it’s a not a particularly attractive display… but the reason that we haven’t invested in it is precisely because it’s not in the spirit of OpenLayers to solve this problem for users. As an API — not a UI development tool — OpenLayers isn’t seeking to solve all the problems of user interaction. Instead, we have left things like a more clean layer switching interface to third parties like GeoExt — with reasonable success. We’ve seen people create alternative layerswitchers for mobile applications, as well as other similar alternatives, ranging all the way down to just a simple dropdown. For complex usage, these may be insufficient — but that’s where libraries like GeoExt pick up the slack. In fact, Tom makes my argument for me: “See Modest Maps’s controls: there aren’t any, because designers will make prettier ones.” So it’s okay to have *no* controls, but not okay to have *unattractive* controls… that argument seems a bit flawed.

In this vein, there is a followup later in the post:

The configuration of the theme requires a line of straight-imperative code that sets a global variable. What if you want a different imgpath for different maps? Shouldn’t this be a per-map setting?

For panels — that is, for controls developed in the past 3 years, rather than 4 or 5 years ago — styling is entirely via CSS. This includes the Editing toolbars, it includes the panpanel and zoompanel controls, etc. Since these controls are via CSS, you can use different configurations per map, simply by using CSS rules.

Comparing to ModestMaps — which Tom describes as having no controls at all — and Leaflet — which has only a Zoom control — OpenLayers doesn’t appear to be lacking in the ‘easily customizable controls’ department

Now, Tom describes this as part of a systemic problem: “an assumption that you already know the scope of the library and the parts that you can customize.” This is a criticism that I am totally willing to grant — and feel falls directly under the heading of poor documentation, especially prose documentation, which I feel is a significant lack in OpenLayers for most users.

The growth of OpenLayers is addressed by code compression, not by changing the code, ever. … the bloat of OpenLayers and its ever-more-complex API is the core problem that shows no signs of changing.

This is completely untrue. The OpenLayers project has made significant efforts in three different areas to reduce code size: Compression (via better tools like the Closure compiler), Better build tools (which simplified building build profiles), and reducing interdependencies in code to limit build size. In fact, a significant portion of the most recent release was an attempt to minimize builds — simplifying default controls, pulling various portions that people use most out of the larger components so they could be built separately, and creating alternative build profiles to help users get started with building smaller OpenLayers builds for their applications.

In addition to making the OpenLayers build itself smaller, we have concentrated a large amount of our recent work on improving performance (simplifying operations while dragging to improve responsiveness) and limiting download size (using more sensible defaults, providing more example build profiles, and doing things like shrinking our default images to save space there). Overall, a simplified OSM map designed for a mobile interface, using our documented example profiles, will save over a megabyte of space, due to improved defaults and documentation.

In addition, the description of the OpenLayers API as ‘ever more complex’ feels confusing to me. An OpenLayers 2.0 application works just as well with OpenLayers 2.11 — so the core API can’t have changed much. There is certainly new functionality in the API, but looking at other APIs — like the leaflet quick start example — I just don’t see that much difference in APIs between OpenLayers and its competition.

While Polymaps and Modest Maps are great drawing frameworks, OpenLayers is an abysmal one. Try to get individual tiles from a drawn map? No way. Interact with anything on the map like a DOM element? Nope. “Unbind” the map from the HTML element it “owns”?

This would be what I would consider contrary to the design of OpenLayers, yes I can’t think of *any* case where someone has come to the mailing list and asked a question which would be more easily answered if these things are easier — and I read a lot of mailing list posts.

New-style Javascript isn’t there in OpenLayers.

A perfectly reasonable criticism — OpenLayers is a child of the era from which it was born, and I think that if this is important to you, then I expect other tools will be more useful to you. That said, OpenLayers isn’t alone in this regard — Google, for example, follows much the same style of code, so far as I can tell, so at least we’re in good company. (Note that leaflet’s quick start reads very much like an OpenLayers example, so I’m not sure I see where this comparison is coming from.)

Recommending GeoExt/MapQuery is a good idea for people building RIAs, but it’s the opposite of that for most others. What people want (people, as in, me, and people building things on mapping frameworks) is a lower-level, simpler API, rather than a higher one.

I think that this is simply untrue, based on the types of questions I regularly see. I think that OpenLayers provides a reasonable API for most tasks at the low level — many of the complaints are indeed only about the higher level tasks (like the layerswitcher styling). Given the rapid growth of usage for GeoExt, I see a lot of basis in the argument that OpenLayers was insufficiently providing for that niche — while I don’t see widely used reasonable competition to OpenLayers as a low level API.

The ability to make slim builds of OpenLayers (something we’ve also done, with openlayers_slim[5]) hasn’t been trumpeted nearly to the level that it should have been.

I don’t really know what more we can do. It’s the first response to every question we’ve ever had on speeding up your application, it’s in our deploying documentation, it’s been the first performance increasing technique mentioned every time we’ve talked about OpenLayers.

Short of not providing a pre-built OpenLayers at all, I don’t know what more we can do to help developers understand how important this is.

I’m positive that OpenLayers is not the best tool for all problems. But I think it’s actually a pretty good tool for a lot of problems. And one of the key things that it does — that nothing else does — is provide support for layers that you don’t *get* anywhere else. OpenLayers is designed to make it possible to use Bing, Google, and OSM in the same map — and as far as I can tell, nothing else does that. I still don’t see a library which offers support for parsing OSM, KML, GeoJSON, and GML in Javascript — and in my experience 50%+ of OpenLayers users end up using either the format parsing code or the commercial layer code.

For the few users who will never need to use anything other than OSM and some markers drawn in Javascript — great. But comparing OpenLayers to these other frameworks for anything other than dropping pre-drawn tiles in a map is sort of a non-sequiter for real usage, no matter what the flaws of OpenLayers are.

So, apparently at WhereCampEU this weekend, there was some discussion of “Why people use web mapping libraries other than #openlayers”. (I can’t find any other references to this in Googling; someone who was there may be able to provide more context which might alter the content of this post.)

There are a number of good points about OpenLayers identified in the whiteboard snapshot; the ones listed there that I can read are:

• Documentation
• Default Look
• Viability
• Examples are not Good
• No explanation of the general architecture

(Throughout this post, I will be comparing OpenLayers to other mapping tools listed on the whiteboard snapshot; with the exception of “Leaflet”, which I can’t seem to find by googling things like “Leaflet” and “Leaflet web mapping”. Help is gladly accepted.)

Some of these things I would be interested in understanding more deeply — and some I’d like to take on trying to address.

Documentation and Examples Are Not Good

I think a large difference of opinion exists between different people in the OpenLayers using and developing community about how documentation should be managed. In general, in OpenLayers, we have taken the approach of saying that the best form of documentation is via examples that clearly demonstrate how to use a given set of functionality. (Note that in this case I say ‘we’, but I won’t claim this is a considered decision on behalf of the project. Instead, it is one that I believe fell into place in large part because of how we started documenting the project, and has not drastically changed.)

In OpenLayers, the examples are the first line of documentation. Always check the examples first; not doing so is doing yourself a disservice. This is not to say the examples are perfect; they are far from it. Especially as the project has grown, the examples are not an ideal way to understand all of the functionality of OpenLayers; as the library has grown, the complexity has grown correspondingly. However, I have often been told “Well, I looked at the API docs first and they weren’t clear about how I was supposed to use a specific feature” — and I think that particular approach is one that is doomed to fail with the documentation currently available in OpenLayers. The best advice I can give to beginners starting with OpenLayers is: start with the examples. Search for the thing you’re looking for — and if you can’t find what you’re looking for with the search term you’re using, tell the OpenLayers users list.

Now, as I’ve said, the examples haven’t kept up. OpenLayers has grown tremendously, and although we are relatively diligent about writing API documentation, prose documentation has never been high on our list, and prose documentation outside of the examples probably never will be. I think that this is a problem that really needs a solid champion outside of the current developer pool if any progress is going to be made; no one on the current development team is likely to be strongly in favor of making sweeping changes, simply because the choices are to continue to add features — things like improved performance, better mobile support, improved functionality for new features like geolocation APIs — or to work on documentation, and the current developers will tend towards the former.

That said, I think it’s fair to compare OpenLayers to some of the other mapping APIs listed on this point.

• Google Maps: Blows OpenLayers out of the water on API documentation. Google has done an excellent job of integrating reference documentation with prose documentation, and has literally hundreds of pages of API docs for their mapping APIs. Their examples are solid — though comparing this page to the OpenLayers equivalent seems to have a certain something lacking. There is, however, no doubt that Google has done great work on documenting their API, and that there are many simple problems for which Google is a better solution than OpenLayers for some users simply because it’s better documented.
• Polymaps: Again, a relatively well done set of documentation. Their API docs are much more descriptive than the equivalent in OpenLayers, and their examples like this one provide a simple, easy to read overview, example, and code in one page, and the example overview is quite nice, with screenshots. I think that this is well in line with what you would expect from Stamen. I will say that their API docs do lack information about what properties you are actually supposed to pass into functions; the map documentation for example, says that “map.center([x])
  
  Sets or gets the location of the map center. If the argument x is specified, it sets the new map center”… but doesn’t explain what x is supposed to be. So long as functions have clear examples, this probably isn’t a problem; it’s just something that I could see being a problem with more complex functions that don’t take obvious arguments.
• Web Maps Lite (I’m assuming this is the Cloudmade offering): This API offers a pretty reasonable set of documentation; their Overview is slightly more comprehensive than the OpenLayers equivilant which uses Naturaldocs. I would describe most of the brief API method descriptions as being similar in quality to those in OpenLayers; no obvious winner or loser there. The examples seem clear and well done for the few that exist; not particularly comprehensive or entirely obvious, but probably a win for most simple use cases over OpenLayers.
• Mapstraction: Documentation was a bit hard to find; I didn’t even see it linked from the homepage, but Googling turned up javadoc-like content; limited, but since Mapstraction is a pretty simple library, probably not insufficient. The examples are… brief, but workable, looking at the mapstraction sandbox. I’m not a huge fan of the sandbox form of examples personally, but for people who are, this probably is a perfectly workable set of docs — but I don’t think there’s an obvious win over OpenLayers here.
• Tile5: The Tile5 API documentation seems, to me, to be less informative and more confusing than OpenLayers. It seems hard to navigate, and even once you’re in it, it’s hard to understand exactly what you’re meant to do with things like T5.Marker objects. The ‘examples’/tutorials section has two sections, approximately equivalent to the OpenLayers introduction, but nothing further; it’s not clear that these examples offer any sort of view of a more comprehensive idea of how to start using the library. I may be missing something, but I feel like this is actually a pretty major lack in this particular project — and given that OpenLayers is coming from deep in the hole on this problem, that’s saying something

Overall, reviewing the various APIs shows one thing to me: I believe that the integrated API reference and prose documentation of Google Maps works best, and that it looks like the tools for generating Javascript documentation from code appear to have improved greatly since OpenLayers started. However, no project has an obvious win in all categories; the different approaches generally leave something to be desired. OpenLayers is probably one of the worst in this regard — with the exception that it does appear to have the broadest example support, providing at least limited documentation for a number of ready-built use cases. However, in prose documentation, most APIs do better, and in API docs, Naturaldocs seems like it may simply not be cutting it for many users, and switching to something else may be viable way to help increase developer engagement with the documentation.

General Architecture

Unfortunately, without being there, it’s hard for me to know what people want here; I’ve heard requests for a general architecture description before, but when I try to give them, people usually end up saying something like “No, that’s not what I meant, I wanted something… else.” I’d be interested to hear thoughts on what this means.

In general, the architecture of OpenLayers is: You pass an HTML element to OpenLayers, at which point, it owns that element. You inform OpenLayers of a set of x/y coordinates you would like to render data into, and it takes layers of content, and allows you to lay down multiple layers in whatever set of x/y coordinates you defined (via your base layer). In general, OpenLayers tries to provide comprehensive tools for interacting with the map — using browser gestures, clicks, etc. — with an API to implement more advanced functionality beyond that. Where possible, OpenLayers will make it as easy as practical to interface with a wide variety of data sources to provide the data you are able to browse, and make interacting with the map to increase the amount of data possible.

Beyond that, OpenLayers isn’t supposed to be much of anything; it’s an API and a tool for rendering map data, and little else. (Sometimes the problem is simply overly high expectations for OpenLayers because people have used it to do a lot of complex things!)

In searching out the various map APIs described above, I didn’t see anything obviously filling this role, which probably means that I don’t know what I’m looking for — but I’d be happy to learn, and explore how to solve this problem, because it doesn’t really sound that complex to solve.

Default Look and Feel

This one is interesting. In general, OpenLayers doesn’t *have* much of a look and feel; in fact, the default look and feel is generally limited to 7 icons which are easily replaceable by CSS. (One of the things we even have docs for!) That said, I’m willing to go head to head against most of the other APIs here for that issue:

• Google: Google has smaller controls than OpenLayers; these may be more attractive to some people. I’m willing to accept that some people prefer Google’s style to OpenLayers standard control styling.
• Tile5: When I look at Tile5’s example at simple map marker, the only ‘default style’ I see is a checkered grey and black background (easily implemented via CSS in any webpage).
• Polymaps: Seems to have a default +/- icon, but no others; the +/- look very similar to what we have in our mobile example, styled using entirely easily-modifiable CSS.
• Mapstraction: This uses the native control style for everything, it looks like, and turns controls off by default; since OpenLayers can have its controls turned off with one map option, this feels like they are the same in this regard.
• Cloudmade: basic example seems to have no default look at all; again, no difference.

Overall, it seems there is a preference for… no controls. Perhaps we should simply make that a better-documented OpenLayers mode of operation, and then people could complain about how their map didn’t have any default styling

Viability

Edit: After reading Volkler’s post on the topic, I realized that this point was *Usability*, not Viability. I’ll expand on that after this section, but I think that the Viability section here is an important concern, if you ignore the off-base ranty nature of it: OpenLayers is an extremely broadly used and widely supported tool, so using any other open source solution is risking joining a smaller community, which may have its own dangers.

This is the one I actually have the strongest feelings about. I think that if you are going to talk about problems with OpenLayers, viability is the one that has the least of a leg to stand on, on any API other than a commercially supported (non-free) one.

OpenLayers has:

• 2000 members of its users list, with dozens of messages a day.
• Code contributions from more than 40 users
• New releases multiple times a year, closing hundreds of bug reports
• A steadily growing list of features and enhancements

OpenLayers is the de facto web mapping library used by almost every open source geo related project. OpenLayers has a broad community, with broad support. There are two books introducing OpenLayers, there is an active community around OpenLayers, and earlier this year, more than a dozen organizations came together to sponsor an OpenLayers code sprint, funding 18 developers to stay in Switzerland for a week and work solely on supporting OpenLayers.

OpenLayers is a huge success story. The biggest evidence of that, in my mind, is simply the fact that no one talks about using OpenLayers anymore — they just do it. OpenLayers has become the de facto web client at FOSS4G; OpenLayers is used by tens of thousands of people every day around the world, from OpenStreetMap to the US Government, to the Portland public transit system, to consulting companies selling services in Australia.

I’m sure that there are many people who would claim that other APIs win on various aspects — and they’d be right. Documentation is a pain. Since OpenLayers caters to thousands of different use cases, it’s not as trivial to get started with. The library is complex, and there are many features that are confusing and complex to get started with — but overall, OpenLayers is a reasonably usable library for doing a relatively complex task that achieves in a way that I don’t think any of the open source competitors come close.

Someone questioning the viability of OpenLayers makes me wonder what they mean — because the project continues to make regular, large strides in new developments and features, and I can’t see how anyone could look carefully at the project in comparison to its competitors and think that it fails on that merit.

I’d be glad to here about this point, or any others, from people who were at the WhereCampEU session — perhaps I’m misunderstanding the complaint, for example — but I would say that to almost everyone looking to create a web map: OpenLayers is a strong starting point, if you’re a competent developer, upon which it is possible to build great applications.

Usability

It appears that “usability” here is a description of the API as it applies to developer usability. I think that this is an area where the main problem is “OpenLayers is a broadly used library supporting dozens, or hundreds, of different use cases, and is not a narrowly designed tool designed to solve a specific problem.” I’ve theorized for a long time that a lot of people (especially people who are likely to attend WhereCamp events) would be much happier with OpenLayers if I were to write a simple “OSM.Map” wrapper around it that picked sensible defaults for users of OSM maps.

The biggest thing that makes usability hard is targeting too many users. OpenLayers is trying to solve many hard problems — from integrating with OSM and parsing OSM data, to integrating with OGC catalog services, to interoperating with WFS. The most ’sensible defaults’ for one application are not the best ones for another, and with weak documentation, usability suffers.

Some ways to fix this include better docs, writing wrappers for specific purposes to simplify defaults for some users, etc. But overall, because OpenLayers is a library which supports many different use cases, it’s somewhat harder to get started with for all use cases. This is unfortunate, but not unexpected; I think it’s a necessary evil of having an API which targets such broad use cases.

Overall, many of the complaints about OpenLayers are reasonable: our documentation is weak, our default styling apparently doesn’t appeal to some people (I still like it, but I’m apparently in the minority), and it’s harder to get started than with some other tools. However, we’ve seen continued transitions of people from other tools to OpenLayers as they encounter harder problems — because as your problems get harder, OpenLayers becomes a more attractive solution to prevent you from having to reinvent the wheel.

�So, this week, I have learned a couple things:

1. Nobody in this house does the dishes but me.
2. My 10 year old daughter loves playing with the younger kids in the sandbox at the park… and the other parents seem to love her for it most of the time.
3. There are at least 7 different major POI data sources, and the type and quality of data varies from “enh” to unusable — but once you get them working, there is actually some interesting data contained in them.

I’m going to do a bit of an exploration of the various place APIs, how difficult they were to get started with, and some thoughts on when you might use each of them. The order listed here is the order in which I implemented them for a new toy that I’m playing with. In each of these things, my goal is to provide place listings in response to search queries from users. The way I am doing this is to take each place API search, and wrap up the title, lat, lon, and URL into a GeoJSON object.

FourSquare

The FourSquare Venue API is actually surprisingly simple to get started with. Although for user access, FourSquare depends on OAuth — a bit weird given that I’m required to type my username and password in directly on my phone for the official app — for venue search, you’re not required to go the full OAuth route, and can instead use the Venues Project, which makes searching for Venues easy. In addition, the documentation on the FourSquare venues project makes it clear that using the API is pretty open; you can do most of what you want to do (except copying Foursquare places wholesale into your own database), including copying and storing IDs, displaying information, etc.

The documentation and examples with the FourSquare API were easy to get started with. The process for getting a key was well linked from the docs, so getting there was easy. There was no need for complex OAuth loops, and the licensing is clear and obvious. Overall, the FourSquare Venues API made getting started with place search easy.

FourSquare Data

The FourSquare data, at least for the areas that I’m currently looking at, are pretty great. I have been able to find results for things that don’t exist in any other data using FourSquare. The primary problem I have with FourSquare is that it is somewhat too generous in attempting to find results — appropriate for searching on a mobile phone, but less for a web interface. (A search for “Cambridge, MA” from Europe will give you many results for ‘ma’, but nothing actually relevant to the town.) Excepting this — which applies to almost all of the APIs in this post — FourSquare provides an easy to use API with a large quantity of interesting data in the areas where I’ve looked at it so far.

Facebook

Facebook is all noise. While I appreciate the social aspects of what Facebook place results provide — they have the benefit that you can have all of the aspects of Facebook linked into them — Facebook places coverage is limited, and what little coverage there is tends to be absolutely filled with duplicates. (A local hospital has 4 duplicates in 3 surrounding towns, even though it only has one location.) The API is relatively easy to use — just fetch a temporary token, and you can then make additional calls back to the API — and Facebook provides a familiar interface for Facebook users, but overall, the quantity and quality of data tends to be low in a way that makes it difficult to use for any serious purpose. Additionally, using the API was made slightly more difficult because getting a human-readable link required an additional call to fetch more details about a place, which almost all of the other APIs did not require. However, it was possible to bunch all these requests in one ?ids= request, so it wasn’t particularly painful. Total number of HTTP calls to get the data I needed: 3.

Google Places

Google Places is a weird entry. First of all, unlike every other API, Google does not give you a link to the place, nor does it have a way of getting details about a batch of places. In order to get a link, you must make a place details call for each place you are interested in. This makes Google one of the slower APIs to get the information I needed for my UI.

Google has a relatively high level of data quality. Although it uses a narrower definition of places than something like FourSquare, it is much broader than something like CitySearch. It also has a smaller problem with duplicates and other data issues than many other providers with user-submitted data tend to.

Overall, Google has high quality data with low noise, for their definition of places. However, for maximum coverage — to support services like check-in — other options like FourSquare have Google beat, at least in the urban areas I’m looking at.

CitySearch

The CitySearch API is interesting; it is a deeper/richer set of data than many other place providers — because it appears to be centered around a strongly curated set of results. Many places have reviews, and additional details, that can be gathered through the APIs, and due to strong categorization efforts, it is a good choice to use to find things like cuisines, which the full text search also indexes.

From the point of view of how I’m using it, the results are reasonable, though it is sometimes not obvious exactly how things are being matched due to the depth of data being searched. Additionally, the coverage is centered around restaurants and the like; because adding places is not something that users are expected to do (the API is centered around having businesses submit, rather than users), the coverage for things like public transit stations, or other things that users would consider important, is lower than competing APIs.

Additionally, the terms of service make it clear that using the API is really designed around creating your own CitySearch clone; they describe how you have to create links to CitySearch, how many ads you have to display on your pages, and the like. From reading the terms, it’s not even clear if it’s allowed to display CitySearch content without also displaying at least 2 ads on the page — see section 3.5 of Usage Requirements. Overall, though there is a great depth of metadata — reviews, attributes, etc. — for places in the CitySearch API, from the perspective of someone looking for a comprehensive POI source, this is probably not the best place to start.

Yelp

Yelp falls under pretty much the same descriptions as CitySearch, minus the weird/confusing Usage Requirements and the lack of user-submitted content. Because the site is review driven, the search is also searching full text of reviews, which has a somewhat weird impact on the results you get. (Searching for “Green Line” will get you Green Line stops… but also places which are close to the green line, because people mention it in reviews.)

Yelp uses OAuth for their API, but there are solid examples of how to make calls in Python, which made getting started easy. Signing up was also easy.

The content in the Yelp API has more user generated content — which means, as usual, more coverage and more noise. It is also an API with a lot of depth — reviews, etc. — if that’s the type of thing you’re interested in.

GeoNames

GeoNames is a perfectly reasonable set of data, but POI coverage is low. For navigation style queries to addresses, towns, etc. GeoNames is fine; for POI coverage, you’ll probably need to look elsewhere. Additionally, the API is not the best documented, and getting set up with an account took longer than I wish it had; additionally, the API is not particularly full featured. There is no way to pass a location into the query, so results are always worldwide, with the expected results. Additionally, while developing, I found that the API was down for about an hour one evening. All of this is in line with the expectations that I’d have of a non-commercial service like this, but it makes using it in any kind of application somewhat less tenable.

OSM/Nominatim

OSM’s coverage is primarily towards address-like things, as is the case for GeoNames, but there is an interesting amount of POI data in it. The search API is relatively fully featured, and you have a benefit of any object you find in this way having a full edit history, and additional data/attributes available, via the OSM APIs. This also allows for easy correction of this data in the case of mistakes — something that doesn’t exist in any of the other APIs. However, overall the coverage for POIs is spotty, and this is not an appropriate source for someone looking to build on top of a comprehensive place database.

SimpleGeo

I almost didn’t bother with SimpleGeo; despite the use of GeoJSON (my preferred format) as transport format, for two reasons: 1. It didn’t offer any particularly exciting data. 2. Required OAuth, but … in a weird way without examples. In general, the documentation around the SimpleGeo *tried* really hard, but failed in too many important ways; without working a way to do browsing of example requests, it was just too much effort to integrate. However, I did eventually find the SimpleGeo Python client (after a lot of looking around), which solved that particular problem. (In general, I don’t like the idea of depending on a specific API client to do things like ‘Fetch JSON’, but this was easier than the alternative.)

SimpleGeo did offer the benefit of using GeoJSON natively — for this reason, SimpleGeo is the only API for which I have full data available. Unfortunately, due to the difficulty of getting started with their API, I almost didn’t bother to implement it.

When I did implement it, I found it lacking in about the way I’d expect. It looks like someone glommed together a bunch of free data with some user entered data, with about the expected result. (For example, “Harvard Square Eye Care” seems to have two copies of the location in Porter Square… and none of the one in Harvard Square.)

Factual

After reading a bit about Factual, I actually got far enough to try their API — setting up a token, etc. — and found that it does make getting set up with the API relatively easy. However, because their local data is split by country, there is no practical way to treat Factual as a worldwide POI data source via their APIs, which makes it impractical without a lot more work than most of these APIs require.

But Don’t take my word for it!

Realizing that everyone has different needs, and even different aspects of a project have different needs, I didn’t just explore these APIs: I built something with them. The result is Anymaps.

Anymaps will let you search through the aggregated result list of all of the place APIs described in this post. So whether you’re looking for Four Burgers in Cambridge, or Pune Central, or even the Taj Mahal, you can see the results from all the various place APIs that are out there — and decide for yourself which one has the right data for you.

And that’s how I spent my Spring Vacation.

The Stack Exchange network is already up to 51 sites on diverse topics, from math to cooking to science fiction. Each site is a community on its own, and each community has its own needs and values. Pouring a big fat algorithm in equal measures on top of 51 different groups of people does not always work the way you might hope it would work. Maybe that’s why the super-algorithm companies (like Google) tend to suck when they try to build social applications.

Our goal as a company is to incubate each of these 51 communities—to get them to critical mass. Critical mass is that magic moment when the community has enough activity that it grows by itself.

Building communities on the Internet is a new kind of profession. There are an awful lot of technology companies, founded by programmers, who think they are building communities on the Internet, but they’re really just building software and wondering why the community doesn’t magically show up.

Stack Exchange is trying really hard not to suck at building communities. I would say we’re earning a solid B so far, but we’re working really hard at learning... doing little experiments and getting early results. And one thing we noticed is that the pure, algorithmic approach can’t possibly work for different communities: you need a political/social approach. That is, you need smart human beings to use smart human judgment and cultivate each community individually.

Or, to use a metaphor that has been on my mind, you can’t use a robot to train a puppy. Every puppy is different.

With 51 communities and a new one opening almost every week, our small team of four community managers are doing a great job but they just don’t have the bandwidth to help cultivate every site. So we depend on the most active, enthusiastic users to promote their own communities and help them flourish. But these users are usually domain experts, not community organization experts.

So what I plan to do is build a team of super-evangelists here at Stack Exchange to serve as backup. Sort of like Lady Gaga’s backup dancers, but probably without as many muscles, they are not onstage to lead; they’re there to fill up the stage with more hotness than one person can provide.

This job will be sort of like being a community organizer at a non-profit. It combines elements of marketing, PR, and sales, but it’s really something different. I don’t expect that there are a lot of people out there who already kn0w how to do this well, so I’m going to train them, personally. Not that I know how to do this, but we’ll learn together. Every workday is going to start with a huddle at 9am and a plan for the day’s activities and an intensive six hours of work. Every workday is going to end with an hour of learning... reading Kawasaki and Godin and Ries and Trout, talking with invited experts, meeting with members of the community about what worked and what didn’t worked. Everyone who joins the program (and survives for a year) will come out with an almost supernatural ability to take a dead, lifeless site on the internet and make it into the hottest bar in town. That’s a skill worth learning for the 21st century.

If you or someone you know is enthusiastic, energetic, super-outgoing (a social connector), a great communicator (capable of sending 50 personal emails in an afternoon), with some training in psychology, political science, economics, philosophy, or the humanities in general, and you’re looking for an alternative to a dead-end mailroom job at a PR agency, this is a rare opportunity... please apply.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

I’m one step further along on the quest for the Holy Grail of kiosk displays. We use a lot of iPads here at the museum, most of them are either in Lab Shield brackets or built into cabinets. The whole idea behind building a kiosk is to make it hard for people to break into your device, so naturally the power and home buttons are behind lock and key.

The trouble is, there are a few quirks in the iPads. One is that after running for a few days, the display seems to lose its sensitivity to touch. The only way so far to deal with that seems to be to open the bracket, hit the power button to make it sleep, then hit it again to make it wake up. Then launch the app again and lock the whole thing back up. Not something the visitor services people want to do a lot. Nor I.

Sometimes visitors also break out of the kiosk app (we’re using iCab Mobile, an absolutely fantastic app, by the way). I have no idea how, but sometimes the iPad will be doing something completely different. One of ours now has a spiffy new home screen background thanks to someone who got in.

I’m very hopeful that my newly found method of remote booting/app launching will help at least take some of the pain out of the reset process.  Here’s how:

1. Jailbreak the iPad (only works on iPad 1′s for now)
2. Install OpenSSH (I installed the whole BigBoss Recommended Tools suite)
3. Install Activator
4. Determine the IP address of the iPad
5. ssh in as root, change the password right away!
6. Set up Activator to launch whatever app (e.g. iCabMobile) you want to have start up at boot time. Hook the app to the “Anywhere -> Power -> Connected” event.
7. If you’re still ssh’ed in from step 5, type ‘reboot<return>

Voilá. Your iPad will reboot, and if you’re crossing your fingers just right, the app will launch after it boots. It seems to do this pretty reliably unless you do it too many times in too short a time.

I’ve set up ssh keys for the iPad so that I can run a command like this from my desktop machine:

ssh -i .ssh/id_rsa_ipadkiosk root@192.168.1.52 reboot

Next up on my list is to build a web app that lets us reboot any iPad. Then we can carry around an iPad running the web app, and reboot troublesome exhibits with the swipe of a finger.

Update: I should note that for this to work, the iPad has to be plugged in to a power source. During the boot sequence the hardware must sense the power source and generate the same event that gets generated if you plug it in after it’s booted.

For all the benefits of KML, it is decidedly a step backwards for handling large vector datasets. Most KML clients, including the cannonical Google Earth application, experience debilitating slow-down when viewing a couple dozen MB of vector data - datasets that I could easily open on a Pentium 4 in ArcView 3.2 10 years ago!

The unfortunate reality is that optimizing the performance of KML datasets is conflated with the structure of the data and is thus the responsibility of the data publisher. The wisdom of combining styling, performance-related structure, organizational structure, geometry and attributes into a single file format may be questionable, but KML has become the defacto geographic markup language due to it’s other benefits.

Anyways, back to performance enhancements on big vector datasets… The concept of “regionation” is used by several KML software to improve performance. From the Google LatLong Blog:

You can think of Regionation as a hierarchical subdivision of points or tiles, which shows less detail from afar, and more detail as you zoom in to the globe. This dynamic loading creates clearer visualizations by minimizing clutter, while simultaneously speeding up the rendering process.

In most implementations, there is a generic strategy for determining this hierarchy based on attributes or geometry size (in the case of vectors) or by a tile system. Neither is ideal when you want to preserve the vector nature of the data, split it into small, easily-loadable files and determine it’s view based on the natural hierarchy that is built into the data structure.

Specifically I am thinking about watersheds here - the US Hydrologic Units. Hydrologic units are watershed boundaries that are organized in a nested hierarchy; higher levels contain smaller watersheds that are contained within a single watershed from a “parent” level. The unique identifiers (hydrologic unit codes or HUCs) are rather ingenious as well; Each level is represented by 2 digits and are concatenated to form a single identifier that can be used to determine it’s “parent”. For example:

Level 4 HUCs e.g. 17090011	Level 5 HUCs e.g. 1709001104	Level 6 HUCs e.g. 170900110403

Instead of fabricating a hierarchy of features, why not just use this natural hierarchy to structure the KML documents?

Or as KML markup:

    <placemark>
        <name>17090009</name>
        <styleurl>#HUC_8-default</styleurl>
        <polygon><outerboundaryis><linearring><coordinates>...
        </coordinates></linearring></outerboundaryis></polygon>
    </placemark>

    <networklink>
    <name>17090009_children</name>
    <region>
      <latlonaltbox>
        <west>-123.001645628</west>
        <south>44.8300083641</south>
        <east>-122.203351254</east>
        <north>45.298653051</north>
      </latlonaltbox>
      <lod>
        <minlodpixels>256</minlodpixels>
        <maxlodpixels>1600</maxlodpixels>
      </lod>
    </region>
    <link>
      <href>./17090009_children.kml</href>
      <viewrefreshmode>onRegion</viewrefreshmode>
    </link>
    </networklink>

The advantages to this design are that you don’t have to break the geometries up to fit into a square tiling pattern, data loads and renders in a logical pattern and there will always be 100 or less (usually far less) placemarks per file due to the design of the HUC data structure. File sizes stay low, network links load quickly and request/rendering occurs only when they come into view. For this example dataset totaling 300M of shapefiles, there are several hundred resulting kmz files without any repeated features and all less than ~ 150K each. In essence, it achieves optimal performance by its very design.

Here’s a video of it in action:

This was all done with a fairly “hackish” python script. I’ll continue to refine it as needed for this particular application but, at this time, it’s not intended to be a reusable tool - if you want to use it, be prepared to dig through the source code and get your hands dirty. The same concept could theoretically be applied to any spatially-hierarchical vector data (think geographic boundaries … country > state > county > city).

So, a friend asked me if there was a list of distances-to-the-sea for worldwide cities. I wasn’t aware of one, but thought it should be relatively quick to build one — then actually thought a bit more, and actually did it, and found out instead of the ‘a while’ estimate, it was actually closer to about 10 minutes. So I thought I’d write up what I did and get a critique on it.

This approach only works for the area of interest, Spain.

So, first I grabbed the world_borders file from http://www.mappinghacks.com/data/world_borders.zip — we’re looking for a gross estimate, so this is a reasonable dataset to use. I unzipped the data, and loaded it into postgres:

$ createdb world
$ psql -f $POSTGIS/postgis.sql world
# Crap, forgot the plpgsql
$ createlang plpgsql world
$ psql -f $POSTGIS/postgis.sql world
$ ogr2ogr -f PostgreSQL PG:dbname=world world_borders.shp

Then assembled a geometry of France, Spain, and Portugal, using ST_Union:

select ST_Union(wkb_geometry)
   FROM world_borders
   WHERE cntry_name IN ('Spain', 'France', 'Portugal');

Once I did that, I grabbed the point for Madrid, and calculated the distance, after converting that shape to a boundary:

select ST_Distance_Sphere(ST_GeomFromText('POINT(-3.683333 40.4)'),
    Boundary(ST_Union(wkb_geometry)))
    FROM world_borders
    WHERE cntry_name IN ('Spain', 'France', 'Portugal');

Which gave me the ballpark answer I was expecting, of 303.2km.

Dependencies: PostgreSQL, PostGIS, OGR installed on your machine (easy on a mac — just grab the KyngChaos libraries — and also easy on Debian derivatives).