ogoglio

Making models for Ogoglio spaces

I'm receiving a fair amount of interest from people who want to make spaces for the Ogoglio platform, especially since we checked in changes which make "baked in" shadows look good (like so).  On the wiki there's a helpful introduction to how Obj is used in Ogoglio spaces.

log4j ranting with some bonus apache and sun rants for good measure

There is really no excuse for log4j.  I've always said it was vastly too complex and somebody should be punished for it; today I almost wrote my own logging package it's behavior was so bad.

1) log4j does nothing when you don't have a proper configuration file.  This is unacceptable--throwing away log messages is never a good idea, at least they should go to the console or /var/log4-rejects or be burned into the machine's bios or something... 

2) log4j helpfully tells you what the name is of the logger that it couldn't figure out what to do with (as well as saying "please initialize log4j 'properly', gee thanks) but it doesn't tell you where it looked for the configuration file! This wouldn't be so bad if it was consistent at the way it fails, but it's not.  You would have thought, you blundering idiot, that if you put the configuration file in the CLASSPATH directly, things would work.  You'd be wrong.  It appears that through some number of levels of indirection the classloader gets changed (see stupid flexibility in #5, below) and thus the CLASSPATH in your shell gets ignored. Lovely.  So you must put the log4j.properties inside jar file that actually links against the logging cruft (see useless flexibility #4, below) to have it actually find the configuration file.  I suspect that if you had the apache gene on chromosome 12 you would have intuitively guessed that was the case.

3) Minor side rant: The fact that the minimal useful configuration file is several lines long is pretty sad.  You should be able to just put something like "BE SENSIBLE" in the file and have dump things to the terminal or send up flares from your machine or something that's likely to help a developer make forward progress.  Oh, and don't forget your specification of the logging format--you can't possibly expect a logging package to choose something that critical for you.

4) It would be bad enough if log4j was our only burden to bear, but of course the apache folks had to add extra layers of indirection between me and my beloved log4j package.  Three isn't a crowd here, let's add an arbitrary number of extra wheels! They have thoughtfully provided a meta-logging package (commons-logging) that allows multiple, true logging back-ends to work simultaneously (you getting this?).  This is in case you need to switch out your logging package for one of the many (I'm sure) vastly more powerful implementations that somebody is fiendishly working on right now. In his mom's basement.  Ugh.  Log4j is bad enough, but allowing extra flexibility is just going to insure that I can't get anything useful to happen without a lifetime supply of Tums.

5) I would be remiss if I didn't mention that the underlying culprit here was actually the uselessly flexible XML parsing meta-infrastructure...oh, and that's from sun, another bunch that never seemed to have anything better to do than make things that should be simple, complex. Naturally, no meta infrastructure that allows you to dynamically load or choose your XML parser implementation at run-time (gee, I do that a lot) would be complete without a dependency on the meta logging layer! So, the poor soul that just wants to parse a config file in XML (which was the case here, where I was using someone else's code) ends up with this huge set of dependencies on meta-layers and true implementations that he/she doesn't even control and possibly has never heard of. Great, so there are at least 2 extra moving parts here, just to parse a config file.  Sun, if you are listening and wondering why people are fleeing J2EE (and to some extent Java entirely), for Ruby on Rails this is the type of cruft that keeps people cursing your name.  Oh, and that boil on your left buttock, that's from me.

6) The XML parsing infrastructure (the sun provided one that the poor sod who wrote this code never even new about) had one, final trick up it's sleeve:  when it fails to parse a document correctly and is called from it's simple API (like 'parseDocument') it just exits. Now, overall, I'm in favor of programs not continuing to run when they can't run correctly--fail early, fail often, I always chant.  However, when you are swallowing any error messages that the poor user could have possibly used to figure out what's wrong (see #2) then you are probably going to surprise the user a great deal by having problems simply stop.  In other words, if you both fail to initialize the document and the logging infrastructure, you should either print something to the console before exiting or set the CPU afire to prevent the user from touching the machine and throwing it out the window.  It's the least you could do.

Why all this anger, you may ask?  I had edited the XML config file for a program that (inadvertently) used all this machinery and I had foolishly added a character to make a line like this:

<foo... /></foo>

Note the extra / I added causes the XML parse to fail and the rest, they say, is history. 5 hours of it, to be precise.

The Java Experience

One thing that's clear is that we need to improve the initial Ogoglio experience for people using Microsoft Windows.  On OS X this is a zero step process, since Apple cares about the idea that things should Just Work, but the fine people at Microsoft and Sun have taken their infighting to a new level of inconvenience for their users and the result is a confusing 17 step process to go from a vanilla XP installation to something which can render Java3D.  It only takes seven minutes or so if you just blindly click "Ok", "Run", "Accept" and "Install", but if you pause for a second to consider what's happening you'll end up drunk in the gutter with despair at the possibilities flushed down the toilet because two big corporations can't stop shooting at each other.

Read this to understand why we're seriously looking to Adobe Flash for accelerated 3D in the browser, like so.  We think it's possible to lower the number of steps to three or four if we invest in a custom installer and take on the costs of supporting that across the myriad of Windows flavors, but frankly I find that to be unacceptable.

First Microsoft shows you a lame dialog when a page tries to load Java:

If you click "More Info" (the default) you are taken to a page which suggests that you turn on Java, despite the fact that they could test and see that you don't have Java installed.  Thanks, Microsoft!

Assuming that you clicked "Ok" instead of "More Info", our page suggests that you either turn on or install Java (we can't test to see which you need from a web page):

So, if you click the "Install Java" link you're taken to the overly complex and self-aggrandizing java.com:

At least there's a Big Green Button, which if clicked shows you a scary IE Security Warning:

Assuming you're not scared away (though most people will be) and you hit "Install" you'll have to accept a EULA, sit while it downloads (3 minutes on DSL), then watch the dozens of completely freaking useless progress bars stripe across your screen.  Eventually you will be presented with an exclamation of their success:

When you hit "Finish" it doesn't take you back to what you were doing, you know, before you dedicated your life to installing Java.  Instead, it takes you to a verification page just in case they didn't actually succeed in installing the JRE:

If you hit the Big Green Button it chunks a bit before telling you that the installation worked, though of course sometimes it doesn't.

By now you've completely forgotten what it was that you were trying to do and so you pop 'round the pub to get a drink and talk about the good old days when you could fit the HTML spec in the palm of your hand and a luggable computer in the trunk of your car.

But let's pretend that you're really dedicated and you remember to return to the page from which all this pain originated.  Let's also assume that the authors of said page did everything in their power to test whether you have Java and Java3D and in fact their scripts noted that you don't have the latter so they presented you with a link directly to its installer (ignoring the god awful mess which is java.sun.com's Java3D page) which you then click:

You ignore the security warning because at this point you feel like you've already given Sun the opportunity to steal your facebook password and tell all your friends to buy thin clients, and you click "Download" and wait for it to do so for a couple of minutes.

(sidenote: At this point in the writing process I had to stop and drink martinis because it's just too, too much)

Then you'll see Yet Another Security Warning which will drive away %50 of everyone who has heard about viruses because the installer isn't even signed and so it could be from your neighborhood Spam King.

But pretend that you don't care whether random code runs on the same machine as your tax returns and you click "Run".  Except that it doesn't run without presenting you with an end user license agreement which in all likelihood has no legal basis whatsoever but companies like to include anyway:

Then you have to confirm that you actually want the install:

Then it proclaims its success and (assuming that you ignore the readme about graphics card flags which can be passed in the command line to turn on pixelation flabjulibation) you're returned to your regularly scheduled web page. 

Unfortunately, the browser still thinks there's no Java3D because there's no way in hell Java will notice a new Java3D install without restarting completely, so you shut down your browser and go read a book.

Power & Computing

I have been thinking quite a bit lately about the new relationship that we, the computing community),  have to power. I don't mean <I>power</I> like the ability to bomb third-world nations, but rather <I>power</I> like the grid that delivers electricity to your home and office. 

In this day an age, I don't think anyone really wants to generate their own power.  There is that really funny sportscenter ad with Lance Armstrong on a bike, but I think the closest most people want to get is solar panels on their roof and maybe occasionally checking the meter.  They sure don't want to be dealing with the distribution grid, failovers, safety, and spot auction pricing.

<object width="425" height="355"><param name="movie" value="http://www.youtube.com/v/2XHl-WrefNE&rel=1"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/2XHl-WrefNE&rel=1" type="application/x-shockwave-flash" wmode="transparent" width="425" height="355"></embed></object>

Why do people not want to generate power? Well, for one thing it's a big hassle to do safely and it scales well in terms of cost and transmission.  Admittedly, Thomas Edison wanted us to  use DC power and have a generating plant in every little community, but these days we'd rather have it done centrally (by professionals with the right training) and have it distributed to us.  We use metering to keep people from wasting and, in some cases, to make a small profit from the enterprise.   Can you imagine the computing world if we didn't have standard power generation and distribution? 

Intern Job Opportunity: Mill wheel slave at Transmutable networks! You and your friends can keep our servers running!

In manufacturing, there is the notion of a value-chain.  I'd argue that bending the strict definition is ok when thinking about how a customer gets an internet service delivered.  At the front end of that chain are people getting paid for generating electrical power and getting to your domain's servers; one would hope they get paid a bit more than the amount it cost to make and deliver the juice or that they at least feel satisfied by providing a public service.  On the other "end" of the internet service are the bit providers that actually route packets between your customers and your systems.  Again, these folks get paid for doing something that most people would rather not do themselves.  Slightly closer to the middle of this mental image of a value chain, but next to network pipe, might be other network services that one pays for like name service, content caching, etc.  Again, these are things best done by professionals with special tools and training, not the typical web developer.

There is, though, a big gap in the middle.  A typical web N.0 service hopes that customers come to it's web site and are willing to trade money (or something believed by somebody to equate to it) for whatever the service provides.  To my way of thinking, that is bottom line for building a web service.  The interesting thing is that, at least for most people, there are a bunch of software layers involved that add value that the developers are currently required to pay for either in cash or time.  Simple examples here are operating systems, database servers, web development tools, graphics editing tools, compilers, etc.  Sure, all of these are valuable and necessary, no doubt, but why can't we apply the 'I don't want to generate power' argument to all of them? Why are they not scaled up and delivered centrally, metered to make sure people get paid, and generally taking out of the view of the service developer?

We are part of the way there. Amazon's Elastic computing cloud ("EC2") is a good start.  (If you don't know you better ask somebody!)  EC2 allows you to rent computing resources of various sizes and pay for them by the minute.  When you want to do something that needs computing you fire up an instance of EC2 and tell it what "AMI" to boot.  An AMI is snapshot of an operating system on disk, so when you boot the image it starts up the OS and runs the usual initialization scripts.  The person who booted the instance can access the instance via SSH or whatever other services (such as the web) are desired.  Amazon provides you with tools to take an existing AMI and modify it, creating your own custom image. In simplest terms, your custom image boots up a web server, a database server, installs your application and away you go as a web service!

EC2 removes a bunch of the layers in the (or lack of a) value chain on the side near the electrical outlet.  True, there were co-location and hosting companies before, but amazon steps it up a level from just the physical part--maintaining power, cooling, monitoring--to include the delivery of operating systems and some built in software.  (This certainly is going to be a challenge for the licensing model of Vista since Amazon can make a great deal of free, open-source software available to developers with this model.)  As a very satisfied user and paying customer of EC2, I feel that I can criticize it though for not going far enough.

Amazon has two great features in addition to being able to rent computing by the minute.  One is the ability to have simple, long term, managed storage via Simple Storage Service (S3).  Why in the world do I want to mess with "value chain" of folks who try to provide backup services when amazon is already had to develop all the infrastructure to do this for their own use anyway?  I'd rather focus on my application, thank you.  S3, however, is probably one order of magnitude slower than the local disk on an instance, so this is a good place to store big things that you want to be sure can be accessed, but not for something that is hit often. (Of course Amazon EC2 AMIs are themselves stored in S3!) For something more like a database that you are going to use often, they have the Simple DB.  It's pretty much a quick hashtable that they manage for you. These pretty much cover the needs of most developers on the storage side since there is a fast, small one and a big, slow one. 

I guess you knew that given my introduction in terms of electric power and 19th century, I was going to take this all the way.  Why stop at these services and offer them as separate, non-integrated pieces? (There are a number of other cool services they offer that I didn't mention.) Trevor and I have been kicking around "the War Room" as the endgame of this process for the Java community that is developing web services.  (In fairness, the rails world has got a pretty good stab at this via RightScale.) Our idea for the War Room is that everything that isn't your application's functionality is handled like electrical power: somebody else does the nasty work and you pay for what you use.  This includes fixed infrastructure internet services like DNS and email, app servers like Tomcat, as well as simple mappings to service objects as Plain Old Java Objects (POJOs) that correspond to things you will need, such as the simple database or payment services.  Basically, you would just upload a java-based WAR file that meets our specification--quite different than the existing deeply complex specifications--and it gets installed in your domain as your application.  All the infrastructure is provided by the various amazon services and linked to java objects that you can use in your application.  You still have to supply the big idea, though!

Critical to the idea is that the WARs that the War Room supports are pretty simple and known to have a great many properties--for example that the web APIs are REST and do not have state associated with each user's web session.  (If you want the full flexibility of servlets and the like, use a generic AMI and feel free to do it yourself!) This allows the automatic use of load-balancing DNS services.  As the web service grows in terms of number of concurrent users, app servers (instances) can be added to the application server tier as needed.  This process, bringing up app servers loaded with your software, currently can be done pretty easily on the granularity of 4-5 minutes.  If higher response rates to user load are needed for your application, it would require "over provisioning" the service and could be done if needed.  Since you get billed only for what you use, you can choose if you want to over-provision or not.

I really think that the War Room could simplify web service development substantially by allowing a developer focus on the key problems to their service--their particular mojo--without any distractions.  It allows an easy way to meter and pay for all the other cruft the developer doesn't need to mess with and allows scalable, reliable systems to be built quickly.  If only I didn't have a day job already....