ogoglio

Power & Computing

I have been thinking quite a bit lately about the new relationship that we, the computing community),  have to power. I don't mean <I>power</I> like the ability to bomb third-world nations, but rather <I>power</I> like the grid that delivers electricity to your home and office. 

In this day an age, I don't think anyone really wants to generate their own power.  There is that really funny sportscenter ad with Lance Armstrong on a bike, but I think the closest most people want to get is solar panels on their roof and maybe occasionally checking the meter.  They sure don't want to be dealing with the distribution grid, failovers, safety, and spot auction pricing.

<object width="425" height="355"><param name="movie" value="http://www.youtube.com/v/2XHl-WrefNE&rel=1"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/2XHl-WrefNE&rel=1" type="application/x-shockwave-flash" wmode="transparent" width="425" height="355"></embed></object>

Why do people not want to generate power? Well, for one thing it's a big hassle to do safely and it scales well in terms of cost and transmission.  Admittedly, Thomas Edison wanted us to  use DC power and have a generating plant in every little community, but these days we'd rather have it done centrally (by professionals with the right training) and have it distributed to us.  We use metering to keep people from wasting and, in some cases, to make a small profit from the enterprise.   Can you imagine the computing world if we didn't have standard power generation and distribution? 

Intern Job Opportunity: Mill wheel slave at Transmutable networks! You and your friends can keep our servers running!

In manufacturing, there is the notion of a value-chain.  I'd argue that bending the strict definition is ok when thinking about how a customer gets an internet service delivered.  At the front end of that chain are people getting paid for generating electrical power and getting to your domain's servers; one would hope they get paid a bit more than the amount it cost to make and deliver the juice or that they at least feel satisfied by providing a public service.  On the other "end" of the internet service are the bit providers that actually route packets between your customers and your systems.  Again, these folks get paid for doing something that most people would rather not do themselves.  Slightly closer to the middle of this mental image of a value chain, but next to network pipe, might be other network services that one pays for like name service, content caching, etc.  Again, these are things best done by professionals with special tools and training, not the typical web developer.

There is, though, a big gap in the middle.  A typical web N.0 service hopes that customers come to it's web site and are willing to trade money (or something believed by somebody to equate to it) for whatever the service provides.  To my way of thinking, that is bottom line for building a web service.  The interesting thing is that, at least for most people, there are a bunch of software layers involved that add value that the developers are currently required to pay for either in cash or time.  Simple examples here are operating systems, database servers, web development tools, graphics editing tools, compilers, etc.  Sure, all of these are valuable and necessary, no doubt, but why can't we apply the 'I don't want to generate power' argument to all of them? Why are they not scaled up and delivered centrally, metered to make sure people get paid, and generally taking out of the view of the service developer?

We are part of the way there. Amazon's Elastic computing cloud ("EC2") is a good start.  (If you don't know you better ask somebody!)  EC2 allows you to rent computing resources of various sizes and pay for them by the minute.  When you want to do something that needs computing you fire up an instance of EC2 and tell it what "AMI" to boot.  An AMI is snapshot of an operating system on disk, so when you boot the image it starts up the OS and runs the usual initialization scripts.  The person who booted the instance can access the instance via SSH or whatever other services (such as the web) are desired.  Amazon provides you with tools to take an existing AMI and modify it, creating your own custom image. In simplest terms, your custom image boots up a web server, a database server, installs your application and away you go as a web service!

EC2 removes a bunch of the layers in the (or lack of a) value chain on the side near the electrical outlet.  True, there were co-location and hosting companies before, but amazon steps it up a level from just the physical part--maintaining power, cooling, monitoring--to include the delivery of operating systems and some built in software.  (This certainly is going to be a challenge for the licensing model of Vista since Amazon can make a great deal of free, open-source software available to developers with this model.)  As a very satisfied user and paying customer of EC2, I feel that I can criticize it though for not going far enough.

Amazon has two great features in addition to being able to rent computing by the minute.  One is the ability to have simple, long term, managed storage via Simple Storage Service (S3).  Why in the world do I want to mess with "value chain" of folks who try to provide backup services when amazon is already had to develop all the infrastructure to do this for their own use anyway?  I'd rather focus on my application, thank you.  S3, however, is probably one order of magnitude slower than the local disk on an instance, so this is a good place to store big things that you want to be sure can be accessed, but not for something that is hit often. (Of course Amazon EC2 AMIs are themselves stored in S3!) For something more like a database that you are going to use often, they have the Simple DB.  It's pretty much a quick hashtable that they manage for you. These pretty much cover the needs of most developers on the storage side since there is a fast, small one and a big, slow one. 

I guess you knew that given my introduction in terms of electric power and 19th century, I was going to take this all the way.  Why stop at these services and offer them as separate, non-integrated pieces? (There are a number of other cool services they offer that I didn't mention.) Trevor and I have been kicking around "the War Room" as the endgame of this process for the Java community that is developing web services.  (In fairness, the rails world has got a pretty good stab at this via RightScale.) Our idea for the War Room is that everything that isn't your application's functionality is handled like electrical power: somebody else does the nasty work and you pay for what you use.  This includes fixed infrastructure internet services like DNS and email, app servers like Tomcat, as well as simple mappings to service objects as Plain Old Java Objects (POJOs) that correspond to things you will need, such as the simple database or payment services.  Basically, you would just upload a java-based WAR file that meets our specification--quite different than the existing deeply complex specifications--and it gets installed in your domain as your application.  All the infrastructure is provided by the various amazon services and linked to java objects that you can use in your application.  You still have to supply the big idea, though!

Critical to the idea is that the WARs that the War Room supports are pretty simple and known to have a great many properties--for example that the web APIs are REST and do not have state associated with each user's web session.  (If you want the full flexibility of servlets and the like, use a generic AMI and feel free to do it yourself!) This allows the automatic use of load-balancing DNS services.  As the web service grows in terms of number of concurrent users, app servers (instances) can be added to the application server tier as needed.  This process, bringing up app servers loaded with your software, currently can be done pretty easily on the granularity of 4-5 minutes.  If higher response rates to user load are needed for your application, it would require "over provisioning" the service and could be done if needed.  Since you get billed only for what you use, you can choose if you want to over-provision or not.

I really think that the War Room could simplify web service development substantially by allowing a developer focus on the key problems to their service--their particular mojo--without any distractions.  It allows an easy way to meter and pay for all the other cruft the developer doesn't need to mess with and allows scalable, reliable systems to be built quickly.  If only I didn't have a day job already....

Our Stack

Trevor and I were talking about the "stack" that we use to build ogoglio and transmutable applications.  His mastery of OmniGraffle should impress everyone.

PowerDNS [snippet]

Working on getting round-robin front-end load balancing in our test domain via DNS.  Also improved the scripts that launch webapp servers so that they can be more automated. More work on unified logging.

Ogoglio on Parallels

My main development machine is a MacBook Pro so to maintain support for Internet Explorer I rebooted my machine into XP, updated the development environment, built, and then tested.  This would take about 45 minutes so I often wouldn't do it before checking in new features of Ogoglio and inevitably I wouldn't take into account some quirk of Windows and IE support then would break.

This last week I decided to stomp out all of the remaining IE bugs in the Ogoglio viewer and I started by booting XP in Parallels Desktop and fixing all of the browser test pages.  Much to my surprise, all of the new browser tests pass, including the Java3D initialization, but when I tried to open the ogoglio viewer it failed.

To make a long story short, I poked around and found that we could make Java3D boot on machines with low-end graphics cards which can't antialias or double-buffer.  Now Parallels renders Ogoglio (badly) and I can debug WinIE against my OS X development kit.  It isn't pretty, but it beats rebooting.

Comet for Everyone

Though I have an active hand in platform building, my central role on the Ogoglio project is to keep my eye on our long term goals and to understand the culture and infrastructure in which we deliver on those goals.  In that role I have the occasional pleasure of pointing to someone's work and talking about cultural ramifications.

The other day Ian Smith posted a technical demonstration of Comet, which is a low level technology for communication between your web browser and an Ogoglio server.  What he didn't mention is why we're using Comet in the Ogoglio platform and what it buys us.

A little background: When you load your average web page there are a lot of connections made to the remote servers using a common language called HTTP.  If there's one thing the Internet can pass quickly and cheaply, it's HTTP.  More importantly, every IT department in every organization makes very sure that their networks and firewalls allow HTTP traffic.  Without HTTP there is no web.

The problem with using HTTP for Ogoglio spaces is that it wasn't originally built to send events like "Body started moving North" and "Door slammed" which need to move quickly so people don't see odd, jerky motion and which need to move in both directions between the client and server.  For this reason no MMO or social world uses HTTP for world events and instead they use hand-tuned systems.  If you've ever tried to play an MMO from inside a corporate network the chances are that it didn't work unless someone in the IT department was also a player and secretly opened the firewall to non-HTTP traffic, which they're loathe to do for security reasons.

Enter Comet.  Enough web developers needed our flavor of bi-directional, fast event communication that they've worked out a way to juggle the details of HTTP a bit to use them in a new way.  It's a bit like putting serious English on a pool ball, because Comet does things with HTTP which are a bit unexpected but which stay within the rules of the game.

Ogoglio spaces now use HTTP for all communication, which means that they're going to work where other spaces fail and they're going to be delivered without asking your IT department to make your network less secure.  This is directly in line with our goal of enabling diverse communities of people to collaborate in online 3D spaces because diversity doesn't stop at the firewall.

Comet Protocol

O Comet, Where Art Thou?

I have been working to move the ogoglio server and viewer to the comet protocol.  The advantages of comet seem pretty clear, it's a standard, asynchronous and pretty fast protocol built on top of (or perhaps underneath, depending on how you think about it) HTTP. When we I went googling to try to find some simple, complete comet programs with a client and server that did something, there were surprisingly few and basically none written solely in Java.

So, I've written one for the community with my own commentary about what I've learned about comet.  You can download the client-source and server-source, or if you just want to run the code, the og-ex (for the server) and the client-jar.

How to install it

If you are just running the war, you can just drop it into your tomcat (6.0.14) webapp directory and it will get installed into /og-ex/add on your tomcat server.  The client is run by using

java -cp client-jar com.ogoglio.example.AddClient http://localhost:8080/og-ex/add 1

You may need to modify the URL based on how you have your local tomcat configured.

If you want to build from source and you have a copy of maven 2, you  just go into the top level directory of each  project (the one with  the .pom file) and type

mvn clean install

This will result in the war file in the server project's target directory and a jar file in the client project's target directory--just like the ones you can download above.

If you are building without maven, you'll to go into the directory structure and compile the sources (3 files for the server, 2 for the client) by hand (ugh).  That done, you'll need to build the war file with the included web.xml file for the server.  It's probably best to download the war file above and mimic its directory layout when you do the war creation.  When you build the client, you'll either to create a jar file or point the -cp option of java at the directory where the client class files reside.  I suppose you could also do CLASSPATH environment variable shenanigans if you prefer.

What Does It Do?
The server in this example maintains a collection of 10 integers.  These are supplied by clients and the collection is updated when a client sends a new value so that the collection is always the 10 most recent values.  Further, when a client sends a value the server asynchronously (!) updates all other clients--but not the sender--with the set of values it currently holds.  For fun, when a client receives this update it dumps out the sum.

The client side program (com.ogoglio.example.AddClient) prints on the terminal a display like this:

$ java -cp target/add-client.jar com.ogoglio.example.AddClient http://localhost:8080/og-ex/add 2

[purple 7] SERVER ---------->Current sum is 0
[gray   1] SERVER ---------->Current sum is 0
[purple 7] SERVER <---------  35
[gray   1] SERVER ---------->0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 35 [ sum is 35]

Let's explain what you are seeing here.  We gave the program a second parameter of 2, meaning that  we would like to  run two copies of the client. Each one runs on its own thread and we give them names like "purple 7" so it's easier to understand the output.  You can see that the server responds to the two connections with it's initial greeting, "Current sum is 0."  This is a normal HTTP-style request/response.  However, because we are using comet the two clients stay connected.  Every few seconds (chosen randomly) one of the clients sends a message with an integer to the server.  In this example purple 7 sends the server "35".  You then see that the server updates gray 1 with the new collection of values which is the initial set of 0's the server starts with, save for the last value which has been replaced with purple 7's 35.  This will continue until you kill the program.

Peering Inside With A Browser
If you want, you can retreive the current sum with a GET (e.g. by using Firefox) on the same URL.  That is why clients initially get the "sum" string above--that is so that we can immediately display that string in a browser.  The server then immediately terminates the comet session for clients that use GET, so the browser's user gets what she expects.

Further, you can add integers to the queue from the browser.  You do this by adding them to the url, like this:

http://localhost:8080/og-ex/add/9/11/13

This puts 9, 11, and 13 in the queue.  This happens before the sum is calculated so the browser sees the effect of this insertion.

I find it informative to run a single client (2nd parameter to the AddClient of 1) and use the browser with extra integers in the URL to play with things.  This small number of clients makes it easier to see whats happening.

Server Logs
The example server dumps many interesting things out to the server logs, usually catalina.out in your logs directory.  These are more to help a developer understand what things happen at what times inside the server processes various comet events. 

We have been playing around with custom avatars for guests.  Here you can see silhouette avatars mingling in one of the event halls we're putting together for Tomorrow Space:

Of course, we'll need to do something about the hovering names running together, but that's something for tomorrow.

Ogoglio Space Publication

Now that the Ogoglio platform is seeing serious improvements in terms of usability and avatar presentation I've been looking ahead to the day when there is a one-click installer which can turn any desktop machine or server into a host for multiple rich social media spaces with many simultaneous users. With the right people creating cheap or free user-oriented creation tools this will radically change the industry and art of online spaces. 

However, I can't help but feel that it's the prelude to the real story, in which these islands are joined to form something else entirely.  I remain dedicated to the goal of a global network of interconnected spaces which foster a community of creative collaboration and which can be encouraged to emerge using the lessons we're learning from the new understanding of civil planning and architecture on this newly networked Earth.  Along those lines I've been tinkering with ideas about how to use the web to interconnect Ogoglio spaces using a combination of massive automated discovery and action based valuation. 

Starting from the painful lessons we're learning from Google's PageRank, watching the blossoming culture around apps which communicate with mDNS, and finding a fascinating standing wave of conversation among twitterers, I believe that we have the technical and social foundations on which to base Ogoglio City.

Relevant patterns:

dynamic local discovery: The ability for software to find software running nearby in terms of network topology.

manual linking: The ability for a person to indicate that a place in a space should lead to another place in another space.

automated crawling: The ability for software to expose links and content to other software which collects, analyzes and provides search functions.

global reference: an Ogoglio space has a unique URL, as does every user, coordinate, and thing in that space

A rough draft: Right now you can manually create doors which hold URLs to remote spaces and on the pages containing spaces link via HTML.  Public Ogoglio spaces make their contents public via crawl-able XML, and in fact this is the primary form of communication of that information to Ogoglio clients.  This XML includes the text descriptions of the things and people in a space, as well as the names, position and outward links of all doors.

Assume that in the near future every Ogoglio server MAY publish its spaces over mDNS and also MAY create a space which is filled with doors to spaces automatically discovered via mDNS.  Each multicast locale becomes a fully connected graph of spaces.  These interconnection spaces are crawl-able using the same mechanism as any other space, so long lived services will make their way into the crawl indices and may be ranked using identical ranking algorithms which are used with any other hypertexts.

I put this out as a question as much as an answer.  What am I missing?

ogoglio: the nature of the "sim"

If you haven't been following the commit messages to the ogoglio repository, we've mad a few changes and I thought it might be good to explain why on the blog.  Lately, we've added support for running the database/media service, the webapp service, and the "sim" service on different machines.    The three members of this "triumvirate" need to pointed at each other by configuring some URLs in server.xml.  (For people doing development and running the whole setup on one machine this should no ill effects--you just point everybody at 127.0.0.1 and it'll be fine. We do it all the time.)

The first two members of the triumvirate, database and webapp, are pretty straightforward.  They work in exactly the way anyone would expect of a two-tier, RESTful service that is backed by MySQL.  Web requests come in to the REST API of the webapp server and it may need to fetch data from the MySQL server to handle the request before replying.  Adding just a bit of a twist, there is a different web application, the media service, that runs on the same host as the mysql server.  This service is basically just a way to store big blobs of unstructured data in a bag.  This is used for holding model data (which can easily get to be several hundred K), textures, and other bits that you don't want in the mysql store.  The properties of these files are stored in mysql and can be queried, but the actual data itself resides in the humble media store and is accessed by name.

More unusual for people familiar with web services is the third triumvirate member, the sim service. This is also a web service but it is used only for simulating spaces.  When a space is created and entered by a user, a "sim" (simulator) is fired up on the sim server via the sim web api.  This simulator encodes the system's knowlege about the state of the space including all objects in the space and all users.  When a user "moves around" in the space, the user's client sends the movement info (a spline path) to the server.  The server is responsible for informing other clients about the motion of the client that moved.  Further, the sim itself does calculations about things like collision detection and is responsible for running scripts that need to be activated.  The sim can be quite computationally/IO expensive due to the amount of networking and geometry computing it needs to do--thus we allow it now to split onto a separate machine.

There are only three interesting numbers to a computer scientist, zero, one, and infinity.  When we enabled ogoglio's sim server to move to a separate machine, we also allowed you to run many of them.  So, if you have many spaces with many users you can spread the compute load around.  At the current time, though, the front-end code isn't smart enough to choose a sim based on it's load or anything like that, it simply assigns a space to a randomly chosen (but known to be up) sim server.  Further, when all the users have left a space and it is idle for a period of time, the space is "shutdown" and it state saved for later, freeing up the sim server to compute on a space that is in use.

The sim server, while not directly visible to the clients, is the heart and soul of an ogoglio system.  It does the "heavy lifting" of the computation of the virtual world while web front end and mysql server handle the more traditional web parts of the system.

So you wanna be an ogoglio developer?

Goal Of This Tutorial

After you finish this tutorial, you should be able to build and deploy the ogoglio system from the source repository using maven, see that the system is work properly on your local machine, and use your newly built instance of the system via the web.  In addition, it shows some of the typical methods that seasoned ogoglio developers use to develop the software, if you would like to follow the well-trodden path.

What You Need To Know Before You Start

You need to know enough subversion to be able to get a copy of the sources, enough about grabbing stuff off the net to download and install some files, enough shell to know how to change environment variables and your what your PATH is, enough java to know what a java "build" is and what it looks like when it goes wrong, and enough about the web to know if a server is working or not.

---

Setup Your Development Machine

You'll need a copy of Sun's JDK if you don't already have it. At the current time, all ogoglio developers are using version 1.5 but it is likely to work on Java 6 as well--let us know if you get Java 6 to work! The source code of the main system requires only jdk 1.4 to compile and run, but the use of maven below implies java 1.5 so we'll assume in this document you are using 1.5. Also, be very sure that your JAVA_HOME environment is set to point to the JRE (not the JDK if you also have that installed). On my Linux development system, I have to set the JAVA_HOME variable to point "inside" the installation of the JDK:

iansmith@mediasvr:~$ echo $JAVA_HOME
/usr/local/jdk1.5.0_12/jre
 

You'll also need to have maven 2 installed on your machine. All the current ogoglio developers use version 2.0.7, although it is likely to work on any Maven 2 version. If you are not familiar with maven, this might be a good time to read an article or two about it, although this tutorial will try to explain maven concepts as needed (since you won't bother reading anything else before getting the system working). Be sure that your path is setup to include maven's driver program, mvn:

iansmith@mediasvr:~$ mvn --version
Maven version: 2.0.7
Java version: 1.5.0_12
OS name: "linux" version: "2.6.20-16-386" arch: "i386"
 

If you don't have it already, you'll need a copy of java 3d. If you running on a Mac with recent version of OS X, you can ignore this step since Apple (correctly) ships its java development configuration with java 3d installed properly. On other OSes, you'll need to download this Java extension and install it according to the directions. Ogoglio has been tested with java 3d versions 1.4 and 1.3 but is likely to work with later versions as well. The ogoglio server and clients do not expect a later version of the API than 1.3. Be very careful that you got your copy of java 3d installed in the right place, as it is easy to accidentally install it in the JDK not the JRE. You should be able to issue a shell command like this and get similar results:

iansmith@mediasvr:~$ ls $JAVA_HOME/lib/ext/{j3d*,vecmath*}
/usr/local/jdk1.5.0_12/jre/lib/ext/j3dcore.jar
/usr/local/jdk1.5.0_12/jre/lib/ext/j3dutils.jar
/usr/local/jdk1.5.0_12/jre/lib/ext/vecmath.jar
 

On any operating system, you should end up with binary files (the fast part of the java 3d graphics system) in your JRE's lib also. The exact files installed varies by version and operating system, but for example on linux:

iansmith@mediasvr:~$ ls $JAVA_HOME/lib/i386/libj3d*
/usr/local/jdk1.5.0_12/jre/lib/i386/libj3dcore-ogl-cg.so
/usr/local/jdk1.5.0_12/jre/lib/i386/libj3dcore-ogl.so
/usr/local/jdk1.5.0_12/jre/lib/i386/libj3dutils.so
 

Finally, you'll need to download some jar files from Sun that neither we or maven are allowed, for licensing reasons, to just ship to you. We recommend downloading the Java Transaction API (be sure to get 1.0.1B, not the latest version), Java Mail, and the Java Activation Framework and putting them in a directory called lib. You should then also copy your "plugin.jar" file from your JRE to this lib directory with a command like this:

iansmith@mediasvr:~$ cp $JAVA_HOME/lib/plugin.jar lib
 

This last jar file is the the netscape "live connect" api that allows applets to communicate with their containing browser and web page.
Now you can run the little shell script below and maven (mvn) will know about these APIs. You can run this script many times without a problem, so don't worry if you break something at first.

#!/bin/bash
#java transactions (for hibernate)
mvn install:install-file -DgroupId=javax.transaction -DartifactId=jta  -Dversion=1.0.1B -Dpackaging=jar -Dfile=lib/jta.jar
#live connect
mvn install:install-file -DgroupId=netscape -DartifactId=liveConnect  -Dversion=1.0 -Dpackaging=jar -Dfile=lib/plugin.jar
#java mail
mvn install:install-file -DgroupId=com.sun -DartifactId=mail  -Dversion=1.4 -Dpackaging=jar -Dfile=lib/mail.jar
mvn install:install-file -DgroupId=com.sun -DartifactId=jaf  -Dversion=1.1 -Dpackaging=jar -Dfile=lib/activation.jar
 

When it works correctly, you should see one copy of this "success output" for each of the four files:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2 seconds
[INFO] Finished at: Sun Sep 02 17:15:57 PDT 2007
[INFO] Final Memory: 2M/5M
 

What is Maven doing?

You can skip this section if you are not of a curious nature. If this was the first time you have ever run maven, you probably will have noticed a great many output messages something like this:

[INFO] org.apache.maven.plugins: checking for updates from central
[INFO] org.codehaus.mojo: checking for updates from central
[INFO] artifact org.apache.maven.plugins:maven-install-plugin: checking for updates from central
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.2/maven-install-plugin-2.2.pom
2K downloaded
 

Maven comes with some knowlege of how to download dependencies from the internet (for example, from http://repo1.maven.org/maven2). Naturally, maven is implemented in terms of itself, so when you first use a maven plugin, it would be better called a "feature", it will attempt to fetch the latest version of the part of maven you need. Our little script above uses the maven "install" plugin, so it is fetched from a repository on the internet, and some of the output shown in the example immediately above.

All of the binaries that maven fetches, you install as jar files, or you build and then install end up a directory $HOME/.m2/repository. This is your local repository of binary data (mostly jar files and some meta-data for maven's bookkeeping). You can find the files you installed with script above by looking around in the repository like this:

iansmith@mediasvr:~/$ ls $HOME/.m2/repository/javax/transaction/jta/1.0.1B
jta-1.0.1B.jar
 

This repository is just a cache. If things go wrong and you want to start over, you can simply blow away the $HOME/.m2/repository directory and start again by using the small shell script above to restart your repository.

Install Tomcat 5.5

You need to make sure that you download and install version 5.5 of Apache's Tomcat. Ogoglio should work with many other servlet containers/web servers but the ogoglio developers all use version 5.5 of Tomcat and there are some deviations from this tutorial that would have be made to the maven configuration to use other versions or other servlet containers (jboss, resin, geronimo, jetty, etc).

Install mysql and create a database with a user

Ogoglio makes heavy use of a database and we recommend mysql version 5 because that is the version we use and we are sure works properly. Many developers already have a copy of this on their machines or, on linux, can easily get access to it via the package managers of most distributions. Since ogoglio uses hibernate for all of its database actions, in principle it should be possible to get ogoglio to work with almost any SQL database. We don't recommend trying to get ogoglio to work with another database on your first try at building/running.

With mysql installed and assuming you know the administrator username and password, create the new ogoglio database. We'll use root as the example administrator account for mysql and og as the name of the database:

iansmith@mediasvr:$ mysqladmin -u root -p create og
 

Now you need to get into the mysql shell and create the ogoglio account that can manipulate the mysql database:

iansmith@mediasvr:~$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 238
Server version: 5.0.38-Ubuntu_0ubuntu1-log Ubuntu 7.04 distribution
 
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
 
mysql> GRANT ALL PRIVILEGES ON og.* TO 'oguser'@'localhost' IDENTIFIED BY 'secret';

With that, you are ready to start being a real developer of ogoglio.

---

Building Ogoglio

To build ogoglio from source, you'll need to copy this file to $HOME/.m2/settings.xml. (If you are already a maven developer, you may have a file there and this should be added to it.) This files controls all the configuration options for ogoglio. You should look through and set the variables to match your development system, for example pointing cargo.tomcat.home at your tomcat 5.5 installation. The key variables to check are ogoglio.mysql.url (the part after the final slash should be the name of the database you created above), ogoglio.mysql.user, ogoglio.mysql.password (these two should much the account you created above), ogoglio.mediaDirectory (should be a path where ogoglio can store media files), and ogoglio.mediaURL (should be a URL form of the same path where ogoglio can store files). You may want to change the bootstrap account mentioned in ogoglio.bootstrapUser, ogoglio.bootstrapUserPW, ogoglio.allBootstrapUsersPW, and ogoglio.allBootstrapUsers. If you change those, be sure sure to be consistent when changing so that the pairs ogoglio.allBootstrapUsers,ogoglio.allBootstrapUsers, ogoglio.bootstrapUser, and ogoglio.allBootstrapUsersPW, ogoglio.bootstrapUserPW match. This account (and password) is an ogoglio account and will allow you to log in to the ogoglio server via the web when you get the server up.

You'll need to grab a copy of the ogoglio sources from the repository, like this:

svn co https://ogoglio.svn.sourceforge.net/svnroot/ogoglio/maven
 

This will take a few moments to pull over all the source files to your machine; many of these files are actually 3d models or other bits of art needed to make a reasonably functional system on your machine.

You need to go to the "master project" called ogoglio (all the subprojects have hyphenated names describing them) and we'll assume you were in /tmp when you grabbed the sources from subversion above. Let's be sure everything looks ok:

iansmith@mediasvr:/tmp$ cd maven/trunk/ogoglio
iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ ls
pom.xml  src
 

You'll note that there is only the pom.xml file that describes our project to maven, and a directory of source code. This is a guiding principle of all "mavenized" projects, only the source ever gets checked in. There are no jars in the source repository; they all live in $HOME/.m2/repository on the developer's machine and are either fetched or built from source as needed.

Using maven, you can build and test all the source code in one fell swoop. The output regarding ogoglio that you see should look something like this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ mvn clean install
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO]   Unnamed - com.ogoglio:ogoglio:pom:0.0.1-SNAPSHOT
[INFO]   Unnamed - com.ogoglio:ogoglio-common:jar:0.0.1-SNAPSHOT
[INFO]   Unnamed - com.ogoglio:ogoglio-appdev:jar:0.0.1-SNAPSHOT
[INFO]   Unnamed - com.ogoglio:ogoglio-body-editor-applet:jar:0.0.1-SNAPSHOT
[INFO]   Unnamed - com.ogoglio:ogoglio-test-applet:jar:0.0.1-SNAPSHOT
[INFO]   Unnamed - com.ogoglio:ogoglio-viewer-applet:jar:0.0.1-SNAPSHOT
[INFO]   Unnamed - com.ogoglio:dev-plugins:maven-plugin:0.0.1-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
 

Again, if this is the first time you have run the maven tool, mvn, it may download a great many files to bring itself up to date. At the end of the process you should see a success message like this:

[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 58 seconds
[INFO] Finished at: Sun Sep 02 18:35:04 PDT 2007
[INFO] Final Memory: 9M/28M
[INFO] ------------------------------------------------------------------------
i

You will probably have noticed that the integration tests briefly put up a window on your screen. This window is testing that the system can actually display a 3d space, and is this "flash" is expected and nothing to worry about. (If you are running this on a headless box and thus the test suite bombs when it tries to display this window, you can run a copy of Xvfb and set your DISPLAY variable to point at the Xvfb instance.)

Strictly speaking, you didn't need to do an "mvn clean" since this is your first build and there is nothing leftover from previous efforts, but its good practice if you have to come back to this step. The "install" phase implies compiling, testing, and installing the created jar files into your local maven repository. You can see the parent directories of these newly minted and installed jars with this code. You find it interesting to go exploring in these directories to see what information maven stores about the various projects.

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ ls $HOME/.m2/repository/com/ogoglio
dev-plugins                 ogoglio                     ogoglio-common
maven-metadata-central.xml  ogoglio-appdev              ogoglio-test-applet
maven-metadata-local.xml    ogoglio-body-editor-applet  ogoglio-viewer-applet
 

If Something Goes Wrong

Normally, you should see the success message above. If something goes wrong, you'll get some type of output from maven. A few key points of information may be helpful here.

• Figure out what maven sub-project (such ogoglio-server, ogoglio-common, etc) you were building at the time of the failure. You can do this by scrolling upward (sometimes quite a ways) until you see a message like this:
  code
  [INFO]

  ---

  
  [INFO] Building Unnamed - com.ogoglio:ogoglio-integration-test:war:0.0.1-SNAPSHOT
  [INFO]    task-segment: [install]
  [INFO]

  ---

  
  
  code
  
• If this was a unit test failure, you find the output of the tests in [subproject-name]/target/surefire-reports/ and this information is likely to be helpful in tracking down problems.
• If this was an integration test failure (which means it occurred in the integration-test project shown in the code above) then not only are the surefire reports likely to be helpful, but also the files tomcat5x.out and tomcat5x.log in the root of the subproject directory. These are the outputs of the tomcat server that was run to test the integration of the client and the server. Server errors are located in the two tomcat output files, client side errors are in the surefire reports. However, almost any time there is a server failure, some response gets sent back to the client (such as a "500 Internal Server Error") that causes it to fail also. When in doubt, try to solve the server problem first.

Booting The Ogoglio Server And Deploying The Web Application

We will assume you have now built and tested all the code and that you continue to work in /tmp. For this task you'll need to open two more shell windows, which we will call the "boot" window and the "tail" window. In both of these go to the directory that is the home of the master project, such as /tmp/maven/trunk/ogoglio in our above examples. In the boot window, you will be using maven to boot up a copy of tomcat, like this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ mvn -Pboot-server cargo:start
 

This will eventually end up with a message like this:

[INFO] [cargo:start]
[INFO] Press Ctrl-C to stop the container...
 

tomcat is now running and you can now iconify this window because not much exciting is going to happen here in the rest of the tutorial. However, it is vital that you understand that can reach the newly booted server (assuming you kept your $HOME/.m2/settings.xml file unchanged from the example) via http://127.0.0.1:8080/. For some reasons related to cookies, trying to access this server as http://localhost:8080/ will work just enough to convince you it's ok, then break your heart. You must access the server with the exact same URL you put in your settings.xml file. The instance of tomcat that is running is fully configured and running in /tmp/tomcat5x, as mentioned by the $HOME/.m2/settings.xml file. At this point, this server has only one web application installed in it, one used by the maven infrastructure itself, called cargopc. You can see any and all installed web applications like this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ ls /tmp/tomcat5x/webapps/
cargocpc  cargocpc.war
 

The war file is a standard format for java web applications--which is really just an archive file. The unpacked version of the war file is in the directory that bears its name.

To monitor the state of tomcat's output, you should use your other shell window, the "tail" window, to do this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ tail -f tomcat5x.out
 

Under normal conditions this "tail" window will never show stack backtraces or other serious errors. If they are present, it's probably worth looking carefully at this window's output (or the underlying tomcat5x.out file in your ogoglio master project directory) to try to see what's wrong.

Let's try deploying the ogoglio-server web application to our running tomcat instance. This is referred to as a "hot deploy" by ogoglio developers (keep up on your lingo!).

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ cd ../ogoglio-server
iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ mvn -Phot-deploy cargo:deploy

After some work, maven should respond with a message like this:

[INFO] [stalledLocalDeployer] Deploying [/tmp/maven/trunk/ogoglio-server/target/ogoglio-server-0.0.1-SNAPSHOT.war] to [/tmp/tomcat5x/webapps]...
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35 seconds
[INFO] Finished at: Sun Sep 02 18:54:17 PDT 2007
[INFO] Final Memory: 10M/19M
[INFO] ------------------------------------------------------------------------
 

The "local deployer" line indicates that the war for this application has been sent to the tomcat server running in your other windows. Since tomcat watches /tmp/tomcat5x/webapps for changes, shortly after you see this build message you'll see your tail window going crazy with messages as tomcat brings up the new web application. You can see the effect in the webapps directory:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio$ ls /tmp/tomcat5x/webapps
cargocpc  cargocpc.war  og  og.war
 

Assuming everything has gone ok this far, you can point your browser at http://127.0.0.1:8080/og and you should see the ogoglio welcome banner (or "coming soon" if the welcome banner hasn't been implemented yet!).

Working On The Server Code Proper

The previous example we booted an instance tomcat and used the "hot-deploy" to push new server code (in a war file) to the running tomcat. This is the basic way to send the server code to be tested with a browser. If you are developing server-side code or (gasp) fixing bugs, you go into the ogoglio-server project, make your src code changes in src/main/java/com/ogoglio, and then use a maven command at the top-level directory like this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ mvn clean install

This builds all the sources, runs the server's unit tests, and (if everything went ok) builds a war file that you can run the integration tests on. This war file is put in the maven repository as a war file so it can be accessed by other projects, if referenced. You see this with:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ ls ~/.m2/repository/com/ogoglio/ogoglio-server/0.0.1-SNAPSHOT/
maven-metadata-local.xml           ogoglio-server-0.0.1-SNAPSHOT.war
ogoglio-server-0.0.1-SNAPSHOT.pom
 

A bit tricky, though, is the hidden fact that some of the unit tests destroy the database's data. They do this so the tests can be sure of the system state when performing tests, but any running server instance has no idea what has occurred "behind it's back." If you return to your web browser connected to a previously run server, you may get strange results. (See the next section for how to easily reconstruct the data you want to work with.) You can take a bit more dangerous shortcut if you are working solely on the web UI, e.g. the HTML and javascript files in src/main/resources/siteTemplates:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ mvn -Phot-deploy compile war:war cargo:deploy

This runs the compile phase, builds the war file and deploys it to the running server but skips running the tests. We don't recommend doing this for server-side java code, although it's likely to work in the short term because avoiding the unit or integration tests is a recipie for disaster. Trust us, we know. This should only be used for tweaking the look and feel of ogoglio's web UI. Without the shortcut it can be quite tedious to go through the full test regime to get a change of a pixel here or there.

Minor Secret
The web UI for ogoglio that is contained in src/main/resources/siteTemplates understands Velocity templates. If you put velocity directives like this in ogoglio's html files

#parse("header.vm")

they will be correctly expanded, assuming of course you have a "header.vm" file in that directory also. This expansion runs during the "compile" phase of building the server. After that point, the HTML used for ogoglio's UI is frozen and doesn't change. At no point does the server ever "generate" HTML for consumption by any client--it only servers up static content or speaks XML via it's web API.

Returning to our discussion of the server's java code, if you have successfully passed the server's unit tests (with 'mvn clean install' above) you should run the integration tests. During this process, a real copy of tomcat gets launched and (your presumably new) server code is deployed to it. The integration test suite behaves like a client: it uses real HTTP calls on the server code to insure that the new code hasn't broken anything. You can run the integration test by switching to the integration test project's top level directory and doing this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-integration-test$ mvn clean install

In fact, this doesn't actually "install" anything much in your maven repository. We use the "clean install" idiom in every directory to mean "do everything you need to do." As we said above, the integration tests will actually put a window on your screen briefly. It should now be more clear why: that test is testing that a client application can actually connect to space on the server with an applet.

Committers
The ogoglio project's developers are pretty serious about thorough testing; our testing constantly improves. If you want to be a commiter on the project, we'd ask that as you fix bugs or add features please add unit and functional (integration) tests that show that you have fixed the bug or correctly implemented the new feature.

Working With 3-D Objects and Spaces During Development

Ogoglio comes with a tool for batch uploading of templates (3-D models and their scripts) and automatically creating spaces on a running server. You can see this in action with:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ mvn og:populate

When you run this command you should see a bunch of output like this as it does its work (if you don't, check and be sure that your server is running--'mvn -Pboot-server cargo:start' in ogoglio's master project directory):

[INFO] Uploading template #9 to http://127.0.0.1:8080/og/
[INFO] Created template from EamesLoungeChair.obj  [9 -> 5]
[INFO] Uploaded support file:EamesLoungeChair.mtl
[INFO] Uploaded support file:wood11.jpg
 

This output shows that the various bits of the Eames Lounge Chair were uploaded and although it was listed as template number 9 in the src/main/resources/populate directory, its template ID on the running server is 5.

The "og:populate" command in maven is configured to look look in the directory src/main/resources/populate for files and directories that have names that might be spaces or templates. If it finds them, it uploads them to a running server. It is careful to try to not upload templates that have the same name as ones already present on the server. It does this based on the names and dates, so you'll need to touch the templates object file, thus updating its last changed time, before the upload process will destroy the one on the server and replace it with your newer version . The contents of a template directory looks like this:

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ ls src/main/resources/populate/template-1
Moon.jpg  Moon.js  Moon.mtl  Moon.obj
 

The template when it is uploaded will not be numbered 1. The ogoglio server will chose the template's ID based on its state. as we saw in the example above with the lounge chair changing to ID number 5. The resulting template will be named "Moon." The ".obj" file has the 3-D model in a textual form, the ".mtl" is the materials used in the model, the ".jpg" is a texture, and the ".js" file is javascript code associated with that template. These are the primary file types used by the system itself; some of the templates supplied with ogoglio have other types of files in their directories like (.blend or .3DS) that can be used to edit the model with tools like blender or 3-D Studio. These are just for your convenience, the template uploader during og:populate will ignore them.

Also allowed in the src/main/resource/populate directory of the ogoglio-server project are spaces. Let's look at a space document that will get uploaded when we do "mvn og:populate":

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ more src/main/resources/populate/space-1
 
<space ownerusername="library" sealevel="0.0" simid="1"
        displayname="Sitting" maxguests="0" displaysea="false" spaceid="1"
        published="true">
        <thing templateid="9" rz="0.0" ry="0.0" rx="0.0" rw="1.0"
                scalez="1.0" displayname="EamesLoungeChair" scaley="1.0"
                templateowner="library" scalex="1.0" possessionid="1" thingid="1
"
                z="-5.0" y="0.0" x="-18.0" ownerusername="library" />
 
        [several more thing elements elided for clarity]
</space>
 

When og:populate runs, it creates a space on the server with the name "Sitting." Again the server's final ID for the new space will be of it's own choosing, it's not likely to be '1' as this document's filename might suggest. However, the uploader does change the templates used in the final, resulting space document to be sensible with respect to the template numbers in the directory names. In other words, you can write this space document as if the template numbers were indicated correctly by the directory names, such as '9' above in the space document. It will get changed to the correct number at the time the space is constructed, which might be '5' as we saw in our prior discussion of the Eames Lounge Chair.

If you go visit this 'Sitting' space on your running server, you should see 5 chairs in the space, all arranged so that they are facing you when you enter the space (just spin around with the left/right arrow keys to see them). You can change the chairs properties (like orientation and location) in the resulting space by changing properties in the space-1 document, deleting the space with the ogoglio web UI, and re-running "mvn og:populate."

This process can also be done in reverse. This means that you can use the ogoglio server's web UI to manipulate objects in a space, chosing and arranging the objects in the space to get them the way you want. Then, you have to expose (ta-da!) that ogoglio is really a web application! You do this by going to a URL that looks like this

http://127.0.0.1:8080/og/space/1?children=true

This URL asks the server to respond with its space document for space number 1. You can figure out a space's number by choosing to edit the space from the web UI and looking at the URL your browser goes to. It will look like this and the space's ID will be pretty obvious.

http://127.0.0.1:8080/og/spaceEditor.html?spaceID=1

When you go to the URL to get the space document, you should use your browser's "save as" function to save the resulting XML into a file named "/tmp/reverse.xml". This filename is controlled, like everything about maven in the pom.xml file in the root directory of the ogoglio-server project. You can then do the following to ask for this, newly saved, space document to be "converted" to use the templates in your populate directory.

iansmith@mediasvr:/tmp/maven/trunk/ogoglio-server$ mvn og:reverse

This command will output a new space document into the src/main/resources/populate directory in which the server's "true" mapping has been reversed back to the ones you see on disk. If you think about the Eames Lounge Chair in prior examples, it is contained in a directory called "template-9". We saw that when we uploaded the chair the true template number was 5 on the server, but our space document space-1 could use the 9 to indicate which template it was referring to and the upload would be "patched" to use the server's number. When doing a reverse, a space document is patched the other way, so that the "real" space document you just downloaded into /tmp.reverse.xml will be copied into the populate directory of the source code and will refer to the numbers of the templates in that same directory.

If you have been following this carefully, you should realize that this whole process of og:populate and og:reverse allows you to create data--spaces and templates--that persist across clearing the database.