Google Appengine Staging Server Howto

Out of the box, Google’s App Engine supports versioned deployments. You can switch back and forth between revisions very easily, which is a great feature for properly testing an application before going live. There is one major problem: All versions of the application share the same datastore. So if you’re migrating your data you run a serious risk of influencing your current production application. Hence the need for a proper staging environment.

It’s no secret, I am a fan of Google’s App Engine. Once you get used to its peculiarities, it has a number of major advantageous. Since I started incorporating some of the continuous integration/lean startup ideas in my own project. I’ve run into the shared datastore issue and the need for a properly isolated staging environment has become apparent.

Here’s how I did it.

Setting up the Staging Application

It’s possible to use namespaces to create an isolated datastore, however I didn’t want to create additional code for testing. So I took another approach, which I believe is a lot easier and less error-prone:

1. In the appengine control panel, create a second application. You have 10 free ones so that shouldn’t be a problem. I added the “-staging” suffix to the name of the application under test, so I won’t mistake one for the other.
2. If you want to start from a copy of the existing datastore, you can export the entire datastore using the Python development kit. Even if you’re using the Java development kit, it’s worth setting this up. It allows you to make backups of your datastore, which might come in handy when something is really messed up.
3. Next, import the database into your staging application using the same tool.
4. And finally, deploy your application to the staging application. If you’re using Eclipse, just change the application id, if not, you can find the property in the appengine-web.xml.

A small note on using production data in your tests: Be very careful about it. You may want to anonymize some of the data and remove anything that could be remotely confidential.

That should be it. There really wasn’t much to it, but you now should have a fully functioning copy of your production application. Surf around a little to make sure everything is working swiftly.

When you’re happy, lets automate it.

Automating Deployments

I was about to throw out Maven, but I’ve now created a setup that I’m pretty happy with. So Maven is here to stay for now. As are the Maven Eclipse plugin and the GAE plugin for Maven.

It’s thanks to the maven-gae-plugin that I could automate the staging and production deployments. Which has given me a very reproducible build and deployment set up.

To seamlessly create a build for both the staging and production server, I’m using Maven profiles and its ability to filter resources while copying them.

In the appengine-web.xml I added a gae.application variable:

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
        <application>${gae.application}</application>
...

Next up I enabled filtering of the appengine-web.xml (all of the next few bits go into the pom.xml):

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-war-plugin</artifactId>
    <configuration>
        <webResources>
            <resource>
                <directory>src/main/webapp</directory>
                <filtering>true</filtering>
                <includes>
                    <include>**/appengine-web.xml</include>
                </includes>
            </resource>
        </webResources>
    </configuration>
</plugin>

In the properties section, I added the default application, which is the staging one. This gives me the assurance that I’ll always be deploying to the staging environment, unless I really want to go to production:

<properties>
    <gae.application>myapp-staging</gae.application>
</properties>

And for the production deployment I created a profile:

<profiles>
    <profile>
            <id>production</id>
            <properties>
                    <gae.application>myapp</gae.application>
            </properties>
    </profile>
</profiles>

With this configuration, I can easily run the local development server:

> mvn gae:run

Deploy to the staging server:

> mvn gae:deploy

And when I’m happy, deploy it to the production server:

> mvn gae:deploy -Pproduction

In addition to the name of the application, you can also configure other properties that differ between a test setup and a production one. For instance, I use the PayPal development servers locally and on the staging server, but the real PayPal site in production.

Conclusion

With a pretty simple Maven configuration, it’s possible to create a very reproducible build and deployment environment. Add a continuous integration server and you’re on your way to the perfect lean setup.

Reference: A Google Appengine Staging Server Howto from our JCG partner Peter Backx at the Streamhead blog.