This is a republished guest blog post by Ed Ropple. Ed is a platform engineer with Localytics. His ambition is to enable other software developers to be more productive and less error-prone. You can find his original article here.
Since starting at Localytics in Februmarch or so, I’ve found myself thrown into a bunch of new-ish stuff. My prior ops experience was much more “developer moonlighting as a sysadmin”, rather than buzzword-compliant DevOps Ninja Powers. At Localytics I’ve been leveling those up pretty quick, though, and there’s some fun stuff I’d like to talk about a little. But we need to figure out what we want to make public first, and it’ll probably end up on the company blog before it ends up here, so I’m going to natter on a bit about something I’m doing in my spare time: setting up a Mesos cluster and populating it with apps for a side project or two.
What’s Mesos?
Glad you asked, figment of my imagination! Mesos is a compute management framework that was born in the AMP Lab at UC Berkeley. The idea behind Mesos and other platform-as-a-service (PaaS) projects is to take your entire data center, the whole kit and caboodle of it, and treat it as a single heterogeneous compute environment. Apps don’t run on servers (well, they do, but only after a fashion); instead they run on the whole damn data center. There are a few other tools that act vaguely similar to Mesos, among them Deis and Flynn. All of them are deeply in hock to the Google Omega paper, which you should go read because Google does this stuff at scale. The differences between the various clusterization options are largely in scope–Mesos is a fair bit more ambitious than Deis and Flynn—and the tooling each project’s selected for their stuff.
You’ll also see references to Marathon along the way, too. It’s a scheduler for Mesos that acts sort of like an init system—it provides a lot of the brains of the system. I’ll be adding it as I go along.
I found this video on Mesos, Marathon, and Docker to be really helpful for thinking about this stuff, I suggest you watch it. I’m funnier than they are, but they actually know what they’re doing. (How dare they.)
Hardware
Unfortunately, I don’t have a data center in my house. I mean, I’ve considered a rack, but I pay enough for air conditioning during the summer as it is. So my available hardware is a little constrained. My former VM server, now sort-of-Docker playground, is an Ivy Bridge i5 with 32GB of RAM and a few terabytes of disk, is the best candidate for the job, but it’s also running my home VPN and I don’t want to hose the machine beyond my meager powers of resuscitation. (Yet.) So for the first bits of this series, I’m going to be building a three-node Mesos cluster on my MacBook Pro.
Getting Started
First off: I’m going to be uploading my stuff as I go to mesos-experiments on Github. I’ll tag the state as of each post; this post will contain blog-1 and blog-2.
Despite Mesos being an Apache project, most of the useful documentation is instead on Mesosphere‘s site. Mesosphere is the commercial arm for Mesos. Their “learn” page is a little intimidating. Like, I know what all the words mean, but I’ll be damned if I know what they mean all strung together. But not knowing the first thing about something has never stopped me before, so on we go. Since I’m using Vagrant on OS X , I was tempted to give vagrant-mesos a spin, but that has a real problem for me in that it comes pre-built and I won’t understand what I’m doing with it. So, soup-to-nuts it is.
Mesos provided instructions for Ubuntu 14.04, so I went ahead and grabbed a box from the depths of the Internet.
vagrant box add --name ubuntu-14.04-chef https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vbox.box
(Thanks for the box, Phusion! You’re the best company I know absolutely nothing about.)
Anyway, this Vagrant box has Chef and Puppet installed; I’m not a partisan, unless the parti against which I’m sanning is Chef Server, ’cause I have had enough of that for one lifetime. So Chef Solo it is. Let’s init us some Vagrant:
vagrant init
The default Vagrantfile is filled with comments that at this point in my Vagrant life I don’t need or want, so after deleting approximately one Pete worth of comments and cargo-culting me some chef-solo, here’s what I’ve got:
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu-14.04-chef"
config.vm.synced_folder '.', '/vagrant', disabled: true
config.vm.synced_fonder './scripts', '/opt/scripts'
config.vm.provision "chef_solo" do |chef|
chef.cookbooks_path = [ "./chef/cookbooks", "./chef/librarian-cookbooks" ]
chef.add_recipe "edcanhack_mesos"
chef.json = {}
end
endThe astute among us will notice that this refers to a nonexistent Chef cookbook. And the astute among us can shaddap, thankyouverymuch, I’m getting to that. To avoid spamming your eyeballs out, I’ll omit the connective tissue of the system setup I’m working with (it’s in the repo) and just include the interesting bits of my cookbooks1.
For now, the recipe is pretty straightforward:
include_recipe "ubuntu"
include_recipe "python"
MESOS_DEB_URL="http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.19.0~ubuntu14.04%2B1_amd64.deb"
MESOS_EGG_URL="http://downloads.mesosphere.io/master/ubuntu/14.04/mesos-0.19.0_rc2-py2.7-linux-x86_64.egg"
case node[:platform]
when "ubuntu"
%w"zookeeperd default-jre python-setuptools python-protobuf curl".each { |p| package p }
else
raise "ubuntu only!"
end
remote_file "Mesos .deb" do
source MESOS_DEB_URL
path "/tmp/mesos.deb"
end
dpkg_package "Installing Mesos .deb" do
source "/tmp/mesos.deb"
end
remote_file "Mesos .egg" do
source MESOS_EGG_URL
path "/tmp/mesos.egg"
end
bash "Installing Mesos .egg" do
code "easy_install /tmp/mesos.egg"
endThese are the underlying prerequisites, as per Mesosphere’s prerequisites page. I haven’t yet extended this for multi-server, obviously, so we will be running what they call a singleton cluster for the time being. Unfortunately, while you can (inadvisably) reboot a machine within Chef, Vagrant will lose all its marbles when you try–guess how I found that one out–so we’ll do that ourselves too.
vagrant up && vagrant halt && vagrant up # AND UP AND DOWN AND UP AND DOWN vagrant ssh -c "curl -i localhost:5050/master/health"
And… now I have a machinegunMesos-ready box. (This is tag blog-1.)
HTTP/1.1 200 OK Date: Fri, 11 Jul 2014 06:32:28 GMT Content-Length: 0
App app app
So I came to Localytics as a Scala Person, and I remain so, but I’ve ended up becoming acquainted with Ruby in no small part through doing an uncomfortable amount of Chef wizardry. Still no clue as to Rails, though. Good news for me right now, though, is that Mesosphere has a tutorial for putting a Play app onto a Mesos cluster. So lemme get to it.
If you’ve clicked on the barrage of links thus far, you probably know that Mesos uses a two-level scheduler. The top-level scheduler is the Mesos master. Its job is to fairly (or unfairly, if that’s your bag) allocate resources across the apps running in its frameworks. Frameworks generate jobs for the Mesos slaves to run, passing them back up through the master and down to the slave nodes to run them. Mesos comes with support for Hadoop and Spark and a bunch of other fashionable belles at the distributed computing ball. Mesosphere has also developed a framework of their own, called Marathon, which is designed to execute long-running tasks.
Mesosphere differs from Flynn, Deis, and Heroku in that it doesn’t use a git push model; instead you just point it at a zip file and it does its thing off of that. Mesosphere has a PlayHello project that we’ll start with. More pressingly, though, Step 1 of their tutorial says “go download Marathon!” and I can’t be bothered to do that manually, so let’s go drop that into our Chef cookbook. (And, while we’re at it, gem install marathon_client so we have that for later…)
user "marathon" remote_file "Marathon .tar.gz" do source MARATHON_URL path "/tmp/marathon.tar.gz" end bash "Installing Marathon" do code <<-ENDCODE tar xzf /tmp/marathon.tar.gz mv /tmp/marathon /opt/marathon chown -R marathon /opt/marathon ENDCODE end cookbook_file "Adding Marathon upstart script" do source "marathon_upstart.conf" path "/etc/init/marathon.conf" end
Once again, the eagle-eyed will see a cookbook_file directive in there. The blog-2 tag includes a “marathon_upstart.conf” file. It originates from a gist by jalaziz that handles things pretty much as I would have–though it has some neat idiomatic tricks I’d never seen, like how it handles file-existence tests (which are in retrospect pretty obvious, but new to me). I updated it to use Marathon’s start script rather than calling it directly and chopped out some extraneous bits.
The directories in that upstart script are somewhat magical, in that Mesos dumps some stuff to the file system in places that Marathon expects them. When we get to a cluster that isn’t a singleton we’ll need to provide some configuration in /etc/default/marathon.
Anyway, reprovision with vagrant provision, bounce the servers again, and you should have Marathon running on localhost:8080. (I’ve also added a port forward to/from port 8080 and upped the VM RAM to 2GB in the Vagrantfile.) Click the friendly ‘new app’ button.
- ID: hello-test
- Command: ./Hello-*/bin/hello -Dhttp.port=$PORT
- Memory: 512
- CPUs: 0.5
- Instances: 1
- URIs: http://downloads.mesosphere.io/tutorials/PlayHello.zip
You should see an instance of the app come up along with a randomized port for it to play on.
The first time I did this, I did not get the wonderful deployed app I was hoping for. Instead–nothing. Zero instances came up in my app info. Scaled up, no good; scaled down, ditto. Figured out I’d misconfigured Vagrant with too little RAM for the instance, so went and bumped that up. New failure: the app would start and immediately die2. Ordinarily, Mesos installs its logs to $MESOS_HOME, but (distressingly) installing it via the .deb package puts things in…places. Eventually I tracked down the output of my workers to a directory six levels deep within /tmp and for a moment doubted my fortitude, but continued on nevertheless.
Look what I found:
root@ubuntu-14:/tmp/mesos/slaves/20140712-044358-16842879-5050-1169-0/frameworks/20140711-071659-16842879-5050-1167-0000/executors/hello-test_0-1405141392233/runs/latest# cat stderr
SONOFA:
WARNING: Logging before InitGoogleLogging() is written to STDERR I0712 05:03:12.295583 3228 fetcher.cpp:73] Fetching URI 'http://downloads.mesosphere.io/tutorials/PlayHello.zip' I0712 05:03:12.296902 3228 fetcher.cpp:123] Downloading 'http://downloads.mesosphere.io/tutorials/PlayHello.zip' to '/tmp/mesos/slaves/20140712-044358-16842879-5050-1169-0/frameworks/20140711-071659-16842879-5050-1167-0000/executors/hello-test_0-1405141392233/runs/4750cdfa-fcc2-44fb-a078-819edc3fdad7/PlayHello.zip' sh: 1: unzip: not found
But! That mystery solved, we now have… the stock Play hello-world app.
This is the blog-2 tag in mesos-experiments, and that’ll do it for now. For my next trick (and next post), I’ll build a separate secondary slave and add some Vagrant magic to let it scale to an arbitrary number of slave nodes.
| Reference: | A Virtual Mesos Cluster with Vagrant and Chef from our JCG partner Manuel Weiss at the Codeship Blog blog. |
