We blogged about the alpha launch of Shipped back in June, but today I want to talk about the service it relies on: Mantl.

Mantl's goal is to provide a fully functional, instrumented, and portable container based PaaS for your business at the push of a button. It's an opinionated set of services, configurations and tools that provide a standard interface for you to operate your container-based applications and infrastructure. Typically companies write a lot of unnecessarily repeated, similar-looking glue code to deploy, inspect and manage their services and the infrastructure they run on. This code is difficult to effectively version control and is seldom the same between teams. Mantl is an attempt to turn these glued together pieces into an easy to use complete solution for managing your services.

The user experience that addresses this glue is called "Shipped" and it provides an integrated Build, Deploy, and Run experience. The Build experience is integrated to the developers existing tools, frameworks, and development languages (so there are no changes to how you develop your application). Deploy has become very complex over the last several years as cloud services and providers require you to refactor or rewrite your application for their environment. We do not believe this is the right approach.

Mantl's design is to enable you to deploy your application across any internal, private, public, or hybrid combinations of environments and technologies. You get the benefit of not being locked in and the reduced complexity of having a consistent deployment process. Finally, to support any deployment, the Run experience is very key. The goal of the Run experience is to be able to support your application across deployment locations in a CI/CD & Application Intent Framework focused on business objectives and not technologies.

You may find Mantl useful if:

• You're operating in an inter-cloud environment and want to deploy services in a consistent way across clouds.
• You're decomposing a monolithic application into micro-services, like SoundCloud's approach.
• You currently deploy many micro-services in an ad-hoc way and are having trouble managing all the disparate systems.
• You're looking for a standard platform to deploy many high-availability oriented big-data applications easily.

You can adopt as much or as little of it as you'd like, but before we get into those decisions, let's look at what it provides out of the box.

What's does it do?

Mantl is a collection of tools from a lot of great service providers, the big ones are below.

Provisioning

Terraform

Terraform, by HashiCorp, provides an abstraction layer that allows you to provision compute resources on a large number of providers. It bootstraps the raw infrastructure that will run Mesosand all the corresponding services on top of it.

Scheduling

Mesos

Mesos manages resources in your cluster and handles scheduling and isolating your tasks. It allows you to bin-pack your servers, efficiently running multiple tasks on each node. It's the key to making all of this work. Mesos relies on Zookeeper for high availability, so you'll also get that for free.

Kubernetes

Kubernetes solves similar problems as Mesos and support is planned in the near future.

Service Discovery

Consul

Consul, another Hashicorp product, handles service discovery. It makes it easy to coordinate cluster membership and also does some other handy things like arbitrary key value storage for configuration management, etc. A lot of the coordination in Mantl uses Consul to make the magic happen.

Logging & Search

Mantl uses the "ELK" stack to handle logging and indexing.

Elastic Search

This handles indexing logs and metrics from each host and each container. It's how you'll monitor the performance of your services.

Logstash

This little daemon runs on each node and basically pipes logs into elastic search for later consumption/analysis.

Kibana

Kibana is basically a visualization layer for elastic search. If you annotate your logs nicely, Kibana will make pretty graphs for you (yay!).

Monitoring

CollectD

This is another daemon that runs on each node and collects information so you can monitor your cluster in a standard way.

Container Management

Docker

Mesos has built-in support for Docker, a very popular portable container implementation. It's typical to build your application into a deployable docker container.

Marathon

Marathon manages long running tasks, like your applications. It's the typical deploy target for consumers of Mantl. If your servers or containers die, marathon takes care of rescheduling them so your services are always available.

Application Routing

HAProxy

Marathon allows you to scale the number of containers running your application, but once you've done so, you have a new problem: routing traffic between instances of your application. Mantl manages this for HTTP(S) traffic by keeping  the HAProxy configuration up to date with Consul entries for each service running in Marathon.

Glue

Mesos-Consul / Marathon-Consul

These are the data-bridges that keep Consul up to date. Whenever Mesos or marathon runs a new task, these lets Consul know about it so the DNS entries are always current.

HAProxy-Consul

This is a Docker container purpose built as a load balancer for your application deployments. The proxy configuration is kept up to date by Consul-Template to load balance between containers running the same application. For now, this only works for stateless HTTP(s) services, but eventually it may support other protocols.

Security

Calico

This handles virtual network isolation, which is useful for multi-tenant environments or maintaining strict isolation for security and compliance reasons.

Vault

Most systems have secrets, Mantl provides Vault out of the box to set you up to handle your secrets and personally identifiable information in a safe way.

How does Shipped use Mantl?

Shipped is purpose built to make deploying docker applications super easy, so Mantl was built alongside it to make hosting these applications easy.

Shipped actually runs itself alongside other tenant applications on Mesos workers. Shipped allows developers to build their application into a docker image. Once you have a containerized app, you can deploy with the push of a button. Behind the scenes, this goes something like this:

1. POST a request to Marathon that starts at least one new task with the docker container. We include some special environment variables that are consulted by HAProxy-Consul later to get hostnames correct.
2. Marathon-Consul watches the Marathon event bus to update Consul with the running services and the ports they started on.
3. HAProxy-Consul re-writes the HAProxy configuration to route hostnames to the correct containers (for now this is a round-robin setup).
4. All the worker nodes run Logstash. It slurps up Mesos logs and output from each container's stdout/err and forwards them on to Elastic Search. This allows us to make deployment failures. It also gathers logs from every container and makes them available in the management dashboard
5. CAdvisor collects metrics from each container and sends them to Elastic Search for later consumption.

Mantl Architecture

Shipped wraps all this up for you in a tidy management interface:

How do I get started?

Cisco has prepared pretty solid getting started docs in the Mantl repository. You can provision a cluster on OpenStack, Google Compute Engine, Amazon Web Services, DigitalOcean, VMware vSphere or SoftLayer without too much trouble. After that, getting to know the tools and how they all fit together is still left as a bit of an exercise to the user. I've done my best to link to the components above, and the docs on the repo are constantly improving. The repo has been starred 1500 times, so get involved! You can also try it out from Shipped

Let me know your thoughts and give it a try!

Looking for help managing your highly available micro-services?

Reach out to Must Win!

-

Mike Ihbe is a founding partner of The Must Win All Star Web & Mobile Consultancy.

Mike is an expert in a dizzying array of technologies and has loads of experience managing fast-moving dev teams, designing systems, and scaling large applications. When not on the job, he can be found cooking delicious meals on ski slopes in exotic locales.