This content is part of the Essential Guide: The essential guide to open source virtualization platforms

Apache Mesos better utilizes resources, improves scalability

Apache Mesos has become popular with major enterprises for its ability to scale linearly, reduce wasteful resource consumption and abstract storage, CPU and memory.

Created by developers at the University of California, Berkeley and embraced by major enterprises, including Twitter, Apple and Netflix, Apache Mesos is open source software that abstracts storage, CPU and memory across a cluster of machines. One of the major draws of Apache Mesos is that it scales linearly, meaning that as the load level increases, response time increases proportionally -- essentially, scaling without limit. Mesos refers to itself as a "distributed systems kernel" because it takes the core principles of a Linux kernel, but applies them to a different level of abstraction.

Stop wasting VM resources

Data center administrators underutilize VMs all too often, causing organizations to waste money on resources they aren't using. This issue is usually addressed through partitioning, which sets aside a specific set of servers to run specific functions. In a traditional environment, you would use partitioning to define a requirement, such as the need for a specific amount of servers, and then assign VMs and storage accordingly. In a public cloud -- where users are billed on a resource usage basis and don't have access to the cloud OS -- partitioning applies larger or smaller templates to the VM configuration.

As you can imagine, an enterprise as large as Uber has some heavy-duty data processing needs. Uber uses the Apache Cassandra database -- a NoSQL column-oriented database -- to store location data.

While this method is effective enough, it's more practical to colocate services, which is where Apache Mesos enters the picture. Partitioning dedicates a machine to a specific task, such as a database server, and another to run, say, a web server. Colocating is more efficient, because it allows you to run more than one service on one VM or server. It also cuts down costs, because running more than one service on the same server reduces the number of servers you require. Rather than relying on partitioning to run services, Mesos uses colocation to allow the software to take resources on an as-needed basis. Put in technical terms, Mesos replaces whichever resource manager you're using with its own framework and implements scheduling and execution interfaces.

Mesos works with individual software, Docker containers and big data clusters that are configured to use Mesos as the resource manager. Apache Mesos is not an orchestration system for VMs. Mesos also uses Linux control groups, also known as cgroups, to limit resources, prioritize processes and do accounting. This is useful in the public cloud because it allows vendors to charge customers based on how many resources they use. Cgroups are helpful in traditional environments as well, because they can limit processes so they don't take over a machine.

Solve the problem of partitioning with Mesos and YARN

Apache Hadoop Yet Another Resource Negotiator (YARN), the resource manager for Apache Hadoop MapReduce, performs roughly the same function as Mesos. In fact, Myriad, an open source project, enables data centers to use both products at the same time. You would use Mesos and YARN together if, for example, you wanted to run container applications with Mesos, but use YARN to run Hadoop.

If we take a closer look at why Yahoo rewrote Hadoop to add YARN, we can get a better understanding of what both Mesos and YARN do.

Programming the framework for YARN is a complex task, one better suited for engineers of large software products, like Apache Spark, than end users. The Hadoop configuration makes it much easier to use YARN. In a clustered environment, you can simply edit a few configuration files on the name node and then copy the entire Hadoop installation to the date nodes -- YARN works without any further configuration changes needed.

The major problem with earlier releases of Hadoop was partitioning. With partitioning, you can designate slots to run either map jobs or reduce jobs to a machine in a Hadoop cluster. Once you're assigned a slot to run a map job, you can't use it to run a reduce job, and vice versa. Suppose you've assigned 10 slots to run map jobs and 10 slots to reduce jobs to a machine in a Hadoop cluster. Now, let's say Hadoop needs to run 11 map jobs -- you'll find yourself in a bit of a bind because you haven't allocated enough slots for map jobs, and you can't use any of the additional slots allotted for reduce jobs. This problem made it clear that there needed to be a better way to colocate services and do away with partitioning. Apache responded by making it so YARN and Mesos could dole out services.

The Apache Mesos architecture

The Apache Mesos architecture consists of a master daemon, which manages the agent daemons running on each cluster node. The agent daemons also use cgroups to keep them working within their allocated memory, CPU and storage. Each of these agents uses a Mesos framework to run tasks. This framework is made up of two components: the scheduler, which registers with the master to receive resources, and the executor, which takes these resources from the scheduler and uses them to run the framework's tasks. Essentially, the executor identifies whichever application you're running that resources are available.

Mesos and container orchestration

Much like Kubernetes and Docker Swarm, Mesos also performs container orchestration. Mesos works with three types of container technologies: Composing, which allows different container technologies to run together, Docker and Mesos's own containerization, which is the default configuration.

Popular Mesos frameworks for container orchestration

Developed by Twitter to run stateless services, like Java VMs and web servers, Apache Aurora is a framework designed for both long-running and cron jobs. Apache Chronos is an elastic distributed system that expresses dependencies between jobs. Written by Mesosphere, Apache Marathon is a container orchestration system that can scale to thousands of physical servers. Aurora, Chronos and Marathon all interface with Mesos using JSON and a REST API.

One of the Twitter engineers who helped create Mesos said at DockerCon that he recommended Mesos users use a minimum of three machines in their home lab to run a working Mesos prototype. Apache offers the minimesos experimentation and testing tool for those interested in trying out Mesos. Minimesos includes a bash shell in which you can deploy a Mesos Elasticsearch framework.

Major companies embrace Mesos

As you can imagine, an enterprise as large as Uber has some heavy-duty data processing needs. Uber uses the Apache Cassandra database -- a NoSQL column-oriented database -- to store location data. A column-oriented database writes one row/column combination at a time rather than writing an entire row of columns, so it wastes no space on empty columns. A column-oriented database also keeps columns together for rapid retrieval.

In addition to relying on Mesos for tracking data, Uber also regularly contributes code to Mesos. In 2016, Uber wrote a framework for Apache Cassandra, which makes it easier to deploy Cassandra on DC/OS, an open source distribution for Apache Mesos. You can run Hadoop, Spark, Cassandra and more on top of DC/OS because it's extendable.

Netflix is also a major user of and contributor to Apache Mesos. Netflix says it runs on Amazon Elastic Cloud Compute (EC2) and uses Mesos to deliver "fine-grained resource allocation to tasks of various sizes that can be bin packed to a single EC2 instance." In 2015, Netflix developed Fenzo, an open source scheduler for Apache Mesos frameworks. Fenzo manages scheduling and resource assignment for deployments and adds cluster auto scaling to Mesos.

Next Steps

Twitter eradicates bottlenecks with Apache Mesos

The past, present and future of Apache Hadoop

YARN broadens what Apache Hadoop can do

Dig Deeper on Open source virtualization