This is an integration framework which allows running and managing Crate database through Mesos.

First, the jar file needs to be built from source:

./gradlew fatJar

The framework can be deployed in two ways: (1) executing the jar file from the command line and (2) launching it via Marathon.

If the framework is deployed using Marathon the jar needs to be copied to the mesos-slaves. Otherwise it needs to be copied to the mesos-master from where it is launched using the command line.

In both cases the Main method requires a --crate-version argument, which is the Crate version that should be used. The version number must be in the format x.y.z. Current Crate version is 0.47.8.

Alternatively you can specify a full download URL for Crate. In that case the URL needs to be in the format: http(s)://<HOST><PATH>/crate-<X>.<Y>.<Z><SUFFIX>.tar.gz

java -Djava.library.path=/usr/local/lib -jar /path/to/crate-mesos-0.x.x.jar --crate-version 0.x.x [OPTIONS]"

Create a Marathon configuration file:

{
  "id": "/crate/dev",
  "instances": 1,
  "cpus": 0.5,
  "mem": 256,
  "ports": [0],
  "uris": [],
  "env": {},
  "cmd": "java -Djava.library.path=/usr/local/lib -jar /path/to/crate-mesos-0.x.x.jar --crate-version 0.x.x [OPTIONS]"
}

For this to work java needs to be available on the mesos-slave. If java isn't available it can be included as dependency in the Marathon configuration file by listing it in uris and by changing the cmd:

"uris": [
    "https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz"
],
"cmd": "$(pwd)/jre*/bin/java $JAVA_OPTS -jar /path/to/crate-mesos-0.x.x.jar --crate-version 0.47.7",

And submit it to a running Marathon master:

curl -s -XPOST http://localhost:8080/v2/apps -d@CrateFramework.json -H "Content-Type: application/json"

Crate has 2 options for persistent data paths: one for data (tables) and one for blobs.

You can specify both paths (--crate-data-path, --crate-blob-path) when starting the framework.

If the paths are specified, the executor will check if the path exists on the slave. If the path does not exist, the executor won't start Crate on that slave.

Configuration options for crate instances can also be passed to the framework. These options will be passed to the Crate processes which are launched by the framework.

All options starting with -Des. are considered crate configuration options.

For example in order to get the framework to launch instances that will have stats-collecting enabled use the following:

java ... -jar crate-mesos-0.x.x.jar --crate-version 0.x.x -Des.stats.enabled=true

The Crate Framework is run as user ``crate`` and role ``*`` by default. However, this is configurable using ``--framework-user`` and ``--framework-role``.

This means, that a user crate (or other specified user) is required to be present on all instances, both master and slaves. The user does not need to have any specific permissions. You can add a user with the simplest configuration:

useradd crate -s /bin/bash

If you specify a role different to the default "*" you need to add it the mesos-master configuration, e.g.:

echo "crate" > /etc/mesos-master/roles

If you are using Persistent Data Paths (which is recommended), you need to make sure that the user crate has write permissions at these locations. For example:

chown crate:crate /path/to/persistent/disk

Crate uses by default a the ports 4200 and 4300. In order to get offers you need to add the resource reservation for a port range that includes these ports, e.g. writing it into the resources file:

echo 'ports(*)[31000-31099, 31101-32000, 4200-4399]' > /etc/mesos-slave/resources

or starting the slave with the option:

--resources=ports(*)[31000-31099, 31101-32000, 4200-4399]

Then restart the slave and clean the old slave state if necessary (rm -f /tmp/mesos/meta/slaves/latest).

The ports can be configured on startup of the Framework, which means that you need adopt the resource port range according to your configured ports.

The API is availble on port 4040 (default, but can be set via the --api-port command line option).

You can get information about the cluster from the /cluster endpoint:

curl -X GET http://localhost:4040/cluster

You can resize the cluster by setting the number of desired instances:

curl -X POST -H "Content-Type: application/json" localhost:4040/cluster/resize -d '{"instances": 5}'

You can force shut down the cluster:

curl -X POST http://localhost:4040/cluster/shutdown

A Crate cluster can be resized by changing the number of instances using the Framework API (see API Usage).

Increasing the number of instances is always possible, unless the number of desired instances is greater than the number of slaves. Each instance of the Crate Framework enforces the contraint that there is only one Crate instance prunning on each host.

The Crate Framework shuts down Crate instances gracefully (see Configuration and Zero Downtime Upgrade) when decreasing the number of instances in a cluster.

If you want to ensure green health (full data + replica availability), you need to change the cluster.graceful_stop.min_availability setting to full. This option will cause the Crate node to try move all shards off the node before shutting down. If this is not possible, the node will not shut down and run into the timeout (cluster.graceful_stop.timeout). However the Crate Framework will continue to try to shut down the node again. Such a state is indicated by the Framework API when the number of running instances does not approach the number of desired instances when scaling down. Please keep in mind that the cluster can not be resized to zero instances.

In order to shut down the a cluster you need to use the /cluster/shutdown API endpoint.

A zero downtime upgrade of a Crate cluster running on Mesos is currently not possible, however it is still possible to upgrade the cluster with downtime.

An upgrade requires a few steps:

Assuming you've started the Crate Framework with version 0.47.7 and want to upgrade to version 0.47.8 (or any other greater version), you will first need to set the minimum availability to full (see Resizing a Cluster) if not already done. Also check to other options for graceful shutdown.

This will ensure that you are able to resize your cluster to the minimum amount of nodes.

The minimum amount of nodes is equal the highest number of replicas of a table plus 1:

min_nodes = max_replicas + 1

E.g. if your cluster has 5 nodes and your table with the most configured replicas has 2 replicas, you can resize your cluster down to 3 nodes.

It is highly recommended to shut down Crate nodes one by one! In this way you are in better control if a node does not shut down gracefully, e.g. runs into the timeout.

Now you can re-start the Crate Framework with the new Crate version number. The Crate instances with the old version are still running at this point. If you'd upscale your cluster, new Crate instances would still use the old version, but that is not what we want.

In order to be able to use the new version set with the restarted framework, you need to kill the remaining instances using the /cluster/shutdown API endpoint.

Once there are no more instances, you can resize the cluster and new Crate instances will use the new version from the framework.

Because the framework stores the information on which slaves Crate instances with data were running and when you up-scale the cluster again, it will prefer offers from these slaves.

In order for applications to discover the Crate nodes Mesos-DNS can be used.

If Mesos-DNS is running it will automatically retrieve information about the instances launched by the Crate framework and then the client applications can connect to the crate cluster using the following URL: <cluster-name>.crateframework.<domain>:<http-port>

Both <cluster-name> and <http-port are options that can be specified when the Mesos Crate Framework is launched. The <domain> is part of the Mesos-DNS configuration.

One Crate Framework can only be used to manage one crate cluster.In order to be able to manage multiple crate clusters it is possible to run the crate framework multiple times.

The easiest and recommended way to do so is to deploy the Crate Framework using Marathon. This also has the advantage that the Crate Framework itself will be HA.

In order to deploy something on Marathon create a json file. For example crate-mesos.json with the following content:

{
    "id": "crate-demo",
    "instances": 1,
    "cpus": 0.25,
    "mem": 50,
    "ports": [4040],
    "cmd": "java -Djava.library.path=/usr/local/lib -jar /tmp/crate-mesos-0.1.0.jar --zookeeper mesos-master-1:2181,mesos-master-2:2181,mesos-master-3:2181 --crate-cluster-name crate-demo --crate-version 0.47.7 --api-port $PORT0",
    "healthChecks": [
        {
            "protocol": "HTTP",
            "path": "/cluster",
            "gracePeriodSeconds": 3,
            "intervalSeconds": 10,
            "portIndex": 0,
            "timeoutSeconds": 10,
            "maxConsecutiveFailures": 3
        }
    ]
}

In order to instruct marathon to deploy the crate framework curl can then be used:

curl -s -XPOST http://marathon-url:8080/v2/apps -d@crate-mesos.json -H "Content-Type: application/json"

If Mesos-DNS is available the launched Crate Framework can then be accessed using crate-demo.marathon.mesos. Where crate-demo is the id specified in the crate-mesos.json and mesos is the configured Mesos-DNS domain.

nslookup -querytype=srv _crate-demo._tcp.marathon.mesos

Now for each additional cluster an additional "crate framework app" can be deployed using Marathon. Keep in mind that each cluster should have its unique ports so the port configuration options should be set in each clusters cmd definition.

Any attributes that are defined on a Mesos-Slave will be passed to the Crate processes as node tag with a mesos_ prefix.

For example if a Mesos-Slave is launched with --attributes=zone:a the Crate instance will have the node.mesos_zone=a tag set.

This is can be used to setup a Multi Zone Crate Cluster.

Assuming there are 4 slaves, 2 with the attribute zone:a and 2 with the attribute zone:b. In this case the framework would have to be launched with the following options to have a working multi zone setup:

java ... -jar crate-mesos-0.1.0.jar --crate-version x.x.x \
    -Des.cluster.routing.allocation.awareness.attributes=mesos_zone \
    -Des.cluster.routing.allocation.awareness.force.mesos_zone.values=a,b

• As there is no official crate-mesos release yet the jar file isn't hosted but needs to be built locally and somehow copied to the slaves.
• There is no automatic handling of cluster failures.
• The overall cluster health needs to be monitored separately, using the Crate Admin UI (running on port 4200 at path /admin) or other third party tools.
• The cluster does not automatically resize depending on used resources.
• Although Crate requires a minimum disk size to start, the disk usage is not monitored inside the framework further more. This can be done using the Admin UI or plain SQL.
• A zero-downtime upgrade is not possible at the moment.

You can build Crate-Mesos-Framework on your own with the latest version hosted on GitHub. To do so, please refer to DEVELOP.rst for further information.