Running a Python 3 REST application with Statsd and Consul support in Docker

Post to Twitter

Where I currently work we build a lot of RESTful services in either Go (golang) or Python. Many of these services are deployed to Apache Mesos via Docker containers with everything tied up nicely using tools like Marathon and more. For our Mesos based services we also like to track metrics as well as have these services load their configurations from Consul which in turn helps keep our services as ephemeral as possible.

Today, I’m going to go over setting up a very simple way to spin up a Python Falcon REST service that reports to Statsd as well as registering as a service with Consul along with setting up health checks. I’ve borrowed some ideas/code from several places and changed as needed.

Docker Images Used:
https://hub.docker.com/r/kamon/grafana_graphite/
https://hub.docker.com/_/consul/

Note: What follows is a simple/basic way to get up and running for development. I’ve purposely kept things very simple (no Consul cluster, using WSGIref to run the REST service, Python code all in one file, etc.) so take this as a starting point and something to play with to get familiar then adjust as needed to suit your production deployments if applicable.

First, I setup a Python 3 venv to work inside of. The steps on how I do this can be found in this article.

Inside the new src folder (or whatever your using) I add a folder called python_app and then a file called Dockerfile which for this scenario kicks off the WSGIref server (good for development, not your best choice for production).

FROM python:3.4.4-onbuild
EXPOSE 8080
CMD ["python", "./app.py"]

Next, I crafted a docker-compose.yml file and added the images I want to use and build.

version: "2"
services:
  app1:
    depends_on:
      - consul1
    build: ./python_app/
    ports:
      - "8080:8080"

  consul1:
    image: "consul:latest"
    container_name: "consul1"
    hostname: "consul1"
    ports:
      - "8400:8400"
      - "8500:8500"
      - "8600:53"
    command: "agent -dev -client=0.0.0.0 -bind=127.0.0.1"

  grafana-statsd:
    image: "kamon/grafana_graphite"
    container_name: "grafana-statsd"
    hostname: "grafana-statsd"
    ports:
      - "80:80"
      - "8125:8125/udp"
      - "8126:8126"

We haven’t added the Python code yet so lets start with the consul1 and grafana-statsd services.

For Consul its just the single non-clustered server running in development mode. This is using the recently released official Consul image. Ports we want to expose are set and not much more is needed here.

Moving onto the grafana-statsd image we expose the ports and all the other hard work has already been done.

Now we need a Python service. Add a file called app.py in the python_app folder.

import json
from time import sleep
from wsgiref import simple_server

import falcon
import requests
from statsd import StatsClient


_statsd_client_singleton = None


def init_statsd_client(host='grafana-statsd', port=8125,
                       prefix='PythonApp.app1'):

    global _statsd_client_singleton

    if not bool(_statsd_client_singleton):
        _statsd_client_singleton = StatsClient(host, port, prefix)

    return _statsd_client_singleton


class BaseResource(object):
    def __init__(self):
        self.statsd_client = init_statsd_client()


class HomeResource(BaseResource):
    def on_get(self, req, resp):
        self.statsd_client.incr('HomeResource')
        resp.status = falcon.HTTP_200
        resp.body = 'Hello World'


class HealthResource(BaseResource):
    def on_get(self, req, resp):
        data = {
            'status': 'healthy'
        }
        resp.body = json.dumps(data)


def register_service():
    url = 'http://consul1:8500/v1/agent/service/register'
    data = {
        'id': 'app1',
        'tags': ['v1'],
        'name': 'PythonApp',
        'address': 'app1',
        'check': {
            'http': 'http://app1:{port}/health'.format(port=8080),
            'interval': '10s'
        }
    }
    resp = requests.put(
        url,
        data=json.dumps(data)
    )
    return resp.text

app = falcon.API()


if __name__ == '__main__':
    sleep(10)

    home = HomeResource()
    health = HealthResource()

    app.add_route('/', home)
    app.add_route('/health', health)

    try:
        print(register_service())
    except Exception as ex:
        print(ex)

    httpd = simple_server.make_server('0.0.0.0', 8080, app)
    httpd.serve_forever()

Note: Keep in mind for this simple example I’m using the WSGIref server (development). Also, in the register_service function the call to register the service in Consul and setup health checks is done by manually building the URL and JSON we will pass. This for learning is more clear of whats going on. Feel free to utilize a Python library to handle this for you: Python-Consul or Consulate.

Three resources are setup. The BaseResource sets up the statsd client so we can re-use that. The HomeResource simply returns hello world when called. The HealthReource returns the service’s status (hard coded right now to always be healthy). This Python 3 REST service is using the very fast Falcon Framework (which recently went v1.0).

You might be curious about this line of code:

sleep(10)

Typically if you are spinning up Consul the same time as your service that wants to register there is a bit of a lag time before Consul will actually listen. Usually a few seconds works, I use 8-10 seconds to play it safe. You could do a limited retry here if you wanted in a production scenario.

The only other file needed in the python_app folder is the requirements.txt with the following contents:

falcon
requests
statsd

Incidentally my folder layout looks like this:

Project Layout

Now you should be able to run everything.

$ docker-compose up

You can then hit the HomeResource endpoint a few times to generate some statsd traffic and verify the Consul health checks are working.

Home Resource: http://127.0.0.1:8080/
Consul: http://127.0.0.1:8500/ui/#/dc1/services
Grafana: http://127.0.0.1 (make sure to add a Graphite datasource and then add a new dashboard with a graph for your metrics)

Consul Python App Status

Grafana PythonApp App1 Metrics

This offers a lot of possibilities and I’ve hopefully kept it simple enough to get started easily.

Post to Twitter

This entry was posted in Open Source, Python. Bookmark the permalink.