Building a self-contained Python 3 application using PyPy

Post to Twitter

A few articles ago I showed you how to create a Python 3 Microservice (a simple Minecraft service). At the end of that article I mentioned in the future I’d show you how to build a self-contained Python application/service. Well, today is that day.

I’ll use PyPy (the Python 3.2.5 compatible PyPy3 2.4.0 version) to show how this can easily be done and how you can install packages into the self-contained instance of Python. What this means is you easily package this up and simply extract it to another machine (same architecture or you need to modify some steps) and it will just run even if Python is not installed there. In fact there are several ways to do what I’m going to show you how to do today and several ways to package this up. I’m just going to focus on one possible way to do this as easily as possible.

Let’s create the project folder first:

$ mkdir pypy-self-contained
$ cd pypy-self-contained/

Copy the extracted contents of the PyPy package (Python 3.2.5 compatible PyPy3 2.4.0) http://pypy.org/download.html into the pypy-self-contained folder.

Now we can test if PyPy is working, so assuming OS X (if not make the appropriate change to the correct path):

$ ./pypy3-2.4.0-osx64/bin/pypy3 --version

Expected output:

Python 3.2.5 (b2091e973da6, Oct 19 2014, 18:30:58)
[PyPy 2.4.0 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)]

So far so good. Let’s add the rest of the files we will need:

$ mkdir myapp && cd myapp
& mkdir myapp
$ touch myapp/__init__.py
$ touch app.py
$ cd myapp
$ wget https://raw.githubusercontent.com/bottlepy/bottle/master/bottle.py
$ touch mtwsgi.py
$ cd ..

Your project structure should be like this:

Project Structure

Note: Ignore the __pycache__ folder as it gets generated automatically when you run the app.

Add the following to mtwsgi.py:

"""
WSGI-compliant HTTP server.  Dispatches requests to a pool of threads.
"""
 
from wsgiref.simple_server import WSGIServer, WSGIRequestHandler
import multiprocessing.pool
 
__all__ = ['ThreadPoolWSGIServer', 'make_server']
 
 
class ThreadPoolWSGIServer(WSGIServer):
    """
    WSGI-compliant HTTP server. Dispatches requests to a pool of threads.
    """
 
    def __init__(self, thread_count=None, *args, **kwargs):
        """
        If thread_count == None, we'll use multiprocessing.cpu_count() threads.
        """
        WSGIServer.__init__(self, *args, **kwargs)
        self.thread_count = thread_count
        self.pool = multiprocessing.pool.ThreadPool(self.thread_count)
 
    # Inspired by SocketServer.ThreadingMixIn.
    def process_request_thread(self, request, client_address):
        try:
            self.finish_request(request, client_address)
            self.shutdown_request(request)
        except:
            self.handle_error(request, client_address)
            self.shutdown_request(request)
 
    def process_request(self, request, client_address):
        self.pool.apply_async(
            self.process_request_thread, args=(request, client_address))
 
 
def make_server(
        host, port, app, thread_count=None, handler_class=WSGIRequestHandler):
    """
    Create a new WSGI server listening on `host` and `port` for `app`
    """
    httpd = ThreadPoolWSGIServer(thread_count, (host, port), handler_class)
    httpd.set_app(app)
    return httpd

Add the following to app.py:

"""
Multithreading Bottle server adapter
"""
from myapp import bottle
from myapp import mtwsgi


class MTServer(bottle.ServerAdapter):
    def run(self, handler):
        thread_count = self.options.pop('thread_count', None)
        server = mtwsgi.make_server(
            self.host, self.port, handler, thread_count, **self.options)
        server.serve_forever()


if __name__ == '__main__':
    app = bottle.Bottle()

    @app.route('/')
    def version():
        app_ver = {
            "version": "1.0.0"
        }
        return app_ver

    app.run(server=MTServer, host='0.0.0.0', port=8081, thread_count=3)

Run the wsgi app:

$ ../pypy3-2.4.0-osx64/bin/pypy3 app.py

With a browser or cURL hit the URL: http://0.0.0.0:8081/

Expected output:

{"version": "1.0.0"}

That works well for a simple app, but what if you need a Python package installed?

$ cd ..
(you should be in the "pypy-self-contained" folder now)
$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$ ./pypy3-2.4.0-osx64/bin/pypy3 get-pip.py
(now you can install whatever packages you want)
$ ./pypy3-2.4.0-osx64/bin/pip install PACKAGE-NAME-HERE

So a super easy example would be to install the markdown package and use it from the app:

$ ./pypy3-2.4.0-osx64/bin/pip install markdown
$ cd myapp

Modify app.py to:

"""
Multithreading Bottle server adapter
"""
from myapp import bottle
from myapp import mtwsgi

import markdown


class MTServer(bottle.ServerAdapter):
    def run(self, handler):
        thread_count = self.options.pop('thread_count', None)
        server = mtwsgi.make_server(
            self.host, self.port, handler, thread_count, **self.options)
        server.serve_forever()


if __name__ == '__main__':
    app = bottle.Bottle()

    @app.route('/')
    def version():
        html = markdown.markdown('your text string')
        print(html)
        app_ver = {
            "version": "1.0.0"
        }
        return app_ver

    app.run(server=MTServer, host='0.0.0.0', port=8081, thread_count=3)

Run app.py:

$ ../pypy3-2.4.0-osx64/bin/pypy3 app.py

Refresh your browser if it’s still open (http://0.0.0.0:8081/)

Output in console:

<p>your text string</p>

A very trivial example but this shows the basics on how you can use PyPy to host a completely self-contained Python application. If you haven’t done so, please consider a donation to PyPy.

Post to Twitter

This entry was posted in Open Source, Python. Bookmark the permalink.