Talking HTTP

Intro to Web Modules

Me

  • I work at uStudio.
  • We make video management easy.
  • We are always looking for a few good devs.

WSGI

Web Server Gateway Interface

Why Do You Care?

Web applications are just interfaces

Browsers are ubiquitous GUI frameworks (tk sucks)

Managing cloud operations (beats screen sessions)

WSGI is a PEP. (You want to know them all, right?)

Pronunciation

Some people pronounce it "Whiz-gi".

(rhymes with whiskey)

I don't.

Web Servers for Non-Webheads

HTTP is simple in theory.

Accept a socket connection

Parse request payload

Generate response

Buffer response payload

Close socket

Request Example


POST /articles HTTP/1.1
User-Agent: my-awesome-client/1.3
Content-Type: application/json
Content-Length: 49
Host: www.myblog.com
Accept: */*

{"title": "foobar", "content": "This is a post."}

Response Example


HTTP/1.1 201 Created
Date: Wed, 10 Jul 2013 02:03:00 GMT
Expires: -1
Content-Type: application/json
Content-Length: 37

{"status": 201, "message": "created"}

The Reality


sock = socket(AF_INET, SOCK_STREAM)
sock.bind(("", 8080))
sock.listen(5)

while True:
    (client, address) = sock.accept()
    body = client.recv(BUFFER_SIZE)
    while "\r\n\r\n" not in body:
        body += client.recv(BUFFER_SIZE)
    headers, body = body.split("\r\n\r\n")
    method, path, headers = extract_headers(headers)
    content_length = int(headers["content-length"]) - len(body)
    if content_length:
        body += client.recv(content_length - len(body))
    client.send(handle_request(method, path, headers, body))
    client.close()

Lots of Concerns...

  • thread pool
  • non-blocking
  • routing
  • special headers
  • keep alive
  • buffering
  • network issues
  • static files
  • caching
  • middleware
  • ssl
  • stability
  • proxies
  • expiration
  • range requests

Not Invented Here

The year was 2003...

The dark ages of Python on the web.

Everybody rolled their own thing.

Twisted Web, Zope, Webware, etc.

Custom servers, CGI, "proprietary" middleware...

Open-source vendor lock in

Enter WSGI

Stupid-Simple Standard

Ideas of WSGI

Took cues from Java servlet standardization

Minimal design implications for existing frameworks

Extensible for future hotness

Easy to understand / work with for (early) adopters.

WSGI is a function

The most basic app just takes two arguments.

  • environ is a dictionary of request information
  • start_response handles... starting the response.
  • The return value is an iterable of strings.

WSGI is a function


def application(environment, start_response):
    status = "200 OK"
    headers = [("Content-Type", "application/json")]
    body = json.dumps({"status": 200, "message": "ok"})
    start_response(status, headers)
    yield body

environ contains... a lot.

  • Apple_PubSub_Socket_Render: /tmp/launch-XuHI0f/Render
  • Apple_Ubiquity_Message: /tmp/launch-SdX70z/Apple_Ubiquity_Message
  • COMMAND_MODE: unix2003
  • CONTENT_LENGTH: 21
  • CONTENT_TYPE: application/json
  • DISPLAY: /tmp/launch-yI0RmK/org.macosforge.xquartz:0
  • GATEWAY_INTERFACE: CGI/1.1
  • HOME: /Users/jmarshall
  • HTTP_ACCEPT: */*
  • HTTP_ACCEPT_ENCODING: gzip,deflate,sdch
  • HTTP_ACCEPT_LANGUAGE: en-US,en;q=0.8
  • HTTP_CONNECTION: keep-alive
  • HTTP_HOST: localhost:8000
  • HTTP_ORIGIN: http://localhost:8000
  • HTTP_REFERER: http://localhost:8000/
  • HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4)
  • ITERM_PROFILE: Default
  • ITERM_SESSION_ID: w0t1p1
  • LANG: en_US.UTF-8
  • LOGNAME: jmarshall
  • MXMLC_PATH: ~/bin/flex/bin/mxmlc
  • OLDPWD: /Users/jmarshall/Presentations/APUG-20130709/code
  • PATH:
  • PATH_INFO: /process
  • PWD: /Users/jmarshall/Presentations/APUG-20130709/code/wsgi_process
  • QUERY_STRING:
  • REMOTE_ADDR: 127.0.0.1
  • REMOTE_HOST: 1.0.0.127.in-addr.arpa
  • REQUEST_METHOD: POST
  • SCRIPT_NAME:
  • SECURITYSESSIONID: 186a5
  • SERVER_NAME: josh-macbook-pro.local
  • SERVER_PORT: 8000
  • SERVER_PROTOCOL: HTTP/1.1
  • SERVER_SOFTWARE: WSGIServer/0.1 Python/2.7.2
  • SHELL: /bin/bash
  • SHLVL: 1
  • SSH_AUTH_SOCK: /tmp/launch-4QZWdj/Listeners
  • TERM: xterm
  • TERM_PROGRAM: iTerm.app
  • TMPDIR: /var/folders/f_/7fnqd5r14zngq914cxzt_6q00000gn/T/
  • USER:
  • VERSIONER_PYTHON_PREFER_32_BIT: no
  • VERSIONER_PYTHON_VERSION: 2.7
  • VIRTUALENVWRAPPER_HOOK_DIR: /Users/jmarshall/.virtualenvs
  • VIRTUALENVWRAPPER_PROJECT_FILENAME: .project
  • WORKON_HOME: /Users/jmarshall/.virtualenvs
  • _: /usr/bin/python
  • __CF_USER_TEXT_ENCODING: 0x1F5:0:0
  • wsgi.errors: <open file '<stderr>', mode 'w' at 0x1069ee270>
  • wsgi.file_wrapper: wsgiref.util.FileWrapper
  • wsgi.input: <socket._fileobject object at 0x106aec7d0>
  • wsgi.multiprocess: False
  • wsgi.multithread: True
  • wsgi.run_once: False
  • wsgi.url_scheme: http
  • wsgi.version: (1, 0)

Valuable Keys

Some of the more valuable entries in environ

  • PATH_INFO: Contains the path part of a URL
  • REQUEST_METHOD: "GET", "POST", etc.
  • QUERY_STRING: Everything past the ?
  • REMOTE_ADDR: The client IP address
  • wsgi.input: File-like object (request body)

WSGI is a callable

Raw WSGI makes simple apps very easy

Makes complex apps... interesting.

Most modern frameworks support this at the core.

Most abstract away the application itself.

Request and response objects, templating...

More Complicated Example

Show JSON Example

Complications Arise

  • Raw WSGI is great if you want to...
  • roll your own routing
  • parse request arguments yourself
  • tightly control body parsing
  • intercept and redirect application behavior
  • rule your own world
  • tear all your hair out
  • It's a bad choice if you want to just 'get stuff done'.

WSGI is an adapter

Really, WSGI enables frameworks.

Abstract container, support custom interfaces.

Most basic are wrappers like WebOb and Werkzeug

Simple frameworks like Flask to monsters like Django.

Roll Your Own

There are a lot of pieces out there.

Use what you know, or what you want to learn.

Jinja2, Cheetah, Mako, etc. for templates.

SQLAlchemy, Storm, MongoEngine, etc. for models.

Werkzeug, custom regex, etc. for routing.

Custom Router

Example of Simple "Roll Your Own"

Use Frameworks that Support WSGI

It's hard to find a framework that DOESN'T.

Your usage may constrain what frameworks to use.

Limited dependencies? Raw WSGI.

Simple APIs? Flask.

Content management / massive app? Django.

Middleware Changes Things

One of the more powerful aspects of WSGI

You can nest apps.

Environ can be modified.

You can call start_response multiple times.

This means, you can intercept and change things.

Simple Proxy Example


def proxy_host(app):
    def proxy_app(environ, start_response):
        if "HTTP_X_FORWARDED" in environ:
            environ["REMOTE_ADDR"] = environ["HTTP_X_FORWARDED"]
        return app(environ, start_response)
    return proxy_app

@proxy_host
def application(environ, start_response):
    # do actual stuff...

Extending Environ

Some WSGI adapters extend functionality.


# wrapped with gevent's websocket handler
def websocket_app(environ, start_response):
  socket = environ["wsgi.websocket"]
  socket.send(json.dumps({"foo": "bar"})

Servers

There are LOTS of WSGI deployment options.

gevent, mod_wsgi, gunicorn, cgi...

Depends on what you want to do.

Frequently middleware will be involved.

WSGIREF Example

For development ONLY


from wsgiref.simple_server import make_server
def app(environ, start_response):
    start_response("200 OK", [("Content-Type", "text/json")])
    yield "Hello World!"

server = make_server("", 8000, app)
server.serve_forever()

Simple Gunicorn Example

Running an existing app

Links

IF you want to do raw WSGI... wsgiref

Werkzeug utilities are very helpful

Heroku deployment is easy here.

Questions?

@joshmarshall on the Twitters.

@joshmarshall on the GitHub.