Wherein we discover the gateways to dynamic processes on a server.

Homework Review and Questions

• You've learned about passing messages back and forth with sockets
• You've created a simple HTTP server using sockets
• You may even have made your server dynamic by returning the output of a python script.

A computer has an environment:

$ printenv
TERM_PROGRAM=iTerm.app
...

C:\> set
ALLUSERSPROFILE=C:\ProgramData
...

PS C:\> Get-ChildItem Env:
ALLUSERSPROFILE             C:\ProgramData
...

These new values are now part of the environment

$ printenv
...
VARIABLE=some value

C:\> set
...
VARIABLE='some value'

PS C:\> Get-ChildItem Env:
...
VARIABLE                    'some value'

We can see this environment in Python, too:

$ python

>>> import os
>>> print(os.environ['VARIABLE'])
some_value
>>> print(os.environ.keys())
['VERSIONER_PYTHON_PREFER_32_BIT', 'VARIABLE',
 'LOGNAME', 'USER', 'PATH', ...]

You can alter os environment values while in Python:

>>> os.environ['VARIABLE'] = 'new_value'
>>> print(os.environ['VARIABLE'])
new_value

>>> ^D

$ echo this is the value: $VARIABLE
this is the value: some_value
<OR>
C:\> \Users\Administrator\> echo %VARIABLE%
'some value'

CGI is little more than a set of standard environmental variables

First discussed in 1993, formalized in 1997, the current version (1.1) has been in place since 2004.

From the preamble:

This memo provides information for the Internet community. It does not
specify an Internet standard of any kind.

-- RFC 3875 - CGI Version 1.1: http://tools.ietf.org/html/rfc3875

You have a couple of options:

In the class resources for this session, you'll find a directory named cgi.

• Your browser might show a 404 or 403 error
• If you see something like that, check the permissions for cgi-bin and cgi_1.py
• The file must be executable, the cgi-bin directory needs to be readable and executable.

Problems with permissions can lead to failure. So can scripting errors

1 / 0

CGI is famously difficult to debug. There are reasons for this:

• CGI is designed to provide access to runnable processes to the internet
• The internet is a wretched hive of scum and villainy
• Revealing error conditions can expose data that could be exploited

Back in your editor, add the following lines, just below import cgi:

import cgitb
cgitb.enable()

Let's fix the error from our traceback. Edit your cgi_1.py file to match:

#!/usr/bin/env python
import cgi
import cgitb

cgitb.enable()

cgi.test()

Servers like http.server --cgi run CGI scripts as a system user called nobody.

$ ./cgi_bin/cgi_1.py

There are a couple of important facts about CGI that derive from this:

• The script must include a shebang so that the system knows how to run it.
• The script must be executable.
• The executable named in the shebang will be called as the nobody user.
• This is a security feature to prevent CGI scripts from running as a user with any privileges.
• This means that the executable from the script shebang must be one that anyone can run.

CGI is largely a set of agreed-upon environmental variables.

Let's find 'em. In a terminal fire up python:

In [1]: from http import server
In [2]: server.__file__
Out[2]: '/Users/cewing/pythons/parts/opt/lib/python3.5/http/server.py'
In [3]: !subl '/Users/cewing/pythons/parts/opt/lib/python3.5/http/server.py'

From http/server.py, in the CGIHTTPRequestHandler class, in the run_cgi method:

env = copy.deepcopy(os.environ)
env['SERVER_SOFTWARE'] = self.version_string()
env['SERVER_NAME'] = self.server.server_name
env['GATEWAY_INTERFACE'] = 'CGI/1.1'
...
if self.have_fork:
    # Unix -- fork as we should
    ...
    pid = os.fork()
    ...
    try:
        ...
        os.execve(scriptfile, args, env)
    ...
else:
    # Non-Unix -- use subprocess
    import subprocess
    ...
    p = subprocess.Popen(cmdline,
                         ...
                         env = env
                         )
...

And that's it, the big secret. The server takes care of setting up the environment so it has what is needed.

def test(environ=os.environ):
    ...
    print("Content-type: text/html")
    print()
    try:
        form = FieldStorage()   # Replace with other classes to test those
        print_directory()
        print_arguments()
        print_form(form)
        ...
    except:
        print_exception()

What the Server Does:

• parses the request
• sets up the environment, including HTTP and SERVER variables
• sends a HTTP/1.1 200 OK\r\n first line to the client
• figures out if the URI points to a CGI script and runs it
• appends what comes from the script on stdout and sends that back

What the Script Does:

• names appropriate executable in the shebang line
• uses os.environ to read information from the HTTP request
• builds any and all extra HTTP Headers
  (Content-type:, Content-length:, ...)
• prints the headers, empty line and script output (body) to stdout

You've seen the output from the cgi.test() method from the cgi module. Let's make our own version of this.

All this is well and good, but where's the dynamic stuff?

http://localhost:8000/cgi_bin/somescript.py?a=23&b=46&b=92

In the cgi module, we get access to the query string with the FieldStorage class:

import cgi

form = cgi.FieldStorage()
stringval = form.getvalue('a', None)
listval = form.getlist('b')

• The values in the FieldStorage are always strings
• getvalue allows you to return a default, in case the field isn't present
• getlist always returns a list: empty, one-valued, or as many values as are present

Let's create a dynamic adding machine.

• In the cgi-bin directory you'll find cgi_sums.py.
• In the index.html file in the cgi directory, the third link leads to this file.
• You will use the structure of that link, and what you learned just now about cgi.FieldStorage.
• Complete the cgi script in cgi_sums.py so that the result of adding all operands sent via the url query is returned.
• Return the results as plain text, with the appropriate Content-Type header.

Let's take a break here, before continuing

The Web Server Gateway Interface

CGI is great, but there are problems:

The most popular approach is to have a long-running process inside the server that handles CGI scripts.

Enter WSGI, the Web Server Gateway Interface.

WSGI consists of two parts, a server and an application.

Where the simplified server above is not functional, this is a complete app:

def application(environ, start_response)
    status = "200 OK"
    body = "Hello World\n"
    response_headers = [('Content-type', 'text/plain'),
                        ('Content-length', len(body))]
    start_response(status, response_headers)
    return [body]

A third part of the puzzle is something called WSGI middleware

• Middleware implements both the server and application interfaces
• Middleware acts as a server when viewed from an application
• Middleware acts as an application when viewed from a server

The WSGI Stack can thus be expressed like so:

HTTP <---> WSGI <---> app code

The Python standard lib provides a reference implementation of WSGI:

You can also deploy with Apache as your HTTP server, using mod_wsgi:

Finally, it is also common to see WSGI apps deployed via a proxied WSGI server:

REQUEST_METHOD:

The HTTP request method, such as "GET" or "POST". This cannot ever be an empty string, and so is always required.

SCRIPT_NAME:

The initial portion of the request URL's "path" that corresponds to the application object, so that the application knows its virtual "location". This may be an empty string, if the application corresponds to the "root" of the server.

PATH_INFO:

The remainder of the request URL's "path", designating the virtual "location" of the request's target within the application. This may be an empty string, if the request URL targets the application root and does not have a trailing slash.

QUERY_STRING:

The portion of the request URL that follows the "?", if any. May be empty or absent.

CONTENT_TYPE:

The contents of any Content-Type fields in the HTTP request. May be empty or absent.

CONTENT_LENGTH:

The contents of any Content-Length fields in the HTTP request. May be empty or absent.

SERVER_NAME, SERVER_PORT:

When combined with SCRIPT_NAME and PATH_INFO, these variables can be used to complete the URL. Note, however, that HTTP_HOST, if present, should be used in preference to SERVER_NAME for reconstructing the request URL. See the URL Reconstruction section below for more detail. SERVER_NAME and SERVER_PORT can never be empty strings, and so are always required.

SERVER_PROTOCOL:

The version of the protocol the client used to send the request. Typically this will be something like "HTTP/1.0" or "HTTP/1.1" and may be used by the application to determine how to treat any HTTP request headers. (This variable should probably be called REQUEST_PROTOCOL, since it denotes the protocol used in the request, and is not necessarily the protocol that will be used in the server's response. However, for compatibility with CGI we have to keep the existing name.)

HTTP_ Variables:

Variables corresponding to the client-supplied HTTP request headers (i.e., variables whose names begin with "HTTP_"). The presence or absence of these variables should correspond with the presence or absence of the appropriate HTTP header in the request.

Seem Familiar?

Let's start simply. We'll begin by repeating our first CGI exercise in WSGI

• Find the wsgi directory in the class resources. Copy it to your working directory.
• Open the file wsgi_1.py in your text editor.
• We will fill in the missing values using Python and the wsgi environ, just as we use os.environ in cgi

But First

You can run this script with python:

$ python wsgi_1.py

WSGI is a long-running process.

So now we've learned a bit about the WSGI specification and how a WSGI application can get data that comes in via an HTTP request.

When viewing our first wsgi app, do we see the name of the wsgi application script anywhere in the URL?

We have to write an app that will map our incoming request path to some code that can handle that request.

The wsgi environment gives us access to PATH_INFO.

http://localhost:8080/book/<identifier>

Let's write a function, called resolve_path in our application file.

• It should take the PATH_INFO value from environ as an argument.
• It should return the function that will be called.
• It should also return any arguments needed to call that function.
• This implies of course that the arguments should be part of the PATH

We need to hook our new dispatch function into the application.

• The path should be extracted from environ.
• The dispatch function should be used to get a function and arguments
• The body to return should come from calling that function with those arguments
• If an error is raised by calling the function, an appropriate response should be returned
• If the router raises a NameError, the application should return a 404 response

Once you've got your script settled, run it:

$ python bookapp.py

The function books should return an html list of book titles where each title is a link to the detail page for that book

• You'll need all the ids and titles from the book database.
• You'll need to build a list in HTML using this information
• Each list item should have the book title as a link
• The href for the link should be of the form /book/<id>

Quit and then restart your application script:

$ python bookapp.py

http://localhost:8080/

The next step of course is to polish up those detail pages.

Quit and restart your script one more time

Next steps for an app like this might be:

• Create a shared full page template and incorporate it into your app
• Improve the error handling by emitting error codes other than 404 and 500
• Swap out the basic backend here with a different one, maybe a Web Service?
• Think about ways to make the application less tightly coupled to the pages it serves

Next week we will be installing Python packages that are not part of the standard library.

For educational purposes, you might wish to take a look at the source code for the wsgiref module. It's the canonical example of a simple wsgi server

>>> import wsgiref
>>> wsgiref.__file__
'/full/path/to/your/copy/of/wsgiref.py'
...

See you Next Time