In the first part of this course, you were introduced to the concept of Object Oriented Programming

OOP was first formalized in the 1970s in Smalltalk, invented by Alan Kay at Xerox PARC

Smalltalk was also the first language which utilized the Model View Controller design pattern.

This pattern (like all design patterns) seeks to provide a way of thinking that helps to make software design easier.

In this case, the goal is to help clarify the high-level separation of concerns in a system.

A web browser provides a convenient container for views of data.

These views are created by controller software hosted on a server.

This controller software accepts input from users via HTTP requests, channeling it into a data model, often stored in some database.

The controller returns information about the state of the data model to the user via HTTP responses

Web frameworks abstract away the specifics of the HTTP request/response cycle, leaving simple MVC components for the developer to use.

Web frameworks exist in nearly all modern languages.

Python has scores of them.

Over the weeks to come, we'll learn about two of them, Pyramid and Django.

You can create compelling one-page applications, much like in microframeworks like Flask

You can also create powerful, scalable applications using the full power of Python

Created by the combined powers of the teams behind Pylons and Zope

It represents the first true second-generation web framework in existence.

Begin by creating a location where you'll do your work.

I generally put all my work in a folder called projects in my home directory:

$ cd
$ mkdir projects
$ cd projects
$ mkdir learning-journal
$ cd learning-journal
$ pwd
/Users/cewing/project/learning-journal

Again, this will help us to keep our work here isolated from anything else we do.

Remember how to make a new venv?

$ pyvenv ljenv

c:\Temp>python -m venv myenv

And then, how to activate it?

$ source ljenv/bin/activate
(ljenv)$

C:> ljenv/Scripts/activate.bat

We can do this with the pip in our active ljenv:

(ljenv)$ pip install pyramid
Collecting pyramid
  Downloading pyramid-1.5.2-py2.py3-none-any.whl (545kB)
    100% |################################| 548kB 172kB/s
...
Successfully installed PasteDeploy-1.5.2 WebOb-1.4
pyramid-1.5.2 repoze.lru-0.6 translationstring-1.3
venusian-1.0 zope.deprecation-4.1.1 zope.interface-4.1.2

Once that is complete, we are ready to create a scaffold for our project.

Pyramid does not.

However, our application will require a database and handling that does require some.

Pyramid provides a system for creating boilerplate called pcreate.

You use it to generate the skeleton for a project based on some pattern:

(ljenv)$ pcreate -s alchemy learning_journal
Creating directory /Users/cewing/projects/learning-journal/learning_journal
...
Welcome to Pyramid.  Sorry for the convenience.
===============================================================================

Let's take a quick look at what that did

Start by initializing a new git repository in the learning_journal folder you just created:

(ljenv)$ cd learning_journal
(ljenv)$ git init
Initialized empty Git repository in
 /Users/cewing/projects/learning-journal/learning_journal/.git/

There are many other files you don't want or need in your repository

You can ignore this in git with the .gitignore file.

Create one now, in this same directory, and add the following basic lines:

*.pyc
.DS_Store

Finally, add this new file to your repository, too.

(ljenv)$ git add .gitignore

This will make a first commit here in this local repository.

For homework, you'll put this into GitHub, but this is enough for now.

Let's move on to learning about what we've built so far.

This folder contains your project.

At the top level, you have configuration (.ini files)

You also have a file called setup.py

This file turns this collection of Python code and configuration into an installable Python distribution

Let's take a moment to look over the code in that file

These files contain sections ([app:main]) containing name = value pairs of configuration data

This data is parsed with the Python ConfigParser module.

The result is a dict of values:

{'app:main': {'pyramid.reload_templates': True, ...}, ...}

The default section of the file is passed in as global_config, the section for this app as settings.

Our connection is set up using settings read from the .ini file.

Can you find the settings for the database?

The DBSession ensures that each database transaction is tied to HTTP requests.

The Base provides a parent class that will hook our models to the database.

It uses app-specific settings passed in from the .ini file

We can also include configuration from other add-on packages

Additionally, we can configure routes and views needed to connect our application to the outside world here (more on this next week).

Finally, the Configurator instance performs a scan to ensure there are no problems with what we've created.

Pyramid configuration is powerful and flexible.

We'll use a few of its features

But there's a lot more you could (and should) learn.

Read about it in the configuration chapter of the Pyramid documentation.

These capture relevant details about the information we want to preserve and how we want to interact with it.

In Python-based MVC applications, these Models are implemented as Python classes.

The individual bits of data we want to know about are attributes of our classes.

The actions we want to take using that data are methods of our classes.

Together, we can refer to this as the API of our system.

But what happens when you want to save information.

What happens to a instance of a Python class when you quit the interprer?

When your script stops running?

The code in a website runs when an HTTP request comes in from a client.

It stops running when an HTTP response goes back out to the client.

So what happens to the data in your system in-between these moments?

The data must be persisted

Any of these might be useful for certain types of applications.

On the web, you tend to see two used the most:

• NoSQL
• SQL

In general, the telling factor is going to be how you intend to use your data.

In systems where the dominant feature is viewing/interacting with individual objects, a NoSQL storage solution might be the best way to go.

In systems with objects that are related to eachother, SQL-based Relational Databases are a better choice.

Our system is more like this latter type (trust me on that one for now).

We'll be using SQL (sqlite to start with).

And a storage system where our data must be rendered as database tables

Python provides a specification for interacting directly with databases: dbapi2

And there are multiple Python packages that implement this specification for various databases:

• sqlite3
• python-mysql
• psycopg2
• ...

With these, you can write SQL to save your Python objects into your database.

SQL, while not impossible, is yet another language to learn.

And there is a viable alternative in using an Object Relational Manager (ORM)

An ORM provides a layer of abstraction between you and SQL

You instantiate Python objects and set attributes on them

The ORM handles converting data from these objects into SQL statements (and back)

You can find SQLAlchemy among the packages in requires in setup.py in our new learning_journal package.

However, we don't yet have that code installed.

To do so, we will need to "install" our own package

Make sure your ljenv virtualenv is active and then type the following:

(ljenv)$ python setup.py develop
running develop
running egg_info
creating learning_journal.egg-info
...
Finished processing dependencies for learning-journal==0.0

You can also install the package using python setup.py install

But using develop allows us to continue developing our package without needing to re-install it every time we change something.

It is very similar to using the -e option to pip

Now, we'll only need to re-run this command if we change setup.py itself.

The egg-info directory that was just created is an artifact of installing a Python egg.

It should never be committed to a repository.

Let's add *.egg-info to our .gitignore file and then commit that change

Remember how?

Our class inherits from Base

We ran into Base earlier when discussing configuration.

We were binding it to the database we wanted to use (the engine)

It will be connected through the ORM to a table in our database.

The name of the table is determined by the __tablename__ special attribute.

Other aspects of table configuration can also be controlled through special attributes

Instances of the class, once saved, will become rows in the table.

Attributes of the model that are instances of Column will become columns in the table.

You can learn much more in the Declarative chapter of the SQLAlchemy docs

Each instance requires at least a specific data type (such as Integer).

Additionally, you can control other aspects of the column such as it being a primary key.

In the declarative style we are using, the name of the column in the database will default to the attribute name you assigned.

If you wish, you may provide a name specifically. It must be the first argument and must be a string.

But we're still missing one ingredient here.

We need to create our database, or there will be nowhere for our data to go.

Luckily, our pcreate scaffold also gave us a convenient way to handle this:

# in setup.py
entry_points="""\
[paste.app_factory]
main = learning_journal:main
[console_scripts]
initialize_learning_journal_db = learning_journal.scripts.initializedb:main
""",

The console_script set up as an entry point will help us.

When we exectute the script at the command line, we will be running this function.

But before we try it out, let's update the name we use so we don't have to type that whole big mess.

In setup.py change initialize_learning_journal_db to setup_db:

entry_points="""\
[paste.app_factory]
main = learning_journal:main
[console_scripts]
setup_db = learning_journal.scripts.initializedb:main
""",

Then, as you have changed setup.py, re-install your package:

(ljenv)$ python setup.py develop
...

We'll need to provide a configuration file name, let's use development.ini:

(ljenv)$ setup_db development.ini
2015-01-05 18:59:55,426 INFO  [sqlalchemy.engine.base.Engine][MainThread] SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
...
2015-01-05 18:59:55,434 INFO  [sqlalchemy.engine.base.Engine][MainThread] COMMIT

The [loggers] configuration in our .ini file sends a stream of INFO-level logging to sys.stdout as the console script runs.

(ljenv)$ ls
...
learning_journal.sqlite
...

We've now created an sqlite database.

You'll need to add *.sqlite to .gitignore so you don't inadvertently add that file to your repository.

Once you've done so, commit the change to your repository

But before we do so, let's make a nicer interpreter available for our project

You've been using iPython in class, we can use it here too.

Just install it with pip:

(ljenv)$ pip install ipython pyramid_ipython

Once that finishes, you'll be able to use iPython as your interpreter for this project.

And Pyramid provides a way to connect your interpreter to the application code you are writing:

The pshell command

This object represents the connection to the database.

All database queries are phrased as methods of the session.

In [8]: query = session.query(MyModel)
In [9]: type(query)
Out[9]: sqlalchemy.orm.query.Query

Arguments to the query method can be a model class or columns from a model class.

In [14]: str(q1)
Out[14]: 'SELECT models.id AS models_id, models.name AS models_name, models.value AS models_value \nFROM models'

In [15]: str(q2)
Out[15]: 'SELECT models.name AS models_name, models.id AS models_id, models.value AS models_value \nFROM models'

You can use this to check that the query the ORM is constructing looks like you expect.

It can be helpful in debugging.

1. Methods that return a new Query object
2. Methods that return scalar values or model instances

Let's start by looking quickly at a few methods from the second category

It takes a primary key as an argument:

In [16]: session.query(MyModel).get(1)
Out[16]: <learning_journal.models.MyModel at 0x105f30208>
In [17]: session.query(MyModel).get(10)
In [18]:

If no item with that primary key is present, then the method returns None

query.all() returns a list of all rows returned by the database:

In [18]: q1.all()
Out[18]: [<learning_journal.models.MyModel at 0x105f30208>]

In [19]: type(q1.all())
Out[19]: list

query.count() returns the number of rows that would have been returned by the query:

In [20]: q1.count()
Out[20]: 1

In [24]: session.add(new_model)
In [25]: session.new
Out[25]: IdentitySet([<learning_journal.models.MyModel object at 0x105f4af28>])

We can even bulk-add new objects:

In [26]: new = []
In [27]: for name, val in [('bob', 34), ('tom', 13)]:
   ....:     new.append(MyModel(name=name, value=val))
   ....:
In [28]: session.add_all(new)
In [29]: session.new
Out[29]: IdentitySet([<learning_journal.models.MyModel object at 0x105f4af28>,
                      <learning_journal.models.MyModel object at 0x105f4a4a8>,
                      <learning_journal.models.MyModel object at 0x105f30550>])

In order for these new objects to be saved to the database, the session must be committed:

In [30]: other_session = Session()
In [31]: other_session.query(MyModel).count()
Out[31]: 1
In [32]: session.commit()
In [33]: other_session.query(MyModel).count()
Out[33]: 4

When you are using a scoped_session in Pyramid, this action is automatically handled for you.

The session that is bound to a particular HTTP request is committed when a response is sent back.

(don't worry if this seems confusing, more to come next week)

Simply change the values of a persisted attribute, the session will know it's been updated:

In [34]: new_model
Out[34]: <learning_journal.models.MyModel at 0x105f4af28>
In [35]: new_model.name
Out[35]: 'fred'
In [36]: new_model.name = 'larry'
In [37]: session.dirty
Out[37]: IdentitySet([<learning_journal.models.MyModel object at 0x105f4af28>])

Commit the session to persist the changes:

In [38]: session.commit()
In [39]: [model.name for model in other_session.query(MyModel)]
Out[39]: ['one', 'larry', 'bob', 'tom']

This method allows you to reduce the number of results, based on criteria:

In [40]: [(o.name, o.value) for o in session.query(MyModel).filter(MyModel.value < 20)]
Out[40]: [('one', 1), ('larry', 3), ('tom', 13)]

In [43]: q1 = session.query(MyModel).filter(MyModel.value < 20)
In [44]: q1 = q1.order_by(MyModel.name)
In [45]: [(o.name, o.value) for o in q1]
Out[45]: [('larry', 3), ('one', 1), ('tom', 13)]

Note that you can do this inline as well (s.query(Model).filter().order_by())

Also note that when using chained queries like this, no query is actually sent to the database until you require a result.

It's good to remember that none of the information we've persisted to our database is vital to us.

For homework this week we'll be making new models, and the data we have in our current database will only get in the way.

Until you have real production data it is always safe simply to delete the database and start over:

$ rm learning_journal.sqlite

You can always re-create it by executing setup_db