Note
For this tutorial, you will want to have the documentation for fabric at hand
We’ve looked at creating and destroying new AWS EC2 instances using the Python wrapper for the AWS API, boto.
Once you have a server, you want to do something useful with it. Possible tasks might include:
Manually performing these tasks is tedious. Once you have figured out how to configure a system properly for your application, why would you want to do it over and over again?
Instead, you should automate these tasks.
Automation ensures that you can accomplish a task with a single command, which encourages actually doing that task more often.
One package available for Python to accomplish this sort of task automation is fabric. It provides a uniform interface for performing tasks either locally or on one or more remote server.
Fabric uses SSH to handle communications between your local machine and the system you are working on, so any command you can run from a terminal is available to be automated.
In this quick tutorial, we’ll walk through the process of automating the provisioning of a new AWS EC2 instance and then installing some software on the new system.
To begin with, let’s create an environment suitable for exploring fabric and using boto to set up remote servers.
Begin by creating a new virtualenv project:
heffalump:~ cewing$ mkproject fabrictests
New python executable in fabrictests/bin/python
Installing setuptools, pip...done.
Creating /Users/cewing/projects/fabrictests
Setting project for fabrictests to /Users/cewing/projects/fabrictests
[fabrictests]
heffalump:fabrictests cewing$
Once that’s handled, let’s install both boto and fabric:
[fabrictests]
heffalump:fabrictests cewing$ pip install boto fabric
Downloading/unpacking boto
Downloading boto-2.25.0-py2.py3-none-any.whl (1.1MB): 1.1MB downloaded
Downloading/unpacking fabric
Downloading Fabric-1.8.2.tar.gz (251kB): 251kB downloaded
...
... (lots of c-compilation stuff)
Successfully installed boto fabric paramiko pycrypto ecdsa
Cleaning up...
[fabrictests]
heffalump:fabrictests cewing$
Notice that fabric, since it operates over SSH, requires the Python ssh library paramiko.
Because fabric will use SSH to connect to all hosts, including your local machine, you’ll need to enable SSH access for your computer.
We are going to set up an HTTP server on our ubuntu instance. In order to prove that we’ve done it, we need to be able to connect to that server via HTTP (port 80).
Before you go any further, open your AWS web console. Navigate to the EC2 dashboard, and then select the ‘Security Groups’ controls.
Find your ssh-access security group and add a new TCP rule. Allow HTTP access via port 80 from any address.
Don’t forget to apply your rules changes after adding the new rule, or it won’t take effect.
Fabric operates by providing a single command fab.
This command looks for a file in your present working directory called fabfile or fabfile.py. If there is none, it will throw an error:
[fabrictests]
heffalump:fabrictests cewing$ fab
Fatal error: Couldn't find any fabfiles!
Remember that -f can be used to specify fabfile path, and use -h for help.
Aborting.
[fabrictests]
heffalump:fabrictests cewing$
Let’s start by creating a new fabfile here in the fabrictests project directory:
fabrictests]
heffalump:fabrictests cewing$ touch fabfile.py
[fabrictests]
heffalump:fabrictests cewing$
What goes into that file?
Python code.
Let’s start by creating a very simple test Python function we can use to show how fabric works in general.
Open fabfile.py in your editor and add the following code:
from fabric.api import run
def host_type():
run('uname -s')
Once you’ve saved this, the name of the new Python function becomes available as a subcommand for the fab command:
[fabrictests]
heffalump:fabrictests cewing$ fab -H localhost host_type
[localhost] Executing task 'host_type'
[localhost] run: uname -s
[localhost] Login password for 'cewing':
[localhost] out: Darwin
[localhost] out:
Done.
Disconnecting from localhost... done.
[fabrictests]
heffalump:fabrictests cewing$
Notice that you’re local machine has prompted you to login with a password. That’s a good thing. If you enable publickey login then fabric will skip this part for you. This also means that you won’t be sending your password across a connection, so that’s nice.
Having to specify a host to connect to is a bit awkward, so fabric provides a mechanism for setting this up automatically.
The fabric.api module provides a symbol env that allows you to share state between different fabric tasks. This state can be mutated locally, both at module level and from within a given task.
You can hang pretty much any symbol onto this object, which is really just a fancy Python dictionary.
However, there are a number of environmental settings that are present from the get-go, and which fabric will refer to automatically.
One of these is hosts. When we invoked the fab command with -H localhost we were pre-populating this env attribute with the string ‘localhost’. We can do this manually. Update your fabfile.py as follows:
from fabric.api import run
from fabric.api import env
env.hosts = ['localhost', ]
def host_type():
run('uname -s')
Save those changes and you should now be able to run the host_type subcommand without providing the host explicitly:
[fabrictests]
heffalump:fabrictests cewing$ fab host_type
[localhost] Executing task 'host_type'
[localhost] run: uname -s
[localhost] Login password for 'cewing':
[localhost] out: Darwin
[localhost] out:
Done.
Disconnecting from localhost... done.
[fabrictests]
heffalump:fabrictests cewing$
Our interactions with boto are all oriented around creating and destroying servers.
In fabric, our orientation will be toward manipulating a server once it exists.
Before we do that, though we need to get a server running.
Any attempt we make at dealing with AWS EC2 will require that we have a connection, so our first step should be to make a connection to the region we want to use. Let’s add a function to do this to our fabfile.py
# add an import
import boto.ec2
# add an environmental setting
env.aws_region = 'us-west-2'
# add a function
def get_ec2_connection():
if 'ec2' not in env:
conn = boto.ec2.connect_to_region(env.aws_region)
if conn is not None:
env.ec2 = conn
print "Connected to EC2 region %s" % env.aws_region
else:
msg = "Unable to connect to EC2 region %s"
raise IOError(msg % env.aws_region)
return env.ec2
We can call fabric functions from other fabric functions, so this will be of use to us as we try to provision, manage and deprovision our servers.
In the boto walkthrough, we saw how to provision a new instance and set it running. Let’s convert that code to a fabric task.
We’ll need to start by getting a connection, then use that connection to create a new instance.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | # add an import at the top
import time
# add this function
def provision_instance(wait_for_running=False, timeout=60, interval=2):
wait_val = int(interval)
timeout_val = int(timeout)
conn = get_ec2_connection()
instance_type = 't1.micro'
key_name = 'pk-cpe'
security_group = 'ssh-access'
image_id = 'ami-d0d8b8e0'
reservations = conn.run_instances(
image_id,
key_name=key_name,
instance_type=instance_type,
security_groups=[security_group, ]
)
new_instances = [i for i in reservations.instances if i.state == u'pending']
running_instance = []
if wait_for_running:
waited = 0
while new_instances and (waited < timeout_val):
time.sleep(wait_val)
waited += int(wait_val)
for instance in new_instances:
state = instance.state
print "Instance %s is %s" % (instance.id, state)
if state == "running":
running_instance.append(
new_instances.pop(new_instances.index(i))
)
instance.update()
|
There’s something new in this function. It accepts arguments. We can pass arguments like this from the command-line when we run the command:
[fabrictests]
heffalump:fabrictests cewing$ fab provision_instance
[fabrictests]
heffalump:fabrictests cewing$ fab provision_instance:wait_for_running=1
[fabrictests]
heffalump:fabrictests cewing$ fab provision_instance:wait_for_running=1,interval=5
The important part to know about this is that arguments passed in from the command line appear in your function as strings. You are responsible for converting them if needed.
Let’s provision a new server with this function:
[fabrictests]
heffalump:fabrictests cewing$ fab provision_instance:wait_for_running=1
[localhost] Executing task 'provision_instance'
Connected to EC2 region us-west-2
Instance i-8c424a85 is pending
Instance i-8c424a85 is pending
Instance i-8c424a85 is pending
Instance i-8c424a85 is pending
Instance i-8c424a85 is pending
Instance i-8c424a85 is pending
Instance i-8c424a85 is pending
Instance i-8c424a85 is running
Done.
[fabrictests]
heffalump:fabrictests cewing$
Once the function completes, you should be able to load your EC2 dashboard and see the new instance you just created running happily.
Now that we have an instance running, we want to run commands on it to configure it.
Fabric provides the run command to support this.
But there’s an issue. When you execute a function using fabric, it is actually run repeatedly, once for each host you list in the env.hosts environment setting.
At the moment, our script lists ‘localhost’, but we don’t actually want to run this command on ‘localhost’, so we need to get around this.
Fabric provides the execute command specifically for this purpose. We can pass it the list of hosts on which we want to run a particular command, and it will do so for us.
In order to play with that, though, we need to interactively select the instance we want to use (after all, we might have more than one, right?).
We’ll begin by building a function that allows us to list instances, optionally filtering for a particular state:
def list_aws_instances(verbose=False, state='all'):
conn = get_ec2_connection()
reservations = conn.get_all_reservations()
instances = []
for res in reservations:
for instance in res.instances:
if state == 'all' or instance.state == state:
instance = {
'id': instance.id,
'type': instance.instance_type,
'image': instance.image_id,
'state': instance.state,
'instance': instance,
}
instances.append(instance)
env.instances = instances
if verbose:
import pprint
pprint.pprint(env.instances)
If we run this command from the command line, we should see our running instance:
[fabrictests]
heffalump:fabrictests cewing$ fab list_aws_instances:verbose=1,state=running
[localhost] Executing task 'list_aws_instances'
Connected to EC2 region us-west-2
[{'id': u'i-ab5159a2',
'image': u'ami-d0d8b8e0',
'instance': Instance:i-ab5159a2,
'state': u'running',
'type': u't1.micro'}]
Done.
[fabrictests]
heffalump:fabrictests cewing$
Nice.
Now we can create a function that will take the output of that function and give us a choice of running instances on which to take a given action:
# add an import
from fabric.api import prompt
def select_instance(state='running'):
if env.active_instance:
return
list_aws_instances(state=state)
prompt_text = "Please select from the following instances:\n"
instance_template = " %(ct)d: %(state)s instance %(id)s\n"
for idx, instance in enumerate(env.instances):
ct = idx + 1
args = {'ct': ct}
args.update(instance)
prompt_text += instance_template % args
prompt_text += "Choose an instance: "
def validation(input):
choice = int(input)
if not choice in range(1, len(env.instances) + 1):
raise ValueError("%d is not a valid instance" % choice)
return choice
choice = prompt(prompt_text, validate=validation)
env.active_instance = env.instances[choice - 1]['instance']
The fabric.api.prompt function allows us to use an interaction like raw_input to ask for input from the fabric user. Here, we build a list of the available instances that are ‘running’, and then ask the user to choose among them. After a valid choice is made, we can then hang that instance on our env so that we can access it from other functions. We can also short-circuit this behavior by checking first to see if we’ve already selected an instance.
Noe, let’s set up a function that will run a given command on the server we select:
# add an import
from fabric.api import execute
def run_command_on_selected_server(command):
select_instance()
selected_hosts = [
'ubuntu@' + env.active_instance.public_dns_name
]
execute(command, hosts=selected_hosts)
This function begins by asking the user to select an instance. Then it sets up a list of hosts that includes that server (with the username ‘ubuntu’ hard-coded). Finally, it attempts to execute the command we provide as an argument on the list of hosts we just built.
In order for us to run a command on this server, we need to ensure that the keypair we set up for our AWS instances is available to the ssh agent. We can do that at the system level on our local machine:
[fabrictests]
heffalump:fabrictests cewing$ ssh-add ~/.ssh/pk-aws.pem
Identity added: /Users/cewing/.ssh/pk-cpe.pem (/Users/cewing/.ssh/pk-cpe.pem)
[fabrictests]
heffalump:fabrictests cewing$
Now our local ssh agent knows that this is a file we might use to connect. When the AWS server asks us to use public-key authentication, the agent will then try this key along with any others the agent knows about. If no known key works, ssh will bomb out.
Execute will take the name of a fabric command to run. We need a command that it will run that will install nginx for us and then start it.
# add an import
from fabric.api import sudo
def _install_nginx():
sudo('apt-get install nginx')
sudo('/etc/init.d/nginx start')
Finally, we need to wrap this function in a function we might call from the command line that will run it on the server we select:
def install_nginx():
run_command_on_selected_server(_install_nginx)
Now, if we run this fabric command from our command line, we can get nginx installed on our AWS instance:
[fabrictests]
heffalump:fabrictests cewing$ fab install_nginx
[localhost] Executing task 'install_nginx'
Connected to EC2 region us-west-2
Please select from the following instances:
1: running instance i-ab5159a2
Choose an instance: 1
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] Executing task '_install_nginx'
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] sudo: apt-get install nginx
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out:
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: Reading package lists... 0%
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out:
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: Reading package lists... 0%
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out:
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: Reading package lists... 24%
...
...
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: Setting up nginx-full (1.1.19-1ubuntu0.5) ...
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: Setting up nginx (1.1.19-1ubuntu0.5) ...
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: Processing triggers for libc-bin ...
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out: ldconfig deferred processing now taking place
[ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com] out:
Done.
Disconnecting from ubuntu@ec2-54-185-44-188.us-west-2.compute.amazonaws.com... done.
[fabrictests]
heffalump:fabrictests cewing$
At this point, you should be able to open a web browser and point it at your EC2 instance and see the default nginx web page. Remember, the public_dns_name of your instance is the way to get at it, so use that in your browser.
Challenge yourself to add two more functions:
Good Luck!