Hello World: AWS with Python

Virtualization is nice, but it gets really amazing when you automate it. In this guide, I’ll show you how to setup the AWS Python library and write your first AWS automation program in Python. Refer to my earlier posts for Hello World examples of Python, AWS Linux and AWS Windows.

Install The AWS Python Library

  1. Install the AWS Python Library
  2. users-Mac:~ user$ pip install boto3
    Collecting boto3
      Downloading boto3-1.4.2-py2.py3-none-any.whl (126kB)
        100% |████████████████████████████████| 133kB 563kB/s 
    Collecting botocore<1.5.0,>=1.4.1 (from boto3)
      Downloading botocore-1.4.85-py2.py3-none-any.whl (3.1MB)
        100% |████████████████████████████████| 3.1MB 192kB/s 
    ...
    users-Mac:~ user$
    

    Create an AWS IAM User and Access Key

  3. From the AWS Console, select Services→IAM
  4. Click Users
  5. Click Add user
  6. Add user

  7. Enter a User name (example: script)
  8. Check Programmatic access under Access type
  9. User Details

  10. Click Next: Permissions
  11. Select an existing group or create a new group with sufficient permissions. (For example, one with the AdministratorAccess policy.)
  12. User Permissions

  13. Click Next: Review
  14. Confirm the settings are correct and click Create user
  15. Click Show under Secret access key
  16. Show Access Key

  17. Save the Access key ID and the Secret access key for use in the next
    step.
  18. Click Close

Setup the AWS Python Library

  1. Set the default AWS Credentials that Python scripts should use.
  2. users-Mac:~ user$ mkdir ~/.aws
    users-Mac:~ user$ vi ~/.aws/credentials
    
    [default]
    aws_access_key_id = xxxxxxxxxxxxxxxxxxxx
    aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    
  3. Set the default AWS Region in which Python scripts should perform tasks.
  4. users-Mac:~ user$ vi ~/.aws/config
    
    [default]
    region = us-west-2
    

Create an S3 Bucket for Artifacts

  1. From the AWS Console, select Services→S3
  2. S3

  3. Click Create Bucket
  4. Create S3 Bucket

  5. Enter a Bucket Name and a Region for your bucket and click Create. (Note: The bucket name you choose must be unique across all existing bucket names in Amazon S3.)
  6. Create S3 Bucket 2

Confirm Python Library and AWS Credentials

  1. Create a new Python Project and Python file for connection testing. You can download this here.
  2. # Import the AWS Python library
    import boto3
    
    # Connect to AWS S3
    s3c = boto3.client('s3')
    s3r = boto3.resource('s3')
    # Print out bucket names
    print s3c.meta.endpoint_url
    for bucket in s3r.buckets.all():
        print(bucket.name)
    
  3. Run the Python script. You should see the artifact container that you created earlier.
  4. /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/user/PycharmProjects/hello_aws_world/test.py
    https://s3-us-west-2.amazonaws.com
    ex-artifacts
    
    Process finished with exit code 0
    

Create Web Artifacts

  1. Create a PHP and an ASP page to deploy to Linux and Windows servers respectively, or download these web pages here. These web artifacts are designed to display the internal instance meta-data. You can read more about accessing instance metadata here.
  2. index.php
    ----------------
    <?php
      $url = "http://169.254.169.254/latest/meta-data/";
      $instance_id = file_get_contents($url . "instance-id");
      $local_hostname = file_get_contents($url . "local-hostname");
      $local_ipv4 = file_get_contents($url . "local-ipv4");
      $public_hostname = file_get_contents($url . "public-hostname");
      $public_ipv4 = file_get_contents($url . "public-ipv4");
      echo "Hello AWS World for Linux<br/>";
      echo "<b>instance-id:</b> " . $instance_id . "<br/>";
      echo "<b>local-hostname:</b> " . $local_hostname . "<br/>";
      echo "<b>local-ipv4:</b> " . $local_ipv4 . "<br/>";
      echo "<b>public-hostname:</b> " . $public_hostname . "<br/>";
      echo "<b>public-ipv4:</b> " . $public_ipv4 . "<br/>";
    ?>
    
    default.asp
    ----------------
    <html>
    <body>
    <%
    url = "http://169.254.169.254/latest/meta-data/"
    
    set XmlObj = Server.CreateObject("Microsoft.XMLHTTP")
    
    XmlObj.open "POST", url & "instance-id", false
    XmlObj.send
    instance_id = XmlObj.responseText
    
    XmlObj.open "POST", url & "local-hostname", false
    XmlObj.send
    local_hostname = XmlObj.responseText
    
    XmlObj.open "POST", url & "local-ipv4", false
    XmlObj.send
    local_ipv4 = XmlObj.responseText
    
    XmlObj.open "POST", url & "public-hostname", false
    XmlObj.send
    public_hostname = XmlObj.responseText
    
    XmlObj.open "POST", url & "public-ipv4", false
    XmlObj.send
    public_ipv4 = XmlObj.responseText
    
    Response.write("Hello AWS World for Windows<br/>")
    Response.write("<b>instance-id:</b> " & instance_id & "<br/>")
    Response.write("<b>local-hostname:</b> " & local_hostname & "<br/>")
    Response.write("<b>local-ipv4:</b> " & local_ipv4 & "<br/>")
    Response.write("<b>public-hostname:</b> " & public_hostname & "<br/>")
    Response.write("<b>public-ipv4:</b> " & public_ipv4 & "<br/>")
    %>
    </body>
    </html>
    
  3. Now, upload these files to the bucket you created for artifacts. Make the files public. Later, these will be downloaded programmatically into your AWS Instances

Create Instance Bootstrap Scripts

  1. Create a shell and a PowerShell script to auto-configure Linux and Windows servers respectively, or download these scripts here. The OS bootstrap script will automatically partition and format the EBS volume, install the web server, deploy artifacts to the web server root on the EBS volume and start the web server.
  2. bootstrap.sh
    ----------------
    echo 'Initializing, Partitioning and Formatting all raw disks.'
    sudo parted -s /dev/sdf mklabel gpt
    sudo parted -s -a optimal /dev/sdf mkpart primary ext4 0% 100%
    sudo mkfs.ext4 -L www /dev/sdf1
    sudo mkdir /mnt/www
    sudo echo 'LABEL=www         /mnt/www    ext4     defaults        0   2' | sudo tee -a /etc/fstab
    sudo mount -a
    
    echo 'Installing Apache.'
    sudo yum -y update
    sudo yum -y install httpd
    sudo yum -y install php
    sudo chkconfig httpd on
    
    echo 'Setting Up New Default Web Site Virtual Directory on EBS at /mnt/www/html'
    sudo mkdir /mnt/www/html
    cd /mnt/www/html
    sudo wget https://s3-us-west-2.amazonaws.com/MyBucket/index.php
    
    echo 'Setting Apache DocumentRoot to /mnt/www'
    sudo sed -i 's%/var/www/html%/mnt/www/html%g' /etc/httpd/conf/httpd.conf
    
    echo 'Starting httpd server.'
    sudo service httpd start
    
    bootstrap.ps1
    ----------------
    Write-Host "Initializing, Partitioning and Formatting all raw disks."
    Get-Disk |
    Where partitionstyle -eq 'raw' |
    Initialize-Disk -PartitionStyle MBR -PassThru |
    New-Partition -AssignDriveLetter -UseMaximumSize |
    Format-Volume -FileSystem NTFS -NewFileSystemLabel "disk2"; -Confirm:$false
    
    Write-Host "Installing IIS."
    Install-WindowsFeature -Name Web-Server -IncludeAllSubFeature -IncludeManagementTools
    
    Write-Host "Downloading File System Security PowerShell Module to C:\Program Files\WindowsPowerShell\Modules"
    $down = New-Object System.Net.WebClient
    $url  = 'https://s3-us-west-2.amazonaws.com/MyBucket/NTFSSecurity.zip';
    $file = 'C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity.zip';
    $down.DownloadFile($url,$file);
    
    Write-Host "Unzipping C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity.zip"
    New-Item -ItemType directory -Path 'C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity'
    $shell = new-object -com shell.application
    $zip = $shell.NameSpace($file)
    foreach($item in $zip.items())
    {
        $shell.Namespace("C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity").copyhere($item, 0x14)
    }
    
    Write-Host "Setting Up New Default Web Site Virtual Directory on EBS at d:\inetpub"
    New-Item -ItemType directory -Path 'd:\inetpub'
    $url  = 'https://s3-us-west-2.amazonaws.com/MyBucket/default.asp';
    $file = 'D:\inetpub\default.asp';
    $down.DownloadFile($url,$file);
    Add-NTFSAccess -Path $file -Account 'IUSR' -AccessRights ReadAndExecute
    
    Write-Host "Setting Default Web Site Virtual Directory to D:\IIS"
    Import-Module WebAdministration
    Set-ItemProperty 'IIS:\Sites\Default Web Site' -Name physicalPath -Value d:\inetpub
    
  3. Change the URIs in the scripts to reflect the bucket name that you are using for artifacts. (i.e. “$url = …”)
  4. Download NTFSSecurity.zip from Microsoft Technet here. The Windows bootstrap script needs this to set permissions on the Web content.
  5. Upload bootstrap.sh, bootstrap.ps1, and NTFSSecurity.zip to the bucket that you created for artifacts. Make the files public. These will be downloaded and executed programmatically by your instance Auto-Run scripts.

Write a Python Script to Deploy Instances

  1. Create a new Python file for the deployment of instances. You can download this here. This script will do the following:
    1. Create a security group for your instance type if it doesn’t already exist.
    2. Create an Elastic Block Store (EBS) and attach it to the deployed instance. This volume will be used to store artifacts.
    3. Configure the instance to download and execute an OS bootstrap script. The OS bootstrap script will automatically partition and format the EBS volume, install the web server, deploy artifacts to the web server root on the EBS volume and start the web server.
    4. Attempt to assign an EIP. First it will attempt to use an allocated, but unused EIP. Otherwise, it will attempt to allocate and use a new EIP.
    import boto3
    import botocore
    
    # Settings (Configure these to match your environment.)
    KeyName = 'MyKeyPair2'
    BaseName = 'Hello AWS World'  # Base string of Name tag
    ImageId = 'ami-b04e92d0'    # Amazon Linux AMI 2016.09.0 (HVM), SSD Volume Type
    #ImageId = 'ami-9f5efbff'      # Microsoft Windows Server 2016 Base
    
    
    # No user serviceable parts below #
    
    # Connect
    ec2c = boto3.client('ec2')
    ec2r = boto3.resource('ec2')
    
    # Get Image Info
    Amis = ec2c.describe_images(ImageIds=[ImageId])
    ImageDescription = Amis['Images'][0]['Description']
    # Determine image type
    ImageType = ""
    if 'Linux' in ImageDescription:
        ImageType = 'Linux'
    elif 'Windows' in ImageDescription:
        ImageType = 'Windows'
    else:
        print 'Unknown image type, exiting...'
        exit(1)
    print "The AMI image type is " + ImageType
    
    # Create Security Group
    SecGroupName = ImageType + 'Web'
    SecGroupDescription = ImageType+' Web'
    try:
        Groups = ec2c.describe_security_groups(
            GroupNames=[SecGroupName]
        )
    
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == 'InvalidGroup.NotFound':
            print "Creating security group"+SecGroupName
            NewSecGroup = ec2c.create_security_group(
                GroupName=SecGroupName,
                Description=SecGroupDescription
            )
            if ImageType == 'Linux':
                ec2c.authorize_security_group_ingress(
                    GroupId=NewSecGroup['GroupId'],
                    IpProtocol="tcp",
                    CidrIp="0.0.0.0/0",
                    FromPort=22,
                    ToPort=22
                )
            elif ImageType == 'Windows':
                ec2c.authorize_security_group_ingress(
                    GroupId=NewSecGroup['GroupId'],
                    IpProtocol="tcp",
                    CidrIp="0.0.0.0/0",
                    FromPort=3389,
                    ToPort=3389
                )
            ec2c.authorize_security_group_ingress(
                GroupId=NewSecGroup['GroupId'],
                IpProtocol="tcp",
                CidrIp="0.0.0.0/0",
                FromPort=80,
                ToPort=80
            )
            ec2c.authorize_security_group_ingress(
                GroupId=NewSecGroup['GroupId'],
                IpProtocol="tcp",
                CidrIp="0.0.0.0/0",
                FromPort=443,
                ToPort=443
            )
        else:
            print "Unexpected error: %s" % e
    
    # Set UserData to bootstrap OS
    UserData = ''
    if ImageType == 'Linux':
        UserData = '''#!/bin/sh
    wget https://s3-us-west-2.amazonaws.com/ex-artifacts/bootstrap.sh
    chmod u+x bootstrap.sh
    ./bootstrap.sh'''
    elif ImageType == 'Windows':
        UserData = '''<powershell>
    New-Item -ItemType directory -Path 'c:\\temp'
    $down = New-Object System.Net.WebClient
    $url  = 'https://s3-us-west-2.amazonaws.com/ex-artifacts/bootstrap.ps1'
    $file = 'c:\\temp\\bootstrap.ps1'
    $down.DownloadFile($url,$file)
    Invoke-Expression $file
    </powershell>'''
        UserData.replace("\r", "\r\n")
    
    # Define block devices
    DeviceName1 = ''
    DeviceSize1 = ''
    if ImageType == 'Linux':
        DeviceName1 = '/dev/xvda'
        DeviceSize1 = 8
    elif ImageType == 'Windows':
        DeviceName1 = '/dev/sda1'
        DeviceSize1 = 30
    
    # Deploy the instance
    print "Creating instance."
    Instances = ec2r.create_instances(
        ImageId=ImageId,
        MinCount=1,
        MaxCount=1,
        KeyName=KeyName,
        SecurityGroups=[SecGroupName],
        UserData=UserData,
        InstanceType='t2.micro',
        BlockDeviceMappings=[
            {
                'VirtualName': 'root',
                'DeviceName': DeviceName1,
                'Ebs': {
                    'VolumeSize': DeviceSize1,
                    'DeleteOnTermination': True,
                    'VolumeType': 'standard',
                },
            },
            {
                'VirtualName': 'www',
                'DeviceName': '/dev/sdf',
                'Ebs': {
                    'VolumeSize': 1,
                    'DeleteOnTermination': True,
                    'VolumeType': 'standard',
                },
            },
        ],
    )
    Instance = Instances[0]
    
    # Tag the Instance
    # Apply the tag to the instance
    Instance.create_tags(
        Tags=[
            {
                'Key': 'Name',
                'Value': BaseName + " " + ImageType
            },
        ]
    )
    
    # Assign EIP to the instance.
    print 'Waiting for instance to start.'
    Instance.wait_until_running()
    
    Result = ec2c.describe_addresses()
    Addresses = Result['Addresses']
    FreeAddresses = []
    for Address in Addresses:
        if Address not in 'AssociationId':
            FreeAddresses.insert(len(FreeAddresses), Address)
    
    if FreeAddresses:
        Eip = FreeAddresses[0]
        print "Using existing EIP: ", Eip['PublicIp']
    else:
        print "Creating and assigning EIP to the instance."
        Eip = ec2c.allocate_address()
        print "Created EIP: ", Eip['PublicIp']
    AllocationId = Eip['AllocationId']
    InstanceId = Instance.instance_id
    
    Response = ec2c.associate_address(
        InstanceId=InstanceId,
        AllocationId=AllocationId,
    )
    
    # Display the instance info
    # Refresh Instance Info
    Instance.load()     # Reload instance details
    PublicDnsName = Instance.public_dns_name
    
    print "The web server will be available in a few minutes."
    print "    Public DNS: ", PublicDnsName
    print "    InstanceId: ", InstanceId
    print "    Image:      ", ImageDescription
    
  2. Be sure that the ImageId variable is set to a Linux image and run the script.
  3. Linux Image Output
  4. -------------------------------- /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/user/PycharmProjects/hello_aws_world/launch.py The AMI image type is Linux Creating instance. Waiting for instance to start. Using existing EIP: 35.165.238.48 The web server will be available in a few minutes. Public DNS: ec2-35-165-238-48.us-west-2.compute.amazonaws.com InstanceId: i-0cde95b2268bf752f Image: Amazon Linux AMI 2016.09.0.20160923 x86_64 HVM GP2 Process finished with exit code 0
  1. Wait 1-2 minutes and connect to the Public DNS in a browser.
  2. Hello AWS World for Linux

  3. Be sure that the ImageId variable is set to a Windows image and run the script.
  4. Windows Image Output
    --------------------------------
    /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/user/PycharmProjects/hello_aws_world/launch.py
    The AMI image type is Windows
    Creating instance.
    Waiting for instance to start.
    Using existing EIP:  35.165.207.95
    The web server will be available in a few minutes.
        Public DNS:  ec2-35-165-207-95.us-west-2.compute.amazonaws.com
        InstanceId:  i-01cefe7ef1a088899
        Image:       Microsoft Windows Server 2016 with Desktop Experience Locale English AMI provided by Amazon
    
    Process finished with exit code 0
    
  5. The Windows prep takes a lot longer than Linux. So, you need to wait 10-15 minutes and then connect to the Public DNS in a browser.
  6. Hello AWS World for Windows

Conclusion

Congratulations, you have automated the deployment of an OS, installation of a web server, and deployment of artifacts. By deploying your application with a script, you have created a completely consistent deployment workflow allowing you to scale and repair application servers with the push of a button.

You may find the following additional resources helpful:

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.