In this guide, I’ll show you how to setup the AWS Java library and write your first AWS automation program in Java. Refer to my earlier posts for Hello World examples of Java, AWS Linux and AWS Windows.
Create an AWS IAM User and Access Key
Your Java application will need AWS credentials to run. Set up an AWS user that has sufficient permissions for your script.
- From the AWS Console, select Services→IAM
- Click Users
- Click Add user
- Enter a User name (example: script)
- Check Programmatic access under Access type
- Click Next: Permissions
- Select an existing group or create a new group with sufficient permissions. (For example, one with the AdministratorAccess policy.)
- Click Next: Review
- Confirm the settings are correct and click Create user
- Click Show under Secret access key
- Save the Access key ID and the Secret access key for use in the next
step. - Click Close
Install The AWS Java Library
- Launch Eclipse
- Open Help → Install New Software
- Click Add
- Set the Name as AWS and the Location as https://aws.amazon.com/eclipse
- Expand AWS Core Management Tools and Check AWS Toolkit for Eclipse Core
- Click Next and complete the remaining installation steps
Setup the AWS Library
- Set the default AWS Credentials that the API should use.
- Set the default AWS Region in which the API should perform tasks.
users-Mac:~ user$ mkdir ~/.aws users-Mac:~ user$ vi ~/.aws/credentials
[default] aws_access_key_id = xxxxxxxxxxxxxxxxxxxx aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
users-Mac:~ user$ vi ~/.aws/config
[default] region = us-west-2
Create an S3 Bucket for Artifacts
- From the AWS Console, select Services→S3
- Click Create Bucket
- Enter a Bucket Name and a Region for your bucket and click Create. (Note: The bucket name you choose must be unique across all existing bucket names in Amazon S3.)
Confirm AWS Library installation and AWS Credentials
- Create a new AWS Java Project by clicking the AWS Icon and selecting New AWS Java Project…
- Set the Project name (example: launch) and click Finish
- Create a new Java package for connection testing: Right click your project and select New → Class
- Set the class Name (example: Test) and click Finish
- Enter the following into your class file. You can download this here
- Run the Java program. You should see the artifact container that you created earlier in the output.
package launch;
import java.io.IOException;
import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.Bucket;
public class test {
public static void main(String[] args) throws IOException {
AWSCredentials credentials = null;
try {
credentials = new ProfileCredentialsProvider("default").getCredentials();
} catch (Exception e) {
throw new AmazonClientException(
"Cannot load the credentials from the credential profiles file. " +
"Please make sure that your credentials file is at the correct " +
"location (/Users/user/.aws/credentials), and is in valid format.",
e);
}
AmazonS3 s3c = new AmazonS3Client(credentials);
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
s3c.setRegion(usWest2);
System.out.println("region: " + s3c.getRegion());
for (Bucket bucket : s3c.listBuckets()) {
System.out.println(bucket.getName());
}
}
}
region: us-west-2 ex-artifacts
Create Web Artifacts
- Create a PHP and an ASP page to deploy to Linux and Windows servers respectively, or download these web pages here. These web artifacts are designed to display the internal instance meta-data. You can read more about accessing instance metadata here.
- Now, upload these files to the bucket you created for artifacts. Make the files public. Later, these will be downloaded programmatically into your AWS Instances
index.php ---------------- <?php $url = "http://169.254.169.254/latest/meta-data/"; $instance_id = file_get_contents($url . "instance-id"); $local_hostname = file_get_contents($url . "local-hostname"); $local_ipv4 = file_get_contents($url . "local-ipv4"); $public_hostname = file_get_contents($url . "public-hostname"); $public_ipv4 = file_get_contents($url . "public-ipv4"); echo "Hello AWS World for Linux<br/>"; echo "<b>instance-id:</b> " . $instance_id . "<br/>"; echo "<b>local-hostname:</b> " . $local_hostname . "<br/>"; echo "<b>local-ipv4:</b> " . $local_ipv4 . "<br/>"; echo "<b>public-hostname:</b> " . $public_hostname . "<br/>"; echo "<b>public-ipv4:</b> " . $public_ipv4 . "<br/>"; ?>
default.asp
----------------
<html>
<body>
<%
url = "http://169.254.169.254/latest/meta-data/"
set XmlObj = Server.CreateObject("Microsoft.XMLHTTP")
XmlObj.open "POST", url & "instance-id", false
XmlObj.send
instance_id = XmlObj.responseText
XmlObj.open "POST", url & "local-hostname", false
XmlObj.send
local_hostname = XmlObj.responseText
XmlObj.open "POST", url & "local-ipv4", false
XmlObj.send
local_ipv4 = XmlObj.responseText
XmlObj.open "POST", url & "public-hostname", false
XmlObj.send
public_hostname = XmlObj.responseText
XmlObj.open "POST", url & "public-ipv4", false
XmlObj.send
public_ipv4 = XmlObj.responseText
Response.write("Hello AWS World for Windows<br/>")
Response.write("<b>instance-id:</b> " & instance_id & "<br/>")
Response.write("<b>local-hostname:</b> " & local_hostname & "<br/>")
Response.write("<b>local-ipv4:</b> " & local_ipv4 & "<br/>")
Response.write("<b>public-hostname:</b> " & public_hostname & "<br/>")
Response.write("<b>public-ipv4:</b> " & public_ipv4 & "<br/>")
%>
</body>
</html>
Create Instance Bootstrap Scripts
- Create a shell and a PowerShell script to auto-configure Linux and Windows servers respectively, or download these scripts here. The OS bootstrap script will automatically partition and format the EBS volume, install the web server, deploy artifacts to the web server root on the EBS volume and start the web server.
- Change the URIs in the scripts to reflect the bucket name that you are using for artifacts. (i.e. “$url = …”)
- Download NTFSSecurity.zip from Microsoft Technet here. The Windows bootstrap script needs this to set permissions on the Web content.
- Upload bootstrap.sh, bootstrap.ps1, and NTFSSecurity.zip to the bucket that you created for artifacts. Make the files public. These will be downloaded and executed programmatically by your instance Auto-Run scripts.
bootstrap.sh ---------------- echo 'Initializing, Partitioning and Formatting all raw disks.' sudo parted -s /dev/sdf mklabel gpt sudo parted -s -a optimal /dev/sdf mkpart primary ext4 0% 100% sudo mkfs.ext4 -L www /dev/sdf1 sudo mkdir /mnt/www sudo echo 'LABEL=www /mnt/www ext4 defaults 0 2' | sudo tee -a /etc/fstab sudo mount -a echo 'Installing Apache.' sudo yum -y update sudo yum -y install httpd sudo yum -y install php sudo chkconfig httpd on echo 'Setting Up New Default Web Site Virtual Directory on EBS at /mnt/www/html' sudo mkdir /mnt/www/html cd /mnt/www/html sudo wget https://s3-us-west-2.amazonaws.com/MyBucket/index.php echo 'Setting Apache DocumentRoot to /mnt/www' sudo sed -i 's%/var/www/html%/mnt/www/html%g' /etc/httpd/conf/httpd.conf echo 'Starting httpd server.' sudo service httpd start
bootstrap.ps1
----------------
Write-Host "Initializing, Partitioning and Formatting all raw disks."
Get-Disk |
Where partitionstyle -eq 'raw' |
Initialize-Disk -PartitionStyle MBR -PassThru |
New-Partition -AssignDriveLetter -UseMaximumSize |
Format-Volume -FileSystem NTFS -NewFileSystemLabel "disk2"; -Confirm:$false
Write-Host "Installing IIS."
Install-WindowsFeature -Name Web-Server -IncludeAllSubFeature -IncludeManagementTools
Write-Host "Downloading File System Security PowerShell Module to C:\Program Files\WindowsPowerShell\Modules"
$down = New-Object System.Net.WebClient
$url = 'https://s3-us-west-2.amazonaws.com/MyBucket/NTFSSecurity.zip';
$file = 'C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity.zip';
$down.DownloadFile($url,$file);
Write-Host "Unzipping C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity.zip"
New-Item -ItemType directory -Path 'C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity'
$shell = new-object -com shell.application
$zip = $shell.NameSpace($file)
foreach($item in $zip.items())
{
$shell.Namespace("C:\Program Files\WindowsPowerShell\Modules\NTFSSecurity").copyhere($item, 0x14)
}
Write-Host "Setting Up New Default Web Site Virtual Directory on EBS at d:\inetpub"
New-Item -ItemType directory -Path 'd:\inetpub'
$url = 'https://s3-us-west-2.amazonaws.com/MyBucket/default.asp';
$file = 'D:\inetpub\default.asp';
$down.DownloadFile($url,$file);
Add-NTFSAccess -Path $file -Account 'IUSR' -AccessRights ReadAndExecute
Write-Host "Setting Default Web Site Virtual Directory to D:\IIS"
Import-Module WebAdministration
Set-ItemProperty 'IIS:\Sites\Default Web Site' -Name physicalPath -Value d:\inetpub
Write a Java Program to Deploy Instances
- Create a new Java package to deploy EC2 instances: Right click your project and select New → Class
- Set the class Name (example: Launch) and click Finish
- Enter the following into your class file. You can download this here
- Be sure that the ImageId variable is set to a Linux image and run the script.
- Wait 1-2 minutes and connect to the Public DNS in a browser.
- Be sure that the ImageId variable is set to a Windows image and run the script.
- The Windows prep takes a lot longer than Linux. So, you need to wait 10-15 minutes and then connect to the Public DNS in a browser.
package launch;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Arrays;
import java.io.UnsupportedEncodingException;
import org.apache.commons.codec.binary.Base64;
import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.AmazonEC2Client;
import com.amazonaws.services.ec2.model.DescribeInstancesRequest;
import com.amazonaws.services.ec2.model.DescribeInstancesResult;
import com.amazonaws.services.ec2.model.DescribeImagesRequest;
import com.amazonaws.services.ec2.model.DescribeImagesResult;
import com.amazonaws.services.ec2.model.DescribeSecurityGroupsRequest;
import com.amazonaws.services.ec2.model.DescribeSecurityGroupsResult;
import com.amazonaws.services.ec2.model.CreateSecurityGroupRequest;
import com.amazonaws.services.ec2.model.AuthorizeSecurityGroupIngressRequest;
import com.amazonaws.services.ec2.model.DescribeInstanceStatusRequest;
import com.amazonaws.services.ec2.model.DescribeInstanceStatusResult;
import com.amazonaws.services.ec2.model.DescribeAddressesResult;
import com.amazonaws.services.ec2.model.AssociateAddressRequest;
import com.amazonaws.services.ec2.model.EbsBlockDevice;
import com.amazonaws.services.ec2.model.BlockDeviceMapping;
import com.amazonaws.services.ec2.model.Filter;
import com.amazonaws.services.ec2.model.Instance;
import com.amazonaws.services.ec2.model.Image;
import com.amazonaws.services.ec2.model.SecurityGroup;
import com.amazonaws.services.ec2.model.IpPermission;
import com.amazonaws.services.ec2.model.RunInstancesRequest;
import com.amazonaws.services.ec2.model.RunInstancesResult;
import com.amazonaws.services.ec2.model.CreateTagsRequest;
import com.amazonaws.services.ec2.model.Tag;
import com.amazonaws.services.ec2.model.Address;
import com.amazonaws.services.ec2.model.AllocateAddressResult;
public class Launch {
public static void main(String[] args) {
// ** Settings (Configure these to match your environment.)
String keyName = "MyKeyPair2";
String baseName = "Hello AWS World"; // Base string of Name tag
String imageId = "ami-b04e92d0"; // Amazon Linux AMI 2016.09.0 (HVM),
// SSD Volume Type
// String imageId = "ami-9f5efbff"; //Microsoft Windows Server 2016 Base
// **** No user serviceable parts below ****
AmazonEC2 ec2;
AWSCredentials credentials = null;
try {
credentials = new ProfileCredentialsProvider("default").getCredentials();
} catch (Exception e) {
throw new AmazonClientException("Cannot load the credentials from the credential profiles file. "
+ "Please make sure that your credentials file is at the correct "
+ "location (/Users/thornj/.aws/credentials), and is in valid format.", e);
}
ec2 = new AmazonEC2Client(credentials);
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
ec2.setRegion(usWest2);
//// Get Image Info
List<String> imageFilterValues = new ArrayList<String>();
imageFilterValues.add(imageId);
Filter imageFilter = new Filter("image-id", imageFilterValues);
DescribeImagesRequest imageRequest = new DescribeImagesRequest();
DescribeImagesResult imageResult = ec2.describeImages(imageRequest.withFilters(imageFilter));
String imageDescription = "";
if (imageResult != null) {
List<Image> images = imageResult.getImages();
if (images.size() > 0) {
imageDescription = images.get(0).getDescription();
System.out.println(imageDescription);
} else {
System.out.println("Can't find image, exiting...");
return;
}
}
// Identify Image Type
String imageType;
if (imageDescription.contains("Linux")) {
imageType = "Linux";
} else if (imageDescription.contains("Windows")) {
imageType = "Windows";
} else {
System.out.println("Unknown image type, exiting...");
return;
}
System.out.println("The AMI image type is " + imageType);
//// Create Security Group
String secGroupName = imageType + "Web";
String secGroupDescription = imageType + " Web";
List<String> secGroupFilterValues = new ArrayList<String>();
secGroupFilterValues.add(secGroupName);
Filter secGroupFilter = new Filter("group-name", secGroupFilterValues);
DescribeSecurityGroupsRequest secGroupRequest = new DescribeSecurityGroupsRequest();
DescribeSecurityGroupsResult secGroupResult = ec2
.describeSecurityGroups(secGroupRequest.withFilters(secGroupFilter));
String userData = "";
String encodedUserData = "";
if (secGroupResult != null) {
List<SecurityGroup> secGroups = secGroupResult.getSecurityGroups();
if (secGroups.size() < 1) {
System.out.println("Creating security group " + secGroupName);
CreateSecurityGroupRequest createSecGroupRequest = new CreateSecurityGroupRequest();
createSecGroupRequest.withGroupName(secGroupName).withDescription(secGroupDescription);
ec2.createSecurityGroup(createSecGroupRequest);
IpPermission ipPermission1 = new IpPermission();
IpPermission ipPermission2 = new IpPermission();
IpPermission ipPermission3 = new IpPermission();
if (imageType == "Linux") {
ipPermission1.withIpRanges("0.0.0.0/0").withIpProtocol("tcp").withFromPort(22).withToPort(22);
} else if (imageType == "Windows") {
ipPermission1.withIpRanges("0.0.0.0/0").withIpProtocol("tcp").withFromPort(3389).withToPort(3389);
}
ipPermission2.withIpRanges("0.0.0.0/0").withIpProtocol("tcp").withFromPort(80).withToPort(80);
ipPermission3.withIpRanges("0.0.0.0/0").withIpProtocol("tcp").withFromPort(443).withToPort(443);
AuthorizeSecurityGroupIngressRequest authorizeSecurityGroupIngressRequest = new AuthorizeSecurityGroupIngressRequest();
authorizeSecurityGroupIngressRequest.withGroupName(secGroupName).withIpPermissions(ipPermission1,
ipPermission2, ipPermission3);
ec2.authorizeSecurityGroupIngress(authorizeSecurityGroupIngressRequest);
}
}
//// Set UserData to bootstrap OS
if (imageType == "Linux") {
userData = "#!/bin/sh\n" + "wget https://s3-us-west-2.amazonaws.com/qst-bootstrap/bootstrap.sh\n"
+ "chmod u+x bootstrap.sh\n" + "./bootstrap.sh\n";
} else if (imageType == "Windows") {
userData = "<powershell>\r\n" + "New-Item -ItemType directory -Path \"c:\\temp\"\r\n"
+ "$down = New-Object System.Net.WebClient\r\n"
+ "$url = \"https://s3-us-west-2.amazonaws.com/qst-bootstrap/bootstrap.ps1\"\r\n"
+ "$file = \"c:\\temp\\bootstrap.ps1\"\r\n" + "$down.DownloadFile($url,$file)\r\n"
+ "Invoke-Expression $file\r\n" + "</powershell>\r\n";
}
try {
encodedUserData = new String(Base64.encodeBase64(userData.getBytes("UTF-8")), "UTF-8");
} catch (UnsupportedEncodingException e) {
throw new AssertionError("UTF-8 not supported");
}
//// Define block devices
String deviceName1 = "";
Integer deviceSize1 = 0;
if (imageType == "Linux") {
deviceName1 = "/dev/xvda";
deviceSize1 = 8;
} else if (imageType == "Windows") {
deviceName1 = "/dev/sda1";
deviceSize1 = 30;
}
EbsBlockDevice ebsBlockDevice1 = new EbsBlockDevice();
ebsBlockDevice1.setVolumeSize(deviceSize1);
ebsBlockDevice1.setVolumeType("standard");
ebsBlockDevice1.setDeleteOnTermination(true);
EbsBlockDevice ebsBlockDevice2 = new EbsBlockDevice();
ebsBlockDevice2.setVolumeSize(1);
ebsBlockDevice2.setVolumeType("standard");
ebsBlockDevice2.setDeleteOnTermination(true);
BlockDeviceMapping blockDeviceMapping1 = new BlockDeviceMapping();
blockDeviceMapping1.setDeviceName(deviceName1);
blockDeviceMapping1.setEbs(ebsBlockDevice1);
BlockDeviceMapping blockDeviceMapping2 = new BlockDeviceMapping();
blockDeviceMapping2.setDeviceName("/dev/sdf");
blockDeviceMapping2.setEbs(ebsBlockDevice2);
Collection<BlockDeviceMapping> blockDeviceMappings = Arrays.asList(blockDeviceMapping1, blockDeviceMapping2);
//// Deploy the instance
Collection<String> securityGroupIds = Arrays.asList(secGroupName);
int minInstanceCount = 1;
int maxInstanceCount = 1;
RunInstancesRequest runInstancesRequest = new RunInstancesRequest(imageId, minInstanceCount, maxInstanceCount);
runInstancesRequest.setKeyName(keyName);
runInstancesRequest.setSecurityGroupIds(securityGroupIds);
runInstancesRequest.setUserData(encodedUserData);
runInstancesRequest.setInstanceType("t2.micro");
runInstancesRequest.setBlockDeviceMappings(blockDeviceMappings);
RunInstancesResult runInstancesResult = ec2.runInstances(runInstancesRequest);
//// Tag the Instance
// Apply the tag to the instance
List<Instance> Instances = runInstancesResult.getReservation().getInstances();
Instance instance = Instances.get(0);
CreateTagsRequest createTagsRequest = new CreateTagsRequest();
createTagsRequest.withResources(instance.getInstanceId()) //
.withTags(new Tag("Name", baseName + " " + imageType));
ec2.createTags(createTagsRequest);
String instanceId = runInstancesResult.getReservation().getInstances().get(0).getInstanceId();
//// Assign EIP to the instance.
System.out.println("Waiting for instance to start.");
wait_until_running(instanceId);
// Get list of unused addresses.
List<Address> freeAddresses = new ArrayList<Address>();
// DescribeImagesResult imageResult =
// ec2.describeImages(imageRequest.withFilters(imageFilter));
DescribeAddressesResult describeAddressResult = ec2.describeAddresses();
if (describeAddressResult != null) {
List<Address> addresses = describeAddressResult.getAddresses();
for (Address address : addresses) {
if (address.getAssociationId() == null) {
System.out.println("public ip " + address.getPublicIp());
freeAddresses.add(address);
}
}
}
Address eip;
String eipPublicIP;
if (freeAddresses.size() > 0) {
eip = freeAddresses.get(0);
eipPublicIP = eip.getPublicIp();
System.out.println("Using existing EIP: " + eipPublicIP);
} else {
System.out.println("Creating and assigning EIP to the instance.");
AllocateAddressResult allocateAddressResult = ec2.allocateAddress();
eipPublicIP = allocateAddressResult.getPublicIp();
System.out.println("Created EIP: " + eipPublicIP);
}
AssociateAddressRequest associateAddressRequest = new AssociateAddressRequest().withPublicIp(eipPublicIP)
.withInstanceId(instanceId);
ec2.associateAddress(associateAddressRequest);
//// Display the instance info
// Refresh Instance Info
// Instance.load() //Reload instance details
List<String> instanceFilterValues = new ArrayList<String>();
instanceFilterValues.add(instanceId);
Filter instanceFilter = new Filter("instance-id", instanceFilterValues);
DescribeInstancesRequest instanceRequest = new DescribeInstancesRequest();
DescribeInstancesResult instanceResult = ec2.describeInstances(instanceRequest.withFilters(instanceFilter));
String publicDnsName = instanceResult.getReservations().get(0).getInstances().get(0).getPublicDnsName();
System.out.println("The web server will be available in a few minutes.");
System.out.println(" Public DNS: " + publicDnsName);
System.out.println(" InstanceId: " + instanceId);
System.out.println(" Image: " + imageDescription);
}
private static boolean wait_until_running(String instanceId) {
AmazonEC2 ec2;
AWSCredentials credentials = null;
try {
credentials = new ProfileCredentialsProvider("default").getCredentials();
} catch (Exception e) {
throw new AmazonClientException("Cannot load the credentials from the credential profiles file. "
+ "Please make sure that your credentials file is at the correct "
+ "location (/Users/thornj/.aws/credentials), and is in valid format.", e);
}
ec2 = new AmazonEC2Client(credentials);
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
ec2.setRegion(usWest2);
System.out.println("Waiting for instance to transition to running");
for (int i = 0; i < 5; ++i) {
DescribeInstanceStatusRequest describeInstanceRequest = new DescribeInstanceStatusRequest()
.withInstanceIds(instanceId);
DescribeInstanceStatusResult describeInstanceResult = ec2.describeInstanceStatus(describeInstanceRequest);
if (!describeInstanceResult.getInstanceStatuses().isEmpty()) {
String state = describeInstanceResult.getInstanceStatuses().get(0).getInstanceState().getName();
System.out.println("Instance state is " + state);
if (state.equals("running")) {
return true;
}
}
if (describeInstanceResult.equals("running")) {
return true;
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("got interrupted!");
}
}
return false;
}
}
Linux Image Output
--------------------------------
Amazon Linux AMI 2016.09.0.20160923 x86_64 HVM GP2
The AMI image type is Linux
Waiting for instance to start.
Waiting for instance to transition to running
Instance state is running
Creating and assigning EIP to the instance.
Created EIP: 35.165.238.48
The web server will be available in a few minutes.
Public DNS: ec2-35-165-238-48.us-west-2.compute.amazonaws.com
InstanceId: i-0cde95b2268bf752f
Image: Amazon Linux AMI 2016.09.0.20160923 x86_64 HVM GP2
Windows Image Output
--------------------------------
Microsoft Windows Server 2016 with Desktop Experience Locale English AMI provided by Amazon
The AMI image type is Windows
Waiting for instance to start.
Waiting for instance to transition to running
Creating and assigning EIP to the instance.
Created EIP: 35.165.207.95
The web server will be available in a few minutes.
Public DNS: ec2-35-165-207-95.us-west-2.compute.amazonaws.com
InstanceId: i-01cefe7ef1a088899
Image: Microsoft Windows Server 2016 with Desktop Experience Locale English AMI provided by Amazon
Conclusion
Congratulations, you have automated the deployment of an OS, installation of a web server, and deployment of artifacts. By deploying your application with a script, you have created a completely consistent deployment workflow allowing you to scale and repair application servers with the push of a button.
You may find the following additional resources helpful: