How to make EC2 Spot Instance Data Persistent? (To Save Budget and For Fun)

This will be a way to run Spot Instance as an On Demand Instance, it can save money for people who run services on EC2 which does not requires a high SLA. There are a couple of ideas to solve the problem.

• Use another attached instance for user and application data, root instance will be only used for OS.
• Backup the data on root EBS all the time, when re-launch, restore the data from the backup.
• Set to not delete the EBS, when Spot instance is terminated, either restores it automatically or manually.

Surely, this discussion is only for the cases when the EC2 instance contains state data or persistent data, if the instance is only a stateless application layer, you do not need to worry about it at all.

Use Second Attached EBS for User and Application Data

Pro:

• Easy on AWS Configuration part, set a user script to attach the volume and restart, this is probably all are required for configuration on AWS side.

#!/bin/bash
      OUTPUT=$(curl http://169.254.169.254/latest/meta-data/instance-id)
      aws ec2 attach-volume --volume-id vol-xxxxxxxxxxxx --device /dev/xvdf --instance-id $OUTPUT --region ap-southeast-1

Cons:

• Tons of work to update the path for storing application/user data

I gave up this options, because for me it is almost impossible to make it, because there are so many configuration I need to update, for example, docker, many sides under `/var/www` etc.

Sync or Backup the Data all the time, When re-launch, restore the data from backup

Pro:

• Simple and easy to understand, backup all and restore all

Cons:

• Required more researching to backup the whole disk, not sure it is possible through tools/scripts or not
• Restoring might be slow

Set to Not Delete EBS after instance termination

Pro:

• Less work
• Flexible, the EBS is there with all your data, this can be either re-attached manually or by a script

Cons:

• Automatic re-attaching needs some work for wring the script
• If do it manually, the down time might be longer
• Needs researching for the trigger to re-attach the persistent EBS

Some ides to check and re-attach the EBS volume

The steps and the logic:

if instance (tag spot) unhealthy and the attached volumn is not the same with the existing root volume (tag spot-persistance)
  shutdown the instance (because the instance is respawned from spot request with new EBS)
  set the elastic ip to new instance
  detach the volume
  attach the volume with the tag
  start the instance
  delete the detached volume

These steps should be same for both manual step and script steps.

• Route53 health Check can be used here as a trigger, (the instance need to have a health endpoint), the basic health check costs $0.75/month.
• Lambda function can be used for the re-attach EBS volume, (it has a free tier, and the spot instance termination supposed to be rare)

Overall, it would be (your current EC2 on demand cost * 0.3 ~ 0.5 ) * 24 * 30 + 0.75. It would be dependents on what kinds of EC2 instance, but overall, it would save a small amount of money spending on AWS infrastructures.