Changing the partition / disk of the /var folder in AWS EC2 Instances

This procedure is usually pretty straight forward in normal environments - it would go something like follows:

1. Attach, parition and format the disk
2. Drop down to runlevel 1
3. Mount the new parition and copy all data from /var into it.
4. Delete the /var folder, ensure the new parition is mounted as /var in fstab
5. Reboot.

However we are unable to drop down to runlevel 1 on an AWS EC2 instance so we are forced to take a slightly different approach.

Warning: Before doing anything like this you should ensure that you take a snapshot of the volume / disk before proceeding.

This has been tested on CentOS 7 - however this should work for the vast majority of Linux variants.

To start with we'll firstly create a new EBS volume and attach it to the relevant EC2 instance. These typically appear as /dev/xvdX - to retrieve the block device name we issue:

lsblk

We'll use LVM to manage our additional disks. For some reason the LVM toolset is not included in the CentOS distribution from the AWS Marketplace, so we'll need to install them manually:

sudo yum install lvm2

pvcreate /dev/xvdb

vgcreate data-vg /dev/xvdb

lvcreate -n data-lv -l 90%FREE data-vg

mkfs.xfs /dev/data-vg/data-lv

We can then mount this new filesystem with:

sudo mount -t auto /dev/data-vg/data-lv /mnt

We should also make a note of the current security context which will effect how SELinux (if enabled - and it really should be!) treats the directory:

ls -Zd /var

system_u:object_r:var_t:s0

We'll need to ensure that /mnt has the same security context - so we do:

sudo chcon -t var_t /mnt

Before we proceed we'll also want to ensure the nfs service is not running:

sudo systemctl stop nfs

We will then want to move all of the files within /var are copied to our new mountpoint.

To ensure the mv command copies all of the hidden files in /var we should use shopt to set the relevent option for bash:

shopt -s dotglob

and then copy the files:

sudo rsync -aulvXpogtr /var/* /mnt

Proceed by unmounting the /mnt volume:

sudo umount /mnt

Add an entry into fstab - for example:

/dev/data-vg/data-lv    /var    xfs    defaults,noatime,nofail 0   2

Move the existing /var directory to something like:

sudo mv /var /var.old

Re-create the /var directory and reboot the system:

sudo mkdir /var

Ensure it mounts properly:

sudo mount -a

fd -h

and finally if all looks well restart the system:

sudo shutdown -r now

Credit / based on the post here.