Tuesday, 20 February 2018

Changing the partition / disk of the /var folder in AWS EC2 Instances

This procedure is usually pretty straight forward in normal environments - it would go something like follows:

1. Attach, parition and format the disk
2. Drop down to runlevel 1
3. Mount the new parition and copy all data from /var into it.
4. Delete the /var folder, ensure the new parition is mounted as /var in fstab
5. Reboot.

However we are unable to drop down to runlevel 1 on an AWS EC2 instance so we are forced to take a slightly different approach.

Warning: Before doing anything like this you should ensure that you take a snapshot of the volume / disk before proceeding.

This has been tested on CentOS 7 - however this should work for the vast majority of Linux variants.

To start with we'll firstly create a new EBS volume and attach it to the relevant EC2 instance. These typically appear as /dev/xvdX - to retrieve the block device name we issue:


We'll use LVM to manage our additional disks. For some reason the LVM toolset is not included in the CentOS distribution from the AWS Marketplace, so we'll need to install them manually:

sudo yum install lvm2

pvcreate /dev/xvdb

vgcreate data-vg /dev/xvdb

lvcreate -n data-lv -l 90%FREE data-vg

mkfs.xfs /dev/data-vg/data-lv

We can then mount this new filesystem with:

sudo mount -t auto /dev/data-vg/data-lv /mnt

We should also make a note of the current security context which will effect how SELinux (if enabled - and it really should be!) treats the directory:

ls -Zd /var


We'll need to ensure that /mnt has the same security context - so we do:

sudo chcon -t var_t /mnt

Before we proceed we'll also want to ensure the nfs service is not running:

sudo systemctl stop nfs

We will then want to move all of the files within /var are copied to our new mountpoint.

To ensure the mv command copies all of the hidden files in /var we should use shopt to set the relevent option for bash:

shopt -s dotglob

and then copy the files:

sudo rsync -aulvXpogtr /var/* /mnt

Proceed by unmounting the /mnt volume:

sudo umount /mnt

Add an entry into fstab - for example:

/dev/data-vg/data-lv    /var    xfs    defaults,noatime,nofail 0   2

Move the existing /var directory to something like:

sudo mv /var /var.old

Re-create the /var directory and reboot the system:

sudo mkdir /var

Ensure it mounts properly:

sudo mount -a

fd -h

and finally if all looks well restart the system:

sudo shutdown -r now

Credit / based on the post here.

Wednesday, 7 February 2018

Tagging traffic with DSCP on CentOS 7 / iptables

Fortunately this is very easy to do in-guest with iptables.

To do this we make use of the mangle table - which allows us to modify packets before they leave the system.

sudo iptables -t mangle -A OUTPUT -j DSCP --set-dscp-class AF21  -m comment  --comment "set dscp class to AF21 for all outbound traffic"

Or if you use Puppet / hiera to manage your configuration - you'd use something like:

    name: "058 Set DSCP AF21 for QoS"
    chain: OUTPUT
    table: mangle
    set_dscp_class: af21
    jump: DSCP

We can now use tcpdump to verify the outbound  traffic is being marked.

To do this however we need to firstly work out what the (decimal) ToS value of the DSCP class is.

In this case it's AF21 - so in decimal this equates to 72 - so we do:

sudo tcpdump -i eth0 -v ip[1]==72

Example output:

16:23:38.749359 IP (tos 0x48, ttl 64, id 35827, offset 0, flags [DF], proto TCP (6), length 316)
    test.server.ssh > Flags [P.], cksum 0x17ef (incorrect -> 0x9d68), seq 8127740:8128016, ack 5509, win 289, length 276

Note that the tos value in the above output is in decimal.

Thursday, 18 January 2018

Setting up MPIO with iSCSI on Server 2012/2016 via Powershell

MPIO (in regards in iSCSI) in Windows Server allows us to utilise two or more connectors rather than using a single connector to hook up to our storage appliance.

By default MPIO is not installed / enabled on Server 2012/2016 - so we'll firstly enable it with:

Enable-WindowsOptionalFeature –Online –FeatureName MultiPathIO

To enable automatic use of MPIO we should issue:

Enable-MSDSMAutomaticClaim -BusType iSCSI

and then set the load balancing type we'll want to use with it (in this case we'll use a round robin style):

Set-MSDSMGlocalDefaultLoadBalancePolicy -Policy RR

and finally set the disk timeout period used for MPIO:

Set-MPIOSetting -NewDiskTimeout 45

And reboot the system:

shutdown /r -t 0

Set the interface IP addresses by firstly obtaining the interface index for the relevant ports:


and assign the relevant IP addresses:

New-NetIPAddress -InterfaceIndex <int> -IPAddress -PrefixLength 24

We'll also need to ensure that iSCSI support has been enabled:

Set-Service -Name msiscsi -StartupType Automatic
Start-Service msiscsi

We'll now proceed by configuring our iSCSI connectors via Powershell:

New-IscsiTargetPortal -TargetPortalAddress <SAN-IP-1> -InitiatorPortalAddress <LOCAL-SAN-IP-1> -InitiatorInstanceName "ROOT\ISCSIPRT\0000_0"
New-IscsiTargetPortal -TargetPortalAddress <SAN-IP-2> -InitiatorPortalAddress <LOCAL-SAN-IP-1> -InitiatorInstanceName "ROOT\ISCSIPRT\0000_0"

Now obtain the node address with:


and connect to them with:

Connect-IscsiTarget -IsMultipathEnabled $true -NodeAddress <node-address> -IsPersistent $true
Connect-IscsiTarget -IsMultipathEnabled $true -NodeAddress <node-address> -IsPersistent $true

Then get a list of the disks with:

Get-IscsiSession | Get-Disks

Wednesday, 27 December 2017

Hiera and how it works within Puppet

Hiera and how it works within Puppet

Hiera allows us to modify settings from modules within Puppet - for this example I will be tweaking some of the default settings from the saz/ssh module.

Let's start by firstly install hiera:

puppet module install puppet/hiera

Hiera makes use of hierarchies - for example servers in a specific location might need a paticular DNS server, however all servers might require a specific SSH configuration. These settings are defined within the hiera.yaml file:

cat /etc/puppetlabs/code/environments/production/hiera.yaml

version: 5
  # The default value for "datadir" is "data" under the same directory as the hiera.yaml
  # file (this file)
  # When specifying a datadir, make sure the directory exists.
  # See https://docs.puppet.com/puppet/latest/environments.html for further details on environments.
  # datadir: data
  # data_hash: yaml_data
  - name: "Per-node data (yaml version)"
    path: "nodes/%{::trusted.certname}.yaml"
  - name: "Other YAML hierarchy levels"
      - "common.yaml"

We're going to modify the heirarchy a little - so let's back it up firstly:

cp /etc/puppetlabs/code/environments/production/hiera.yaml /etc/puppetlabs/code/environments/production/hiera.yaml.bak

and replace it with:

version: 5
  # The default value for "datadir" is "data" under the same directory as the hiera.yaml
  # file (this file)
  # When specifying a datadir, make sure the directory exists.
  # See https://docs.puppet.com/puppet/latest/environments.html for further details on environments.
  # datadir: data
  # data_hash: yaml_data
  - name: "Per-Node"
    path: "nodes/%{::trusted.certname}.yaml"
  - name: "Operating System"
    path: "os/%{osfamily}.yaml"
  - name: "Defaults"
      - "common.yaml"

We now have the ability to set OS specific settings - for example some (older) operating systems might not support specific cipher suites.

Let's run the following on our client to identify what Puppet classifies it as:

facter | grep family

  family => "RedHat",

So let's create the relevent structure:

touch /etc/puppetlabs/code/environments/production/data/os/RedHat.yaml
touch /etc/puppetlabs/code/environments/production/data/os/Debian.yaml
touch /etc/puppetlabs/code/environments/production/data/os/common.yaml

We'll proceed by installing the saz/ssh module:

puppet module install saz/ssh

In this example we will concentrate on hardening the SSH server:

cat <<EOT > /etc/puppetlabs/code/environments/production/data/common.yaml
ssh::storeconfigs_enabled: true

    Protocol: '2'
        - ''
        - '%{::hostname}'
    PasswordAuthentication: 'no'
    SyslogFacility: 'AUTHPRIV'
    HostbasedAuthentication: 'no'
    PubkeyAuthentication: 'yes'
    UsePAM: 'yes'
    X11Forwarding: 'no'
    ClientAliveInterval: '300'
    ClientAliveCountMax: '0'
    IgnoreRhosts: 'yes'
    PermitEmptyPasswords: 'no'
    StrictModes: 'yes'
    AllowTcpForwarding: 'no'

We can check / test the values with:

puppet lookup ssh::server_options --merge deep --environment production --explain --node <node-name>

Finally restart the puppet server:

sudo service puppetserver restart

and poll the server from the client:

puppet client -t

Creating files from templates with Puppet

To utilise a templates when creating new files we can issue something like: /etc/puppetlabs/code/environments/production/manifests/site.pp

 file { '/etc/issue':
    ensure  => present,
    owner   => 'root',
    group   => 'root',
    mode    => 0644,
    content => template($module_name/issue.erb),

The source / content must be stored within a puppet module - so in the case we were using Saz's SSH module - we would place the template in:


touch /etc/puppetlabs/code/environments/production/modules/ssh/templates/issue.erb

and the site.pp file would look something like:

 file { '/etc/issue':
    ensure  => present,
    owner   => 'root',
    group   => 'root',
    mode    => 0644,
    content => template($ssh/issue.erb),

Reload the puppet server:

sudo service puppetserver reload

and pull down the configuration on the client:

puppet agent -t

Friday, 22 December 2017

Changing regional settings (locate, time zone, keyboard mappings) in CentOS 7 / RHEL

Quite often when deploying new instances of CentOS the process of setting regional settings like time is often hidden from the user behind the OS installer and more often than not it is not necessary to change these.

However this post will outline the steps that need to be taken if a server has been moved geographically or has simply not been configured correctly in the first place!

We'll start my changing the time zone - this is pretty straight forward and you can find timezone settings available to the system in:

ls -l /usr/share/zoneinfo/

In this case we'll use the GB (Great Britain) - by creating a symbolic link:

sudo rm /etc/localtime

ln -s /usr/share/zoneinfo/GB /etc/localtime

We'll also ensure NTP is installed:

sudo yum install ntp && sudo service ntpd start

sudo systemctl enable ntpd OR chkconfig ntpd on

We'll also want to change the system locale - we can view the current one with:



cat /etc/locale.conf

and change it by firstly identifying locales available to us:

localectl list-locales

and then set it with:

localectl set-locale en_IE.utf8

or manually change locale.conf:

cat "en_IE.utf8" > /etc/locale.conf

and confirm with:

localectl status

Finally we need to change the key mappings - to view the current selection issue:

localectl list-keymaps

and then set with:

localectl set-keymap ie-UnicodeExpert

Wednesday, 20 December 2017

vi(m) Cheat Sheet

The following is a list of common commands that I will gradually compile for working with vi / vim.

Find and Replace (Current Line)


Find and Replace (All Line)