Wednesday, 21 March 2018

Applying QoS on a Cisco 3650 / 3850

Note: In older models / OS's we needed to issue 'mls qos' in order to enable QoS on the switch - however with the 3650 QoS is enabled by default.

The aim is to tag voice traffic with DSCP that hits either gi0/1 or gi0/2 from VLAN 10 and ensuring data traffic is even spread between vlan20 and vlan30:

vlan 10
desc priority traffic
name VLAN10

vlan 20
desc non-priority data traffic 1
name VLAN20

vlan 30
desc non-priority data traffic 1
name VLAN20

We'll need to use MQC (Modular QoS CLI) in order to apply the QoS - so let's firstly create our class map:

####### Set ingress service policy

class-map qos_vlan10_cm_in
match vlan 10

class-map qos_vlan20_cm_in
match vlan 20

class-map qos_vlan20_cm_in
match vlan 30

policy-map qos_pm_in
class qos_vlan10_cm_in
set ip dscp 40
class qos_vlan20_cm_in
set ip dscp 0
class qos_vlan30_cm_in
set ip dscp 0

int gi1/0/1
service-policy input qos_pm_in

int gi1/0/2
service-policy input qos_pm_in

And now for the egress QoS:

class-map qos_vlan10_cm_out
match ip dscp 40

class-map qos_vlan20_cm_out
match ip dscp 0

class-map qos_vlan30_cm_out
match ip dscp 0

policy-map qos_pm_out
class qos_vlan10_cm_out
priority 1 percent 10
class qos_vlan20_cm_out
bandwidth percent 20
class qos_vlan30_cm_out
bandwidth percent 20
class class-default
bandwidth remaining percent 100

# The 'bandwidth percent' command provides 20% of the interfaces bandwdith to vlan20 and 20% to vlan30 - however during times were there is no contention other classes can utilise additional bandwidth. I.e. The command doesn't reserve bandwidth. While the remaining 100% of the bandwidth is shared between other traffic.

# The 'priority level' command always ensures that traffic from VLAN10 is always served before anything else and reserves 10% of the link (i.e. 100 megabits) for the voice traffic - however it is not able to utilise any greater than 10%!

Finall apply the service policy to the outgoing interface with:

int gi1/0/24
service-policy output qos_pm_out

Monday, 12 March 2018

Understanding VFS (Virtual File System), inodes and thier role

VFS (Virutal File System)

The virtual file system manages all of the real filesystems mounted at a given time for example xfs, ext4 etc. Each file system registers itself with the VFS during initialization. Each real file system either requires support built directly into the kernel or in the form or modules.

Each filesystem mounted by the VFS has a corrosponding superblock (that is a VFS superblock - not an EXT3 superblock - they are similar in nature however distinct.) A VFS superblock contains the following information:

- Device (e.g. /dev/sda1)
- Inode Pointers (The mounted inode pointers points to the first inode on the filesystem)
- Block size of the filesystem
- Superblock operations (A pointer to a set of superblock routines for this file system)
- File system type (A pointer to the mounted file system's file_system_type  data structure e.g. XFS)
- File system specific (A pointer to information needed by this file system)

VFS inodes

Inodes are used to describe each file system object on a systems - for example a folder or directory. They consist of the following information:

- File Size
- File Type (e.g. regular file, directory, block device etc.)
- Group
- Number of links
- Permissions
- File Access / Modify and Change times
- Extended Attributes
- ACL's

Each inode is identified by a unique inode number - for example we can inspect the inode information about '/etc/passwd' with the stat command:

>> stat /etc/passwd
File: ‘/etc/passwd’
Size: 1578            Blocks: 8          IO Block: 4096   regular file
Device: ca01h/51713d    Inode: 18082       Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:passwd_file_t:s0
Access: 2018-03-12 11:46:01.201484317 +0000
Modify: 2017-01-19 11:45:27.790745205 +0000
Change: 2017-01-19 11:45:27.793745185 +0000

It's also important to realise that the inode itself does not hold any data - it's purely used for descriptive purposes. 

We can delete a file system object directly with it's assosiated inode number - for example:

cd / && find . -inum 18082 -exec rm -i {} \;

Note: inodes in VFS are different from that of filesystems such as ext2, ext3 etc. An inode within the VFS can be referencing a file situated on one  several different file systems.

Friday, 9 March 2018

Manually adding a host SSH fingerprint into known_hosts

I noticed that when connecting to a non-standard port for SSH e.g. that the SSH host fingerprint was not being added to the users known_hosts file. So in order to perform this manually we should issue:

ssh-keygen -p 2020 ssh-rsa AAAABBBBBBCCCCCC.....

and then append it to our known_hosts

echo AAAABBBBBBCCCCCC..... >> ~/.ssh/known_hosts

Tuesday, 27 February 2018

Quickstart: mod_evasive and mod_security for httpd / apache

ModSecurity provides protection against common attacks on websites.

To install we should issue:

sudo yum install mod_security mod_security_crs

and then before production we should set 'SecRuleEngine On' to 'SecRuleEngine On'

vi /etc/httpd/conf.d/mod_security.conf

By default events will be logged to:


and rules (from mod_security_crs) can be found in:


while if you wish to create your own custom rules - these should be placed in:


ModEvasive attempts to help with mitigating DoS/DDoS attacks.

Note: At this time I do not believe the mod_evasive module supports Event driven MPM (mpm_event_module) out of the box. However it should still work in prefork and worker modes.

You can verify which mode you are running it under with:

cat /etc/httpd/conf.modules.d/00-mpm.conf | grep LoadModule

To install we should issue:

sudo yum install mod_evasive

and then at the end of your httpd.conf file define your settings:

<IfModule mod_evasive20.c>
  DOSHashTableSize 3097
  DOSPageCount 2
  DOSSiteCount 50
  DOSPageInterval 1
  DOSSiteInterval 1
  DOSBlockingPeriod 60

For changes to take effect we should ensure httpd is reloaded:

sudo service httpd reload

Monday, 26 February 2018

Running a non-https Wordpress site behind nginx that performs TLS termination

I was in the situation the other day where I had a fresh installation of a wordpress site which was sitting behind a reverse proxy (nginx in this case) which was terminating the SSL / TLS.

Now my initial thought was that this should work: users connect over HTTPS to the nginx, nginx proxies the request over HTTP to the wordpress server (httpd.) However after setting up the site I noticed that the formatting of the site was out and on close inspection noticed that my browser was blocking style sheets because they were being referenced as http://..... rather than https://.... We can verify this from within the Network Tab in the chrome developers tools (F12.)

So instead we need to instruct wordpress to use https when it's serving clients from nginx. However be aware that we will need to instruct nginx to send the 'X-Forwarded-Proto' HTTP header to httpd - although not all load balancers support this.

The 'X-Forwarded-Proto' header allows downstream servers (httpd) in this case to be aware of what protocol the client connecting to the upstream server (nginx) is using (https in this case.)

We should firstly add the following stanzas just below:

if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')

if (isset($_SERVER["HTTP_X_FORWARDED_FOR"])) {

The second stanza ensures the clients IP is recorded accurately in the logs (opposed to the upstream load balancers.)

We need to add the following headers in our nginx vhost config:

location / {
  proxy_pass ...........
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;

Restart nginx and httpd and we should now have a fully working wordpress site!

Tuesday, 20 February 2018

Changing the partition / disk of the /var folder in AWS EC2 Instances

This procedure is usually pretty straight forward in normal environments - it would go something like follows:

1. Attach, parition and format the disk
2. Drop down to runlevel 1
3. Mount the new parition and copy all data from /var into it.
4. Delete the /var folder, ensure the new parition is mounted as /var in fstab
5. Reboot.

However we are unable to drop down to runlevel 1 on an AWS EC2 instance so we are forced to take a slightly different approach.

Warning: Before doing anything like this you should ensure that you take a snapshot of the volume / disk before proceeding.

This has been tested on CentOS 7 - however this should work for the vast majority of Linux variants.

To start with we'll firstly create a new EBS volume and attach it to the relevant EC2 instance. These typically appear as /dev/xvdX - to retrieve the block device name we issue:


We'll use LVM to manage our additional disks. For some reason the LVM toolset is not included in the CentOS distribution from the AWS Marketplace, so we'll need to install them manually:

sudo yum install lvm2

pvcreate /dev/xvdb

vgcreate data-vg /dev/xvdb

lvcreate -n data-lv -l 90%FREE data-vg

mkfs.xfs /dev/data-vg/data-lv

We can then mount this new filesystem with:

sudo mount -t auto /dev/data-vg/data-lv /mnt

We should also make a note of the current security context which will effect how SELinux (if enabled - and it really should be!) treats the directory:

ls -Zd /var


We'll need to ensure that /mnt has the same security context - so we do:

sudo chcon -t var_t /mnt

Before we proceed we'll also want to ensure the nfs service is not running:

sudo systemctl stop nfs

We will then want to move all of the files within /var are copied to our new mountpoint.

To ensure the mv command copies all of the hidden files in /var we should use shopt to set the relevent option for bash:

shopt -s dotglob

and then copy the files:

sudo rsync -aulvXpogtr /var/* /mnt

Proceed by unmounting the /mnt volume:

sudo umount /mnt

Add an entry into fstab - for example:

/dev/data-vg/data-lv    /var    xfs    defaults,noatime,nofail 0   2

Move the existing /var directory to something like:

sudo mv /var /var.old

Re-create the /var directory and reboot the system:

sudo mkdir /var

Ensure it mounts properly:

sudo mount -a

fd -h

and finally if all looks well restart the system:

sudo shutdown -r now

Credit / based on the post here.

Wednesday, 7 February 2018

Tagging traffic with DSCP on CentOS 7 / iptables

Fortunately this is very easy to do in-guest with iptables.

To do this we make use of the mangle table - which allows us to modify packets before they leave the system.

sudo iptables -t mangle -A OUTPUT -j DSCP --set-dscp-class AF21  -m comment  --comment "set dscp class to AF21 for all outbound traffic"

Or if you use Puppet / hiera to manage your configuration - you'd use something like:

    name: "058 Set DSCP AF21 for QoS"
    chain: OUTPUT
    table: mangle
    set_dscp_class: af21
    jump: DSCP

We can now use tcpdump to verify the outbound  traffic is being marked.

To do this however we need to firstly work out what the (decimal) ToS value of the DSCP class is.

In this case it's AF21 - so in decimal this equates to 72 - so we do:

sudo tcpdump -i eth0 -v ip[1]==72

Example output:

16:23:38.749359 IP (tos 0x48, ttl 64, id 35827, offset 0, flags [DF], proto TCP (6), length 316)
    test.server.ssh > Flags [P.], cksum 0x17ef (incorrect -> 0x9d68), seq 8127740:8128016, ack 5509, win 289, length 276

Note that the tos value in the above output is in decimal.