Quickstart: mod_evasive and mod_security for httpd / apache

ModSecurity provides protection against common attacks on websites.

To install we should issue:

sudo yum install mod_security mod_security_crs

and then before production we should set 'SecRuleEngine On' to 'SecRuleEngine On'

vi /etc/httpd/conf.d/mod_security.conf

By default events will be logged to:

/var/log/httpd/modsec_audit.log

and rules (from mod_security_crs) can be found in:

/etc/httpd/modsecurity.d/activated_rules

while if you wish to create your own custom rules - these should be placed in:

/etc/httpd/modsecurity.d/local_rules

ModEvasive attempts to help with mitigating DoS/DDoS attacks.

Note: At this time I do not believe the mod_evasive module supports Event driven MPM (mpm_event_module) out of the box. However it should still work in prefork and worker modes.

You can verify which mode you are running it under with:

cat /etc/httpd/conf.modules.d/00-mpm.conf | grep LoadModule

To install we should issue:

sudo yum install mod_evasive

and then at the end of your httpd.conf file define your settings:

<IfModule mod_evasive20.c>
  DOSHashTableSize 3097
  DOSPageCount 2
  DOSSiteCount 50
  DOSPageInterval 1
  DOSSiteInterval 1
  DOSBlockingPeriod 60
  DOSEmailNotify [email protected]
</IfModule>

For changes to take effect we should ensure httpd is reloaded:

sudo service httpd reload

Read more

Running a non-https Wordpress site behind nginx that performs TLS termination

I was in the situation the other day where I had a fresh installation of a wordpress site which was sitting behind a reverse proxy (nginx in this case) which was terminating the SSL / TLS.

Now my initial thought was that this should work: users connect over HTTPS to the nginx, nginx proxies the request over HTTP to the wordpress server (httpd.) However after setting up the site I noticed that the formatting of the site was out and on close inspection noticed that my browser was blocking style sheets because they were being referenced as http://..... rather than https://.... We can verify this from within the Network Tab in the chrome developers tools (F12.)

So instead we need to instruct wordpress to use https when it's serving clients from nginx. However be aware that we will need to instruct nginx to send the 'X-Forwarded-Proto' HTTP header to httpd - although not all load balancers support this.

The 'X-Forwarded-Proto' header allows downstream servers (httpd) in this case to be aware of what protocol the client connecting to the upstream server (nginx) is using (https in this case.)

We should firstly add the following stanzas just below:

if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
 $_SERVER['HTTPS']='on';

if (isset($_SERVER["HTTP_X_FORWARDED_FOR"])) {
 $_SERVER['REMOTE_ADDR'] = $_SERVER["HTTP_X_FORWARDED_FOR"];
}

The second stanza ensures the clients IP is recorded accurately in the logs (opposed to the upstream load balancers.)

We need to add the following headers in our nginx vhost config:

location / {
  proxy_pass ...........
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
}

Restart nginx and httpd and we should now have a fully working wordpress site!

Read more

Changing the partition / disk of the /var folder in AWS EC2 Instances

This procedure is usually pretty straight forward in normal environments - it would go something like follows:

1. Attach, parition and format the disk
2. Drop down to runlevel 1
3. Mount the new parition and copy all data from /var into it.
4. Delete the /var folder, ensure the new parition is mounted as /var in fstab
5. Reboot.

However we are unable to drop down to runlevel 1 on an AWS EC2 instance so we are forced to take a slightly different approach.

Warning: Before doing anything like this you should ensure that you take a snapshot of the volume / disk before proceeding.

This has been tested on CentOS 7 - however this should work for the vast majority of Linux variants.

To start with we'll firstly create a new EBS volume and attach it to the relevant EC2 instance. These typically appear as /dev/xvdX - to retrieve the block device name we issue:

lsblk

We'll use LVM to manage our additional disks. For some reason the LVM toolset is not included in the CentOS distribution from the AWS Marketplace, so we'll need to install them manually:

sudo yum install lvm2

pvcreate /dev/xvdb

vgcreate data-vg /dev/xvdb

lvcreate -n data-lv -l 90%FREE data-vg

mkfs.xfs /dev/data-vg/data-lv

We can then mount this new filesystem with:

sudo mount -t auto /dev/data-vg/data-lv /mnt

We should also make a note of the current security context which will effect how SELinux (if enabled - and it really should be!) treats the directory:

ls -Zd /var

system_u:object_r:var_t:s0

We'll need to ensure that /mnt has the same security context - so we do:

sudo chcon -t var_t /mnt

Before we proceed we'll also want to ensure the nfs service is not running:

sudo systemctl stop nfs

We will then want to move all of the files within /var are copied to our new mountpoint.

To ensure the mv command copies all of the hidden files in /var we should use shopt to set the relevent option for bash:

shopt -s dotglob

and then copy the files:

sudo rsync -aulvXpogtr /var/* /mnt

Proceed by unmounting the /mnt volume:

sudo umount /mnt

Add an entry into fstab - for example:

/dev/data-vg/data-lv    /var    xfs    defaults,noatime,nofail 0   2

Move the existing /var directory to something like:

sudo mv /var /var.old

Re-create the /var directory and reboot the system:

sudo mkdir /var

Ensure it mounts properly:

sudo mount -a

fd -h

and finally if all looks well restart the system:

sudo shutdown -r now

Credit / based on the post here.

Read more

Tagging traffic with DSCP on CentOS 7 / iptables

Fortunately this is very easy to do in-guest with iptables.

To do this we make use of the mangle table - which allows us to modify packets before they leave the system.

sudo iptables -t mangle -A OUTPUT -j DSCP --set-dscp-class AF21  -m comment  --comment "set dscp class to AF21 for all outbound traffic"

Or if you use Puppet / hiera to manage your configuration - you'd use something like:

firewall_rules_common:
  dscp_markings:
    name: "058 Set DSCP AF21 for QoS"
    chain: OUTPUT
    table: mangle
    set_dscp_class: af21
    jump: DSCP

We can now use tcpdump to verify the outbound  traffic is being marked.

To do this however we need to firstly work out what the (decimal) ToS value of the DSCP class is.

In this case it's AF21 - so in decimal this equates to 72 - so we do:

sudo tcpdump -i eth0 -v ip[1]==72

Example output:

16:23:38.749359 IP (tos 0x48, ttl 64, id 35827, offset 0, flags [DF], proto TCP (6), length 316)
    test.server.ssh > 10.11.12.13.42632: Flags [P.], cksum 0x17ef (incorrect -> 0x9d68), seq 8127740:8128016, ack 5509, win 289, length 276

Note that the tos value in the above output is in decimal.

Read more