Thursday, 22 June 2017

Setting up snort, DAQ and PF_RING on CentOS 7

Let's firstly download and build the PF_RING kernel module:

yum -y install kernel-devel kernel-headers libtool automake autoconf flex bison gcc

cd /tmp
wget http://packages.ntop.org/rpm7/x64/PF_RING/pfring-6.7.0-1286.x86_64.rpm

Download and install DAQ from the snort site:

cd /tmp
wget https://www.snort.org/downloads/snort/daq-2.0.6-1.f21.x86_64.rpm
rpm -i daq*

and then build the DAQ module for PF_RING:

git clone https://github.com/ntop/PF_RING.git
cd PF_RING/userland/snort/pfring-daq-module
autoreconf -ivf
./configure
make & make install

This should copy the library to: /usr/local/lib/daq/daq_pfring.so

Finally download and configure snort:

cd /tmp
wget https://www.snort.org/downloads/snort/snort-openappid-2.9.9.0-1.centos7.x86_64.rpm
yum install snort-openappid-2.9.9.0-1.centos7.x86_64.rpm

We can then run snort in either IDS mode:

snort --daq-dir=/usr/local/lib/daq --daq pfring --daq-mode passive -i ethX -v -e -c /etc/snort/snort.conf

or IPS mode:

snort --daq-dir=/usr/local/lib/daq --daq pfring  -i ethX:ethY -e -Q -c /etc/snort/snort.conf

We can also update the SNORT definitions with:

cd /tmp
wget https://www.snort.org/downloads/registered/snortrules-snapshot-2990.tar.gz
tar zxvf snortrules*
cd snortrules*
cd etc
cp * /etc/snort
cd ../rules
cp * /etc/snort/rules
cd ..
cp -R preproc_rules /etc/snort
cp -R so_rules /etc/snort

After attempting to start snort again I received a number of complaints about bad folder paths - so I ended up creating several sym links to get it working correctly:

ln -s /usr/lib64/snort-2.9.9.0_dynamicengine/ /usr/local/lib/snort_dynamicengine
ln -s /usr/lib64/snort-2.9.9.0_dynamicpreprocessor/ /usr/local/lib/snort_dynamicpreprocessor
ln -s /etc/snort/so_rules/precompiled/Centos-5-4/x86-64/2.9.9.0/ /usr/local/lib/snort_dynamicrules

And also modifying some of the directory variables in snort.config like so_rules and rules.

And finally creating a few files:

touch /etc/snort/rules/white_list.rules
touch /etc/snort/rules/black_list.rules

Tuesday, 20 June 2017

Identifying a network bottleneck / packet loss on CentOS

We can check for for packet loss at the hardware level using ethtool:

ethtool -S eth0

Typically (although can vary) you are looking for a counter such as:

rx_*_errors

When the packets are received from the NIC they are then placed into send and receive queues - we should ensure that none of them are currently full - we can view these with:

ss -nmp

and finally check for protocol errors with netstat:

netstat -s

Sources

https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf

Packet Capture: Fine tuning Linux for 10gb NIC's / busy networks

Below I have outlined some of the more important tweaks that can be applied on a Linux system in order to optimise performance with 10gb NICs and busy networks where there is a high volume of throughput.

As a fornote when capturing packets with 10gb cards you should also ensure that you have a sufficient CPU and available IOPS - I'd recommend an SSD for best performance.

Hardware

While libpcap will work with pretty much any NIC if you want to use PF_RING (which is strongly recommended due to performance benefits) you will need an Intel 82599-based NIC and ensure the Linux kernel is above 2.6.31 (which should be pretty much every mainstream distribution these days.)

There are also other specialist NIC's that are supported and can also perform hardware packet filtering - however for the purposes of this tutorial we will be sticking with an Intel based chip.

Tuning

Firstly we should run a network performance tool - such as iperf to benchmark throughput:

sudo yum install iperf

and on the server side issue:

iperf -s

and the client side:

iperf -c server.ip.address -w64k -t60

You'll also want to monitor the cpu during this period e.g.:

mpstat 5

This will also provide us with something to contrast performance with when we have finished performing the tweaks.

RX Descriptor Sizes

The descriptors do not hold any packet data - rather contain information about the whereabouts of the data is in memory. These values are often not set at the maximum - in order to verify your current descriptor levels you can run:

ethtool -g eth0

Example output:

Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 2048
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 128
TX: 512

We can then increase the descriptors as follows:

ethtool -G eth0 rx 4096 tx 4096

Jumbo Frames

One of the obvious considerations is enabling Jumbo frames on the interface - although this is presuming that the application(s) support them! We can enable this on a per interface level with:

vi /etc/sysconfig/network-scripts/eth0

and append / change:

MTU=9000

sudo service network restart


RX and TX Checksum Offload

Each time a packet is received or sent the CPU calculates a checksum - enabling this feature forces the NIC to calculate this instead - hence freeing up CPU.

This can be enabled on a per interface level with:

ethtool --offload eth0 tx on rx on

* Note: Saving CPU with TX checksum offload is dependant on how large the frame packet sizes are - larger packets equate to a greater saving.

Kernel Tweaking

Removing TCP time-stamping is another way to reduce CPU load - however you (obviously) lose the round trip time of the segment:

sysctl -w net.ipv4.tcp_ timestamps=0

And increasing the syn and network driver backlog with:

net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 2500

and tcp read, write limits:

net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

and socket buffer space limits:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

and backlogged sockets with (default is 100):

net.core.somaxconn = 1024

Sources

https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf

Disabling the PC speaker / beep sound with CentOS 7 / Core

One of the not so common issues I encountered with the Core edition of CentOS was the pc speaker - - the vast majority of the time sound means little when the machine has been virtualised.

I quickly realised that the relevant alsa tools are (expectantly) not provided  and need to be installed.

Doing a quick search for alsamixer with yum brings up the 'alsa-utils' package:

sudo yum provides alsamixer

sudo yum -y install alsa-utils

We can then use the alsamixer utility to adjust the speaker volume:

alsamixer

or if we prefer to do it via command line we can issue:

amixer set <sound-card-name> Mute | Unmute

(Use the 'axmixer' command on it's own to view available sound devices.)

To ensure the changes persist across reboot we should issue:

alsactl store

Unfortunately the system beeps (which will quickly drive you mad when working from the console) persisted  - this can be done either by using the setterm utility:

setterm -blength 0

or we can simply disable the speaker module all together with:

rmmod -v pcspkr

and to ensure the changes persist we should add this module to the blacklist:

echo 'blacklist pcspkr' >> /etc/modprobe.d/blacklist

I realise there are alternative methods to do this - however this was the easiest and most efficient for myself.

Wednesday, 7 June 2017

Building VMWare Kernel Modules Fails: Fail to find version.h

After upgrading the kernel the other day I attempted to recompile the VMWare kernel module although it ended up failing. After reviewing the logs I noticed that it was complaining about not finding 'version.h' - although the kernel-header package was installed:

2017-06-07T10:18:44.574+01:00| vthread-4| I125: Setting header path for 4.11.3-200.fc25.x86_64 to "/lib/modules/4.11.3-200.fc25.x86_64/build/include".
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Validating path "/lib/modules/4.11.3-200.fc25.x86_64/build/include" for kernel release "4.11.3-200.fc25.x86_64".
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Failed to find /lib/modules/4.11.3-200.fc25.x86_64/build/include/linux/version.h

Turns out that this file is actually kept in: /usr/include/linux/

So in order to get the VMWare module to compile I ended up copying it to the current kernel's header directory:

sudo cp /usr/include/linux/version.h /lib/modules/`uname -r`/build/include/linux/

Monday, 5 June 2017

QoS: Traffic Policing vs Traffic Shaping

This article will focus on understanding how QoS techniques such as traffic policing and shaping are performed (and contrasted) and how values such as burst rates can be calculated.

One of the fundamental differences between the two is that policing involves dropping traffic when the bucket is full - while shaping put excess traffic into a queue  for submission and is gradually released resulting in a smoother flow of traffic.

This can be illustrated below (credit to Cisco for this diagram):



It is also worth noting that traffic shaping only works on outbound traffic (traffic leaving the device) - while policing will work on both ingress (traffic coming to the device) and egress traffic.

Key Terms / Formulas

I'll firstly go over some of the key terms:

Committed Information Rate (CIR) = 10000000 (10Mbps)
Burst Commit Bucket (Bc) = CIR * 0.125s = 1875000
Time Interval (Tc) = Bc / CIR = 0.125s

In this case the bucket will be emptied after 0.125s - which for some purposes might be perfectly fine - however if you are supporting a large file server you'd want the TC to be much higher - the burst rate and Tc will depend greatly on your network type and quite often you will have to tweak it for best performance. The formular for burst rate is above is the default one provided by Cisco in their documentation.

Traffic Policing Example (Egress)

Presuming the port speed is 100mb - the following configuration would limit egress traffic to 20mb:

int fa0/1-48
srr-queue bandwidth limit 80

Traffic Policing Example (Ingress)

We have to create a service policy for ingress policing:

mls qos

class-map match-all rate-limit
  description Bandwidth Control
 match ip dscp default

policy-map GeneralTraffic
 class rate-limit
  police 10000000 192000 exceed-action drop

int range fa01-48
service-policy input GeneralTraffic

Traffic Shaping Example (Egress)





Sources

http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html

Saturday, 3 June 2017

Setting up / configuring an NTP client on CentOS 7

Firstly ensure that the appropriate timezone is set on the system - for example:

ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime

Install the ntp deamon:

yum install ntp -y && systemctl enable ntp

and then start the service with:

ntpdate pool.ntp.org && systemctl start ntpd