When initially setting snort up you will likely come across one or two (or several) false positives.
For example in my case a specific server was being flagged when users were downloading a specific file from it over the network. The 'alert' being generated was consistent and so I wanted to ensure that this rule is not applied when the traffic was being sourced from this particular server.
Fortunately snort allows us to do this without having to completely disable the rule all together.
This can be applied in the 'thresholds.conf' file and is known as a 'supression.'
sudo vi /etc/snort/thresholds.conf
and adding something like:
suppress gen_id 1, sig_id 39463, track by_src, ip 10.11.12.13
and reload snort with:
sudo systemctl reload snort
Pages
▼
Friday, 30 June 2017
Thursday, 29 June 2017
A simple systemd service template
Using the below template you can easily create a simple systemd init / service script:
sudo vi /usr/lib/systemd/system/myservice.service
and add:
[Unit]
Description=Snort NIDS Daemon
After=syslog.target network.target
[Service]
Type=simple
ExecStart=/usr/bin/myprogram -myparams
sudo vi /usr/lib/systemd/system/myservice.service
and add:
[Unit]
Description=Snort NIDS Daemon
After=syslog.target network.target
[Service]
Type=simple
ExecStart=/usr/bin/myprogram -myparams
Ensure it start on boot:
sudo systemctl enable myservice
and attempt to start it:
sudo service myservice start && sudo service myservice status
Setting up Barnyard2 using postgresql / postgres on CentOS 7
Important Fornote: Ensure that your snort instance output mode is set to unified in the snort.conf file e.g.:
output unified2: filename merged.log, limit 128
Do not include the 'nostamp' option with the above statement (as it is by default) otherwise barnyard will fail to read the logs.
and restart with:
sudo service snort restart
yum install autogen libpcap-devel postgresql-devel daq-devel libdnet-devel
cd /tmp
git clone https://github.com/firnsy/barnyard2.git
cd barnyard2
./autogen
./configure --with-postgresql
make
sudo make install
We can then copy the example config:
cp etc/barnyard.conf /etc/barnyard.conf
And ensure the following lines are set appropriately:
config daemon
config hostname: localhost
config interface: eth0
config logdir: /var/log/barnyard2/
config waldo_file: /var/log/barnyard2/barnyard2.waldo
and ensure the following line is present / uncommented:
output alert_fast: stdout
Important Note: Ensure 'deamon' mode is commented out in the barnyard.conf file otherwise you won't be able to debug easily when running the following command.
We can then test the configuration with:
sudo /usr/local/bin/barnyard2 -c /etc/barnyard2.conf -d /var/log/snort -f snort.log -w /var/log/barnyard2/barnyard2.waldo
We can also add a test rule into snort e.g. the following:
vi /etc/snort/rules/local.rules
alert icmp any any -> any any (msg: "ICMP Packet found"; sid:1000001; rev1;)
and restart snort with:
sudo service snort restart
This should hopefully generate a fair few events - even on smaller networks - if sending some ICMP traffic yourself with a ping or traceroute.
Once you are happy with the results we can then comment out the following (in barnyard.conf):
output alert_fast: stdout
and replace it with our postgresql server:
output database: log, postgresql, user=<username> password=<password> dbname=<snorby-database> host=localhost sensor_name=sensor1
Stop and start barnyard2 in using 'daemon' mode (-D) this time:
sudo /usr/local/bin/barnyard2 -c /etc/barnyard2.conf -d /var/log/snort -f snort.log -w /var/log/barnyard2/barnyard2.waldo -D
Then verify events are present in Snorby - you might also need to restart the worker in some cases.
I ended up purging some of the snort logs and had to delete the .waldo (bookmark) file in order to get barnyard2 to start picking up logs again - once you've deleted the file simply 'touch' it:
touch /var/log/barnyard2/barnyard2.waldo
If all goes to plan we can now create a systemd service as follows:
vi /usr/lib/systemd/system/barnyard2.service
and add the following:
[Unit]
Description=Snort NIDS Daemon
After=syslog.target network.target
Requires=snort.service
Requires=snort.service
[Service]
Type=simple
ExecStart=/usr/local/bin/barnyard2 -c /etc/barnyard2.conf -d /var/log/snort -f snort.log -w /var/log/barnyard2/barnyard2.waldo
[Install]
WantedBy=multi-user.target
Enable and start the service:
sudo systemctl enable
sudo service barnyard2 start
Tuesday, 27 June 2017
Quickstart: Installing and configuring snorby on CentOS 7
This tutorial will demonstrate how to build and configure Snorby on CentOS 7.
Fornote: I will initally be disabling SELinux for the installation of Snorby - however at the end we will re-enable it and adjust the relevant rules in order to get it running nicely with Snorby.
Lets firstly install the libraries needed to compile some of the ruby gems:
yum install mysql-devel libpqxx-devel ruby-devel
cd /tmp
yum install ruby
gem update
gem install rails
(Again - I had to install an older version of rails in order to get it working with Ruby 1.9.3)
Also - ensure selinux is set to permissive mode with:
setenforce 0
vi /etc/selinux/config # set mode to 'permissive'.
We'll now download and configure Snorby:
mkdir -p /opt/snorby/app
cd /opt/snorby/app
git clone git://github.com/Snorby/snorby.git
cd snorby
bundle install
yum install wkhtmltopdf
We'll now sort of the configuration file - firstly copy the template as follows:
cd etc
cp snorby_config.yml.example snorby_config.yml
and under 'Production' we'll change the 'wkhtmltopdf' variable to: /usr/bin/wkhtmltopdf
You can also specify mail setting within:
config/initializers/mail_config.rb
I will be using postgres for the database portion - there is a post here that demonstrates how to get posgres up and running.
We need to ensure that the snorby rake setup can connect with the database - so we should edit pg_hba.conf - something like the below should do the trick:
vi /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all postgres peer
local all snorby md5
host all snorby 127.0.0.1/32 md5
host all snorby ::1/128 md5
and reload the server for changes to take effect:
sudo service postgresql reload
We'll now edit the database configuration - ensuring the 'adapter' variable is set to 'postgresql', user and passwords are set accordingly etc.
cp database.yml.example database.yml
and then run the Snorby setup with:
rake snorby:setup
We'll now attempt to start the rails application in production mode:
cd config
sudo rails s -e production
With any luck you should now be able to access Snorby on http://<ip>:3000
The default username / password is: [email protected] / snorby.
Once logged in go to 'Administration' >> 'Worker and Job Queue' and ensure the work has started.
Our next step is to configure nginx with passenger so we can let nginx server our pages. However we will either need to compile passenger and nginx from scratch or we can use the Phusion Pasenger repository - which in this case to save time (and my sanity) we will do:
yum install yum-utils
sudo curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
sudo yum install -y nginx passenger || sudo yum-config-manager --enable cr && sudo yum install -y nginx passenger
Uncomment 'passenger_root', 'passenger_ruby' and 'passenger_instance_registry_dir' from /etc/nginx/conf.d/passenger.conf
sudo vi /etc/nginx/conf.d/passenger.conf
Restart and enable nginx with:
sudo systemctl enable nginx
sudo service nginx restart
and validate the install with:
sudo /usr/bin/passenger-config validate-install
We can do add a server block for snorby (after testing you should setup a proper virtual host etc.)
Edit nginx.conf:
vi /etc/nginx/nginx.conf
and comment out the default server block and add something like follows:
server {
listen 80 default_server;
server_name localhost;
root /opt/snorby;
rails_env production;
passenger_app_root /opt/snorby/application
passenger_enabled on;
passenger_ruby /usr/local/rvm/rubies/ruby-2.2.2/bin/ruby;
passenger_sticky_sessions on;
}
Make sure the data directory is writable by nginx:
chown -R nginx:nginx /opt/snorby/
Tes the configuration and reload nginx:
nginx -t
sudo service nginx reload
When initially attempting access the site it bombed out and after going through the nginx error logs the following line caught my attention:
Missing proper 'which' command. Make sure it is installed before using RVM!
and also lines like (which was causing the Snorby Worker process from starting):
stderr: sh: env: command not found
Now - I was pretty sure this was already installed - and to confirm we can check with:
rpm -qa which
which-2.20-7.el7.x86_64
So my next though was that it was likely an environment variable (specifically PATH) issue. It turns out nginx (by default) 'nukes' all environment values and in order to preserve them (specifically 'PATH' in this case) we need to add the following to the 'http' stanza in nginx.conf:
env PATH;
and reload:
sudo service nginx reload
Although the above may seem pretty trivial it actually look me a while to get it all up and running with a lot of trial and error - I really hope the developers will streamline the process and fix a number of outstanding bugs as I can imagine a lot of people would be put of my the amount of work involved getting it running!
Performing a hard reset
If you want to start from scratch or simply just purge the snorby database you can do so by doing:
cd /opt/snorby/application
and issuing:
bundle exec rake snorby:hard_reset
SELinux Rules
Will be available soon...
Sources
https://www.phusionpassenger.com/library/install/nginx/install/oss/trusty/
https://stackoverflow.com/questions/32071190/cant-deploy-passenger-with-nginx
Fornote: I will initally be disabling SELinux for the installation of Snorby - however at the end we will re-enable it and adjust the relevant rules in order to get it running nicely with Snorby.
Lets firstly install the libraries needed to compile some of the ruby gems:
yum install mysql-devel libpqxx-devel ruby-devel
cd /tmp
yum install ruby
gem update
gem install rails
(Again - I had to install an older version of rails in order to get it working with Ruby 1.9.3)
Also - ensure selinux is set to permissive mode with:
setenforce 0
vi /etc/selinux/config # set mode to 'permissive'.
We'll now download and configure Snorby:
mkdir -p /opt/snorby/app
cd /opt/snorby/app
git clone git://github.com/Snorby/snorby.git
cd snorby
bundle install
yum install wkhtmltopdf
We'll now sort of the configuration file - firstly copy the template as follows:
cd etc
cp snorby_config.yml.example snorby_config.yml
and under 'Production' we'll change the 'wkhtmltopdf' variable to: /usr/bin/wkhtmltopdf
You can also specify mail setting within:
config/initializers/mail_config.rb
I will be using postgres for the database portion - there is a post here that demonstrates how to get posgres up and running.
We need to ensure that the snorby rake setup can connect with the database - so we should edit pg_hba.conf - something like the below should do the trick:
vi /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all postgres peer
local all snorby md5
host all snorby 127.0.0.1/32 md5
host all snorby ::1/128 md5
and reload the server for changes to take effect:
sudo service postgresql reload
We'll now edit the database configuration - ensuring the 'adapter' variable is set to 'postgresql', user and passwords are set accordingly etc.
cp database.yml.example database.yml
and then run the Snorby setup with:
rake snorby:setup
We'll now attempt to start the rails application in production mode:
cd config
sudo rails s -e production
With any luck you should now be able to access Snorby on http://<ip>:3000
The default username / password is: [email protected] / snorby.
Once logged in go to 'Administration' >> 'Worker and Job Queue' and ensure the work has started.
Our next step is to configure nginx with passenger so we can let nginx server our pages. However we will either need to compile passenger and nginx from scratch or we can use the Phusion Pasenger repository - which in this case to save time (and my sanity) we will do:
yum install yum-utils
sudo curl --fail -sSLo /etc/yum.repos.d/passenger.repo https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger.repo
sudo yum install -y nginx passenger || sudo yum-config-manager --enable cr && sudo yum install -y nginx passenger
Uncomment 'passenger_root', 'passenger_ruby' and 'passenger_instance_registry_dir' from /etc/nginx/conf.d/passenger.conf
sudo vi /etc/nginx/conf.d/passenger.conf
Restart and enable nginx with:
sudo systemctl enable nginx
sudo service nginx restart
and validate the install with:
sudo /usr/bin/passenger-config validate-install
We can do add a server block for snorby (after testing you should setup a proper virtual host etc.)
Edit nginx.conf:
vi /etc/nginx/nginx.conf
and comment out the default server block and add something like follows:
server {
listen 80 default_server;
server_name localhost;
root /opt/snorby;
rails_env production;
passenger_app_root /opt/snorby/application
passenger_enabled on;
passenger_ruby /usr/local/rvm/rubies/ruby-2.2.2/bin/ruby;
passenger_sticky_sessions on;
}
Make sure the data directory is writable by nginx:
chown -R nginx:nginx /opt/snorby/
Tes the configuration and reload nginx:
nginx -t
sudo service nginx reload
When initially attempting access the site it bombed out and after going through the nginx error logs the following line caught my attention:
Missing proper 'which' command. Make sure it is installed before using RVM!
and also lines like (which was causing the Snorby Worker process from starting):
stderr: sh: env: command not found
Now - I was pretty sure this was already installed - and to confirm we can check with:
rpm -qa which
which-2.20-7.el7.x86_64
So my next though was that it was likely an environment variable (specifically PATH) issue. It turns out nginx (by default) 'nukes' all environment values and in order to preserve them (specifically 'PATH' in this case) we need to add the following to the 'http' stanza in nginx.conf:
env PATH;
and reload:
sudo service nginx reload
Although the above may seem pretty trivial it actually look me a while to get it all up and running with a lot of trial and error - I really hope the developers will streamline the process and fix a number of outstanding bugs as I can imagine a lot of people would be put of my the amount of work involved getting it running!
Performing a hard reset
If you want to start from scratch or simply just purge the snorby database you can do so by doing:
cd /opt/snorby/application
and issuing:
bundle exec rake snorby:hard_reset
SELinux Rules
Will be available soon...
Sources
https://www.phusionpassenger.com/library/install/nginx/install/oss/trusty/
https://stackoverflow.com/questions/32071190/cant-deploy-passenger-with-nginx
Installing and setting up Postgres with Centos 7
The following will outline how to quickly (and securely) get postgres and up and running.
We'll firstly need to add the epel repositories with:
sudo yum install epel-release
and then install postgres with:
sudo yum install postgresql postgresql-server postgresql-contrib
Initialise the database with:
postgresql-setup initdb
Let's firstly ensure that only local applications can access the server:
sudo vi /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all postgres peer
local all all peer
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
The first line permit permit unix domain socket access for the postgres user - while the second and third line provide network access to localhost IPv4 and IPv6.
Enable and start the service:
sudo systemctl enable postgresql
sudo service start postgresql
Unlike MySQL / MariaDB - postgres by default uses 'peer' accounts - that are simply local user accounts that have been mapped with a corresponding postgres account.
By default postgres create a user called 'postgres' which we can use to access it's CLI:
sudo su - postgres
psql
quit the console using '\q'
We will then create a new user for our application (from the shell) with:
createuser --interactive
(ensuring the user is NOT a super admin, however can create databases.)
and set a password for the user:
psql
\password <username>
\q
We can this test this from shell with:
psql -U <username> -h localhost -W --dbname=postgres
Note: We haven't created a database for the user yet - so we use the 'postgres' database above.
Once logged in we can create our down database with:
CREATE DATABASE <database-name>;
We'll firstly need to add the epel repositories with:
sudo yum install epel-release
and then install postgres with:
sudo yum install postgresql postgresql-server postgresql-contrib
Initialise the database with:
postgresql-setup initdb
Let's firstly ensure that only local applications can access the server:
sudo vi /var/lib/pgsql/data/pg_hba.conf
# TYPE DATABASE USER ADDRESS METHOD
local all postgres peer
local all all peer
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
The first line permit permit unix domain socket access for the postgres user - while the second and third line provide network access to localhost IPv4 and IPv6.
Enable and start the service:
sudo systemctl enable postgresql
sudo service start postgresql
Unlike MySQL / MariaDB - postgres by default uses 'peer' accounts - that are simply local user accounts that have been mapped with a corresponding postgres account.
By default postgres create a user called 'postgres' which we can use to access it's CLI:
sudo su - postgres
psql
quit the console using '\q'
We will then create a new user for our application (from the shell) with:
createuser --interactive
(ensuring the user is NOT a super admin, however can create databases.)
and set a password for the user:
psql
\password <username>
\q
We can this test this from shell with:
psql -U <username> -h localhost -W --dbname=postgres
Note: We haven't created a database for the user yet - so we use the 'postgres' database above.
Once logged in we can create our down database with:
CREATE DATABASE <database-name>;
Thursday, 22 June 2017
Setting up snort, DAQ and PF_RING on CentOS 7
Let's firstly download and build the PF_RING kernel module:
yum -y install kernel-devel kernel-headers libtool automake autoconf flex bison gcc
cd /tmp
wget http://packages.ntop.org/rpm7/x64/PF_RING/pfring-6.7.0-1286.x86_64.rpm
Download and install DAQ from the snort site:
cd /tmp
wget https://www.snort.org/downloads/snort/daq-2.0.6-1.f21.x86_64.rpm
rpm -i daq*
and then build the DAQ module for PF_RING:
git clone https://github.com/ntop/PF_RING.git
cd PF_RING/userland/snort/pfring-daq-module
autoreconf -ivf
./configure
make & make install
This should copy the library to: /usr/local/lib/daq/daq_pfring.so
Finally download and configure snort:
cd /tmp
wget https://www.snort.org/downloads/snort/snort-openappid-2.9.9.0-1.centos7.x86_64.rpm
yum install snort-openappid-2.9.9.0-1.centos7.x86_64.rpm
We can then run snort in either IDS mode:
snort --daq-dir=/usr/local/lib/daq --daq pfring --daq-mode passive -i ethX -v -e -c /etc/snort/snort.conf
or IPS mode:
snort --daq-dir=/usr/local/lib/daq --daq pfring -i ethX:ethY -e -Q -c /etc/snort/snort.conf
We can also update the SNORT definitions with:
cd /tmp
wget https://www.snort.org/downloads/registered/snortrules-snapshot-2990.tar.gz
tar zxvf snortrules*
cd snortrules*
cd etc
cp * /etc/snort
cd ../rules
cp * /etc/snort/rules
cd ..
cp -R preproc_rules /etc/snort
cp -R so_rules /etc/snort
After attempting to start snort again I received a number of complaints about bad folder paths - so I ended up creating several sym links to get it working correctly:
ln -s /usr/lib64/snort-2.9.9.0_dynamicengine/ /usr/local/lib/snort_dynamicengine
ln -s /usr/lib64/snort-2.9.9.0_dynamicpreprocessor/ /usr/local/lib/snort_dynamicpreprocessor
ln -s /etc/snort/so_rules/precompiled/Centos-5-4/x86-64/2.9.9.0/ /usr/local/lib/snort_dynamicrules
And also modifying some of the directory variables in snort.config like so_rules and rules.
And finally creating a few files:
touch /etc/snort/rules/white_list.rules
touch /etc/snort/rules/black_list.rules
systemd Service
Finally lets create our own service for snort:
sudo vi /lib/systemd/system/snort.service
and add the following (presuming you want IDS mode enabled):
[Unit]
Description=Snort NIDS Daemon
After=syslog.target network.target
[Service]
Type=simple
ExecStart=/usr/sbin/snort --daq-dir=/usr/local/lib/daq --daq pfring --daq-mode passive -i ethX -e -c /etc/snort/snort.conf
[Install]
WantedBy=multi-user.target
Then enable and start it with:
sudo systemctl enable snort
sudo service snort start
and check the status with:
sudo service snort status
yum -y install kernel-devel kernel-headers libtool automake autoconf flex bison gcc
cd /tmp
wget http://packages.ntop.org/rpm7/x64/PF_RING/pfring-6.7.0-1286.x86_64.rpm
Download and install DAQ from the snort site:
cd /tmp
wget https://www.snort.org/downloads/snort/daq-2.0.6-1.f21.x86_64.rpm
rpm -i daq*
and then build the DAQ module for PF_RING:
git clone https://github.com/ntop/PF_RING.git
cd PF_RING/userland/snort/pfring-daq-module
autoreconf -ivf
./configure
make & make install
This should copy the library to: /usr/local/lib/daq/daq_pfring.so
Finally download and configure snort:
cd /tmp
wget https://www.snort.org/downloads/snort/snort-openappid-2.9.9.0-1.centos7.x86_64.rpm
yum install snort-openappid-2.9.9.0-1.centos7.x86_64.rpm
We can then run snort in either IDS mode:
snort --daq-dir=/usr/local/lib/daq --daq pfring --daq-mode passive -i ethX -v -e -c /etc/snort/snort.conf
or IPS mode:
snort --daq-dir=/usr/local/lib/daq --daq pfring -i ethX:ethY -e -Q -c /etc/snort/snort.conf
We can also update the SNORT definitions with:
cd /tmp
wget https://www.snort.org/downloads/registered/snortrules-snapshot-2990.tar.gz
tar zxvf snortrules*
cd snortrules*
cd etc
cp * /etc/snort
cd ../rules
cp * /etc/snort/rules
cd ..
cp -R preproc_rules /etc/snort
cp -R so_rules /etc/snort
After attempting to start snort again I received a number of complaints about bad folder paths - so I ended up creating several sym links to get it working correctly:
ln -s /usr/lib64/snort-2.9.9.0_dynamicengine/ /usr/local/lib/snort_dynamicengine
ln -s /usr/lib64/snort-2.9.9.0_dynamicpreprocessor/ /usr/local/lib/snort_dynamicpreprocessor
ln -s /etc/snort/so_rules/precompiled/Centos-5-4/x86-64/2.9.9.0/ /usr/local/lib/snort_dynamicrules
And also modifying some of the directory variables in snort.config like so_rules and rules.
And finally creating a few files:
touch /etc/snort/rules/white_list.rules
touch /etc/snort/rules/black_list.rules
systemd Service
Finally lets create our own service for snort:
sudo vi /lib/systemd/system/snort.service
and add the following (presuming you want IDS mode enabled):
[Unit]
Description=Snort NIDS Daemon
After=syslog.target network.target
[Service]
Type=simple
ExecStart=/usr/sbin/snort --daq-dir=/usr/local/lib/daq --daq pfring --daq-mode passive -i ethX -e -c /etc/snort/snort.conf
[Install]
WantedBy=multi-user.target
Then enable and start it with:
sudo systemctl enable snort
sudo service snort start
and check the status with:
sudo service snort status
Tuesday, 20 June 2017
Identifying a network bottleneck / packet loss on CentOS
We can check for for packet loss at the hardware level using ethtool:
ethtool -S eth0
Typically (although can vary) you are looking for a counter such as:
rx_*_errors
When the packets are received from the NIC they are then placed into send and receive queues - we should ensure that none of them are currently full - we can view these with:
ss -nmp
and finally check for protocol errors with netstat:
netstat -s
Sources
https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf
ethtool -S eth0
Typically (although can vary) you are looking for a counter such as:
rx_*_errors
When the packets are received from the NIC they are then placed into send and receive queues - we should ensure that none of them are currently full - we can view these with:
ss -nmp
and finally check for protocol errors with netstat:
netstat -s
Sources
https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf
Packet Capture: Fine tuning Linux for 10gb NIC's / busy networks
Below I have outlined some of the more important tweaks that can be applied on a Linux system in order to optimise performance with 10gb NICs and busy networks where there is a high volume of throughput.
As a fornote when capturing packets with 10gb cards you should also ensure that you have a sufficient CPU and available IOPS - I'd recommend an SSD for best performance.
Hardware
While libpcap will work with pretty much any NIC if you want to use PF_RING (which is strongly recommended due to performance benefits) you will need an Intel 82599-based NIC and ensure the Linux kernel is above 2.6.31 (which should be pretty much every mainstream distribution these days.)
There are also other specialist NIC's that are supported and can also perform hardware packet filtering - however for the purposes of this tutorial we will be sticking with an Intel based chip.
Tuning
Firstly we should run a network performance tool - such as iperf to benchmark throughput:
sudo yum install iperf
and on the server side issue:
iperf -s
and the client side:
iperf -c server.ip.address -w64k -t60
You'll also want to monitor the cpu during this period e.g.:
mpstat 5
This will also provide us with something to contrast performance with when we have finished performing the tweaks.
RX Descriptor Sizes
The descriptors do not hold any packet data - rather contain information about the whereabouts of the data is in memory. These values are often not set at the maximum - in order to verify your current descriptor levels you can run:
ethtool -g eth0
Example output:
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 2048
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 128
TX: 512
We can then increase the descriptors as follows:
ethtool -G eth0 rx 4096 tx 4096
Jumbo Frames
One of the obvious considerations is enabling Jumbo frames on the interface - although this is presuming that the application(s) support them! We can enable this on a per interface level with:
vi /etc/sysconfig/network-scripts/eth0
and append / change:
MTU=9000
sudo service network restart
RX and TX Checksum Offload
Each time a packet is received or sent the CPU calculates a checksum - enabling this feature forces the NIC to calculate this instead - hence freeing up CPU.
This can be enabled on a per interface level with:
ethtool --offload eth0 tx on rx on
* Note: Saving CPU with TX checksum offload is dependant on how large the frame packet sizes are - larger packets equate to a greater saving.
Kernel Tweaking
Removing TCP time-stamping is another way to reduce CPU load - however you (obviously) lose the round trip time of the segment:
sysctl -w net.ipv4.tcp_ timestamps=0
And increasing the syn and network driver backlog with:
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 2500
and tcp read, write limits:
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
and socket buffer space limits:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
and backlogged sockets with (default is 100):
net.core.somaxconn = 1024
Sources
https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf
As a fornote when capturing packets with 10gb cards you should also ensure that you have a sufficient CPU and available IOPS - I'd recommend an SSD for best performance.
Hardware
While libpcap will work with pretty much any NIC if you want to use PF_RING (which is strongly recommended due to performance benefits) you will need an Intel 82599-based NIC and ensure the Linux kernel is above 2.6.31 (which should be pretty much every mainstream distribution these days.)
There are also other specialist NIC's that are supported and can also perform hardware packet filtering - however for the purposes of this tutorial we will be sticking with an Intel based chip.
Tuning
Firstly we should run a network performance tool - such as iperf to benchmark throughput:
sudo yum install iperf
and on the server side issue:
iperf -s
and the client side:
iperf -c server.ip.address -w64k -t60
You'll also want to monitor the cpu during this period e.g.:
mpstat 5
This will also provide us with something to contrast performance with when we have finished performing the tweaks.
RX Descriptor Sizes
The descriptors do not hold any packet data - rather contain information about the whereabouts of the data is in memory. These values are often not set at the maximum - in order to verify your current descriptor levels you can run:
ethtool -g eth0
Example output:
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 2048
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 128
TX: 512
We can then increase the descriptors as follows:
ethtool -G eth0 rx 4096 tx 4096
Jumbo Frames
One of the obvious considerations is enabling Jumbo frames on the interface - although this is presuming that the application(s) support them! We can enable this on a per interface level with:
vi /etc/sysconfig/network-scripts/eth0
and append / change:
MTU=9000
sudo service network restart
RX and TX Checksum Offload
Each time a packet is received or sent the CPU calculates a checksum - enabling this feature forces the NIC to calculate this instead - hence freeing up CPU.
This can be enabled on a per interface level with:
ethtool --offload eth0 tx on rx on
* Note: Saving CPU with TX checksum offload is dependant on how large the frame packet sizes are - larger packets equate to a greater saving.
Kernel Tweaking
Removing TCP time-stamping is another way to reduce CPU load - however you (obviously) lose the round trip time of the segment:
sysctl -w net.ipv4.tcp_ timestamps=0
And increasing the syn and network driver backlog with:
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 2500
and tcp read, write limits:
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
and socket buffer space limits:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
and backlogged sockets with (default is 100):
net.core.somaxconn = 1024
Sources
https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf
Disabling the PC speaker / beep sound with CentOS 7 / Core
One of the not so common issues I encountered with the Core edition of CentOS was the pc speaker - - the vast majority of the time sound means little when the machine has been virtualised.
I quickly realised that the relevant alsa tools are (expectantly) not provided and need to be installed.
Doing a quick search for alsamixer with yum brings up the 'alsa-utils' package:
sudo yum provides alsamixer
sudo yum -y install alsa-utils
We can then use the alsamixer utility to adjust the speaker volume:
alsamixer
or if we prefer to do it via command line we can issue:
amixer set <sound-card-name> Mute | Unmute
(Use the 'axmixer' command on it's own to view available sound devices.)
To ensure the changes persist across reboot we should issue:
alsactl store
Unfortunately the system beeps (which will quickly drive you mad when working from the console) persisted - this can be done either by using the setterm utility:
setterm -blength 0
or we can simply disable the speaker module all together with:
rmmod -v pcspkr
and to ensure the changes persist we should add this module to the blacklist:
echo 'blacklist pcspkr' >> /etc/modprobe.d/blacklist
I realise there are alternative methods to do this - however this was the easiest and most efficient for myself.
I quickly realised that the relevant alsa tools are (expectantly) not provided and need to be installed.
Doing a quick search for alsamixer with yum brings up the 'alsa-utils' package:
sudo yum provides alsamixer
sudo yum -y install alsa-utils
We can then use the alsamixer utility to adjust the speaker volume:
alsamixer
or if we prefer to do it via command line we can issue:
amixer set <sound-card-name> Mute | Unmute
(Use the 'axmixer' command on it's own to view available sound devices.)
To ensure the changes persist across reboot we should issue:
alsactl store
Unfortunately the system beeps (which will quickly drive you mad when working from the console) persisted - this can be done either by using the setterm utility:
setterm -blength 0
or we can simply disable the speaker module all together with:
rmmod -v pcspkr
and to ensure the changes persist we should add this module to the blacklist:
echo 'blacklist pcspkr' >> /etc/modprobe.d/blacklist
I realise there are alternative methods to do this - however this was the easiest and most efficient for myself.
Wednesday, 7 June 2017
Building VMWare Kernel Modules Fails: Fail to find version.h
After upgrading the kernel the other day I attempted to recompile the VMWare kernel module although it ended up failing. After reviewing the logs I noticed that it was complaining about not finding 'version.h' - although the kernel-header package was installed:
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Setting header path for 4.11.3-200.fc25.x86_64 to "/lib/modules/4.11.3-200.fc25.x86_64/build/include".
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Validating path "/lib/modules/4.11.3-200.fc25.x86_64/build/include" for kernel release "4.11.3-200.fc25.x86_64".
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Failed to find /lib/modules/4.11.3-200.fc25.x86_64/build/include/linux/version.h
Turns out that this file is actually kept in: /usr/include/linux/
So in order to get the VMWare module to compile I ended up copying it to the current kernel's header directory:
sudo cp /usr/include/linux/version.h /lib/modules/`uname -r`/build/include/linux/
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Setting header path for 4.11.3-200.fc25.x86_64 to "/lib/modules/4.11.3-200.fc25.x86_64/build/include".
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Validating path "/lib/modules/4.11.3-200.fc25.x86_64/build/include" for kernel release "4.11.3-200.fc25.x86_64".
2017-06-07T10:18:44.574+01:00| vthread-4| I125: Failed to find /lib/modules/4.11.3-200.fc25.x86_64/build/include/linux/version.h
Turns out that this file is actually kept in: /usr/include/linux/
So in order to get the VMWare module to compile I ended up copying it to the current kernel's header directory:
sudo cp /usr/include/linux/version.h /lib/modules/`uname -r`/build/include/linux/
Monday, 5 June 2017
QoS: Traffic Policing vs Traffic Shaping
This article will focus on understanding how QoS techniques such as traffic policing and shaping are performed (and contrasted) and how values such as burst rates can be calculated.
One of the fundamental differences between the two is that policing involves dropping traffic when the bucket is full - while shaping put excess traffic into a queue for submission and is gradually released resulting in a smoother flow of traffic.
This can be illustrated below (credit to Cisco for this diagram):
It is also worth noting that traffic shaping only works on outbound traffic (traffic leaving the device) - while policing will work on both ingress (traffic coming to the device) and egress traffic.
Key Terms / Formulas
I'll firstly go over some of the key terms:
Committed Information Rate (CIR) = 10000000 (10Mbps)
Burst Commit Bucket (Bc) = CIR * 0.125s = 1875000
Time Interval (Tc) = Bc / CIR = 0.125s
In this case the bucket will be emptied after 0.125s - which for some purposes might be perfectly fine - however if you are supporting a large file server you'd want the TC to be much higher - the burst rate and Tc will depend greatly on your network type and quite often you will have to tweak it for best performance. The formular for burst rate is above is the default one provided by Cisco in their documentation.
Traffic Policing Example (Egress)
Presuming the port speed is 100mb - the following configuration would limit egress traffic to 20mb:
int fa0/1-48
srr-queue bandwidth limit 80
Traffic Policing Example (Ingress)
We have to create a service policy for ingress policing:
mls qos
class-map match-all rate-limit
description Bandwidth Control
match ip dscp default
policy-map GeneralTraffic
class rate-limit
police 10000000 192000 exceed-action drop
int range fa01-48
service-policy input GeneralTraffic
Traffic Shaping Example (Egress)
Sources
http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html
One of the fundamental differences between the two is that policing involves dropping traffic when the bucket is full - while shaping put excess traffic into a queue for submission and is gradually released resulting in a smoother flow of traffic.
This can be illustrated below (credit to Cisco for this diagram):
It is also worth noting that traffic shaping only works on outbound traffic (traffic leaving the device) - while policing will work on both ingress (traffic coming to the device) and egress traffic.
Key Terms / Formulas
I'll firstly go over some of the key terms:
Committed Information Rate (CIR) = 10000000 (10Mbps)
Burst Commit Bucket (Bc) = CIR * 0.125s = 1875000
Time Interval (Tc) = Bc / CIR = 0.125s
In this case the bucket will be emptied after 0.125s - which for some purposes might be perfectly fine - however if you are supporting a large file server you'd want the TC to be much higher - the burst rate and Tc will depend greatly on your network type and quite often you will have to tweak it for best performance. The formular for burst rate is above is the default one provided by Cisco in their documentation.
Traffic Policing Example (Egress)
Presuming the port speed is 100mb - the following configuration would limit egress traffic to 20mb:
int fa0/1-48
srr-queue bandwidth limit 80
Traffic Policing Example (Ingress)
We have to create a service policy for ingress policing:
mls qos
class-map match-all rate-limit
description Bandwidth Control
match ip dscp default
policy-map GeneralTraffic
class rate-limit
police 10000000 192000 exceed-action drop
int range fa01-48
service-policy input GeneralTraffic
Traffic Shaping Example (Egress)
Sources
http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html
Saturday, 3 June 2017
Setting up / configuring an NTP client on CentOS 7
Firstly ensure that the appropriate timezone is set on the system - for example:
ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
Install the ntp deamon:
yum install ntp -y && systemctl enable ntp
and then start the service with:
ntpdate pool.ntp.org && systemctl start ntpd
ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
Install the ntp deamon:
yum install ntp -y && systemctl enable ntp
and then start the service with:
ntpdate pool.ntp.org && systemctl start ntpd
Replacing your modem / router with a Raspberry Pi and a DSL-320B
One of the major draw backs to Linux (although by no fault of its own) is that there are so few PPPoA drivers available - largely due to the vast majority of them not being open sourced. However there a few Thompson model chip sets that work - but these are ancient.
Instead the easiest option is to bridge the connection with a dedicated PPPoA modem - however there are not many around that support this - the Draytec Vigor 120 and D-Link DSL-320B both do what we need.
I chose the D-Link since you can pick these up fairly cheaply from Amazon.
The first hurdle was working out which settings need to be configured on the DSL-320B - although this might differ slightly from ISP to ISP - below are the settings I used on the device to get it running correctly in bridging mode:
Firstly go to the web-based configuration portal and hit: Setup >> ADSL Setup
Manual ADSL Connection = Bridge Mode
Bridge Mode / Connection Type = 1483 Bridged IP LLC (VC-Mux didn't work for me.)
VPI: 0 (may differ)
VCI: 38 (may differ)
Virtual Circuit = Enable
Service Category = UBR
Sources:
Hostapd on CentOS 6: http://jasonmaur.com/hostapd-centos-6/
Instead the easiest option is to bridge the connection with a dedicated PPPoA modem - however there are not many around that support this - the Draytec Vigor 120 and D-Link DSL-320B both do what we need.
I chose the D-Link since you can pick these up fairly cheaply from Amazon.
The first hurdle was working out which settings need to be configured on the DSL-320B - although this might differ slightly from ISP to ISP - below are the settings I used on the device to get it running correctly in bridging mode:
Firstly go to the web-based configuration portal and hit: Setup >> ADSL Setup
Manual ADSL Connection = Bridge Mode
Bridge Mode / Connection Type = 1483 Bridged IP LLC (VC-Mux didn't work for me.)
VPI: 0 (may differ)
VCI: 38 (may differ)
Virtual Circuit = Enable
Service Category = UBR
Now we want to configure a PPP connection on our Raspberry Pi - I'm using CentOS 7 on mine - however the instructions are pretty generic.
We'll need to firstly install the ppp client etc:
sudo yum -y install rp-pppoe pppd
To get us up and running quick we can run 'pppoe-setup' from the terminal (as root) and we will be prompted for PPPoA username and password among other options.
For the firewall choice we will typically want Option 2 / MASQUERADE - however we will be tweaking the rules in a bit.
All of the ppp configuration is stored under /etc/ppp - there are a few noteworthy files:
chap-secrets: This holds your PPPoA username / password
pap-secrets: Again, holds your PPPoA username / password
firewall-masq: The firewall script (if you chose option 2 during thr setup wizard)
/etc/sysconfig/network-scripts/ifcfg-pppX: The interface configuration script
Before bringing up the connection we will need to modify the firewall rules - since they are not setup very well for a general purpose home router. You will need to add some extra lines into the firewall script (that gets executed when the pppX interface comes up) - this is because existing firewall rules are flushed:
vi /etc/ppp/firewall-masq
# Allow incoming SSH
iptables -t filter -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT
# Allow established connections inbound
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow all traffic on localhost
iptables -A INPUT -i lo -j ACCEPT
Now let's attempt to bring the ppp connection up with:
sudo ifup ppp0
and review the connection with:
ip link ppp0
With any luck you will see it's come up and now assigned an IP.
The next step is to setup a local DHCP server that will serve our LAN:
sudo yum install dhcpd
and configure like follows (replacing where necessary):
# name server(s)
option domain-name-servers 8.8.8.8;
# default lease time
default-lease-time 600;
# max lease time
max-lease-time 7200;
# this DHCP server to be declared valid
authoritative;
# specify network address and subnet mask
subnet 10.11.12.0 netmask 255.255.255.0 {
# specify the range of lease IP address
range dynamic-bootp 10.11.12.10 10.11.12.254;
# specify broadcast address
option broadcast-address 10.11.12.255;
# specify default gateway
option routers 10.11.12.1;
}
Now I also want wireless clients to be able to connect to my network - so I ended up purchasing a high gain USB NIC (RTL8188CUS to be precise).
We will need to install the following packages:
sudo yum -y install hostapd iw bridge-utils openssl-devel libnl-devel
and the build tools:
yum groupinstall "Development Tools"
Unfortunately the CentOS ARM repo's don't currently have a package for hostapd - so we'll need to compile this from source:
cd /tmp
yum install git
git clone git://w1.fi/srv/git/hostap.git
cd ~/hostap/hostapd
git checkout hostap_2_3
cp defconfig .config
We will also need to apply a patch in order to get hostapd working with the RTL8188CUS chipset from: https://github.com/pritambaral/hostapd-rtl871xdrv
cd to the parent directory (the one with the src and hostapd folders) and run the patch e.g.:
patch -Np1 -i /path/to/rtlxdrv.patch
We will now need to tweak the .config file a little - ensure the following are set:
CONFIG_DRIVER_NL80211=y # enable netlink interface
CONFIG_IEEE80211N=y # enable 802.1n
CONFIG_IEEE80211AC=y # enable 802.1ac
CONFIG_ACS=y # enable automatic channel selection
CONFIG_DRIVER_RTW=y # enable RTL8188CUS support
make && make install
and then create a configuration file for it:
mkdir /etc/hostapd
vi /etc/hostapd/hostapd.conf
and add something like the following:
driver=rtl871xdrv
device_name=RTL8192CU
manufacturer=Realtek
interface=wlan0 # the interface used by the AP
hw_mode=g # g simply means 2.4GHz band
channel=10 # the channel to use
ieee80211d=1 # limit the frequencies used to those allowed in the country
country_code=GB # the country code
ieee80211n=1 # 802.11n support
wmm_enabled=1 # QoS support
ssid=somename # the name of the AP
auth_algs=1 # 1=wpa, 2=wep, 3=both
wpa=2 # WPA2 only
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP
wpa_passphrase=somepassword
We will also ensure that the wlan0 interface is excluded from control by the Network Manager:
vim /etc/NetworkManager/NetworkManager.conf
and adding:
[keyfile]
unmanaged-devices=mac:<wlan0-mac-address>
And turn of wifi with:
nmcli radio wifi off
sudo rfkill unblock wlan
and start / test it with:
hostapd /etc/hostapd/hostapd.conf
Everything seemed to look OK initially until I attempted to connected to the AP - and I got the following error message on the console:
wlan0: STA 11:22:33:44:55:66 IEEE 802.11: deauthenticated due to local deauth request
This turns out to be due to lack of entropy so we can install haveged to overcome this - however - again it's not readily available as a package so we'll need to compile it from source:
cd /tmp
wget https://netix.dl.sourceforge.net/project/haveged/haveged-1.9.1.tar.gz
tar zxvf hav*
cd haveged*
./configure
make && make install
and retry with:
hostapd /etc/hostapd/hostapd.conf
Note: When starting hostapd it wipes the IP configuration on wlan0! So we will need to manually configure the interface after it's started and also restart the DHCP service - I wrote the following up (very quickly):
#!/bin/bash
echo Killing of any existing hostapd instances...
pkill hostapd
echo
echo Ensuring wifi is turned off
# make sure wlan interface is offline
nmcli radio wifi off
echo
echo Starting hostapd...
nohup /usr/local/bin/hostapd /etc/hostapd/hostapd.conf >/dev/null 2>&1 &
echo
echo Assiging ip address to wlan interface
# assign ip address to interface
ip addr add 10.55.55.1/24 dev wlan0
echo
echo Restarting the DHCP service
# restart dhcp server
systemctl restart dhcpd
echo
Ideally (when I get the time) I will create it's own service unit for systemctl - but for now the above will do!
Sources:
Hostapd on CentOS 6: http://jasonmaur.com/hostapd-centos-6/
Thursday, 1 June 2017
Where did /etc/default/xyz go?! (CentOS 7 / RHEL)
Many services that now rely on systemd now look for their default options elsewhere than /etc/default/...
Instead commonly an 'environment' file is used instead to specify specific switches that a service should use instead.
You can find the environment file by inspecting the .service file e.g.:
cat /usr/lib/systemd/system/keepalived.service | grep ^EnvironmentFile
Although they should be kept in /etc/sysconfig/xyz typically.
Instead commonly an 'environment' file is used instead to specify specific switches that a service should use instead.
You can find the environment file by inspecting the .service file e.g.:
cat /usr/lib/systemd/system/keepalived.service | grep ^EnvironmentFile
Although they should be kept in /etc/sysconfig/xyz typically.