Friday 31 March 2017

The difference between a Unix Domain Socket and a Named Pipe (fifo)

named pipes (fifo's) allow communication between different processes. They allow you to pipe data directly into a special file that is then read immediately by another process.

As an example we can use the 'mkfifo' command:

mkfifo /tmp/fifo

ls -la /tmp/fifo


Notice the precense of the 'p' bit - this indicates that the file is a named pipe / fifo.

Now let's tail it and pipe some data into it:

tail -f /tmp/fifo &

echo test data > /tmp/fifo

test data

A socket file in Linux (AKA a Unix Domain Socket) allows communication between - for example an server and multiple clients - in some respects it is similar to a named pipe (fifo) - however they provide some advantages such as bi-directional communication, passing file descriptors between processes and support packet and sequenced packet modes.

For example haproxy can utilise sockets:

ls -la /run/haproxy_admin.sock

srw-rw----. 1 root root 0 Mar 28 14:17 /run/haproxy_admin.sock

The 's' bit indicates that the file is a socket.

Socket are typically employed when network communication is not required, rather all communication is performed by processes on the local system.

Tuesday 28 March 2017

Fixed: Unable to negotiate with x.x.x.x port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

Modern versions of OpenSSH  will typically exclude insecure cipher suites by default - however this can cause problems with older devices that are using obsolete cipher suites - in my case an older generation ASA:

ssh admin@

Unable to negotiate with port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

However the long term solution (if possible) is to try and upgrade the firmware if available.

In order to access the system in the meantime we can instruct the OpenSSH client to use a weak cipher suite:

ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 admin@

Tuesday 21 March 2017

Safely purging / pruning old mysql binary logs

While binary logging provides an excellent way to roll back a database to a specific point in time, to recover the database in the event of corruption or for use with replication, by default MySQL binary logs are not purged and over time can accumulate using a huge amount of disk space.

Binary logs (as the name suggests) store data in binary format and are consumed by mysqlbinlog.

In order to prune the logs lets firstly locate the appropraite directory -- in my case this is /var/lib/mysql

cd /var/lib/mysql

-rw-rw----  1 mysql mysql   104871967 Jan 01 00:00 BINLOG.000001
-rw-rw----  1 mysql mysql   104885618 Jan 02 00:00 BINLOG.000002
-rw-rw----  1 mysql mysql   104866713 Jan 03 00:00 BINLOG.000003

We can then either purge by file:

mysql -u user -p

or date:

PURGE BINARY LOGS BEFORE '2017-01-02 00:00:00';

*Warning* Simply deleting the files is extremely dangerous - since the logs do not get removed from the index!

Wednesday 15 March 2017

Configuring IP Source Guard on a 2960X

IP Source Guard is a layer 3 security feature that prevents IP spoofing. It like DAI relies on the DHCP snooping binding table to function.

DAI however works on Layer 2 / ARP and is not able to inspect layer 3 / IP traffic - hence IP Source Guard was introduced.

IP Source Guard is applied on a per-interface level:

int gi0/4
ip verify source

If you have statically assigned IP's you can create a 'static binding' so that IP Source Guard can confirm which IP it is expecting on the specific port.

ip source binding 1111.2222.3333 vlan 100 interface gi0/20

We can show interface that have been configured with IP Source Guard with:

do show ip verify source

We can also view the IP Source Guard binding table at any time with:

show ip source binding

Setting up DHCP snooping a long side a DHCP relay agent (Option 82 / giaddr)

DHCP snooping allows the switch to maintain it's own (binding) table that links a MAC address to an IP, switchport, vlan and lease time a long with restricting specific ports that send DHCP server messages.

This is performed to attempt to mitigate any rouge DHCP servers on the network - however a lot of other services such as DAI also depend on DHCP snooping.

We should firstly enable DHCP snooping with:

ip dhcp snooping

DHCP snooping is configured on a per-VLAN basis - however we must firstly define which VLAN we want to enable it on:

ip dhcp snooping vlan 100

We can then review the VLAN's and trusted ports with:

do show ip dhcp snooping

and assign the port connected to the DHCP server as a 'trusted' port:

int gi0/15
desc DHCP_Server
ip dhcp snooping trusted

We can now plug the DHCP server into gi0/15 (vlan 100) and a client machine gi0/2 (vlan 100) - on the client tail the syslog / messages log:

tail -f /var/log/messages &

remove any existing leases with:

sudo dhclient -r

and attempt to lease an address with:

sudo dhclient

Once the client machine has obtained an IP address we can then review the binding database with:

show ip dhcp snooping binding

Let's now move the DHCP server patch cable into gi0/16 (which is not currently trusted.)

We'll also enable dhcp snooping packet debugging so we can identify if the switch is dropping the DHCP packets:

debug ip dhcp snooping packet

Again - release and renew the ip on the client machine and tail /var/log/messages:

and you should see a series of DHCPDISCOVER messages - however nothing being offered from the DHCP server - these packets should have been blocked by DHCP snooping - we should see the packets being dropped in the console e.g.:

*Mar  1 01:11:11.722: DHCP_SNOOPING_SW: bridge packet output port set is null, packet is dropped.

Note: If you are using a DHCP relay agent (ip helper-address) - you should be aware that 'Option 82' / the 'giaddr' (gateway) address is modified when relaying DHCP requests and as such will be seen as inconsistent by the DHCP snooping service - so in order to work-around this we can disable Option 82 for the DHCP relay agent with:

no ip dhcp snooping information option

Tuesday 14 March 2017

Converting pk7 to pem encoded certificate / key pair to PKCS12 format

Often if a key pair is pem encoded and you wish to transport this securely for use on a Windows server you will need to convert the pem encoded files to the PKCS12 format - which is effectively an encrypted container for the key pair.

We can perform the conversion for this with openssl - by firstly converting the p7b file (which is simply a container with in most cases the main certificate plus any other certificates in the chain):

openssl pkcs7 -print_certs -in certifcates.p7b -out certifcates.cer

and then from PEM to pkcs12 format:

openssl pkcs12 -export -inkey certifcate.key -in certifcates.cer -out certificates.p12

Generating a SAN (Subject Alternative Name) certificate with OpenSSL

Firstly create a new file (e.g. /tmp/csr_yourdomain.conf) as follows (replacing the relevant information.)

default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
O=Your Company
CN = *

[ req_ext ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = *
DNS.2 = *

The above information is typically taken when generating a single FQDN - however the inclusion of the 'alt_names' section is where SAN (subject alternative domains) are listed.

We can easily add additional domains by simply incrementing the 'DNS.' prefix e.g. DNS.3, DNS.4 etc.

Finally generate the new certificate - ensuring the configuration is pointed at the conf file you have just created:

openssl req -new -sha256 -nodes -out \* -newkey rsa:2048 -keyout \*.yourdomain.key -config /tmp/csr_yourdomain.conf

Friday 10 March 2017

Setting up and using gpg (GnuPG) with CentOS 7

PGP was first introduced by Phil Zimmermann & Associates and was a proprietary piece of software that allowed users to encrypt their files - although later on one of the original developers began work on OpenPGP (an open-source alternative.)

These days GPG (GnuPG) is also extremely popular and is based open OpenPGP standards and is widely available across most popular operating systems.

For this tutorial I will focus on CentOS 7 - however much of the processes are exactly the same on most Linux distributions.

GPG makes use of PKI - so can be used to encrypt data, but also allows us to sign data so that it can be independently verified.

Let's go ahead and install the relevant packages:

sudo yum install gnupg2

There is also the 'gnupg1' package available that is *internally* quite different from gnupg2 and is typically more suited for server / embedded environments as it requires less dependencies.

gnupg2 keeps your public keys and secret keys in a 'keyring.' A keyring is simply a container that stores certificates, keys etc. securely.

We'll proceed by generating the private / public key pair:

gpg2 --gen-key

For this demonstration we'll choose RSA for the algorithm, a key size of 4096 bits and an expiry of 1 year.

When you first generate your keys you will notice two files are created:

~/.gnupg/pubring.gpg (Stores the public keys)

~/.gnupg/secring.gpg (Stores the 'secret' / private keys)

We can view our public keys with:

gpg2 --list-keys

and our public ones with:

gpg2 --list-secret-keys

In order to delete key pairs we should firstly delete the secret key with:

gpg2 --delete-secret-key <key-id>

and then remove the associated public key:

gpg2 --delete-keys <key-id>

Sending / Receiving files with GPG 

In order to receive a file securely from another party we must firstly provide them with it (preferably via offline media such as a USB stick) export our public key with:

gpg2 --armor --export -a <key-id> > public.key

The 'armor' option instructs GPG to encode the output with ASCII ensuring that it can be emailed.

The other party will then encrypt data with your public key - however they must firstly import your public key:

gpg2 --import public.key

and then encrypt the data with:

gpg2 --encrypt --armor --recipient "Joe Blogs" secret.txt

This will generator a file called secret.txt.asc (you might want to rename this to something more meaningful like secret.txt.gpg)  - this can then be transmitted to yourself - who can then decrypt the file with the following:

gpg2 --decrypt secret.txt.asc > secret_decrypted.txt

You should then be prompted for the password associated with the private / secret key.

Windows Task Manager, RAM and the Windows Metafile

I have sometimes noticed slight discrepancies between the available RAM presented in Windows Task Manager and the sum of private and shared bytes processes use - although on one particular server this discrepancy was a whopping 4GB!

Doubting the accuracy of what the Windows Task Manager was reporting I checked a few other tools, including Process Hacker - which provides me with a more granular look at the system's RAM consumption, private, shared memory etc. However the figures still did not add up - so I ended up coming across a tool called RAMMap by sysinternals.

This tool provides you with everything included by the above, but in addition detailed paging and kernel consumption information.

However there was an item called 'Metafile' - This metafile is used part of the Windows System cache and holds NTFS metadata - which is used to provide increased performance when accessing files.

For each file that is accessed on the disk there is a corresponding block of 1kb (or if a file has e.g. 3 attributes - this would be 3kb). Now on a system such as a fileserver that potentially has millions of files this could soon add up and could easily be 10GB plus.

You do have the ability to empty the metafile - however doing this could have adverse (but temporary) effects on the server - if they are cleared out of RAM - there will likely be a surge of activity reading the metafile again from the disk! If you really need to perform this you can click on the 'Empty' drop down on RAMMap and select 'System working-set.'

The recommended approach is to install the 'Dynamic Cache Service' - which can be downloaded from here.

Extract the archive and copy 'DynCache.exe' to '%SystemRoot%\System32'

and then cd to the Systemc32 directory and and create a service for it by running the following:

sc create DynCache binpath= %SystemRoot%\System32\DynCache.exe start= auto type= own DisplayName= "Dynamic Cache Service"

Now open regedit and go to the following key:


and ensure the 'MaxSystemCacheMBytes' are set to (at least) half of the amount of physical RAM you have - for example if you had 8GB of RAM you could set it to: 4096 (mb)

and the minimum cache size ('MinSystemCacheMBytes') to 512 (mb).

Finally start the service with:

sc start DynCache

Note: You can increase / decrease the values above without having to restart the DynCache service!


Fixing High Memory Usage by Metafile on Windows Server 2008 R2:

Wednesday 8 March 2017

Kicking local / remote users out of a linux system

Get a list of current tty (console users) / pts (remote users e.g. ssh etc.) with:

who -a

LOGIN      tty1         2017-02-23 13:14              2432 id=tty1
           system boot  2017-02-23 13:13
           run-level 5  2017-02-23 13:14
youruser ? :0           2017-02-23 13:14   ?          2733 (console)
youruser + pts/0        2017-03-08 15:15   .         32548 (

and 5th column represents the session PID - we can then kill the user's SSH session with:

kill -9 32548

Manually performing DNSSEC validation with dig

Let's firstly obtain the DNSKEY for the root namespace '.':

dig DNSKEY . | grep -Ev '^($|;)' > root.keys

. 172800 IN DNSKEY 256 3 8 AwEAAYvgWbYkpeGgdPKaKTJU3Us4YSTRgy7+dzvfArIhi2tKoZ/WR1Df w883SOU6Uw7tpVRkLarN0oIMK/xbOBD1DcXnyfElBwKsz4sVVWmfyr/x +igD/UjrcJ5zEBUrUmVtHyjar7ccaVc1/3ntkhZjI1hcungAlOhPhHlk MeX+5Azx6GdX//An5OgrdyH3o/JmOPMDX1mt806JI/hf0EwAp1pBwo5e 8SrSuR1tD3sgNjr6IzCdrKSgqi92z49zcdis3EaY199WFW60DCS7ydu+ +T5Xa+GyOw1quagwf/JUC/mEpeBQYWrnpkBbpDB3sy4+P2i8iCvavehb RyVm9U0MlIc=
. 172800 IN DNSKEY 257 3 8 AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQbSEW0O8gcCjF FVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh/RStIoO8g0NfnfL2MTJRkxoX bfDaUeVPQuYEhg37NZWAJQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaD X6RS6CXpoY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3LQpz W5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGOYl7OyQdXfZ57relS Qageu+ipAdTTJ25AsRTAoub8ONGcLmqrAmRLKBP1dfwhYB4N7knNnulq QxA+Uk1ihz0=

From the output we can see the tag number '172800', the security entry point - this is either 0, 256 (indicates ZSK / zone-signing key) or 257 (indicates it's a KSK / key signing key), the number '3' which defines the protocol field and '8' which defines the security algorithm used (RSA/SHA-256 in this case.)

The important key here is the KSK that ultimately signs the DNSSEC RRSet for the root zone.

Note: In a lab environment obtaining the DNSKEYS via dig is alright - but in production you would want to obtain these keys via a more secure method and possibly even have them built into the application itself.

Now there are two ways of verifying a record with DNSSEC - top-to-bottom (where the root domain is validated first and lastly the actual domain in question) or bottom-to-top (the domain in question is validated firstly and lastly the root domain is validated.)

dig +sigchase +trusted-key=./root.keys A | cat -n

Note: Make sure your DNS server supports DNSSEC validation otherwise you'll get something like:

'Launch a query to find a RRset of type RRSIG for zone:
RRSIG is missing for continue validation: FAILED'

Google public DNS servers now support DNSSEC valiation - so we can perform the query through them instead:

dig @ +sigchase +trusted-key=./root.keys A

If successful you should see a message at the bottom the output like:

';; Ok this DNSKEY is a Trusted Key, DNSSEC validation is ok: SUCCESS'

DNSSEC: A quick introduction

DNS is a pretty old protocol - although used extensively even today. However it's major let down (from a security perspective) is that it transmits data in plain text and the receiver does not have means to validate the content - this makes it extremely easy to manipulate requests with techniques like DNS Poisoning.

DNSSEC was introduced to overcome this problem - and provides a way of validating the responses from a nameserver with help from PKI.

DNSSEC introduces a few new record types (plus a few more not mentioned here):

RRSIG: This provides a signature of the RRSet record

DNSKEY: This provides the public key portion of the ZSK

DS: This provides a summary of a child zone's DNSKEY

With DNNSEC all specific types of records for example: A, MX, SOA etc. are bundled into RRSet's (Resource Record Set) which have a corrosponding RRSIG record - which is simply a digital signature of the RRSet - this is created by the private portion of a zone-signing key pair (ZSK). In order to verify records there the public key from the ZSK is used within a DNSKEY record - the response from the DNS server is checked against the RRSIG with the help of the DNSKEY.

We also have to provide a way of validating that the zones DNSKEY is valid / not forged - this is where the DS (Delegation of Signing) key comes into play - it holds a summary of the DNSKEY in the parent zone - that is then protected by the parent zones DNSKEY.

So each parent domain you are requesting is also validated - for example if you are attempting to request the A record for '' the following RR will be validated:

As seen above - even the TLD nameservers are validated - this is part of the 'chain of trust' (we need to ensure that none of the parent zones have been compromised either.) - however the '.' / root namespace does not (obviously) have any parent - so instead it is treated as trusted - this is because of a procedure called the 'root signing ceremony' - more can be read about this here:

We can check the DS and RRSIG records using dig:

dig +dnssec

com. 172800 IN NS
com. 172800 IN NS
com. 86400 IN DS 30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com. 86400 IN RRSIG DS 8 1 86400 20170321050000 20170308040000 61045 . CfqhL197dQg2mf0u+ak7qP/ZjEwzyn0TIjDQ3CJJpLdGmfrbuHuNynNf vnCMr8ca17ZKzBTijE0faAH41V1e64C20/kxuYfYjDlsk7ZFQXQhpE6X 5yHmbApyB1UbtSNYTSPzE9bhbcWxAlRcMSZ+R+ABINX4xRUvBUFAX8MH 20YRHBJMZlRDLznUTDGBSyrjWNzVINYF48G6a/qaN1bTzrldVFYeLCTV rAsYQ4glaUhwi+amtjnubYNNAfIyIqWu2MOhfaOCGfz6s0weD1OPKomx i0X4CV1+Nu+qDB8Ud02H0Z8OJWFqHDkU75hNwn47/09I/qQ9GvHCZWfL TZOetA==

The root nameserver does not know the A record for '' but however it can provide us with the nameservers for the .com tld.

dig +dnssec

;; AUTHORITY SECTION: 172800 IN NS 172800 IN NS 172800 IN NS 86400 IN DS 64326 8 2 02E7FEF4C3BBB0A0FA52F0F8E5774C44B243739D1AB7B3B426A417C3 88F45ACF 86400 IN RRSIG DS 8 2 86400 20170315043628 20170308042628 31697 com. xVFjovZdqjyYSf7N0yglKwSrfBKCDmQVQiMw6U94q7+3UlbyzzB95QJI TmhHJ8gvrnGfSSoj7rsRmW1xiGMWBrRCL6VQB5Go9UuEvOHnz6FFMv04 q8FNtjokD/k+nEJ70h5LHc295dxL+xQ3aKF/wWu9ZxpjwLi0MA3d20OM eoU=

;; ADDITIONAL SECTION: 172800 IN AAAA 2001:500:7967::2:33 172800 IN A 172800 IN A 172800 IN AAAA 2620:74:19::33 172800 IN AAAA 2001:502:cbe4::33 172800 IN A

Again - no luck - although this time we have the details of some nameservers and their corresponding addresses!

We'll now query the verisign nameserver:

dig +dnssec

;; ANSWER SECTION: 86400 IN A 86400 IN RRSIG A 8 3 86400 20170321010646 20170307010646 1547 aqPIS0bEw4pLOcqYqWjNM8VMNfE6UQq6n/FCW40u/1x/CMwYtkfA47ZG ZtFxXQzD0YgEZLXWHETWrl22dIZ2MetXDa4N036XxJf5ZNRe9S3Bcgp/ 2QJAze0mfCFA/XWAEfxgZAN7y6mSuOahFHEOD4gNL5Y+v5SB4OwP70qs S4A=

;; AUTHORITY SECTION: 86400 IN NS 86400 IN NS 86400 IN NS 86400 IN RRSIG NS 8 2 86400 20170321010646 20170307010646 1547 BJJIQ5HCrjtilACZdBTqheNIJr6jOJS/RpFYGrRW/qtsNCPLMg1J8sE2 0xyKflNTB6pkUE3QJxv0yQ8Qn7Q83GMmlRFmdlmvkE7dSGUxhOC20mjt yOY9kW5ozYhhquGJzTbcxLgvG1Ss2jGokoJxf+eoaw0NUSR5M44Q8gda 2KY=

Now is an authoritative name server and knows the address of along with its RRSIG record.


Wednesday 1 March 2017

Setting up logging / syslog on the Cisco 2960X

Fortunately logging can be setup very easily on IOS devices - a syslog server can be configured as follows:

conf t

and also configure the logging buffer in memory for example:

logging buffered 64000 debug

The above will ensure that all messages are logged to the buffer and are forwarded to our syslog server.

We can also filter out annoying messages on the console - ensuring only important messages disrupt us:

logging console warnings

The log levels are as follows:

  <0-7>          Logging severity level
  alerts         Immediate action needed           (severity=1)
  critical       Critical conditions               (severity=2)
  debugging      Debugging messages                (severity=7)
  discriminator  Establish MD-Console association
  emergencies    System is unusable                (severity=0)
  errors         Error conditions                  (severity=3)
  filtered       Enable filtered logging
  guaranteed     Guarantee console messages
  informational  Informational messages            (severity=6)
  notifications  Normal but significant conditions (severity=5)
  warnings       Warning conditions                (severity=4)
  xml            Enable logging in XML

We can view the log buffer from global mode with:

show logging

Cisco Switch Setup Checklist (Best Practise, Hardening etc.)

This is a short list of initial management / setup tasks that can be used a a base template:

Setting up AAA

Setting up remote access (SSH)

Setting up SNMP

Setting up logging / syslog

Automatic configuration backups

Setting up NTP

If (like me) you prefer to disable DNS lookups - you can find the IP addresses of stratum 1 and 2 providers here.

(Ideally it's best that the stratum 1 provider takes precedence)

ntp server prefer
ntp server
ntp server

Hardening / Disabling Unnecessary Services 

no ip http server

no ip http secure-server

Ensure there are no vty lines with telnet enabled.

no ip domain-lookup

If you do not need any DHCP services - including DHCP relay (ip helpers) - you can issue:

no service dhcp

Unless you are connecting to an X.25 network - you can safely issue the 'pad' (packet assembler/disassembler service):

no service pad

EXEC Timeout: This defines how long on the session will remain available on a line before logging you out - by default this is commonly set at 10 minutes - although this should generally be much shorter:

line vty 0
exec timeout <minutes>

TCP Keepalives: By enabling these it allows the switch to identify if remote connections (inbound or outbound via SSH, Telnet etc.) are still active or not:

service tcp-keepalives-in
service tcp-keepalives-out


Cisco Guide to Harden Cisco IOS Devices:

NTP Servers UK List: