Using the 'nice' command to control a proceses CPU utilization

The nice command can be used to control the CPU utilization of a specific process.

There are varying levels ranging from -20 (which favours the process's CPU utilization) and 19 which assigns a process the least favouritable for CPU utilization.

For example you might want to ensure a large report does not consume all of the available CPU - so you could issue something like:

nice -n 19 /bin/bash /path/to/report.sh

or if we wanted to ensure that the process had priority over others:

nice -n -20 /bin/bash /path/to/report.sh

Read more

Vulnerability scanning with Google

Google can be a huge asset for both white-hat and black-hat hackers - with some carefully crafted queries we can help identify various security issues with a website.

I have outlined some of the common ones below.

Identifying publically accessable directory lsitings:
site:website.com intitle:index.of

Identifying configuration files:
site:website.com ext:xml | ext:conf | ext:cnf | ext:reg | ext:inf | ext:rdp | ext:cfg | ext:txt | ext:ora | ext:ini

Database files exposed:
site:website.com ext:sql | ext:dbf | ext:mdb

Log files exposed:
site:website.com ext:log

Backups and old files:
site:website.com ext:bkf | ext:bkp | ext:bak | ext:old | ext:backup

Login pages:
site:website.com inurl:login

SQL errors:
site:website.com intext:"sql syntax near" | intext:"syntax error has occurred" | intext:"incorrect syntax near" | intext:"unexpected end of SQL command" | intext:"Warning: mysql_connect()" | intext:"Warning: mysql_query()" | intext:"Warning: pg_connect()"

Publicly exposed documents:
site:website.com ext:doc | ext:docx | ext:odt | ext:pdf | ext:rtf | ext:sxw | ext:psw | ext:ppt | ext:pptx | ext:pps | ext:csv

PHP configuration:
site:website.com ext:php intitle:phpinfo "published by the PHP Group"

Read more

Microsoft Exchange: Checking a specific mailbox for corruption

Using the 'New-MailboxRepairRequest' we can quickly detect any corruption in a user mailbox.

I recommend using the '-DetectOnly' switch initially to firstly identify what has (if anything) been corrupted e.g.:

New-MailboxRepairRequest -Mailbox user001 -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView -DetectOnly

and then finally - to actually perform the repairs:

New-MailboxRepairRequest -Mailbox user001 -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView

We can also check all mailboxes within a specific database with the '-Database' switch:

New-MailboxRepairRequest -Database MBDB01 -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView -DetectOnly

If you are on Exchange 2013 / 2016 you can use the 'Get-MailboxRepairRequest' cmdlet to view the status of any repair requests:

Get-MailboxRepairRequest | FL

Otherwise (if you are on an older version) you will need to check the event log for the status - specifically event code: 10062 and 10048.

If corruption is detected in the database you will see something like the following:

Corruptions detected during online integrity check for request 1111111-2222-3333-4444-5555555555555
Mailbox:1111111-2222-3333-4444-5555555555555 (Joe Bloggs)
Database:Mailbox Database 01
Corruption Is Fixed FID Property Resolution
"Folder View", No, "1680-E3FFF4 (Inbox)", 0x00000001, "Delete the corrupted view"
"Folder View", No, "1680-E3FFF4 (Inbox)", 0x00000001, "Delete the corrupted view"

I recommend repeating the procedure until the request come back clean.

Read more

Get the total amount of memory used by a program running several threads

We should firstly get the pid of the parent process e.g.:

ps aux | grep mysqld

mysql     1842  1.1  7.2 2385476 748852 ?      Sl   Jun15.............

and then use ps to retrieve all of its threads and residential memory:

ps -p 1842 -L -o rss

and use awk to add up all of the output and convert to mb (or gb) e.g.:

ps -p 1842 -L -o rss | awk '{s+=$1/1024} END {print s}'

And to put it into a script we can do:

#!/bin/bash

application=$1

if [ -z "$1" ]; then

    echo "Please provide a process name in args!"

    exit

fi

pid=`ps aux | grep -v mysqld_safe | grep $application | grep -v grep | awk '{print $2}' | head -n1 | awk '{print $1;}'`

echo The total residential memory consumed by this application is `ps -p $pid -L -o rss | awk '{s+=$1/1024} END {print s}'`MB

Read more

Memory: Understanding the difference between VIRT, RES and SHR in top / htop

VIRT refer to the virtual size of a process - which is the total amount of memory it is using. For example this could be files on the disk that have been mapped to it, shared libariries and other memory shared with other processes. Hence it is not reliable when attempting to work out the total amount of memory used by multiple processes - since some of it can be shared amoungst them.

RES (Resident Memory) reperesents a more accurate picture of what the process is actually using in terms of physical RAM - it does not include any shared memory - such as shared libraries. And as a result is typically always smaller that the VIRT size since most programs will rely on libraries such as libc.

SHR reperesents how much memory other shared memory and libraries reperesent.

We can get an overview and totals of what memory is mapped to a process with the 'pmap' utility - for example:

sudo pmap <pid>

or

sudo pmap <pid> | grep libc

Read more

Quick Reference: umask examples

Like chmod - umask is used to set permissions - but in a slightly different way. The umask utility applies to files / folders that do not exist - while in contrast chmod is applied to files / folders that already present of the filesystem.

umask stands for 'user file-creation mode mask' - which allows you to define the default set of permissions for a user or system-wide.

The normal user umask is typically set to 002 - which chmod's directories as 775 (everyone can read them but only group and owner can write) and 664 for files - again effect.

The root user on the other hand is usually set to 022 - which instead chmod's the permissions for folders as 755 and 644 - which is as above - but prevents the group from writing to the files or folders.

You can convert the umask into chmod format by performing the following for directories:

777 - umask = chmod

777 - 022 = 755

and similarly for files:

666 - umask = chmod

666 - 002 = 664

You can view the umask for the current user in the terminal by simply issuing:

umask

The umask can be set from a number of locations - although there is a specific order that they are searched and as a result if you have conflicting values - the first one it detects will be applied.

You can configure the system-wide umask within: /etc/login.defs e.g.:

grep UMASK /etc/login.defs

UMASK                   077

This umask will be applied if there is not another umask defined for a user elsewhere e.g.:

cat /etc/profiles

We can see the logic that sets the umask - and checks whether the user is root or not:

if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then
    umask 002
else
    umask 022
fi

umask is evaluated with the following preferences:
local users profile, entry in the users GECOS field, entry in /etc/default/login, entry in /etc/login.defs (Source: https://linux.die.net/man/8/pam_umask)

Although if we wish to set an individual users umask we can edit:

sudo vi ~/.bashrc

and (after verifying it doesn't already exist) add the following at the end of the file:

umask 022

Example Use Case

Lets say we have a script user that pulls configuration (using rysnc or something similar) from one node to another - the configuration residing on the source host is read and resides in /etc/myapp - 

Now usually with a fairly static configuration you might issue something like:

chown -R /etc/myapp root:mygroup

* Where the script user is present in 'mygroup'

although the application on the server writes additional files that only the owner can view and also does not include the ensure that the 'mygroup' group has ownership of the file - when the script user polls the newly created file it is unable to read it.

So - in order to ensure that the 'mygroup' group has ownership and is able to read the newly created files we can issue the following:

Read more

Allowing a script (non-privileged) user to execute privileged commands on Linux

Quite often scripts that are performed by non-privileged users (as they always should where possible) will need to perform a privileged action such as reload a service.

In order to do this 'as securely' as possible we can employ sudo - and telling sudo that the user can run the command xyz as the 'root' user and nothing else.

To do this we need to edit the sudoers file:

vi /etc/sudoers

and add the following line

myusername ALL = (root) NOPASSWD: /usr/sbin/service myservice reload

Read more

Preventing user from logging in via SSH although allowing SCP operations (scponly)

I have come across numerous scenarios where scripts and programs will require SCP to work properly - although do not require SSH access.

By default on CentOS there is not a shell that allows you to restrict SSH but allow SCP - so instead we have to install the 'scponly' shell from EPEL:

yum install scponly

Once installed it should be added to: /etc/shells

/bin/sh
...
/usr/bin/scponly
/bin/scponly

proceed by creating a group for it:

sudo groupadd scponly

Create directory you wish to serve e.g.:

/var/secure/uploads

and ensure the appropriate ownership information is applied (I only want the script to read the files):

sudo chown root:scponly

and permissions e.g.:

sudo chmod 770 /var/secure/uploads

sudo chmod 640 /var/secure/uploads/*

and create a new user and ensure they are part of the 'scponly' group and the appropriate shell is assigned to them:

sudo useradd -m -d /home/script -s "/usr/bin/scponly" -c "script" -G scponly script

Read more

Error: The file '/etc/nginx/ssl/example.pem' is mislabeled on your system

#!!!! The file '/etc/nginx/ssl/example.pem' is mislabeled on your system.

The above error was generated because the specific file type (.pem / certificate) was placed in a location that was not allowed via a SELinux module ruleset or alternatively the file permissions were incorrect.

In order to rectify the problem we should issue:

 restorecon -R -v /etc/nginx/ssl/example.pem

Read more

Quick tip: Generating a human readable report for audit.log (SELinux)

The following command returns a list (ordered by module) that describes what the audit log (/var/log/auid/audit.log) contains:

ausearch -m avc -ts today | audit2allow

Read more

Basics: Locking down a Linux system

(See baseline also)

Below is a quick must-do (or baseline if you will) list I have compiled that should be applied to any server running linux:

1. Remove all unnecessary services - the smaller attack surface you have the better! If you can help it try to install 'minimal' or 'core' versions of the OS - both CentOS and Debian provide them!

2. Harden SSH configuration in /etc/ssh/sshd_config:

- Disable root logins
PermitRootLogin no

- Disable plain-text logins (i.e. private key auth only)
PasswordAuthentication no

- Only listen on necessary interfaces
ListenAddress 10.11.12.13

- Ensure only SSH v2 is enabled (enabled by default on most modern versions of OpenSSH)
Protocol 2

- Ensure UsePrivilegeSeparation is set to 'sandbox' mode
UsePrivilegeSeparation sandbox

- Ensure logging is enabled
SyslogFacility AUTHPRIV
LogLevel INFO

- Set login grace time (can help mitigate DDoS attacks)
LoginGraceTime 1m

- Ensure strict mode is enabled (ensures SSH keys etc. are setup correctly for users)
StrictMode yes

- Public key authentication:
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

- Ensure user can't login with empty passwords:
PermitEmptyPasswords no

- Enable / disable challenge response if needed (e.g. Google Authenticator, two-factor auth etc.):
ChallengeResponseAuthentication no

- Disable weak ciphers (source: https://calomel.org/openssh.html)
Ciphers aes256-ctr,aes192-ctr,aes128-ctr

- Unless absolutley needed - disable X11 forwarding:
X11Forwarding no

and in /etc/ssh/ssh_config:

- Unless absolutely needed - disable X11 forwarding:
ForwardAgent no
ForwardX11 no
ForwardX11Trusted no

3. Kernel tweaks

# Disable IPv6 if not needed
echo 1 > /proc/sys/net/ipv6/conf/<interface-name>/disable_ipv6
Reboot and verify - if all OK make the changes permanent with:
echo 'net.ipv6.conf.all.disable_ipv6 = 1' > /etc/sysctl.conf

# Disable IP forwarding (i.e. if not using anything that requires routing e.g. vpn server etc.)
net.ipv4.ip_forward = 0

# log any malformed packets
net.ipv4.conf.all.log_martians = 1

# disable source routing
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0

# disable icmp re-directs / icmp dos mitigation
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0

# enable syn cookies for synflood protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1280

# enable reverse-path forwarding filter
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# disable tcp timstamping
net.ipv4.tcp_timestamps = 0

Now make the changes permanent with:

sysctl -p

4. Ensure selinux is enabled:

sestatus

If it is not turned on we modify:

/etc/sysconfig/selinux

and ensure 'SELINUXTYPE=targeted'

5. Setup IPTables properly

Ensure that a default-accept rule is in place on all of the default chains:

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

and then flush the chains as well as any non-standard chains:

sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -F
sudo iptables -X

Now we will start by allowing traffic to freely flow out and in from our loopback interface:

sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A OUTPUT -o lo -j ACCEPT

We will also want to ensure that already established connections can get back to the server - i.e. allow stateful connections.

sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

We will likely also want to allow all outbound traffic from connections that are currently established:

sudo iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT

We will also likely wish to allow SSH access from a specific host network:

sudo iptables -A INPUT -p tcp -s 10.11.12.13/32 --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

and allow the incoming SSH connection outbound back to the SSH initiator:

sudo iptables -A OUTPUT -p tcp --sport 22 -m conntrack --ctstate ESTABLISHED -j ACCEPT

We might also wish to accept ICMP echo requests:

sudo iptables -A INPUT -p icmp -j ACCEPT

Allow outbound requests for DNS and HTTP/S:

sudo iptables -A OUTPUT -p udp --dport 53 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 53 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 80 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

And also log and then drop any other packets:

sudo iptables -N LOGGING
sudo iptables -A INPUT -j LOGGING
sudo iptables -A FORWARD -j LOGGING
sudo iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
sudo iptables -A LOGGING -j DROP

Finally we should remove our default-allow rules we inserted prior:

sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT DROP

I also wrote an article here: http://blog.manton.im/2016/06/securing-iptables-in-centos-7-attack.html that might be of interest.

6. Ensure ntp / datetime information is configured correctly.

7. Ensure you have brute-force detection / prevention system in place e.g. fail2ban or sshguard.

8. Utilize tcpwrappers

9. Setup syslog forwarding to a centralized server.

10. Harden other services - dependent on intended usage e.g. web server (apache, nginx etc.)

Read more

OpenVPN AS: 'Address already in use'

Came across a rather annoying problem today with the OpenVPN Access Server when separating out admin and user access to the server.

After adding a secondary interface to the server (for admin access) and configuring OpenVPN AS to run the 'Admin Web UI' on it
The fatal mistake was after I removed the secondary interface - the admin interface defaulted back to my primary interface and the 'Client Web Server' IP address was set to '0.0.0.0' - hence I assume OpenVPN AS was not aware that the 'Client Web Server' port was also bound to the 'Admin Web Server' (usually it is fine to share the same port if the exact IP is specified in both sections.)

So the access-server GUI suddenly became unavailable. Tailing the openvpnas logs returned:

2016-12-06 15:41:24+0000 [-] OVPN 0 OUT: 'Tue Dec  6 15:41:24 2016 Exiting due to fatal error'
2016-12-06 15:41:24+0000 [-] OVPN 0 ERR: 'Tue Dec  6 15:41:24 2016 TCP/UDP: Socket bind failed on local address [AF_INET]172.30.0.194:443: Address already in use'
2016-12-06 15:41:24+0000 [-] Server Agent initialization status: {'errors': {u'openvpn_0': [('error', "process started and then immediately exited: ['Tue Dec  6 15:41:24 2016 TCP/UDP: Socket bind failed on local address [AF_INET]1.2.3.4:443: Address already in use']"), ('error', 'service failed to start or returned error status')]}, 'service_status': {'bridge': 'started', 'log': 'started', 'license': 'started', 'iptables_web': 'started', 'iptables_openvpn': 'started', 'ip6tables_openvpn': 'started', 'auth': 'started', 'ip6tables_live': 'started', 'client_query': 'started', 'api': 'started', 'daemon_pre': 'started', 'web': 'started', 'db_push': 'started', 'iptables_live': 'started', u'openvpn_0': 'off', 'crl': 'started', 'user': 'started'}}

Since I was unable to edit the configuration via the GUI I ended up examining the 'config.json' configuration file:

cat /usr/local/openvpnas/etc/config.json | grep 943

    "admin_ui.https.port": "443",
    "cs.https.port": "443",
 
Although changes to this file didn't seem to work and the error persisted.

So eventually I found out about the 'confdba' command - wich let me view the current database configuration (/usr/local/openvpn_as/etc/db/):

/usr/local/openvpn_as/scripts/confdba -a

and then modify (either the ip address keys or in my case I simply changed the ports aroudn so they would not conflict with eahc other):

/usr/local/openvpn_as/scripts/confdba -mk "cs.https.port" -v "446"

restart the service:

sudo service openvpnas restart

and viola - the web GUI was back up and running!

Read more

Creating swap partition in CentOS 7

We should firstly examine the current swap space with:

swapon -s

Identify how much space we have available for swap with:

df -h

and use the fallocate command to quickly create our swap file e.g.:

sudo fallocate -l 128m /swpfile

Set the relevent permissions on the swap file (we don't want anyone else except root to have access to the swap file!):

sudo chmod 600 /swpfile

And then instruct the system to create the swap space with:

sudo mkswap /swpfile

and enable the swap space with:

sudo swapon /swpfile

and then confirm with:

swapon -s

and

freemem

Read more

Checking whether selinux is available / enabled

The easiest way to check the current status of selinux is to either issue:

sestatus

or

cat /etc/sysconfig/selinux

If neither work we should check the kernel name / version with:

uname -a

and confirm whether the kernel supports selinux (i.e. if it's runnung above version 2.6 it should support SELinux)

If you get a 'command not found' error we can try to install:

yum install setools policycoreutils selinux-policy

and issue the following again:

sestatus

You should now (hopefully) be able to enable by ensuring 'SELINUX=enforcing' and 'SELINUXTYPE=targeted' is present in:

/etc/sysconfig/selinux

Read more

Manually creating the .ssh directory

In some cases I have found that the .ssh directory is not automatically present and hence requires us to create this manually - although it is important permissions are set appropriately, otherwise the OpenSSH will reject the 'authorized_keys' file when a user attempts to login (assuming that 'strict_mode' is enabled within the sshd-config.)

So to create the file structure manually we should issue:

su username

mkdir ~/.ssh
chmod 700

touch ~/.ssh/authorized_keys
chmod 700 ~/.ssh/authorized_keys

Read more