Saturday, 22 August 2020

Recovering from a replication failure in a MariaDB Master/Master replication setup

For the purposes of this post I'll assume we are working with two MariaDB servers that have been configured to perform master/master replication and one of them has failed. In this case Server01 is healthy while Server02 has stop replicating.

We need to firstly ensure that no queries are hitting Server02 / the failed server - this will typically be a case of stopping services / blocking network access to services that hit it. e.g. stopping httpd.

We'll also want to ensure replication is stopped on the failed server (Server02):

SERVER02> stop slave;

Now on Server01 / the working server issue:

SERVER01> stop slave;

SERVER01> flush tables with read lock; (This will temporarily stop it updating)

SERVER01> show master status;

We'll make a note of the above command - it should read something like:

File: mysql-bin.123456

Position 123

Binlog_Do_DB: <replicated_database>

Then on Server01 / the working server take a backup of the database:

SERVER01> mysqldump -u<username> -p --lock-tables --databases <database-name[s]> > export.sql

and on Server02 / the failed server - restore the backup:

SERVER02> mysql -u root -p < export.sql

Now on Server01 / the working server issue the following command to start processing changes again:

SERVER01> UNLOCK TABLES;

Then on Server02 / the failed server issue the following to repoint the logs (use the information above we recorded from Server01):

SERVER02> CHANGE MASTER TO master_log_file='mysql-bin.xxxxxx', master_log_pos=yy;

SERVER02> START SLAVE;

To verify we can issue:

SERVER02> show slave status \G

Now we need to do the reverse by ensuring Server01 / working server replicates from Server02 / failed server. On Server02 issue:

SERVER02> show master status \G

Record the output again.

Now on Server01 / the working server set the logs:

SERVER01> CHANGE MASTER TO master_log_file='mysql-bin.xxxxxx', master_log_pos=yy;

SERVER01> START SLAVE;

and then to verify replication issue:

SERVER01> SHOW SLAVE STATUS \G

Finally reverse anything you performed at the start to block comms with Server02 / the bad server e.g. start services, update firewall etc.


Friday, 10 July 2020

Monday, 16 March 2020

Invoking sysprep (Generalising a Windows install) on AWS EC2

  1. From the Windows Start menu:
    For Windows Server 2008 through Windows Server 2012 R2, open EC2ConfigService Settings, and then choose the Image tab.
    For Windows Server 2016 or later, open EC2 Launch Settings.
  2. For Administrator Password, choose Random.
  3. Choose Shutdown with Sysprep.
  4. Choose Yes.
    Note: You must retrieve the new password from the EC2 console at the next service start.

Friday, 13 March 2020

Instruct AWS EC2 'User Data' to be invoked on startup (Server 2016+)

When launching Amazon EC2 images 'user-data' (effectively a bootstrapper) is invoked on first launch. However if you create a custom AMI from one of these images you'll need to run the following to ensure user data is invoked (as the task that invokes it gets disabled prior) with:

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 –Schedule

Shutdown the instance and then create AMI again.

Sunday, 1 March 2020

Quick Start: Upgrading Juniper SRX Devices

Once you've obtained the relevant firmware from Juniper you can either download it via https:

file copy https://cdn.juniper.net/software/junos/XX/XX.tgz /tmp XX.tgz

or alternatively if you need to download it from a named routing instance you'll need to download it over ftp firstly:

ftp routing-instance <instance-name> <ftp-host>

start shell
md5 /tmp/<firmware>.tgz

request system software add validate /tmp/<firmware>.tgz

The system will then extract the firmware and reboot immediately.

To verify the Junos firmware version after reload issue:

show version

Tuesday, 18 February 2020

[Solved] Snort: ERROR: Can't initialize DAQ pfring (-1) -

I came across this error after performing a regular system update on CentOS 7. Although it's a rather generic looking error message it turned out to be quite a trivial problem.

The pfring driver (provided by daq_pfring) had been compiled against the latest kernel version - however for whatever reason an older kernel was being loaded by default by the bootloader.

This can be evidenced by running:

uname -r

and a rpm -qa | grep kernel

To correct this issue:

grub2-set-default 0 # presuming menu item 0 is the kernel you want listed in: /boot/efi/EFI/centos/grub.cfg (which is usually the case.)

and then confirm with:

grub2-editenv list

Restart the machine and then check the kernel / test snort again:

shutdown -r now

sudo service snort status


Tuesday, 14 January 2020

Visualising data from iperf with rrd

The purpose of this test was to test the availability of bandwidth on a leased line while ensuring that the test itself didn't saturate the line.

We'll firstly run our iperf server in daemon mode:

iperf3 --server --daemon --logfile iperf_stdout.txt --pidfile iperf3.pid

Since this will be a long term test we'll ensure that there is no timeout on the test and that intervals of 1 second are reported (since we'll be using this for rrd input):

iperf3 -b 20M -c hlxscript01.hlx.int -i 1 -t 0 -V --logfile log.txt &

In the above example I'm sending a stream traffic equalling 20Mbits. If you wish to saturate the line you will need to remove this and also likely tweak with threads and the TCP window size in order to get optimum results.

Now in order to use our client log (log.txt) for use with rrd we'll need to extract the timestamp along with the recorded speed, feed it into the rrd file and finally generate the graph. I've created a simple shell script to do just that:

#!/bin/bash

epoc=$(date "+%s")

IFS=$'\n'
iperf_results=( $(cat log.txt | grep -o '[0-9]\+\.[0-9]\+ Mbits\/sec' | cut -d " " -f 1) )
results_count="${#iperf_results[@]}"

rrdtool create iperf.rrd --step=1 --start=$epoc-$results_count DS:ds1:GAUGE:1:U:U RRA:AVERAGE:0.5:1:$results_count

START=$(expr $epoc - $results_count)
COUNT=$results_count
for (( i = 0; i < ${COUNT}; i++ )); do
VALUE=${iperf_results[i]}
rrdtool update iperf.rrd ${START}:${VALUE}
START=$(expr ${START} + 1)
done

rrdtool graph iperf.png --start $epoc-$results_counts --end now DEF:ds1a=iperf.rrd:ds1:AVERAGE LINE1:ds1a#FF0000:"Sinus line"

Sources