Tuesday, 22 May 2018

vSphere Replication 'Not Active' status after initial setup of replication (manually install VIB file)

In my experience this error is caused due to the VIB file not being installed on one or more hosts.

Following the below steps will help you get replication up and running again:

1. Firstly identify which ESXI host the VM resides on that you are attempting to replicate.

2. Download the VIB file for vSphere Replication:

wget https://<vsphere-replication-address>/vib/vr2c-firewall.vib

2. Enable SSH on the ESXI host, scp the file to it:

scp vr2c-firewall.vib root@<esxi-host>/tmp

3. Login to the ESXI host via SSH and install the vib file:

esxcli software vib install -v /tmp/vr2c-firewall.vib

4. Usually the replication should start working immediately - but if it doesn't try restarting the vSphere Replication Appliance.

Wednesday, 25 April 2018

Changing the baud rate on a Cisco 3650

Unfortunately the only way to do this is from ROMMON mode - so in order to easily access it we can instruct the switch to automatically enter it on the next reload:

conf t
boot enable-break

or alternatively power of the switch, hold down the 'Mode' button, turn the switch on (while still holding the 'Mode' button) for around 15 seconds.

Then set the baud rate appropriately:

set BAUD 9600

boot the IOS image:


and finally once it's booted up ensure that we revoke the 'enable-break' command:

conf t
no boot enable-break

Wednesday, 4 April 2018

QoS for telephony on the 3650

Below is an example of QoS you can apply for telephony on the 3650. Ingress traffic on gi1/0/1 is marked accordingly and then queued according to the service-policy on gi1/0/24.

# Input QoS

ip access-list extended VOIP
 permit udp any range 16384 32767 any range 16384 32767

ip access-list extended MULTIMEDIA-CONFERENCING
 permit udp any any range 16384 32767

ip access-list extended CALL-SIGNALING
 10 permit tcp any any range 2000 2002
 ! SIP
 20 permit tcp any any range 5060 5061
 30 permit udp any any range 5060 5061
 ! H.323
 40 permit udp any any range 1718 1719
 permit tcp any any eq 1720
 50 permit tcp any any eq 2428
 60 permit tcp any eq 2428 any
 70 permit udp any any eq 2427
 80 permit udp any eq 2427 any

ip access-list extended TRANSACTIONAL-DATA
 10 permit tcp any any eq 443
 20 permit tcp any any eq 1521
 30 permit udp any any eq 1521
 40 permit tcp any any eq 1526
 50 permit udp any any eq 1526
 60 permit tcp any any eq 1575
 70 permit udp any any eq 1575
 80 permit tcp any any eq 1630
 90 permit udp any any eq 1630
 100 permit tcp any any eq 1527
 110 permit tcp any any eq 6200
 120 permit tcp any any eq 3389
 130 permit tcp any any eq 5985
 140 permit tcp any any eq 8080

ip access-list extended BULK-DATA
 10 permit tcp any any eq 22
 20 permit tcp any any eq 465
 30 permit tcp any any eq 143
 40 permit tcp any any eq 993
 50 permit tcp any any eq 995
 60 permit tcp any any eq 1914
 70 permit tcp any any eq ftp
 80 permit tcp any any eq ftp-data
 90 permit tcp any any eq smtp
 100 permit tcp any any eq pop3

ip access-list extended SCAVENGER
 10 permit tcp any any range 2300 2400
 20 permit udp any any range 2300 2400
 30 permit tcp any any range 6881 6999
 40 permit tcp any any range 28800 29100
 50 permit tcp any any eq 1214
 60 permit udp any any eq 1214
 70 permit tcp any any eq 3689
 80 permit udp any any eq 3689
 90 permit tcp any any eq 11999

class-map VOIP
 match access-group name VOIP
 match access-group name MULTIMEDIA-CONFERENCING
 match access-group name CALL-SIGNALING
 match access-group name TRANSACTIONAL-DATA
class-map BULK-DATA
 match access-group name BULK-DATA
class-map SCAVENGER
 match access-group name SCAVENGER

 class VOIP
 set dscp ef
 set dscp af41
 set dscp cs3
 set dscp af21
 class BULK-DATA
 set dscp af11
 set dscp cs1
 class class-default
 set dscp default

int gi1/0/1
 service-policy input MARKING-POLICY

# Output QoS

class-map match-any VOICE-QUEUE
 match dscp ef
 match dscp cs5
 match dscp cs4
 match dscp af41 af42 af43
 match dscp af31 af32 af33
class-map match-any NETWORK-CONTROL-QUEUE
 match dscp cs7
 match dscp cs6
class-map match-any SIGNALING-QUEUE
 match dscp cs3
 match dscp cs2
class-map match-all TRANSACTIONAL-DATA-QUEUE
 match dscp af21 af22 af23
class-map match-all BULK-SCAVENGER-DATA-QUEUE
 match dscp af11 af12 af13 cs1

policy-map qos_pm_2P6Q3T_out
 priority level 1
 police rate percent 10
 bandwidth remaining percent 10
 queue-buffers ratio 10
 queue-limit dscp af43 percent 80
 queue-limit dscp af42 percent 90
 queue-limit dscp af41 percent 100
 bandwidth remaining percent 10
 queue-buffers ratio 10
 queue-limit dscp af33 percent 80
 queue-limit dscp af32 percent 90
 queue-limit dscp af31 percent 100
 bandwidth remaining percent 7
 queue-buffers ratio 10
 bandwidth remaining percent 3
 queue-buffers ratio 10
 bandwidth remaining percent 30
 queue-buffers ratio 10
 queue-limit dscp af23 percent 80
 queue-limit dscp af22 percent 90
 queue-limit dscp af21 percent 100
 bandwidth remaining percent 5
 queue-buffers ratio 10
 queue-limit dscp values af13 cs1 percent 80
 queue-limit dscp values af12 percent 90
 queue-limit dscp values af11 percent 100
class class-default
 bandwidth remaining percent 25
 queue-buffers ratio 25

int gi1/0/24
 service-policy output qos_pm_2P6Q3T_out

Friday, 30 March 2018

Tuesday, 27 March 2018

Policing / Shaping traffic in CentOS with tc

This can be achieved using the tc command - below is a simple example that polices all traffic on the interface. However you can also tag traffic with iptables and apply throttling based on these tags for more complicated scenarios.

Policing Example

tc qdisc add dev enp0s25 handle ffff: ingress
tc filter add dev enp0s25 parent ffff: protocol ip prio 50 u32 match ip src police rate 256mbit burst 0k drop flowid :1
tc qdisc add dev enp0s25 root tbf rate 256mbit latency 25ms burst 0k

Friday, 23 March 2018

QoS Congestion Avoidance

An excellent article on QoS Congestion Avoidance from netcerts:

QoS Congestion Avoidance

Tail Drop
Tail Drop is when the packets are dropped when they arrive on a congested interface.Tail Drop is not just bad for voice packets but for data packets as well. It also impacts the efficiency of network bandwidth utilization. When the Output Queue is full and packets  arrive in on the Input Queue, then the packets which are arriving on the interface will be dropped. It does not matter if its a voice packet or a data packet, everything will be dropped by default when Tail Drop is in action. TCP has a mechanism where it can identify the packet loss and send those packets again.  TCP Senders will start sending packets, and they will increase the rate at which they send packets when there is no congestion and no packet loss. The increase is exponential over a time interval. When packets are being dropped, they sense that network is congested and then TCP Slow start kicks in, the sending rate is  suddenly dropped by a big margin until there is no more packet loss and again the sending rate picks up slowly after certain time interval where TCP does not see any packet loss.
When TCP Slow Start kicks in, all senders on the network back off and you can see a drop in the bandwidth, then slowly everyone starts sending packets at higher rate as they see no more packet loss, so all senders on the network start sending the packets again at higher rate and you see peaks in the network bandwidth.  At this time the interfaces can get congested again and packets can be dropped, which then makes all senders to drop their sending rate and wait for certain time interval where they see no more packet loss, this leads TCP Senders to again increase the sending rate. This goes on in cycles and this behavior  means a lot of bandwidth is just getting wasted. If you are monitoring the bandwidth with a graph, you will something like below graph in  the utilization charts. This behavior is also called as “Global TCP Synchronization” and it is responsible for a lot of network bandwidth wastage.
Tail Drop is the default congestion avoidance mechanism. More efficient congestion avoidance techniques can be configured on Cisco routers which are
1. RED – Random Early Detection
2. WRED – Weighted Random Early Detection
Congestion avoidance techniques basically monitor the network traffic and analyze the bandwidth loads, then they anticipate and avoid congestion at common network bottle-necks by dropping packets in advance.
Random Early Detection (RED)
With RED, the packets are discarded randomly before the Queue gets full, and increase the random drop rate when the queue is getting full. With RED implementation the problem of global TCP Synchronization can be effectively avoided.
RED Terminology
1.Minimum Threshold:
Is the point where the average queue hits or goes over this threshold value and random packet dropping starts.
2. Maximum Threshold:
Is the point where tail drop behavior comes into action.
3. Mark Probability Denominator:
Is the fraction of packets to be randomly dropped. That is how many packets to be dropped.
Weighted Random Early Detection (WRED)
WRED is Cisco implementation of RED. Cisco does not do the random dropping using RED algorithm, instead it used its own WRED.
All WRED does is to make sure that important packet does not get dropped in the random dropping process, instead a not so important packet is dropped.
With WRED multiple RED profiles are automatically created by the router based on IP Precedence or DSCP. If you decide that WRED is based on IPP then router will generate 8 different profiles for the dropping decision. If you decide to use DSCP then the router will create 64 different profiles for the dropping decision.
WRED Profile consists of :
1. Minimum Threshold: Is the point where random packet dropping starts
2. Maximum Threshold : Is the point where tail drop behavior comes into action.
3. Drop Probability: Determines how many packets to be dropped.
All these three values are calculated for each of  the profile – which is 8 profiles when IPP is used and 64profiles when DSCP is used.
By Default:
1. EF will get a very high minimum threshold.
2. AF Classes get their minimum thresholds as per their Drop Priorities.
These values can also be set manually.
Command to Configure WRED
# random-detect
Class Based WRED (CB-WRED)
When CB-WRED is used in conjuction with CBWFQ, it allows DiffServ Assured Forwarding Per Hop Behavior.
Configuration of CB-WRED
under policy-map (MQC)
# random-detect ! –> This will give the IPP based WRED by default
To give DSCP based CB-WRED
# random-detect dscp-based
To modify the thresholds
# random-detect precedence <precedencevalue> <min-threshold> <max-threshold> <mark-probability>
# random-detect dscp <dscpvalue> <min-threshold> <max-threshold> <drop-probability>
Explicit Congestion Notification (ECN )
DSCP uses 6 bits of ToS Byte in IP header, and the last 2 bits are used for ECN.
ECN marks the packets when the average queue length exceeds a specified value and the Routers and the end hosts can lower the sending rates based on ECN. But all routers and the end hosts in the network must support this feature to work.
ECN-Bits:     Description:
00                  Not ECN Capable
01                   ECN Capable
10                   ECN Capable
11                    Indicates Congestion
If the average queue length is between minimum threshold and maximum threshold then ECN bits are marked and ECN process begins. If end-points (routers and hosts) are not ECN capable then packets may be dropped
Under Policy-map
# random-detect ecn
To allows for bursts
Exponential Weighting Constant is the value which determines the burst rate. You can control the WRED sensitivity to bursts.
Command: # random-detect exponential-weighting-constant <N>
By Default the value of N is 9
Lower Values of N – long bursts that is WRED will be more burst sensitive and can cause more packet loss.
Higher Values of N – allows for short bursts
1. If the value of n gets too low, WRED will overreact to temporary traffic bursts and drop traffic unnecessarily.
2. If the value of n gets too high, WRED will not react to congestion. Packets will be sent or dropped as if WRED were not in effect.
Monitoring CB-WRED
# show policy-map interface 
Full credit to netcerts.net

Wednesday, 21 March 2018

Example: Applying QoS on a Cisco 3650 / 3850

Note: In older models / OS's we needed to issue 'mls qos' in order to enable QoS on the switch - however with the 3650 QoS is enabled by default.

The aim is to tag voice traffic with DSCP that hits either gi0/1 or gi0/2 from VLAN 10 and ensuring data traffic is even spread between vlan20 and vlan30. Note: In most cases AutoQoS is a much better solution and greatly simplifies the configuration - however I created this lab to demonstrate a simple example manually:

vlan 10
desc priority traffic
name VLAN10

vlan 20
desc non-priority data traffic 1
name VLAN20

vlan 30
desc non-priority data traffic 1
name VLAN20

We'll need to use MQC (Modular QoS CLI) in order to apply the QoS - so let's firstly create our class map:

####### Set ingress service policy

class-map qos_vlan10_cm_in
match vlan 10

class-map qos_vlan20_cm_in
match vlan 20

class-map qos_vlan20_cm_in
match vlan 30

policy-map qos_pm_in
class qos_vlan10_cm_in
set ip dscp 40
class qos_vlan20_cm_in
set ip dscp 0
class qos_vlan30_cm_in
set ip dscp 0

int gi1/0/1
service-policy input qos_pm_in

int gi1/0/2
service-policy input qos_pm_in

And now for the egress QoS:

class-map qos_vlan10_cm_out
match ip dscp 40

class-map qos_vlan20_cm_out
match ip dscp 0

class-map qos_vlan30_cm_out
match ip dscp 0

policy-map qos_pm_out
class qos_vlan10_cm_out
priority 1 percent 10
class qos_vlan20_cm_out
bandwidth percent 20
class qos_vlan30_cm_out
bandwidth percent 20
class class-default
bandwidth remaining percent 100

# The 'bandwidth percent' command provides 20% of the interfaces bandwdith to vlan20 and 20% to vlan30 - however during times were there is no contention other classes can utilise additional bandwidth. I.e. The command doesn't reserve bandwidth. While the remaining 100% of the bandwidth is shared between other traffic.

# The 'priority level' command always ensures that traffic from VLAN10 is always served before anything else and reserves 10% of the link (i.e. 100 megabits) for the voice traffic - however it is not able to utilise any greater than 10%!

Finall apply the service policy to the outgoing interface with:

int gi1/0/24
service-policy output qos_pm_out