Determining the cause of an ESXI host power failure / restart

Firstly ensure that there are no warning / error lights on the physical host.

Check the event log for the specific ESXI host by going to;

Host >> Tasks and Events >> Tasks

We should then proceed by enabling SSH from the vSphere Client:

Host >> Configuration >> Security Profile >> Services >> Properties and enable SSH.

SSH into the host and run:

cat /var/log/vmksummary.log

You should typically see a regular heart-beat message - although around the time in question we encountered the folloeing event:

2013-01-01T12:30:04Z bootstop: Host has booted

To determine if it was a deliberate reboot we should check for the following line:

localhost vmkhalt: (1268148282) Rebooting system...

* This line would indicate that the boot was initiated by a user.

We should also be able to tell whether it was initiated by a user from the vCenter logs accessable via the vSphere client:

vCenter (Root node) >> Tasks and Events.

Since this line was absent from the vmksummary.log log file it appears that there might have been a power failure at this point.

TCP / UDP Ports Required for Active Directory in an off-premise environment like AWS or Azure

Below are the required ports to get a new domain controller (Server 2008 and above) up and running:

TCP 389
UDP 389
TCP 636
TCP 3268
TCP 3269
TCP 88
UDP 88
TCP 53
UDP 53
TCP 445
UDP 445
TCP 25
TCP 135
TCP 5722
UDP 123
TCP 464
UDP 464
UDP 138
TCP 9389
UDP 67
UDP 2535
UDP 137
TCP 139

Dynamic Ports:

TCP 49152-65535
UDP 49152-65535

Manually configuring DC replication with Active Directory

Firstly we should ensure that all firewall ports are as should be if the replication will be between two different sites. So we go to Sites and Services >> Select our site >> Seelct our server >> Right-hand click on NTDS Settings  >> 'New Active Directory Connection' and select the DC you wish to replicate too.

We then proceed to open up the newley created connection and on the General tab ensure that 'IP' for transport is selected and that the revelent naming contexts are being replicated.

We can then do a repadmin /syncall on the target host to ensure that replication finishes correctly.

Forcing replication of the SYSVOL share

The other day I identified a newly installed domain controller that had not created the SYSVOL share - in order to initiate this I did the following:

Open regedit and go to the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters and set the value of 'SysvolReady' to 0 and then set it again to '1'.

Failure to replicate the SYSVOL folder will cause some pretty serious problems with features such as group policy and the like.

To perform a manual / force replication we run:

ntfrsutl.exe forcerepl "sourcedc" /r "Domain System Volume" /p targetdc

** The "Domain System Volume" is AKA "SYSVOL" **

If you wanted to replicate a specific folder you would run something like:

ntfrsutl.exe forcerepl "sourcedc" /r "my.domain\mysharedfolder" /p targetdc

Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have Replicating Directory Changes In Filtered Set

The other day I came across an error while troubeshooting a problem I had from a run of dcdiag:

Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
   Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=ForestDnsZones,DC=my,DC=domain
Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have
   Replicating Directory Changes In Filtered Set
access rights for the naming context:
DC=DomainDnsZones,DC=my,DC=domain

This indicates a permission problem with the ENTERPRISE DOMAIN CONTROLLERS security group and it's ability to replicate directorty changes in a filtered set.

 To resolve this issue we go to adsiedit on our PDC >> Action >> "Connect to..." >> "Select a type or a Distinguished Name or Naming Context" and enter (replacing the obvious):

DC=ForestDnsZones,DC=my,DC=domain

Expand the new tree node and right hand-click on "DC=ForestDnsZones,DC=my,DC=domain" >> Properties >> Security

and identify the security group "ENTERPRISE DOMAIN CONTROLLERS" and ensure that the "Replicating Directory Changes In Filtered Set" is ticked / set to allowed.

We should then do exactly the same for the "DC=DomainDnsZones,DC=my,DC=domain" partition.

Ensure dcdiag now returns OK and then....




We then proceed by going onto the DC with the permission issues and syncing the changes while specifying the source sever as our PDC:

repadmin /syncall myPDC /APed

Using a vmdk / virtual disk from a VMWare Workstation / Player product in ESXI

I attempted to use a vmdk from a virtual machine hosted on a PC running VMWare Player with an ESXI instance by simply copying the vmdk over SFTP directly to the ESXI datastore and then attaching the disk to a newly created VM on the ESXI host.

Although unfortuantely it wasn't as simple as that as when attempting to turn on the VM I recieved the following errror message in the vSphere client:

"An unexpected error was received from the ESX host while powering on VM XXXXX. Reason: Failed to lock the file. Cannot open the disk '/vmfs/volumes/11fed2c5-81a6f17c-558h-553f/VM01/DISK01.vmdk' or one of the snapshot disks it depends on."

I learnt that the vmdk files that ESXI uses are slightly different from the ones used with the Workstation/Player products and hence have to be converted.

Fortunately VMWare make this pretty easy to do - simply login to the ESXI host, CD to the directory with your VMDK in and run the following command:

vmkfstools -i <original-vmdk> <vmdk-for-esxi>

Introducing the first Windows Server 2008 R2 DC into a Server 2003 domain.

1. Firstly ensure that all DC's are 2003 and decomission any older versions e.g. NT 4.0, 2000 etc.

2. Raise the domain functional level to 'Windows Server 2003' by going to 'AD Domains and Trusts' MMC snapin and right-hand clicking the domain node and select "Raise Domain Functional Level..."

3. Find out which DC holds the schema and infrastructure FSMO roles:

http://blog.manton.im/2015/02/how-to-query-and-move-fsmo-roles-with.html

4. Ensure that there are no outstanding issues with the domain / forest with dcdiag e.g.:

dcdiag /v

and ensure that replication is happening successfully with:

repadmin /showrepl /all /verbose

5. Run the adprep tool on the DC with the above to FSMO roles - the AD prep tool can be found within the 'support\adprep' folder on the root of the Server 2008 R2 disk.

There are too version - adprep.exe (for 64bit OS's) and adprep32 (for 32bit OS's).

** NOTE: You should ensure that the user context launching the adprep tool is a member of the 'Schema Admins', "Enterprise Admins" and "Domain Admins' security group in A.D **

*** WARNING: Before performing something like this it is imporant that (if possible) you can perform this in a similar on ideally mirrored development environment before making changes to the schema OR at least making a backup of AD firstly! ***

So we shall copy the adprep folder directly onto the Server 2003 host and login with the user who hold the schema admin privilages and run the following:

adprep32 /forestprep

or

adprep /forestprep

We can now OPTIONALLY run the 'adprep32 /rodcprep' statement that will prepare the domain / forest for read-only DC's (a feature introudced in Server 2008) with:

adprep32 /rodcprep

And then proceed by preparing the domain with:

adprep32 /domainprep /gpprep

Once this has completed we can then promote our Windows Server 2008 DC's successfully!

Delete specific email meesage from a server / mailbox database with Exchange shell

We should firstly ensure that the user has necessary permissions by assigning thier security group the relvent role:

New-ManagementRoleAssignment -Name "Import Export Mailbox Admins" -SecurityGroup "*SecurityGroupName*" -Role "Mailbox Import Export"

To find an email sent by a user to a numbero users on a specific date / subject we can use:

Get-Mailbox -Server  ExchangeServer | Search-Mailbox -SearchQuery 'Subject:"*My Subject*" AND From:"Joe Bloggs" AND Sent:"11/13/2015"' -targetfolder "Inbox" -targetMailbox "Admin Email" -logonly -loglevel full > C:\temp\results.txt

* The 'targetMailbox' command simply states where the results will be sent too.

Once you have verified from the output that the relevent (and only relevent!) mail items are there we can then use the '-deletecontent' switch to then delete the messages:

** Note: Within the output you should be looking for nodes with something like 'ResultItemsCount' : >0 **

Get-Mailbox -Server  ExchangeServer | Search-Mailbox -SearchQuery 'Subject:"*My Subject*" AND From:"Joe Bloggs" AND Sent:"11/13/2015"' -targetfolder "Inbox" -logonly -loglevel full -deletecontent > C:\temp\results.txt

OR alternatively we can leave the 'targetMailbox' switch in which will save all deleted messages to a mailbox before deleting them:

Get-Mailbox -Server  ExchangeServer | Search-Mailbox -SearchQuery 'Subject:"*My Subject*" AND From:"Joe Bloggs" AND Sent:"11/13/2015"' -targetfolder "Inbox" -targetMailbox "Admin Email" -logonly -loglevel full -deletecontent > C:\temp\results.txt

Moving and removing public folder database replication with Exchange 2010

If you have recently upgraded from an earlier version of Exchange too Exchange 2010 and you have now decided to decomission the oldere version of Exchange you might be required to move all of your existing public folders to the newer server.

We should firstly add our Exchange 2010 server as a replica to ensure the migration goes smoothly by making use of the following script:

.\AddReplicaToPFRecursive.ps1 -TopPublicFolder "\" -ServerToAdd "Exchange 2010 Server"

and also ensuring the 'SYSTEM' public folders are added as well:

.\AddReplicaToPFRecursive.ps1 -TopPublicFolder "\NON_IPM_SUBTREE" -ServerToAdd "Exchange 2010 Server"

We should ensure that after adding the replica server that all of the heiracry and content has been replicated and is upto date with:

Update-PublicFolderHierarchy –Server “<Exchange 2010 Server>”

 Then confirm that the appropraite the public folders are listed as being replicated with the new Exchange 2010 server:

Get-PublicFolder -recurse \ | fl name, replicas

and

Get-PublicFolder -recurse \non_ipm_subtree | fl name, replicas

We can then run the following script with Exchange 2010 to move all public folders to the 2010 instance:

.\MoveAllReplicas.ps1 -Server Exchange2003 -NewServer Exchange2010

 * Note: Do not include the source server if Exchange 2003 as the script will throw a tantrum complaining: "Server 'Exchange 2003' is running Microsoft Exchange 2003 or earlier." ) *

Checking for bad sectors with badblocks and fsck

Badblocks in a linux utility that scan storage media for bad blocks.

It can be operated in serveral modes:

- Destructive mode: Where block data will be wiped, as each sector is overwritten by random data and read. This mode is potentially very dangerous and should typically be only applied on disks that are brand new or you are not worried about losing the data on them!

- Non-destructive mode: Where block data is checked, although rather than overwriting the original block data (effectively wiping it) the block data is firstly backed up. This mode is useful if you have data on the disks you are testing which you don't want to lose! - Although takes slightly longer than the destructive mode.

!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*
READ CAREFULLY!
!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*

To perform a DESTRUCTIVE block data test we can issue the following:

badblocks -wsv -t random /dev/<device>

!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*
READ CAREFULLY!
!*!*!*!*!*!*!*!*!*!*!*!*!*!*!*

We can perform a NON-DESTRUCTIVE test with the following:

badblocks -nsv /dev/<device>

(where the 'n' indicates it's a non-destructive test)

We can also tell our filesystem not to include any badblocks with the fsck utility:

fsck -vcck /dev/<device-PARTITION>

This command ends up calling 'badblocks' although will perform a non-destructive test as we have included the '-cc' option.

Performing an off-site backup with AWS using Veeam Backup and Replication

It is now possible to backup from Veeam to AWS through the use of the AWS Storage Gateway service.

To explain how it works: AWS Storage Gateway allows you to create a Virutal Tape Library Gateway - than simply speaking is a way of creating a virtual tape drive in the cloud that can hook upto other AWS services such as S3 and Glacier.

You are required to download the AWS Storage Gateway virtual appliance to act as an intermetriaty to effectively proxy the data between the Veeam server and AWS. Although in order to hook Veeam upto the virtual appliance you are required to install specific tape drive drivers that emulate a physical tape drive - but actually hook into the AWS storage gateway appliance.

Please refer to the below article for help deploying / provisioing the Storage Gateway appliance:

http://docs.aws.amazon.com/storagegateway/latest/userguide/GettingStartedSetupVMware-common.html

and the following article on how to install the appropraite drivers for your backup solution:

http://docs.aws.amazon.com/storagegateway/latest/userguide/resource_vtl-devices.html#update-vtl-device-driver

Create the relevent tape drives and select your cache and upload buffers disks:

http://docs.aws.amazon.com/storagegateway/latest/userguide/GettingStartedCreateTapes.html

Configure the Veeam server's drivers / iSCSI settings:

http://docs.aws.amazon.com/storagegateway/latest/userguide/GettingStartedAccessTapesVTL.html

Veeam makes use of the STK-L700 device - of which I had issues getting to work under Server 2008 - the medium changer showed up as 'Unknown Medium Changer' - so I ended up downloading the xxxxxxx driver from the Microsoft Update Catalog via IE:

http://catalog.update.microsoft.com/v7/site/Home.aspx

The driver is called: StorageTek - Storage - Sun/StorageTek Library

I then re-scanned the tape server and only then I was able to right-hand click on the medium changer (now called STK L700 0103) and click 'Import Tapes'.

For convienicne you can download the x64 version of the driver here (labelled as support server 2003 and 2008 R2)

and finally on how to configure Veeam:

http://docs.aws.amazon.com/storagegateway/latest/userguide/backup-Veeam.html#veeam-configure-software

How to enable an AD security group for use with Exchange

By default when you want to use an AD security group within Exchange - lets say for example within a transport rule you will notice that by default they are not available.

So in order to make the security groups accessable we need to 'mail-enable' them: AKA mail enabled security groups. In order to do this we should firstly ensure that the security group's scope is 'Universal' NOT 'Global' as it is by default.

We can then proceed to go to the Exchange Management Console >> Recipient Configuration >> right-hand click 'Distribution Group' >> New Distribution Group >> Select 'Existing Group' >> Next ensure 'Security' is selected for the group type and seelct the relevent security group.

You should now be able to specify your mail enabled security group within Exchange e.g. when creating transport rules.

Deleting old backups from Windows Backup Sets

Although you can do this via the control panel >> Windows Backups - if you are using a thrid party product that is utilizing the Windows Backup engine you will need to use the wbadmin tool.

I was recently required to clear out several older windows backups to free some space on the disk.

Firstly we can  view all backups within a backup set with something like:

wbadmin get versions -backupTarget:"B:\"

(where B: is the root of the backup.)

We can use the vssadmin tool to list all of our VSS backups with:

vssadmin list shadows /for=b:

and delete the oldest VSS backup using diskshadow in interactive mode:

diskshadow
delete shadows oldest b:

Exchange Routing Groups

Routing groups are used to provide communication between to Exchange servers - typically between two different versions of Exchange e.g. Exchange 2010 and Exchange 2003.

The two servers that form the source and destination of the routing are reffered to as 'bridgehead servers'.

In order to view information about current routing groups we can use:

Get-RoutingGroupConnector | FL

Routing group connectors are unidrectional routes between two bridgehead servers i.e. a seperate routing group has to be defined for both incoming and outgoing mail.

In the event of a post migration the task of removing the old Exchange server would require removing any redundent routing groups with something like:

Remove-RoutingGroupConnector "My Routing Group"

Just like send connectors routing group connectors also have a cost associated with them allowing you to specify preferred routes.