Linux: Server Not Syncing with NTP (Stuck in INIT state)

The service was confirmed running and set to start on boot:

sudo service ntpd status

Redirecting to /bin/systemctl status ntpd.service
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-01-25 12:11:16 GMT; 13min ago
  Process: 32649 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 32650 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─32650 /usr/sbin/ntpd -u ntp:ntp -g

We can quickly check if NTP is not properly synchronised with the 'ntpstat' command:

> ntpstat

unsynchronised

  time server re-starting

   polling server every 8 s

We can also check the connection status of the ntp server with:

ntpq -p

     remote           refid      st t when poll reach   delay   offset  jitter

================================================

 meg.magnet.ie   .INIT.          16 -    -  512    0    0.000    0.000   0.000

 ec2-52-53-178-2 .INIT.          16 -    -  512    0    0.000    0.000   0.000

 chris.magnet.ie .INIT.          16 -    -  512    0    0.000    0.000   0.000

From the above (specifically the INIT state) my immediate thought was that it was a firewall issue somewhere. 

It's worth checking the EC2 instance SG to ensure that the server can reach udp/123 outbound. However remember to also check the Network ACL (it's stateless) and ensure that udp/123 can get out as well.

ntpd was attempting to initiate a connection with the ntp servers however never got past this phase. After confirming the firewall rules, sg's, ACL's etc.  (i.e. 123/udp outbound and ensuring that session states were maintained) I decided to directly query one of the NTP servers with:

ntpdate -d meg.magnet.ie

This was successful so it seemed something else was causing this.  

In the end I realised it was failing because ntpd was binding with localhost and was attempting to access the external NTP servers (obviously failing because they are unroutable from a loopback device!)

Changing:

listen interface 127.0.0.1

to

listen interface 10.11.12.13

in /etc/ntp.conf  and restarting ntpd resolves the issue:

     remote           refid      st t when poll reach   delay   offset  jitter

===============================================

 x.ns.gin.ntt.ne 249.224.99.213   2 u   27   64    1   11.206   52.795   0.000

 213.251.53.217  193.0.0.229      2 u   26   64    1   12.707   53.373   0.000

 ntp3.wirehive.n 195.66.241.3     2 u   25   64    1   14.334   53.125   0.000

 h37-220-20-12.h 82.69.97.89      2 u   24   64    1   19.211   53.350   0.000