Rebuilding a GPS NTP Stratum 1 Server in Two Hours: Our Log

On 23 October 2026, we set out to replace our Stratum 1 server, ntp-0-hq.rdem-systems.com, gently: a Raspberry Pi with a GPS receiver, in service since 2018. The plan was to build a new machine next to the old one, compare them for a few days, then switch over. By the end of the day, the new machine was idle (the GPS receiver ordered for it does not come up on the new Pi; we will come back to that), the old one had lost its SD card, and it was the old Raspberry Pi, reinstalled from scratch, that was serving Stratum 1 time again, this time with chrony and NTS, over IPv4 and IPv6.

This page follows the log kept during the work: the commands, what they returned, and the pitfalls, including the ones we set ourselves. GPS coordinates, internal addresses and secrets are removed; everything else is as recorded.

Before and after

Backing up the old server, and what it revealed

Before touching anything, we archived /etc, /boot, /usr/local, /root and /home from the old server, with a snapshot of its state (NTP peers, open ports, packages) and SHA-256 hashes of the files. Twenty key files were compared with their originals on the machine: identical. That archive came in handy later in the day, when the SD card gave out.

Rereading an eight-year-old configuration has its surprises:

• the PPS did not go through the kernel, but through a user-space program started from rc.local;
• the same rc.local stopped ntpd, reset the clock with ntpdate, then restarted ntpd;
• two typos in ntp.conf (inurst instead of iburst, npt.polytechnique.fr instead of ntp.): two cross-check sources had never been used;
• a fixed IPv6 address was configured and served, but no AAAA record published it.

The old server's SD card dies

For a test on the new machine, the old GPS board was taken off the old server, so the old server was shut down. With the board back in place, it no longer booted: its 2018 SD card, which had just been through a power cut (our UPS no longer holds the load) and then the shutdown, was dead. During that outage ntp-0-hq did not answer; our Stratum 2 servers fell back to their other sources, and the time they serve stayed correct.

Restoring the old system would have meant reinstating Raspbian 10, which is no longer maintained, and ntpd without NTS. We chose to reinstall the old Pi from scratch on a new SD card, with the old GPS board and the outdoor antenna: the target planned for the new machine, on the hardware that worked. From the loss of the SD card to being back at Stratum 1, including bringing IPv6 back and setting up NTS, the rebuild took two hours.

The rebuild, step by step

1. Base system

Raspberry Pi OS Lite Trixie, apt full-upgrade, then raspi-config: SSH, hostname, time zone Europe/Paris, keyboard, predictable interface names, and under Interface Options → Serial Port, no console on the serial port but the hardware port enabled. On Trixie, a fixed address is set with NetworkManager (nmcli or nmtui): raspi-config can no longer do it. On a Pi 3, the Ethernet interface goes through USB and is named enx followed by the MAC address.

2. Serial port and PPS

In /boot/firmware/config.txt, section [all]:

enable_uart=1
dtoverlay=disable-bt
dtoverlay=pps-gpio,gpiopin=4

On a Pi 3, disable-bt gives the PL011 UART, which Bluetooth holds by default, back to the GPS; otherwise the GPS ends up on the mini-UART, whose speed depends on the CPU frequency. This board's PPS arrives on GPIO4. cmdline.txt must no longer contain console=serial0,115200, or the console talks to the GPS. After a reboot:

$ ls -l /dev/serial0 /dev/pps0
crw------- 1 root root 249, 0 Oct 23 16:54 /dev/pps0
lrwxrwxrwx 1 root root      7 Oct 23 16:54 /dev/serial0 -> ttyAMA0

The sentences arrive at 9600 baud, with a corrected fix (quality 2, SBAS) on 8 satellites and $GPTXT,01,01,02,ANTSTATUS=OK: the antenna is detected. gpsd identifies the module:

$ gpspipe -w -n 6 | grep DEVICES
"driver":"MTK-3301","subtype":"AXN_2.5_3339_15092100,0004,QUECTEL-L80,1.0"

A Quectel L80 module on a MediaTek MT3339 chip. "MTK-3301" is the name of gpsd's MediaTek driver, not of the chip: that is where the wrong mention on our old page came from. The PPS, one pulse per second:

$ ppstest /dev/pps0
source 0 - assert 1792767887.000497314, sequence: 296 - clear  0.000000000, sequence: 0
source 0 - assert 1792767888.000499248, sequence: 297 - clear  0.000000000, sequence: 0
source 0 - assert 1792767889.000502276, sequence: 298 - clear  0.000000000, sequence: 0

This reading predates chrony: the Pi's clock, left to itself, runs about 0.5 ms ahead of the GPS, and the gap grows by about 2.4 µs per second.

3. gpsd

Packages gpsd gpsd-clients pps-tools chrony, then /etc/default/gpsd:

DEVICES="/dev/serial0 /dev/pps0"
GPSD_OPTIONS="-n"
USBAUTO="false"

-n makes gpsd read the GPS continuously, even with no client connected: chrony is not a gpsd client, it reads the shared memory that gpsd fills. The service must also be enabled; we learnt that at the reboot (below).

4. Ansible

The rest of the configuration comes from our Ansible role: users and keys, chrony (sources, reference clocks, client access, leap seconds through leapseclist), NTS certificate and monitoring. One remark that applies to any chrony managed this way: sources.d/*.sources files only accept the server, pool and peer directives. refclock lines go in conf.d/.

chrony: PPS, NMEA and the wrong second

The two reference clocks in service:

# /etc/chrony/conf.d/refclock.conf
refclock SHM 0 refid NMEA noselect poll 4 precision 1e-1 delay 0.2
refclock PPS /dev/pps0 refid PPS poll 4

NMEA, set to noselect, is never chosen as the reference: the 9600-baud serial line delays it by about 175 ms. The PPS carries the precision. Six outside servers act as witnesses, four of them over NTS (Paris Observatory, École polytechnique, PTB ×2, Netnod, Cloudflare), alongside our own servers: they do not set the time while the PPS is there, but they would get a lying PPS rejected.

Why we removed lock NMEA

The first version of the configuration had lock NMEA on the PPS line: chrony then numbers each pulse with the second from the NMEA sentences. Yet a few minutes after the cold start, the receiver stamped its sentences one second ahead: the sentence announcing 15:20:34 arrived at 15:20:33.170 system time, the system clock itself being set from the network to within 26 µs. The fault was gone after 22 minutes of operation:

15:35:55.259 UTC  <-  $GPZDA,153555.000
15:35:56.258 UTC  <-  $GPZDA,153556.000

Our hypothesis, consistent with this behaviour but not checked in the firmware: until it has received the UTC parameters broadcast by the satellites (up to 12.5 minutes), the receiver applies an old value of the offset between GPS time and UTC, 17 seconds instead of 18.

With the network available, the wrong second does no harm: the witnesses answer within seconds, the PPS takes one to two minutes to be accepted, and chrony rejects it. The dangerous case is a power cut, where the Pi reboots before the ISP box and the router. With lock NMEA, the PPS would then be the only source, and chrony, allowed to step the clock at any time, would put it on a wrong second before serving it at Stratum 1. Without lock, chrony only uses the PPS once another source has set the clock to within half a second. With no network, the server declares itself unsynchronised and its clients ignore it. No time rather than a wrong time. As far as we can read its configuration, that was already how the old server worked: its PPS was numbered by the system clock.

NTS and IPv6

The NTS certificate is issued by Let's Encrypt with DNS-01 validation, for the names ntp-0-hq.rdem-systems.com (the public name) and ntp-0.hq.rdem-systems.com (the management name). Its renewal is part of the centralised renewal of our certificates. NTS-KE listens on TCP port 4460, defined by RFC 8915; the protected time requests then use the usual NTP port, UDP 123.

Over IPv4, the server sits behind a double NAT: UDP 123 and TCP 4460 have to be forwarded on both devices. Over IPv6 there is nothing to translate, only a firewall to open. The Pi got a fixed address, added next to its automatic one. An mtr from outside showed where the traffic stopped: it crossed the ISP box (hop 10) and reached the router (hop 11), then nothing. An ICMPv6 rule pointed at the wrong zone, and an address change on the ISP box side was still blocking NTP. Once fixed:

10.|-- 2a01:e0a:808:97c0::1       0.0%     3    3.1   3.4   3.1   3.8   0.4
11.|-- 2a01:e0a:808:97c0:265a:4c  0.0%     3    3.9   3.8   3.7   3.9   0.1
12.|-- 2a01:e0a:808:97c4::123:12  0.0%     3    4.5   4.3   3.9   4.5   0.3

An mtr only proves ping. The service is checked with a real client, here chronyd -Q from one of our servers in Paris (AS206014), which queries without touching the local clock:

$ chronyd -Q -4 'server ntp-0-hq.rdem-systems.com nts iburst'
System clock wrong by -0.000062 seconds (ignored)
$ chronyd -Q -6 'server ntp-0-hq.rdem-systems.com nts iburst'
System clock wrong by -0.000308 seconds (ignored)

Queried by its IPv6 address rather than its name, NTS-KE answers The name in the certificate does not match the expected. That is expected, since the certificate carries names, not addresses, and it incidentally proves that NTS-KE answers over IPv6.

The two reboot pitfalls

We finished with a test reboot. It revealed two faults that continuous operation would have hidden.

chrony was running the old configuration

The Ansible role had written its files, but chronyd, started an hour earlier, had not been restarted: it was still following the Debian pool, with no reference clock and no client access. The files were right; the running state was not. A simple check: systemctl show chrony -p ActiveEnterTimestamp must be later than the last configuration change.

gpsd did not start

After the reboot, NMEA and PPS stayed at Reach 0. On Debian 13, the gpsd package enables gpsd.socket but not gpsd.service: gpsd only starts when a client connects. chrony not being one, the shared memory stayed empty. Before the reboot, everything worked because gpsd had been started by hand. Fix: systemctl enable gpsd.service.

Another finding from that reboot: this board has no battery-backed clock. The Pi restarted with the time of its last shutdown, 1 h 36 behind, and chrony stepped it in one go. The GPS, on the other hand, stayed powered during this warm reboot, kept its UTC parameters, and the wrong second did not come back.

Result, and what is left to do

$ chronyc sources
#? NMEA        0   4   377    22   +182ms[ +182ms] +/-  200ms
#* PPS         0   4   377    23  +2670ns[+4937ns] +/-  213ns
$ chronyc tracking
Reference ID    : 50505300 (PPS)
Stratum         : 1

Still open:

• replace the UPS, which no longer holds the load, and keep a copy of the SD card;
• find out why the new GPS receiver does not come up on the new Pi (the subject of a future log), then return to the original plan: parallel running, comparison, switch-over.

Sources

• RFC 8915 — Network Time Security for NTP (port 4460, NTS-KE)
• chrony.conf documentation (4.6.1) — refclock (lock option), sourcedir, makestep directives
• GPSD Time Service HOWTO — gpsd, shared memory and chrony
• Internal — Our Stratum 1 server, our infrastructure, NTS on chrony