A second GNSS Stratum 1 with Galileo: our log

Since 27 October 2026, we have two Stratum 1 servers side by side: ntp-0 (renamed ntp-0-gps on the 27th), the GPS Raspberry Pi rebuilt on 23 October, and ntp-0-gnss, a second Raspberry Pi with a u-blox receiver that picks up GPS, GLONASS and Galileo, the European system. The receiver meant for it had not worked on the 23rd; correctly refitted on the 27th, it answered first time.

As for the rebuild, this page follows the log kept during the work: commands, outputs and measurements, times in UTC. GPS coordinates, internal addresses and secrets are removed.

The hardware

The two machines only share the Raspberry Pi model and the network: receivers from different makers and generations, separate antennas. A receiver or antenna failure on one does not affect the other; they do share the rack, the power supply, the network and the ISP box.

First readings: I²C, serial, PPS

Both I²C lines idle high, and the bus answers instantly with its two devices: the u-blox receiver at 0x42 and the RV-3028 RTC at 0x52.

$ pinctrl get 3,15,18
 3: a0    -- | hi // GPIO3 = SCL1
15: a0    -- | hi // GPIO15 = RXD0
18: ip    -- | lo // GPIO18 = input
$ i2cdetect -y 1
40: -- -- 42 -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- 52 -- -- -- -- -- -- -- -- -- -- -- -- --

At 9600 baud, the speed of ntp-0's L80, the serial line only gives garbage: this module talks at 115200 baud. At the right speed, the sentences come in, with an SBAS-corrected 3D fix on 12 satellites (GPS and GLONASS) and a horizontal dilution of precision of 0.90. The PPS comes in too, one pulse per second, read by the kernel driver:

$ ppstest /dev/pps0
source 0 - assert 1793115109.000209308, sequence: 304
source 0 - assert 1793115110.000210946, sequence: 305
source 0 - assert 1793115111.000211714, sequence: 306

The firmware, read with ubxtool:

$ ubxtool -f /dev/serial0 -s 115200 -p MON-VER
  swVersion ROM CORE 3.01 (107888)
  extension FWVER=SPG 3.01
  extension PROTVER=18.00
  extension GPS;GLO;GAL;BDS

ROM CORE: the firmware is in read-only memory, and the module has no flash memory. Its configuration only survives a power cut in the memory kept alive by the HAT's two backup batteries. That is what will matter for Galileo.

Galileo

Galileo is the European Union's satellite navigation system. Received on top of GPS and GLONASS, it brings more satellites, hence a more robust fix, and above all a time that no longer depends on a single system: if one degrades, the receiver keeps the others. The M8 receives up to three at once. Out of the box, ours received GPS, GLONASS, SBAS and QZSS, but not Galileo:

$ ubxtool -P 18 -p CFG-GNSS
  gnssId 0 ... GPS L1C/A enabled
  gnssId 2 ... Galileo E1
  gnssId 6 ... GLONASS L1 enabled
$ ubxtool -P 18 -e GALILEO
UBX-ACK-ACK:
  ACK to Class x06 (CFG) ID x3e (GNSS)
$ ubxtool -P 18 -p CFG-GNSS
  gnssId 2 ... Galileo E1 enabled

The module accepts the setting. But when the Pi was moved a few hours later, a power cut erased it: Galileo was disabled again, and the receiver saw no Galileo satellite. The setting had not been saved to the backed-up memory. Rather than depend on it, a small service, gnss-constellations, re-enables Galileo at every boot if it is off, and retries if the receiver is not ready yet. Checked twice, after a reboot then after a power cut: Galileo re-enabled, and 5 Galileo satellites used in the solution.

chrony: the same design as ntp-0

The configuration follows ntp-0's, described in the rebuild log: PPS numbered by NMEA (lock NMEA), held as noselect at startup by a logical interlock until three NTS sources have confirmed the NMEA second, and our fleet servers seen at stratum 8 at least. ntp-0 and ntp-0-gnss query each other directly on the local network: each is a witness for the other.

# /etc/chrony/conf.d/refclock.conf
refclock SHM 0 refid NMEA noselect poll 4 offset 0.05 precision 1e-1 delay 0.2
refclock PPS /dev/pps0 lock NMEA refid GNSS poll 4 noselect

The NMEA sentence lags the second it announces by about 56 ms at 115200 baud, against 160 ms for the L80 at 9600 baud: offset 0.05 compensates for it. Once the interlock released, around 15:50:

$ chronyc sources
#? NMEA        0   4   377    17    -45ms[  -45ms] +/-  200ms
#* GNSS        0   4   377    16  +4554ns[ +225us] +/-  572ns

The NMEA at −45 ms is from the first configuration, with offset 0.1: this measurement set the value of 0.05. After correction, it is within a few milliseconds.

The move, a power cut

To join ntp-0 in the rack, the Pi was shut down, unplugged, moved and plugged back in. Three findings:

• The PPS came back a few seconds after power returned. The HAT's backup batteries kept the navigation data, and the receiver made a hot start. ntp-0, with its L80, takes about a minute.
• Galileo was lost, as described above.
• The Pi's clock restarted 2341 s behind, then was set from the network. The RTC was not read at boot (see below).

The logical interlock did its job: during those minutes, the Pi followed ntp-0, at Stratum 2, without ever serving the PPS before confirmation.

The RTC: present, but not backed up

The HAT carries a battery-backed clock, an RV-3028, and the driver sees it once dtoverlay=i2c-rtc,rv3028 is added to config.txt: /dev/rtc0 appears, and rtcsync, already in Debian's chrony configuration, keeps it on time. Yet at every boot the Pi restarted with a wrong time: 889 s behind the first time, 2341 s after the move, 7708 s after a one-minute power cut, each time caught up by chrony in a single step.

A false lead first

At boot, the kernel wrote:

kernel: rtc-rv3028 1-0052: hctosys: unable to read the hardware clock

We read it as an ordering problem: the driver, loaded by the overlay, would come too late for the kernel to read the RTC. So we added a unit running hwclock --hctosys before chrony (on Debian 13, hwclock is in the util-linux-extra package). At the next boot, it waited 90 s for /dev/rtc0 then gave up, delaying chrony and gpsd by as much: udev does not report RTCs to systemd. A one-line rule removed the wait:

# /etc/udev/rules.d/70-rtc-systemd.rules
SUBSYSTEM=="rtc", KERNEL=="rtc0", TAG+="systemd"

But hwclock then failed too, like the kernel: the RTC itself refused to be read.

hwclock: ioctl(RTC_RD_NAME) to /dev/rtc0 to read the time failed: Invalid argument

The real cause: switchover to battery, disabled

Read through the kernel interface (ioctl RTC_PARAM_GET, read-only), the RV-3028's switchover mode was the factory one:

BACKUP_SWITCH_MODE 0      → RTC_BSM_DISABLED

The RTC therefore never used the HAT's batteries. Without power, it loses its time and raises its power-loss flag, and the driver refuses to read it until it has been set again. A few minutes after each boot, rtcsync rewrote it and it became readable again: that is what made the fault so misleading. The fix is one overlay parameter, asking for "level switching" mode:

# /boot/firmware/config.txt
dtoverlay=i2c-rtc,rv3028,backup-switchover-mode=3

At the next boot, the driver reports mode 2, and the kernel reads the RTC itself as soon as it registers it, with no unit at all; chrony starts with no offset to catch up:

The RTC parameter 0x2 is set to 0x2.
kernel: rtc-rv3028 1-0052: setting system clock to 2026-10-27T17:59:20 UTC

The message at the start therefore did not come from a driver loaded too late: the kernel was refusing an invalid RTC. The hwclock unit has become redundant; it does no harm. Two precautions: do not enable the RTC's charger with batteries that are not rechargeable, and do not mix up the numbering, since the same mode is 3 for the overlay, which follows the chip's bits, and 2 for the kernel (RTC_BSM_LEVEL).

That reboot had not cut the power. The proof came from a real power cut of about one minute: the RTC held on its batteries, the kernel read it about 12 s after boot, and chrony had no offset to catch up. The PPS came back in 4 s.

# after the power cut (level switching)
kernel: rtc-rv3028 1-0052: setting system clock to 2026-10-27T18:04:40 UTC
$ chronyc tracking
System time     : 0.000000000 seconds slow of NTP time

Seen from outside

At the first test, ntp-0-gnss was only reachable from outside over IPv6: the site's single public IPv4 address still served ntp-0-gps. From one of our servers in Paris, an ordinary NTP query gets the answer of a Stratum 1 with refid GNSS, and chronyd -Q measures a 0.18 ms offset. By the same evening, NTS was live: a certificate shared by both machines, issued on the 27th, carries their public and local names, and authenticated time is obtained through the name ntp-0-gnss-hq.rdem-systems.com, within 0.15 ms from Paris:

$ chronyd -Q -6 'server ntp-0-gnss-hq.rdem-systems.com iburst nts'
System clock wrong by -0.000145 seconds (ignored)

In the fleet's sources

Since the evening of 27 October, our fleet's Stratum 2 servers query ntp-0-gnss next to ntp-0-gps, over NTS, with sessions established to both machines. The same evening, the site's single public IPv4 address, that of ntp-0-hq, was redirected to ntp-0-gnss. On one of them, in Paris, it was already the selected source:

$ chronyc -n sources
^* 2a01:e0a:808:97c4::123        1  10   317   805   +290us[ +398us] +/- 1970us
^- 2a01:e0a:808:97c4::123:123    1  10   377   164   +441us[ +441us] +/- 2015us

::123 is ntp-0-gnss, ::123:123 ntp-0-gps: both answer at Stratum 1. On most of our servers, the GNSS is the preferred source and the GPS a witness; on the others, chrony picks between them from its measurements alone. As of today, nothing remains open on this machine.

Sources

• Galileo — European Union Agency for the Space Programme (EUSPA)
• ubxtool — u-blox receiver configuration examples (gpsd)
• chrony.conf documentation (4.6.1) — refclock directive (lock, offset options), stratumweight
• Internal — the rebuild of ntp-0, our Stratum 1 servers, our infrastructure