We are currently testing out Ubuntu Core for a system we are developing. It runs on an industrial PC (Intel Atom CPU). One of the tests we are doing is repeated boot-ups: the PC gets powered on, we try to SSH in and see if everything is running, see how long it takes to full operational state, and then power it off again. The PCs are not power down gracefully but hard (which might be part of the problem, but is unavoidable). These devices don’t have a permanent internet connection so we test them “offline” (just local LAN).
What happens is that after maybe 100 cycles, the system clock and real time clock are messed up. The RTC jumps back to (I’m guessing the production date) 2012 and the system clock jumps back to the last time the device had a internet connection (some weeks ago) and, I guess, received something on the NTP service.
I tried deactivating and masking the NTP-service for a test. I read that in some Linux distributions the NTP service updates the RTC in regular intervals to make sure they stay in sync and I was wondering if something gets messed up, if just during such an update the device gets shut off. But it didn’t help. Now the system clock jumps back to some date beginning of 2018 (I’m guessing that might have something to do with the image using core18 and the compilation time of some component of it).
Has anybody experienced something similar and/or knows what might cause it? Does anybody know how the interaction between RTC, NTP and system clock works on Ubuntu Core? I’m far from being an expert in these matters, so any help or pointer what to check next would be greatly appreciated.
the Ubuntu Core images have an initramfs script that forces the clock to be set to the image build time (note it looks at the writable filesystem timestamp, not at the core18 snap or anything) if it is detected that the clock is too far off (which breaks gpg and thus makes seeding of snaps impossible). Many embedded devices do not come with an RTC on board at all which is why we picked this route for UbuntuCore.
if a formerly properly working RTC is all of a sudden off, this indicates that you powered down during a write …
NTP is automatically run by systemd-networkd and systemd-timesyncd during the connection bringp AFAIK we do not have any specific code for this beyond what systemd does by default here …
Could you please clarify, did you mean this could happen when powering down while writing to the /writable filesystem or while writing to the RTC? If it’s the later, which service/daemon might be doing this writing?
Btw. do you know how much the clock can be off before it makes problems for gpg?
the latter … not sure what exactly writes to it in the systemd world though … (it used to be the hwclock service in pre-systemd times but i think nowadays something like systemd-timesyncd might be in charge)
the latest code of the fixrtc script is at:
specifically for Ubuntu Core there is additionally:
the latter … not sure what exactly writes to it in the systemd world though … (it used to be the hwclock service in pre-systemd times but i think nowadays something like systemd-timesyncd might be in charge)
I masked systemd-timesyncd on that system to make sure it can’t interfere:
but that didn’t solve the problem. After that change the system clock just jumps back further (so it had some influence).
I did another test yesterday and after roughly 90 reboots I got:
date = 28th Jan 2018
RTC = 1st Jan 2012
It’s not jumping back to the creation date of /writable, since that would be 30th March 2020.
Thanks for the links, I will have a look if I can figure out, what is going on.
For my understanding, maybe someone can tell me if I’m right or not: It seems the kernel when it gets shut down writes the system time to the RTC. Is that correct? If yes, is there a way to deactivate that, so that the kernel only reads the RTC on bootup and then leaves it be and never writes to it? Would changing access permissions to the device help?
Because I’m running out of ideas, I tried do block the RTC from access after boot by having a service that has it permanently open (RTC can only be open by one process at a time if I understand correctly). But that didn’t help either.
did you consider that you might be actually hitting a BIOS bug ? did you check for any BIOS updates for the device yet ?
also, perhaps try your “100 reboots” test with a classic ubuntu server install too, in case it fails there as well (and you have ruled out the BIOS) there might actually be some systemd bug
Thanks for the Tip.
I tested the System with a classical Ubuntu (Ubuntu 18.04.4 LTS, Kernel Linux 5.3.0-53-generic) and in roughly 2800 bootcycles that problem never occurred. The clock drifted a bit (to be expected), but nothing more.
Checking the BIOS version is a bit more difficult at the moment, since I’m mostly not on side, but honestly, since it was fine with the normal Ubuntu, I would find it strange if there is an underlying BIOS problem.
Is there a difference in how Core deals with time management, RTC, ect. compared to the normal Ubuntu?
Hi Tobid, I know this was a log time ago, but do you have any more information about what the BIOS bug was, or what version of what BIOS fixed the problem? I feel like I have the same problem but I have a custom BIOS, so I need to figure out what specifically changed in your BIOS and try to make the same change, if applicable. Any info would be appreciated.
