[Solved] Why does my Orange Pi Zero get into endless config loop post update?

Problem occurs on fresh install of Ubuntu Core 16, essentially load SD card into device, turn it on, do nothing but wait until first forced update.

  • Ubuntu CORE 16 on Orange Pi Zero (armf).
  • Downloaded and registered.
  • Found that on fresh image, with no snaps installed, eventually the device drops its IP address and cannot be sshed into.
  • That might be the pushed updates? The system was warning a reboot was coming through but the device did not appear to come up again after the pushed refresh - except that the serial port persisted with original IP address.
  • If I hard reboot, the device recovers with a new IP address.
  • If I sudo reboot before the forced reboot the device reboots and retains the original IP address. So there is some subtle difference between the user soft reboot happening and the reboot from the refresh.
  • To top it off, this morning the serial port prompted for the config again. There was no prompt for ubuntu email address. It prompted for the IP setup.
  • I answered yes to all questions, and an IP was assigned, the device fell back into the config prompt. As often as I pushed through the prompts the device would be reassigned and IP and then fall back into the config prompt.
  • I have tried this on two OPiZ boards to confirm that the board was not flaky.
  • Ubuntu CORE 16 (armf) for Orange Pi Zero is not release ready.

Posted also on Ubuntu forum under ubuntu-core tag and on Orange Pi forum.

Does the orange pi use sdcard for it’s storage? Did you move the one sdcard to the second pi or did you use a different card? It could be that your sdcard has become readonly.

I’ve had that happen on a card I used in a raspberry pi - writes would claim to be written but upon reboot the data was back to the pre-written state.

OPIZ has micro SD card so no physical lock on it.

So unless you are suggesting that the software marked the SD card as read-only?

I have used a fresh burn of image on both OPiZ.

Still a problem then with the Ubuntu Core 16 image for OPiZ.

I find now even a hard reboot (power off/power on) does not clear the problem. I cannot otherwise get into the board. Ctrl-C/Z etc. do not break out of the config loop. I will need to re-burn the image. Not much point in reburning, since first update that comes through will frag the board.

Who in the end should this be reported to? OPiZ forum has not bothered answering my post. Ubuntu has pushed me here.

OPiZ community is notorious for not responding. I was actually surprised Ubuntu “teamed” up with them.

The problem is with the Ubuntu update mechanism and delays on the internet. There is what I call the doldrums which is southern hemisphere is blocked out of access to northern hemisphere servers when northern hemisphere is awake and users are throttling servers.

Whenever I try connecting to ubuntu core site with device at night time (southern hemisphere) I will often get a snap does not exist message or connection timeout with some message about missing headers. This is just round trip problem over the internet over distances and server loading.

This morning (saturday morning southern hemisphere and northern hemisphere asleep) I reburnt the Ubuntu CORE 16 Orange Pi onto SD, loaded it, started device, let device go through it’s obligatory first update and it came back up! That is it did not lock into config loop after update came through.

This indicates that the update, also prone to having to travel across planet, is sensitive to timeouts and frags the device that is updating.

Therefore, snap technology, binding to commercial entity in northern hemisphere appears unfriendly to southern hemisphere.

The only solution, since you cannot avoid the updates, is to set the updates to quiet time in northern hemisphere.

The problem is that this is crap since you cannot guarantee not fragging the device and having to re-burn SD card and re-build device.

Can’t recommend Ubuntu CORE and snap for any business critical applications on Orange Pi Zero and its associated CORE 16 image - at least in southern hemisphere.

To be precise, the SD card’s firmware makes itself readonly. Note that flash memory will wear-out by limited write cycles and most of them will lock down into readonly protect mode when such case occurrs.

Note that we have plenty of Ubuntu Core users in the southern hemisphere including commercial and non-commercial projects without any problems using any of the reference images (pi2/3 (including cm3) and dragonboard) as well as some commercial images with imx6 hardware.

none of them expose such a behaviour.

where does the orangepi image you are using come from exactly, do you have a pointer to the kernel snap or gadget snap in use ?

1 Like

#ogra

I have reported it.

So “none” is too strong my friend. It only then makes my reporting one occurance to act as a means to overturn your argument. It is a weak argument you offer since we are talking probabilistically both because it is software reliability of a community that offers software “as is” and leaves it to users to debug and also because its over network traffic and servers likely loaded to max during certain times of day.

Your use of “most” is closer but so what if it works sometimes? That leaves it open for not working other times. So we are back to probabilities since mine is a “sometimes something goes wrong” message.

There is only one orange pi zero image for CORE 16 my friend. From the orange pi site.

I am otherwise used to debugging real time distributed systems. Since I have simply downloaded the img from orange pi and done nothing with it but left it running over night then the problem comes with the image.

I have seen serious connection problems in southern hemisphere (at least from my home office) with connecting to snap store (and other sites including github). Those connection problems are clearly time of day related.

As the snap update would need be over the net and likely suffering timeouts etc, I don’t think it is a long stretch to get to the conclusion I have.

I have simply offered my observations.

The device can otherwise get into a fit where it falls back into a perpetual config loop likely caused by a broken update. Update over the internet.

If you have a better account at how the Ubuntu CORE 16 Orange Pi Zero image frags itself that way I do notice you have not offered it?

I manage many operational aspects of the snap store, and I live in Melbourne. Rest assured that we’d notice and be on it pretty quickly if there were issues accessing the store from the southern hemisphere.

If you have trouble accessing both the snap store and GitHub, it sounds like your Internet connection might have some problems. But I’ll be happy to investigate from our end if you can explain exactly what store problems you were seeing (including error messages) and when.

1 Like

#wgrant

Messages were either that the snap I was requesting didn’t exist or I got timeouts and connection attempts dropped because of missing headers. Since I had block copied the snap commands from snapcraft.io I was happy enough I hadn’t injected any typos.

Sequence of events:

  • Monday I burnt an SD with Ubuntu CORE 16 for OPiZ from orange pi site.
  • I followed the prompts to register and configure the ethernet.
  • I tried to add mosquitto from the snap store using the snap command. I actually blocked copied the command line from the store page and pasted it into the shell.
  • I tried probably a dozen times to run the command to install mosquitto, it eventually worked after a run of can’t find mosquitto, timeouts and missing header reports.
  • I then tried to snap in node-red. There is a whole other story about node-red for armhf but I was not able to get a connection to store Monday night. All I got again was either snap did not exist or timeouts and missing headers.
  • After dinner Tuesday I had no luck getting node-red installed.
  • After dinner Wednesday, after again getting no solid connection to store, I gave up and re-burnt the SD with Ubuntu CORE 16 for OPiZ.
  • I started the device, registered, left it running over night. Before I went to bed I ssh’d into device so it was up.
  • Thursday morning, with morning coffee in hand, the OPiZ was in tight ethernet configuration prompt loop on the serial port and no device seen on the LAN.

Now I don’t mind hearing my network connection is flaky (I am running device off a wifi extender with an ethernet cable). But not sure that excuses the fragging of the board.

Someone who knows the CORE should be able to fathom what would force the board back into the ethernet config prompt again. It wasn’t asking for my login email, it would just ask for a key to start configuration, accept a key, grab an IP address and then fall back into asking for a key to start configuration.

Note in amongst the failed attempts to snap in node-red I did remove mosquitto and after a few attempts got mosquitto snapped back in from the store. I did also ping 8.8.8.8 to find device was still talking to outside world etc. From that I got as node-red is much bigger than mosquitto there were more opportunities to lose connection.

Building an image is trivial with Ubuntu Core (doing it from scratch probably takes an afternoon for an experienced person and a few days (with a bit of learning) for someone who has never done it), i have about ten different experimental images on my disk for this device. You did not provide any information if you built it yourself or where you downloaded it. how would anyone know what you are using over there without giving us any details.

… have we ever met that you call me “my friend” ?

nobody doubts that, but since you are “used to debugging real time distributed systems” you will also know that debugging needs information that you have not bothered to give us … syslog, dmesg, the output of snap list, snap version and snap changes would be helpful …

instead you are philosophing about hemispheres (with no further proof) and mark your case solved with foot stamping and blaming the store…

… and theories about them without any proof or data …

That said and despite your tone … i now took a look at the image you pointed to above and can tell you a bunch of observations without even booting it …

kernel and gadget snap are manually side-loaded from some locally built snaps not in the store at all … circumventing the actual ubuntu-core build process (which would be: using only published snaps from the store, which does also mean they would be quality and security checked by the store systems) … this is an image someone glued together using stings and duct tape on his home desktop … (neither gadget nor kernel would ever update on this image, leaving you with a non-secure kernel permanently)

the kernel snap included looks like someone used my experimental nanopi kernel snap as base for it. that has never been released as stable snap and is based off an old experimental 4.11 pre-release tree:

$ snap info linux-generic-allwinner
name:      linux-generic-allwinner
summary:   The generic armhf kernel built for allwinner boards
publisher: Oliver Grawert (ogra)
contact:   ogra@ubuntu.com
license:   GPL-2.0
description: |
  The Ubuntu linux-generic kernel package as a snap
type:    kernel
snap-id: vvs4L387pDB9dvc9emQgIICUvLRfCObi
channels:                            
  stable:    –                       
  candidate: 4.11.0-13.19 (16) 133MB -
  beta:      4.11.0-13.19 (16) 133MB -
  edge:      4.11.0-13.19 (16) 133MB -

given i never built this snap with orangepi in mind it can well bee that the network card drivers have plenty of issues (you never originally bothered to tell us if you are using wlan or wired, so it is very hard to tell but could very likely be the root of your issue)

1 Like

#ogra

You do know that if the device is caught in an endless config loop on the serial port and you cannot break out of that into terminal on the serial port and you cannot ssh into the device via the LAN then you cannot get to logs et al.

There might have been a forensic thing I could do with the SD card perhaps. Too late now I have burnt on a Armbian image.

The device was otherwise bricked, essentially.

So, your factoids are “interesting” my friend but irrelevant, so, in that vein did you know peanuts are used in the production of dynamite?