I’m getting a lot of emails today about failed scans in the store for my automated builds:
Launchpad uploaded this snap package to the store, but the store failed to
scan it:
Waiting for previous upload(s) to complete their review process. If you want to prioritize this last one, go to the other upload(s) page in https://dashboard.snapcraft.io/ and click on the ‘Reject and remove from review queue’ button.
This doesn’t actually sound like a failure, it sounds like the review will happen as soon as the other one is done. But I’m getting spammed about it, so I’m wondering if something is actually wrong. Do I need to do something about this?
Can you tell me more about the snap in question? does it actually need manual review or was the “held” upload a bad one?
You could do two things:
1- Follow the instructions in that report you quoted (reject/remove). This will move processing to the next uploaded snap. But this won’t help if the next upload has the same characteristics that made the first one fail automated review (say, it uses a restricted interface or is classic-confined).
2- Poke reviewers and have the snap reviewed If it uses a restricted interface and its use is approved for this upload, it will automatically pass for the subsequent ones.
I’m sorry about the spam, though that sounds a consequence of automated builds pushing to a snap for which there are pending uploads. Maybe we could be silent after the first “held waiting for another review” message, but I think that’d give the false impression that they are being processed, which is not the case.
In fact, it looks like the automated reviews have now passed, but they didn’t go into the channels I requested. So what exactly happened? I got no emails about any sort of failure, just several emails containing the exact text I quoted. I have no more details.
Were the uploads made in rapid succession? (if using Launchpad maybe you’re building for various architectures and they were all uploaded quickly, while at least one of them was still waiting for the automated review). This would explain why they got “queued”.
As for the requested channels not being honored - not sure how this works on Launchpad; using snapcraft, “snapcraft push --release” does the two operations (push, release) internally, but will only do the release if the push succeeded and the package is in a “ready to release” state. If the package is held for review, the store immediately returns with the message you saw (so snapcraft doesn’t stall endlessly waiting for review as it did before). So the “intent” of the channels you wanted is lost. This was filed as https://bugs.launchpad.net/snapstore/+bug/1684529. I suspect Launchpad may be doing something similar behind the scenes and is susceptible to the same behavior.
I’m not giving any solutions here (yet), just gathering information and explaining how things work right now so we can better identify if anything needs fixing and if so, how to go about it.
Yes, multiple architectures and multiple tracks. If we can only scan one at a time, being queued as a result makes total sense. This seems pretty typical though-- not releasing to the channels I requested and actually emailing me an error just because I’ve built up a queue seems like a broken experience to me. It seems that anyone publishing revisions via CI is going to hit this. Such a thing is supposed to be automated, and this requires a person to be in the loop and figure out what revision is supposed to go where once the store makes it through its queue.
I agree it’s not ideal but the bug I quoted has a couple of scenarios showing why it’d be undesirable to release post-facto if an upload was held waiting for automated review. I realize it’s not great but for now manual intervention will be needed to get each snap into its channel
One more question - is this the first time this happens to you? I imagine nextcloud builds are frequent, yet I don’t recall any mention of this issue happening before to nextcloud.
Thanks for putting up with all my questions, we’ll find a good solution for this
The bug talks about being put into the manual review queue-- not automated. Whether that’s because of classic or because the automated review failed. That’s different than simply sitting in line for an automated review that passes in the end. In the case of requiring manual review, I agree that releasing once approved is probably undesired. But in the case of simply queuing automated reviews (which pass), I think it is.
No I get these pretty often, I just try to keep my whining to a certain level .
This is a great point. In principle, yes, we can see if any preceding upload is pending automated review (unscanned, I think is what the state would be) and not bail, but send the “still processing” response. It can still create a crappy experience if snapcraft spins while a potentially large upload queue is processed, and I’ll need to check what Launchpad would do in this case. But sounds like a possible approach to a solution.
Launchpad works more or less the same way as @roadmr describes snapcraft behaving. The implementation details are a bit different for various reasons, but it’s the same idea, and in particular Launchpad responds to a “still processing” response by backing off and trying again later (currently 20 retries at intervals of 15/15/30/30/60/60/60/… seconds; we can adjust this if need be). And yes, I agree that this proposed approach is exactly what we need.
To expand: the basic issue here is that Launchpad is waiting for the store to process the pushed snap before it can do the equivalent of snapcraft release. The store doesn’t notify Launchpad of this; Launchpad has to poll. If manual review of the first thing in the queue takes (say) two days, it isn’t reasonable for Launchpad to have to sit there polling for two days before it can eventually release it.
So, we need the store to return distinct responses in the following two cases:
The upload is queued behind something that’s been identified as requiring manual review
The upload is queued behind something that just hasn’t been processed yet
In the former case, we need to treat that like any other kind of manual review (i.e. the user will have to do a separate release step once the review has been completed). In the latter case, the program doing the upload (whether that be snapcraft or Launchpad) should try again later.
Hello, so the above bug is now “fix released”, and as long as all the uploads in the queue ahead of us are still “pending review”, the store will now send a “processing” response. Known clients (snapcraft/launchpad) then wait and re-poll later, as Colin described. The store will only send the “stop processing” reply with the same message you’d been seeing when one of the uploads gets held for manual review.
Let me know how this works for you! for Launchpad this should now “just work” (modulo actual manual review needs), for Snapcraft you could do something like doing “snapcraft push” for two revisions of the same snap (which you know will pass automated review) simultaneously. Previously, one of them would explode with the message you mentioned. If you do this now, one will wait while the other processes, then the second one will get processed too.
for the last few weeks i had at least one of the daily core snap builds (6 arches) fail to upload every day, today all 6 got through for the first time in a while, good work
If it uses a restricted interface and its use is approved for this upload, it will automatically pass for the subsequent ones.
.