Error downloading snap [stage-snaps on remote build]

snapcraft
1

Hi,

For the past week there’s been an issue while remote building the snap on the following architectures;

  • arm64
  • amd64

Failing with the following:

Downloading snap 'yq'
Downloading snap 'jq'
[05/Oct/2021:12:40:35 +0000] "CONNECT canonical-bos01.cdn.snapcraftcontent.com:443 HTTP/1.1" 200 3822594 "-" "snapd/2.51.7 (series 16; classic) snap ubuntu/20.04 (arm64) linux/4.15.0-159-generic"
[05/Oct/2021:12:40:35 +0000] "CONNECT api.snapcraft.io:443 HTTP/1.1" 200 4593 "-" "snapd/2.51.7 (series 16; classic) snap ubuntu/20.04 (arm64) linux/4.15.0-159-generic"
[05/Oct/2021:12:40:35 +0000] "CONNECT api.snapcraft.io:443 HTTP/1.1" 200 13210 "-" "snapd/2.51.7 (series 16; classic) snap ubuntu/20.04 (arm64) linux/4.15.0-159-generic"
[05/Oct/2021:12:40:36 +0000] "CONNECT canonical-lgw01.cdn.snapcraftcontent.com:443 HTTP/1.1" 403 1952 "-" "snapd/2.51.7 (series 16; classic) snap ubuntu/20.04 (arm64) linux/4.15.0-159-generic"
error: Get https://canonical-lgw01.cdn.snapcraftcontent.com/download-origin/canonical-lgw01/8X4ytHZ2xX4kNkr8V2NU3AQuoMlglwED_8.snap?token=1633449600_1012cd336c446516a1608a05753f621691df1705: Forbidden
Error while downloading snap 'jq' from channel 'latest/stable'
[05/Oct/2021:12:40:36 +0000] "CONNECT api.snapcraft.io:443 HTTP/1.1" 200 5058 "-" "snapd/2.51.7 (series 16; classic) snap ubuntu/20.04 (arm64) linux/4.15.0-159-generic"
[05/Oct/2021:12:40:36 +0000] "CONNECT api.snapcraft.io:443 HTTP/1.1" 200 4584 "-" "snapd/2.51.7 (series 16; classic) snap ubuntu/20.04 (arm64) linux/4.15.0-159-generic"
Build failed

They are both snaps that is staged (stage-snaps) in my snap. There’s been no issues regarding this when building before.

I’ve checked on both systems that those snaps exists in the snap store. It also says forbidden, so I guess there’s an issue somewhere else.

If I build locally on my machine (amd 64) it builds successfully.

Anyone knows what can cause this?

2

I think there is a magic environment variable you can set to force it to a certain CDN but I can’t find it right now, perhaps @cjwatson can chime in here …

1 Like
3

Thanks.

I also see this

UserWarning: Attempted to set unknown attribute "location" on instance of "Container"

several places in the logs, but that seems to be something that I can ignore.

4

SNAPPY_STORE_NO_CDN=1. This is https://bugs.launchpad.net/launchpad-buildd/+bug/1945712.

1 Like
5

how does one tell the launchpad builders to do that? it looks to me that @joachimmg is using snapcraft remote-build. Also, this is specifically for stage-snaps where we have no control over what snapcraft sets for the environment when calling snap download or equivalent.

6

Hi,

Yeah. My first though was that I could override-stage, but then I realized the packages is pulled before the stage-stage. Then tried to override-pull, but the packages is pulled before this stage too. Now trying to set it in the build-environment-part.

My next try will be to create a nil part and set this environment-variable, that the part with the snaps depends on (using: after).

7

If you look at the bug report you’ll see a linked merge proposal where I’ve proposed a fix for launchpad-buildd to do this automatically.

8

i would expect build-environment: to work for it … (curious to hear if it does though)

9 
     plugin: nil
     build-environment:
       - SNAPPY_STORE_NO_CDN: '1'
     stage-snaps:
       - jq
       - yq

Did not work.

10

ah, too bad, I guess because it applies to the build step but not to pull … I had hoped (despite the name of the snapcraft.yaml option) it would set for all steps …

11

Yeah, I’m doing a additional test now with:

  pre-dependencies:
    plugin: nil
    build-environment:
      - SNAPPY_STORE_NO_CDN: "1"
  dependencies:
    after: [pre-dependencies]
    plugin: nil
    stage-snaps:
      - jq
      - yq

Hoping that the build-environment from the part «pre-dependencies» isn’t cleared before the pull stage of «dependencies», since the snaps is pulled before this step.

12

No luck on this either; but after x trial and errors, there have eventually been a successful build for each architecture.

13

Hi @cjwatson ,

I saw that the fix now is merged few days ago. I’m just wondering if it’s live now or when we can expect it to become live?

I’m still having the CDN-issue and wanted to ask here before I file a bug, in case the fix is live already.

Hope it’s ok to ask here.

EDIT:

Screenshot from 2021-10-11 14-42-21
Screenshot from 2021-10-11 14-42-21365×544 26.6 KB

Found out that if I clicked the «Fix commited» label it would give me a description, and it’s not released yet :slight_smile:

14

It’s not deployed yet, no. I have a ticket in with our sysadmins to deploy it, and I’ll change the bug status to “Fix Released” (and hopefully also remember to post here) once that’s done.

15

This should now be fixed. Let us know if not!

16

@cjwatson it’s not fixed for me, assuming I’m dealing with the same problem. See build logs for the ‘node’ snap, which I’ve just retried right now: https://launchpad.net/~openjs/+snap/node12

Downloading core
[13/Oct/2021:01:47:46 +0000] "CONNECT canonical-lgw01.cdn.snapcraftcontent.com:443 HTTP/1.0" 403 1762 "-" "-"
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',))': /download-origin/canonical-lgw01/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_11743.snap?token=1634101200_a4e3aac806ada657281e39cb7c4b4511d508f8a2
[13/Oct/2021:01:47:47 +0000] "CONNECT canonical-lgw01.cdn.snapcraftcontent.com:443 HTTP/1.0" 403 1762 "-" "-"
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',))': /download-origin/canonical-lgw01/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_11743.snap?token=1634101200_a4e3aac806ada657281e39cb7c4b4511d508f8a2
...
17

The download is not done over snap download but using the Snap Store APIs themselves. There were some issues at the time, among those, I believe snap download was not machine callable friendly. We may be able to now communicate with the snapd socket since we use that for the snap injection logic.

1 Like
18

Failing that, would you consider a patch to honour SNAPPY_STORE_NO_CDN=1 in snapcraft's environment and turn it into an HTTP header in the same way that snapd does?