Building with snapcraft in docker

knocte · December 18, 2019, 4:01am

Hey, I wanted to try layouts, and I found some problems.

First, the docs in https://snapcraft.io/docs/snap-layouts say Layouts require snap version 2.36+., however, I’m using snapcraft version 2.43.1+18.4 (and snap version 2.42.1+18.04), and they still not work. I’ve had to google quite a bit to find that they are still marked as “experimental”, and need to be enabled via snap set core experimental.layouts=true, can this be added to the docs?

The next problem I have is that I cannot even use the above line to enable them, because that line uses the snap command instead of snapcraft command. Why not enable a flag to snapcraft command too? My workflow was to setup a CI lane to create my snap package, but the CI lane is based on gitlab-ci infrastructure, which under the hood uses docker, which conflicts with the snap command apparently (see https://stackoverflow.com/a/55804965/544947 ).

degville · December 18, 2019, 10:05am

Hello - welcome to the forum, and thanks for bringing this to our attention.

It looks like you’re using the standard deb package of snapcraft (2.43.1+18.04) rather than the latest version from the snapcraft snap (currently 3.8), which does support layouts without requiring the experimental flag.

This should be solved with sudo apt remove snapcraft and sudo snap install snapcraft --classic.

However, it’s our fault for not communicating this clearly - especially when Ubuntu 18.04 bundles snapcraft 2.43 by default (you’re the second person to bring this to our attention this week).

Sorry for the confusion. We’ll update the docs to try and make it clearer people need to migrate to the snap rather than using the deb that’s pre-installed.

knocte · December 18, 2019, 12:54pm

Thanks for the heads up and your willingness to update the docs.

However, as my second part of the message explains, your solution doesn’t help me

I mean, it’s already bad that I cannot test a snap package programatically because my CI infrastructure conflicts with snap’s under-the-hood infrastructure (https://stackoverflow.com/a/55804965/544947), now if I need to use snap to install snapcraft, I cannot use even ContinuousIntegration to build the package like I’ve been doing?

I guess what I’m asking is, can I obtain a newer version of snapcraft through apt instead? Is there a PPA? Or would adding 19.10 ubuntu packages of snapcraft be a good workaround?

lucyllewy · December 18, 2019, 2:14pm

I’ve not tried this yet, but maybe this project can help for CI scenarios where systemd cannot run.

ijohnson · December 18, 2019, 4:04pm

Note that if all you need is to run snapcraft as a snap inside a docker environment, there are examples of how to do this in other parts of the forum, such as https://snapcraft.io/docs/build-on-docker. This works because snapcraft is a classic snap and can just be “unpacked” on a system and used without the necessary setup that snapd does for strict snaps.

knocte · December 19, 2019, 2:27am

Thanks for your reply Johnson, but I’m afraid that doesn’t help me either. I’m using GitLabCI, not Docker, so I don’t have any control of docker from outside. It’s just that GitLabCI-runners spawn docker instances under the hood.

It’s very unfortunate that this renders all the snap work to be manual (creating the snap, and testing it), if using GitLabCI. Think about the consequences of this when you think that many opensource projects are switching to GitLab to run their infrastructure (Debian and GNOME already did).

ijohnson · December 19, 2019, 2:51am

Sure, if you don’t have control over what docker images are being run, as long as you have some access to the root filesystem of the container you’re running in, you can still use snapcraft as a snap to build your snap. You would just take the steps from the Dockerfile I linked to above in the docs, and instead of encoding them in a docker container, you would just run them manually as part of your build to “install” snapcraft as a snap inside your docker container.

knocte · December 19, 2019, 7:24am

Not sure what you really mean. Have you ever created yaml files for gitlab-CI?

ijohnson · December 19, 2019, 1:20pm

Sorry if I was unclear, I just meant something like this in your gitlab yaml:

test:
  script:
    - apt update
    - apt install curl jq squashfs-tools
    - curl -L $(curl -H 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/core' | jq '.download_url' -r) --output core.snap
    - mkdir -p /snap/core
    - unsquashfs -d /snap/core/current core.snap
    - curl -L $(curl -H 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable' | jq '.download_url' -r) --output snapcraft.snap
    - mkdir -p /snap/snapcraft
    - unsquashfs -d /snap/snapcraft/current snapcraft.snap
    - mkdir -p /snap/bin
...

using the steps from https://snapcraft.io/docs/t/creating-docker-images-for-snapcraft/11739 that is explained in the tutorial I linked above at https://snapcraft.io/docs/build-on-docker

knocte · December 20, 2019, 3:10am

Does that really help here? AFAIU a script like that would allow me to install snapcraft snap without snap, however I still need snap running to be able to run it! Which would yield the original problem I mentioned (https://stackoverflow.com/a/55804965/544947).

Anyway I have put it in gitlab-CI yaml script, and after it performs all the steps, I run /snap/bin/snapcraft --version and I’m getting this error: /usr/bin/python3: can't open file '/bin/snapcraft': [Errno 2] No such file or directory, which I guess is because $SNAP env var is empty (because of not running snapcraft inside snap?).

knocte · December 20, 2019, 2:54pm

I was forgetting to include these:

ENV PATH="/snap/bin:$PATH"
ENV SNAP="/snap/snapcraft/current"
ENV SNAP_NAME="snapcraft"
ENV SNAP_ARCH="amd64"

(Converting them to “export FOO=bar” lines of course.)

But now I’m getting two problems more:

“The following snaps are required but not installed as snapcraft is running inside docker or podman container: core18.”
A python exception:

Staging foo 
4150 Sorry, an error occurred in Snapcraft:
4151 'utf-8' codec can't encode character '\udcc5' in position 62: surrogates not allowed
4152 Traceback (most recent call last):
4153   File "/snap/snapcraft/current/bin/snapcraft", line 11, in <module>
4154     load_entry_point('snapcraft==3.8', 'console_scripts', 'snapcraft')()
4155   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 764, in __call__
4156     return self.main(*args, **kwargs)
4157   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 717, in main
4158     rv = self.invoke(ctx)
4159   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 1114, in invoke
4160     return Command.invoke(self, ctx)
4161   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 956, in invoke
4162     return ctx.invoke(self.callback, **ctx.params)
4163   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 555, in invoke
4164     return callback(*args, **kwargs)
4165   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
4166     return f(get_current_context(), *args, **kwargs)
4167   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/cli/_runner.py", line 103, in run
4168     snap_command.invoke(ctx)
4169   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/cli/_command.py", line 87, in invoke
4170     return super().invoke(ctx)
4171   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 956, in invoke
4172     return ctx.invoke(self.callback, **ctx.params)
4173   File "/snap/snapcraft/current/lib/python3.5/site-packages/click/core.py", line 555, in invoke
4174     return callback(*args, **kwargs)
4175   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/cli/lifecycle.py", line 261, in snap
4176     _execute(steps.PRIME, parts=[], pack_project=True, output=output, **kwargs)
4177   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/cli/lifecycle.py", line 66, in _execute
4178     lifecycle.execute(step, project_config, parts)
4179   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/lifecycle/_runner.py", line 94, in execute
4180     executor.run(step, part_names)
4181   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/lifecycle/_runner.py", line 148, in run
4182     self._handle_step(part_names, part, step, current_step, cli_config)
4183   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/lifecycle/_runner.py", line 162, in _handle_step
4184     getattr(self, "_run_{}".format(current_step.name))(part)
4185   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/lifecycle/_runner.py", line 226, in _run_stage
4186     self._run_step(step=steps.STAGE, part=part, progress="Staging")
4187   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/lifecycle/_runner.py", line 281, in _run_step
4188     getattr(part, step.name)()
4189   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 740, in stage
4190     self._do_runner_step(steps.STAGE)
4191   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 242, in _do_runner_step
4192     return getattr(self._runner, "{}".format(step.name))()
4193   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 84, in stage
4194     "override-stage", self._override_stage_scriptlet, self._stagedir
4195   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 137, in _run_scriptlet
4196     scriptlet_name, function_call.strip()
4197   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 193, in _handle_builtin_function
4198     function(**function_args)
4199   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 767, in _do_stage
4200     self.mark_stage_done(snap_files, snap_dirs)
4201   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 777, in mark_stage_done
4202     self._scriptlet_metadata[steps.STAGE],
4203   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 394, in mark_done
4204     f.write(yaml_utils.dump(state))
4205   File "/snap/snapcraft/current/lib/python3.5/site-packages/snapcraft/yaml_utils.py", line 38, in dump
4206     data, stream, _SafeOrderedDumper, default_flow_style=False, allow_unicode=True
4207   File "/snap/snapcraft/current/lib/python3.5/site-packages/yaml/__init__.py", line 200, in dump
4208     return dump_all([data], stream, Dumper=Dumper, **kwds)
4209   File "/snap/snapcraft/current/lib/python3.5/site-packages/yaml/__init__.py", line 188, in dump_all
4210     dumper.represent(data)
4211   File "/snap/snapcraft/current/lib/python3.5/site-packages/yaml/representer.py", line 27, in represent
4212     self.serialize(node)
4213   File "ext/_yaml.pyx", line 1348, in _yaml.CEmitter.serialize
4214   File "ext/_yaml.pyx", line 1510, in _yaml.CEmitter._serialize_node
4215   File "ext/_yaml.pyx", line 1509, in _yaml.CEmitter._serialize_node
4216   File "ext/_yaml.pyx", line 1428, in _yaml.CEmitter._serialize_node
4217 UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc5' in position 62: surrogates not allowed
4218 You can find the traceback in file '/tmp/tmpf6fg3bnh/trace.txt'.

ijohnson · January 6, 2020, 5:23pm

This message is expected and can be safely ignored AFAIK since you’re building inside a docker container.

The issue you have now is something else with UTF-8, can you please start a new topic in the snapcraft category with details on that, including your snapcraft.yaml?