Dirty snap release?

xet7 · August 28, 2019, 10:56pm

Wekan snapcraft.yaml is here:

wekan/wekan/blob/master/snapcraft.yaml

name: wekan
version: 0
version-script: git describe --tags |  cut -c 2-
summary: The open-source kanban
description: |
   Wekan is an open-source and collaborative kanban board application.

   Whether you’re maintaining a personal todo list, planning your holidays with some friends, or working in a team on your next revolutionary idea, Kanban boards are an unbeatable tool to keep your things organized. They give you a visual overview of the current state of your project, and make you productive by allowing you to focus on the few items that matter the most.
   Depending on target environment, some configuration settings might need to be adjusted.
   For full list of configuration options call:
   $ wekan.help

confinement: strict
grade: stable

architectures:
  - amd64

plugs:
  mongodb-plug:

This file has been truncated. show original

It has this code:
version-script: git describe --dirty --tags | cut -c 2-

Previously when I built releases, it had generated versions like:
3.24
3.25-15-ga97360d

Now when it does not find any errors when building, it generates versions like:
3.29-dirty

I presume I can not fix this just by going to shower more often.

Any other ways to fix this?

xet7 · August 28, 2019, 11:07pm

Hmm, maybe this is why:
https://stackoverflow.com/questions/16035240/why-is-git-describe-dirty-adding-a-dirty-suffix-when-describing-a-clean-ch

xet7 · August 28, 2019, 11:12pm

xet7 · August 29, 2019, 12:09pm

Oops, it did not work.
Determining the version from the project repo (version-script).
The version-script failed to run (exit code 1)

https://launchpadlibrarian.net/439398441/buildlog_snap_ubuntu_xenial_amd64_wekan-devel_BUILDING.txt.gz

popey · August 29, 2019, 12:20pm

Couple of options, perhaps you’re running git in the wrong place. Maybe you need to:-

version-script: git describe --dirty --tags -C parts/wekan | cut -c 2- to coerce it to look in parts/wekan.

Alternatively, remove version and version-script and use adopt-info: wekan then in the override-build section of the wekan part, at the end:

snapcraftctl set-version $(git describe --dirty --tags | cut -c 2-)

No need for -C there as you’ll be in the right place.

lucyllewy · August 29, 2019, 12:26pm

It’s worth mentioning that adopt-into combined with snapcraftctl set-version is the way to do this going forward, because version-script is deprecated and will at the least raise a warning when building.

xet7 · August 29, 2019, 12:52pm

Running pull phase…
Failed to load plugin: properties failed to load for wekan: Additional properties are not allowed (‘adopt-info’ was unexpected)

https://launchpadlibrarian.net/439447515/buildlog_snap_ubuntu_xenial_amd64_wekan-devel_BUILDING.txt.gz

github.com

wekan/wekan/blob/master/snapcraft.yaml#L223


name: wekan
version: 0
version-script: git describe --tags |  cut -c 2-
summary: The open-source kanban
description: |
   Wekan is an open-source and collaborative kanban board application.
   Whether you’re maintaining a personal todo list, planning your holidays with some friends, or working in a team on your next revolutionary idea, Kanban boards are an unbeatable tool to keep your things organized. They give you a visual overview of the current state of your project, and make you productive by allowing you to focus on the few items that matter the most.
   Depending on target environment, some configuration settings might need to be adjusted.
   For full list of configuration options call:
   $ wekan.help
confinement: strict
grade: stable
architectures:
  - amd64
plugs:
  mongodb-plug:
    interface: content
    target: $SNAP_DATA/shared
hooks:
  configure:
    plugs:
      - network
      - network-bind
slots:
  mongodb-slot:
    interface: content
    write:
      - $SNAP_DATA/share
apps:
    wekan:
        command: wekan-control
        daemon: simple
        plugs: [network, network-bind]
    mongodb:
        command: mongodb-control
        daemon: simple
        plugs: [network, network-bind]
    caddy:
        command: caddy-control
        daemon: simple
        plugs: [network, network-bind]
    help:
        command: wekan-help
    database-backup:
        command: mongodb-backup
        plugs: [network, network-bind]
    database-list-backups:
        command: ls -al $SNAP_COMMON/db-backups/
    database-restore:
        command: mongodb-restore
        plugs: [network, network-bind]
parts:
    mongodb:
        source: https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1604-3.2.22.tgz
        plugin: dump
        stage-packages: [libssl1.0.0]
        filesets:
            mongo:
                - usr
                - bin
                - lib
        stage:
            - $mongo
        prime:
            - $mongo
    wekan:
        source: .
        plugin: nodejs
        node-engine: 12.22.1
        node-packages:
            - node-gyp
            - node-pre-gyp
            - fibers
        build-packages:
            - ca-certificates
            - apt-utils
            - python
#            - python3
            - g++
            - capnproto
            - curl
            - execstack
            - nodejs
            - npm
#            - p7zip-full
        stage-packages:
            - libfontconfig1
        override-build: |
            echo "Cleaning environment first"
            rm -rf ~/.meteor ~/.npm /usr/local/lib/node_modules
            rm -rf .build
            echo "Installing meteor"
            curl https://install.meteor.com/ -o install_meteor.sh
            chmod +x install_meteor.sh
            sh install_meteor.sh
            rm install_meteor.sh
            rm -rf .build
            chmod u+w *.json
            npm install
            meteor build .build --directory --allow-superuser
            # Remove legacy webbroser bundle, so that Wekan works also at Android Firefox, iOS Safari, etc.
            rm -rf .build/bundle/programs/web.browser.legacy
            # Change to directory .build/bundle/programs/server
            cd .build/bundle/programs/server
            chmod u+w *.json
            npm install
            cd ../../../..
            # Cleanup
            cd .build/bundle
            find . -type d -name '*-garbage*' | xargs rm -rf
            find . -name '*phantom*' | xargs rm -rf
            find . -name '.*.swp' | xargs rm -f
            find . -name '*.swp' | xargs rm -f
            cd ../..
            # Add fibers multi arch
            #cd .build/bundle/programs/server/node_modules/fibers/bin
            #curl https://releases.wekan.team/fibers-multi.7z -o fibers-multi.7z
            #7z x fibers-multi.7z
            #rm fibers-multi.7z
            #cd ../../../../../../..
            # Copy to Snap
            cp -r .build/bundle/* $SNAPCRAFT_PART_INSTALL/
            cp .build/bundle/.node_version.txt $SNAPCRAFT_PART_INSTALL/
            rm -f $SNAPCRAFT_PART_INSTALL/lib/node_modules/wekan
            #rm -f $SNAPCRAFT_PART_INSTALL/programs/server/npm/node_modules/meteor/rajit_bootstrap3-datepicker/lib/bootstrap-datepicker/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs
            # Delete phantomjs that is in accounts-lockout
            #rm -rf $SNAPCRAFT_PART_INSTALL/programs/server/npm/node_modules/meteor/lucasantoniassi_accounts-lockout/node_modules/phantomjs-prebuilt
            # Delete temporary files
            #rm -f $SNAPCRAFT_PART_INSTALL/programs/server/npm/node_modules/tar/lib/.mkdir.js.swp
            #rm -f $SNAPCRAFT_PART_INSTALL/lib/node_modules/node-pre-gyp/node_modules/tar/lib/.mkdir.js.swp
            #rm -f $SNAPCRAFT_PART_INSTALL/lib/node_modules/node-gyp/node_modules/tar/lib/.mkdir.js.swp
            # Meteor 1.8.x additional .swp remove
            #rm -f $SNAPCRAFT_PART_INSTALL/programs/server/node_modules/node-pre-gyp/node_modules/tar/lib/.mkdir.js.swp
            # Delete fibers for other archs
            #rm -rf $SNAPCRAFT_PART_INSTALL/programs/server/node_modules/fibers/bin/linux-ia32*
            # ostrio tmp remove
            #rm -rf $SNAPCRAFT_PART_INSTALL/programs/server/npm/node_modules/meteor/ostrio_files/node_modules/request-libcurl/.node_modules-garbage*
        organize:
            README: README.wekan
        prime:
            - -lib/node_modules/node-pre-gyp/node_modules/tar/lib/.unpack.js.swp
    helpers:
        source: snap-src
        plugin: dump
    caddy:
        plugin: dump
        ## Caddy v1 is not developed anymore. TODO: Sometime migrate to Caddy v2.
        ## https://caddy.community/t/caddyfile-v1-adapter/9129
        ## https://github.com/caddyserver/caddy/tree/v1
        #source: https://caddyserver.com/download/linux/amd64?license=personal&telemetry=off
        #source-type: tar
        # Using last working binary that was downloaded from above URL to Wekan Snap,
        # and .txt files from https://github.com/caddyserver/caddy/tree/v1/dist
        source: https://releases.wekan.team/caddy/caddy-v1-linux-amd64.7z
        source-type: 7z
        organize:
          caddy: bin/caddy
          CHANGES.txt: CADDY_CHANGES.txt
          EULA.txt: CADDY_EULA.txt
          LICENSES.txt: CADDY_LICENSES.txt
          README.txt: CADDY_README.txt
        stage:
          - -init

This file has been truncated. show original

ogra · August 29, 2019, 1:04pm

try this:

name: wekan
...
adopt-info: mypart
...
summary: The open-source kanban
description: |

part:
  mypart:
  ...
    override-build: |
      snapcraftctl set-version "$(git describe --dirty --tags | cut -c 2-)"
      snapcraftctl build
   stage-packages:
      - ...
...

popey · August 29, 2019, 1:25pm

Sorry, I wasn’t clear. Yes, adopt-info is in the metadata, snapcraftctl goes in the override section.

xet7 · August 29, 2019, 1:27pm

Currently I’m trying to build with these changes, I don’t know are they correct:

github.com

wekan/wekan/blob/master/snapcraft.yaml

name: wekan
version: 0
version-script: git describe --tags |  cut -c 2-
summary: The open-source kanban
description: |
   Wekan is an open-source and collaborative kanban board application.

   Whether you’re maintaining a personal todo list, planning your holidays with some friends, or working in a team on your next revolutionary idea, Kanban boards are an unbeatable tool to keep your things organized. They give you a visual overview of the current state of your project, and make you productive by allowing you to focus on the few items that matter the most.
   Depending on target environment, some configuration settings might need to be adjusted.
   For full list of configuration options call:
   $ wekan.help

confinement: strict
grade: stable

architectures:
  - amd64

plugs:
  mongodb-plug:

This file has been truncated. show original

xet7 · August 29, 2019, 1:29pm

My changes did not work:
https://launchpadlibrarian.net/439451580/buildlog_snap_ubuntu_xenial_amd64_wekan-devel_BUILDING.txt.gz

popey · August 29, 2019, 1:33pm

You added a new part called mypart which wasn’t necessary. Just add the snapcraftctl line to the override-build section in the wekan part, and make adopt-info: wekan then remove the mypart. I think @ogra was giving you generic advice about the placement of pieces in a yaml, rather than copy/pastable code.

xet7 · August 29, 2019, 8:01pm

It seems I have failed.

https://launchpadlibrarian.net/439488689/buildlog_snap_ubuntu_xenial_amd64_wekan-devel_BUILDING.txt.gz

github.com

wekan/wekan/blob/master/snapcraft.yaml

name: wekan
version: 0
version-script: git describe --tags |  cut -c 2-
summary: The open-source kanban
description: |
   Wekan is an open-source and collaborative kanban board application.

   Whether you’re maintaining a personal todo list, planning your holidays with some friends, or working in a team on your next revolutionary idea, Kanban boards are an unbeatable tool to keep your things organized. They give you a visual overview of the current state of your project, and make you productive by allowing you to focus on the few items that matter the most.
   Depending on target environment, some configuration settings might need to be adjusted.
   For full list of configuration options call:
   $ wekan.help

confinement: strict
grade: stable

architectures:
  - amd64

plugs:
  mongodb-plug:

This file has been truncated. show original

It did build snap and upload to store, but snap store rejected because of these files, that AFAIK are deleted in snapcraft.yaml file. I did also try recursive delete of .swp files, but it did not work, those did still exists.

found potentially sensitive files in package: lib/node_modules/wekan/.build/bundle/programs/server/npm/node_modules/tar/lib/.mkdir.js.swp, lib/node_modules/wekan/.build/bundle/programs/server/node_modules/node-pre-gyp/node_modules/tar/lib/.mkdir.js.swp, lib/node_modules/wekan/node_modules/tar/lib/.mkdir.js.swp lint-snap-v2_iffy_files

And Wekan release still has word dirty in it.

I did about 8 releases of Wekan today, trying to make it work in various ways.

I did also go to shower and sauna.

How can I get it cleaned up?

cjp256 · August 30, 2019, 7:58pm

While you should be using adopt-info as suggested above, it may be that you are seeing “dirty” because package-lock.json is tracked and might be modified by npm?

If any files in the git tree end up modified, it will result in the “dirty” label.

xet7 · September 5, 2019, 9:48pm

I did have problems with deleting temporary .swp files with those newer version codes, build service did not accept Wekan package at all, so I had to go back to original version script and remove from command --dirty to get Wekan released.

Is there some reason there is plans to remove support for version script? Could it still be kept? I would like to still have version detection working, and to be able to release new Snap packages. It has taken countless of hours to get build service even to accept Wekan package at all. Or can someone help with making required changes to snapcraft.yaml ?

Wekan also still uses core-16. I was not able to get core-18 working at all with snap. Is it possible that someone would help with getting core-18 version of script working?

Can also someone help make Wekan snap for other CPU architectures? For that, Wekan source bundle is made like in current snapcraft.yaml on x64, then that bundle that is also at https://releases.wekan.team as wekan-3.xx.zip is on other CPU architecture unzipped and run npm install commands as described at bundle section of page https://github.com/wekan/wekan/wiki/Platforms .