When we run checkbox-snappy.checkbox-cli , it gives a graphical interface to choose the tests.
is it possible to have a configuration file by default to run the specific tests , say example i want to run audio and wifi tests when i run the checkbox snap.
i do not find any example to create a plainbox.conf or checkbox.conf. Could you please give some suggestion how to create .conf and add requires tests to run ?
vinaysimha@localhost:~$ checkbox-snappy.checkbox-cli check-config
Configuration files:
Checkbox does not provide a way to fine tune the list of tests to run using the configuration file.
One possible workaround is to call checkbox-snappy.checkbox-cli run with a test plan identifier.
For snappy we have created several small test plans to build up the main QA test plans (basically one per device/category).
You can list them using checkbox-snappy.checkbox-cli list 'test plan'
Then to run for instance audio only test cases (those fully automated), type:
checkbox-snappy.checkbox-cli run com.canonical.certification::audio-automated
References:
i am able to run the jobs individually.
checkbox-snappy.checkbox-cli run com.canonical.certification::monitor/hdmi
checkbox-snappy.checkbox-cli run com.canonical.certification::ethernet-full
checkbox-snappy.checkbox-cli run com.canonical.certification::cpuinfo
checkbox-snappy.checkbox-cli run com.canonical.certification::memory/info
checkbox-snappy.checkbox-cli run com.canonical.certification::disk/detect
i want to run all the above jobs in a automated way and get the report in .xml, html, tar.xz
Do i need to create a snap?
I think checkbox should learn to look at $SNAP_USER_DATA for configuration.
Yes, with a custom test plan. See links that spineau pasted.
Loading from $SNAP_DATA has landed a few days ago.
i am running this in snapdragon 410c board (arm64)
snap list
Name Version Rev Developer Notes
checkbox-snappy 2.2 860 ce-certification-qa devmode
vinaysimha@localhost:~$ checkbox-snappy.checkbox-cli check-config
Configuration files:
i am not able to create plainbox.conf or checkbox.conf in the above patch specified
even i tried export SNAP_DATA=$PWD and SNAP_USER_DATA=$PWD ,
created a sample plainbox.conf with contents
[environment]
OPEN_BG_SSID=my-ap-ssid
But still check-config is not able to find any .conf file.
please suggest.
please give some references where checkbox.conf and plainbox.conf used to test in any platform?
@kissiel for convenience I would suggest using $SNAP_COMMON which doesn’t move across revisions.
@vinaysimha it seems the copy you are using does not yet support the new location, you may need to get a new release from the checkbox team.
i tried to build the checkbox for arm64, but not able to build.
git+ssh://git.launchpad.net/checkbox-snappy
checkbox-snappy [arm64] $ snapcraft --target-arch arm64
Setting target machine to 'arm64’
Cannot find the definition for part ‘plainbox-provider-snappy-resource’.
It may be a remote part, run snapcraft update to refresh the remote parts cache.
i am trying to clone , but it is failing
git clone https://git.launchpad.net/plainbox-provider-snappy-resource
Cloning into ‘plainbox-provider-snappy-resource’…
Username for ‘https://git.launchpad.net’: vinaysimha@inforcecomputing.com
Password for ‘https://vinaysimha@inforcecomputing.com@git.launchpad.net’:
remote: Authorisation required.
fatal: Authentication failed for ‘https://git.launchpad.net/plainbox-provider-snappy-resource/’
Quoting https://docs.snapcraft.io/reference/env :
SNAP_COMMON
(…)
Unlike SNAP_DATA this directory does is not backed up and restored across snap refresh and revert operations.
i am not able to access the link https://docs.snapcraft.io/reference/env:
404 not found : nginx/1.10.3 (Ubuntu)
Colon (’:’) got into the url, sorry. Edited.
by referring http://checkbox.readthedocs.io/en/latest/launcher-tutorial.html
i created a test file , copied this example launcher 1 com.canonical.certification::smoke, i got this malfunciton error
./test
Application Malfunction Detected
Traceback (most recent call last):
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/core.py”, line 248, in eat
return self._dispatch()
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/core.py”, line 298, in _dispatch
result = ingredient.dispatch(self.context)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 152, in dispatch
return self._dispatch(context, 0)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 179, in _dispatch
return self._dispatch_None(context, level, retval, command)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 208, in _dispatch_None
return self._dispatch(context, level + 1)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 165, in _dispatch
retval = command.invoked(context)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 242, in invoked
self._export_results()
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 681, in _export_results
self._create_transport(params[‘transport’])
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 609, in _create_transport
tr_type = self.launcher.transports[transport][‘type’]
KeyError: ‘outfile’
i can run the memory-automated
[test plan]
unit = com.canonical.certification::memory-automated
forced = yes
But i cannot run only the memory/info test plan . please suggest
[test plan]
unit = com.canonical.certification::memory/info
forced = yes
Traceback (most recent call last):
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/core.py”, line 248, in eat
return self._dispatch()
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/core.py”, line 298, in _dispatch
result = ingredient.dispatch(self.context)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 152, in dispatch
return self._dispatch(context, 0)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 179, in _dispatch
return self._dispatch_None(context, level, retval, command)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 208, in _dispatch_None
return self._dispatch(context, level + 1)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 165, in _dispatch
retval = command.invoked(context)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 220, in invoked
self._start_new_session()
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 368, in _start_new_session
self.ctx.sa.select_test_plan(tp_id)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/decorators.py”, line 142, in wrapper
raise exc
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/decorators.py”, line 136, in wrapper
return func(*args, **kwargs)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/session/assistant.py”, line 812, in select_test_plan
test_plan = self._context.get_unit(test_plan_id, ‘test plan’)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/session/state.py”, line 463, in get_unit
raise KeyError(unit_id)
KeyError: ‘com.canonical.certification::memory/info’
You can use triple backtick (```) to make long pasted text look nicer, try it
i can run the memory-automated
[test plan]
unit = com.canonical.certification::memory-automated
forced = yes
But i cannot run only the memory/info test plan . please suggest
[test plan]
unit = com.canonical.certification::memory/info
forced = yes
Traceback (most recent call last):
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/core.py”, line 248, in eat
return self._dispatch()
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/core.py”, line 298, in _dispatch
result = ingredient.dispatch(self.context)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 152, in dispatch
return self._dispatch(context, 0)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 179, in _dispatch
return self._dispatch_None(context, level, retval, command)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 208, in _dispatch_None
return self._dispatch(context, level + 1)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/guacamole/ingredients/cmdtree.py”, line 165, in _dispatch
retval = command.invoked(context)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 220, in invoked
self._start_new_session()
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/checkbox_ng/launcher/subcommands.py”, line 368, in _start_new_session
self.ctx.sa.select_test_plan(tp_id)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/decorators.py”, line 142, in wrapper
raise exc
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/decorators.py”, line 136, in wrapper
return func(*args, **kwargs)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/session/assistant.py”, line 812, in select_test_plan
test_plan = self._context.get_unit(test_plan_id, ‘test plan’)
File “/snap/checkbox-snappy/860/lib/python3.5/site-packages/plainbox/impl/session/state.py”, line 463, in get_unit
raise KeyError(unit_id)
KeyError: ‘com.canonical.certification::memory/info’how to set particular job in launchers?
There is only option to set the [test plan] , is there any option to set the job only
[job]
unit = com.canonical.certification::memory/info
forced = yes
There’s no option in launchers (yet) to set a list of jobs to run. You can only select a test plan and decide if you want to skip to manual selection to jump straight to the tests execution.
Since you can build your own checkbox snap, may I suggest to clone one of our providers and create a new test plan unit (See http://checkbox.readthedocs.io/en/latest/units/test-plan.html).
This way you’ll be able to create a launcher and call your custom test plan.
I’d recommend the plainbox-provider-checkbox provider. you can put the new test plan unit anywhere under the /units directory.
Finally clone https://code.launchpad.net/checkbox-snappy to point the checkbox provider to your local repo before calling snapcraft.
Update the checkbox-snappy snapcraft.yaml as follow:
parts:
plainbox-provider-resource:
after: [plainbox-dev, checkbox-support-dev, checkbox-ng-dev]
plainbox-provider-checkbox:
source: /path/to/your/checkbox-provider-local-repo
after: [plainbox-provider-resource]
plainbox-provider-snappy:
after: [plainbox-provider-checkbox]
plainbox-provider-ipdt:
after: [plainbox-provider-snappy]
plainbox-provider-docker:
after: [plainbox-provider-ipdt]
plainbox-provider-tpm2:
after: [plainbox-provider-docker]
plainbox-provider-sru:
after: [plainbox-provider-tpm2]
plainbox-provider-engineering-tests:
after: [plainbox-provider-sru]
Reference: https://docs.snapcraft.io/reference/plugins/source
checkbox is not able to build for arm64. there is no snapcraft for the snap i am running on the target.
i need to figure out how to build in launchpad builders. please suggest.
git clone git+ssh://git.launchpad.net/plainbox-provider-checkbox
git clone git+ssh://git.launchpad.net/checkbox-snappy
modified the snapcraft.yaml
+ source: ../plainbox-provider-checkbox
snapcraft --target-arch arm64
The plugin used by 'plainbox-provider-ipdt' does not support cross-compiling to a different target architecture
The plugin used by 'plainbox-provider-resource' does not support cross-compiling to a different target architecture
The plugin used by 'plainbox-provider-checkbox' does not support cross-compiling to a different target architecture
currently in target (arm64)
Name Version Rev Developer Notes
checkbox-snappy 2.2 860 ce-certification-qa devmode
bogomips: 38
cache: -1
count: 4
model: aarch64
model_number: 0x0
model_revision: 0
model_version: 8
other: fp asimd evtstrm crc32
platform: aarch64
speed: 1209
type: aarch64