I’m looking into options for regression testing our UC images, either virtually or on hardware.
This is not so much for hardware approval [ although that may also be good ], but to ensure all our snaps are working as they should with automated tests to ensure functionality. The idea would be to spin up the image on something, run the test, and return a report that validates the image that can be automatically validated.
I think, this is a great topic to discuss. As you mentioned, the checkbox is one of the alternatives but it mostly focuses on the stability of the hardware and OS functionalities. But, I think it is even possible to extend it with some additional test cases.
When you look at the problem essentially, what you need is a way to communicate with the device which is generally ssh, then run several commands and get its results. There are quite much open source projects which are designed for automated integration tests (e.g. https://robotframework.org/ )
At the end, there are too many alternatives and you need to choose a suitable tool for you/your team’s capabilities. You can even combine different solutions together. You could use checkbox for hardware verification and robot framework for applications etc.
ok !
Is the Spread config you use for testing Ubuntu Core published anywhere ? Seems like that would be a good starting point for our own UC image regression testing
Yes, it’s right in the snapd tree, we probably have the most complex spread.yaml in the world given that we are the primary users of spread, so I’m not sure how useful it is to you, but if you want to test Ubuntu Core via QEMU with spread, you can setup Ubuntu Core 20 grade dangerous images with cloud-init config setting up a root user password (that spread needs to login with) and then use the QEMU backend.
If you are interested in using spread to test against a real live device running Ubuntu Core 20, you’ll likely need to use the adhoc backend to setup SSH details and such, and just live with the fact that you may have issues with test failing and putting the device into a weird state and have to reprovision the device again.
Is the Spread config you use for testing Ubuntu Core published anywhere ? Seems like that would be a good starting point for our own UC image regression testing 