Hello, I am creating some tests for console-conf using spread.
The current approach to execute them is to create to spawn the console-conf process from a expect file which is called from the tasks by doing “expect -d -f task.exp”
When I run the tests from spread I see that the console-conf starts and sudenly dies before showing the first screen (no errors in any log). Looking at the subiquity code I think it is happening because it needs a terminal to display its content. Then I executed manually the call to the expect file from a terminal in the test machine and it is working.
I already tested many different wayt to spawn the process but with the same result. Currently I am testing a new approach by using xvfb, but still in progress.
I would like to hear any ideas about which approach I could follow to implement those tests.
Yes, it is an alternative but I wanted to use first spread to be able to run those test in the real devices. The main objective is to reduce the validation time for promote to candidate.
In the small tests I’ve run I’m not really sure console-conf dies http://paste.ubuntu.com/24460962/ the entries expect: does "\u001b]4;0;rgb:00/00/00;1;rgb:cd/00/00;2;rgb:..." match glob pattern "Ubuntu core"? no seem to show that it is alive, but the output is not readable by expect, do you get this kind of output? This is tests/main/console-conf/task.yaml
If you execute the expect manually from the console you will see in the logs that the texts are added and match. The problem is that console conf is not showing the welcome screen, so you can’t see the welcome text in the logs.
Could you try running “expect -d console-conf.exp” from the test machine, to see if you get the same result than me?
Sure, running from an open session with a non root user I get the same result as you with the above console-conf.exp, with a modified console-conf-exp (removing the initial set``and the -nottyinit -nottycopy` spawn options, and invoking console-conf with sudo in the spawn call http://paste.ubuntu.com/24466912/) I get the actual textual output http://paste.ubuntu.com/24466891/, seems that in the other case the output is cut before the text is received?
Based on the /var/log/console-conf/subiquity-debug.log, the console-conf is stopped, otherwise the log should show more details after the line “subiquitycore.core:95 moving to screen Welcome”
mmm maybe it doesn’t show more details in the log because expect doesn’t find the pattern and doesn’t send the “\n” to continue to the next screen, this is the log from a regular execution:
Well I found a solution to run the tests from spread.
The idea is to add a new output mode, which is a mix between the standart mode and the shell mode, so by doing that a terminal is created as in the shell mode but in this case that terminal receives the script to execute and it’s not interactive. All the output of the execution is shown.
I already changed spread with this change and seems to be working, I would to like to hear the @niemeyer opinion about that change before proposing that.
I don’t think we want to allocate a terminal during normal operation of spread tasks, as part of the point of having a terminal is the assumption that you have a person observing and interacting with the tool. But spread is a batch execution tool, and benefits from choices those scripts make when deciding it’s not actually a human in a terminal there.
That said, the whole point of expect and similar tools is precisely to emulate the human there, so I’m sure it will internally allocate a pty and run the child command under that environment.
As we discussed before, I suggest taking one of the expect scripts we already have working and slowly modifying them until it breaks, so you’ll have a clear idea of what the reason is. Also note Federico’s comments above. Federico has battle-tested a lot of those issues in our long snapd test suite, so his input is worth paying attention to.
Well the idea is just to do that allocation when it is required, not always, something similar to the shell mode. The change that I am proposing does not require any human interaction, just it will create/open the terminal, use it and close it automatically.
Well We already did that, I just left the spawn and 1 expect line, and the spect is not receiving the text that it should. If I run the same but from a terminal in the test machine it works fine. I suspect that the problem is on how console-conf is showing the screen, it needs a terminal, and it does not work when we call it from a batch script with no access to the terminal output.
Again, it is expect itself that creates and manipulates the pty that the child application runs under. If that wasn’t the case you’d see the whole output on your terminal while expect was executing, and its own stdin/stdout interaction would be mixed with that of the child. That’s definitely not how it works.
The fact we have several expect-based tasks is also proof that it does work fine without spread having to allocate a pty externally.
To run the tests you will need to use this spread branch: https://github.com/sergiocazzolato/spread/tree/rawpty
This branch allows to run the tasks in a rawpty, the output that you will see is the same than the regular one, but the difference is on how the terminal is created to run the tasks.
This is a temporal solution while we find out how to make expect to create a raw pty when it is called from spread. To run the tests add -rawpty to the execution command.
@cachio tests look great, thanks a lot for pushing forward with them! As we talked online we can start trying to replace the manual ones with them in the next validation, compare results and see how things go.
I’ll start looking into expect config, as @niemeyer mentioned we should be able to configure appropriately the pty expect’s allocates to work with console-conf’s output, will let you know how it goes.
Again, we don’t want to patch spread for that, because we don’t want to run a tasks assuming a terminal and a human is available when they aren’t.
The task at hand has full root access on that machine, though, and should be able to do what it needs.
If expect is broken for some reason, we can pick something else that works. I still wonder what’s the difference between your task that doesn’t work and the tasks we already have that do work.
Did you figure that out? How far did you get into figuring it?
Well, I tested all the parameters to make expect run with a raw pty but it is like expect is not able to start a raw pty if there is not a tty associated.
This is from the expect documentation:
Internally, spawn uses a pty, initialized the same way as the
user’s tty. This is further initialized so that all settings are
“sane” (according to stty(1)). If the variable stty_init is
defined, it is interpreted in the style of stty arguments as fur-
ther configuration. For example, “set stty_init raw” will cause
further spawned processes’s terminals to start in raw mode.
-nottycopy skips the initialization based on the user’s tty.
-nottyinit skips the “sane” initialization.
When I run with spread the variable stty_init is not defined, so the raw pty is not initialized. I tried different things and hacks to make that variable available for expect but I could not make it work. So the change that I did in spread is just a workaround until we find out another solution.
Also I was discussing that with the subiquity developers to see if something else could be done. Sadly no other ideas came up from that discussion, just that expect is failing to create a raw pty. Something interesting could be to see any internal log of expect.
This documentation confirms the points mentioned above: expect allocates its own pty. It also says that the settings are copied from the user terminal, which means ssh is doing a better job setting up the pty configuration in a way that makes your tests happy than expect itself.
This still doesn’t make clear what the specific setting is though (again, we have other tasks working fine with expect). The expect pty needs to be tuned so it suits the application being tested, or replace expect altogether with something that does exactly that.
It seems that the pty created by expect somehow inherits some of the features of the terminal from which the process is called, no matter if -nottycopy is used in spawn, the calls to stty_init raw or direct calls to stty, maybe I’m not doing them right though (will keep trying). If the calling terminal is non-interactive, part of the output is lost and the expect assertions fail, I’ve verified that the console-conf process is alive before expect’s timeout but the complete output is not captured, trying the most straight configuration approaches didn’t seem to work for me.
In order to unblock us I’ve created a supersimple utility that launches a given command on a remote host over ssh after requesting an interactive pty with width and height set https://github.com/fgimenez/rawpty/blob/master/cmd/rawpty/main.go. Using it you can successfully run expect without needing to modify spread:
