Testing console-conf

[cachio]

did you consider using qemu

Yes, it is an alternative but I wanted to use first spread to be able to run those test in the real devices. The main objective is to reduce the validation time for promote to candidate.

[fgimenez]

@cachio thanks for working on this!

In the small tests I’ve run I’m not really sure console-conf dies http://paste.ubuntu.com/24460962/ the entries expect: does "\u001b]4;0;rgb:00/00/00;1;rgb:cd/00/00;2;rgb:..." match glob pattern "Ubuntu core"? no seem to show that it is alive, but the output is not readable by expect, do you get this kind of output? This is tests/main/console-conf/task.yaml

summary: console-conf

execute: |
    snap install --edge --devmode expect

    expect -d console-conf.exp

and this tests/main/console-conf.exp:

set stty_init sane
spawn -nottyinit -nottycopy /usr/bin/console-conf

expect {
    "Ubuntu core" {
        send "\n"
        expect {
            "network"
            exit 0
               } default { exit 1 }
    } default {
        exit 1
    }
}

[cachio]

I tested that and I see the same result.

Timeout got from the expect session log (log_file -noappend -a /home/test/all.log):
https://paste.ubuntu.com/24466688/

Log /var/log/console-conf/subiquity-debug.log:
https://paste.ubuntu.com/24466693/

[cachio]

If

match glob pattern “Ubuntu core”? no seem to show that it is alive, but the output is not readable by expect

If you execute the expect manually from the console you will see in the logs that the texts are added and match. The problem is that console conf is not showing the welcome screen, so you can’t see the welcome text in the logs.

Could you try running “expect -d console-conf.exp” from the test machine, to see if you get the same result than me?

Thanks for helping

[fgimenez]

Could you try running “expect -d console-conf.exp” from the test machine, to see if you get the same result than me?

Sure, running from an open session with a non root user I get the same result as you with the above console-conf.exp, with a modified console-conf-exp (removing the initial set``and the-nottyinit -nottycopy` spawn options, and invoking console-conf with sudo in the spawn call http://paste.ubuntu.com/24466912/) I get the actual textual output http://paste.ubuntu.com/24466891/, seems that in the other case the output is cut before the text is received?

[cachio]

seems that in the other case the output is cut before the text is received?

Based on the /var/log/console-conf/subiquity-debug.log, the console-conf is stopped, otherwise the log should show more details after the line “subiquitycore.core:95 moving to screen Welcome”

[fgimenez]

mmm maybe it doesn’t show more details in the log because expect doesn’t find the pattern and doesn’t send the “\n” to continue to the next screen, this is the log from a regular execution:

2017-04-27 15:34:14,853 subiquitycore.core:88 Known Signals: ['quit', 'refresh', 'next-screen', 'prev-screen', 'menu:network:main:set-default-v4-route', 'menu:network:main:set-default-v6-route']
2017-04-27 15:34:14,853 subiquitycore.core:95 moving to screen Welcome
2017-04-27 15:34:15,748 subiquitycore.core:95 moving to screen Network
2017-04-27 15:34:15,749 subiquitycore.views.network:151 interfaces: [<subiquitycore.models.network.Networkdev object at 0x7f6ca7e1d2e8>]
2017-04-27 15:34:15,750 subiquitycore.ui.container:218 _focus_changed <FocusTrackingListBox selectable box widget>
2017-04-27 15:34:15,751 subiquitycore.ui.container:218 _focus_changed <FocusTrackingListBox selectable box widget>
2017-04-27 15:34:15,752 subiquitycore.ui.container:218 _focus_changed <FocusTrackingListBox selectable box widget>
2017-04-27 15:34:16,447 subiquitycore.ui.container:218 _focus_changed <FocusTrackingListBox selectable box widget>

It seems that there are no entries between moving to screen Welcome and moving to screen Network, what do you think?

[cachio]

Well I found a solution to run the tests from spread.

The idea is to add a new output mode, which is a mix between the standart mode and the shell mode, so by doing that a terminal is created as in the shell mode but in this case that terminal receives the script to execute and it’s not interactive. All the output of the execution is shown.

I already changed spread with this change and seems to be working, I would to like to hear the @niemeyer opinion about that change before proposing that.

[niemeyer]

I don’t think we want to allocate a terminal during normal operation of spread tasks, as part of the point of having a terminal is the assumption that you have a person observing and interacting with the tool. But spread is a batch execution tool, and benefits from choices those scripts make when deciding it’s not actually a human in a terminal there.

That said, the whole point of expect and similar tools is precisely to emulate the human there, so I’m sure it will internally allocate a pty and run the child command under that environment.

As we discussed before, I suggest taking one of the expect scripts we already have working and slowly modifying them until it breaks, so you’ll have a clear idea of what the reason is. Also note Federico’s comments above. Federico has battle-tested a lot of those issues in our long snapd test suite, so his input is worth paying attention to.

[cachio]

Thanks for answering

I don’t think we want to allocate a terminal during normal operation of spread tasks

Well the idea is just to do that allocation when it is required, not always, something similar to the shell mode. The change that I am proposing does not require any human interaction, just it will create/open the terminal, use it and close it automatically.

As we discussed before, I suggest taking one of the expect scripts we already have working and slowly modifying them until it breaks

Well We already did that, I just left the spawn and 1 expect line, and the spect is not receiving the text that it should. If I run the same but from a terminal in the test machine it works fine. I suspect that the problem is on how console-conf is showing the screen, it needs a terminal, and it does not work when we call it from a batch script with no access to the terminal output.

[niemeyer]

Again, it is expect itself that creates and manipulates the pty that the child application runs under. If that wasn’t the case you’d see the whole output on your terminal while expect was executing, and its own stdin/stdout interaction would be mixed with that of the child. That’s definitely not how it works.

The fact we have several expect-based tasks is also proof that it does work fine without spread having to allocate a pty externally.

@fgimenez has some suggestions above.

[cachio]

I already tried those suggestions and sadly none of them have worked.

I’ll continue digging in tomorrow with @fgimenez to see if we can understand the problem better and find out another solution.

Thanks

[cachio]

I have pushed a first version of the console conf tests: https://github.com/sergiocazzolato/console-conf-tests
Please take a look to this. This is WIP but I would like to hear any suggestion.

To run the tests you will need to use this spread branch: https://github.com/sergiocazzolato/spread/tree/rawpty
This branch allows to run the tasks in a rawpty, the output that you will see is the same than the regular one, but the difference is on how the terminal is created to run the tasks.

This is a temporal solution while we find out how to make expect to create a raw pty when it is called from spread. To run the tests add -rawpty to the execution command.

Thanks

[fgimenez]

@cachio tests look great, thanks a lot for pushing forward with them! As we talked online we can start trying to replace the manual ones with them in the next validation, compare results and see how things go.

I’ll start looking into expect config, as @niemeyer mentioned we should be able to configure appropriately the pty expect’s allocates to work with console-conf’s output, will let you know how it goes.

[niemeyer]

Again, we don’t want to patch spread for that, because we don’t want to run a tasks assuming a terminal and a human is available when they aren’t.

The task at hand has full root access on that machine, though, and should be able to do what it needs.

If expect is broken for some reason, we can pick something else that works. I still wonder what’s the difference between your task that doesn’t work and the tasks we already have that do work.

Did you figure that out? How far did you get into figuring it?

[cachio]

Did you figure that out? How far did you get into figuring it?

Well, I tested all the parameters to make expect run with a raw pty but it is like expect is not able to start a raw pty if there is not a tty associated.

This is from the expect documentation:

Internally, spawn uses a pty, initialized the same way as the
user’s tty. This is further initialized so that all settings are
"sane" (according to stty(1)). If the variable stty_init is
defined, it is interpreted in the style of stty arguments as fur-
ther configuration. For example, “set stty_init raw” will cause
further spawned processes’s terminals to start in raw mode.
-nottycopy skips the initialization based on the user’s tty.
-nottyinit skips the “sane” initialization.

When I run with spread the variable stty_init is not defined, so the raw pty is not initialized. I tried different things and hacks to make that variable available for expect but I could not make it work. So the change that I did in spread is just a workaround until we find out another solution.

Also I was discussing that with the subiquity developers to see if something else could be done. Sadly no other ideas came up from that discussion, just that expect is failing to create a raw pty. Something interesting could be to see any internal log of expect.

[niemeyer]

This documentation confirms the points mentioned above: expect allocates its own pty. It also says that the settings are copied from the user terminal, which means ssh is doing a better job setting up the pty configuration in a way that makes your tests happy than expect itself.

This still doesn’t make clear what the specific setting is though (again, we have other tasks working fine with expect). The expect pty needs to be tuned so it suits the application being tested, or replace expect altogether with something that does exactly that.

[fgimenez]

It seems that the pty created by expect somehow inherits some of the features of the terminal from which the process is called, no matter if -nottycopy is used in spawn, the calls to stty_init raw or direct calls to stty, maybe I’m not doing them right though (will keep trying). If the calling terminal is non-interactive, part of the output is lost and the expect assertions fail, I’ve verified that the console-conf process is alive before expect’s timeout but the complete output is not captured, trying the most straight configuration approaches didn’t seem to work for me.

In order to unblock us I’ve created a supersimple utility that launches a given command on a remote host over ssh after requesting an interactive pty with width and height set https://github.com/fgimenez/rawpty/blob/master/cmd/rawpty/main.go. Using it you can successfully run expect without needing to modify spread:

# tests/main/console-conf/task.yaml
summary: console-conf
execute: |
    snap install --edge --devmode expect
    snap install rawpty
    rawpty "/snap/bin/expect -d -f ${PWD}/console-conf.exp"

and

# tests/main/console-conf/console-conf.exp
spawn sudo /usr/bin/console-conf

expect {
    "Ubuntu Core" {
        exit 0
    } timeout {
        exit 1
    }
}

@cachio let me know if this works for you, I’ll keep digging on expect config.

[niemeyer]

Thanks for helping on it, but why is ssh needed? We should be able to create ptys without requiring ssh.

[fgimenez]

Yeah, sorry but I haven’t found any reference about how to create an interactive raw pty with non null height and width (the dimensions seem to be critical for console-conf and in general any urwid based application) from a noniteractive session. Opening an ssh session allows us to request our desired terminal, send a command to it and get the result easily, it also works fine with console-conf (sergio’s tests are working with very small modifications) but if there’s a simpler way of doing things any ideas are more than welcome