Snapd 2.28 REST API: Endpoints with large response issue

snapd
3

Did a test to check if this issue originate from Nginx or not with below go snippet to forward snapd.socket to 127.0.0.1:9000

testfwd.go

import (
	"io"
	"log"
	"net"
)

const (
	UNIX_ADDR = "/run/snapd.socket"
	TCP_ADDR  = "127.0.0.1:9000"
)

func main() {
	l, err := net.Listen("tcp", TCP_ADDR)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("listening on %s", TCP_ADDR)
	for {
		uconn, err := l.Accept()
		if err != nil {
			log.Printf("accept failed: %s", err)
			continue
		}
		go fwd(uconn)
	}
}

func fwd(uconn net.Conn) {
	defer uconn.Close()
	tconn, err := net.Dial("unix", UNIX_ADDR)
	if err != nil {
		log.Printf("tcp dial failed: %s", err)
		return
	}
	log.Printf("connected to %s", tconn.RemoteAddr())
	go io.Copy(uconn, tconn)
	io.Copy(tconn, uconn)
	log.Printf("disconnected")
}

Running above go implementation on the host works fine and retrieved the whole body in one large HTTP data-chunk.

After some further debugging, I noticed that the whole HTTP body were retrieved but the issue seems to be that the body is missing details about the HTTP chunk-boundary/chunk-size’s and Wireshark is unable to even interpret the response data as an HTTP response.

While running snapd 2.28, by proxy snapd through Nginx, the response is missing all the chunk-boundary/chunk-size in the responses that are usually there on the 2.27.4.

Using above go snippet, I can verify that the HTTP chunk-boundary/chunk-size is still present just running above go snippet, so this seems to be some behavior in Nginx and does not necessarily need to be an issue in the Snapd REST API socket since it have the chunk-boundary/chunk-size defined but still something that affect Nginx to be able to handle these requests while reaching snapd version 2.28.

Still unable to narrow down why Nginx would have problem to proxy these snapd responses on 2.28 but no problem in 2.27.4.

Reproduce steps:

  1. Create Gadget Model Device base on Core 18 which will have snapd as separate snap.

  2. Run above GO (./testfwd) implementation on the target with Core18 + “snapd” snap. $ sudo ./testfwd

  3. Port Forward the listing port from the “./testfwd” (port 9000) implementation on your host that run Nginx.

$ ssh user@<Gadget with Core18 as base> -R 9000:127.0.0.1:9001

  1. Install/configure Nginx. (Ubuntu 18.04 LTS, deb: 1.14.0-ubuntu1.2)

$ sudo apt-get install nginx

$ vim /etc/nginx/sites-available/default
######################################
server 
{
    listen 127.0.0.1:9001;
    location /
    {
        proxy_pass http://127.0.0.1:9002;
    }
}
######################################

$ sudo systemctl restart nginx

  1. Check the communication via tcpdump/wireshark

    #Test snapd via ./testfwd
    $ curl -i http:127.0.0.1:9001/v2/snaps
    $ curl -i http:127.0.0.1:9001/v2/snaps/snapd
    $ curl -i http:127.0.0.1:9001/v2/changes?select=all

    #Test snapd Nginx endpoint
    $ curl -i http:127.0.0.1:9002/v2/snaps
    $ curl -i http:127.0.0.1:9002/v2/snaps/snapd
    $ curl -i http:127.0.0.1:9002/v2/changes?select=all

Tested snapd Revisions
Snapd Revision (2567)
Snapd Revision (2827) # Unable to proxy snapd response via Nginx properly