Snapd 2.28 REST API: Endpoints with large response issue

Currently experience issues with endpoints that have larger responses but only on the snapd 2.28 version.

The previous version worked fine but now I’m unable to retrieve the full response from the endpoint such a “/v2/snaps” “/v2/changes?select=all”.

Below curl responses/output are an update between Snapd 2.37.4(2567) to 2.28(2827), no other snaps or settings was updated/refresh during this moment. The snapd REST API is running behind an Nginx in this case but the request and responses headers are the same except the size of the body content.

Similar behavior on “/v2/changes?select=allsnapd 2.28. This endpoint doesn’t respond with an proper json. The response is capped/stuck in the middle of the json response and the connection times-out in the same way as the “/v2/snaps/”.

I did suspect that this was a question of the Nginx configuration but switching back to 2.37.4 via.
$ snap revert snapd --revision=2567
changed the behavior of the endpoint “/v2/snaps” and started to work again as expected.

I only experience this issue on endpoints that have larger responses such as “/v2/snaps” and “/v2/changes?select=all”. Using “/v2/snaps/snapd” still works as expected.

snapd 2.37.4 Response

$ curl -i http://127.0.0.1:9000/v2/snaps --trace-ascii -
#################################
== Info:   Trying 127.0.0.1...
== Info: Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
=> Send header, 86 bytes (0x56)
0000: GET /v2/snaps HTTP/1.1
0018: Host: 127.0.0.1:9000
002e: User-Agent: curl/7.47.0
0047: Accept: */*
0054:
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
HTTP/1.1 200 OK

<= Recv header, 15 bytes (0xf)
0000: Server: nginx
Server: nginx

<= Recv header, 37 bytes (0x25)
0000: Date: Wed, 17 Apr 2019 12:42:57 GMT
Date: Wed, 17 Apr 2019 12:42:57 GMT

<= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
Content-Type: application/json

<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
Transfer-Encoding: chunked

<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
Connection: keep-alive



<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 11566 bytes (0x2d2e)
0000: 2d21
0006: {"type":"sync","status-code":200,"status":"OK","result":[{"id":"
.....
2d06: 30892792Z"}],"sources":["local"]}
2d29: 0
2d2c:
{
  "type": "sync",
  "status-code": 200,
  "status": "OK",
  "result": [
....
  ],
  "sources": [
    "local"
  ]
}
== Info: Connection #0 to host 127.0.0.1 left intact
#################################

snapd 2.38 response

$ curl -i http://127.0.0.1:9000/v2/snaps --trace-ascii -
#################################
== Info:   Trying 127.0.0.1...
== Info: Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
=> Send header, 86 bytes (0x56)
0000: GET /v2/snaps HTTP/1.1
0018: Host: 127.0.0.1:9000
002e: User-Agent: curl/7.47.0
0047: Accept: */*
0054:
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
HTTP/1.1 200 OK

<= Recv header, 15 bytes (0xf)
0000: Server: nginx
Server: nginx

<= Recv header, 37 bytes (0x25)
0000: Date: Wed, 17 Apr 2019 12:44:02 GMT
Date: Wed, 17 Apr 2019 12:44:02 GMT

<= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
Content-Type: application/json

<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
Transfer-Encoding: chunked

<= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
Connection: keep-alive



<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 8112 bytes (0x1fb0)
0000: 1fa8
0006: {"type":"sync","status-code":200,"status":"OK","result":[{"id":"
1f86: n","validation":"unproven"},"developer":
{
    "type":"sync",
    "status-code":200,
    "status":"OK",
    "result":[
....
== Info: transfer closed with outstanding read data remaining
== Info: Closing connection 0
#################################

How can I reproduce this?

Did a test to check if this issue originate from Nginx or not with below go snippet to forward snapd.socket to 127.0.0.1:9000

testfwd.go

import (
	"io"
	"log"
	"net"
)

const (
	UNIX_ADDR = "/run/snapd.socket"
	TCP_ADDR  = "127.0.0.1:9000"
)

func main() {
	l, err := net.Listen("tcp", TCP_ADDR)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("listening on %s", TCP_ADDR)
	for {
		uconn, err := l.Accept()
		if err != nil {
			log.Printf("accept failed: %s", err)
			continue
		}
		go fwd(uconn)
	}
}

func fwd(uconn net.Conn) {
	defer uconn.Close()
	tconn, err := net.Dial("unix", UNIX_ADDR)
	if err != nil {
		log.Printf("tcp dial failed: %s", err)
		return
	}
	log.Printf("connected to %s", tconn.RemoteAddr())
	go io.Copy(uconn, tconn)
	io.Copy(tconn, uconn)
	log.Printf("disconnected")
}

Running above go implementation on the host works fine and retrieved the whole body in one large HTTP data-chunk.

After some further debugging, I noticed that the whole HTTP body were retrieved but the issue seems to be that the body is missing details about the HTTP chunk-boundary/chunk-size’s and Wireshark is unable to even interpret the response data as an HTTP response.

While running snapd 2.28, by proxy snapd through Nginx, the response is missing all the chunk-boundary/chunk-size in the responses that are usually there on the 2.27.4.

Using above go snippet, I can verify that the HTTP chunk-boundary/chunk-size is still present just running above go snippet, so this seems to be some behavior in Nginx and does not necessarily need to be an issue in the Snapd REST API socket since it have the chunk-boundary/chunk-size defined but still something that affect Nginx to be able to handle these requests while reaching snapd version 2.28.

Still unable to narrow down why Nginx would have problem to proxy these snapd responses on 2.28 but no problem in 2.27.4.

Reproduce steps:

  1. Create Gadget Model Device base on Core 18 which will have snapd as separate snap.

  2. Run above GO (./testfwd) implementation on the target with Core18 + “snapd” snap. $ sudo ./testfwd

  3. Port Forward the listing port from the “./testfwd” (port 9000) implementation on your host that run Nginx.

$ ssh user@<Gadget with Core18 as base> -R 9000:127.0.0.1:9001

  1. Install/configure Nginx. (Ubuntu 18.04 LTS, deb: 1.14.0-ubuntu1.2)

$ sudo apt-get install nginx

$ vim /etc/nginx/sites-available/default
######################################
server 
{
    listen 127.0.0.1:9001;
    location /
    {
        proxy_pass http://127.0.0.1:9002;
    }
}
######################################

$ sudo systemctl restart nginx
  1. Check the communication via tcpdump/wireshark

    #Test snapd via ./testfwd
    $ curl -i http:127.0.0.1:9001/v2/snaps
    $ curl -i http:127.0.0.1:9001/v2/snaps/snapd
    $ curl -i http:127.0.0.1:9001/v2/changes?select=all

    #Test snapd Nginx endpoint
    $ curl -i http:127.0.0.1:9002/v2/snaps
    $ curl -i http:127.0.0.1:9002/v2/snaps/snapd
    $ curl -i http:127.0.0.1:9002/v2/changes?select=all

Tested snapd Revisions
Snapd Revision (2567)
Snapd Revision (2827) # Unable to proxy snapd response via Nginx properly