Proposal: Use casync for delta updates

snapd
1

Please consider moving away from xdelta and use casync

Currently snapd keeps at least one old revision of a snap. This is good for making sure that a bad update doesn’t result in snap breakage. This however also wastes disk space. So instead of keeping the whole snap of the old revision, snapd should only maintain the delta and reconstruct the snap in case an update goes bad, using the newly installed snap as the seed (yep, that’s possible with casync).

This could work great for both IoT use cases as well as desktop as we don’t lose the current functionality.

3 Likes
2

I am using desync (alternate implementation of casync) for OS OTA updates. I am also using it for snap updates in my custom store implementation. I will share a few commands on how that can be done.

Using Casync also means that all updates can be delta based, without much effort. A user who is 10 revisions behind of the current update can apply the delta update very reliably.

1 Like
3

I am going to come up with a few working examples to show how this could work… during the weekend.

4

That took a while, I was super busy with work. I understand that “proposing” such big changes is generally not aligned with a project’s vision and requires work both on the store side and the client side. However the benefits are unmatched. Here is the basic steps that I took to “emulate” what I am proposing.

Download two different revisions of htop snap

snap download htop --stable
snap download htop --edge

Rename the snap file for clarity

om26er@HomePC:~/Desktop/desync$ ls
htop_3161.assert  htop_3161.snap  htop_3181.assert  htop_3181.snap
om26er@HomePC:~/Desktop/desync$ mv htop_3161.snap a.snap
om26er@HomePC:~/Desktop/desync$ mv htop_3181.snap b.snap
om26er@HomePC:~/Desktop/desync$ ls
a.snap  b.snap  htop_3161.assert  htop_3181.assert

Create casync indexes (a.caibx and b.caibx) and chunks for both snaps in store directory

om26er@HomePC:~/Desktop/desync$ mkdir store
om26er@HomePC:~/Desktop/desync$ desync make a.caibx a.snap -s store
Chunking [==========================================================================================================================================================] 100.00% 0s
Storing [===========================================================================================================================================================] 100.00% 0s
om26er@HomePC:~/Desktop/desync$ desync make b.caibx b.snap -s store
Chunking [==========================================================================================================================================================] 100.00% 0s
Storing [===========================================================================================================================================================] 100.00% 0s
om26er@HomePC:~/Desktop/desync$ ls store
0001  07e5  0fd5  19ba  1ebf  27fa  3791  406c  4a42  533a  5cf5  6e3a  7ee8  8533  9304  9841  a3c5  a978  b44d  ba8a  c94c  d8e7  e236  e7f7  ed62  fa5b
0195  0953  10a1  1ab5  1ee0  28fc  3954  42f9  4ba9  55a1  6017  7042  7f4b  8673  942d  9958  a510  acf2  b489  bca7  ca0c  d9b4  e248  e86d  eeba  fa9a
0352  0998  11e5  1adc  2122  2bd6  3a9b  453a  4c5a  575d  65d9  71c7  814e  86a9  9463  9a08  a707  aebe  b62f  bf69  ce14  da05  e341  e983  ef14  fb9f
0385  0b69  1221  1c18  2256  2c46  3b05  464a  50d6  5795  67a1  7766  8256  86f8  94f9  9b27  a736  b0cd  b665  c296  ce3e  dc3d  e434  ea9b  f049  fd7f
0403  0d5d  140c  1cf3  23f7  31dc  3cfc  481d  50d9  58a6  69b5  7976  826a  8a98  96e0  9f4c  a77c  b147  b74d  c4f0  cec5  ddaa  e656  ebf6  f4a3  fe0c
0754  0d99  171a  1de4  257d  36b6  3d1b  49be  5104  5c28  6b7a  7c5d  8314  90a8  977a  9f8c  a879  b296  b9e0  c61a  d27c  dfb0  e76f  ec01  f4fa
07ac  0e38  184c  1df0  278a  3760  3dde  49ee  5165  5c3c  6b83  7e79  8337  92e4  97da  a27f  a8ee  b3c8  ba1f  c718  d575  e02b  e7e6  ec96  f88a
om26er@HomePC:~/Desktop/desync$ ls
a.caibx  a.snap  b.caibx  b.snap  htop_3161.assert  htop_3181.assert  store

Save chunks that are missing between newer revision (edge) and old (stable)

om26er@HomePC:~/Desktop/desync$ rm a.snap
om26er@HomePC:~/Desktop/desync$ mkdir a-snap-chunks-minus-b
om26er@HomePC:~/Desktop/desync$ desync cache -s store -c a-snap-chunks-minus-b/ a.caibx --ignore b.caibx
om26er@HomePC:~/Desktop/desync$ ls a-snap-chunks-minus-b/
0001  0d5d  0e38  171a  184c  1ee0  3760  3a9b  3b05  4c5a  5c3c  67a1  6b83  8337  90a8  96e0  9958  a77c  a8ee  b147  c4f0  e86d
om26er@HomePC:~/Desktop/desync$ du -sh a-snap-chunks-minus-b
1.1M    a-snap-chunks-minus-b

Let’s reconstruct the old revision of the snap from the chunks (in case a revert is needed).

om26er@HomePC:~/Desktop/desync$ desync extract --seed b.caibx a.caibx a.snap -s a-snap-chunks-minus-b
[===================================================================================================================================================================] 100.00% 0s
om26er@HomePC:~/Desktop/desync$ du -sh a.snap
9.5M    a.snap
om26er@HomePC:~/Desktop/desync$ ls
a.caibx  a.snap  a-snap-chunks-minus-b  b  b.caibx  b.snap  htop_3161.assert  htop_3181.assert  store
om26er@HomePC:~/Desktop/desync$ sudo snap ack htop_3161.assert
[sudo] password for om26er: 
om26er@HomePC:~/Desktop/desync$ sudo snap install a.snap
htop 3.1.0 from Maximiliano Bertacchini (maxiberta) installed

That basically shows it all. The store directory aka the chunk store, should live on a http service or S3.

The above commands basically show how, instead of keeping a whole copy of a snap on the host, we can get away with only keeping the diff and reconstructing the snap “ondemand”, when needed.

Such an approach will definitely help with the narrative that snaps waste disk space, especially for the desktop apps, it is also useful for the IoT use case where the disk space is mostly limited.

5

The great thing about using desync is that it’s written in golang and is written as a library, which makes it relatively simpler to integrate into snapd.

Here is the project https://github.com/folbricht/desync

6

I need to mention that using casync/desync is not a hard requirement, as long as we can get away with keeping the delta of older revisions instead of full snaps. I guess at least the revert implementation could be done using xdelta as well.

The biggest advantage of casync is that we can skip N revisions as all the chunks are stored in a single store, like git.