Snapcraft Metrics

Abstract

Provide an interface to publishers and authorized collaborators to retrieve metrics gathered by the Snap Store for a given snap.

Rationale

The Snap Store has APIs for requested metrics based on date ranges. The Snap Store provides these metrics in the form of graphs on the web UI. This specification is to extend Snapcraft CLI to provide access the underlying data for various metrics.

Background: Store APIs

Store URL:

Available Metrics:

  • daily_device_change: contains the 3 series representing the number of new, continued and lost devices with the given snap installed compared to the previous day.
  • installed_base_by_channel: contains one series per channel representing the number of devices with the given snap installed, channels with no data across the entire interval are omitted.
  • installed_base_by_country: contains one series per country representing the number of devices with the given snap installed.
  • installed_base_by_operating_system: contains one series per operating_system representing the number of devices with the given snap installed.
  • installed_base_by_version: contains one series per version representing the number of devices withe the given snap installed.
  • weekly_device_change: similar to the ‘daily_device_change’ metric but operates on a 7 day window. i.e. new contains the number of devices that were seen during the last 7 days but not in the previous 7 day and so on for continued and lost.
  • weekly_installed_base_by_channel: similar to the installed_base_by_channel metric but operates in a 7 day window.
  • weekly_installed_base_by_country: similar to the installed_base_by_country metric but operates in a 7 day window.
  • weekly_installed_base_by_operating_system: similar to the installed_base_by_operating_system metric but operates in a 7 day window.
  • weekly_installed_base_by_version: similar to the installed_base_by_version metric but operates in a 7 day window.

Each metric has a query that includes a start & end date. The returned data, if available, will include all days in between. Note that the weekly installed numbers are still provided on a daily basis, the provided numbers are the averages for the 7 day window ending on the specified day.

Snapcraft

Snapcraft will introduce a new metrics command to execute a query.

Usage

snapcraft metrics <snap-name> --name <metric-name> --start <start-date> --end <end-date> [--format=(json|table)]
parameter required/optional type description
snap-name required string Name of snap
metric required string Name of metric (see below).
start optional string of format YYYY-MM-DD Start of date range to request (must not be later than today’s date). Defaults to yesterday.
end optional string of format YYYY-MM-DD End (inclusive) of date range to request (must not be later than today’s date). Defaults to yesterday.
format required string Output format.

Supported metrics:

  • daily_device_change
  • installed_base_by_channel
  • installed_base_by_country
  • installed_base_by_operating_system
  • installed_base_by_version
  • weekly_device_change
  • weekly_installed_base_by_channel
  • weekly_installed_base_by_country
  • weekly_installed_base_by_operating_system
  • weekly_installed_base_by_version

Supported formats:

  • json
  • table

Output

JSON Formatting

When using --format=json, Snapcraft shall output will be a json string matching the response from the server (for the given metric). As such, the API is driven by the Store and may be extended in the future.

The relevant API documentation may be found here:

If response is the json object returned from the API,
Snapcraft shall the output the pertinent data at response["metrics"][0].

Example daily_device_change:

$ snapcraft metrics my-snap --name daily_device_change --start 2021-07-01 --end 2021-07-01 --format=json
{'buckets': ['2021-07-01'], 'metric_name': 'daily_device_change', 'series': [{'name': 'continued', 'values': [66]}, {'name': 'lost', 'values': [55]}, {'name': 'new', 'values': [77]}], 'snap_id': '<snap-id>', 'status': 'OK'}

Table Formatting

When using --format=table, Snapcraft shall output a table-based interpretation of the returned data with columns for the requested date ranges with rows of the requested data series. As the output for most queries will be large, a pager will be utilized. It is recommended that the user output this to a file and use an editor of choice.

if the query returns with a “None” data point, it is replaced with a “-” to indicate zero (or not applicable depending on context).

Example daily_device_change:

$ snapcraft metrics my-snap --name daily_device_change --start 2021-07-01 --end 2021-07-01 --format=table
Devices    2021-07-01
Continued  49
Lost       21
New        19

Example installed_base_by_channel:

$ snapcraft metrics my-snap --name installed_base_by_channel --start 2021-07-01 --end 2021-07-01 --format=table
Channel    2021-07-01  2021-07-02  2021-07-03
Beta       245         255         240
Candidate  1           1           0
Edge       68          78          85
Stable     401         405         409

Example installed_base_by_country:

$ snapcraft metrics my-snap --name installed_base_by_country --start 2021-07-01 --end 2021-07-01 --format=table
Country  2021-07-01  2021-07-02  2021-07-03
Ar       6           6           6
At       2           2           1
Au       6           6           3
Be       3           3           2
Bg       1           2           1
Br       14          14          10
Ca       12          13          12
Ch       3           3           2
Cl       0           1           1
Cn       3           2           2
Co       1           1           1
Cy       1           1           0
Cz       1           1           0
De       12          9           10
Dk       1           2           1
Es       9           10          8
Fi       1           1           1
Fr       8           8           8
Gb       30          27          21
Ge       0           1           1
Gr       4           4           3
Hk       1           1           1
Hu       3           2           2
Id       1           1           1
Ie       2           2           2
Im       3           3           3
In       12          12          11
It       6           9           4
Jp       1           1           1
Ke       1           1           1
Lt       1           0           0
Nl       4           4           6
None     50          48          31
Np       1           0           0
Nz       2           3           3
Pk       2           2           1
Pl       5           5           4
Pt       5           5           3
Qa       1           1           1
Ro       1           1           1
Ru       5           5           5
Se       8           7           6
Sg       2           2           2
Sk       4           2           2
Tr       6           5           6
Tw       5           5           5
Us       53          44          39
Uy       2           2           2
Vn       1           0           0

Example installed_base_by_operating_system:

$ snapcraft metrics my-snap --name installed_base_by_operating_system --start 2021-07-01 --end 2021-07-01 --format=table
OS                2021-07-01  2021-07-02  2021-07-03
Arch/             2           1           1
Centos/7          2           2           2
Debian/10         3           3           2
Elementary/5.1.7  2           2           2
Elementary/6      1           0           0
Fedora/34         1           1           1
Linuxmint/20.1    1           1           1
Manjaro/          1           2           1
Pop/20.10         1           1           1
Pop/21.04         1           1           1
Ubuntu/16.04      10          10          11
Ubuntu/18.04      68          66          77
Ubuntu/19.04      1           1           0
Ubuntu/19.10      1           1           1
Ubuntu/20.04      255         260         250
Ubuntu/20.10      9           9           7
Ubuntu/21.04      88          92         99
Ubuntu/21.10      2           2           5

Example installed_base_by_version:

$ snapcraft metrics my-snap --name installed_base_by_version --start 2021-07-01 --end 
Version  2021-07-01  2021-07-02  2021-07-03
2.4.3             1           0           0
2.4.4             1           1           1
2.4.5             4           4           4
2.5.0            28          28          16

Credentials Handling

The metrics APIs requires the package_metrics permission granted for given credentials. If the current credentials does not have this permission granted, the user will be prompted to login (again). If the user is not logged in, they will also be prompted to do so.

Note that any currently logged in user will not have this permission granted to their existing cached credentials made prior to this change. As such, they will require re-authentication to obtain it. Future logins will not require re-authentication as Snapcraft will request this permission during all future logins.

Error Handling

On error, Snapcraft shall exit with error:

Failed to query metric {metric!r} for snap {snap!r} with range {start}..{end}.

Snapcraft shall include additional details pertaining to error, if available.

Code changes

Store API

  • Add support for /dev/api/snap/metrics endpoint.

  • Add ACL support for package_metrics.

CLI

  • Add metrics command with support for table and json output formats.

When using --format=table, Snapcraft shall output a table-based interpretation of the returned data with columns for the requested date ranges with rows of the requested data series.

Use >

If the query returns with a “None” data point, it is replaced with a “-” to indicate zero (or not applicable depending on context).

1 Like

/me would love some of this info for snapstats ;-p Unfortunately none of this data is public :frowning: (I get that it is sensitive, I’m just pining for something I can’t have :-p)

3 Likes

You should try and become a collaborator to ALL the SNAPS :smiley:

2 Likes

Hmm, couldn’t there be an option for snap authors to make this information public? Maybe even opt-out, since the majority of snaps package FOSS projects?

I think there’d be too much potential for showing data publically to backfire.

Even though other repos/stores collect the same data and often much much more, what they show publically for a given application usually comes down to a generic number of installations and a rating. If the Snap Store suddenly starts showing what relatively looks like a lot vs the competition, its easy and unfair criticism.

In a world where e.g Google knows what apps I have installed, what websites I browse, my voice, my face, my friends, and my heart rate, the Snap Store would be the invasive one for knowing what country you’re downloading from, purely because it’s not something you’d usually see.

So while I think the data is minimally scoped and valuable to the developers, I’d imagine the public perception by users could easily be swayed by one malicious user on some public forum. “The Snap Store knows when you’re sleeping!”.

I see your point though I think if that were true it would already be a problem because the store website already displays the world map and a bar plot showing users by distribution (if the developer didn’t disable it) just without any quantities. Therefore, it is already clear to users that those metrics are collected.
In addition to that, developers can already publish the stats and some have done so (e.g. in blog posts). Hence, I don’t really see any harm in adding an option for making this data public.
Anyways, just a suggestion :slight_smile: