External repositories


#44

@cprov? Or is there someone else we can tag?


#45

@mickmack This is supported and you’ll see it being referred to as a “branded store”. It’s basically the same store and the same code, and even the same data, but it offers your devices a customized view of the world. This already works today and there are some well known companies relying on it.

This is also not in conflict with the points made above, as it doesn’t affect the usability points we care about, both for you and for the users of that device, and it’s also pretty simple to implement and support, because it’s not really a federated view, but rather a partial view of the same world.

I don’t know who’d be the best person to present those details to you, but @noise or @JamieBennett should point you towards the right direction.


#46

@niemeyer: I regard to your comment in the other thread proprietary code is a problem. Due to the terms I have to agree on to publish the proprietary code to the default store. I am not allowed to give Canonical:

5 Licence. You hereby grant to Canonical and its group companies, for the Term, a worldwide, non-exclusive licence to install, deploy, reproduce, and run your Apps for the purposes of testing and evaluation and to reproduce and distribute your Apps to end users through the Client Software. You hereby grant to Canonical and its group companies, for the Term, a licence to use any trade marks and branding you provide as part of the App in the Client Software and in associated uses such as use in marketing materials and advertising relating to the App, the Developer Site, or the Client Software.

as written in Ubuntu Store’s Terms of Service

Due to my non transferable licenses.

But if it is possible to host our repositories our selves in the Branded Store it might be a solution. But it sound like either the “multiple stores are dangerous or impossible to handle” argument or the “there is an other store named the Brand Store” argument will contradict the other.


#47

@mickmack I see, so you are not the author distributing the software, but the user trying to manage it locally on your own premises? Yes, there’s a solution for that in the works as well as part of the code clean ups I quickly discussed above. It still a branch of the main store rather than a completely independent store, but it’s supposed to allow you to make the snap available inside your premises without having to share its data with Canonical or anyone else. The details are still in the works, so you may need to share the digest inside an assertion so that we can make the overall system work as intended, but you won’t have to share the bits themselves at least.

Depending on the exact licensing terms, we might also adapt the terminology of the ToS so that you’re covered. I’m not the best person to discuss this, though. Perhaps @olli can help.


#48

What about this?


#49

Hello,

You may have seen my “External Repositories” forum post, which asked why Snap doesn’t support adding external repositories to the snapd program. There were 2 claims I would like to address:

  1. Adding External Repositories provides a bad user experience
  2. The Snap Store has open-source implementations available, and you can run your own snap store

The first point I would like to address is the claim that external repositories provide a bad user experience. This I would classify as mostly true. PPAs were a good experiment, but they have not been a good experience for the end user, and neither has adding repositories. But, the fact that previous attempts at repositories have been a bad experience does not mean the entire repositories concept should be thrown away, rather, it needs to be reborn for a new age of Linux packaging, and will be addressed later in this post.

The second point I would like to address is the claim that there are implementations of the Snap Store available which are open source, and you can run your own Snap Store. This is a half truth. There are open-source Snap Store attempts on GitHub, however, I have found none that have worked, being out-of-date or very limited in power. However, if there was more interest in this, this would not be such a problem. The second problem is that even if you can run your own Snap Store, you can’t install it on user’s machines. There is an override flag for environment variables to force Snap to use your store (which is not easy to do), but with that flag set, you can’t install snaps from the Ubuntu Snap Store at all. In other words, if you set up an alternative store, you are locked into that alternative store.

Now, what are the benefits of the current Snap design? Snap currently contains a very nice installation method, supports IoT, Server, and Desktop, and has many technical merits. The user experience of “snap install” or the Snap Store is very good, but flawed. Namely, Snap is a very Ubuntu-centric system right now in both appearance and function. Every distribution that uses Snap is forced to use the main “Ubuntu Store”. The “Ubuntu Store” is has all the snaps, and Ubuntu gets paid when you buy a snap. If you dare to override this by using the forced different store flag, you can’t install any snaps from the “Ubuntu Store” at all.

You might bring up Canonical’s “Branded Stores”, and this is where things, I believe, could really improve for the benefit of other distributions interested in using Snap while keeping a similar user experience and helping to relieve the fear of Canonical’s centralized design. Here is the proposal:

  1. Every distribution interested in Snap has the opportunity to run a branded store, which will have that distribution’s branding and imagery. If a Snap is purchased from a branded distribution store while running on Canonical’s servers, the distribution and Canonical split the 30% they pull off the Snap price 50/50. So if a snap cost $1.00 and was bought from a “Fedora Linux” branded store, Fedora gets $.15, and Canonical gets $.15. *
  2. Every distribution running a branded store should have the right to run that branded store on their own servers. This means that if someone downloads a snap from, say, a “Solus Linux” branded store and Solus ran their own servers, Solus was the one powering the servers. If a distribution is running their own server, they get all of the 30% cut from each Snap’s price, because they ran the servers.
  3. Updating snaps are symmetrical. If a developer updates a Snap on the Solus-branded store, it sends the update to the Ubuntu-branded store. If a developer updates a Snap on the Ubuntu-branded store, it updates on the Solus-branded store.
  4. If Ubuntu shows they are an excellent provider of Snaps, I can see how many small distributions would choose to leave their branded Stores in the hands of Canonical’s servers. This benefits Canonical and the distribution.
  5. If a distribution wants to sign up, they should be able to sign up no matter their distribution size as long as they aren’t, say, a single-person effort. In other words, users shouldn’t be able to claim they “run a distro” to save 15% on Snaps, but small distributions which actually are real (and have website, actual developers) should be able to sign up.
  6. Every distribution ships with it’s own store, and it is set using an environment variable flag for the Branded Store which already exists in current versions of Snap. This is a different flag than mentioned earlier, and only supports Branded Stores Ubuntu set up. Repositories, without the repository.

Comments?

  • All prices without obvious payment fees, taxes

#50

Ubuntu’s PPAs were the first, but not the only attempt to address this. SUSE, Fedora, and other lesser-known distributions have incrementally improved on this model. Fundamentally, the issue with extra repositories is that you have to give the user more trust. And this is something that doesn’t work in some cases. However, I believe it can be done well.

This is critical. While the UX is “better” with a single store model, it doesn’t scale very well across the known universe of Linux systems. And if, for example, Fedora wants to provide a lifecycle and infrastructure around offering Snaps (not saying they do or don’t, but just roll with me here), then Fedora needs to be able to run everything themselves to provide the level of assurance they try to offer now with RPMs. They’re trying this now with Flatpaks, and we’ll see how that goes.

I want to tweak this idea a bit: I don’t like environment variables for configuring this. They’re a huge risk and in my view, it’s a stupid way to configure things. I have enough experience with boneheaded things with Docker because of this way of configuring the Docker daemon, and even though Debian popularized it with the /etc/default/foo files, it’s a really bad way to configure things because it depends on what’s running the outer environment. The Snappy daemon absolutely should read from a proper config file and process the configuration like most applications do. That’s less brittle and less likely to be accidentally screwed up by humans. It also allows for configuration validation and other nice things.

Now, here’s my addendum to this proposal: Snappy stores should have a means of establishing a federation of trust. For example, the Canonical store would trust and acknowledge the Fedora store and vice versa. This does add a slight degree of complexity, but it allows for things running on their own infrastructure to be trusted by the mainline code. And the current single-store model (which is not designed to handle conflicts at all) can be extended a little further to ensure that there’s still no such thing as conflicting snaps.

There’s a couple of ways to handle this:

  • Namespace the snap install with the store it came from (like how Docker and Flatpak do it)
  • Rely on the global index to prevent conflicts from happening

I think the latter is probably just not doable beyond maybe four major stores, but the former would scale quite well.

This would also make it possible for companies like Red Hat and SUSE to actually have a business model around Snaps, which would enable further interest and development in the ecosystem.


#51

While discussing these ideas is completely fair, please note that the most important reason why we won’t focus on federated stores ourselves right now is because that’s a distraction. It complicates the design significantly and creates new usability issues which we know too well from other systems.

In other words, spending time making plans and then designing a solution and then implementing a solution for Fedora to have a snap store is a huge waste of time until Fedora wants to have a snap store in the first place. That’s why focus is important. We need to rely on the few full time developers we have to focus on making sure other people will want to do that in the first place.


#52

The catch-22 here is that Fedora doesn’t want to invest in snaps unless it’s possible for such a thing in the first place. @zyga and I have had discussions with Matthew Miller (Fedora Project Leader) about it, and that was one of the sticking points. The other points were that it’s not possible to build snaps based on Fedora and have snapd run fully Fedora all the way down (aka, a core snap built on Fedora infrastructure for a potential Snappy Fedora Modular Edition).

The base snaps thing will resolve one of those issues, and I’m not really sure how to deal with the core snap issue for making a Snappy Fedora Modular system. All that would be left is the store aspect.


#53

Indeed this does feel like catch 22. I can only hope that we are seen as honestly working on addressing those issues and as base snaps become available and snapcraft work is started all the perceived roadblocks go away, one by one.

I also honestly hope that if there are people that are interested in snaps and the functionality that they can bring to a system can work with us on implementing missing things. We really need people coding, not just talking about it.


#54

I completely believe that this is one of N points that people will put their finger when they want to find reasons to not actually engage. This is a widespread reaction to almost anything… “Oh, we would like to engage but you see, there are those N reasons why we won’t.”. When people do want to engage, though, the interaction is different. It comes via direct communication, and it’s expressed as “Look, this is the sticking point. Let’s sort it out and we will engage in those terms.”. It’s a story with two interested parties, instead of one interested party heavily vested and another one watching the work.

So, we have a non-trivial number of cases in the latter stage, and those are the ones we’re putting the most work on. Unsurprisingly (to me at least), all of the cases we have in that stage today care about snaps working well for them and for their users, rather than who’s spending money to run and maintain the servers.

In fact, let me get back to helping those people out. :slight_smile:


#55

I think that if the snappy store wasn’t centralised in Canonical’s hands then Fedora would be much more open to the snappy project. This decision will block more widespread use of snappy. But yes, of course there are other things to work on…


#56

That’s what I just responded to.


#57

Yeah sorry I should’ve liked one of the other comments rather than making my own :frowning:
The other approach would be to say something like ‘if we made it easier to open your own snappy store would you be willing to support snappy in Fedora out-of-the-box’ to see if it’s something worth doing…
You’re right that there may be other blocking issues.


#58

In the first place there should be more simple solution which I suppose will suit all without writing complex logic in snapd:

  1. ubuntu-image should be able to change the default store web address from search.apps.ubuntu.com to any provided from command line.
  2. There should exist some open source snap store (to be implemented) that will serve snapd requests.
  3. It serves only those snaps that contain in that store.
  4. For all others it acts like a proxy to default Ubuntu Store.
  5. It can override any snap from Ubuntu Store even the core.snap if someone wants.
  6. It can have its own while lists or black lists to mask Ubuntu Store’s snaps if desired.

This proxying concept is the existing logic from noise’s simple store. It is simple and very efficient. Simple store is too simple now it should be rewritten all over again. But this is the question not related to snapd at all.

Complex relationship between different stores is not required at all.

The only thing to do from the snapd perspective now is to have a tooling to change the default store and add additional cryptographic signatures of the new store so that it’ll be trusted. Is it that complex to implement such a tool? I doubt it.

You already have the possibility to change the main store to brand store. Why not allow to change to external? I suspect some marketing/sales restrictions imposed.


#59

@denis Because the branded store is actually just a trivial mapping on top of the exact same store, code, deployment and all. That also means the exact same snaps are available to the branded store, if whoever manages the branded store so desires. If you replace the store with a different store, then you cannot use the same snaps anymore. It’s a split and conflicting world rather than a federated world. It’s indeed very cheap, but unfortunately that’s true both in terms of cost and in terms of result, which is why we don’t support it.


#60

This would be fantastic, but is it possible to implement in ubuntu-image right now? The snap store URL seems to be baked into the core snap, which is then signed by canonical. I am not sure what means the ubuntu-image tool could use to affect that configuration unless the core snap exposes a way to do that. (Outsider’s perspective here - if there is something I am missing, I would love to learn about it.)


#61

Well, I’m here, as a member of the Fedora community, trying desperately (and sometimes, what feels like in vain) to improve the situation. I don’t like how you’re posturing here, because it makes it look like that you’re not actually interested in making snaps attractive for anyone…

Sometimes, I wonder whether it’s even worth it when I get responses like that constantly…


#62

@Conan_Kudo Both Zygmunt and I are talking to you as the actual well known snapd and Fedora contributor that you are. We all appreciate your efforts a lot, as we hope you appreciate ours into making sure things work well for you. The fact we’d like to have more people collaborating and that we won’t detour our focus and perform a major investment into federating stores without more clear engagements is not about you.


#63

@lance, indeed what is hardcoded in the core.snap we can’t deal with it.
But! snapd could just look first in some independent place. In the gadget snap for example. This is the perfect place for that.
The gadget snap should define the base URL of a store. And all specific store assertions/signatures. All that snapd should do is just obey to these credentials, that’s all that necessary.

Why’s that? Not true at all.

It seems that you’re trying to confuse and obscure everything in that field.

So to conclude:

  1. gadget snap defines store base URL and its keys
  2. snapd obeys that info
  3. That’s all.

All additional functions is up to the external store. This external store could even resign snaps from Ubuntu Store with his own cryptokeys to match those defined in gadget snap. (In the case that you continue to insist that it’s oh-oh-oh so impossible to implement an array of dictionaries defining multiple stores) The first candidate for this re-sign procedure is the core snap of course.