Store offline - 2018-09-26 - RESOLVED

Due to a database server failure, the store is currently offline. We’re working on it.

3 Likes

After some manual recovery, we have restored the databases in read-only mode, such that API requests and basic snapd commands are functional.

The dashboard site is down whilst we continue to recover completely.

Thank you for your patience and understanding :heart:

1 Like

I can now confirm that full functionality in the store has been restored.

snapcraft pushes, releases and automatic builds from the build service should now be working as expected.

Please let us know if you see any further issues.

1 Like

Well done, everyone involved :slight_smile:

I’m rather curious what happened that caused such a complete failure…

2 Likes

A hardware failure took out the primary DB host. We failed over successfully at first to the replica, but then a secondary problem took out the replica. We then restored the replica from last backup and got the API services back up in read-only mode as we needed to try to recover or at least assess the state of replication from the primary before going read-write to avoid any data loss. Getting the primary repaired and booting again took some time, but once that was up we were able to determine the replica logs were complete and brought the services back to read-write against the replica, now primary. And we are now cleaning up to resync the original primary as the new replica.

5 Likes

it’s nearly beer o’clock! a well deserved evening cool-down is in order, methinks.