Issue preventing users from signing in via third party auth systems

Incident Report for Pugpig

Postmortem

We released a code change to our Distribution Server Fleet at 9:00 am. However, one of our API servers had previously failed to upgrade correctly to PHP 7.3, so a trailing comma in the code release (which is only valid in 7.3) caused errors for all users hitting this server. We rolled back the code change to fix the issue quickly. We have now upgraded all the servers, and the updated code is running successfully. As our development and staging servers had all successfully upgraded to 7.3, this issue was not caught earlier.

Posted Sep 17, 2020 - 09:57 UTC

Resolved

This issue has now been resolved. We'll follow up with a more detailed explanation shortly. This issue would have affected all new users trying to sign in to their apps for about 20 minutes - they will have seen an error saying "Request failed with status code 500". Users that were already logged in would not have been affected.

Posted Sep 17, 2020 - 09:19 UTC

Identified

We've identified the issue and are rolling back to a previous version.

Posted Sep 17, 2020 - 09:15 UTC

Update

We are continuing to investigate this issue.

Posted Sep 17, 2020 - 09:07 UTC

Investigating

We're aware of an issue that is preventing most or all users from signing in via third-party auth services. We're actively investigating this issue.

Posted Sep 17, 2020 - 09:06 UTC

This incident affected: Pugpig API (Third Party Auth Gateways).