Distribution Service suffering intermittent failures

Incident Report for Pugpig

Resolved

This incident has been resolved.
Posted Mar 12, 2025 - 08:55 UTC

Monitoring

A new AWS Redis Cache has been brought online and is primed. According to our monitoring, service has returned to normal. However, we will keep an eye on this.
Posted Mar 11, 2025 - 23:33 UTC

Investigating

The Distribution Service is currently suffering intermittent failures. The incident started at around 21:40 UK time. The root cause is related to an issue with an AWS Redis Cache.

The issues will affect:
- new users trying to log in to the apps
- anyone trying to perform a search in the apps

New content will also process more slowly than usual

We are currently deploying new instance and expect this to be resolved in the next 30 minutes at most.
Posted Mar 11, 2025 - 23:23 UTC
This incident affected: Pugpig Distribution Service (Distribution Service Access) and Pugpig API (Pugpig Search API, Third Party Auth Gateways).