Distribution content ingestion from Express delays
Incident Report for Pugpig
Postmortem

We’re are following up on the incident that occurred yesterday. At around 5:00 am UTC we had an unexpected influx of content. This caused our Bulk Processing queue to only process content extremely slowly. At around 10:00 am is was returning to normal and by 12:00 am it was fully restored. We have discovered the root cause of the slow down caused by this influx (an extremely inefficient database query) and have rolled out a fix to prevent this particular issue reoccurring.

Most customers would not have been impacted if their content was processing on the normal priority queues. However, some customers that are updating many editions and timelines repeatedly would have had content delayed in these slower queues. We are reaching out to these affected customers individually to determine if we can improve the frequency of updates.

Please do get in touch with our support team if you did experience delays and we are not already discussing these with you.

Thank you for your understanding, and we appreciate your patience as we work to improve our systems.

Posted Dec 04, 2024 - 14:33 UTC

Resolved
This incident has been resolved.
Posted Dec 03, 2024 - 15:46 UTC
Monitoring
We believe that content updates are now processing as usual but we are continuing to monitor.
Posted Dec 03, 2024 - 13:33 UTC
Investigating
We are currently investigating delays with content updating through the Distribution Service from Pugpig Express.
Posted Dec 03, 2024 - 09:39 UTC
This incident affected: Pugpig Distribution Service (Distribution App Content Ingestion).