Significant build queue on MacOS stacks, due to recent GitHub and other service issues and delayed webhooks

Incident Report for Bitrise

Resolved

This incident has been resolved.

Posted Oct 21, 2016 - 16:03 UTC

Monitoring

GitHub and other popular web services had network (DNS) issue recently, due to an upstream DDoS attack.

Webhooks were not delivered for hours, but now that GitHub is recovering it started to send the previously missing webhooks in large amount, which causes significant queue right now on our MacOS stacks.

The issue will recover automatically over time (shouldn't take too long), we're monitoring the system to make it sure the queues are processed as fast as possible.

Posted Oct 21, 2016 - 15:10 UTC