System is stable, queues on the Mac stacks are slowly but steadily decreasing. There are no queues on the Linux/Android stacks.
Posted Oct 22, 2018 - 21:36 UTC
Update
Queue depleted on the Linux/Android stacks. macOS stacks are still working through the build queue but that's decreasing as well.
Posted Oct 22, 2018 - 20:40 UTC
Update
GitHub still processing and firing webhooks, which keeps queueing builds. System remains stable but due to the high build queue builds take a while to start.
Posted Oct 22, 2018 - 19:51 UTC
Identified
GitHub had an all-day issue (https://status.github.com/messages) and during that they disabled webhook delivery. They just re-enabled the webhook delivery and are now sending all their queued webhook events from the last couple of hours. This means that an unusually large amount of builds are triggered at the same time in our system. We try our best to process these builds as fast as we can, scaling our infrastructure where we can, but it might take some time for our system as well to process all these events and the builds these trigger.