Monitoring - Due to unusually high amount of concurrent builds there's a minor build queue on the macos builders. System is stable and processing the builds, and we're working on adding more capacity.
Sep 13, 16:07 UTC

Monitoring - System remained stable, every part working as expected. Builds which were running during the incident might be in a hanging state (still running on the UI, but no logs are generated). Our system will abort these automatically in the next two hours, if you'd find one you can also abort it manually and then initiate a rebuild, that will go through immediately.
Sep 12, 18:14 UTC

Monitoring - It seems there was a temporary issue on GCE which affected our virtual machine cleanup.

We found a couple of virtual machines which got stuck in running state. If you see that, abort the build and do a rebuild, alternatively our system will abort it eventually but that will take some time.

We also fixed a related quota limit configuration issue, as due to the original issue we ended up having too many VMs and reached our quota.

System is fully functional now, we're not aware of any new issues on the Linux builders, but we're still monitoring.
Sep 10, 16:52 UTC