Sporadic virtual machine creation issue on Linux stacks

Incident Report for Bitrise

Resolved

This incident has been resolved.

Posted Sep 10, 2018 - 17:13 UTC

Monitoring

It seems there was a temporary issue on GCE which affected our virtual machine cleanup.

We found a couple of virtual machines which got stuck in running state. If you see that, abort the build and do a rebuild, alternatively our system will abort it eventually but that will take some time.

We also fixed a related quota limit configuration issue, as due to the original issue we ended up having too many VMs and reached our quota.

System is fully functional now, we're not aware of any new issues on the Linux builders, but we're still monitoring.

Posted Sep 10, 2018 - 16:52 UTC

Investigating

We detected sporadic virtual machine creation issues on the Linux stacks, investigating the issue.

Posted Sep 10, 2018 - 16:20 UTC