Mac infra Elite machines: elevated build restart rate

Incident Report for Bitrise

Resolved

This incident has been resolved.
Posted Apr 19, 2018 - 16:34 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 19, 2018 - 14:31 UTC

Identified

The hosts are back online, we're back at 100% capacity in the Elite clusters. We're still performing a review to ensure the issue does not happen again and that everything's working as expected.
Posted Apr 19, 2018 - 12:30 UTC

Update

We moved a couple of stand by machines into the Elite clusters, so builds should go through now but might be a bit slower than usual. We're working with our infrastructure provider to restore all the Elite machines to get back to 100% capacity.
Posted Apr 19, 2018 - 11:55 UTC

Investigating

Due to a sudden issue with multiple host machines used for the MacOS Elite stacks builds targeting those machines / stacks might be restarted multiple times until a suitable host is found.

We're working on fixing the issue ASAP, sorry for the inconvenience.
Posted Apr 19, 2018 - 11:29 UTC