High build queue on the Xcode 10.1 stack
Incident Report for Bitrise CI
This incident has been resolved.
Posted Mar 15, 2019 - 12:39 UTC
Queues cleared out.
Posted Mar 15, 2019 - 11:46 UTC
We are experiencing high queues on our Xcode 10.1 stack.

Unfortunately, we are still seeing queues on the Xcode 10.1 stacks. In the past couple of days our infra team deployed multiple possible solutions to our production system and have also been working on upgrading our machine park along with the underlying storage layer itself. Sadly the previous configuration changes only managed to ease the issue with the build times and queues, but didn't manage to fully solve them. Currently we are working on adding new machines and additional storage to our machine park as well as reworking our whole build load balancing logic to be able to handle the increased load generated by the current traffic.

For more in-depth information on the cause of this issue and the steps we're taking to ensure it won't happen again, please check out our blogpost on this topic: https://blog.bitrise.io/update-mac-infrastructure-queues-march-2019

We are really sorry for the inconvenience caused by this issue, but are doing everything in our power to ensure performance is where it needs to be as soon as possible.
Posted Mar 15, 2019 - 10:44 UTC
This incident affected: Build Processing (OS X Stack (iOS, Xamarin, ...)).