High queue on the Xcode 10.1 stack
Incident
Report for Bitrise CI
Resolved
The queues are back to normal and the builds should be starting as usual.
Identified
Our infra team deployed a fix to our production servers and we are seeing an improvement on build performance and build processing. Unfortunately, we are still seeing queues on the Xcode 10.1 stacks. In the past couple of days our infra team deployed multiple possible solutions to our production system and have also been working on upgrading our machine park along with the underlying storage layer itself. Sadly the previous configuration changes only managed to ease the issue with the build times and queues, but didn't manage to fully solve them. Currently we are working on adding new machines and additional storage to our machine park as well as reworking our whole build load balancing logic to be able to handle the increased load generated by the current traffic.
For more in-depth information on the cause of this issue and the steps we're taking to ensure it won't happen again, please check out our blogpost on this topic: https://blog.bitrise.io/update-mac-infrastructure-queues-march-2019
We are really sorry for the inconvenience caused by this issue, but are doing everything in our power to ensure performance is where it needs to be as soon as possible.
For more in-depth information on the cause of this issue and the steps we're taking to ensure it won't happen again, please check out our blogpost on this topic: https://blog.bitrise.io/update-mac-infrastructure-queues-march-2019
We are really sorry for the inconvenience caused by this issue, but are doing everything in our power to ensure performance is where it needs to be as soon as possible.
This incident affected: Build Processing (OS X Stack (iOS)).