Queues on our Xcode 11.3 Elite stacks
Incident
Report for Bitrise CI
Resolved
Queue times should be returning to acceptable levels.
Update
Our current MacOS infrastructure setup requires us to manually balance the available resources for specific stacks for the week ahead, every weekend. For the second week of 2020, we failed to anticipate the rate at which some teams would migrate from Xcode 11.2 to 11.3, leading to queues on our Elite Xcode 11.3 stacks while Elite 11.2 stacks sat idle.
We wanted to let you know that we will do an “aggressive” reconfiguration this weekend to ensure we have the right capacity allocation configuration in place for 2020 Week 3, and we’ll roll this out during this weekend.
Our long term plans to mitigate this issue include improving the software that’s responsible for managing the capacity allocation, as well as finalizing a second datacenter for our Mac build infrastructure to de-risk significant configuration changes.
Again, we apologize for any inconvenience this may have caused. Thank you for your continued patience.
We wanted to let you know that we will do an “aggressive” reconfiguration this weekend to ensure we have the right capacity allocation configuration in place for 2020 Week 3, and we’ll roll this out during this weekend.
Our long term plans to mitigate this issue include improving the software that’s responsible for managing the capacity allocation, as well as finalizing a second datacenter for our Mac build infrastructure to de-risk significant configuration changes.
Again, we apologize for any inconvenience this may have caused. Thank you for your continued patience.
Identified
We are currently experiencing higher-than-average queues on our Elite Xcode 11.3 stacks. Our infrastructure team is aware of the issue and is currently working to mitigate it. Thank you for your patience!
This incident affected: Build Processing (OS X Stack (iOS)).