Queues on our Xcode standard and elite stacks
Incident
Report for Bitrise CI
Resolved
The queues are completely down for every stack and the system remained completely stable and operating on full capacity. We keep monitoring our system for any out of the ordinary behaviour.
Update
Currently the queues are down for most of our stacks, except for the Xcode 11.1 Elite stack that had the largest number of builds queued and for which the VMs are still processing these builds. Our Infra team is still monitoring our build system and everything is still functioning on full capacity, our logs aren't showing anything out of the ordinary. Thank you for your patience during this incident and while our system is processing the queued builds
Update
The system is still running on full capacity and currently processing steadily the builds in the queue. Our monitoring tools aren't showing anything out of the ordinary.
Monitoring
The deploy finished and all clusters are up and running on full capacity. Due the previously queued builds the queue time is going down slowly but steadily. Our Infra team is monitoring the system to make sure that everything is functioning correctly and that the queues continue to decrease.
Update
Our Infra team is working on the deployment of the fix. Currently we are seeing the queues steadily decrease, but due to the large amount of builds that were originally in the queue it would take longer until the queue times would decrease to the normal amounts. Thank you for your patience on this issue!
Update
Our Infra team is continuing the deploy of the fix to the other parts of our system as well. We are monitoring the components that currently have the fix deployed and the metrics are showing the expected improvements. We are working on deploying the fix to the whole system as soon as possible, thank you for your patience on this issue!
Identified
Our Infra team is in the process of deploying a fix. Currently we are seeing machines starting up on the parts of our system that already have the fix applied and the team is working on deploying to the remaining parts of the system as well. Due to the fix it's possible that some of the builds and the related logs would be lost. We are really sorry for the inconvenience caused by this and we are working on rolling out the fix to the whole system as soon as possible.
Update
Currently the builds fail to start up on our Xcode and VS4Mac stacks. Our Infra team is currently working together with our hosting provider to identify the issue and create a fix for it as soon as possible.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating this issue.
Identified
We are currently experiencing unexpected queues on our Xcode and VS4Mac Standard and Elite stacks. Our infra team is currently investigating the situation.
This incident affected: Build Processing (OS X Stack (iOS)).