Issue with Linux/Android stacks which can delay virtual machine creation
Incident
Report for Bitrise CI
Resolved
Linux builds are running again as expected.
Update
Builds are starting properly, we don't experience queues anymore. Some builds might still report "Waiting for worker", those will be started soon.
Monitoring
We have applied the fix to all control machines. There is no more build queue for Elite users; Standard build queue is slowly being resolved as well.
Update
We have identified the root cause and working on the fix right now. The Elite build queue should already be getting shorter.
Update
The issue now affects our other region we use as well, we're working on fixing the issue as fast as we can. Sorry for the inconvenience.
Update
A temporary fix was deployed, new builds should now start properly, but previously started builds might be in an aborted state with the note "failed to connect to the right build environment". Those builds will start eventually but might take some time due to re-routing between clusters.
Identified
We identified the problem and working on resolution.
Investigating
High virtual machine creation error rate detected, we're working on identifying the cause.
This incident affected: Build Processing (Linux Stack (Android, web projects, ...)).