High latency issues on our Xcode stacks

Incident Report for Bitrise

Resolved

Our Infra team deployed a fix and resolved the latency issue. Build performance is back to normal and there aren't any queues on the build machines.
Posted Jun 18, 2019 - 17:54 UTC

Update

Currently the queues are down, our Infra team deployed a fix and the latency decreased but we are still seeing some performance degradation on the Xcode 10.2 stack. Our Infra team is working on further decreasing the latency and resolving the issue as soon as possible, in the meantime a possible mitigation of the performance degradation would be to use the Xcode Edge stack that also has the same version of Xcode installed as the Xcode 10.2 stack.
Posted Jun 18, 2019 - 16:38 UTC

Update

We are currently experiencing queues on our Xcode 10.2 machines, that is related to the ongoing high latency issues. Our Infra team is working on resolving the issue as soon as possible.
Posted Jun 18, 2019 - 13:12 UTC

Investigating

We discovered latency issues on our Xcode 8.3, Xcode 10.1, Xcode 10.2, Xcode 11.0 stacks that can cause builds to run slower. Our Infra team is currently investigating the issue.
Posted Jun 18, 2019 - 11:49 UTC