Major (network) outage on the Mac stacks
Incident
Report for Bitrise
Resolved
We added additional resources to the build cluster and the build queue is now successfully depleted and back at normal levels. Build processing remained stable.
Update
Build processing remained stable and our infrastructure provider also confirmed that network connection is restored now, although it might be slower than usual.
Due to the outage there's a significant queue of builds waiting to be processed, so you might still have to wait a bit before the build would actually start ("waiting for a worker").
Due to the outage there's a significant queue of builds waiting to be processed, so you might still have to wait a bit before the build would actually start ("waiting for a worker").
Update
First batch of builds are processing now, we're still closely monitoring the system and waiting for an update from our infrastructure provider.
Update
Root cause of the issue is identified, it is a major power failure.
We'll post an update as soon as we gain additional information.
We'll post an update as soon as we gain additional information.
Update
The network issue also affects the processing of the builds on the Mac stacks, we're working on mitigating the issue as fast as possible. In the meantime you will most likely see your build "get stuck" in the "waiting for worker" state.
We're terribly sorry and doing everything we can to resolve the issue as soon as possible.
We're terribly sorry and doing everything we can to resolve the issue as soon as possible.
Identified
We detected intermittent network issues on our Mac stacks, which was verified by our infrastructure provider, they are working on the resolution.