Elevated Queue Times on Linux Machines
Incident
Report for Bitrise
Resolved
This incident is now resolved, and impacted services are now operating without issues.
Our teams identified the root cause, released a fix, and were monitoring our systems continuously to ensure the mitigation is safe. We'll conduct an internal post-mortem to understand what happened and learn how to avoid a similar outage in the future.
Thank you for your patience and understanding. If you encounter any issues, please reach out to our Support team:
https://support.bitrise.io/en/
Our teams identified the root cause, released a fix, and were monitoring our systems continuously to ensure the mitigation is safe. We'll conduct an internal post-mortem to understand what happened and learn how to avoid a similar outage in the future.
Thank you for your patience and understanding. If you encounter any issues, please reach out to our Support team:
https://support.bitrise.io/en/
Identified
We identified the root cause of elevated build queues on Bitrise. Some users may experience longer-than-usual wait times for builds to start, as queued jobs take longer to be picked up.
Services:
Build start times and queue wait times; build execution itself remains normal once a worker is allocated.
Scope:
Queues on standard, elite, and elite-xl machines
Severity:
SEV-2
What we’re doing?
Our Engineering teams identified the root cause and are now actively working on mitigations to restore stability across all affected services. We will perform validated mitigations as soon as they are determined safe.
When will the next update be available?
You can expect a status update approximately 1 hour from now.
Services:
Build start times and queue wait times; build execution itself remains normal once a worker is allocated.
Scope:
Queues on standard, elite, and elite-xl machines
Severity:
SEV-2
What we’re doing?
Our Engineering teams identified the root cause and are now actively working on mitigations to restore stability across all affected services. We will perform validated mitigations as soon as they are determined safe.
When will the next update be available?
You can expect a status update approximately 1 hour from now.
Investigating
We are currently investigating elevated build queues on Bitrise. Some users may experience longer-than-usual wait times for builds to start, as queued jobs take longer to be picked up.
Services:
Build start times and queue wait times; build execution itself remains normal once a worker is allocated.
Scope:
Queues on standard, elite, and elite-xl machines
Severity:
[SEV-2]
What we’re doing?
Our Engineering teams are actively investigating causes, reviewing worker pool utilization, autoscaling events, recent deploys, and any upstream provider issues. We are analyzing metrics and logs to identify bottlenecks like unusually high build volume, throttling from third-party providers, etc. We are working on immediate actions to remediate this issue. We will perform validated mitigations as soon as they are determined safe.
When will the next update be available?
You can expect a status update approximately 1 hour from now.
Services:
Build start times and queue wait times; build execution itself remains normal once a worker is allocated.
Scope:
Queues on standard, elite, and elite-xl machines
Severity:
[SEV-2]
What we’re doing?
Our Engineering teams are actively investigating causes, reviewing worker pool utilization, autoscaling events, recent deploys, and any upstream provider issues. We are analyzing metrics and logs to identify bottlenecks like unusually high build volume, throttling from third-party providers, etc. We are working on immediate actions to remediate this issue. We will perform validated mitigations as soon as they are determined safe.
When will the next update be available?
You can expect a status update approximately 1 hour from now.
This incident affected: ⚙️ Builds (CI).