Long waiting times on M4 Pro machine (queue) builds
Incident
Report for Bitrise
Resolved
Wait times have now dropped to approximately 30 seconds across most queues. This incident has been resolved.
Update
We’re continuing to address the elevated build queue times, but wait times have now dropped to approximately 6-7 minutes across most queues. Our engineering team is monitoring the queues closely and is working on full remediation.
We’ll keep you posted with further updates. Thank you for your patience.
We’ll keep you posted with further updates. Thank you for your patience.
Identified
We identified the root cause of elevated build queues on M4 Pro. Some users may experience longer-than-usual wait times for builds to start, as queued jobs take longer to be picked up due to the Cloudflare outage experienced earlier today, which has resulted in a larger number of builds being run at this time.
Services:
Build start times and queue wait times; build execution itself remains normal once a worker is allocated.
Scope:
M4 Pro Machines
Severity:
SEV-2
What we’re doing?
Our Engineering teams identified the root cause and are now actively working on mitigations to restore stability across all affected services. We will perform validated mitigations as soon as they are determined safe. We will be adding more M4 Pro capacity by next week.
When will the next update be available?
You can expect a status update approximately 1 hour from now.
Services:
Build start times and queue wait times; build execution itself remains normal once a worker is allocated.
Scope:
M4 Pro Machines
Severity:
SEV-2
What we’re doing?
Our Engineering teams identified the root cause and are now actively working on mitigations to restore stability across all affected services. We will perform validated mitigations as soon as they are determined safe. We will be adding more M4 Pro capacity by next week.
When will the next update be available?
You can expect a status update approximately 1 hour from now.
This incident affected: ⚙️ Builds (CI).