As more developers rely on Bitrise and as those developers trust our platform to handle more of the tasks involved in testing and deploying their apps, our responsibility to ensure that always works, grows as well. In that regard, we failed you and we’re sorry.

What Happened

On September 12th around 16:00 UTC, as many of you were preparing for the Apple Event, we experienced an issue with our build system - actions related to our virtual machines started to fail. After some digging, we found that the issue was caused by the storage volume under the database. The disk was full and the automatic measures pruning unnecessary information from the database weren’t aggressive enough to prevent this problem. After we extended the size of the volume and restarted the appliance, the API came back online.

How Did We Get Here?

We’re currently performing a root cause analysis to find that out. You’ll know that we experienced similar issues on August 14th, which prompted us to institute manual and automatic checks to catch early warning signs. These measures failed to flag this occurrence in the time we needed to be able to prevent the problem from escalating, though.

To ensure we do better, we are investigating the underlying cause, its impact and additional preventive measures. We will share our findings with you through https://status.bitrise.io/

Added Transparency

Starting today, we commit to being extra transparent. This means regular public posts - including our notes and insights - as our investigation and our plans progress, but also a commitment to post mortems on issues with customer impact going forward.

For now, we want to thank you for your patience, your support and your trust as we get back to helping you build amazing apps.