Incident Report

Summary

Beginning on 18 October 2024, users of the Gmail application for iOS experienced increased application crash rates for a duration of 1 day, 5 hours, 45 minutes. To our Gmail for iOS users who were impacted by this issue, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The Gmail iOS application uses a Google-wide SSO client library to authenticate all accounts on a given device with a backend authentication server. This includes a lookup of account IDs and their corresponding email addresses. Email addresses are returned from the authentication server normalized to avoid potential formatting conflicts. For example, “TestUser@gmail.com” would be normalized to “testuser@gmail.com.”

A recent server-side feature release introduced an issue where the backend authentication server began returning non-normalized email addresses. This resulted in a formatting mismatch where the Gmail iOS application was unable to match account IDs to the normalized email addresses stored in memory when the application was initially launched, causing the application to crash.

Remediation and Prevention

Google engineers were alerted to the issue via internal monitoring on 18 October 2024 at 11:58 US/Pacific and immediately started an investigation.

Engineers rolled back two recent updates to identity endpoints used by Gmail iOS to mitigate impact on 19 October 2024 at 17:15 US/Pacific.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Make Gmail iOS more resilient to identity endpoint changes
• Limit these identity changes to in-scope users
• Improved attribution of future backend production changes

Detailed Description of Impact

Between 18 October 2024 at 11:30 US/Pacific and 19 October 2024 at 17:15 US/Pacific, approximately 2.6M users of Gmail for iOS experienced foreground crashes on application startup.

Impact was limited to the Gmail application for the iOS platform. Users did not face any errors while using Gmail in other platforms, including Android, Web interfaces.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 18 October 2024 11:30

Incident End: 19 October 2024 17:15

Duration: 1 day, 5 hours, 45 minutes

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

The Gmail application for iOS experienced an elevated crash rate for a duration of 1 day, 5 hours, 45 minutes.

From preliminary analysis, the root cause of the issue was two changes to identity endpoints used by Gmail iOS that caused it to crash.

Google will complete a full incident report in the following days that will provide a full root cause.

Customer Impact:

Approximately 1 million Gmail for iOS platform users had foreground crashes on start of the application during this period of impact.

Impact was limited to the Gmail application for the iOS platform, and users did not face any errors while using Gmail in other platforms, including Android, Web interfaces.

Summary:

Gmail iOS application crashes

Description:

We are experiencing an issue with Gmail for iOS.

Our engineering team is working on multiple streams of investigation to mitigate the issue at the earliest.

We will provide an update by Monday, 2024-10-21 10:00 US/Pacific with current details.

Diagnosis:

Customers using Gmail for iOS may experience application crashes.

Workaround:

The affected users should access Gmail through the web interface.

We are experiencing an issue with Gmail for iOS.

Our engineering team is working on multiple streams of investigation to mitigate the issue at the earliest.

We will provide an update by Friday, 2024-10-18 22:00 US/Pacific with current details.

Symptom: Customers using Gmail for iOS may experience application crashes.

We are experiencing an issue with Gmail for iOS.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2024-10-18 18:00 US/Pacific with current details.

Symptom: Customers using Gmail for iOS may experience application crashes.

We are experiencing an issue with Gmail for iOS.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2024-10-18 16:00 US/Pacific with current details.

Symptom: Customers using Gmail for iOS may experience application crashes.