INCIDENT REPORT

SUMMARY

On December 8th, Google Calendar experienced a service degradation focused in Europe but also extending to other areas for a duration of 1 hour and 40 minutes. We sincerely apologize for the inconvenience this incident may have caused to our customers and we are taking steps to ensure this doesn’t happen again.

ROOT CAUSE

The majority of Google Calendar’s core user data resides in our own Spanner database. Spanner databases have several layers of isolation. To improve reliability, the back end Spanner databases which serve Google Calendar, were partitioned and split into isolated groupsets. In order to provide further resiliency and fault tolerance, Google decided to further separate the Calendar data into allocations.

During the process of migration of data into allocations, Google inadvertently pushed the Spanner servers over the limit in terms of resources. While migrations have brought Spanner servers over the limit before, the size of the data being migrated was larger which caused several hours of queueing which used more memory. This eventually created a cascading effect and the servers ran out of memory.

REMEDIATION AND PREVENTION

Google was alerted to the issue on Wednesday December 8, 2021 00:46 and immediately started an investigation. At 01:25 the ongoing allocation migration was paused. At 01:39 Google Engineers increased resources for Cloud Spanner servers and the impact was mitigated at 02:20.

Google is taking the following steps to ensure that this type of failure does not happen again:

• Immediately increase Spanner Servers resources in partitioned allocations to achieve the same capacity across all regions.
• Implement an auto-scaling mechanism which would automatically resolve any resource usage growth before it becomes a problem.
• Improve Spanner’s behavior when the resource pressure is extremely high.

DETAILED DESCRIPTION OF IMPACT

On December 8th, 2001 from 00:40 to 02:20 US/Pacific:

Google Calendar

Approximately 1.5% of Google Calendar customers globally (mostly focused in Europe) experienced HTTP 500 errors when using the Google Calendar interface.

Google Calendar API

Approximately 10.4% of Google Calendar customers using the public APIs (APIv3, CalDAV) experienced HTTP 500 errors which would include 3rd party email and calendar applications which used the public APIs.

Google Meet

Up to 10K Google Meet users were unable to join meetings.

Mini Incident Report - Full Incident Report To Follow

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 08 December 2021 00:40

Incident End: 08 December 2021 02:20

Duration: 1 hours, 40 minutes

Affected Services and Features:

Google Calendar

Regions/Zones: Europe

Description:

Some Google Calendar users experienced HTTP 500 errors when using the web interface or when using public APIs (APIv3, CalDAV), which can also affect third party email clients. From preliminary analysis, the root cause of the issue is related to an internal Spanner database resourcing issue.

Customer Impact:

Affected customers would see HTTP 500 and other errors when:

• Using the Google Calendar Interface.
• Using the public APIs (APIv3, CalDAV).
• Using 3rd party email and calendar applications which used the public APIs.