Typesense - Connectivity issues in one data center in the Bahrain region – Incident details

Experiencing partial outage

Connectivity issues in one data center in the Bahrain region

Identified
Major outage
Started 10 days ago

Affected

Cluster Regions

Operational from 5:05 AM to 3:34 AM, Partial outage from 3:34 AM to 7:21 PM, Major outage from 7:21 PM to 12:00 AM

Bahrain

Operational from 5:05 AM to 3:34 AM, Partial outage from 3:34 AM to 7:21 PM, Major outage from 7:21 PM to 12:00 AM

Updates
  • Update
    Update

    The situation has not improved in the Middle East regions affected by the recent drone strikes. Our infra provider does not have an ETA on when they will be able to restore these data center facilities, due to the extraordinary nature and fluidity of the situation.

    So we recommend that users with clusters in the UAE and Bahrain regions provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions.

    Since our infra provider's control plane is also down in these regions, it is not possible to access or export the existing data out of these regions. So it is not possible to clone or modify cluster configuration (like RAM, CPU, enabling HA, enabling SDN, etc) in these two regions.

    If you have have a multi-region (Search Delivery Network) cluster with at least one node outside of these two affected regions, please reach out to support at typesense dot org, for assistance in migrating your data out of your SDN cluster.

  • Identified
    Identified

    Our infrastructure provider has completed their initial assessment, and confirmed that one of the three data centers in the Bahrain region have sustained structural damage due to drone strikes in the area.

    They estimate full recovery to take a prolonged amount of time given the extraordinary nature of the situation.

    HA clusters in this region continue to operate fine with two out of three nodes still functioning. However, given the volatility of the conflicts in the area, we recommend that users with clusters in this region plan to provision new clusters in a region outside the Middle East, reindex their data from their primary DB and update their applications to use the new regions.

  • Investigating
    Investigating

    We are observing upstream network connectivity issues in one of the three data centers in the Bahrain region.

    Typesense Cloud clusters with High Availability enabled remain unaffected, as traffic is being routed away from the affected data center.

    Typesense Cloud clusters that do not have High Availability enabled will remain unavailable until network connectivity is restored.