tag:status.maestra.io,2005:/history maestra Status - Incident history 2026-04-08T15:37:00.000+00:00 maestra tag:status.maestra.io,2005:Incident/cmnrocpx300qrnif58pczltok 2026-04-08T15:37:00.000+00:00 2026-04-08T15:37:00.000+00:00 Degraded access to admin panel Type: Incident

Duration: 16 minutes

Affected Components: Admin panel

Apr 8, 15:37:00 GMT+0
Investigating - We are currently investigating elevated error rates affecting approximately 1.5% of traffic to some internal web services. API endpoints are operating normally and are not affected by this incident..

Apr 9, 16:28:00 GMT+0
Resolved - This incident has been resolved. We'll get back with details at a later date..

Apr 17, 13:28:06 GMT+0
Postmortem - **What happened** On April 8, about 1.5% of admin panel traffic started failing. API endpoints were not affected. The issue was fully resolved on April 9. **Why it happened** We recently rolled out a new version of the software that routes user traffic to our services (HAProxy) across our fleet of six load balancers. The rollout had been tested in a test deployment and initially looked healthy in production as well. However, about 7 hours after the upgrade, one of the six load balancers started failing to reach backend services — and that's what caused the errors users experienced. The root cause is a bug in the new HAProxy version that only shows up under a very specific, still-unidentified set of conditions. Our test deployment and the other five production balancers kept working fine, which is why the issue slipped past our pre-rollout checks. **Why it took us a while to notice** When a request failed, our system automatically tried it again, and the retry usually succeeded. So from the outside things looked a bit slow or occasionally flaky rather than clearly broken. That automatic retry behavior is normally a good thing — it hides small, transient glitches from users — but in this case it also hid the growing problem from our monitoring long enough to delay detection. **What we did** \- Rolled back the affected balancer to the previous HAProxy version, which immediately restored normal traffic. \- Paused all further HAProxy upgrades across the fleet until we understand the trigger. **What's next** We're working to reproduce the bug in a controlled test environment so we can pinpoint the trigger, confirm a fix (either a patched version or a config change), and safely resume the rollout..

]]> tag:status.maestra.io,2005:Maintenance/cmmsirjc10695m0kq0psrohsd 2026-03-16T05:00:00.000+00:00 2026-03-16T05:00:00.000+00:00 Database maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Loyalty, Admin panel, Site personalization

Mar 16, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect admin panel, site personalization and loyalty to be unavailable for no more then 35 minutes..

Mar 16, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 16, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Maintenance/cmmsirk0e0617wv24wkpazf0q 2026-03-16T05:00:00.000+00:00 2026-03-16T05:00:00.000+00:00 Database maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Loyalty, Admin panel, Site personalization

Mar 16, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect admin panel, site personalization and loyalty to be unavailable for no more then 35 minutes..

Mar 16, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 16, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Maintenance/cmmiifw8r01b3x6d003rfhtst 2026-03-09T05:00:00.000+00:00 2026-03-09T05:00:00.000+00:00 Database servers maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Loyalty, Admin panel, Site personalization

Mar 9, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect admin panel, site personalization and loyalty to be unavailable for no more then 45 minutes..

Mar 9, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 9, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Maintenance/cmkqsc2st061x8iu91ylxanl7 2026-01-26T06:00:00.000+00:00 2026-01-26T06:00:00.000+00:00 Cluster maintenance Type: Maintenance

Affected Components: Transactional messages, Loyalty, Mass campaigns, Admin panel, Site personalization

Jan 26, 06:00:00 GMT+0
Completed - Maintenance has completed successfully..

Jan 26, 06:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect loyalty, mass campaigns, admin panel, personalization and transaction messages to be unavailable for no more then 20 minutes..

]]> tag:status.maestra.io,2005:Maintenance/cmjch5mya03wx13jw51rzv2hs 2025-12-22T05:00:00.000+00:00 2025-12-22T05:00:00.000+00:00 Network maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Transactional messages, Loyalty, Mass campaigns, Admin panel, Site personalization

Dec 22, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect admin panel, site personalization, loyalty mass campaigns and transactional messages to be unavailable for no more then 10 minutes..

Dec 22, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 22, 06:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Incident/cmja43ook021rafzwk2n9bird 2025-12-17T11:25:00.000+00:00 2025-12-17T11:25:00.000+00:00 Service degradation Type: Incident

Duration: 30 minutes

Affected Components: Loyalty, Transactional messages, Site personalization

Dec 17, 11:25:00 GMT+0
Investigating - We are currently investigating this incident..

Dec 17, 11:55:00 GMT+0
Resolved - **Summary:** On **2025-12-17**, our API Gateway experienced a temporary degradation caused by an unsuccessful update of an ingress component. **Impact:** * Elevated **5xx error rate** on the API Gateway. * **Average 5xx rate over the incident interval:** **\~1.76% of total RPS**. * **Peak 5xx rate:** **\~9–10% of total RPS**. * Partial request failures for API clients during the incident window. **Timeline (UTC):** * **11:25** — 5xx rate begins to increase. * **\~11:35** — Peak degradation observed. * **\~11:55** — Errors drop back to baseline and service stabilizes. **Incident Duration:** From **11:25 to 11:55 (UTC)** — **\~30 minutes**. **Root Cause:** A failed deployment of an ingress component introduced regressions that affected API Gateway request handling, resulting in increased 5xx responses. **Resolution:** We rolled back/reverted the ingress component update and confirmed the API Gateway error rate returned to normal levels..

]]> tag:status.maestra.io,2005:Maintenance/cmheibnxb00i62fcbd85hnhal 2025-11-03T05:00:00.000+00:00 2025-11-03T05:00:00.000+00:00 Network maintenance Type: Maintenance

Duration: 30 minutes

Affected Components: Transactional messages, Loyalty, Mass campaigns, Admin panel, Site personalization

Nov 3, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect admin panel, site personalization, loyalty, mass campaigns and transactional messages to be unavailable for no more then 10 minutes..

Nov 3, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 3, 05:30:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Incident/cmgz955jf02sxvmu4wnsuwduw 2025-10-20T14:50:00.429+00:00 2025-10-20T14:50:00.429+00:00 Transactional communications are sent with delay Type: Incident

Duration: 35 minutes

Affected Components: Transactional messages

Oct 20, 14:50:00 GMT+0
Investigating - We are currently investigating this incident..

Oct 20, 15:24:54 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.maestra.io,2005:Incident/cmgtbornz010gl371d7fg3zc0 2025-10-16T09:19:00.000+00:00 2025-10-16T09:19:00.000+00:00 Delays processing asynchronous operation calls Type: Incident

Duration: 1 hour and 22 minutes

Affected Components: Transactional messages

Oct 16, 09:19:00 GMT+0
Investigating - Some requests to the /v3/operations/async endpoint are experiencing delays. We’re investigating the issue..

Oct 16, 10:41:07 GMT+0
Resolved - The incident has been resolved. All calls received during the outage will be processed once the current queue is cleared. An additional four-minute planned downtime of the async API was required to prevent further issues, from 11:39–11:43 UTC (07:39–07:43 EDT, 04:39–04:43 PDT).

]]> tag:status.maestra.io,2005:Maintenance/cmgh5t7ry008ipv1i2d8h86k2 2025-10-07T22:00:00.000+00:00 2025-10-07T22:00:00.000+00:00 Network equipment upgrade Type: Maintenance

Duration: 5 hours

Affected Components: Admin panel, Site personalization

Oct 7, 22:00:00 GMT+0
Completed - Our hosting provider is upgrading the network equipment, which may cause brief, occasional service interruptions..

]]> tag:status.maestra.io,2005:Incident/cmgcf7rmk0010qgeazox30cdq 2025-10-04T14:39:00.000+00:00 2025-10-04T15:22:00.000+00:00 Platform outage Type: Incident

Duration: 11 minutes

Affected Components: Loyalty, Admin panel, Site personalization, Mass campaigns

Oct 4, 15:22:00 GMT+0
Resolved - This incident has been resolved. We will provide a comprehensive report next week..

Oct 22, 14:03:16 GMT+0
Postmortem - Due to a configuration error, offsite backup uploads fully saturated the network link, causing the gateway to become unavailable. To prevent this from happening again, we introduced rate limiting and moved backup transfers to a private Direct Connect to S3 instead of the production link..

Oct 4, 14:39:00 GMT+0
Identified - The platform is experiencing outages. The admin panel and some functionality are affected. We are actively investigating the issue..

]]> tag:status.maestra.io,2005:Maintenance/cmg4p5j0b012jx2dzo15urka1 2025-09-29T07:00:00.000+00:00 2025-09-29T07:00:00.000+00:00 Network maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Mass campaigns, Site personalization, Loyalty

Sep 29, 07:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. We expect to Site personalization, Loyalty and Mass campaigns be unavailable for no more then 30 minutes..

Sep 29, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 29, 08:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Maintenance/cme6ov3g800b94qsp4d9q09hl 2025-08-11T06:00:00.000+00:00 2025-08-11T06:00:00.000+00:00 Server maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Transactional messages, Mass campaigns, Loyalty

Aug 11, 06:00:00 GMT+0
Identified - We will be rebooting the production server on 11 of August at 7:00am UTC to apply updates and ensure stability. Expected downtime: \~45 minutes. Thank you for your understanding..

Aug 11, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 11, 07:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.maestra.io,2005:Incident/cme004t08002lpaio1xqz4vs3 2025-07-26T13:47:00.000+00:00 2025-07-26T13:47:00.000+00:00 Delays in scenarios execution Type: Incident

Duration: 19 minutes

Affected Components: Transactional messages

Jul 26, 13:47:00 GMT+0
Investigating - We are currently investigating this incident..

Jul 26, 14:06:00 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.maestra.io,2005:Incident/cme00nkzr0041ls3uyo7t5lyy 2025-06-26T15:24:00.000+00:00 2025-06-26T15:24:00.000+00:00 Partial outage of the platform Type: Incident

Duration: 1 hour and 42 minutes

Affected Components: Loyalty, Transactional messages, Mass campaigns, Admin panel, Site personalization

Jun 26, 15:24:00 GMT+0
Investigating - We are currently investigating this incident..

Jun 26, 17:06:00 GMT+0
Resolved - This incident has been resolved..

]]>