Sending schedules and exporting as PDF/PNG delayed
Incident Report for Cluvio
Postmortem

What happened: Sending of dashboard schedule and sql alert emails was delayed for about 2 hours.

Root cause: An attacker created a trial account and by abusing our API triggered spam/phishing email sending. Our ops team was alerted about the abuse and blocked the account, and due to rate-limiting, the number of requests processed was limited, but there was still about 2 hour delay for the email sending due to the job processing queue being affected.

What we did to avoid these types of issues in the future:

To prevent this type of abuse, we tightened some aspects of the trial account usage (which otherwise has full functionality without any restrictions) regarding email sending for alerts, schedules and user invitations.

As a result, abuse attempts like the one causing this issue will not be possible.

Posted Jul 12, 2021 - 12:26 CEST

Resolved
The email sending and dashboard exporting are working properly again. The slowdown was caused by clogging of the dashboard schedule job queue from an attempted abuse for distribution of spam. We have disabled the spamming account and will deploy better protection against this type of abuse later today.
Posted Jul 05, 2021 - 10:43 CEST
Investigating
We are investigating an issue with dashboard schedule emails being delayed and exporting PDF/PNG not completing
Posted Jul 05, 2021 - 10:18 CEST
This incident affected: EU (Germany, AWS eu-central-1) (Cluvio Web Application (EU), Emails for Sql Alerts and Dashboard Schedules (EU)).