Today’s outage
Today a user added a twitter keyword (iPhone) generating 700,000 pushes within a super short amount of time (a few hours?), which went fine. The bad idea was when I tried to delete those to help him running his device. The load on the server just blew up, and the company hosting us being in Paris, I had to wait they wake up to get a hard reboot.
To prevent this, and as of now :
- Twitter keyword are being looked at, if too popular it will be automatically unactivated for you (we warn you with a push).
- If you receive too many pushes within one hour, you will receive an email and won’t get any anymore for one hour, so you don’t generate 700,000 pushes.
- Push tables are sharded and will soon be deployed in production, to prevent a user slowing others
- Another server will be deployed within a week, and should prevent what happened today. Those server are the fastest our host can get, with at least 16Gigs of memory each.
Sorry about this outage, I know for some of you the application is super important and I’m doing my best for reducing issues.
