As of late at around 10: 45pm CET, after a pair of glasses of purple wine, we deleted the production database by likelihood 😨.
Fortunately our database is a managed database from DigitalOcean, which manner that DigitalOcean automatically enact backups once a day. After 5 minutes of hand-wringing and dismay, we went into maintenance mode and had been ready to restore a backup. At around 11: 15pm CET, 30 minutes after the catastrophe, we went back online, however 7 hours of scoreboard data change into gone forever 😵.
To be precise, any scoreboards created or ratings added between 15: 47 CET and 23: 21 CET were lost. We’re extraordinarily sorry about this.
What took web site?
It’s tempting in payment the catastrophe on the couple of glasses of purple wine. On the opposite hand, the feature that wiped the database change into written whilst sober. It’s a feature that deletes the local database and creates the entire required tables from scratch. This evening, whilst doing some slack evening coding, the feature connected to the production database and wiped it. Why? Here’s one thing we’re quiet looking out for to establish out.
Here is the code that triggered the catastrophe:
def database_model_create(): """Handiest works on localhost to forestall catastrophe""" database = config.DevelopmentConfig.DB_DATABASE person = config.DevelopmentConfig.DB_USERNAME password = config.DevelopmentConfig.DB_PASSWORD port = config.DevelopmentConfig.DB_PORT local_db = PostgresqlDatabase(database=database, person=person, password=password, host='localhost', port=port) local_db.drop_tables([Game, Player, Round, Score, Order]) local_db.create_tables([Game, Player, Round, Score, Order]) print('Initialized the local database.')
host is hardcoded to
localhost. This suggests it would perchance perhaps quiet never join to any machine rather than the developer machine. We’re too tired to establish it out exact now. The gremlins won this time.
What enjoy we realized? Why won’t this happen all but again?
We’ve realized that having a feature that deletes your database is just too terrible to enjoy mendacity around. The venture is, you might perchance well also never if truth be told test the safety mechanisms neatly, on fable of testing it would perchance perhaps suggest pointing a gun on the production database.
In fact, we can never be 100% definite that one thing admire this won’t happen all but again. Computer systems are exact too advanced and there are days when the complexity gremlins secure. On the opposite hand, we can establish out what went gross and be certain that that that particular error doesn’t happen all but again.
All but again, we’re very sorry. Correct night.