What happened to Neurostars?
Recently, we had some technical difficulties with Neurostars that resulted in a loss of data.
How did this happen?
A recent hardware issue with our hosting provider caused a temporary outage with the application. In the process of bringing the application back up, a full stop and restart of the server was required. This unknowing resulted in a loss of the ephemeral storage that was being used to back the application server.
Why was this not being backed up?
The server was being backed-up via our standard procedure, but a relatively small change in the configuration moved the application to a volume that was excluded from the backup retention plan.
So Neurostars was running on an ephemeral storage volume that was not being backed-up?
Yes, unfortunately, that was the case.
Can’t we recover the data with some forensic data recovery tools?
We are investigating this with the hopes that it is possible. The problem is that this configuration changed happened some months ago, which means the source drive was constantly being written to after the data was moved, which makes it difficult to recover the data. However, we are exploring options and are hopeful we can get some of the data back from the old site.
How do we know this will not happen again?
First, we are making sure that the backup is working on several levels. We have a data backup of all the underlying server data, and not using any ephemeral storage. We will also be working to publish backups of our user generated content online, making it available for anyone to download and reuse.
We are also ensuring that multiple people are aware of how the application is configured and setup.
Who can I contact with thoughts, suggestions, concerns, complaints, etc?
Please leave your thoughts here or email me or contact me ( Chris Fitzpatrick ) directly. I welcome all feedback.