Neurostars Outage Aug 22


#1

Hi everyone,

We had an outage in Neurostars on Aug 21/22 for about 10 hours. I am travelling abroad and didn’t get the notifications right away.

It looks like a system update ran into a snag because Github ( which hosts some of the dependencies ) had some connectivity issues. The upgrade abort caused the app container’s file system to become corrupted, which made rolling back slightly more complicated.

I went ahead and rebuilt the application with the newer version and restored the previous database backup. I also switched out the container’s file system from devicemapper to overlayfs ( which has been recommended by the Discourse developers because of corruption issues with devicemapper ).

Apologize about the outage. Please let me know if there are any issues.

best, Chris.


#2

#3

I’m afraid some things are still not working reliably. For example loading of images:


#4

Yikes, looks like some of the file uploads had permissions problems on the restore. I corrected that and they should be being served now. Is there anything else that’s not working correctly?


#5

Seems to be fixed. Thanks!