Early last Friday morning, one of the Micro.blog photo storage servers failed. I brought it back up shortly after I noticed, but dealing with the problem motivated me to make a change I’ve been thinking about for how photos are stored. Over the weekend I migrated the photo storage to a new system based on Linode’s Object Storage.
(Luckily, years ago when I was designing Micro.blog, I had anticipated I might eventually make a change like this. The old server was running Minio to provide an S3-compatible interface, internal to Micro.blog, so switching to a new system was much easier.)
When a photo is uploaded to Micro.blog, it has always been kept in at least 2 places: on a primary photo storage server, and on a copy of your published blog, along with HTML and other files. Both of these locations were backed up daily.
Over the summer there was a similar error to what happened last week. For some new uploads, only 1 copy of the photo was kept. If Micro.blog needed to rebuild your published blog files, there was a small window during which it could overwrite the latest photos that had been uploaded. Usually I’d be able to fix this from a backup, but I know at least 1 user who was affected by it, and there might have been a few more.
If you ever see broken images on your site, please reach out to help@micro.blog. You can also re-upload photos under Posts → Uploads in Micro.blog on the web and update any posts from the original photos.
Remember that Micro.blog also has built-in features for making extra backups for yourself, beyond the backups we already handle behind the scenes:
- Save new posts to the Internet Archive. There’s a checkbox to enable this under Posts → Design. When you publish a new post, Micro.blog automatically makes it available in the Wayback Machine.
- Export in Blog Archive Format. Micro.blog will make a .bar file that is actually a ZIP file with all your photos, HTML for posts, and a complete JSON Feed.
The new photo storage that is live now should be even more robust. Photos are a very important part of Micro.blog. I’ll keep improving this and adding storage redundancy wherever I can.
