This week we had some issues with our photo server. On Wednesday black thumbnails appeared for some photos and today we had to shut down the upload function for a short period. In this article we will try to explain what happened.
So what was the deal? Wednesday morning we had problems reaching our server (which is remotely hosted by Amazon) that resizes, uploads and distributes our photos – there was no way connecting to it. So we set up a new server to bring everything back to normal and turned the other server off. There were some changes in the configuration which resulted in a small amount of photos having now a wrong (internal) filename (which makes it hard for us to find these photos or even results in showing a wrong photo). This issue is fixed and the upload is working again.
What are we doing next? We will download all original photos uploaded during that time to our hard drive, calculate the correct (internal) filename and rename them. This will take a while but on Monday it should be finished (so if you still see some black thumbnail this is the reason). If we find any problematic photos we will contact you and let you know about it. In the best case everything is just fine, in the worst case we will let you know which photo was affected and ask you to upload it again.
We are really sorry for the trouble and are focused on resolving it. Thanks for your understanding! If you experience any problems with the upload please let us know in the comments.
Update: we are in the middle of testing the photos – the first ones are ready and it seems some of them are broken. We will keep you updated on the process and as promised contact everyone who was affected. Thanks again!
Another update: we’ve found all photos. Everyone whose photos were affected got a message with the filenames. We’ve deleted those photos – sorry, we had to do this, there is no way to repair these – and gave everyone 10 piggies for the inconvenience. Thanks for your understanding!
Yet another update: it takes a bit of time to invalidate our cache, but we are on it! Errr, sorry again for the delay.
Hopefully final update: all messages are out, please reupload those photos again. And if your are one of the users whose avatar photos seems a bit wrong: please reupload this one again, but resize it a bit (can be 1px smaller) so the checksum is different. Then everything should be fine again! Thanks to you!