Wrestling With Our Photo Server


This week we had some issues with our photo server. On Wednesday black thumbnails appeared for some photos and today we had to shut down the upload function for a short period. In this article we will try to explain what happened.

Server vs. Us

So what was the deal? Wednesday morning we had problems reaching our server (which is remotely hosted by Amazon) that resizes, uploads and distributes our photos – there was no way connecting to it. So we set up a new server to bring everything back to normal and turned the other server off. There were some changes in the configuration which resulted in a small amount of photos having now a wrong (internal) filename (which makes it hard for us to find these photos or even results in showing a wrong photo). This issue is fixed and the upload is working again.

What are we doing next? We will download all original photos uploaded during that time to our hard drive, calculate the correct (internal) filename and rename them. This will take a while but on Monday it should be finished (so if you still see some black thumbnail this is the reason). If we find any problematic photos we will contact you and let you know about it. In the best case everything is just fine, in the worst case we will let you know which photo was affected and ask you to upload it again.

We are really sorry for the trouble and are focused on resolving it. Thanks for your understanding! If you experience any problems with the upload please let us know in the comments.

Update: we are in the middle of testing the photos – the first ones are ready and it seems some of them are broken. We will keep you updated on the process and as promised contact everyone who was affected. Thanks again!

Another update: we’ve found all photos. Everyone whose photos were affected got a message with the filenames. We’ve deleted those photos – sorry, we had to do this, there is no way to repair these – and gave everyone 10 piggies for the inconvenience. Thanks for your understanding!

Yet another update: it takes a bit of time to invalidate our cache, but we are on it! Errr, sorry again for the delay.

Hopefully final update: all messages are out, please reupload those photos again. And if your are one of the users whose avatar photos seems a bit wrong: please reupload this one again, but resize it a bit (can be 1px smaller) so the checksum is different. Then everything should be fine again! Thanks to you!

written by recurving on 2010-12-03 #news #relaunch #server #problems


  1. squamy
    squamy ·

    Oh no! Servers always like to cause trouble for the weekend. Hope it hasn't been too stressful!

  2. illy
    illy ·

    Ah, this might also explain why there were some fisheye-photos in my uploaded pictures, though I don't even have a fisheye camera :)

  3. gnarlyleech
    gnarlyleech ·

    i like the photo illustration. lol

  4. beanieboi
    beanieboi ·

    @squamy you are damn right, always before weekend! :(

  5. recurving
    recurving ·

    @illy: errr, exactly that… :) sorry again!

  6. eatcpcks
    eatcpcks ·

    This is the Murphy's law:D

  7. fivedayforecast
    fivedayforecast ·

    Tis fine

  8. clickiemcpete
    clickiemcpete ·

    Thanks for all the hard work, I'm sure it is a nightmare keeping up with all the stuff you guys having going on.

  9. idigtulsa
    idigtulsa ·

    Let's flip this around and look at it like a good thing.....If non of us were taking any pictures, then the servers would be lonely, board and everything would still be fine in the land of The Lomo Internets. Thank for your hard work behind the scenes so we can keep at it!!

  10. discodrew
    discodrew ·

    Appreciate the heads up!

  11. dogma
    dogma ·

    Let's hope everythings gone be fine!

  12. brittany
    brittany ·

    @idigtulsa ahah i totally agree with you

  13. rodrigoalmeida
    rodrigoalmeida ·

    Hi there! Thanks for the information and for trying to solve the problem. I'm still having problems with this album :www.lomography.tw/homes/rodrigoalmeida/albums/1657232-super…, which I uploaded last Thursday. Most photos have black rectangles at their bottoms and I can't use any of them on lomowalls. Would you check it for me please? Thanks a lot!

  14. recurving
    recurving ·

    @all: thanks for your understanding – we really appreciate that!
    @rodrigoalmeida for sure – we're gonna take a look tomorrow when we are all back in office!
    @clickiemcpete: oh, far from being a nightmare, it's a lot of fun! just sometimes – well – challenging… :) but we are very happy our servers don't get bored and have lots to do…

  15. snug
    snug ·

    Wow... that sounds very complicated and quite a long job for you... certainly makes us appreciate the great service you do offer us.. your still much better than flickr!

  16. rodrigoalmeida
    rodrigoalmeida ·

    @recurving: thanks!

  17. recurving
    recurving ·

    @all: I've updated the article – we are still checking all photos (around 2,000 are done already), some have issues we are afraid… but gonna compile a full list and get in touch with everyone!

  18. wkdroots
    wkdroots ·

    Almost all of my pictures are affected. I have the same problems as rodrigoalmeida =( I didn't notice this posting so I wrote your Facebook page asking for help and I got a very fast response from Christopher Zinck who took a look at my page, emailed me a link to this article and answered all my questions. You guys rock!!

  19. recurving
    recurving ·

    Hey guys – we've found the photos and will send out a message to all of you – thanks again for your understanding!

  20. recurving
    recurving ·

    @wkdroots: Sorry to hear that – you (and all others) will get a notification with all affected filenames and albums! And high-five to @dont_think (Christopher) for being so fast… :)

  21. niko_fuzzy
    niko_fuzzy ·

    how to know which picture got deleted?

  22. recurving
    recurving ·

    If you didn't get a message all your photos seem to be fine. If there are affected photos and you didn't get a notice please post the urls to these here…

  23. bass_clarinet_2000
    bass_clarinet_2000 ·

    I'm uploading mine now. I was surprised to see how many of mine were effected but that is ok. I'm fine with it:D Also, the piggies make me a happy girl.

  24. calvin_hardy
    calvin_hardy ·

    Oh no :L

  25. spoeker
    spoeker ·

    hey can you have a look at these 2 albums of mine:

    I got a message first saying that Im affected and that you will contact me again with more details. But so far I didnt get a second message but the photos in these 2 albums seem to be affected - black rectangles at their bottoms and I can't use any of them on lomowalls.
    and these two are showing up as different photos which arent mine:

    but please don't delete them, some of them are submitted to contests where I already reached the limit ;)
    maybe I can reupload the photos and delete the others as soon as the competitions are over? Other ideas? thanks for your help

  26. rdp_ribeirense
    rdp_ribeirense ·

    thankyou :)

  27. wkdroots
    wkdroots ·

    @recurving: My pictures havent been deleted and i didnt get a message. I did get the 10 piggies though. Here are the links to the 2 albums that are affected (my ony 2 albums)

    www.lomography.tw/homes/wkdroots/albums/1649870-cross-proce… (first 8 photos)

    www.lomography.tw/homes/wkdroots/albums/1649339-black-and-w… (first 6 photos)

    Also, i just sent you a private message. Thank you for the help!

  28. rodrigoalmeida
    rodrigoalmeida ·

    Dear @recurving,
    I got the piggies, but didn't receive any message. I still have the same problems with this album: www.lomography.tw/homes/rodrigoalmeida/albums/1657232-super…
    Could you take a look at it please? If I have to upload it again, it's ok. I just want to be sure if I really need to do it. Thanks a lot for your hard work.

  29. tallgrrlrocks
    tallgrrlrocks ·

    Thanks for tirelessly giving the effort to find solutions! Great job.

  30. jabuka
    jabuka ·

    When can I reupload all my delated photos? I reuploaded them but most of them were shown in black in the lab as before. Maybe they uploadeded properly, but I'm really not sure, so I delated them again.

  31. recurving
    recurving ·

    @jabuka: the photos should be fine, but the ones shown are still the old (cached) ones… we are not invalidating our caches so they get refilled.

  32. spoeker
    spoeker ·

    ehm.......now my two albums I posted you here in the comments are all white....?
    Can someone please help me? Should I delete them now and reupload? I didnt get a message with the filenames so far..and some of these photos were competition-submissions, eg. for the "word of the month december" where I can only submit one photo..and the wrangler competitions, Rip Curl and some more I can't remeber....

  33. recurving
    recurving ·

    hey @spoeker: we are on it (your photos, the filenames and the submissions)!

  34. wkdroots
    wkdroots ·

    deleted and reuploaded three times now. Now im having the same problem as @spoeker. All the new photos are white. Im not to sure what to do anymore. I never recieved a message and deleting and reuploading clearly isnt working. When you get a sec can you check it out?


  35. spoeker
    spoeker ·

    @recurving - thank you! looking forward to your message :)

  36. recurving
    recurving ·

    hey @spoeker and @wkdroots: you should have gotten the message now and the photos can be reuploaded!

  37. maria_vlachou
    maria_vlachou ·

    I reuploaded my latest album 3 days ago and it's still all thumbs! Am I being impatient? Thanks for trying...

  38. spoeker
    spoeker ·

    @recurving thanks! I'm not at home at the moment so I can't reupload and resubmit for the wrangler competitions, I hope I made it in time but it says only 1 day left :(

  39. brommi
    brommi ·


More Interesting Articles