Have an account? Login | New to Lomography? Register | Lab | Current Site:

Wrestling With Our Photo Server

This week we had some issues with our photo server. On Wednesday black thumbnails appeared for some photos and today we had to shut down the upload function for a short period. In this article we will try to explain what happened.

Server vs. Us

So what was the deal? Wednesday morning we had problems reaching our server (which is remotely hosted by Amazon) that resizes, uploads and distributes our photos – there was no way connecting to it. So we set up a new server to bring everything back to normal and turned the other server off. There were some changes in the configuration which resulted in a small amount of photos having now a wrong (internal) filename (which makes it hard for us to find these photos or even results in showing a wrong photo). This issue is fixed and the upload is working again.

What are we doing next? We will download all original photos uploaded during that time to our hard drive, calculate the correct (internal) filename and rename them. This will take a while but on Monday it should be finished (so if you still see some black thumbnail this is the reason). If we find any problematic photos we will contact you and let you know about it. In the best case everything is just fine, in the worst case we will let you know which photo was affected and ask you to upload it again.

We are really sorry for the trouble and are focused on resolving it. Thanks for your understanding! If you experience any problems with the upload please let us know in the comments.

Update: we are in the middle of testing the photos – the first ones are ready and it seems some of them are broken. We will keep you updated on the process and as promised contact everyone who was affected. Thanks again!

Another update: we’ve found all photos. Everyone whose photos were affected got a message with the filenames. We’ve deleted those photos – sorry, we had to do this, there is no way to repair these – and gave everyone 10 piggies for the inconvenience. Thanks for your understanding!

Yet another update: it takes a bit of time to invalidate our cache, but we are on it! Errr, sorry again for the delay.

Hopefully final update: all messages are out, please reupload those photos again. And if your are one of the users whose avatar photos seems a bit wrong: please reupload this one again, but resize it a bit (can be 1px smaller) so the checksum is different. Then everything should be fine again! Thanks to you!

written by recurving

39 comments

  1. squamy

    squamy

    Oh no! Servers always like to cause trouble for the weekend. Hope it hasn't been too stressful!

    over 3 years ago · report as spam
  2. illy

    illy

    Ah, this might also explain why there were some fisheye-photos in my uploaded pictures, though I don't even have a fisheye camera :)

    over 3 years ago · report as spam
  3. gnarlyleech

    gnarlyleech

    i like the photo illustration. lol

    over 3 years ago · report as spam
  4. beanieboi

    beanieboi

    @squamy you are damn right, always before weekend! :(

    over 3 years ago · report as spam
  5. recurving

    recurving

    @illy: errr, exactly that… :) sorry again!

    over 3 years ago · report as spam
  6. eatcpcks

    eatcpcks

    This is the Murphy's law:D

    over 3 years ago · report as spam
  7. fivedayforecast

    fivedayforecast

    Tis fine

    over 3 years ago · report as spam
  8. clickiemcpete

    clickiemcpete

    Thanks for all the hard work, I'm sure it is a nightmare keeping up with all the stuff you guys having going on.

    over 3 years ago · report as spam
  9. idigtulsa

    idigtulsa

    Let's flip this around and look at it like a good thing.....If non of us were taking any pictures, then the servers would be lonely, board and everything would still be fine in the land of The Lomo Internets. Thank for your hard work behind the scenes so we can keep at it!!

    over 3 years ago · report as spam
  10. discodrew

    discodrew

    Appreciate the heads up!

    over 3 years ago · report as spam
  11. dogma

    dogma

    Let's hope everythings gone be fine!

    over 3 years ago · report as spam
  12. brittany

    brittany

    @idigtulsa ahah i totally agree with you

    over 3 years ago · report as spam
  13. rodrigoalmeida

    rodrigoalmeida

    Hi there! Thanks for the information and for trying to solve the problem. I'm still having problems with this album :http://www.lomography.com/homes/rodrigoalmeida/albums/1657232-supersampler-1st-try, which I uploaded last Thursday. Most photos have black rectangles at their bottoms and I can't use any of them on lomowalls. Would you check it for me please? Thanks a lot!

    over 3 years ago · report as spam
  14. recurving

    recurving

    @all: thanks for your understanding – we really appreciate that!
    @rodrigoalmeida for sure – we're gonna take a look tomorrow when we are all back in office!
    @clickiemcpete: oh, far from being a nightmare, it's a lot of fun! just sometimes – well – challenging… :) but we are very happy our servers don't get bored and have lots to do…

    over 3 years ago · report as spam
  15. snug

    snug

    Wow... that sounds very complicated and quite a long job for you... certainly makes us appreciate the great service you do offer us.. your still much better than flickr!

    over 3 years ago · report as spam
  16. rodrigoalmeida

    rodrigoalmeida

    @recurving: thanks!

    over 3 years ago · report as spam
  17. recurving

    recurving

    @all: I've updated the article – we are still checking all photos (around 2,000 are done already), some have issues we are afraid… but gonna compile a full list and get in touch with everyone!

    over 3 years ago · report as spam
  18. wkdroots

    wkdroots

    Almost all of my pictures are affected. I have the same problems as rodrigoalmeida =( I didn't notice this posting so I wrote your Facebook page asking for help and I got a very fast response from Christopher Zinck who took a look at my page, emailed me a link to this article and answered all my questions. You guys rock!!

    over 3 years ago · report as spam
  19. recurving

    recurving

    Hey guys – we've found the photos and will send out a message to all of you – thanks again for your understanding!

    over 3 years ago · report as spam
  20. recurving

    recurving

    @wkdroots: Sorry to hear that – you (and all others) will get a notification with all affected filenames and albums! And high-five to @dont_think (Christopher) for being so fast… :)

    over 3 years ago · report as spam
  21. niko_fuzzy

    niko_fuzzy

    how to know which picture got deleted?

    over 3 years ago · report as spam
  22. recurving

    recurving

    If you didn't get a message all your photos seem to be fine. If there are affected photos and you didn't get a notice please post the urls to these here…

    over 3 years ago · report as spam
  23. bass_clarinet_2000

    bass_clarinet_2000

    I'm uploading mine now. I was surprised to see how many of mine were effected but that is ok. I'm fine with it:D Also, the piggies make me a happy girl.

    over 3 years ago · report as spam
  24. calvin_hardy

    calvin_hardy

    Oh no :L

    over 3 years ago · report as spam
  25. spoeker

    spoeker

    hey can you have a look at these 2 albums of mine:
    http://www.lomograph(…)h-03-plus04
    http://www.lomograph(…)c-a-plus-28

    I got a message first saying that Im affected and that you will contact me again with more details. But so far I didnt get a second message but the photos in these 2 albums seem to be affected - black rectangles at their bottoms and I can't use any of them on lomowalls.
    and these two are showing up as different photos which arent mine:
    http://www.lomograph(…)os/12218903
    http://www.lomograph(…)os/12218742

    but please don't delete them, some of them are submitted to contests where I already reached the limit ;)
    maybe I can reupload the photos and delete the others as soon as the competitions are over? Other ideas? thanks for your help

    over 3 years ago · report as spam
  26. rdp_ribeirense

    rdp_ribeirense

    thankyou :)

    over 3 years ago · report as spam
  27. wkdroots

    wkdroots

    @recurving: My pictures havent been deleted and i didnt get a message. I did get the 10 piggies though. Here are the links to the 2 albums that are affected (my ony 2 albums)

    http://www.lomograph(…)oss-process (first 8 photos)

    http://www.lomograph(…)k-and-white (first 6 photos)

    Also, i just sent you a private message. Thank you for the help!

    over 3 years ago · report as spam
  28. rodrigoalmeida

    rodrigoalmeida

    Dear @recurving,
    I got the piggies, but didn't receive any message. I still have the same problems with this album: http://www.lomograph(…)ler-1st-try
    Could you take a look at it please? If I have to upload it again, it's ok. I just want to be sure if I really need to do it. Thanks a lot for your hard work.
    Cheers,
    Rodrigo

    over 3 years ago · report as spam
  29. tallgrrlrocks

    tallgrrlrocks

    Thanks for tirelessly giving the effort to find solutions! Great job.

    over 3 years ago · report as spam
  30. jabuka

    jabuka

    When can I reupload all my delated photos? I reuploaded them but most of them were shown in black in the lab as before. Maybe they uploadeded properly, but I'm really not sure, so I delated them again.

    over 3 years ago · report as spam
  31. recurving

    recurving

    @jabuka: the photos should be fine, but the ones shown are still the old (cached) ones… we are not invalidating our caches so they get refilled.

    over 3 years ago · report as spam
  32. spoeker

    spoeker

    ehm.......now my two albums I posted you here in the comments are all white....?
    http://www.lomograph(…)c-a-plus-28
    http://www.lomograph(…)h-03-plus04
    Can someone please help me? Should I delete them now and reupload? I didnt get a message with the filenames so far..and some of these photos were competition-submissions, eg. for the "word of the month december" where I can only submit one photo..and the wrangler competitions, Rip Curl and some more I can't remeber....

    over 3 years ago · report as spam
  33. recurving

    recurving

    hey @spoeker: we are on it (your photos, the filenames and the submissions)!

    over 3 years ago · report as spam
  34. wkdroots

    wkdroots

    deleted and reuploaded three times now. Now im having the same problem as @spoeker. All the new photos are white. Im not to sure what to do anymore. I never recieved a message and deleting and reuploading clearly isnt working. When you get a sec can you check it out?

    http://www.lomograph(…)oss-process
    http://www.lomograph(…)k-and-white

    over 3 years ago · report as spam
  35. spoeker

    spoeker

    @recurving - thank you! looking forward to your message :)

    over 3 years ago · report as spam
  36. recurving

    recurving

    hey @spoeker and @wkdroots: you should have gotten the message now and the photos can be reuploaded!

    over 3 years ago · report as spam
  37. maria_vlachou

    maria_vlachou

    I reuploaded my latest album 3 days ago and it's still all thumbs! Am I being impatient? Thanks for trying...

    over 3 years ago · report as spam
  38. spoeker

    spoeker

    @recurving thanks! I'm not at home at the moment so I can't reupload and resubmit for the wrangler competitions, I hope I made it in time but it says only 1 day left :(

    over 3 years ago · report as spam
  39. brommi

    brommi

    :-S

    about 3 years ago · report as spam

Read this article in another language

This is the original article written in: English. It is also available in: Spanish.