Wrestling With Our Photo Server

39

This week we had some issues with our photo server. On Wednesday black thumbnails appeared for some photos and today we had to shut down the upload function for a short period. In this article we will try to explain what happened.

Server vs. Us

So what was the deal? Wednesday morning we had problems reaching our server (which is remotely hosted by Amazon) that resizes, uploads and distributes our photos – there was no way connecting to it. So we set up a new server to bring everything back to normal and turned the other server off. There were some changes in the configuration which resulted in a small amount of photos having now a wrong (internal) filename (which makes it hard for us to find these photos or even results in showing a wrong photo). This issue is fixed and the upload is working again.

What are we doing next? We will download all original photos uploaded during that time to our hard drive, calculate the correct (internal) filename and rename them. This will take a while but on Monday it should be finished (so if you still see some black thumbnail this is the reason). If we find any problematic photos we will contact you and let you know about it. In the best case everything is just fine, in the worst case we will let you know which photo was affected and ask you to upload it again.

We are really sorry for the trouble and are focused on resolving it. Thanks for your understanding! If you experience any problems with the upload please let us know in the comments.

Update: we are in the middle of testing the photos – the first ones are ready and it seems some of them are broken. We will keep you updated on the process and as promised contact everyone who was affected. Thanks again!

Another update: we’ve found all photos. Everyone whose photos were affected got a message with the filenames. We’ve deleted those photos – sorry, we had to do this, there is no way to repair these – and gave everyone 10 piggies for the inconvenience. Thanks for your understanding!

Yet another update: it takes a bit of time to invalidate our cache, but we are on it! Errr, sorry again for the delay.

Hopefully final update: all messages are out, please reupload those photos again. And if your are one of the users whose avatar photos seems a bit wrong: please reupload this one again, but resize it a bit (can be 1px smaller) so the checksum is different. Then everything should be fine again! Thanks to you!

written by recurving on 2010-12-03 in #news #problems #relaunch #server

39 Comments

  1. squamy
    squamy ·

    Oh no! Servers always like to cause trouble for the weekend. Hope it hasn't been too stressful!

  2. illy
    illy ·

    Ah, this might also explain why there were some fisheye-photos in my uploaded pictures, though I don't even have a fisheye camera :)

  3. gnarlyleech
    gnarlyleech ·

    i like the photo illustration. lol

  4. beanieboi
    beanieboi ·

    @squamy you are damn right, always before weekend! :(

  5. recurving
    recurving ·

    @illy: errr, exactly that… :) sorry again!

  6. eatcpcks
    eatcpcks ·

    This is the Murphy's law:D

  7. fivedayforecast
    fivedayforecast ·

    Tis fine

  8. clickiemcpete
    clickiemcpete ·

    Thanks for all the hard work, I'm sure it is a nightmare keeping up with all the stuff you guys having going on.

  9. idigtulsa
    idigtulsa ·

    Let's flip this around and look at it like a good thing.....If non of us were taking any pictures, then the servers would be lonely, board and everything would still be fine in the land of The Lomo Internets. Thank for your hard work behind the scenes so we can keep at it!!

  10. discodrew
    discodrew ·

    Appreciate the heads up!

  11. dogma
    dogma ·

    Let's hope everythings gone be fine!

  12. brittany
    brittany ·

    @idigtulsa ahah i totally agree with you

  13. rodrigoalmeida
    rodrigoalmeida ·

    Hi there! Thanks for the information and for trying to solve the problem. I'm still having problems with this album :www.lomography.com/homes/rodrigoalmeida/albums/1657232-supe…, which I uploaded last Thursday. Most photos have black rectangles at their bottoms and I can't use any of them on lomowalls. Would you check it for me please? Thanks a lot!

  14. recurving
    recurving ·

    @all: thanks for your understanding – we really appreciate that!
    @rodrigoalmeida for sure – we're gonna take a look tomorrow when we are all back in office!
    @clickiemcpete: oh, far from being a nightmare, it's a lot of fun! just sometimes – well – challenging… :) but we are very happy our servers don't get bored and have lots to do…

  15. snug
    snug ·

    Wow... that sounds very complicated and quite a long job for you... certainly makes us appreciate the great service you do offer us.. your still much better than flickr!

  16. rodrigoalmeida
    rodrigoalmeida ·

    @recurving: thanks!

  17. recurving
    recurving ·

    @all: I've updated the article – we are still checking all photos (around 2,000 are done already), some have issues we are afraid… but gonna compile a full list and get in touch with everyone!

  18. wkdroots
    wkdroots ·

    Almost all of my pictures are affected. I have the same problems as rodrigoalmeida =( I didn't notice this posting so I wrote your Facebook page asking for help and I got a very fast response from Christopher Zinck who took a look at my page, emailed me a link to this article and answered all my questions. You guys rock!!

  19. recurving
    recurving ·

    Hey guys – we've found the photos and will send out a message to all of you – thanks again for your understanding!

  20. recurving
    recurving ·

    @wkdroots: Sorry to hear that – you (and all others) will get a notification with all affected filenames and albums! And high-five to @dont_think (Christopher) for being so fast… :)

  21. niko_fuzzy
    niko_fuzzy ·

    how to know which picture got deleted?

  22. recurving
    recurving ·

    If you didn't get a message all your photos seem to be fine. If there are affected photos and you didn't get a notice please post the urls to these here…

  23. bass_clarinet_2000
    bass_clarinet_2000 ·

    I'm uploading mine now. I was surprised to see how many of mine were effected but that is ok. I'm fine with it:D Also, the piggies make me a happy girl.

  24. calvin_hardy
    calvin_hardy ·

    Oh no :L

  25. spoeker
    spoeker ·

    hey can you have a look at these 2 albums of mine:
    www.lomography.com/homes/spoeker/albums/1657161-colorsplash…
    www.lomography.com/homes/spoeker/albums/1657173-lc-a-plus-28

    I got a message first saying that Im affected and that you will contact me again with more details. But so far I didnt get a second message but the photos in these 2 albums seem to be affected - black rectangles at their bottoms and I can't use any of them on lomowalls.
    and these two are showing up as different photos which arent mine:
    www.lomography.com/homes/spoeker/albums/1657173-lc-a-plus-2…
    www.lomography.com/homes/spoeker/albums/1657161-colorsplash…

    but please don't delete them, some of them are submitted to contests where I already reached the limit ;)
    maybe I can reupload the photos and delete the others as soon as the competitions are over? Other ideas? thanks for your help

  26. rdp_ribeirense
    rdp_ribeirense ·

    thankyou :)

  27. wkdroots
    wkdroots ·

    @recurving: My pictures havent been deleted and i didnt get a message. I did get the 10 piggies though. Here are the links to the 2 albums that are affected (my ony 2 albums)

    www.lomography.com/homes/wkdroots/albums/1649870-cross-proc… (first 8 photos)

    www.lomography.com/homes/wkdroots/albums/1649339-black-and-… (first 6 photos)

    Also, i just sent you a private message. Thank you for the help!

  28. rodrigoalmeida
    rodrigoalmeida ·

    Dear @recurving,
    I got the piggies, but didn't receive any message. I still have the same problems with this album: www.lomography.com/homes/rodrigoalmeida/albums/1657232-supe…
    Could you take a look at it please? If I have to upload it again, it's ok. I just want to be sure if I really need to do it. Thanks a lot for your hard work.
    Cheers,
    Rodrigo

  29. tallgrrlrocks
    tallgrrlrocks ·

    Thanks for tirelessly giving the effort to find solutions! Great job.

  30. jabuka
    jabuka ·

    When can I reupload all my delated photos? I reuploaded them but most of them were shown in black in the lab as before. Maybe they uploadeded properly, but I'm really not sure, so I delated them again.

  31. recurving
    recurving ·

    @jabuka: the photos should be fine, but the ones shown are still the old (cached) ones… we are not invalidating our caches so they get refilled.

  32. spoeker
    spoeker ·

    ehm.......now my two albums I posted you here in the comments are all white....?
    www.lomography.com/homes/spoeker/albums/1657173-lc-a-plus-28
    www.lomography.com/homes/spoeker/albums/1657161-colorsplash…
    Can someone please help me? Should I delete them now and reupload? I didnt get a message with the filenames so far..and some of these photos were competition-submissions, eg. for the "word of the month december" where I can only submit one photo..and the wrangler competitions, Rip Curl and some more I can't remeber....

  33. recurving
    recurving ·

    hey @spoeker: we are on it (your photos, the filenames and the submissions)!

  34. wkdroots
    wkdroots ·

    deleted and reuploaded three times now. Now im having the same problem as @spoeker. All the new photos are white. Im not to sure what to do anymore. I never recieved a message and deleting and reuploading clearly isnt working. When you get a sec can you check it out?

    www.lomography.com/homes/wkdroots/albums/1649870-cross-proc…
    www.lomography.com/homes/wkdroots/albums/1649339-black-and-…

  35. spoeker
    spoeker ·

    @recurving - thank you! looking forward to your message :)

  36. recurving
    recurving ·

    hey @spoeker and @wkdroots: you should have gotten the message now and the photos can be reuploaded!

  37. maria_vlachou
    maria_vlachou ·

    I reuploaded my latest album 3 days ago and it's still all thumbs! Am I being impatient? Thanks for trying...

  38. spoeker
    spoeker ·

    @recurving thanks! I'm not at home at the moment so I can't reupload and resubmit for the wrangler competitions, I hope I made it in time but it says only 1 day left :(

  39. brommi
    brommi ·

    :-S

More Interesting Articles

  • Playing Detective: An Interview with Derek Woods, the Man Behind the Controversial True Detective Photo (NSFW)

    written by Jill Tan Radovan on 2014-05-03 in #people #lifestyle
    Playing Detective: An Interview with Derek Woods, the Man Behind the Controversial True Detective Photo (NSFW)

    Derek Woods is an Los Angeles-based photographer who previously got involved in a controversy surrounding a photo that was used in the opening credits of the HBO TV series "True Detective." Coincidentally, Woods happens to be a member of the Lomo community, and it became vital to interview him regarding the issue. The interview was successful and was published in May last year. His current project, 365 of Lomography, will chronicle his day-to-day exploits with Lomography cameras. To jog your memory, and to re-acquaint you with Woods, we are republishing our interview with the controversial photographer. Please take note that some of the photos are NSFW.

    4
  • The Lomo'Instant Does Halloween!

    written by camilla_illa on 2014-10-30 in #gear #tipster
    The Lomo'Instant Does Halloween!

    Halloween is fast approaching and we are getting ready to celebrate till dawn with our favorite cameras. I had the lucky chance to try the Lomo'Instant in preparation for this scary night!

  • December 14th Advent Offer: Take Advantage of our Festive 3 For 2 Film Deals (Online Code: 3FOR2HOLIDAYFILM)

    written by jacobs on 2014-12-14 in #news
    December 14th Advent Offer: Take Advantage of our Festive 3 For 2 Film Deals  (Online Code: 3FOR2HOLIDAYFILM)

    Seeing that we love to spread the cheer around here, we're giving you another chance to load up on our awesome film with today's Advent deal! Choose a classy black and white film, like our Lady Grey, or get creative and colorful with one of our Redscale films. We're certain that no matter what you choose, you'll have a great time making memories with tons of lovely analogue photos this year!

  • Shop News

    Summer Inspired Camera at 20% off

    Summer Inspired Camera at 20% off

    The brazilian summer inspired camera is now at 20% off! You can now celebrate life in full color and treasure every culture in a snap! This summer is no exception; make sure you’re prepared to capture all the sporty action with the Fisheye No.2 Brazilian Summer Camera!

  • Awesome Albums: Trip to Laspi on Mountain by lomovan

    written by plasticpopsicle on 2014-05-21 in #news
    Awesome Albums: Trip to Laspi on Mountain by lomovan

    Since many fun cycling events have been happening around the world this month, it's about time for us to take a look at some lovely snaps taken by our community's cyclers during their biking trips! Today's awesome album

  • Lomography x National Portrait Gallery Rumble WINNERS!

    written by hannah_brown on 2015-01-06 in #news
    Lomography x National Portrait Gallery Rumble WINNERS!

    We had a staggering 1110 entries to this rumble. We have picked our way through all of them and were very inspired by the quality of all your photos! It was a real challenge narrowing it down to just 3 winners. Find out who won this great competition.

    1
  • Coming Soon, Happening Now: The New Lomography Mystery Product Part 2

    written by jillytanrad on 2014-05-24 in #news
    Coming Soon, Happening Now: The New Lomography Mystery Product Part 2

    It’s time for our second clue! Before you discover what’s coming soon and happening now, allow us to ask you this question: what photos do you carry round with you?

    11
  • Shop News

    Diana Mini and Flash Petite Noire at 25% off

    Diana Mini and Flash Petite Noire at 25% off

    At 25% off you can take dreamy 35mm images with this little black beauty. Beam coloured light into your shots with its accompanying Diana Flash Back accessory and be the analogue king of the night.

  • Tales from the Trip: People from the Lake

    written by plasticpopsicle on 2014-04-22 in #world #locations
    Tales from the Trip: People from the Lake

    As intrepid travelers, we don't only search for an adventure or a change of scenery -- we also seek to capture fascinating moments in photographs and tell the compelling stories behind them. In this wanderlust-driven series for the Locations section, we showcase some of the most fascinating travel photos by our fellow lomographers, along with their tales from the trip.

    2
  • Tales from the Trip: Purple Camel Smiles

    written by plasticpopsicle on 2014-04-09 in #world #locations
    Tales from the Trip: Purple Camel Smiles

    As intrepid travelers, we don't only search for an adventure or a change of scenery -- we also seek to capture fascinating moments in photographs and tell the compelling stories behind them. In this wanderlust-driven series for the Locations section, we showcase some of the most fascinating travel photos by our fellow lomographers, along with their tales from the trip.

    2
  • Tales from the Trip: The Axis

    written by plasticpopsicle on 2014-05-10 in #world #locations
    Tales from the Trip: The Axis

    As intrepid travelers, we don't only search for an adventure or a change of scenery -- we also seek to capture fascinating moments in photographs and tell the compelling stories behind them. In this wanderlust-driven series for the Locations section, we showcase some of the most fascinating travel photos by our fellow lomographers, along with their tales from the trip.

    1
  • Shop News

    LC-A Instant Back+ at Lesser Price!

    LC-A Instant Back+ at Lesser Price!

    Grab this lovely accessory at discounted price! LC-A instant Back is iideal for your summer getaways to capture memories instantly with your LC-A cameras in hand!

  • Tales from the Trip: Atacama Blue

    written by plasticpopsicle on 2014-05-14 in #world #locations
    Tales from the Trip: Atacama Blue

    As intrepid travelers, we don't only search for an adventure or a change of scenery -- we also seek to capture fascinating moments in photographs and tell the compelling stories behind them. In this wanderlust-driven series for the Locations section, we showcase some of the most fascinating travel photos by our fellow lomographers, along with their tales from the trip.

  • Tales from the Trip: Finding the Right Way

    written by plasticpopsicle on 2014-03-31 in #world #locations
    Tales from the Trip: Finding the Right Way

    As intrepid travelers, we don't only search for an adventure or a change of scenery -- we also seek to capture fascinating moments in photographs and tell the compelling stories behind them. In this wanderlust-driven series for the Locations section, we showcase some of the most fascinating travel photos by our fellow lomographers, along with their tales from the trip.

    1
  • December 20th Advent Offer: There’s still time for a last minute treat! (Online Code: CRACKLINGCAMERAS)

    written by jacobs on 2014-12-20 in #news
    December 20th Advent Offer: There’s still time for a last minute treat!  (Online Code: CRACKLINGCAMERAS)

    Are you still looking for the perfect last minute gift for your loved ones this holiday season? Don't fret! We have another incredible 15% off our cameras today! Surprise and delight with our wide selection of beautiful analogue cameras. But don't wait around to take advantage of this sweet deal — it's only valid for today.