Wrestling With Our Photo Server


This week we had some issues with our photo server. On Wednesday black thumbnails appeared for some photos and today we had to shut down the upload function for a short period. In this article we will try to explain what happened.

Server vs. Us

So what was the deal? Wednesday morning we had problems reaching our server (which is remotely hosted by Amazon) that resizes, uploads and distributes our photos – there was no way connecting to it. So we set up a new server to bring everything back to normal and turned the other server off. There were some changes in the configuration which resulted in a small amount of photos having now a wrong (internal) filename (which makes it hard for us to find these photos or even results in showing a wrong photo). This issue is fixed and the upload is working again.

What are we doing next? We will download all original photos uploaded during that time to our hard drive, calculate the correct (internal) filename and rename them. This will take a while but on Monday it should be finished (so if you still see some black thumbnail this is the reason). If we find any problematic photos we will contact you and let you know about it. In the best case everything is just fine, in the worst case we will let you know which photo was affected and ask you to upload it again.

We are really sorry for the trouble and are focused on resolving it. Thanks for your understanding! If you experience any problems with the upload please let us know in the comments.

Update: we are in the middle of testing the photos – the first ones are ready and it seems some of them are broken. We will keep you updated on the process and as promised contact everyone who was affected. Thanks again!

Another update: we’ve found all photos. Everyone whose photos were affected got a message with the filenames. We’ve deleted those photos – sorry, we had to do this, there is no way to repair these – and gave everyone 10 piggies for the inconvenience. Thanks for your understanding!

Yet another update: it takes a bit of time to invalidate our cache, but we are on it! Errr, sorry again for the delay.

Hopefully final update: all messages are out, please reupload those photos again. And if your are one of the users whose avatar photos seems a bit wrong: please reupload this one again, but resize it a bit (can be 1px smaller) so the checksum is different. Then everything should be fine again! Thanks to you!

written by recurving on 2010-12-03 in #news #relaunch #problems #server


  1. squamy
    squamy ·

    Oh no! Servers always like to cause trouble for the weekend. Hope it hasn't been too stressful!

  2. illy
    illy ·

    Ah, this might also explain why there were some fisheye-photos in my uploaded pictures, though I don't even have a fisheye camera :)

  3. gnarlyleech
    gnarlyleech ·

    i like the photo illustration. lol

  4. beanieboi
    beanieboi ·

    @squamy you are damn right, always before weekend! :(

  5. recurving
    recurving ·

    @illy: errr, exactly that… :) sorry again!

  6. eatcpcks
    eatcpcks ·

    This is the Murphy's law:D

  7. fivedayforecast
    fivedayforecast ·

    Tis fine

  8. clickiemcpete
    clickiemcpete ·

    Thanks for all the hard work, I'm sure it is a nightmare keeping up with all the stuff you guys having going on.

  9. idigtulsa
    idigtulsa ·

    Let's flip this around and look at it like a good thing.....If non of us were taking any pictures, then the servers would be lonely, board and everything would still be fine in the land of The Lomo Internets. Thank for your hard work behind the scenes so we can keep at it!!

  10. discodrew
    discodrew ·

    Appreciate the heads up!

  11. dogma
    dogma ·

    Let's hope everythings gone be fine!

  12. brittany
    brittany ·

    @idigtulsa ahah i totally agree with you

  13. rodrigoalmeida
    rodrigoalmeida ·

    Hi there! Thanks for the information and for trying to solve the problem. I'm still having problems with this album :www.lomography.com/homes/rodrigoalmeida/albums/1657232-supe…, which I uploaded last Thursday. Most photos have black rectangles at their bottoms and I can't use any of them on lomowalls. Would you check it for me please? Thanks a lot!

  14. recurving
    recurving ·

    @all: thanks for your understanding – we really appreciate that!
    @rodrigoalmeida for sure – we're gonna take a look tomorrow when we are all back in office!
    @clickiemcpete: oh, far from being a nightmare, it's a lot of fun! just sometimes – well – challenging… :) but we are very happy our servers don't get bored and have lots to do…

  15. snug
    snug ·

    Wow... that sounds very complicated and quite a long job for you... certainly makes us appreciate the great service you do offer us.. your still much better than flickr!

  16. rodrigoalmeida
    rodrigoalmeida ·

    @recurving: thanks!

  17. recurving
    recurving ·

    @all: I've updated the article – we are still checking all photos (around 2,000 are done already), some have issues we are afraid… but gonna compile a full list and get in touch with everyone!

  18. wkdroots
    wkdroots ·

    Almost all of my pictures are affected. I have the same problems as rodrigoalmeida =( I didn't notice this posting so I wrote your Facebook page asking for help and I got a very fast response from Christopher Zinck who took a look at my page, emailed me a link to this article and answered all my questions. You guys rock!!

  19. recurving
    recurving ·

    Hey guys – we've found the photos and will send out a message to all of you – thanks again for your understanding!

  20. recurving
    recurving ·

    @wkdroots: Sorry to hear that – you (and all others) will get a notification with all affected filenames and albums! And high-five to @dont_think (Christopher) for being so fast… :)

  21. niko_fuzzy
    niko_fuzzy ·

    how to know which picture got deleted?

  22. recurving
    recurving ·

    If you didn't get a message all your photos seem to be fine. If there are affected photos and you didn't get a notice please post the urls to these here…

  23. bass_clarinet_2000
    bass_clarinet_2000 ·

    I'm uploading mine now. I was surprised to see how many of mine were effected but that is ok. I'm fine with it:D Also, the piggies make me a happy girl.

  24. calvin_hardy
    calvin_hardy ·

    Oh no :L

  25. spoeker
    spoeker ·

    hey can you have a look at these 2 albums of mine:

    I got a message first saying that Im affected and that you will contact me again with more details. But so far I didnt get a second message but the photos in these 2 albums seem to be affected - black rectangles at their bottoms and I can't use any of them on lomowalls.
    and these two are showing up as different photos which arent mine:

    but please don't delete them, some of them are submitted to contests where I already reached the limit ;)
    maybe I can reupload the photos and delete the others as soon as the competitions are over? Other ideas? thanks for your help

  26. rdp_ribeirense
    rdp_ribeirense ·

    thankyou :)

  27. wkdroots
    wkdroots ·

    @recurving: My pictures havent been deleted and i didnt get a message. I did get the 10 piggies though. Here are the links to the 2 albums that are affected (my ony 2 albums)

    www.lomography.com/homes/wkdroots/albums/1649870-cross-proc… (first 8 photos)

    www.lomography.com/homes/wkdroots/albums/1649339-black-and-… (first 6 photos)

    Also, i just sent you a private message. Thank you for the help!

  28. rodrigoalmeida
    rodrigoalmeida ·

    Dear @recurving,
    I got the piggies, but didn't receive any message. I still have the same problems with this album: www.lomography.com/homes/rodrigoalmeida/albums/1657232-supe…
    Could you take a look at it please? If I have to upload it again, it's ok. I just want to be sure if I really need to do it. Thanks a lot for your hard work.

  29. tallgrrlrocks
    tallgrrlrocks ·

    Thanks for tirelessly giving the effort to find solutions! Great job.

  30. jabuka
    jabuka ·

    When can I reupload all my delated photos? I reuploaded them but most of them were shown in black in the lab as before. Maybe they uploadeded properly, but I'm really not sure, so I delated them again.

  31. recurving
    recurving ·

    @jabuka: the photos should be fine, but the ones shown are still the old (cached) ones… we are not invalidating our caches so they get refilled.

  32. spoeker
    spoeker ·

    ehm.......now my two albums I posted you here in the comments are all white....?
    Can someone please help me? Should I delete them now and reupload? I didnt get a message with the filenames so far..and some of these photos were competition-submissions, eg. for the "word of the month december" where I can only submit one photo..and the wrangler competitions, Rip Curl and some more I can't remeber....

  33. recurving
    recurving ·

    hey @spoeker: we are on it (your photos, the filenames and the submissions)!

  34. wkdroots
    wkdroots ·

    deleted and reuploaded three times now. Now im having the same problem as @spoeker. All the new photos are white. Im not to sure what to do anymore. I never recieved a message and deleting and reuploading clearly isnt working. When you get a sec can you check it out?


  35. spoeker
    spoeker ·

    @recurving - thank you! looking forward to your message :)

  36. recurving
    recurving ·

    hey @spoeker and @wkdroots: you should have gotten the message now and the photos can be reuploaded!

  37. maria_vlachou
    maria_vlachou ·

    I reuploaded my latest album 3 days ago and it's still all thumbs! Am I being impatient? Thanks for trying...

  38. spoeker
    spoeker ·

    @recurving thanks! I'm not at home at the moment so I can't reupload and resubmit for the wrangler competitions, I hope I made it in time but it says only 1 day left :(

  39. brommi
    brommi ·


More Interesting Articles

  • The Lomo'Instant Does Halloween!

    written by camilla_illa on 2014-10-30 in #gear #tipster
    The Lomo'Instant Does Halloween!

    Halloween is fast approaching and we are getting ready to celebrate till dawn with our favorite cameras. I had the lucky chance to try the Lomo'Instant in preparation for this scary night!

  • 10 Steps to Better Photography with the LC-A+

    written by Julien Matabuena on 2015-06-08 in #gear #tipster
    10 Steps to Better Photography with the LC-A+

    With features that allow one to be as creative as possible and a size compact enough to bring it anytime, anywhere, the LC-A+ is indeed an embodiment of our 10 Golden Rules. In this week's feature, we list down some of the ways you could up your photography game with this wonderful camera.

  • 10 Steps to Better Photography with the Sprocket Rocket

    written by Julien Matabuena on 2015-06-01 in #gear #tipster
    10 Steps to Better Photography with the Sprocket Rocket

    Looking to amp up your photography with the Sprocket Rocket? You've come to the right place! In this feature, we list down some of the ways on how you could make the most out of this humble yet functional 35mm shooter.

  • Shop News

    Diana Mini and Flash Petite Noire at 25% off

    Diana Mini and Flash Petite Noire at 25% off

    At 25% off you can take dreamy 35mm images with this little black beauty. Beam coloured light into your shots with its accompanying Diana Flash Back accessory and be the analogue king of the night.

  • 12 Months, 12 Projects: March Of Caffenol

    written by Maaike van Stratum on 2015-04-12 in #world #lifestyle
    12 Months, 12 Projects: March Of Caffenol

    My list of resolutions for 2015 consists of 12 projects, one for every month. March was for caffenol. You have probably heard of the amazing fact that you can develop black and white photos with coffee, sodium, and vitamin C. I had tried this before but with less than stellar results. Somehow, there's always something going wrong. Time to devote a few rolls to caffenol to finally get the hang of it.

  • December 14th Advent Offer: Take Advantage of our Festive 3 For 2 Film Deals (Online Code: 3FOR2HOLIDAYFILM)

    written by jacobs on 2014-12-14 in #news
    December 14th Advent Offer: Take Advantage of our Festive 3 For 2 Film Deals  (Online Code: 3FOR2HOLIDAYFILM)

    Seeing that we love to spread the cheer around here, we're giving you another chance to load up on our awesome film with today's Advent deal! Choose a classy black and white film, like our Lady Grey, or get creative and colorful with one of our Redscale films. We're certain that no matter what you choose, you'll have a great time making memories with tons of lovely analogue photos this year!

  • Lomography x National Portrait Gallery Rumble WINNERS!

    written by hannah_brown on 2015-01-06 in #news
    Lomography x National Portrait Gallery Rumble WINNERS!

    We had a staggering 1110 entries to this rumble. We have picked our way through all of them and were very inspired by the quality of all your photos! It was a real challenge narrowing it down to just 3 winners. Find out who won this great competition.

  • Shop News

    Immortalize your best shot on Aluminium!

    Immortalize your best shot on Aluminium!

    Carefully enlarged from your negatives onto premium photographic paper by lab professionals, each picture is a unique piece of craftsmanship.

  • A Salute to the Masters: Barkers (A Tribute to Jack Delano)

    written by sirio174 on 2015-05-02 in #world #lifestyle
    A Salute to the Masters: Barkers (A Tribute to Jack Delano)

    This is tribute to the Farm Security Administration photographer, Jack Delano, and his photographic series dedicated to barkers. For this article, I chose a series of photos I took this year at the traditional Easter Fair in my city, Como, using a classic rangefinder camera loaded with a roll of black and white film.

  • My Day in Analog: Bike Rides with My Dad

    written by mmxi on 2015-04-28 in #world #lifestyle
    My Day in Analog: Bike Rides with My Dad

    My dad and I have been riding our bicycles for as long as I can remember. We had no camera back then, so I only have the pictures from our recent rides. Each ride to the island of Khortitsa today is like a reunion with my childhood and my father's care.

  • I am a Mad Analogue Scientist: New Experiments with 35mm Films and Chemicals

    written by blackfairy on 2014-08-07 in #gear #tipster
    I am a Mad Analogue Scientist: New Experiments with 35mm Films and Chemicals

    Unfortunately, it happens sometimes that your resulting pictures are not what you expected - the image doesn't look that good, the colors are bland, and the subject is banal. Indeed, it couldn't be picture of the year! Herein I propose a second chance for your pictures by modifying your 35mm negatives. Just pick up some ideas from here, experiment, and scan your negatives with the Lomography Smartphone Scanner. Anything is possible: burning, scratching, putting on hydrochloric acid, balsamic vinegar, nail polish, bleach, or raspberry juice... use your imagination and write down your new film soup recipe! You can find a sample of the effects in this article.

  • Shop News

    Fisheye One Pearl Blue

    Fisheye One Pearl Blue

    Snap dazzling 35mm fisheye shots at the push of a button with this shimmery blue lightweight beauty.

  • RYU x RIO: World Cup 2014 on Kodak Aerochrome Film

    written by zonderbar on 2014-08-11 in #news
    RYU x RIO: World Cup 2014 on Kodak Aerochrome Film

    Weeks have passed and yet Germans are still celebrating the victory of their heroic football team. Shortly before the World Cup started, we took notice of an interesting photography project on Kickstarter. Berlin-based sports photographer Ryu Voelkel called for help to create a football photography book like no other. The campaign was successfully funded. Ryu made his way to Brazil and came back with amazing shots including some very special Kodak Aerochrome photographs. Meet Ryu and learn more about him and his special moments at the WC 2014.

  • A Bit of Boston in the Heart of Vienna

    written by bgaluppo on 2015-03-05 in #gear #lifestyle
    A Bit of Boston in the Heart of Vienna

    The brand new Lomo'Instant Boston Edition hit the shelves around the world this week. In an attempt to celebrate the home of the ol' bean and the cod, we ventured out in search of some classic Boston flair. While we couldn't find a baseball match or a proper clam chowder, we did have a blast chasing red brick walls and spicing up our photo shooting with some iconic American flavors!

  • Photo Stories: The Main Lodge Interior 1950s by marcus_loves_film

    written by Julien Matabuena on 2015-07-28 in #world #locations
    Photo Stories: The Main Lodge Interior 1950s by marcus_loves_film

    A couple of years ago marcus_loves_film had the opportunity to spend time at a lodge more than half a century old in Woodruff, Wisconsin. Through these photographs, he had documented one night of his stay.