Sintel, the Durian Open Movie Project » Blog Archive » Developer *Fail* Moments

    

Developer *Fail* Moments

on July 14th, 2010, by ideasman42

Over the course of the project its come to my attention that as developers we get it pretty easy with lots of encouraging comments from the community, however I worry that we don’t properly communicate the troubles we invariably cause when poking about with Blender.

To this end, here is a laundry list of memorable mistakes we made during the project.
- Campbell :)

  • Wondering why his IP address has a collision with the server, Brecht changed it so his system could become a contributing render node, for some reason nobody can access the server.
  • Campbell spent 3 hours to find out why converting to mesh wasn’t working, copying from the wrong object, one character change.

  • Campbell spent an hour or so trying to work out why the smaller the pow(size, 0.0001), then pow(size, 0.0000000001) became more accurate. Turns out no multiplication was needed the rotation matrix has the scale already applied: pow(anything, 0.0)==1.0
  • After considered discussion Campbell and Brecht were sure the artists didn’t have custom patches applied and it was save to refresh their builds, Brecht modifys the build script wiping Campbell’s build directory, documentation scripts and WIP patches, Later we discovered Nathan also had 2 uncommitted rig-generating scripts, Luckily he managed use some ninja undelete-fu to get them back.
  • Campbell committed removal of most of blenders source code while attempting a merge.

  • Brecht’s tiling/mipmap system compares file access times which the farm ignored, dancing textures!
  • Campbell updated the RNA API for keyframe paths, forgot to search for function calls and totally broke keyframe insertion.

  • Campbell wrote a multi-processing hair-sim generator to run overnight which managed to overwrite all the freshly generated data while trying to render result for review.

57 Responses to “Developer *Fail* Moments”

  1. wo262 Says:

    hahaha

  2. shrinidhi Says:

    :D

  3. Robbie Losee Says:

    laughing out loud! reminds me of so many of my mistakes, and I’m not even a dev.

  4. ROUBAL Says:

    It also reminds me some mistakes in my former work in electronics : Spending half a day on wiring a huge special plug and discovering when finished that caps and rings had not been put on the cable !

  5. Alexsani Says:

    you know what blender needs, though? Some sort of protect cache button for sims. I really love sitting through an hour of sim-ing and then accidentally change one variable and it has to re-sim EVERYTHING.

  6. D Says:

    This is the most depressing Durian entry ever! D:

  7. ibkanat Says:

    Good to see you guys can laugh at yourselves sometimes it is hard to get over your own mistakes.

  8. Tommy Says:

    Remember Kids, when Campbell is around BACKUP your systems!

  9. MadSquirrel Says:

    # D Says:
    >July 14th, 2010 at 13:57
    >
    >This is the most depressing Durian entry ever! D:

    As a developer, I’d say it’s always encouraging to see I’m not the only one to make such mistakes ;)

  10. MadSquirrel Says:

    Fortunately, the team knows the rule : “Remember: Data you don’t have at least two copies of is data you don’t care about.” :)

  11. nawabz Says:

    i feel ya man. image tell more than a` thousand words.

  12. Lyle Walsh Says:

    I’ll be the first one to admit that the most common programming and ble(u)ndering error is usually resolved and filed under
    resolution code: ID10-T

    Hey that’s a good one: its a combination of Blendering and Blundering -
    so I’m Ble(u)ndering!

  13. nawabz Says:

    i remember opening my lamp because it wasnt working, 3 hours later i needed to switch it on for it to turn on DNNNNN

    :S

  14. alalo Says:

    hahah These moments must be total adrenaline. :>

  15. panzi Says:

    Why are there even uncommitted scripts/patches? Oh, you don’t use a distributed source control management (DSCM) system. I’d say it’s time to change that! :)

  16. OBI_Ron Says:

    Humans? You let humans code?

  17. NB-DanTE Says:

    the fault is a part of human kind

  18. Andy Says:

    Developing software is like skiing… if you don’t fall over, you’re not trying hard enough!

  19. Zazizizou Says:

    And you share this with everyone :o ?

    It’s good to know what kind of problems you face. Thanks for the update :D

  20. Federico Cáceres Says:

    And after all Blender still rocks (and things get commited). Thank you guys, you are awesome!

  21. Dilly Says:

    Its good to see that its not just me that make such “Fail moments”. =P

  22. yoff Says:

    Very funny :-)

    I would be surprised, though, if there are not at least ten times the problems mentioned here, on a project of this size…

  23. DudeFromKansas Says:

    Lol, I applaud you guys for sticking your necks out there like that! To err is to be human though and you and all the Durian team are doing a great job!!

  24. ben Says:

    Developer or artist, some days here can only be summed up by the following:

    http://i524.photobucket.com/albums/cc326/dgiharris/DoubleFacePalm.jpg

    Kudos to the devs for their work BTW. :)

  25. Anonymous Coward Says:

    Should change the title to “Get a blender developer of your own”.

  26. matt Says:

    The only people who don’t make mistakes don’t do anything (and usually consider themselves experts :) )
    As someone who has been programming for nearly 30 years (earning a living at it for 17+ of them) I have to say I’ve had a few double facepalm moments of my own … the trick is to learn from them and find new mistakes to make next time :D

  27. Whimsy Collective Says:

    Being a Render Wrangler, folder name conventions and to which folder on the disk mount is for final or artist review. Can be hell for us when we have to QA team the final output for production quality review.

    Also, the merging of svn’s *OUCH*. Doesn’t Mericurial (Google Code supports it) or GIT handle that better.

    Something to consider reading.

    http://arstechnica.com/open-source/news/2009/04/google-code-adds-mercurial-version-control-system.ars

    Here’s the direct link to Google Code analysis, http://code.google.com/p/support/wiki/DVCSAnalysis

    Be sure to read the comments as there are some chatter and difference over the history point in the comparison.

  28. Whimsy Collective Says:

    *correction* Also, the merging of svn’s *OUCH*. Doesn’t Mericurial (Google Code supports it) or GIT handle that better.

    I was referring to the history aspect of the DVCS. Doesn’t user have the source on their local disk as well, that way they can go back a revision.

    @Nathan What was the ninja undelete-fu?

  29. Blenderism Says:

    Wow, despite all of the set backs and problems, the movie still got done! Durian Team Amazing Feat #26144!

  30. namekuseijin Says:

    oh teh joy of programming!

  31. ideasman42 Says:

    @Alexsani, once cache is external I found it doesn’t get removed, but this definitely is an area that needs work.

    @MadSquirrel, surprisingly very few of our problems have been todo with data loss/backups etc. We have however had the server disk die but luckily was able to mirror it and swap it out.

    @alalo, if you want to raise your heard rate try formatting a USB disk at 2am on a server where switching /dev/sdb with /dev/sdd would wipe all your art assets :D . Now I tend to do this on my workstation instead.

    @panzi, as a rule we commit everything to Subversion, but when working on stuff we don’t always do this immediately. As for DVCS, Id like to try GIT but didn’t get around to it yet.

    @yoff, for sure, things like this happen all the time though normally there more complicated and hard to explain- Fix X breaks feature Y in corner case Z, less facepalm moments and more “Isn’t blender complicated, perhaps we should ***remove some features!***

    @Anonymous Coward, we’re taken!

    @matt, exactly, when approaching problems its good to get a sense of which things to check first to avoid loosing a lot of time on dead-end leads.

    @Whimsy Collective, yep, experiencing this now, lots of files being rendered, flickering, restarting nodes checking farm heat… etc.
    SVN Does merging well enough (though not as well as GIT I hear)
    was totally my fault.
    Thanks for the link, as said above, Id like to try DVCS at some point.

  32. AIBlender Says:

    Entertaining in a dark kind of way. We need a pixar type out takes for sintel :)

  33. J. Says:

    An idea: Scales getting angry and biting in Sintel’s finger. :D

  34. MeshWeaver Says:

    ouch…i’ve had a few moments like that…not nearly as serious, but still, i understand…i think

    but Blender is still awesome, and you devs are still awesome :D really big thanks for all the great work you’ve done – Blender is now even awesome-er than before :D GO DEVS!

  35. ronbravo Says:

    LOL! Quite honestly, this was one of the best and most entertaining posts I’ve read on this project. People don’t usually understand how frustrating development can be at times, and just how easy we can over look things or completely screw things up. Thanks for shedding some light on it by sharing your experiences. Keep up the good work.

  36. ronbravo Says:

    Oh yeah, Campbell if you do investigate version control software in the future, please experiment with Mercurial. I know everyone loves to use GIT, however, mercurial is pretty nice one too. I don’t know which one would suit Blender’s needs better but just thought I’d mention it.

  37. Max Puliero Says:

    eheheheh

    very funny :D

  38. Whimsy Collective Says:

    Link for Git Front-end tools, and what not…

    https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools

    Git-Cola for front-end seems by comparison among the others.

  39. Borgleader Says:

    The joys of software development! Oh man those were pretty hilarious.

    On the same note, in my recent programming classes (I’m a soft. eng. student), I’ve often helped friends who were stuck on an error message for very long to notice that they’d forgotten a semi-colon at the end of a class declaration which once included elsewhere was breaking more than half their code.

    I feel your pain and hope you guys are pulling through all right :)

  40. Juan Romero (Onnevan) Says:

    Am I the only one who thinks that at this stage of the project, there should be much more eye candy posts, even if they were just a couple of pictures or rough renders or whatever?. This kind of posts are good too, but come on:
    world cup–> hire a Durian member–>Developer facepalm—>???
    please, give us some eyecandy, just a little, nice, hd or even sd snapshots!!

  41. joeri67 Says:

    Lol.
    Being a good programmer makes code fluid.
    So yeah, fluid splatters.

    Today I kept wondering why adding an object didnt work, untill I remembered I’m in object select mode and the new objects where added in the scene… Argh.

  42. Cathy & India Says:

    Nawww….Campbell needs a hug….:(

  43. Mike Says:

    Lol. I’m not brave enough to blog about my mistakes, but if I was, you guys would laugh pretty loud too.

  44. maddes Says:

    @ Juan Romero (Onnevan)
    I completely agree with you. I like all the curent posts too, but hey, the team ought to be able to SHOW something interesting.

    For example: Several months (!) ago they promised to keep us informed about the progress of the first minute – anyone remembers about that?

  45. kellpossible Says:

    ideasman42:
    I made that exact same mistake you mentioned when trying to reformat a thumb drive in a hurry! It was hell for a while! Why did they have to make the names so similar?! I thought i’d lost everything on my hard drive but then I discovered photorec. I would recommend it to anyone in a similar situation. Chances are, you’ve probably used it before!

  46. Jack_the_Bear Says:

    LOL…..hilarious yet depressing. are you guys going to have a technical blooper reel on the DVD? that would be awsome…… ya there has been plenty of times where I have been working on a project, then suddenly you would see me screaming and tearing my hair out because one little change and a whole scene is messed up. keep up the good work guys

  47. joe doe Says:

    it’s nice to see, how you guys handle that. Where I’ve been working as a coder, mistakes were handled like “oh my goodness, you’re the most worthless douchebag I’ve ever met. I’ll have to report that to the boss” – when that guys made mistakes it was like “it ‘s just a crappy system… AND the fault of the project manager” :)

  48. Big Fan Says:

    So Ton made only *one* mistake…
    Ton is impressive. ;)

  49. blendNmix Says:

    Big Fan: No this is sade! Because Ton is human then. :)

    DurianTeam: You can even fail to finish the movie that you will remain our heros! :p

  50. J. Says:

    Are you out of your mind?
    Of course they’ll finish the movie, maybe with Ton in the ” angry James Cameron-like mode”, but they will! :P :P :P

  51. Triskelion Says:

    those kind of cock ups are par for the course! well done on weathering them in good humour and not knocking each others lights out.(presumably)

  52. Wray Bowling Says:

    This further is pushing me to learn the Blender source. Ton *did* say to me “we need coders.” and more watchful eyes, and more people to attack bugs (especially with Matt leaving full time bug squashing) certainly can’t hurt.

    You should append a little bit at the end there like “Want to have just as much fun? Try these links to get started with Blender source!” lol

    Secondly, to admit you’re wrong is bold. That deserves a donation. $$$

  53. Manu Järvinen Says:

    @ Juan Romero (Onnevan)
    @ maddes
    I totally agree also. Very simple posts like this
    http://durian.blender.org/goodies/concept-art-dragons/
    or this
    http://durian.blender.org/production/render-gallery-update/
    are just plain gold to look at!

    The first minute might have been converted to more complicated issue to share if it has developed to reveal too much information – but at least more of these “dump stuff from our ‘weekly’ folders” (said in the latter link) would be more than welcome – in addition to these also great techy posts.

  54. thejo Says:

    Shit happens dudes, but the work you’re doing is awesome and very much appreciated. Keep up the good work guys.

  55. Gaurav Says:

    in spite of these failures , Blender developers team had done great job
    :)

  56. farsthary Says:

    LOL :)

    It happens me all the time, at least I’m not alone ;)
    Cheers and very good post, we learn as much from failures than success

  57. Frank_robernson Says:

    I understand how horrible all this is, even using 2.49b it was still hell trying to manage things and keep my mind on track and not want to turn my skin inside out.

Leave a Reply

You must be logged in to post a comment.