Sintel, the Durian Open Movie Project » Blog Archive » Memory & JeMalloc

    

Memory & JeMalloc

on April 15th, 2010, by ideasman42

While memory is a dry topic, many of you would have experienced Blender grind to a halt (or worse), with big files the workstation can’t handle.

Yesterday Beorn was having trouble loading a scene which was taking a lot of memory, After concluding it wasn’t a memory leak in blender I looked into why Blender would show 75mb in use while the system monitor shows over 700mb.
It turns out is because the operating system cant always efficiently deal with applications memory usage and ends up using a lot more ram.

So I tested jemalloc, an opensource drop-in replacement for the operating systems memory allocation calls, used by firefox, facebook and freeBSD according to their site.

I was surprised to find memory usage went down by up to 1gig in some cases, without noticeable slowdown, using less memory in almost every case.
It can also decrease render times in cases where the system starts to use virtual memory.

The first example loads a complex scene, and then a blank file, notice there is over 600mb difference.

The second graph shows rendering the sintel model, I’d like to have made a few more examples but don’t have much time right now.

While testing we found Blender exposed a bug in jemalloc’s thread-cache, Jason Evans was kind enough to look into the problem for us, fixing it the next day in bugfix version 1.01.

JeMalloc is now used on every workstation, if all goes well we may include this with Blender as Firefox does.

For more info see: http://www.canonware.com/jemalloc/

- Campbell

Notes…

  • Hoard was also tested, but it didn’t improve memory usage all that much.
  • Edited, now ‘decrease rendertimes’.
  • Comparing against the default allocator in Linux (modified ptmalloc), one could argue the problems is because of bad memory usage within Blender, from what I have read the default malloc in linux is quite good. Nevertheless it may prove to be an advantage for us.
  • For anyone who wants to test on *nix, you don’t need to rebuild Blender, just pre-load:
    LD_PRELOAD=/usr/lib/libjemalloc.so blender.bin

43 Responses to “Memory & JeMalloc”

  1. Anonymous Says:

    Second graph is broken (Error 404)… :(

  2. Anonymous Says:

    It should be http://durian.blender.org/wp-content/uploads/2010/04/jemalloc_render.png

  3. John R. Nyquist Says:

    Great job, Campbell! That’s a cool find. (FYI, your second graphic is not showing)

    BTW, what is the configuration of the workstations being used to develop Durian?

  4. D Says:

    Stupid system memory allocators. JeMalloc sounds quite impressive. :D Awesome discovery, Campbell!

  5. Revolt_Randy Says:

    Way to go, cutting down on sloppy memory usage is a plus for everyone!!!!!!!!! Two thumbs up!!!

  6. David Says:

    “It can also increases render times in cases where the system starts to use virtual memory.”
    :(
    I hope you meant: decrease
    :)

  7. mjordan Says:

    Nice but on which OS are you comparing those 2 allocators? Linux?
    I know the Hoard allocator is also very efficient on multithreaded multi processors systems. It is also available for Windows and Solaris. Worth a try, expecially on Windows:

    http://www.hoard.org

  8. Felix Says:

    @david, unless I’m interpreting it wrong,
    according to the graph he really meant increase… :/

  9. Anonymous Coward Says:

    …or just install freeBSD on all the boxen.

  10. J. Says:

    @Felix and David:
    It depends on how you interpret it, as it does increase efficiency of memory, hence it decreases the actual use of memory! ;)

  11. Robbie Losee Says:

    Very neat. Sounds like yet another way to raise the ceiling and make room for the big dragon. Viva la render speed!

  12. MrMowgli Says:

    Great job!!!

  13. Martin Lindelöf Says:

    ” if all goes well we may include this with Blender as Firefox does.”
    why not? if it fixes loading big scenes.

  14. sid350 Says:

    Good news. I can’t render new Sintel.blend on 4Gb RAM, may be now I can.

  15. ideasman42 Says:

    @John, Didn’t post about this yet but we’re using much the same setup as with bigbuckbunny. blender compiled locally on each PC, desktop icons point to a shell script on the server for building and running. So we can change how blender starts/builds from one place, making it easy to switch allocators.

    @Felix, Its hard to say if this slows things down much without running more tests (time consuming), To be fair we should also compare on Windows & Mac to see if there is much difference. For now I’m satisfied its at worst slightly slower and at best can save the system from running out of ram :) .

    @Anonymous Coward, sure, but thats a lot of work to setup all these PC’s with BSD, switching the allocator for Blender is trivial.

  16. D Says:

    Stupid system memory allocators. JeMalloc sounds quite impressive. :D Good stuff, Campbell.

  17. ruddy Says:

    Woow, great Campbell! :)

    What do you think of Eskil’s approach who avoids linked lists of small elements and favors arrays (is a hybrid approach with linked list of big arrays instead of standard linked list feasible with blender structure?) to avoid data fragmentation and keep memory aligned for SIMD-purposes and avoid cache-misses? Indeed, even if jemalloc is way better, there’s still a lof of memory “lost” (blender still uses 800 Mb when the blank file is loaded)? (I’m not a programmer, so sorry if this sounds stupids.. I know that one of ansi C array drawback is that they cannot be allocated on the fly, but say that there’s a defined array size in blender, say allocation of 1 Mb chunks, and arrays connected by linked list.. a new memory manager…
    http://news.quelsolaar.com/ (I am adding and subtracting)

    If I remember, Yves Poissant has said that cache misses are responsibles of many slow-down in render engines, mostly caused by fragmented memory, but would this implies a full, mammoth rewrite? (no compatible with the tight Durian schedule)

    Regards

  18. ruddy Says:

    Sorry, I’ve misinterpreted the figure! (about the moment when the blank file is loaded), so I remove my statement about the lost memory…

    Regards

  19. RockyMtnMesh Says:

    Memory allocation is not dry! You take that back!
    Seriously, the combination of hard computer science and 3D art is wonderful. I would love to see some more ‘technical’ posts here. Keep up the great work!

  20. Still Says:

    Hi guys!

    I have tried it here, the machine is a Notebook Turion 64 X2 with 2 Mb of RAM and it didn’t make any difference here.
    I have tried with Blender 2.49 and 2.5 alpha2.
    I’ll be at home in a few hours and I’ll try with one of my models and I’ll post the results here.

    []‘s,

    Still

  21. ideasman42 Says:

    @ruddy, havn’t done benchmarks with this personally but believe your correct. Vertex Groups are one of the main offenders for small allocations. With rendering I think there is more to it then using arrays, we already use array chunks for faces but theres no one correct way to solve this since it depends how the data is accessed in each case, a ray-tree could be optimized group near faces in memory, image tiles may also help.

    @RockyMtnMesh, knew there were some of you lurking out there, otherwise Id not have posted :)

    @Still, to really test this well we would need more configurations. Of course if users can test this is great, what OS are you running ,32/64bit?, how did you test jemalloc?.

  22. Jan de Vries Says:

    The sponsor page mentions that you will be building your own renderfarm. Has that begun?

  23. Zane Runner Says:

    Great job i’m currently working on a very large city scene thats starting to slow my machine down a little. this is something i’ve really been hoping to have looked at.

    Something i noticed between 2.5 beta an new 2.5 builds is that memory usage is up 3 times higher than it was before

  24. AlesWolf Says:

    Good work!!

  25. ideasman42 Says:

    @Jan de Vries, expect some news about this soon :)

    @Zane Runner, this is worrying, is this blenders readout of the memory or from the operating system? Blender can print out its memory usage (SpaceBar, Memory Statistics). This could help you find the cause of the memory usage. Otherwise if blenders readout in the header is very different from the operating systems, this could be because of memory fragmentation.

  26. bruno Says:

    hey, great!, there some patch about memory, page faults and cold start up time enhancements here:

    http://blog.mozilla.com/tglek/

    Good work!

  27. Zane Runner Says:

    i never looked at the system just noticed the readout on the top header in blender next to vert and face count.
    i’ll download the latest 2.5 build i can get for windows 64 bit and see/compare

  28. Zane Runner Says:

    nevermind can’t find alpha 0 anymore and alpha 2 uses just as much memory as the latest

  29. MTracer Says:

    I agree with Rocky 100%. I just bought another 2GB of ram because blender is a bit of a pig. This could really help…

    :D

    MTracer.

  30. Alex Delderfield (AD-Edge) Says:

    Awesome, posts like this are interesting, keep them coming.
    Ram usage is something Ive been hoping to optimize for a while now, ever since playing around with the BBB production files which always crashed Blender when rendering due to Ram, so this certainly looks good.

    It would be great to hear how well this goes for Blender on Windows however.

  31. jesus mejuto Says:

    Great! I have been following blender from the early days and each time is getting better and better. Great job.

  32. matty 686 Says:

    I am a windows user and I am looking forward to making big files with the 64bit workstation i am getting this summer for graduation so improving ram usage would be greatly appreciated I cant figure out how to use these ram optimizers so if you plan to use them please code them into blender so non programing artists like me can reap the benefits too

  33. K-Linux Says:

    I don’t know how they rendered for previous projects. But I hope they are going consider “parallel computing” when they start Final Rendering Process!

  34. B.Tolputt Says:

    This is something I like to see – information on how Durian is benefiting Blender development.

    ideasman: Small allocations are the devil, to quote Mama Boucher (of ‘Water Boy’ fame), both in terms of memory fragmentation & the resulting-side-effect of memory requirements.

    Whilst the jemalloc solution is a good one, and probably should be kept even if it is a tad slower (as I read it), the issue with looking at things from a “global perspective” as opposed to the way memory is allocated for each part is that you can only “fiddle around the edges”. If, for example, the Vertex Groups are making too many small allocations, the best solution is to refactor the way memory is allocated for it’s use or the data structures it uses. In C++, memory allocation can be tweaked per structure (through custom new/delete operators); but in stand-alone C I acknowledge this would be more difficult without strict adherence to allocation functions/macros.

    And yes, I am aware this is all “hand-waving” & academic given the real situation is that you needed a quick, global solution to the problem to get Durian back on track. I commend you for finding the solution the way you did (I have a bad habit at looking into the micro problems before considering macro solutions). In fact, changing the underlying allocator would be one of the LAST things I’d do – wasting alot of time in the process. Perfectionism often loses out against practical realism on the ground!

  35. FishB8 Says:

    WOW. Just tried running with jemalloc. Sculpting is much much smoother experience…

  36. Alex Delderfield (AD-Edge) Says:

    @B.Tolputt

    “This is something I like to see – information on how Durian is benefiting Blender development.”

    You might want to read the about section, especially the technical targets, take particular note of the last one mentioned:

    Technical targets
    * High detail multi-res modeling (sculpting) and render (micropolygons?)
    * Fire/smoke/volumetrics & explosions
    * Compositing using tiles/regions, so it becomes resolution independent
    * Crowd/massive simulation (fix animation system to allow duplicates)
    * Improve library system for managing complex projects
    * Deliver in 4k digital cinema (depending agreement with sponsor)
    * Make the Blender 2.5x series fully production ready.

  37. Another Fellow Says:

    * Crowd/massive simulation (fix animation system to allow duplicates)

    Oooh, me like this :)

  38. Still Says:

    @ideasman42 Sorry about the info. My system is a Debian AMD64, Turion X2, 1.9MHz, 2Gb RAM, 1Gb swap. I have just tested like that LD_PRELOAD=/usr/lib/libjemalloc.so blender.bin
    I’m going to test it in a complex scene here with this notebook and with my desktop Phenon X3, 2.3MHz, 3.5Gb RAM, 1Gb swap, and I’ll post the results.
    Anyway, there is another way to test it?

    []‘s,

    Still

  39. j. Says:

    For windows systems, i think Doug Lea’s malloc can be pretty good. Don’t know about linux systems tho.

    Link is http://g.oswego.edu/ , is under the “software” category.

  40. j. Says:

    i have nearly forgotten, nedmalloc can also be good for testing…

    http://www.nedprod.com/programs/portable/nedmalloc/

  41. ideasman42 Says:

    @bruno, thanks for the link, looks very usefil. I was aware of some of these tools but didn’t try them yet, bookmarked and will look into this this later.

    @Alex, its tricky, sometimes cutting edge graphics + running on a 3+ year old system are mutually exclusive :(

    @matty, thats the idea, include it works out of the box.

    @K-Linux, nor sure what you mean by parallel computing exactly, having each PC rendering a single frame is simple and efficient.

    @B.Tolputt, yep, ideally blender wouldn’t make so many small allocations, but fixing this isnt trivial like swapping out the allocator. Needs someone with some time and good test cases to research, find solutions and address the bottlenecks. At the moment I don’t have this much free time.

    @Still, no, this seems fine.

    @j, tried this a while ago when testing speed where none of the replacements made much difference, since jemalloc is very popular I didn’t try all the others. If users want to test other malloc replacements and report on how they go, I wont complain :)

  42. FishB8 Says:

    Tried out tcmalloc as well. Less wasted memory than ptmalloc2 but not quite as small as jemalloc. Seems faster speed-wise though. And offers a lot of control at runtime via environment variables.

    I think what makes the noticeable difference for sculpting was that there is probably a lot less swapping out the CPU cache than with the standard ptmalloc2. Just a guess though, no data to back it up.

  43. Frank Says:

    You should really give NedMalloc a try.
    Of course it all depends on your workload, but on the long run, I’ve found that NedMalloc was often a winner over TC and JE, both speed- and memory-wise, on Linux systems.

Leave a Reply

You must be logged in to post a comment.