Garbage Collector and GCMemAlloced
BlitzMax Forums/BlitzMax Programming/Garbage Collector and GCMemAlloced
| ||
| Hi! I want to code a memory manager for my current app. To see what was happening with the gc in an "empty" app this is what I wrote: Now... Looking at the GCMemAlloced, it's always changing! The program is doing nothing yet it's always changing. I know it is not an error, I just want to know why is that happening? Thanks! |
| ||
| Because even that is adding stuff to the call stack etc. Plus a load of other behind-the-scenes crap I don't understand. It's normal. |
| ||
| Hi GFK, Thanks for the post. I know it's a normal behavior but I just was curious about what was going on behind the scenes...it's a lot of movement! |
| ||
| Hiya Put a DebugStop before the DrawText command in Debug mode then step into all the DrawText functional code. Don't alter anything though!! There's plenty of background code to look at just in that one command. Before slating it - remember BMax is not built to be the fastest, but built to be respectably fast whilst being multi-platform. |
| ||
| There's plenty of background code to look at just in that one command. Before slating it - remember BMax is not built to be the fastest, but built to be respectably fast whilst being multi-platform. That's true col. I'm overall happy with bmax. I just finished a long game-like project that lengthened to 3 years and a half, and just until now I had no serious problems. I'm trying to find out why that app is 'randomly' crashing with an exception access violation. Brucey and others suggested me to use an external debugger (such as GDB) but I believe I will take more time to just learn how to use GDB that to actually 'reproduce' the bug in a controlled environment. Anyway, back to topic... I believe that learning a little bit more about the GC could help me solve that... |
| ||
| I read your thread about that thinking to myself you must be pulling your hair out of over that issue!! I recently wrote some code that accesses some dlls, ( I was messing with some direct usb access libraries ) and it actually ran faster in debugmode! Turns out an invalid return address just so happened to be a valid value giving me false impressions! So I know how you feel when things crash for no obvious reason! Marks just updated BMax to 1.45 with some fixes to the GC. Maybe it'll help your issue? Last edited 2011 |
| ||
| I read your thread about that thinking to myself you must be pulling your hair out of over that issue!! I am! And I'm using the newest version of BMAX too but sadly, I'm dragging this problem back from V1.42 or something... Right now I'm building a memory manager for my app, and I will incorporate a resource manager too! My problem is when loading/writing stuff. I hope that once I've got all covered, the problem can get isolated... since I have so many modules involved.. it's hard to find the offending one :S Last edited 2011 |
| ||
| I've had plenty of the ole 'intermittent, for no reason, works for 30secs then craps out' bugs in the past :P From my experience with BMax - Exception Access Violations are generally associated with a resource thats a value of zero or Null, and your code or a module is expecting it to be an actual valid value. Your code or the module code ends up accessing a memory address of zero which then causes the OS to fire up the violation - ie your exe is accessing memory not assigned to it by the OS. Something as simple as a duplicate variable name is surprisingly popular at causing this! Bruceys modules are of a high standard, but if you've completely eliminated any errors in your own code then you've got to look at the modules you're using. I'd first validate the values being passed into the mod functions and also validate them on return. Use actual values checking for null and/or zero and also check within an expected value range instead of True or False, anything outside the range and stop the code straight into the debug window. You can NEVER do too many validation checks when bug hunting :D I love bug hunting... got kind of a 'thing' going for it lately :D If you wanted any help, mail me any complete code that produces the error, so I can do a build, and I'll take a look? EDIT:- BTW is the EAV in debug and/or release build? Good luck with it. I'm sure it'll be found soon. Last edited 2011 |
| ||
| Thanks col! I know Brucey's modules are VERY reliable. Last program that "randomly but rarely" crashed with an EAV was a little one that made sprite sheets. My app is divided in 40+ modules because compiling time was slow! and I always build modules with bmk makemods modulename before compiling the final exe BTW, One of the 'lesser' problems of my app is a memory leaking. Believing that the GC was a reason not to worry about managing memory resources was a big mistake, and I also expect to find my answers when dealing with the leak.. Anyway... back to topic I'm seriously considering your help here...but before throwing the towel I'll give a try with the memory manager and resource manager I'm writing, and probably re-write the offending module (because out of the 40, it crashes only while being in it) I'm currently making an application to heavily "test" all the task I normally do in mi app (LoadImages, LoadAnimImages, from streams, from zipstreams, read/write to files, databases etc) in a controlled way. When I finish it, I'll let it run all day long until it crashes (: ... I hope it does EDIT: Ok! little app testing program is taking me somewere! :D The EXCEPTION ACCESS VIOLATION is being thrown by the pixmap module, in this line of code: Case PF_BGRA8888 Return p[2] Shl 16 | p[1] Shl 8 | p[0] | p[3] Shl 24 Now... I really don't know if this is exactly the same EAV I get in my app, but what I know is that this shouldn't be happening... For the imports, I'm using this: SuperStrict Framework brl.max2d Import bah.freeimage Import brl.d3d7max2d Import brl.d3d9max2d Import brl.map Brucey's freeimage is the only one there... I'll have to write an alternate version of the pixmap test w/o the freeimage module and see what happens... Last edited 2011 Last edited 2011 |
| ||
| Ok! Confirmed! the EAV I'm getting comes from using bah.freeimage module! Guess I can leave the GC alone for a while (: This is the link to the test I wrote including the images I use to make the tests: http://www.smartb.mx/freeimage_test.zip It comes with both versions of the test, free image version, and no freeimage version... can you help me confirm the problem, col? I'm using Bmax 1.45, mingw (GCC) 4.6.1 and bah.freeimage 1.07 P.S. Perhaphs I should make another post? This has gone offtopic (non GC Related) But I'm sooo glad I found it! :D |
| ||
| You've got plenty of management going on in there :P I like it even though it is 'over the top' :D It's not as random as you think..... in main.bmx, comment out line 54 and line 56 so that Setup is run immediately and it'll crash out at exactly the same time, every time. This hinted to me that it is in fact a GC issue. Also at the 'crash point', the GC figures are up to the point that it would normally ( in this example ) be collected. I've found a 'work around' fix for you that works in this example code. Try it with the real project :- In freeimage_pixmap_test.bmx, in the GenerateImageStrip(...) function, squeeze a GCCollect() at Line 104 at the start of the function code. Let us know how it pans out in the main project. This would be a great example to post in the Bug Forum ;-) EDIT:- There definitely seems to be a problem with the GC when the code is large with a lot of 'still valid and active variables and data structures', the GC kicks automatically doing some collecting and something gets screwed up. Keeping the GC updated and low helps the problem. Last edited 2011 |
| ||
| Hey! Thanks for testing! The management I'm doing uses 2 clases of my main project (IGF ones) ... They are template classes for state machines... but i guess I found the bug in the 2nd test! (first one was raw image loading) So, it was a GC issue after all huh? How ugly this GC bugs can be! In my app it looks like random because the operations involving pixmaps are not made at the same moment and depend totally on user input. I will report the bug, but the fact that the non freeimage version of the test runs just fine without some GC trickery its good enough for me (: ... I can afford to remove bah.freeimage completely from my project. Besides, in my project I'm not doing exactly that kind of pixel manipulation, but it always crashed when loading images (pixmap?) or stuff. EDIT:- There definitely seems to be a problem with the GC when the code is large with a lot of 'still valid and active variables and data structures', the GC kicks automatically doing some collecting and something gets screwed up. Keeping the GC updated and low helps the problem. lots still valid and active variables is the case of my app lol... So you did made another test code to find that out? ... I guess it is time to report this in the bugs forums... in the meantime I'll remove bah.freeimage from my app until this gets solved. Thank you for you help, Dave! Last edited 2011 Last edited 2011 |
| ||
| I've made and used some BMax sources that access a lot of external code. By external code I mean code that's Imported that doesn't use the BMax language itself:- c/c++/dll/lib etc. When using a fair of amount of data and the code is of a mixed source ( Bmax/c/dll etc ), that's when I notice the GC can sometimes bug out, and it needs to be collected manually otherwise crashes occur with still valid data. As seen in this example. I'm sure it's not anything to do with anyones source code, or your own, but it's the actual GC itself! It just so happens in your case it bombs out using code that accesses a module. I've had the same problem with passing arrays into dll functions. It works ok until the GC auto-collects and screws something up, then in the dll call I had an EAV. In the debug window everything was still showing as alive and active. Real pain in the arse. 'GCCollect'ing before using iterative code that accesses 'external' code fixes it up nicely. lots still valid and active variables is the case of my app Yep, and tons of other large 'practical' and 'real' apps too :P |
| ||
| I'm sure it's not anything to do with anyone's source code, or your own, but it's the actual GC itself! I use a lot of iterative code that accesses 'external' code, putting GCCollect all over it will take some time, but oh well... How do we report this bug? I believe my example is too large and depends on some image files... Maybe I'll do a stripped version of the tester app that doesn't require too much to show the GC bug in action... |
| ||
| To be honest I'd link the the .zip thats used above so that BRL have code thats guaranteed to crash, and also link to this topic page too. It a great example of putting the GC to a real life test. It's not like they will have to wade through your code as we know how to reproduce the crash at the same time, every time. This should make it easier for BRL to dissect whats going on with garbage management. |
| ||
| To be honest I'd link the the .zip thats used above so that BRL have code thats guaranteed to crash, and also link to this topic page too. Alright, then I'll take your advice and just put that in the bugs forum. Thank you for your help Dave! Last edited 2011 |
| ||
| try this modified "freeimage_pixmap_test.bmx" file: with this file freeimage version works. TFreeImage.GetPixmap() returns a static pixmap, and you save that to TImageContainer. And when GC release unneeded "FreeImages"(locals) then those "static pixmap" fields have pixmap that is no longer in memory. Oh, even easier is to modify this line in function GenerateImageStrip: Local imageContainer:TImageContainer = TImageContainer.Create(url + t, pixmap.Copy()) ^^ and that works too. Last edited 2011 |
| ||
| If I understand GC management correctly?... Adding the TPixmap instance to the TImageContainer should increase its internal reference count and not cause the TPixmap memory to be collected? no? or is the only reference the Local variable? and theres no trace in the GC of storing it the TImageContainer then storing that in the TList? Also how come clearing the GC at the beginning works ok? I'm not querying your fix, just trying to gain more knowledge of the BMax garbage collector. cheers. Last edited 2011 |
| ||
| That's a nice workaround Zeke! But still, the problem in my app is not one that involves image manipulation. I get an EAV error for other unknown reason, and for that I'll have to find another particular workaround ... that's why I believe the real problem lies in what Dave said before about GC collecting memory when it shouldn't, in cases where external code is heavily used. Dave's Dll anecdotal example of EAV goes in the same line too... Of course, until GC is fixed, I'll have to make my own workarounds... Either keep GC garbage low OR shutting down GC while loading/writing stuff, and resume when finish? |
| ||
| the real problem lies in what Dave said before about GC collecting memory when it shouldn't, in cases where external code is heavily used. yes. external code is c++ code. and if you check brl.mod/blitz.mod/blitz_gc.h file: // BBRETAIN/BBRELEASE should be used to prevent an object from garbage collection. // // This is mainly of use if an object is being stored outside of BlitzMax's 'sight' - say, in a C++ table. // // You can also use bbGCRetain/bbGCRelease functions above if necessary - MACROS are just faster. so, those "c++ heavy"(like freeimage) modules should use bbretain/bbrelease functions. |
| ||
| Is the problem any different in threaded mode, assuming you're in non-threaded mode? They use different garbage collection methods, so it might be worth a shot: [blitz.h] //Which GC to use... #ifdef THREADED # define BB_GC_MS #else # define BB_GC_RC #endif |
| ||
| For me, One of my most recent experiences of overloading the GC was using a simple array, an array that could change size and its data every frame, when passed into external code ( a proven MS DLL ), sometimes I'd get an EAV, sometimes not. Nulling and GCCollecting the 'old' array before reassigning the new data to it prevented the EAV ( these were Local arrays ). Almost like there was some kind of confusion in which array referrences were valid and which ones needed collecting. Someone recently ( sorry I cant remember your name!! ) did explain that there is a recursion depth in the GC that 'could' get overloaded, causing similar issues. You know the old analogy - 'If something 'can' happen, then it probably will' so if that is the case then I'm surprised there isn't some kind of overload protection. Oh, all in single thread mode, debug and release. Just in debug mode it would also stop at at the line of code that was accessing the 'not supposed to null' data, still with an EAV. I'm personally along the lines of the GC getting overloaded? In Rixarns example above, GCCollecting early seems to make plenty of room for the newer Locals to be kept alive and valid for the next steps? The code is using TPixmap references that are becoming invalid at some point. I understand pulling the TPixmap from another object probably isnt the best thing to do, but I'd expect the TPixmap reference count to increase and internally not allow any parent objects that have a reference to be released too until said TPixmap is completely unreferenced. Am I expecting too much with that school of thought? cheers |
| ||
| For me it's also single threaded mode, debug and release. so, those "c++ heavy"(like freeimage) modules should use bbretain/bbrelease functions. Then it means I have to hack bah.freeimage and add those bbretain/bbrelease to the code in order to safely use it? Haven't checked if bah.freeimage already has them... But if not, it will be a PITA ): As for what to expect from the GC, I rather prefer it would always crash at the same point, or never... but not "randomly". If we as programmers are doing something we aren't supposed to do then make it crash always to remove ambiguity. If not, then it's something worth looking at, as you said James (: Last edited 2011 |
| ||
| no need to play with gc... just modify freeimage.bmx freeimage.bmx->Type TFreeImage->Method Init, line 726: ' build a pixmap... based on the "displayable" bitmap pixmap = CreateStaticPixmap(bmx_freeimage_getImage(displayImagePtr), .. bmx_freeimage_GetWidth(displayImagePtr), .. bmx_freeimage_GetHeight(displayImagePtr), .. bmx_freeimage_GetPitch(displayImagePtr), format) change to this: ' build a pixmap... based on the "displayable" bitmap pixmap = CreateStaticPixmap(bmx_freeimage_getImage(displayImagePtr), .. bmx_freeimage_GetWidth(displayImagePtr), .. bmx_freeimage_GetHeight(displayImagePtr), .. bmx_freeimage_GetPitch(displayImagePtr), format).Copy() [EDIT] hmm.. but then "Freeimage" image will use 2 x memory.. because pixmap isnt static anymore.. damn.. maybe to ask brucey if he know how this can be solved. Last edited 2011 |
| ||
| Hi Zeke, Thanks for research. Although it is a memory expense solution, I think it is a viable solution if someone is urged to use bah.freeimage while things get fixed. Better to work at 2x the amount of memory that work with flaws. I'm my case, I can live w/o that module for the moment. In fact I already removed all the references of bah.freeimage. Biggest problem of mine is still the GC, who releases things when it shouldn't, and doesn't release things when it should (I'm currently trying to Isolate this second case into another code example)... I really hope GC gets revised by BRL. |
| ||
| OK, here I am with yet another example ... but this time it's not a bug because of the "type" of the Garbage Collector bmax uses. From the docs example this is a cyclic class. I can understand why you have to null the variable to make the reference count reach to 0, or else the memory will always leak. Type TCyclic Field cycle:TCyclic=Self Method Remove() cycle=Null End Method End Type For k=1 To 10 Local cyclic:TCyclic=New TCyclic GCCollect Print GCMemAlloced() Next Now, what I have here is a more sophisticated way of cycling references that went unnoticed and was causing all the leaking in my App! In my example, it is a member object of the base class that has a reference to the base class! When I null the obj variable after the keydown, I expected the object in memory is regarded as an "isolated island of references", and thus collected! Java (correct me if wrong) works that way. Cyclic dependencies are not counted as reference so if Object A has reference of object B and object B has reference of Object A and they don't have any other live reference then both Objects A and B will be eligible for Garbage collection. My big serious mistake was to assume Bmax GC worked that way... Geez, I'll have to do a LOT of manual var = null stuff all over my app. BMax GC aint that automatic as I believed... I had red before some post of JoshK talking about this but I never understood what he was saying until now... Last edited 2012 |
   |