Massive Memory Leak in threaded mode
BlitzMax Forums/BlitzMax Programming/Massive Memory Leak in threaded mode
| ||
| I think I may have found a strange bug. When in non threaded mode my program stays at a constant memory, however in threaded mode there is an instant and catastrophic memory leak (we're talking 10s and megabytes a second). I am aware there is a different GC used in threaded mode so are there any issues I should be aware of? Has anyone else had this? The strangest thing is in threaded mode the memallocated function shows a constant amount (ie. doesn't indicate a leak) by windows task manager shows a massive one, and eventually my application crashes. I would post the code, but its part of a massive project and would be quite out of context and hard to follow. Last edited 2011 |
| ||
| Can you make an example to post? I've had many problems with the threaded GC myself but rarely been able to narrow them down to something shareable... |
| ||
| If you have any ideas what type could be causing it, a good start is to make a global count variable in that type that is set to 0. Increase it by 1 in method new, and decrease it by 1 in method delete, and output this value. This will let you see if the type is being collected or not. |
| ||
I know exactly what is causing it, I have narrowed it down to a very specific piece of code:'change over to the new area Method switchArea() currentArea._unloadArea() currentArea = TArea.construct(toId) TPathQueue.updateArea() 'refresh path finder with new area timer = MilliSecs() + delta 'update delta time to current, so logic is executed as normal End Method I am completely stumped because memallocated shows no memory leak. However my program crashing and using huge amounts of RAM begs to differ.. Last edited 2011 |
| ||
| Ok now my program seems to be leaking memory when looping through an empty loop. There is something seriously wrong here. EDIT: Ok I currently think theres something wrong not with my code, but with the GC. Using the EXACT SAME CODE the normal build mode works fine, threaded build causes a memory leak. Should I report this as a bug? Will mark even do anything about this? Is there any alternative? Last edited 2011 |
| ||
| Post it as a bug with example code. The above example calls a number of methods that could be a factor etc. so something portable is needed to highlight the issue. FWIW I don't think you're crazy, this is the kind of gremlin I've been chasing the threaded GC for over a year now... I can never seem to get it into a corner though. |
| ||
| Ok I have gone through all the types and put a counter in them that gets incremented/decremented when objects are created and destroyed. There are no leaks as far as thats concerned. So I thought maybe some aspect of the types need to be nulled for the GC to collect them when objects are destroyed. Thus I went through all the delete methods and nulled every instance variable. Still no dice. I honestly have no idea why this is happening, but I know it happens when the above code is fired. I would post the rest ima747, but its a LOT of code for those methods, isn't that somewhat of a faux par? |
| ||
| I know exactly what is causing it, I have narrowed it down to a very specific piece of code: Can you not see how these two statements are annoying to the helpful?I know it happens when the above code is fired. I would post the rest ima747, but its a LOT of code for those methods. I would guess its not the milli-second bit. |
| ||
| Yeh, sorry, I am just a little hesitant to post a lot of code. I am almost 100% sure the problem happens at this line: currentArea = TArea.construct(toId) I commented out everything else in that section, and just ran a loop that called that function with different ids. It caused a memory leak, and nothing else I tried did. So I think that is it, or at least part of it. Heres that function: 'create a new area for use Function construct:TArea(aid:Int) Local temp:TArea = New TArea temp._loadArea(aid) Return temp EndFunction And here is the method it calls:
'reads through a file with the given id, and loads an area using that data
Method _loadArea(aid:Int)
Local time:Int = MilliSecs()
'set up the general stuff
Local file:TStream = OpenFile("area" + aid + ".txt")
Local sizex:Int = Int(ReadLine(file))
Local sizey:Int = Int(ReadLine(file))
Local tset:Int = Int(ReadLine(file))
tileSet = TImgSet.loadFromNumber(tset)
areaMap = New Short[sizex, sizey]
collisionMap = New Byte[sizex, sizey]
portalList = New TList
entityMap = TAdaptiveEntityMap.construct(sizey)
'load the area tiles
'note they NEED to be square
For Local i:Int = 0 To sizex - 1
For Local j:Int = 0 To sizey - 1
areaMap[i, j] = Short(ReadLine(file))
Next
Next
'load the collision map
For Local i:Int = 0 To sizex - 1
For Local j:Int = 0 To sizey - 1
collisionMap[i, j] = Byte(ReadLine(file))
Next
Next
'now we set up the ais
Local line:String = ReadLine(file)
While Not(line = "endai")
Local n:String = line
Local isid:Int = Int(ReadLine(file))
Local is:TImgSet = TImgSet.loadFromNumber(isid)
Local x:Int = Int(ReadLine(file))
Local y:Int = Int(ReadLine(file))
Local fact:Byte = Byte(ReadLine(file))
addEntity(TAi.construct(n, is, x, y, fact))
line = ReadLine(file)
Wend
'load all the portals
line:String = ReadLine(file)
While Not(line = "endportals")
Local tid:Int = Int(line)
Local tx:Int = Int(ReadLine(file))
Local ty:Int = Int(ReadLine(file))
Local fx:Int = Int(ReadLine(file))
Local fy:Int = Int(ReadLine(file))
portalList.AddLast(TPortal.construct(tid, tx, ty, fx, fy))
line = ReadLine(file)
Wend
CloseFile(file)
'add the player to the collision map
collisionMap[player.xLocation, player.yLocation] = 1
Print "Time taken to load area: " + (MilliSecs() - time)
End Method
I think it might have something to do with the file stream not getting cleared from memory . Last edited 2011 |
| ||
| I'm unable to reproduce a link by writing and reading many files If it can't be reproduced it can't be found, that's the base problem with tracking down the memory issues with the threaded GC... I've got an issue in one of my projects that I can't get out because I can only get the problem to surface when the whole thing runs. When I start trimming it down to any portion of the code the problem vanishes... Last edited 2011 |
| ||
| Thats pretty much what has been happening to me. Whenever I isolate any part of the code it runs as it should, when I run it all together... memory leak. However I thought I had a pinned down. I made a loop that constructed new areas and assigned them to the same variable. However when I go through each line of this function call, and its sub functions... no leak. Its only after the whole function returns that there is a leak, but according to my testing this leak doesn't happen until AFTER the function returns. Last edited 2011 |
| ||
| Its what you call a Heisenbug. It vanishes on close observation! |
| ||
| re: xcessive That would imply the leak is triggered, in part, by the return of the function... a shot in the dark would be that some part of the function return doesn't clear something generated in the function. If there's any way you can make that function portable and maintain the leak it would provide an opportunity to dig into it deeper. |
| ||
| I will certainly give it my best shot. I'll try and cut it back to as little code as possible while still keeping the bug. |
| ||
Odd, I thought I'd give this a shot. Started out with this test program.Global Array:Int[1000,1000] Graphics 640,480 Local Highest:Int = 0 Local MemUsed:Int Local Timer:Int = MilliSecs() + 5000 While Not KeyHit(KEY_ESCAPE) And Not AppTerminate() Cls DrawText Memused,10,10 DrawText Highest,10,40 Flip Array = New Int[1000,1000] Memused = GCMemAlloced() If Memused > Highest And MilliSecs() > Timer Then Highest = Memused Wend In both non threaded and threaded, you see a small memory increase of about 100 bytes per second. The first number printed is the amount allocated that loop, and the second number is the highest allocated. The second number doesn't start showing for 5 seconds to prevent it being set too high as the program grabs memory for its startup procedures. As you can see, each time I make a new array, the memory usage goes up and down, but it goes down a little less each time and up a little more each time. Now I thought I'd try the same with the array in a type Type AType Global Array:Int[1000,1000] Function Init() Array = New Int[1000,1000] End Function End Type Graphics 640,480 Local Highest:Int = 0 Local MemUsed:Int Local Timer:Int = MilliSecs() + 5000 While Not KeyHit(KEY_ESCAPE) And Not AppTerminate() Cls DrawText Memused,10,10 DrawText Highest,10,40 Flip AType.Init() Memused = GCMemAlloced() If Memused > Highest And MilliSecs() > Timer Then Highest = Memused Wend This time the memory usage peaks out after a few seconds. No sign of a memory leak in either non threaded or threaded mode. This was done with BMax 1.44 and Win7 Home Premium 64 bit. Edit: Another odd thing. Just tried the array as a field instead of a global Type AType Field Array:Int[1000,1000] Method Init() Array = New Int[1000,1000] End Method End Type Graphics 640,480 Local Highest:Int = 0 Local MemUsed:Int Local Timer:Int = MilliSecs() + 5000 Local MyType:AType = New AType While Not KeyHit(KEY_ESCAPE) And Not AppTerminate() Cls DrawText Memused,10,10 DrawText Highest,10,40 Flip MyType.Init() Memused = GCMemAlloced() If Memused > Highest And MilliSecs() > Timer Then Highest = Memused Wend Now the result is the same as it was for the Global Array outside of a type in threaded mode (i.e. small 100 byte/sec leak), but no sign of a leak in non threaded mode. Last edited 2011 |
| ||
| You should reset the Timer. Anyway, ran the last one, and the mem goes up for a while, peeks at 8049126 and for threaded peeks at 12066848. (or there abouts each time) NB) The current mem used in nonthread is the same as the peek when the peek is reached, and stays there The Current Mem used in Threaded is alternating by a factor of ten between two numbers at atleast 20hrtz. This second part is worrying to me. However Im using Bmax 1.41 |
| ||
| Hmm.... All tests... NonThreaded - Mem goes up only when the mouse is moved, otherwise it doesnt change, peeks out approx 8049126. Threaded - Mem continuously changing, same with mouse, peeks out about 12065728. BMax 1.44 Vista 32bit. |
| ||
I modded your code a little:Type AType Field Array:Int[1000, 1000] Method Init() Array = New Int[1000, 1000] End Method Method New() Self.Init() End Method End Type Graphics 640,480 Local Highest:Int = 0 Local MemUsed:Int Local Timer:Int = MilliSecs() + 5000 Local MyType:AType = New AType While Not KeyHit(KEY_ESCAPE) And Not AppTerminate() Cls DrawText Memused,10,10 DrawText Highest,10,40 Flip MyType = New AType GCCollect() Memused = GCMemAlloced() If MemUsed > Highest And MilliSecs() > Timer Then Highest = MemUsed Timer:+MilliSecs() + 5000 EndIf Wend Not getting a memory leak, just really erratic memory usage in threaded mode. I will try and cut back my code to something small enough to post that still has a memory leak. Notice I added a gccollect() call. It seems your slow leak was just the GC not being able to keep up with memory being allocated. More in than out in other words - I think so at least. Last edited 2011 |
| ||
| You may or may not be doing this several times... A little problem I've just had was with initializing arrays. The array would be a null array like Global Array#[]. Then elsewhere I'd flesh out the array with something like Array = Array[..newSize]. A couple of lines later I fill that array with data. The array size is accurate and correct to the size that I need at the time ( very important for what I'm doing ). On the next loop around if I need to change the size of that array then the same code runs again using Array = Array[..newSize]. Note this works as expected when it makes the array larger, it won't make an existing array smaller, is this correct???. This would cause an intermittent EAV. Even using Array = Null before the Array = Array[..newSize] wouldn't stop the erratic EAVs. To fix it I have invoke the garbage collector after nulling the array and everything works perfect again. This is in threaded and non-threaded modes. Like I say it may not be the problem, and I could be throwing a spanner in the works of where you're heading, but its something I've experienced in the passed couple of days and kind of relevant in that your initializing a large-ish array every frame. EDIT :- The data would be put into the array without any problems, confirmed in the debugger, the EAV would only occur later when accessing the array, which is accessed in full. In other words the garbage collector couldn't keep up! @excessive I just tried your code. The memory usage is rock steady in your example in threaded and non-threaded, whereas in the earlier examples, the threaded builds have erratic ( but contained ) memory usage. Last edited 2011 Last edited 2011 |
| ||
| I think you all may be on to something regarding the GC not being able to keep up... if I recall from the last time I plumbed it's depths it had a recursion limit, in order to keep it from hanging or causing stalls... I'm guessing some structures can cause the recursion to get too deep and it can't clear them after that point... |
| ||
| Note this works as expected when it makes the array larger, it won't make an existing array smaller, is this correct???. My instinct is that it would allocate a new, smaller array, copy the data and discard the old array. It would have to do this if you spliced the array and stored the result in another variable.Even if you don't allocate any arrays or types, for the first few seconds the memory usage increases when moving the mouse, so bear this in mind. I imagine that things such as draw text, flip, will use memory for their function which will vary. This did not leak memory for me in either mode. It peaked at a certain value after a few seconds. |
| ||
| I want to clear something up here... Note this works as expected when it makes the array larger, it won't make an existing array smaller, is this correct???. I stand corrected here! At the time I was on that part of code, I was relying on the debugger to show me that the array had resized. STUPID ME!! I should have known better. Sorry for the confusion as the array is indeed resized correctly as expected. How silly do I feel now :D EDIT :- Anyway, the main point is it will still rely on the GC to clean up after itself, which can bomb out, as in my code, I get an intermittent EAV if I change the array size everyframe without using GCCollect(). Putting it in solves the EAV problem. Last edited 2011 |
| ||
| Could you post/clarify which code, gives the EAV? I didn't get any. How are you resizing the array? |
| ||
| Yeah sure.... There's obviously more to those functions but that snippets shows exactly what I'm doing with the array that was causing me problems. The EAV only happens when the size of the incoming array changes EVERY frame and there is no GCCollect(). Don't forget this is intermittent too, even when passing in the exact same dataset that changes each frame. Sometimes EAV, sometimes not, in threaded and non-threaded. EDIT:- Just to clarify the SAFE_RELEASE() function in there :- Function SAFE_RELEASE(interface:IUnknown Var) If interface interface.Release_ interface = Null EndFunction Last edited 2011 |
| ||
| Where does _pointarray come from? Can you give some code which I can compile? What line does the EAV come from? If _pointarray is global, be careful of this Global array:Int[30] For Local i = 0 Until 30 array[i] = i Next Local a1:Int[] = array array = array[..10] a1 and array both point to two different arrays, of sizes 30 and 10 respectively. |
| ||
| Everyone can have the whole project soon as its a D3D11Max2DDriver, then you can try it for yourself. I see and understand how your last example could cause issues. But I'm not doing that, 'xcessive' may be in his original source files? The _pointArray never directly 'points' to the inArray[]. My code merely looks at the size of inArray and adjusts the size of the _pointArray accordingly to make the _pointArray proportionally larger than the inArray. Then the data is copied across from the inArray to _pointArray using standard _pointArray[index] = inArray[index]. When the filling of the data is complete then its passed to Dx11 to upload to the GPU as a vertex buffer, hence fleshing out the _pointArray with additional information. The EAV only occurs during the DirectX call, not from any calls inside BMax, although I've NOT tested this fact!! At the time I was just interested in getting my code working rather than dig too deep into the GCCollect() command. I've verified the data and size of the data in the array going to DirectX is exact so I know it's not that. But like I show, using GCCollect() to collect any old scrap data clears up the issue immediately. Again, this only happens when that process to resize the array happens every frame. It's as if the GC gets overloaded. I just put it down to the Garbage Collector simply couldn't keep up and never thought any more of it until I saw this post. Pseudo code... This is pretty much the whole guts of that function that potentially has trouble. There is a size validity check that just makes sure the inArray has at least 2 elements coming in and also that the number of elements even, but thats certainly not the issue. @xcessive Apologies to you as I feel I've kind of hi-jacked your thread a bit. How are you getting on with using GCCollect() in your main source?? Does it stop or help your issue?? Last edited 2011 |
| ||
| None of this is relevant that my problem, I am not using array slices at all. Calling gccollect seems to make no difference to me at all. Whenever I change anything the leak seems to disappear! So its really hard to narrow it down to anything, this getting massively frustrating.. |
   |