TBanks and Memory
BlitzMax Forums/BlitzMax Programming/TBanks and Memory
| ||
| I currently use TBanks to store a great deal of information in a memory efficient way - for example, if I have to store data in the range 0-15, I can store it in 4 bits rather than using up a full byte. I have written a module that sets up a bank for each table of data and provides the functions to access and change it. A single record within a table is split up into what I call "groups", for example a competition's list of entrants would be one group containing a number of fields (in this case, the team record number and a seeding value). A group can have many entries. It's obviously not a well normalised database, but memory usage was a concern. Recently though I've been concerned about performance - if a single record needs to be expanded to store more data, such as a competition's list of participating teams, I need to shift all the data following that. I try to avoid this where possible, knowing where memory is likely to be needed and allocating it in advance, but it's far from perfect. Of course, this is old code, and I think it's time to optimise it for performance. I was thinking today that it might be better if every entry of every group of every record had its own bank. Adding an entry to a group would have no effect on any other information - no shifting of data is required, and thus it is faster. I decided to see what memory overheads were involved with regards to creating many banks, so I wrote the following piece of code: SuperStrict Print GCMemAlloced() Local x:TBank[575000] For Local count:Int = 0 To 574999 x[count] = CreateBank(60) PokeByte(x[count], 59, 255) Next Print GCMemAlloced() This created a bank large enough for 575,000 records, each with a size of 60 bytes. It output: 11966 16111990 Roughly 16MB allocated. MemAlloced doesn't seem to return the size of the banks, which would be just over 34MB. Allocating the memory in one single bank, as my current implementation uses, looks like this: SuperStrict Print GCMemAlloced() Local x:TBank = CreateBank(34500000) PokeByte(x, 0, 255) PokeByte(x, 34499999, 255) Print GCMemAlloced() And results in: 11966 12012 Which certainly seems to indicate that the bank memory isn't considered at all. Which means that a 16MB overhead seems amazingly wasteful just to allocate 34MB in 60 byte chunks, compared to the few bytes used to allocate the whole pile. So, questions: What's the reason for GCMemAlloced() not informing you of the memory used by banks? And secondly, anyone got any good ideas for what I'm wanting to do? If I knew of any database modules out there that would allow me to store values in the minimum amount of space possible, I'd surely give them a try, but I'm not aware of any. Of course, I could just be ignorant of them all. :) |
| ||
| Actually, we can scratch most of this. I'd still be interested to know why the bank memory isn't returned in GCMemAlloced(), but the rest is no longer an issue. I spent the evening working out how much more memory I'd need to store everything in whole bytes, as per normal. It came out at 55.5MB. The vast majority (51.5MB) of that comes from a single table, which holds details of all the people in the game and was the reason I was using such a system in the first place - their attributes are over deliberately small ranges to fit more in memory. I'd assumed that it would be a good idea to extend the theory over the entire database, but it doesn't make sense to do so any more. I can still store the person attributes in an encoded form within a standard database, so I'll be doing that. |
| ||
| There is nothing to be collected. All references are fully valid at the point of gc collect. Even if not: There is no guarantee global scope variables get cleaned up when you null-ify them straight away, even less if they are globals |
   |