SaveBank breaking over 1gb?
BlitzMax Forums/BlitzMax Programming/SaveBank breaking over 1gb?
| ||
| Hi, I am using SaveBank to save out a memory bank that is 1gb in size. The file appears to break and be unusable and wont load either back in to BlitzMax or a binary file editor. I thought the filesize limit in 32bit was 2gb, am I wrong? At the moment I am writing the file out manually with lots of writeFloat commands into numerous files but this is slow and takes about 5 minutes to load back in. I am considering using smaller temporary banks and using CopyBank from my main bank to small sub banks and saving those with the SaveBank command, which I think will be faster than my current solution - but it is still less than ideal. Is anyone else having difficulty with SaveBank not working? |
| ||
| I don't use banks (well, for money I do :D) . This is how I save information into a file: [bbcode]' New write stream Local wstream:TStream = WriteFile(filename) ' Number of atoms wstream.WriteInt(leaf_num_atoms) ' Now we write the vertex data Local i:Int For i=1 To leaf_vertices.Length wstream.WriteFloat(leaf_vertices[i-1]) Next ' Next is color data For i=1 To leaf_colors.Length wstream.WriteFloat(leaf_colors[i-1]) Next ' Writing completed. Close stream wstream.Close()[/bbcode] |
| ||
| Hmm, using this code, I can save (and open the file in a hex editor) but it crashes with EXCEPTION_ACCESS_VIOLATION (even in Debug mode) on loading, at around 400 MB in. I wonder if loading the bank directly needs to allocate tons more memory or something? However, I seem to be getting a different result in that I can at least save a 1GB bank... (What is it for, anyway?) |
| ||
| ... aaaand the code: |
| ||
| What file system is on the disk you are trying to load/save on? What version of presumably windows are you using? |
| ||
| Yes, something like this crashes, as James mentions: But you could explicitly allocate memory and utilise ramstreams instead: That will be faster too, but will require more effort on your part to avoid and handle potential errors... Last edited 2012 |
| ||
| I had a quick look at the LoadBank source and this appears to be a bug. It works by temporarily allocating a byte array, doing a LoadByteArray and copying that to a bank. This is a potential problem since you need enough memory for both the array and the bank. But in this case the real problem seems to be the way arrays, and banks, are dynamically resized, multiplying size by 3 and then dividing by 2 in this style: size = size*3/2 This will be calculated as (size*3)/2, with size*3 overflowing to negative when size is a very large positive number. The safe way to accomplish this is size = size + size/2 LoadByteArray does this repeatedly, beginning with a size of 1024, which then gets increased to 1536, 2304 etc. It eventually reaches 994024549, which is a little less than 2^30. At the next resize it overflows to -656446824. Remarkably, I once fell into exactly the same trap. This was all the way back in the 16-bit era so the problem occurred when 3*something reached 2^15. |
| ||
| (What is it for, anyway?) My first passion after I stopped writing computer games was creating landscapes out of fractal algorythms. The 1gb file is the heightmap data of my latest attempt to go bigger and better :) I'm thinking that once I get everything else working I might double the size, but this would bring the filesize to 4gb which I expect to cause issues on older OS', so I am beginning to think that I need to create daughter banks anyway. But you could explicitly allocate memory and utilise ramstreams instead: That sounds like an interesting idea, I've not played with ramstreams but it all looks simple enough. This will be calculated as (size*3)/2, with size*3 overflowing to negative when size is a very large positive number That's interesting, as a rule I always use brackets for ease of decyphering but I assumed BlitzMax used BEDMAS which would imply size * 1.5. I will take a look at the SaveBank code myself, and if it makes sense to my somewhat smaller brain then i'll hack it about to fix the bug. Thank you for your ingenuity in getting to the bottom of this ;). |
| ||
| Ha, I actually imagined it was some sort of landscape data! Well done to Floyd for actually looking into it -- I was knee-deep in Raspberry Pi today and it was messy. Another alternative -- if you're aiming exclusively at Windows, at least for the processing of the data -- is to use Win32 filemapping. I've done this in PureBasic but never tried porting it, and in theory you can read any size of file on a 32-bit system -- it effectively creates a manageable 'window' into an arbitrarily-sized file stream, and it's faster than normal file reading too: Note that .q and PokeQ refer to 'quads', ie. 64-bit numbers, which would be :Long in BlitzMax, and the commands ending with _ (underscore) are the Win32 API commands. I might have a go at porting this tomorrow, if I get the chance, but it's possible I'm missing something that would cause it to fail! |
| ||
| Yeah, this isn't my most complex landscape ( I did one with multiple planets a few months back with LOD right out to space ), but I'm intending / hoping I can use it for a ... Dare I say it after all these years of not finishing or releasing anything, a game. Should I get that far I will definitely attempt a Mac port at some point - if I can find a Mac 3D engine with shaders anyway, but ATM I'm working with Xors3D which is Windows only (and requires almost no re-learning after B3D and MiniB3D ). So I'm in two minds about addressing the OS directly, I think I'd rather fix my LoadBank command - as at least then it's fixed. I'm not likely to get a chance to code for a few days but I'll report back how I got on. |
| ||
| That's interesting, as a rule I always use brackets for ease of decyphering but I assumed BlitzMax used BEDMAS which would imply size * 1.5. BlitzMax has relatively normal precedence rules, so it would for any other numbers, but the point in this case is that halfway through the calculation the initial "size * 3" causes an overflow, so the following division then causes an unexpected result. (Under normal interpretations of BEDMAS or whatever acronym one uses, division and multiplication are of equal precedence, and proceed left-to-right). The reason it's written like that instead of "* 1.5" will be to guarantee the use of integer math. Can't have floats polluting things like memory sizes. |
| ||
| division and multiplication are of equal precedence, and proceed left-to-righ I feel like such a numpty, I think i've been doing that wrong for years! Hopefully I've been saved by my tendency to bracketise everything :) |
| ||
| I tried size + size / 2 and that failed, so I modified LoadByteArray to accept an optional size parameter and sent through the StreamSize from saveBank, and that came through correct in the debuglog but then failed when allocating a size of 1073741824 (1gb). It seems that you simply cannot allocate to an array that size in BlitzMax, I guess the indexing simply runs out of indexes to allocate. So I am going to switch to a system of loading and saving via smaller staging banks. |
| ||
| Well that turned out to be pretty easy, and loads the 1gb data in chunks of 64mb in about 4 seconds which is much faster than sequentially reading each float. (which took about 5 minutes) |
| ||
| It seems that you simply cannot allocate to an array that size in BlitzMax, I guess the indexing simply runs out of indexes to allocate. Note that a 32-bit windows application can only allocate 2GB of RAM total, even if you have more than that... |
| ||
| Note that a 32-bit windows application can only allocate 2GB of RAM total, even if you have more than that... Yeah, I was thinking about this and thought I might do some shuttle loading but that would make it difficult to write a server application - and mean having to portalise connections to numerous server instances, and that makes a complex project even more of a nightmare! So maybe in version 2... :P |
| ||
| Note that a 32-bit windows application can only allocate 2GB of RAM total, even if you have more than that... The process address space can be increased to 3GB but it does require a boot.ini edit, unfortunately. |
| ||
| I've put a lot of thought into it over the last day and have decided I will get most of the functionality right with the single bank, but later on to increase the area of the map I will segment the map and keep only the local area in memory at once, and the server applet will actually be numerous individual instances each serving an area. The client should be able to seamlessly pass between areas without too much trouble, and the required memory usage will be a lot less than it is now. |
   |