Loading time
BlitzMax Forums/BlitzMax Programming/Loading time
| ||
| It takes like 2 minutes to load the world for my RPG, every time I run the program. The map is 300x300 tiles, and each tile has 1 to 5 layers. It takes almost a minute to create the tiles and a minute to load them. But the collisionsetup function finishes instantly. I can't work like this, its too slow. Does anyone have any tips on how to speed this up? There are no entrances or guys on this map but there will be. The map file is over 5 MB. |
| ||
| There could be several things causing the slowdown, I suggest profiling your code to find out for certain where the speed problem is. By profiling, I mean use the MilliSecs() function to find out how long certain sections of code are taking to execute. Some other info needed: - What format are the images saved in - How large is each tile (in pixels) Something which usually helps in any project - insert a call to FlushMem after creating / loading each tile. |
| ||
| Well first I would suugest not making your tiles Types. Use plain arrays. Second, I just tried this with my own map editor, organize the map file into large common chunks. So for each layer put all of the data together in one chunck then you can do something like this: For Local y:Int=0 Until mapheight Local bptr:Byte Ptr = Varptr newmap.layers[l].tiles[0,y] mfs.ReadBytes( bptr,mapwidth*4 ) Next I found that the reading of each tile value one at a time with ReadInt is extreemly slow. With the above method you are rading huge chunks of data very quickly. In my test I was able to to read a 3 layer map that was 366 megabytes in 2 seconds. Before I changed to doing the above it was taking almost a minute. Note that you have to deal with the fact that the blitz code is no longer deal with big/little endedness in this situation. EDIT: looks like you can even handle the endedness with by using a BankStream. Read the data into a bank and then create a BankStream on that bank. |
| ||
| well I'm trying this full on and not getting good results. I'll let you know if I find the problem. Right now it looks like using Banks and BankStreams is just as slow as normal file streams. .... EDIT Okay I can now load a 5000x5000 map with 3 full layers in the above way in about 12 seconds. This is not using the endian streams. |
| ||
| ok, I'm not at home at the moment so can't do any testing of the code, but I've noticed a lack of flushmem in your for loops (especially ones that are instancing types each iteration). This may well be one of the causes. |
| ||
| Thanks for quick replies OMG I thought yall were crazy I just stuck flushmem in the loop and it loads in like 2 seconds! Could someone tell me what exactly is being flushed that makes this take 60 seconds... FlushMem For fory=1 To sizey For forx=1 To sizex tile=New ttile array[forx,fory]=tile tile.x=forx tile.y=fory tile.guylist=CreateList() Next Next But this takes 2 seconds For fory=1 To sizey FlushMem For forx=1 To sizex tile=New ttile array[forx,fory]=tile tile.x=forx tile.y=fory tile.guylist=CreateList() Next Next Some other info needed: - What format are the images saved in - How large is each tile (in pixels) Its not loading any images. Just makeing types and loading their data for a file. I could have mentioned that, but, nah Well first I would suugest not making your tiles Types. Use plain arrays. Second, I just tried this with my own map editor, organize the map file into large common chunks. So for each layer put all of the data together in one chunck then you can do something like this: Thats interesting Ive never used the readbytes command, but I think I could use it. What does endian mean by the way? I am also loading thousands of x and y handles for character images, 1 by 1, with readint() and I want it sped up, because it takes like 10 seconds. with flushmem ;) What if I just loaded the whole file as a chunk in a 1D array? |
| ||
| Endian refers to the why in which the particular hardware platform stores multibyte values. Intel chips are usually little indian and motorola chips are usually big endian. 0xABCD writting to a file on Windows would be DCBA and on mac ABCD I think I got that right. If you want binary files to be cross platform for your program make sure you write them and read them with the same endian stream types. |
| ||
| I think the flushmem is deleting your unused lists created during each for/next (90,000 of them?) Silly test...
start_time=MilliSecs()
For x = 1 To 300
FlushMem()
For y = 1 To 300
mylist:TList=CreateList()
Next
Next
end_time=MilliSecs()
Print end_time-start_time
and take out the flushmem to see the difference. |
| ||
| well all my lists have a different varaible pointing to them. this is what I am really doing. But ya, flushmem makes it go way faster even though it is not deleteing any lists here. It must be CreateList() that is leaveing all the garbage!
Local listarray:TList[300+1,300+1]
start_time=MilliSecs()
For x = 1 To 300
FlushMem()
For y = 1 To 300
listarray[x,y]=CreateList()
Next
Next
end_time=MilliSecs()
Print end_time-start_time
Scott Shaver: thanks for the help, how did you find out about the readbytes command? |
| ||
| Isn't learning about your new fa. language fun?! Glad that seems to have cured it. Note to self - Flush! Fluch and thrice Flush IPete2. |
| ||
| Scott Shaver: thanks for the help, how did you find out about the readbytes command? Digging through the module code to figure out why it is so slow. :) |
| ||
| There's a thread where the suggestion was a single flushmem should be OK *unless* a function or for/next was doing a lot of work. Maybe this is that situation... possibly? |
| ||
Okay so here is what I ended up doing to speed up the loading of huge maps in my editor:' this is a speed optimization, istead of reading each int one at a time ' we read them in in blocks, much faster but we have to deal with the endian stuff ' ourselves Local temp:Int[] = New Int[mcw] Local bptr:Byte Ptr = Varptr temp[0] ' cell tile index data For Local y:Int=0 Until mch FlushMem mfs.ReadBytes( bptr,mcw*4 ) For Local x:Int=0 Until mcw ?BigEndian Local t ' only do the byte swapping if on a big endian system, file is little endian t=bptr[(x*4)];bptr[(x*4)]=bptr[(x*4)+3];bptr[(x*4)+3]=t t=bptr[1];bptr[(x*4)+1]=bptr[(x*4)+2];bptr[(x*4)+2]=t ? newmap.layers[l].tiles[x,y]=temp[x] Next Next |
| ||
| Arrg, Having readable text for data files is so much more convenient. Can you give us a rundown of how much faster your system was with just a flushmem in the loop. How much better did it get by going to a binary format? |
| ||
| Actually with regards to the endian thing, I believe 0xABCD is stored on Intel as CDAB - remember, each $xx is 1 byte. |
| ||
| Isn't learning about your new fav. language fun?! haha, I dont know what the hell is going on anymore/ |
| ||
| Could someone tell me what exactly is being flushed that makes this take 60 seconds... I would hazzard a guess that the lower memory footprint causes less allocations/swaps/cache misses (take a pick). |
| ||
| http://www.blitzwiki.org/index.php/FlushMem |
| ||
| Arrg, Having readable text for data files is so much more convenient. Can you give us a rundown of how much faster your system was with just a flushmem in the loop. How much better did it get by going to a binary format? Not sure if this was addressed to me but my map format has always been binary. Loading each int with readint was taking around 1 minute 30 seconds. Using a bank to load each horizontal row took about 1 minute. Doing what I did in the code above took it down to about 15 seconds. Understand that I'm test this with a outrageously large map. 5000x5000 with 3 layers so I'm loading 75,000,000 integers (the tile index for each cell) and 75,000,000 bytes (the alpha values for each cell). The total map file size on disk is 366 meg. Adding the flushmem in my case has no effect what so ever. But I'm not creating any objects in my code for the map cells, I use an object that holds straight arrays. |
| ||
| ReadBank() is the new ReadBytes(). |
   |