Streaming engine, progressive loading of data
Blitz3D Forums/Blitz3D Programming/Streaming engine, progressive loading of data
| ||
| Hi, :) I am trying to create a big map similar to the one in "Daggerfall" but only 10kmx10km with several 1024x1024pixels heightmaps. What i have coded so far works well, the Player can walk on the terrain and when he reaches the end of the active zone, the program unloads the terrain, the buildings, the trees and loads the new terrain, the new buildings, the new trees. However this unloading/loading is done in the same loop, and as you can imagine, the last frame rendered to the screen is displayed until the current loop ends. So what i am looking for is a way to unload/load data during several loops so that the Player won't notice it. If some of of you have done something similar, can you please give me your advices about this : To determine if the program can continue to load meshes, textures, sounds, i need to know how much time my loop has used (in ms). I also need to know how many ms a loop can use. (maximum) Because pcs have different setups and hardware, i guess i have to determine the number of ms i allow the loop to use with a low end pc ? (So that the game will run perfectly on better pcs...) Also, the number of ms i allow the loop to use is influenced by the number of frames per second i want to have in the game. So if i choose to have 15fps, then i can increase the time used in each loop ? I hope my english is understandable. Thanks, |
| ||
| This is actually a pretty difficult thing to do well... so good luck! The first thing is a general design principle: high-end PCs shouldn't be affected by intentional slowdown put in place to cater to low-end PCs. Instead, you should have a counter in your loop that measures how long various things take, and uses that to dynamically adjust how much the game attempts to do each frame based on how much it was able to do last frame. This will let every computer operate to the best of its own ability (similar logic to that used by render tweening, if you're familiar with that). The second thing, though, is that unless your assets are organised into chunks of known size and complexity, you can't really use the load time of the previous mesh to gauge how long the next one will take. So you need to divide your world into smaller chunks than the engine actually needs, and have them be almost equal in size, so that it can accurately predict how long the next section will take to load (the other thing is that if the sections are smaller, it can break up the task of loading an area over several frames). Two other points of lesser importance: Firstly, the main idea behind background streaming is that it's not perceptible to the player. So, rather than loading all the necessary assets at once when the player reaches the edge of a cell, you should detect in advance which cells they're likely to enter and start loading them in the "spare" CPU time at the end of each frame before they actually get there. Hopefully, by the time the player reaches the new cell, it will be fully loaded and waiting. Games like the Elder Scrolls usually achieve this by loading a grid of nine cells, with the player in the middle one; when the player changes cell, that's the cue to discard the three that are now furthest away, and load a new "edge" of three, so that the player is never at any point exposed to any unloaded cells. Secondly, you might want to have a buffer of three or four discarded-but-not-unloaded cells waiting in the background. Since the player might be zipping across the border between two several times (if that happens to be where they want to play), it's better if that doesn't result in repeatedly having to load and unload the same set of cells. Instead, if they're waiting in the buffer, they can be quickly retrieved and re-added to the scene. Cells that the player never re-enters will eventually fall back to the end of the queue, and then become unloaded for real if the player never returns. ...hope that helps. Last edited 2011 |
| ||
| Thanks for your advices Yasha :) Here are some comments : The first thing is a general design principle: high-end PCs shouldn't be affected by intentional slowdown put in place to cater to low-end PCs. Instead, you should have a counter in your loop that measures how long various things take, and uses that to dynamically adjust how much the game attempts to do each frame based on how much it was able to do last frame. This will let every computer operate to the best of its own ability (similar logic to that used by render tweening, if you're familiar with that). I understand what you mean but i don't know how to do that. If i understand what you meant, the program starts the game with the low end loop time limit and then if the program notices the fps count is higher than 15fps, then the loop time limit is increased ? The second thing, though, is that unless your assets are organised into chunks of known size and complexity, you can't really use the load time of the previous mesh to gauge how long the next one will take. So you need to divide your world into smaller chunks than the engine actually needs, and have them be almost equal in size, so that it can accurately predict how long the next section will take to load (the other thing is that if the sections are smaller, it can break up the task of loading an area over several frames). I have planned to load first the new terrain, then the new buildings, then the new trees, then the new plants, then the new furnitures, then the new npcs, then the new objects. And because my project will have low quality meshes and textures, and many meshes and textures will be copied rather than unloaded/loaded, from my tests i think it should work. The thing that bother me the most is how to create a new blitz3d terrain from a texture i have progressivly loaded. I have to think about that. Firstly, the main idea behind background streaming is that it's not perceptible to the player. So, rather than loading all the necessary assets at once when the player reaches the edge of a cell, you should detect in advance which cells they're likely to enter and start loading them in the "spare" CPU time at the end of each frame before they actually get there. Hopefully, by the time the player reaches the new cell, it will be fully loaded and waiting. Yes i have planned to start the progressive loading depending on the Player coordinates. That should work. Secondly, you might want to have a buffer of three or four discarded-but-not-unloaded cells waiting in the background. Since the player might be zipping across the border between two several times (if that happens to be where they want to play), it's better if that doesn't result in repeatedly having to load and unload the same set of cells. Instead, if they're waiting in the buffer, they can be quickly retrieved and re-added to the scene. Cells that the player never re-enters will eventually fall back to the end of the queue, and then become unloaded for real if the player never returns. Ok for example the cells which are not the current cell but which have a common seam with the current cell. I don't know if i'm going to try to implement this for the beginning but that's a good idea. Well i think i have grasped the concept, thanks. Last edited 2011 |
| ||
| you'd have to make a queuing system. i remember i think morrowind did something similar would start loading and unloading when you got close enough tot he edge. i guess you'd make a queue type and then have it only unload a few or one or two of the objects in the queue and then also load a few things. |
| ||
| I have a semi-relevant question of my own here. I've been using "realtime" loading for my game, mainly because I have a very simply animation going on rather a static "loading..." bar. To achieve this, I simply did the obvious, which is to say I made an UpdateLoadingAnim() function, and put some code in all of the loops in the level loading code that calls the function every 15ms. Runs nice and smooth, with one exception: although I can break up all the in-game operations, the loading of actual media is a single-step issue that I can't subdivide. I got around this easily enough in my level loading code because the only media that isn't loaded at that point is the randomly selected BGM file, and I very cleverly "hid" the loading code for the music at a point that the player would never notice it taking place. Out of curiousity, though, I'm wondering: IS there any way in Blitz, even using DLLs, that one can "stream" media when loading it so that the loading operation can be interrupted for things like loading animations? |
| ||
| IS there any way in Blitz, even using DLLs, that one can "stream" media when loading it so that the loading operation can be interrupted for things like loading animations? A programmer explained to me that it is possible to progressivly load a file, but you have to know how the file format is made, so that you can load only some part of the file during each loop and then recreate the Terrain/Mesh/Texture in blitz3d with the functions CreateTerrain(), CreateMesh(), CreateTexture(), This is far too complicated for me for the moment. Last edited 2011 |
| ||
| Research on asynchronous loading. All the HD i\o is done in a separate thread, and this won't give the slightest stall to your game. You need thread functionality to do this, and Blitz3D does not offer it out-of-the-box. There is a DLL around but I haven't looked any further. I think Yasha knows more about it. |
| ||
| Have a look at Blitz AL here http://tools.mirage-lab.com/ might have some useful options for what your doing - haven't played around with it enough yet to know. Also as mentioned above there is the fastpointer dll that can thread B3D functions but you will have issues regarding loading a sound in one thread and trying to use it in another (or main) thread - tho this isn't impossible to work around but you really need to understand multi threading to implement it "safely" so it won't crash your application. Last edited 2011 |
| ||
| A programmer explained to me that it is possible to progressivly load a file, but you have to know how the file format is made... ...or, you can just roll your own file format. :) You can do this for images, textures, and meshes (but not sounds or music). |
| ||
| IS there any way in Blitz, even using DLLs, that one can "stream" media when loading it so that the loading operation can be interrupted for things like loading animations? There is! For sound files, there are alternative sound libraries such as Blitz Bass Studio, which allow native streaming of audio files without having to fully load them from disk first. For images, textures and meshes, multithreading would really come in handy - just load your resources in a seperate thread and hand them to the main thread when you're done. Unfortunately though, this is not possible in B3D natively. There is a DLL out there to do this, but as far as I remember it's based on a "get B3D function pointer" code I wrote a while back, and I remember it to be not really stable all the time. Another thing is that B3D is not inteded to be used in a multithreaded environment; none of the standard functions are threadsafe. So what's left is to write a DLL for B3D which can use fancy multihreaded loading without affecting B3D. This works well in some cases. Loading/saving images and textures is quite straight-forward, just pass the image buffer to the DLL and have it work on the pixel data directly (I actually needed this for a game before, so I know this works quite well). Meshes are definitely a lot harder to do - since you're not allowed to do DirectX calls in more than one thread, you'd have to load the mesh data into a temporary buffer, pass that to the main thread and have it copy that data into a B3D mesh. Since your main thread would freeze during the copy step anyway, I think that there's not much of an advantage to loading meshes asynchronously. Making your own 3D format and loading that progressively would be a solution, although as far as I remember Blitz did not allow adding bones to an animated model, so loading these with your own format would probably not be possible. Last edited 2011 |
| ||
| as far as I remember Blitz did not allow adding bones to an animated model, so loading these with your own format would probably not be possible Actually with the help of the same "unsafe hacks" DLL that provides threading functionality, it can be done without too much difficulty. However, that code hasn't been properly field-tested. Because it attacks the binary structure of the entities directly, it also needed to be rewritten for specific versions of B3D after any change Mark made to the entity structures. Not the best way to do things by a long shot, but it can at least be done. |
| ||
| Alright, good to know on all accounts. I don't actually need to be able to do this in my case, so I'll just let things lie as they are, nice and stable, but it's handy to know these possibilities exist. |
   |