Instant Replay (Saving large amounts of data)?

BlitzMax Forums/BlitzMax Beginners Area/Instant Replay (Saving large amounts of data)?

Herbert Filby(Posted 2010) [#1]
Hi all. I just purchased Blitzmax and Leadwerks 2.3 to fulfill a game idea I've been hatching for about 10 years now...

I'm trying to recreate an instant replay code that I prototyped in QBasic, and it needs updating.

I am going to be storing perhaps up to 24 hours of non-continuous instant replay data, and I need to know which is going to be faster:

-SQL database
-XML file
-or if there is another option, please let me know

Also, data may be altered dynamically while the replay is happening (like if I were racing against a "ghost" version of myself in Mario Kart, but I can bump into it, rather than passing through it). So it will be simultaneously recording and playing back at the same time, for multiple objects, and updating old object replays if they are altered.

Anyone have any suggestions which route I should go? I plan on making a full 3D game, so each object needs to be "saved" and timestamped... ie. a lot of data being recorded and played back.

I'm not worried so much about how much space this takes up, but more on performance of read/write on random access data that happens every second...


Volker(Posted 2010) [#2]
If you want to use serialization (storing objects) there
are Bruceys persist mod, which stores data in XLM files
and chaos.clone mod; which is faster and stores data binary.
You should look at both and decide which is better for your purposes.
Serialization uses reflection, which is not the fastes at all.


Czar Flavius(Posted 2010) [#3]
For replays, usually not the entire game state is saved but instead the actions or changes that take place. For example, only when a unit changes direction, speed, fires weapon etc. The rest of the time, the game world updates as per your usual update function. This requires considerably less space, but requires the game to be finely tuned so that replays do not go out of sync.


xlsior(Posted 2010) [#4]
If you're looking into SQL, you may want to consider SQL Lite as well -- it's not as fast/flexible as for example MySQL / MSSQL / Oracle, etc. but it does support the basics and it has one massive advantage: You don't have any external dependencies, but the module will enable you to interface directly with the SQL file, without the need to actually install and configure an external database first.

(Brucey has a SQL Lite module for Blitzmax, plus there are one or two others as well)


Czar Flavius(Posted 2010) [#5]
If the data will be read sequentually, it would be best stored in a flat file I think.


Herbert Filby(Posted 2010) [#6]
Yes I'm experimenting with SQLite and Brucey's module. I will look into this persist mod and see if it's more on what I'm looking at doing.


TaskMaster(Posted 2010) [#7]
The benefit of a database is being able to easily find your data. But if your data is just sequential, there is no reason to use a database. Just use a flat file and stream it. Actually, I think 1 time per second is too much. I would think you could save the instant replay of a whole race in memory, then write it to a file afterwards. Unless your race actually takes 24 hours...


Herbert Filby(Posted 2010) [#8]
It's not just one race, though. I want to be able to save multiple races and simultaneously alter files as they are being read (if changes are made during replay). So in essence, you could end up racing 20 iterations of yourself at once.

My original prototype created a separate file for each object, and using QBasic, the files were quite small, but I had to limit how many objects could be recorded and played back.

It would start recording each movement of the player character, while every 0.1 seconds it would look for any other recordings, find the coordinates of that object, and output it to the screen. There was no tweening, so even updating it every 0.1 seconds, playback was a bit jerky, plus that algorithm did not take collision detection into consideration.

I guess since the files will be sequential, I'll try doing it using simple read/write functions to a file and see how that goes. I was hoping that there was a better way but logically, this makes sense. And I wouldn't have to depend on a third party module.

I will see if I can stream it to memory for recording, and write to a file afterward, which should improve performance during gameplay.


Czar Flavius(Posted 2010) [#9]
Do you want a useable reply or not?

It would start recording each movement of the player character, while every 0.1 seconds it would look for any other recordings, find the coordinates of that object, and output it to the screen. There was no tweening, so even updating it every 0.1 seconds, playback was a bit jerky, plus that algorithm did not take collision detection into consideration.


For replays, usually not the entire game state is saved [per iteration] but instead the actions or changes that take place. For example, only when a unit changes direction, speed, fires weapon etc [is this saved in the replay]. The rest of the time, the game world updates as per your usual update function [changing the behavior of the units according to their actions]. This requires considerably less space, but requires the game to be finely tuned so that replays do not go out of sync.


Analogy: Chess games are not recorded by writing down the positions of every piece on the board every turn. Instead, only the moves that take place are recorded. This takes considerably less space, and for a real-time game completely removes the need for tweening or special collision detection algorithms, as you just use your normal ones. Only the initial state of the game needs to be saved, and then a list of changes to the flow of the game thereafter.

If you want to start up the game at any point in the recording, then perhaps you could save the entire state at regular intervals (of the order of minutes, not seconds!) So save at 5 mins. Then if you want to resume at 6 mins, load the 5 mins state and then process 1 min worth of actions. As you won't be drawing the game you can usually go through this in a couple of seconds.


TaskMaster(Posted 2010) [#10]
Is this some multi-player online sort of thing? If so, I would think a server could be used to keep it all in memory. If not, then that makes it even easier.

I still don't see the need to writing this stuff to files.

Unless you are trying to do multiplayer writing files instead of using a server. And if so, you should reconsider and house all of that data at a server and not use files until opportune moments to save the replays, like when you are done with a race.


Who was John Galt?(Posted 2010) [#11]
I don't see how this could work. Either you race a ghost of yourself that you cannot interact with, or you race AI. The only option if you want to be able to interact with old versions of your plays, is to switch the ghost from recorded play to AI as soon as you have hit it... unless the collision does not alter the course of the recorded version.

Serialisation is definitely the fastest and most efficient way. This can be done without reflection although it is a bit more effort. Reflection may be plenty fast enough depending what you are doing. Usually you would just record user input, then run the game with this recorded input instead of straight user interaction. Assuming your game is purely deterministic, this will recreate the whole game. Even 'random' elements can be recreated by reusing that same random seed from the original game.


Herbert Filby(Posted 2010) [#12]
Does anyone have a tutorial on this "reflection" and serialization John Galt mentions? I've seen those words used on the forums while I've been reading up on this, but don't have a clear perception on what they are. Thanks :)

EDIT: also, it's not multiplayer so all data needs to be stored on the local machine for later retrieval if the user quits the game or wants to take a break.