Lock step networking model
BlitzMax Forums/BlitzMax Programming/Lock step networking model
| ||
| I have read about potential approaches regarding multiplayer games over networks. One of the suggested models was the "lock step networking model". Which collects inputs of the users during a "turn" and sends them out to the other ones which collect the incoming inputs and use them for simulating their next "turn". Random data is using things like mersenne twister and a shared "seed" for the random number generator. Currently my game uses something similar: All clients use the same random number seed and therefor "should" be able to simulate the world exactly similar the other clients (I think "floating point precision" is another thing ... see gafferongames.com for an article about this). For now I directly send out networking packages on user interaction: buying an item sends out an "inventory change" with the item id ... the other clients then automatically check if the action was possible (enough money etc) and add the item to the inventory too. This is done similar for the AI (they send out "I want to buy item X" and the engine returns if successful or an error mentioning what failed). So both: networking players and ai use similar commands to do their things. Now latency will get added to the whole networking: what happens if two players try to do the same thing within a small timeframe. Of course "authorative" clients (eg. the "host") decide who was first (regarding the incoming packets order) but this will add a visual glitch on one of the clients: both think they are able to buy that item, but suddenly (with the incoming authorative packet) one of the clients will recognize that he wasnt able to buy it. Ok ... so this is why I have read about that "lock step networking model" and some questions came up: How to do that whole "turn" thing? Currently I have "update" and "render" calls to all objects. Updates eg. change the position of a figure, while Renders actually draw the figure (on a tweened position). Does "turns" mean I should not directly act to a "mouseHit(1)" but store all of the inputs in something like an InputHistory-Log and then inbetween my "updates" check if it is time for another "turn", process all input from the InputHistory-Log ? If yes, how to do this things regarding individual objects? Whats up with GUI ? should guiObjects also be only processed on "turns" or should the result of a gui (eg. "buy item") be send out as something in the likes of an event stored in a "EventHistory-Log" (similar to the input)? Should Eg. a "GameScreen" have now three methods: - Render() - Update() - Turn() ? For now Update contains: update child elements, update tooltips (fading, content...), update positions, ..., check input (is mousedown then dothing xyz) Am I right assuming that I just extract the whole input (aka "user control") into the new "Turn()" and this will be enough? Any other ideas about this model? I just want to add that "server/client" is surely a nice thing, but for my game a bit of overload (and adding more weight to latency). I do not have a fast paced gameplay - it is just "click here and figure moves there" plus some "user clicks on item and places it in his inventory". Think this should play well with the model described above (input synchronization). What are your thoughts about an global "input history" - doesnt this allow some kind of "replay" ? What happens to AI ? They do not send "keystrokes" but directly call the function which is called when pressing that key manually. So if "KEY_K" moves a figure to Screen K via "ChangeScreen(K)" the AI directly calls "ChangeScreen(K)". Should the AI emit "KeyHit(KEY_K)" to the engine, or stay with directly calling the resulting function? The first option requires knowledge about input from the AI - which might change somewhen. Api-calls (the second option) are of course abstracted from input and may keep more constant over time of the project evolution. Api-calls are not "input" - so how to store them regarding the "lock step networking model". Before you ask: the AI does not run on all clients, because I want the AI to be provided by users (so you can have ai-fights without exposing the "ai source file" to other clients). So the clients have control over their local players (ai or human). Hope some of you can help me finding some conclusions. bye Ron |
| ||
| My currently formed idea of "my games" networking part is the following: each user interaction - is defined in a type extending from "Type TUserAction" (which can serialize to string/int / deserialize from string/int) - is stored in a queue "pending" when getting created at the end of a "turn" - the next pending action is emitted (send to network - or just received when solo playing) - if there was no action in "pending" then a "null action" is send at the begin of a "turn" - receive confirmations of outsend actions (of emitted in last turn) - if all players (maybe ignore for "high latency" players) confirmed an action, add it to a "confirmed queue" - process the confirmed queue (first in first out) - confirm incoming pending actions Is this (mixed) lock-step or did I miss some things? With mixed I mean: high latency players would stop the game because they do not confirm actions in time making it laggish for all players. So I thought about ignoring missing confirmations after a given gone time - so every player above eg. 100ms response time wont be able to confirm an action. As soon as response time lowers again, this player will try to catch up by issuing the upcoming pending actions to all others. Hopefully there is one of you willed to construct a small lock-step-networking-with-gnet-example (we could collaborate on this). bye Ron |
| ||
| The best way I found to handle two people buying at the same time is who's packet got read first, there could be a decider packet so first play gets the 'you got it' packet and the other gets the 'nope try again' packet. This situation shouldnt happen too often so it would work fine, the other way would be purchases have to be ratified by the server so all changes are done from that and reported to the client on how they did. This method does stop cheating a little BUT will send your packet count up. My preference would be the first one. |
| ||
| Ok. For a slow paced game as you describe capturing inputs is fine. Have to be careful about users spamming with clicks. Re replays. Yep thats a good way of doing it and often how ive done it in the past. The alternative if you did have floating point calcs is to record a replay with keyframes instead and interpolate storing keys at both a fixed time interval and when significant events happen. Using replays as input is usually very easy to implement but things can stilk go out of sync if you are not careful about the game logic. ..it only a single number to be wrong once for everything to be different and can be a nightmare to debug in lengthy games. The main issue I see you will have is the speed of the game. As you say you currently have an update and render loop. Perhaps separating network upstream updates from that and lowering their rate would help. A lot of this depends on the type of game. Eg a yacht racing simulator where the physics objects data changes fairly predictably is a lot easier than a fast paced shooter. Also....how many entities are you wanting to keep track of. If the number is small then perhaps you could pass the entire entity state each network update. |
| ||
| Thanks for your comments. @buying first/second This works of course when using a "lock-step" approach, because there is an order of incoming commands which is the same on all computers (your sent commands are in the list of the returned commands - aren't they?) For me it is more a thing of "input = key/mousebutton press" or "input = action". I think they talk about "actions" (similar to a server/client approach). So instead of sending "mouseclick, button=1, x=10, y=20" you send "figure x, change target x=10, y=20". so "input" could also be named "event". Above seems to be the only valid solution regarding "3rd party" (or "common/shared" code). Let me explain on some parts of my game: The game consists of up to 4 players in a skyscraper. This building has rooms with multiple functionalities (like "gamescreens" which can get blocked by other figures). To reach these rooms the player might use an elevator. Other figures (computer controlled) move through the building too (eg janitor sweeping the floors, postman bringing messages from room a to room b). Each player has a collection of programme licences to broadcast, news to make a news show out of it, advertisement contracts for ad-blocks during the broadcast etc. This data can get referenced by "GUID", so it could get spread along on multiplayer game start (if the host uses a custom database else all players already have the same data). Currently a random seed is shared amongst all players on game start and mersenne twister random numbers are used to fetch random data from the collections. So all players have the same start licences, contracts etc. Also the same news events happen on the individual players "worlds". So long - no problem. Now to the interaction part. There is also a room offering "scripts" for custom programme productions (like TV-movies -> asylum and the likes :D). For now the database contains some templates to generate scripts from it (like "A %ROLE1% fights with %ROLE2%" with both variables being arrays of options). This is of course again deterministic possible through mersenne twister. BUT ... our plan is to make things customizeable: people should be able to enter custom titles, descriptions etc: to create THEIR script. Now to the "buying" issue: as rooms are blocked as soon as one player enters the room it is not directly an issue of people buying things during the same time but to validate that only one person is in the room on all clients - if both try to enter the room simultaneously, it might happen that both think it is valid to enter and *bamm* problem. Once they are in the room there should be only one and the only problem is the synchronisation of the period "player a leaves room and player b who waited patiently enters now"). Currently I handle everything this way: - each action (which other players need to know) emits an event - my networkhandler listens to these events - onEvent the networkhandler spreads this information around - on receiving network data, the networkhandler calls the function which emit the corresponding "onEvent" (with silencing it before, so it does not send the event again) -- this one is a bit whacky, normally I should have "CallAction() -> Action()" "Action()" and "NetworkAction() -> Action()" to avoid this silencing issue This means instead of listening to "keypresses" I listen to "onChange"-Events of specific guiObjects (eg "player name"-guiInput). I made my gui-System variable, so many of my game objects extend from this. So when managing the TV channels broadcast plan ("PlayerProgrammePlan") the movies displayed there are "guiListItem"-descendants. As soon as the player drags such an item, an onDrag-event is send, if he (successfully) drops it, an onDrop-event and so on. Each of this events are listened by the room handler and it reacts by calling corresponding functions like "PlayerProgrammePlan.SetProgramme(programme, day, hour)" which then sends out an event "programmeplan.onchange" with the programme, day, hour as data-payload). This is send to the network and so on. The "host" also sends out figure positions periodically and clients adjust their position a bit regarding it (some kind of prediction-mode) - this is surely more a "server/client" mode. Ok ... described enough to sum up for the problems :D Above I hope to have explained some issues which fit into both categories: peer2peer (lock step) or server/client. I started coding the multiplayer more in the likes of a server/client but this ended up in sending more and more data to the clients - as clients were more "terminals" and not processing anything regarding game mechanics (which programme is sold by the movie agency, which news happens ...). But now I think I already use some kind of "lock step" approach, but just not recording it "lock step"-like. While there are actions benefiting from lock-step: figure target changes, figure "room changes" (which is animated and has waiting times... so this would come in handy), ... just all interactions needing to be in the same order on all clients, there are of course also things which could be handled "extraordinary". So a "player name change" does not need to be synchronous on all clients, I think such things could be handled in a "sendToServer(const_namechange, name)" manner. Would such an extra-handling have benefits? Once "lock step" is introduced, there seems to be no effort in sending things "inbetween" especially as the time between "lock step"-steps does not matter in cases like player-renaming. @click-spamming This is only adding to the packet size. If you click 20 times within a "lock step" this will get send what happens because of the 20 clicks. If eg. the click "drags" and the next click "drops", then you will send drag-drop-drag-drop-.... in this lock-step. on receiving the command all clients will wait until the "lock step"-step for this command is the current one - and execute the commands. As long as all clients get the same "actions", there should be no matter with "spamming clicks" (or spamming actions). All execute the same. Only problem I see with "lock step" is the latency. It takes time until your action is done on the client. While this is not problematic for animated things (figure waiting 200 ms in front of a door -- knocking -- then door opens, figure enters, screenchange-animation, figure in room) it is a bit "slowing down" for actions like shift-clicking to drop copies of a movie. While I could position 10 advertisement-blocks in a matter of a second by click-move-30px-click-move... : it is a) not possible because feedback of success/failure is not available after a "mousehit" b) possible but not visible for "lock step"-interval-time (until acknowledgement received) How to handle things which need immediate feedback (I know the answer: rewrite that logic) ? Like described above, the "movie block" is a gui element: on drop it emits an event "onDrop" which is listened by a handler, this handler calls the programme plan to adjust the corresponding programme slot ... which sends out an event the network handler is listening to If nothing stops the "onDrop"-event, the movie block will be no longer dragged and is placed in a guiList representing the current days programmeplan. With "lock step" this would be this way (I assume): On drop of the movie block the event "onDrop" is emit. The handler is listening to it and creates an "action" to call the programmeplan. ... End of the turn, actions send ... start of the turn+2, actions available Programme plan gets called via action to adjust the corresponding programme slot Network handler does not need to do something as all clients already got the information and acted BUT ... in my current code the handler listening to "onDrop" is able to Veto() this onDrop (eg. the underlying slot is occupied by a non-moveable programme). With the "lock step" this veto is not possible as the result of the function result of "programmeplan.changeplan()" is not available in this situation. That whole "onDrop" and "Veto()" is needed for a automatically working "drag n drop" solution. But as each "Drag" or "Drop" might result in game play interactions (programmeplan changes) it is possibly too tightly integrated. All other clients of course do not need to know anything about "gui"-Elements, they are pure for the interaction of the player in front of the client and the game. Ideas how to split apart situations like above to handle them accordingly on the clients? Sorry for the lengthy post, hope you did not stop before reaching this line ;D bye Ron |
| ||
| Hi, I use this method for Phoenix USC (fast action game): http://www.gabrielgambetta.com/fpm1.html I have an authoritative server that is sort of lock-step in that it is always playing the game on it's own map and syncing the clients when out of position. The method above works well for fast paced games. |
| ||
| Hmm seems my longish posts do not convince people to answer my questions or to tackle them together with me ... Here is another question (ok, description of the problem first): I have an ingame "time". The figures in that building have a 24hrs/day and there is a clock displayed in the interface (so players always see the live ingame-time onscreen). At each ingame-minute (or earlier, depending on the game speed, but at least every ingameminute) the AI scripts get called so they could "think about things". Of course they get called on every event regarding their figure (eg. "reached room") or on "onDay" "onHour" ... Also the ingame time is controlling things like broadcasts (new broadcast start, audience calculation ...). Now the question: Should I synchronize that ingame time - is it "logic" or "physics". I assume I have to "tick" only during the lock-step-turn-begins and to interpolate the displayed onscreen-time until the next lock-step comes in. The time gone between two locksteps are then simulated (like "for i = 0 to timeGoneBetweenSteps; doSimulate(time+i)" ). Am I right assuming this? What about physical movement ("reaching a door") - could I handle that in the physics update (x :+ dx * deltatime, then check if reached) or should the figure move "dumb" until the logic stops them? I think I should "stop the figure" in the physics and record the action ("reached room door"). Right? Hope some of you keep answering to this thread. PS: @Richard: I have read that article before, authoritative is used on my side already - eg. regarding "game scenario settings". Things like this (or seed spreading) are of course taks of the host/master. bye Ron |
| ||
| Quoting myself to emphasize my current problem This means instead of listening to "keypresses" I listen to "onChange"-Events of specific guiObjects (eg "player name"-guiInput). I made my gui-System variable, so many of my game objects extend from this. So when managing the TV channels broadcast plan ("PlayerProgrammePlan") the movies displayed there are "guiListItem"-descendants. As soon as the player drags such an item, an onDrag-event is send, if he (successfully) drops it, an onDrop-event and so on. Each of this events are listened by the room handler and it reacts by calling corresponding functions like "PlayerProgrammePlan.SetProgramme(programme, day, hour)" which then sends out an event "programmeplan.onchange" with the programme, day, hour as data-payload). This is send to the network and so on. The "host" also sends out figure positions periodically and clients adjust their position a bit regarding it (some kind of prediction-mode) - this is surely more a "server/client" mode. Ok ... described enough to sum up for the problems :D When using lockstep, I send out delayed actions, so far I understood. So instead of "BuyMovie(x)" I have "AddAction(buymovie, [x])". The action is then run when verified by all others, or now (when in singleplayer-mode). But I still do not know, how to handle immediate reactions - like "drag n drop". So when placing a game element on the spot of another, multiple events are processed: drag of the "another" one and "drop" of the game leement. Both events trigger other game functions (my way of decoupling cyclic dependencies). So a "drag" eg. removes a programme from the programme plan, a "drop" adds the dropped programme to the plan. The problem is, that "GUI" would react immediately (not knowing about lockstep at all): so it drops the element visually, and then would send out an action which would get delayed (because of lockstep). The reaction of the "try to drop" is to "drag" the other element, which would send out an delayed action too. so far, so good. but the "GUI" also checks if it is able to do things, and with an delayed action not having taken place yet, these checks will fail. So a check might be: if the current programme in the plan is the same as the one we want to drag, "drag" it. With the delayed happenings, that current programme is still not replaced and the GUI would fail. So this seems as if the basic problem for me is when GUI things interact with eachother: aka drag'n'drop (used eg. for "sorting" elements). For simple button clicks it isn't that problematic, just create an action (for lockstep) instead of directly executing the game/logic-function of choice. Short: How to solve "drag n drop"/"gui interaction" with lockstep? bye Ron |
   |