PushButton Networking for Monkey

Monkey Forums/User Modules/PushButton Networking for Monkey

Ferdi (Posted 2013) [#1]

The Quick Summary GitHub Repository: https://github.com/ferddi/pbnet-monkey/ License: MIT License (Following the original license) https://github.com/Ferddi/pbnet-monkey/blob/master/License.txt Requirements: Monkey V69 (Between 66b and 69 there is a fix for PutString in brl.DataBuffer), and Diddy (version 2013-03-03, I am using EncodeBase64 and XML) Original Repository: https://github.com/PushButtonLabs/PBNetworking/ Original License: https://github.com/PushButtonLabs/PBNetworking/blob/master/License.txt The Long Epic Post I have been chipping away slowly on this. When I was looking into the different platform to use, I first looked into flash, as a development platform. And I also look into what engine was available. And PushButton Engine came on top of my list. I also noticed it had a PushButton Networking Engine. To cut the long story short, in the end I settle with Monkey, mainly because it just work. I don't know much about networking, but I remember I got the PushButton Networking Engine going. And, I believe Monkey badly needs a networking module. So I decided to port it over to Monkey. PushButton Engine and Networking are written by the people who use to be at Garagegames, I think. And I believe Dynamix before that. They wrote Tribes and Torque Network Library. And the theory behind these engine are used in games like Halo. If you want to learn about the theory behind this, why there is event and ghosting, read and/or watch: http://www.gdcvault.com/play/1014345/I-Shot-You-First-Networking (video) I think you only have to watch the first 30 minutes, the rest he talks about all problems you are going have! =P http://blog.wolfire.com/2011/03/GDC-Session-Summary-Halo-networking (summary of the above video) http://www.pingz.com/wordpress/wp-content/uploads/2009/11/tribes_networking_model.pdf Anyway, to the FUN stuff Server running on GLFW PC Windows 7 with GLFW client, XNA client, ... HTML5 client, Flash client, ... Android client, and iOS client connected. iOS client connect to server running on Android Android client connected to server running on IOS 2 clients HTML 5 and Flash connected to server running on XNA. Here I did stress testing with 16 connections on GLFW. If it is not up there, I have NOT tested it on that device. And sorry I only tested HTML5 on Chrome. Technical Stuff, that I think you should know if you want to use this. 1. I just want to use TCP Socket. But HTML5 does not connect. First, for HTML5 client to connect to a server, it requires to do a handshake. The server requires so send a message back to the client with an accept key. This key is generated as follows: acceptKey = EncodeBase64(Sha1(clientKey + magicString)) 'Well Magic String is the GUID = 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 I posted my Sha1 source code somewhere in the forum, and that is the only reason why I have it, because I need to do a handshake for HTML5 - pain in the ... Second, when you are sending a message from the server to the HTML5 client, you need to write a header field. Last, when you receive a message from HTML5, you will need to unmask it For more info read the RFC 6455, The WebSocket Protocol: http://tools.ietf.org/html/rfc6455 But this guy explain it simpler: http://srchea.com/blog/2011/12/build-a-real-time-application-using-html5-websockets/ These code are in pbnet.core.networkconnection.monkey, and then look for the flag isClientHtml5WebSocket. If this flag is true, then the server will send and receive using the above method. 2. I just want to use TCP Socket. Why am I getting Security Error for Flash? Or why is flash not connecting? Flash also has to do a handshake, flash sends a policy request, like this "<policy-file-request/>" Then as the server, you need to send a XML Response like this: <?xml version="1.0"?> <!DOCTYPE cross-domain-policy SYSTEM "/xml/dtds/cross-domain-policy.dtd"> <cross-domain-policy> <allow-access-from domain="*" to-ports="1337" /> </cross-domain-policy> Remember to change the port. These code are in pbnet.core.networkconnection.monkey 3. I am using Diddy's XML and Base 64 Encoding. Because, I am lazy! =) 4. Where to cut between PushButton Engine and PushButton Networking. PBNet uses the entity systems of PBEng. And, PBEng has some nice features, like it can create an entity from XML files. That is you don't need to recompile if you want to change an entity functionality. But if I port that part of the PBEng, and put it into PBNet, it will be hard to integrate with all the other modules that is available on Monkey. So I have removed the entity system that is in PBNet. It is mostly in the ghosting part of the code. Right now the PBNet entity system is just pure skeleton. 5. Server cannot be on Flash or HTML5. Flash player socket and HTML5 WebSockets API CANNOT listen for connections. Flash and HTML5 are purely for client only. That is why the original PBNet server uses Tamarin (which I never got working). The one that I did get working is using Adobe AIR. Because Adobe Air has ServerSocket, it can listen for connections. 6. How to compile the original PBNet? Read: https://github.com/Ferddi/pbnet-monkey/blob/master/HowToCompileOriginalPBNet.txt And yes, for now, the monkey code can connect to Adobe Air code and vice versa. 7. The core of the module At the heart of the module it is basically pumping 102 bytes (LENGTHFIELDSIZE + PACKETSIZE) via TCP. So if for some reason you are sending more than 102 bytes and your message gets split up, you may want to increase the PACKETSIZE to the appropriate value for your game. LENGTHFIELDSIZE is 2 bytes, so you can have PACKETSIZE up to 64K. Those two consts are in pbnet.core.networkconnection.monkey. 8. This engine uses polling. I didn't want to go against what Mr Sibly has put in place in the Monkey Framework. Basically there is Update and Draw. And I believe at any time during the game, there can only be 1 update or 1 draw. If we have thread going for another update or connection, I felt this go against Mr Sibly design. And from my experience, if you go against the intended design, it will cause other problems somewhere down the line. And that is why it is using polling. 9. Windows 8 or WP8 EDIT: Rone has done tcpsocket for Windows 8 for us. I don't have access to windows 8 or windows phone 8 or the windows 8 developer account. But if someone wants to do these two, it is very simple. First, you will need to write TCPSocket, which is brl.TCPStream with 2 added method, State, and SetupSocket. State is just to return the variable _state. And SetupSocket is for passing the variable _socket from TCPServer to TCPSocket when the TCPServer accepts a connection. Second you will need to write TCPServer. This is more involve. You will need to figure out how to do Select(in C++), Non-Blocking IO(in Java), or Pending (in C#) but for Windows 8. Then you accept the connection and send it over to your Windows 8 TCPSocket via SetupSocket. Lastly, make sure the Nagle's algorithm is off, or No Delay is on. 10. XNA GetChars Enter and backspace doesn't work in XNA. So I added 2 buttons BKSPC and ENTER. ToDo Lists / Roadmap (in no particular order!) 1. When there is a close connection in iOS, the program exits. This is very noticeable when your server is GLFW, and you have a client in an iPhone. When you close the server, the app shuts down. I need to look into it. 2. Bitstream and brl.DataBuffer does not talked very clean with each other. I believe I need to write some native code in here, especially for C++ code, using pointers and all, so these two classes talk nicely to each other. When this comes into effect, it is another reason why you want your server to be on the C++ code base. 3. I have planned to look further into removing the whole reflection from the module. I have another code that was using reflection. I removed it, and I got a 3 - 4 times increase in speed. For example, what used to take 3ms - 4ms using reflection, was now taking only 1ms with out reflection. If the code is at the core, than I think this is required. But this way you have to use ALOT of Const. Basically I need to profile. 4. Currently I split server and client into two difference executable. I need to create an example that is just 1 executable, so server and client it joined together. 5. UDP? - Flash and HTML5 WebSocket can not do UDP, since they can't listen for connections. I am a little reluctant to do this. 6. Bluetooth? I have no clue what so ever about bluetooth. 7. Scoping, I need to take a look at it, I think it is not working. 8. Remove for loop in tcpsocket.html5.js, this can't be good, especially in a scripting language! 9. Remove for loop in tcpsocket.java =) From here and downward, it is mostly examples: 10. Example on how to integrate with Diddy. 11. Example on how to integrate with Flixel. 12. Flixel has a recording and playback system, how to send those data. This will show how to just use Core / Element only. 13. Example on how to integrate with FantomEngine. 14. Example on how to integrate with MiniB3D. 15. Example on how to integrate with Horizon. Any other engine I should look at? Any other feature I should be implementing? I think there should be a sticky post on the User Module forum, listing what engine is available. And may be even link to that post from the front page. So new users know exactly what Modules are available to them. Another thing, I am a very slow programmer. =P I am going to flip flop on this and my game. I started this in August 2012. That's how slow I am =) If there are any bugs let me know, and I will try to fix it right away. Can you please submit bug with source code as example or detailed steps on how to reproduce them? Basically, if I can't reproduce the problem, I can't fix the problem. I know, it is very rough right now, I will try to polish it slowly. Anyway, Hope it will be useful.

FelipeA (Posted 2013) [#2]

It's funny, I was actually listening to The lord of the rings ost when I was watching this post. I felt so epic reading this, it's amazing! Monkey really needed a networking module like this. I'll do some tests with this. Thanks!

DruggedBunny (Posted 2013) [#3]

Interesting, will take a look!

Neuro (Posted 2013) [#4]

This is pretty f*ckin awesome.

Beaker (Posted 2013) [#5]

Sounds great.

Rone (Posted 2013) [#6]

Hi, Nice module! I created a fork and added tcpsocket.win8.cpp, I was bored // author: Sascha Schmidt(Rone) /* Add the following usings to MonkeyGame.cpp using namespace Windows::Networking; using namespace Windows::Networking::Sockets; using namespace Windows::Storage::Streams; using namespace concurrency; using namespace Windows::UI::Core; */ #define _HIDE_GLOBAL_ASYNC_STATUS 100 #include <windows.networking.sockets.h> #include <wrl.h> #include <robuffer.h> #include <vector> #include <mutex> #include <atomic> #include <list> #include <condition_variable> #include <thread> // ***** tcpsocket.h ***** static byte* GetIBufferPointer(IBuffer^ buffer); // used to wrap BBDataBuffer when using DataWriter::WriteBuffer class NativeBuffer : public Microsoft::WRL::RuntimeClass< Microsoft::WRL::RuntimeClassFlags< Microsoft::WRL::RuntimeClassType::WinRtClassicComMix >, ABI::Windows::Storage::Streams::IBuffer, Windows::Storage::Streams::IBufferByteAccess > { public: virtual ~NativeBuffer(); STDMETHODIMP RuntimeClassInitialize(UINT totalSize, byte* ptr); STDMETHODIMP Buffer( byte **value); STDMETHODIMP get_Capacity(UINT32 *value); STDMETHODIMP get_Length(UINT32 *value); STDMETHODIMP put_Length(UINT32 value); private: UINT32 m_length; byte * m_buffer; }; class CircularBuffer{ public: CircularBuffer(); void Write(byte value); byte Read(); bool IsFull(); bool IsEmty(); int Read(byte* buffer, int count); int Length(); private: std::vector<byte> _data; int _start; int _end; int _size; }; class BBTcpSocket : public BBStream { public: BBTcpSocket(); ~BBTcpSocket(); bool Connect( String addr,int port ); int ReadAvail(); int WriteAvail(); int State(); int Eof(); void Close(); int Read( BBDataBuffer *buffer,int offset,int count ); int Write( BBDataBuffer *buffer,int offset,int count ); int SetupSocket(int so); private: void ReceiveLoop(DataReader^ reader, StreamSocket^ socket); IBuffer^ CreateBuffer(UINT32 cbBytes, byte* ptr); std::atomic_int _state; //0=INIT, 1=CONNECTED, 2=CLOSED, -1=ERROR Microsoft::WRL::ComPtr<NativeBuffer> _nativeBuffer; StreamSocket^ _socket; DataWriter^ _writer; DataReader^ _reader; std::mutex _mx; CircularBuffer _cb; }; // ***** tcpsocket.cpp ***** CircularBuffer::CircularBuffer(){ _size = 8096; _data.resize(_size); _start = 0; _end = 0; } void CircularBuffer::Write(byte value) { _data[_end] = value; _end = (_end + 1) % _size; if( _end == _start ){ _start = (_start + 1) % _size; // full, overwrite } } byte CircularBuffer::Read() { byte value = _data[_start]; _start = (_start + 1) % _size; return value; } bool CircularBuffer::IsFull() { return ((_end + 1) % _size == _start); } bool CircularBuffer::IsEmty() { return (_end == _start); } int CircularBuffer::Read(byte* buffer, int count) { if( (_start + count ) >= _end ) { count = _end - _start; } memcpy(buffer, &_data[_start], count); _start = (_start + count) % _size; return count; } int CircularBuffer::Length() { int length = _end - _start; if (length < 0) { length += (_size + 1); } return length; } NativeBuffer::~NativeBuffer() { } STDMETHODIMP NativeBuffer::RuntimeClassInitialize(UINT totalSize, byte* ptr) { m_length = totalSize; m_buffer = ptr; return S_OK; } STDMETHODIMP NativeBuffer::Buffer( byte **value) { *value = &m_buffer[0]; return S_OK; } STDMETHODIMP NativeBuffer::get_Capacity(UINT32 *value) { *value = 0; return S_OK; } STDMETHODIMP NativeBuffer::get_Length(UINT32 *value) { *value = m_length; return S_OK; } STDMETHODIMP NativeBuffer::put_Length(UINT32 value) { m_length = value; return S_OK; } BBTcpSocket::BBTcpSocket(): _socket(nullptr), _writer(nullptr), _reader(nullptr), _nativeBuffer(nullptr) { _state = 0; _socket = ref new StreamSocket(); _socket->Control->NoDelay = true;// Make sure Nagle's algorithm is off! } BBTcpSocket::~BBTcpSocket() { Close(); } bool BBTcpSocket::Connect( String addr,int port ) { if( _state ) return false; auto str_addr = ref new Platform::String(addr.ToCString<wchar_t>(), addr.Length()); auto str_port = ref new Platform::String(String(port).ToCString<wchar_t>(), String(port).Length()); std::condition_variable cond_var; std::mutex m; try { auto hostName = ref new HostName(str_addr); std::thread connector([&]() { auto val = _socket->ConnectAsync(hostName, str_port, SocketProtectionLevel::PlainSocket); task<void>(val).then([this,&cond_var] (task<void> previousTask) { std::unique_lock<std::mutex> lock(_mx); try { previousTask.get(); Print("Connection opened"); _state = 1; } catch (Platform::Exception^ exception) { Print("Error: failed to connect."); _state = -1; } if( _state == 1 ) { // start listening auto reader = ref new DataReader(_socket->InputStream); reader->InputStreamOptions = InputStreamOptions::Partial; //reader->ByteOrder = Windows::Storage::Streams::ByteOrder::LittleEndian; _writer = ref new DataWriter(_socket->OutputStream); ReceiveLoop(reader,_socket); } cond_var.notify_one(); }); }); connector.join(); std::unique_lock<std::mutex> lock(_mx); cond_var.wait(lock); } catch (ThrowableObject* exception) { Print("Error: Invalid host name."); _state = -1; } return _state == 1; } int BBTcpSocket::ReadAvail(){ std::lock_guard<std::mutex> lock(_mx); if( _state!=1 ) return 0; return _cb.Length(); } int BBTcpSocket::WriteAvail(){ std::lock_guard<std::mutex> lock(_mx); if( _state!=1 ) return 0; return 0; } int BBTcpSocket::State(){ std::lock_guard<std::mutex> lock(_mx); return _state; } int BBTcpSocket::Eof(){ std::lock_guard<std::mutex> lock(_mx); if( _state>=0 ) return _state==2; return -1; } void BBTcpSocket::Close(){ if( _socket==nullptr ) return; if( _state==1 ) _state=2; try { delete _socket; } catch(Platform::Exception^ exception) { } _socket=nullptr; } int BBTcpSocket::Read( BBDataBuffer *buffer,int offset,int count ){ std::lock_guard<std::mutex> lock(_mx); if( _state!=1 ) return 0; int i = 0; for (i = 0; i < count; ++i) { if (!_cb.IsEmty() ) { buffer->PokeByte(offset+i,_cb.Read()); } } //int n=_cb.Read( (byte*)buffer->WritePointer(offset), count); if( i>0 || (i==0 && count==0) ) return i; _state=(i==0) ? 2 : -1; return 0; } int BBTcpSocket::Write( BBDataBuffer *buffer,int offset,int count ){ std::lock_guard<std::mutex> lock(_mx); if( _state!=1 ) return 0; _writer->WriteBuffer(CreateBuffer( buffer->Length(), (byte*)buffer->ReadPointer())); try { task<unsigned int>(_writer->StoreAsync()) .then([this] (task<unsigned int> writeTask) { try { writeTask.get(); } catch (Platform::Exception^ exception) { Print("Send Error."); // Send failed with error: " + exception->Message } }); } catch(Platform::ObjectDisposedException ^ ex) { // error } return 0; } IBuffer^ BBTcpSocket::CreateBuffer(UINT32 cbBytes, byte* ptr){ if( _nativeBuffer == nullptr ){ Microsoft::WRL::MakeAndInitialize<NativeBuffer>(&_nativeBuffer, cbBytes, ptr); } else{ _nativeBuffer->RuntimeClassInitialize(cbBytes, ptr); } auto iinspectable = (IInspectable*)reinterpret_cast<IInspectable*>(_nativeBuffer.Get()); IBuffer^ buffer = reinterpret_cast<IBuffer^>(iinspectable); return buffer; } void BBTcpSocket::ReceiveLoop(DataReader^ reader, StreamSocket^ socket) { task<unsigned int>(reader->LoadAsync(1024)).then([this, reader, socket] (unsigned int size) { try { std::lock_guard<std::mutex> lock(_mx); for( int i = 0; i < size; ++i ){ if( _cb.IsFull() ) { Print("Win8 Circular Buffer is full!"); } else { _cb.Write(reader->ReadByte()); } } } catch (Platform::Exception^ exception) { } }).then([this, reader, socket] (task<void> previousTask) { try { // Try getting all exceptions from the continuation chain above this point. previousTask.get(); // Everything went ok, so try to receive more... // The receive will continue until the stream is broken (i.e. peer closed closed the socket). ReceiveLoop(reader, socket); } catch (Platform::Exception^ exception) { // Explicitly close the socket. Close(); } catch (task_canceled&) { // Do not print anything here - this will usually happen because user closed the client socket. // Explicitly close the socket. Close(); } }); } /* static byte* GetIBufferPointer(IBuffer^ buffer) { // Cast to Object^, then to its underlying IInspectable interface. Platform::Object^ obj = buffer; Microsoft::WRL::ComPtr<IInspectable> insp(reinterpret_cast<IInspectable*>(obj)); // Query the IBufferByteAccess interface. Microsoft::WRL::ComPtr<IBufferByteAccess> bufferByteAccess; //ThrowIfFailed(insp.As(&bufferByteAccess)); // Retrieve the buffer data. byte* pixels = nullptr; //ThrowIfFailed(bufferByteAccess->Buffer(&pixels)); return pixels; } */

Ferdi (Posted 2013) [#7]

@Beaker, thanks for the SimpleInput class! @Rone, Thanks for doing that Rone. I have push your changes through. I have not tested it though. Have you tested it? And if so, did you just use "diddy.glfw.cpp"? And do you want me to add you to the repository, so you can push stuff through?

rIKmAN (Posted 2013) [#8]

This sounds great - although I don't need networking just yet this is deffo on my list to check out when the time comes around. I notice in the screenshots you have of the iOS Client that the native keyboard is on-screen - is this through native code or somehow done through Monkey? Good work! :)

Rone (Posted 2013) [#9]

@Ferdi I tested demo/pbnetworkingdemo.monkey with diddy.win8.cpp. Seems to work... I'm going to add tcpserver.win8.cpp this night...probably.

Ferdi (Posted 2013) [#10]

@Rikman - you just have to type this code in "EnableKeyboard()". It is part of Monkey. Just looked it up it is part of mojo.input. Well that is where I found it. @Rone - cool. I must not have the latest diddy then, and/or I just don't have the setup for win 8. I am still new to the whole forking etc. I used to use Rational Clearcase at work, so my brain unfortunately is wired to that - not a good thing. @All, thanks for replying. Hope it will be useful.

Salmakis (Posted 2013) [#11]

Hey, any updates on this? i mean is this be worked on? Since i really would love to use this^^

Ferdi (Posted 2013) [#12]

@Salmakis, I will get back to it right after I decked out my game. I find it very hard to switch my brain around. I liked to concentrate on one think at a time. At this time, it is working. Not very fast, but "functional". =)

Salmakis (Posted 2013) [#13]

okay i can understand that :) do you know a good documentation about how to use PushButton at all?

Ferdi (Posted 2013) [#14]

Try: https://github.com/PushButtonLabs/PBNetworking/tree/master/docs Then download index.html? I haven't read that docs myself. All the code are all in AS3, but hopefully you can try to match it to my code, let me know how it goes? You can always ask me questions. And I can try to answer it as best as possible.

Salmakis (Posted 2013) [#15]

okay i will check it out this week^^

PhillipK (Posted 2013) [#16]

Heyho! Thanks for the work you've done here (8 months ago :O) and sorry for the thread ressurection :3 But i've looked over this and i stuck a bit. I cant figure out, how to extract the needed code from the example. The example itself still works great buts its hard to read, if you are not familiar to the lib. Is there any chance for a step by step tutorial or something? Or otherwise, just a very short explanation for the used classes? The most problem at the moment: Iam just to dumb to see, where the data is sent. I cant track it up the example, so its just a magic something that works :( Anyways, i will still try to understand the exmaple on my own :)