Feature Request: sockets / networking

Monkey Forums/Monkey Programming/Feature Request: sockets / networking

Skn3	(Posted 2011) [#1]

I thought it would be good to get a discussion going.

At some point in the coming months I want to have a look at wrapping iOS networking code to use in monkey. I already had a look into this and it seems like it will be an interesting project. Now this isn't something that is uncommon, multiplayer is essential for games these days. I really think monkey could benefit if some consideration was taken on adding support for networking/sockets to the mojo framework.

Naturally html5 sockets are not very well supported but could this not be implemented so that when the browser does not support sockets, simply return null/false/error for the commands. That way at least the game designer can add in a notification to the user to "Please try in a browser that supports sockets".

I think to cover all bases there would need to be a game match-up API and socket API and then let the programmer do the rest. Maybe the html5 match-up service could use something similar to the gnet system. Naturally iOS could default to the gamecenter service...

Something like:

Local matchup := CreateGameFinder()
If FoundGameServer(matchup)
	Print "Found Game Server: "+GameServerName(matchup)
	Local Socket := CreateUDPSocket(GameServerIp(matchup),GameServerPort(matchup))
	SendInt(Socket,5)
	CloseSocket(Socket)
EndIf

It could be made more flexible by adding match-up "drivers" e.g. CreateOpenFeintGameFinder() so that future game services could be added. The CreateGameFinder() would use the default match-up driver which is specific to the target being compiled to...

This is hopefully similar to the design I will follow and I will have to fill in the blanks for platforms as I release to them. I am by no means an expert in most of the platforms so I suspect my efforts will be very slow in development and probably a bit rubbish. The trouble is I can't afford the time to support such a module and I need to partially hard-code this into my engine. So I wont be much use to the community...

What are peoples thoughts on a networking API, how should it work? should it be an official module? What are monkeys plans for such a module?

Tibit

(Posted 2011) [#2]

I will very much need this in the upcoming months as well.

However I do not want a gnet system. I have not intention to offend Mark or anyone else, but since I built a network library in BlitzMax I know from experience that having bare socket-functionality is enough.

I don't know how many countless e-mail I got from users who could not get gnet to work, and I myself had problem with it. And I have found that high-level libraries can be nice, they do also limit you a lot.

So I much prefer if monkey implements just the essentials socket functions. SendTo/Recv UDP and for TCP send/recv. The closer to the bare minimum we can get the better performance and usability will networklibraries built upon it be.

My view is this. Yes Monkey NEEDS networking - but only very simple stuff.

Skn3, if Mark does not want to add networking basics to monkey in the foreseeable future then maybe we can work together on this? I can provide an SVN server. I'm happy to share my code public if we do this as some kind of community project.

TeaBoy

(Posted 2011) [#3]

Yup, PLEASE :o) Monkey NEEEEEEEEEEEEEEEDS this feature!

Tibit

(Posted 2011) [#4]

Just out of curiosity, what are you specifically going to use the networking for?

I'll use it primarily for remote updating of in-game stuff from my server.

FlameDuck

(Posted 2011) [#5]

Two things.

1) I agree with Tibit that the game-matching is a bad thing to include in your network, unless you want to foot the bill for the infrastructure. You can always use a third party system if you really need it, and if you depend on (for instance) a PHP/MySQL solution, that's not going to work at all on my Google App Engine/BigTable setup.

2) For the network side of thing, take a look at Twisted. While I realize porting that beast to Monkey would be an astronomical (if not impossible) task, you should definitely be inspired by how it works. I mean a library where writing a (simple) server is 20 lines of code? Yes please. I'll take 2.

Xaron

(Posted 2011) [#6]

That would be awesome!

Actually I already started a network module: http://www.monkeycoder.co.nz/Community/posts.php?topic=725

Maybe we could just integrate everything there? I'd like to create some socket communication as well, which won't be available for all targets of course.

Skn3	(Posted 2011) [#7]

Hey Tibit I would love to say yes but I am hella busy and I think it would unavoidably eat up time in development. I would be happy to try and contribute some code/knowledge/discussions as I go along but I don't think I could be a founder of the project.

1) I agree with Tibit that the game-matching is a bad thing to include in your network, unless you want to foot the bill for the infrastructure. You can always use a third party system if you really need it, and if you depend on (for instance) a PHP/MySQL solution, that's not going to work at all on my Google App Engine/BigTable setup.

Yeah it would most definitely be a bad idea to rely on a gnet system as the primary game service, however it would be a good solution for html5. As long as there was a flexible way to add new services that is...

Xaron, aside from plans etc have you got anything running with sockets? Would be interesting to see your progress.

Xaron

(Posted 2011) [#8]

To be honest my only experience with socket programming is using C++ and Windows. Can't be so different on other platforms though. ;)

MikeHart

(Posted 2011) [#9]

If someone is bored, maybe they could create a module for using Pubnub.

Seems that this could be easily ported/imported.

http://www.pubnub.com

Skn3	(Posted 2011) [#10]

Well I started having a play around today on my lunch break with html5. I have not done any part of the sockets itself, instead was experimenting with how the data could be handled.

Just playing around for now so its nothing serious! I have been looking at the websockets support:
https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket#send%28%29

There is not really a sensible way to handle data yet in javascript. There is a ArrayBuffer object in javascript but that has fixed width (you wouldn't be able to mix bytes, ints and floats in one buffer). There is a DataView object but this is not supported very well so finally the solution is to manage the packing into a string.

I had started writing a binary packer but found a nice one already done via the interweb!

sockets.monkey

' Module skn3.sockets
'
' Copyright 2011 Jonathan Pittock, all rights reserved.
' No warranty implied; use at your own risk.

#If TARGET<>"html5"
#Error "Mojo is not currently available for target '${TARGET}'."
#End

'import the native code please :D
Import "native/sockets.${TARGET}.${LANG}"

'there finished

Extern

Class Packet="Skn3SocketPacket"
	Method GetLength:Int()
	Method ToString:String()
	
	Method AddByte:Bool(value:Int)
	Method AddShort:Bool(value:Int)
	Method AddInt:Bool(value:Int)
	Method AddFloat:Bool(value:Float)
	Method AddString:Bool(value:String)
End

Function CreatePacket:Packet(size:Int) = "Skn3SocketCreatePacket"

Public

Function Main()
	Local packet := CreatePacket(1024)
	packet.AddByte(255)
	packet.AddByte(128)
	packet.AddByte(64)
	packet.AddShort(12345)
	packet.AddInt(1111234511)
	packet.AddFloat(1.2345)
	
	Print packet.ToString()
End

native/sockets.html.js

// --- dependancies ---
//+ Jonas Raoni Soares Silva
//@ <a href="http://jsfromhell.com/classes/binary-parser" target="_blank">http://jsfromhell.com/classes/binary-parser</a> [v1.0]
BinaryParser = function( bigEndian, allowExceptions ){
	this.bigEndian = bigEndian;
	this.allowExceptions = allowExceptions;
};
with( { p: BinaryParser.prototype } ){
	with( {p: ( p.Buffer = function( bigEndian, buffer ){ this.bigEndian = bigEndian || 0; this.buffer = []; this.setBuffer( buffer ); } ).prototype } ){
		p.setBuffer = function( data ){
			if( data ){
				for( var l, i = l = data.length, b = this.buffer = new Array( l ); i; b[l - i] = data.charCodeAt( --i ) );
				this.bigEndian && b.reverse();
			}
		};
		p.hasNeededBits = function( neededBits ){
			return this.buffer.length >= -( -neededBits >> 3 );
		};
		p.checkBuffer = function( neededBits ){
			if( !this.hasNeededBits( neededBits ) )
				throw new Error( "checkBuffer::missing bytes" );
		};
		p.readBits = function( start, length ){
			//shl fix: Henri Torgemane ~1996 (compressed by Jonas Raoni)
			function shl( a, b ){
				for( ; b--; a = ( ( a %= 0x7fffffff + 1 ) & 0x40000000 ) == 0x40000000 ? a * 2 : ( a - 0x40000000 ) * 2 + 0x7fffffff + 1 );
				return a;
			}
			if( start < 0 || length <= 0 )
				return 0;
			this.checkBuffer( start + length );
			for( var offsetLeft, offsetRight = start % 8, curByte = this.buffer.length - ( start >> 3 ) - 1, lastByte = this.buffer.length + ( -( start + length ) >> 3 ), diff = curByte - lastByte, sum = ( ( this.buffer[ curByte ] >> offsetRight ) & ( ( 1 << ( diff ? 8 - offsetRight : length ) ) - 1 ) ) + ( diff && ( offsetLeft = ( start + length ) % 8 ) ? ( this.buffer[ lastByte++ ] & ( ( 1 << offsetLeft ) - 1 ) ) << ( diff-- << 3 ) - offsetRight : 0 ); diff; sum += shl( this.buffer[ lastByte++ ], ( diff-- << 3 ) - offsetRight ) );
			return sum;
		};
	}
	p.warn = function( msg ){
		if( this.allowExceptions )
			throw new Error( msg );
		return 1;
	};
	p.decodeFloat = function( data, precisionBits, exponentBits ){
		var b = new this.Buffer( this.bigEndian, data );
		b.checkBuffer( precisionBits + exponentBits + 1 );
		var bias = Math.pow( 2, exponentBits - 1 ) - 1, signal = b.readBits( precisionBits + exponentBits, 1 ), exponent = b.readBits( precisionBits, exponentBits ), significand = 0,
		divisor = 2, curByte = b.buffer.length + ( -precisionBits >> 3 ) - 1;
		do
			for( var byteValue = b.buffer[ ++curByte ], startBit = precisionBits % 8 || 8, mask = 1 << startBit; mask >>= 1; ( byteValue & mask ) && ( significand += 1 / divisor ), divisor *= 2 );
		while( precisionBits -= startBit );
		return exponent == ( bias << 1 ) + 1 ? significand ? NaN : signal ? -Infinity : +Infinity : ( 1 + signal * -2 ) * ( exponent || significand ? !exponent ? Math.pow( 2, -bias + 1 ) * significand : Math.pow( 2, exponent - bias ) * ( 1 + significand ) : 0 );
	};
	p.decodeInt = function( data, bits, signed ){
		var b = new this.Buffer( this.bigEndian, data ), x = b.readBits( 0, bits ), max = Math.pow( 2, bits );
		return signed && x >= max / 2 ? x - max : x;
	};
	p.encodeFloat = function( data, precisionBits, exponentBits ){
		var bias = Math.pow( 2, exponentBits - 1 ) - 1, minExp = -bias + 1, maxExp = bias, minUnnormExp = minExp - precisionBits,
		status = isNaN( n = parseFloat( data ) ) || n == -Infinity || n == +Infinity ? n : 0,
		exp = 0, len = 2 * bias + 1 + precisionBits + 3, bin = new Array( len ),
		signal = ( n = status !== 0 ? 0 : n ) < 0, n = Math.abs( n ), intPart = Math.floor( n ), floatPart = n - intPart,
		i, lastBit, rounded, j, result;
		for( i = len; i; bin[--i] = 0 );
		for( i = bias + 2; intPart && i; bin[--i] = intPart % 2, intPart = Math.floor( intPart / 2 ) );
		for( i = bias + 1; floatPart > 0 && i; ( bin[++i] = ( ( floatPart *= 2 ) >= 1 ) - 0 ) && --floatPart );
		for( i = -1; ++i < len && !bin[i]; );
		if( bin[( lastBit = precisionBits - 1 + ( i = ( exp = bias + 1 - i ) >= minExp && exp <= maxExp ? i + 1 : bias + 1 - ( exp = minExp - 1 ) ) ) + 1] ){
			if( !( rounded = bin[lastBit] ) )
				for( j = lastBit + 2; !rounded && j < len; rounded = bin[j++] );
			for( j = lastBit + 1; rounded && --j >= 0; ( bin[j] = !bin[j] - 0 ) && ( rounded = 0 ) );
		}
		for( i = i - 2 < 0 ? -1 : i - 3; ++i < len && !bin[i]; );
		if( ( exp = bias + 1 - i ) >= minExp && exp <= maxExp )
			++i;
		else if( exp < minExp ){
			exp != bias + 1 - len && exp < minUnnormExp && this.warn( "encodeFloat::float underflow" );
			i = bias + 1 - ( exp = minExp - 1 );
		}
		if( intPart || status !== 0 ){
			this.warn( intPart ? "encodeFloat::float overflow" : "encodeFloat::" + status );
			exp = maxExp + 1;
			i = bias + 2;
			if( status == -Infinity )
				signal = 1;
			else if( isNaN( status ) )
				bin[i] = 1;
		}
		for( n = Math.abs( exp + bias ), j = exponentBits + 1, result = ""; --j; result = ( n % 2 ) + result, n = n >>= 1 );
		for( n = 0, j = 0, i = ( result = ( signal ? "1" : "0" ) + result + bin.slice( i, i + precisionBits ).join( "" ) ).length, r = []; i; j = ( j + 1 ) % 8 ){
			n += ( 1 << j ) * result.charAt( --i );
			if( j == 7 ){
				r[r.length] = String.fromCharCode( n );
				n = 0;
			}
		}
		r[r.length] = n ? String.fromCharCode( n ) : "";
		return ( this.bigEndian ? r.reverse() : r ).join( "" );
	};
	p.encodeInt = function( data, bits, signed ){
		var max = Math.pow( 2, bits );
		( data >= max || data < -( max >> 1 ) ) && this.warn( "encodeInt::overflow" ) && ( data = 0 );
		data < 0 && ( data += max );
		for( var r = []; data; r[r.length] = String.fromCharCode( data % 256 ), data = Math.floor( data / 256 ) );
		for( bits = -( -bits >> 3 ) - r.length; bits--; r[r.length] = "\0" );
		return ( this.bigEndian ? r.reverse() : r ).join( "" );
	};
	p.toSmall    = function( data ){ return this.decodeInt( data,  8, true  ); };
	p.fromSmall  = function( data ){ return this.encodeInt( data,  8, true  ); };
	p.toByte     = function( data ){ return this.decodeInt( data,  8, false ); };
	p.fromByte   = function( data ){ return this.encodeInt( data,  8, false ); };
	p.toShort    = function( data ){ return this.decodeInt( data, 16, true  ); };
	p.fromShort  = function( data ){ return this.encodeInt( data, 16, true  ); };
	p.toWord     = function( data ){ return this.decodeInt( data, 16, false ); };
	p.fromWord   = function( data ){ return this.encodeInt( data, 16, false ); };
	p.toInt      = function( data ){ return this.decodeInt( data, 32, true  ); };
	p.fromInt    = function( data ){ return this.encodeInt( data, 32, true  ); };
	p.toDWord    = function( data ){ return this.decodeInt( data, 32, false ); };
	p.fromDWord  = function( data ){ return this.encodeInt( data, 32, false ); };
	p.toFloat    = function( data ){ return this.decodeFloat( data, 23, 8   ); };
	p.fromFloat  = function( data ){ return this.encodeFloat( data, 23, 8   ); };
	p.toDouble   = function( data ){ return this.decodeFloat( data, 52, 11  ); };
	p.fromDouble = function( data ){ return this.encodeFloat( data, 52, 11  ); };
}

// --- globals ---
var globalBinaryParser = undefined;

// --- class: socket packet ---
function Skn3SocketPacket(size) {
	//create teh binary parser if it has not already been created
	if (typeof globalBinaryParser == "undefined") { globalBinaryParser = new BinaryParser(false,true); }
	
	//setup fields
	this.length = 0;
	this.size = size;
	this.buffer = "";
}

function Skn3SocketCreatePacket(size) {
	return new Skn3SocketPacket(size);
}

	Skn3SocketPacket.prototype.GetLength = function() {
		// --- get the length of the written data ---
		return this.buffer.length;
	}
	
	Skn3SocketPacket.prototype.ToString = function() {
		return this.buffer;
	}
	
	Skn3SocketPacket.prototype.AddByte = function(value) {
		// --- add a byte to the packet ---
		if (this.length+1 <= this.size) {
			this.buffer += globalBinaryParser.fromByte(value);
			this.length += 1;
			return true;
		}
	}
	
	Skn3SocketPacket.prototype.AddShort = function(value) {
		// --- add a short to the packet ---
		if (this.length+2 < this.size) {
			this.buffer += globalBinaryParser.fromShort(value);
			this.length += 2;
			return true;
		}
	}
	
	Skn3SocketPacket.prototype.AddInt = function(value) {
		// --- add a int to the packet ---
		if (this.length+4 < this.size) {
			this.buffer += globalBinaryParser.fromInt(value);
			this.length += 4;
			return true;
		}
	}
	
	Skn3SocketPacket.prototype.AddFloat = function(value) {
		// --- add a float to the packet ---
		if (this.length+4 < this.size) {
			this.buffer += globalBinaryParser.fromFloat(value);
			this.length += 4;
			return true;
		}
	}
	
	Skn3SocketPacket.prototype.AddString = function(value) {
		// --- add a string to the packet ---
		//this.buffer += globalBinaryParser.fromShort(value);
	}

Tibit

(Posted 2011) [#11]

Cool!

Good start :)

Is the packing done using some specific standard?

Since the only thing to do next then is to be able to unpack the data (and pack it) on all other platforms. I know .Net has built-in functions for all those toByte toInt and so on.

It seems to me at first look that packing into a byteArray (using the same means) would be better than using strings since they are often 16bit and that can mean wierdness and inconsitencies depending on language.

However with that said, I have some awesome bitPacking functions from TNet in BlitzMax that I should thank Koriolis for.

And converting these to Monkey would mean that if we ONLY get string-sending to work across all platforms then we will be able to pack anything (int,float,and so on) into that string if that makes sense?

It is the String to ByteArray and ByteArray to String conversions that should be the hardest part of that?

Also another note, html5 sockets are like awesome, however unlike other sockets they require a "initialization" procedure. This needs to be taken into account on the serverside of things as well.

I was also thinking that it might become popular to have the server in PHP, so maybe we should consider that as well? The problem there is only that Monkey does not built to php yet ;)

Skn3	(Posted 2011) [#12]

The packing is done using standards as far as I can see. The library supports little/big endian and also specific variable types eg short, long, float, etc

It would probably be sensible to convert the code in the binary packer to write to a uint8 ArrayBuffer so I guess that is the next step. Once that is done it would be possible to just deal with sending data via packets.. (banks) in the same way other blitz languages do things. Avoiding strings on certain targets is a must!

Also another note, html5 sockets are like awesome, however unlike other sockets they require a "initialization" procedure. This needs to be taken into account on the serverside of things as well.

How do you mean? I have only read the documentation of the api and not really tried to get anything running yet.

A php server would be a bit mental and awesome :D as would a monkey php target ;)

Tibit

(Posted 2011) [#13]

I'd go with a Socket-Class, and my suggestion is that it has a SocketSettings Property which the user can optionally set with data such as initialization data for html5.

Hm.. You have a good idea how we can write a simple test to verify bitPacking rutines across platforms?

Here is a html5 socket example. I haven't tested it but at first glance it looked promising.

* http://net.tutsplus.com/tutorials/javascript-ajax/start-using-html5-websockets-today/

AND

* http://www.codeproject.com/KB/HTML/Web-Socket-in-Essence.aspx

This is made for a php server, but just replace the php-server with an IP and you should be set!

Websockets are so awesome I so want them to be a reality among the larger part of the internet population.

Skn3	(Posted 2011) [#14]

Ah I see, there is no peer 2 peer communication with websockets yet. Interesting that it is so limited. It wouldn't be that hard to write a router. I understand what you were referring to now with the php server!

It could also be a possibility to embed a flash movie to deal with network and let it communicate to javascript. Although that is a bit nasty.

What do you think the best course of action would be. Ignore html5 sockets for now, use flash, write/port a websocket server in blitzmax or use something like node.js?

Tibit

(Posted 2011) [#15]

I think most times you want a server-client solution, unless the concern i latency. For P2P then going bluetooth or built-in wifi is probably a better bet for low latency.

btw talked to the founder of Atari last night, Nolan Bushnell. And He was sure that picoNets (small local wireless networks) will be huge in the future.

I buy that theory since if one game comes out that is popular - and you can play it locally (realtime) with anyone who has a smartphone, its a viral spread setup :)

Playniax

(Posted 2011) [#16]

Nice how you just casually slipped that in, 'btw talked to the founder of Atari last night, Nolan Bushnell.'

He is your neighbour or what?

Just kidding, no harm intended :)

Tibit

(Posted 2011) [#17]

He was in town, GameConference, he is pretty cool, gave an awesome keynote ;)

I actually asked him about it, and we discussed when it would happen. Do we have to wait until there is only one ubiquitous platform or could it work by doing like we are doing on here, cross-platform stuff.

Kieryn

(Posted 2011) [#18]

Any updates to this? I'm new to Monkey and http requests were one of the first things I was looking to try out... how else can you take advantage of web services etc?

Xaron

(Posted 2011) [#19]

I work on that atm... :)

Tibit

(Posted 2011) [#20]

Xaron, please tell more, also do you got anything to share perhaps?

Atm I don't want to start wrapping native code, but I'm very happy to get going building simple networking and testing once there are some raw socket commands.

Skn3	(Posted 2011) [#21]

No update from me for socket stuff I'm afraid!

Tibit

(Posted 2011) [#22]

Xaron, you got an update?

I started building a TCP Server in C# to test against but also to start working on an API that will work across all platforms.

I was thinking that send/receive use byte arrays or strings. So you can send a byte-array or a string, and you can receive a byte-array or a string. Obviously the byte-array allows you to pack and unpack stuff however you want. The String stuff does a simple string conversion and also allows you to pack data into a string-format.

Here is some progress on the Interface Side for TCP. It could look like this (just pseudo code):

TCP.StartServer:TCPServer( port:int, networkInterfaceID:int = 0 )

'Create and Listen to a post is as simple as this:
Local myServer:TCPServer = TCP.StartServer( 5555 )

in OnUpdate:

If myServer.Pending
   Local newConnection:TCPConnection = myServer.AcceptConnection()
   If newConnection.Availible
      newConnection.Read() 'Read to buffer
      Local byteArray:byte[] = newConnection.ReadBuffer
      Local text:string = newConnection.ReadString
   End
End

In this case it is using blocking sockets, however it does not block anything since we check Pending and Available before calling Accept or Read, and when we do we only read the amount we have - so there is no waiting!

Unless someone intends to host a MMO server on their iPad I strongly suspect a synchronous approach like this will be simpler, easier to code, easier to debug, easier to use and in some cases faster, yet might not be supported on all platforms.

UDP will be slightly different due to not requiring Accept, however TCP will require more work than this on a higher level for normal game traffic due to it being a stream protocol - packets are never guaranteed to be complete - they can be cutoff anywhere. However I would not suggest adding a check for this in the lowest level of the library since if someone ever is going to use a server on another platform - then they will be dependent on our protocol.

Good to hear comments on where people here wants and what kind of support in sockets you seek. We'll make this a free-community module.

Xaron

(Posted 2012) [#23]

No update yet, my time is limited but it becomes more and more urgent for me as well so I HAVE to implement more networking, lol.

So what I'm on is: Bluetooth, Socket communication (TCP) for Android, iPhone and Windows Phone 7.

Tibit

(Posted 2012) [#24]

Xaron, if you send me what you have - even if just the interface - names of functions/methods & classes - I'll see if I can help in any way!

This project has so big potential, multiplayer on mobile is the future imo :)