More modules!

Monkey Archive Forums/Monkey Discussion/More modules!

marksibly(Posted 2012) [#1]
Hi,

Ok, there are a few new modules currently under development...


* databuffer (glfw/stdcpp/ios/android/html5)

An updated version of the opengl databuffer, including DataBuffer.Load.

* stream

An abstract stream class, much like bmx's. Can read/write databuffers, bytes, shorts, int, floats.

* datastream (glfw/stdcpp/ios/android/html5)

Wraps a databuffer for streaming data in/out of a databuffer.

* filestream (glfw/stdcpp/ios/android)

For reading/writing files. Currently can only read assets on android, but I believe it's possible to store assets in the 'regular' filesystem (?) so if I get that going you'll be able to access the whole filesystem.

* tcpstream (glfw/stdcpp/ios/android)

For creating client socket streams.

* filesystem( glfw/stdcpp/ios/android)

For examining file types, listing dirs etc.



One issue is just where these module should go in the module tree? I can think of 4 options here:

* Dump them all in the root module namespace, python style. I don't particularly like this idea (I used to), as I think the current monkey approach of having a small number of publisher/proper-noun top level modules works pretty well. It also makes it easier/cleaner to provide an 'import all' style top level module.

* Dump them in the monkey module and auto-import them all, ala lang, list etc. I don't like this idea much either, as it dumps a bunch of new identifiers into the global namespace. You can get around this using module paths, but it's still nasty...

* Dump them in the monkey module and NOT auto-import them. This is better, but I think the monkey modules should only contain modules that 'work everywhere', and that is not gonna be the case with this stuff.

* Dump them in a whole new top-level module, say, 'brl'. 'Import brl' can perform the usual lazy import everything behavior, but you'd also be able to selectively import stuff. This is my current favorite idea, and I now think maybe all non-essential monkey modules (wrt: translators, which is probably only monkey.lang and monkey.math) like list, map, stack etc should be moved here...eventually...?

Also, a few notes on target compatibility:

* None of these work on XNA. There are technical reaons for this (mainly the fact that XNA doesn't seem to be able to do databuffers) but I also think that XNA is reaching the end of the road. I'll continue to support XNA and improve it where possible, but I think my time would be better spent adding this stuff to a win8 target.

* None of these work on Flash. DataBuffers are possible and I'll probably add them, but Flash apps are generally hosted remotely, so local filesystem access is redundant unless you're writing an 'installable' Flash app. This is certainly possible, but I think the time would be better spent improving other installable targets such as GLFW, win8 etc.

* HTML5 has no filesystem/file support. There are new FileReader etc objects for dealing with local files in JS but, like Flash, I think HTML5 should remain a 'remotely hosted' target (logically anyway - browsers can still install/cache HTML5 apps with a manifest). There are also technical issues with synchronously reading files in JS - ie: you can't! The file IO objects all operate asynchronously, which means 'read' actually means 'start reading and call this function when finished'. This is similar to image loading issues in HTML5, and is probably something that will need to be addressed eventually, but I don't want to have to put ALL targets through the hassle of having to declare callbacks or something just for the sake of HTML5.

* HTML5 has no socket support yet. There's the WebSockets API (which I had a hack at a while back with limited success - (could't find a functioning server!) but this will probably entail having to write a custom server that recognizes the WebSocket protocol (which I think is just HTTP on a different port - but still...). Also, it's likely there will be synchronicity problems with WebSockets, ie: you wont be able to just read them, you'll have to start a read operation and (somehow) get notified when it's finished. This is certinaly doable, but the interface probably wont look like the standard socket interface. I should still be able to hack HttpGet and HttpPost functions though...

That's a fair few 'gotchas', but that was always going to be the case when it came to this stuff.


silentshark(Posted 2012) [#2]
Good stuff - mark, I can see this helping open up all kinds of new possibilities.

Btw, Did you ever get any further on an in game ad module that was mentioned a whle back? http://www.monkeycoder.co.nz/Community/posts.php?topic=1652#27878

I appreciate that some of the guys have done some sterling work crafting their own, but things like ads on ios still remain a pain. Easier in game ads would be a big monkey selling point IMHO.


jondecker76(Posted 2012) [#3]
These all look great!

I do have a suggestion though, as I'm not sure what your scope of "local filesystem" is. In the case of flash (or any other target that is hosted remotely) there is still great value in having access to the remote filesystem. I would use monkey for my web apps if I could, but things like this and the inability to dynamically load images is what is holding it back from this market.

Still, all good news - thanks for the update


marksibly(Posted 2012) [#4]
Hi,

> but things like ads on ios still remain a pain.

I agree, but all the ad systems all seem to require low-level tinkering with views/view-controllers etc which makes them very hard to modularize, since these objects are effectively 'singletons'.

I guess it could theoretically be done with, say separate view/view-controller etc. modules, but it'd be a hell of a lot of work and really require the design of 'native APIs' something I really didn't/don't want to attempt (I briefly tried with Mojo - gave up pretty quickly).

So I think ad systems really need to be added at the target level, and I can only think of 2 practical ways to do this.

* Create a 'mega target' that contains ALL possible ad systems and kludges in the correct one at build time based on an app config setting.

* Add some kind of 'subtarget' system, where multiple target dirs can share the same target build code. This way, you could have multiple ios targets, eg: ios_adwords and ios_macads or something, each with customized view controller code and project file.

But, I haven't yet implemented or even tried a Monkey ad system so please correct me if I'm missing something here...

> In the case of flash (or any other target that is hosted remotely) there is still great value in having access to the remote filesystem.

Depends what you mean by 'filesystem', 'access' and 'the'!

By filesystem access, I'm really talking able the ability to create/read/write local files that the app *doesn't* know about - ie: files the app creates at runtime, or files outside the data/ dir etc. In fact, on Android the filesystem can't even see stuff in the data/ dir as assets are (currently anyway) embedded in the apk. Not sure about ios yet.

Everything in data/ is stuff the app already knows about at compile time, so doesn't really need a filesystem to access it. I guess a 'virtual filesystem' that maps to data/ might be useful, but that's a different problem.

If you're talking about accessing binary data, then a Flash DataBuffer should be able to help - I hope, haven't tried it yet.

Again though, I'm not sure what can been done in Flash (eg: perhaps you create files on the server? which API then?) so correct me if I'm wrong.


AdamRedwoods(Posted 2012) [#5]

Depends what you mean by 'filesystem', 'access' and 'the'!

i haven't noticed if this ability is in there yet-- but i think the value of a Flash filesystem is to load files asynchronously and therefore to have a preloader for all the graphic files.

This is something I am noticing that is a need lately for most targets, except for maybe glfw, since synchronous is relatively fast.

There are also technical issues with synchronously reading files in JS - ie: you can't!

this is what is slowing down miniB3D, but I made a preloader to handle this issue. again, i think there's value on preloaders-- which we can make ourselves, we just need the image file loading utils.

but I also think that XNA is reaching the end of the road.

agreed, but if you need ideas, look at Rone's miniB3D handling of databuffer, or the older one i have in there on github for xna. seems to work.


Tibit(Posted 2012) [#6]
Cool module update! :)

Use Case 1: Loading resources async from a server on web and mobile games is getting more useful these days, especially if the game has a virtual shop. On larger games with loads of graphics this is of even greater importance. It reduces size, reduces initial loading time and allows OnTheFly updates.

Use Case 2: Monetize Monkey Games using Ads. Some Ads or libs are super easy to implement, others not so. Those others are because they want the viewcontroller or they want the user to attach delegates and initialize it in a particular place. However this can be circumvented by creating an custom class in ObjectiveC that wraps this. The problem then becomes that essential code are part of the xcode project part of things, and not in a sharable monkey module. Chartboost for example just updated their API to NOT use the viewcontroller - so this problem might solve itself over time if that is a sign that API simplicity rules.

Networking: Async read/write. I see Async using a callback as a benefit on all platforms? A application freeze is usually no good even on windows when reading (for ex) a big map before the next level starts, or maybe I misunderstood that?

I would assume creating a Databuffer struct/class in C# would be quite simple, I don't see why it need to be XNA specific?

> * Dump them in a whole new top-level module, say, 'brl'.
I like this the best. I have found myself using the diddy ArrayList more and more latley instead of the Monkey List. In some weird logical way it do make sense that the non-language parts are separate from the Monkey core.


Gerry Quinn(Posted 2012) [#7]
> * Dump them in a whole new top-level module, say, 'brl'.

I also prefer this. And adding 'import brl' to a file is no real imposition any more than adding 'import mojo'.


slenkar(Posted 2012) [#8]
I see Async using a callback as a benefit on all platforms?


I agree


Skn3(Posted 2012) [#9]
For websockets check out: http://cjihrig.com/blog/websockets-in-node-js-0-8-6-for-windows-7/

I also agree that a callback would be nicer, it seems more natural to respond when messages are received (instead of polling). It also means networking can be done without blocking (depending on target)... One of the major drawbacks with blitzmax networking.

Are we going to get UDP as well? It is still important for game dev.

Thinking about it you could probably implement it as an interface that acts as a message reciever for when file is loaded or message received. You could then potentially wrap this in procedural functions that keep in a loop until the callback is received and a 'finished' flag is set.

That way you give the choice to the user if they want to work procedural or event based.


DruggedBunny(Posted 2012) [#10]
I've been having a go at writing a BlitzMax-based WebSockets server because of this post... You basically do a 'handshake' over TCP and can then send different types of data back and forth in 'frames' -- it's supposed to be really quick, too.

I've got the weird "client key -> SHA-1 -> Base64" handshake working so far with some online text-based clients (in Chrome on Windows) and can receive their data... just need to decode it next as it's all messed up, but I know the data is stored in frames and not too hard to get at. Done for tonight though!


Skn3(Posted 2012) [#11]
Good stuff :D so I wonder the feasibility is to make websockets router invisible to monkey networking commands?


marksibly(Posted 2012) [#12]
Hi,

Regarding async loading, what sort of API would people like to see here? Some pseudo code would be great!

Should load commands be retrofitted to perform asynchronously, so you still get back an image or sound object, you just can't use it yet? Or should preloading returning a different object, or even no object - the path just gets mapped to a preloaded object so that LoadImage works next time you use it?

And I hope you guys aren't talking about callback based async streaming, eg: converting this...

Local count:=ReadInt()
For Local i:=0 Until count
  Local strLen:=ReadInt()
  ...read string etc...


...to an async version would be pretty nasty - you'd end up having to create a state machine that changes state on each 'read complete' event. For file formats that contain 'chunk lengths' etc, this is a pretty big problem.

In general, I think async loading would work for loading 'monolithic' objects like images, sounds, entire databuffer 'blobs' etc - but would not be practical for reading files with variable sized chunks, like the above. Of course, it may be sufficient to load a file in one big chunk, then read through it synchronously with a datastream.

The other approach to all this async stuff is to start looking at threading.

If all you're after is a 'loading' screen, this could probably be done pretty cheaply/easily already (ignoring HTML5 for now) - all native 'loading' commands could, upon entry, enable a thread that regularly invokes App.OnLoading(), and disables it upon return. This would give you some degree of synchronization - ie: OnLoading would never be called while your Monkey code was executing. It would still be possible to lockup the OnLoading thread if you don't call Load often enough though.

This could be expanded to a system with 'fibres' (ie: cooperative threads) that can yield to each other. This'd be my preferred approach in a way - with some kind of thread system it'd be possible to implement image etc preloading in pure Monkey, plus a whole lot more of course.

HTML5 is slowly getting thread support - it already has Worker threads, although these can't access the DOM/canvas/images/sounds etc yet, so aren't useful for async loading problems right now. These threads can however use new FileReadSync objects though which solves the problem above.

As for the brl modules, I've decided to hold back on the filesystem and filestream modules for now - I really need to expand the loading commands to handle external files first. So for now, you're still stuck with only being able to load things in data/.

The good news is that databuffers are now working on all targets (thanks Adam/Rone!), can be loaded synchronously (haven't tested on psm yet) and read stream-style using a datastream. These files still need to be in your data/ dir, and there's a new BINARY_FILES data file filter to flag which files can be loaded into a databuffer (currently defaults t *.bin|*.dat).


slenkar(Posted 2012) [#13]
how about using the async loading functions that are native to html5 and flash (dont know about smartphones), and faking it on the desktop targets

Im talking about images and sounds, not variable chunk files


marksibly(Posted 2012) [#14]
Hi,

That's sort of the first option I mentioned.

One way to do this would be to add something like...

Interface ImageListener
    Method ImageLoaded:Void( image:Image,success:Bool )
End


...to mojo graphics, and add an ImageListener param to LoadImage. Ditto for LoadSound/LoadDataBuffer. Perhaps this could be generalized with a Loadable class and LoadableListener interface?

Should ImageLoaded be allowed to fire at any time? This might cause synchronization problems if it goes off while your code is doing something messy - perhaps the system should guarantee this is only called when your app is idle, ie: has returned from OnUpdate/OnRender etc? Otherwise, confusion arises over which bits of your code need to be re-entrant, as I recently discovered with Qt...


Beaker(Posted 2012) [#15]
I'd definitely vote for the coroutine/yield direction if possible. I'm not a huge fan of the Flash way of loading async images, but it would be better than nothing. The thread method would also be useful for network commands etc. I'd recommend taking a look at the way Unity does coroutines (at least in c#, not the horrid Unityscript), as I think it's got a good balance between power and simplicity.

Examples:
yield return new WaitForSeconds(2);
yield return new WaitForEndOfFrame();

WWW w = new WWW();
yield return w;


ziggy(Posted 2012) [#16]
@Mark: Instead of the whole interfaces approach, wouldn't it be easier to have an "Status" property on the Image (and Sound) classes? I think this would be a lot simpler to implement. A simple status atribute with values like: Empty, Loading, Loaded, and Failed.

Now, if we could have delegates we could easily have a sort of optional call-back method call for the very unusual scenario when you want to perform an additional acction when the load process has ended, so you get the same functionality that with the listeners, but I think it could be a lot simpler to use.


marksibly(Posted 2012) [#17]
Hi,

Images etc could also provide a status method easily enough, but there were a couple of requests for a callback based approach and that's just what I came up with off the top of my head. The callback wouldn't be that much harder to implement anyway - all the fun stuff's gonna be in the behind-the-scenes threading...

If you wanted to skip the callback, you could just pass Null as the ImageListener to LoadImage and use a purely polled approach. The presence of the listener param could still signify an async load though, eg:

Local img:=LoadImage( "blah.png" ) 'sync load
Local img:=LoadImage( "blah.png",myapp ) 'async/callback/polled load
Local img:=LoadImage( "blah.png",Null ) 'async/polled load

I don't think the interface approach would be that hard to use - simplest case, just implement ImageListener in your App subclass and pass your app to LoadImage. Yes, delegates would allow you to wire up different images to different 'loaded' methods of the same object, but I don't think that'd be much of a win since as you say it'd probably be used reasonably rarely anyway.

Another option would be to just skip the interface and add OnImageLoaded, OnSoundLoaded etc to app - or even just a single OnLoaded (more flexible for external mods). Then, a load flag or enable functions could be used to turn async loading on/off.


Tibit(Posted 2012) [#18]
I do like the simplicity of having a State like Ziggy suggested. An interface is also a simple and a more versatile approach. At first it may be uncommon to some, but it is simple. If using the Interface approach, diddy and other modules can easily implement a State:int and probably will.

I do not think OnImageLoaded should be in App - with the exception of very simple games I fear that will complicate code and create hard to avoid dependencies, and where does it end? OnBigFileRead, OnConnect..

The interface approach can be used both by mojo and external libs, additions and improvements will probably be easier to create and share. Simplicity.

[monkeycode]' Send Image. Note: in IRL code I probably would not do new here
Local myTCPStream:Stream = new Stream("192.168.0.1", 6001)
myTCPStream.WriteFile( "blah.png", new OnImageSentCallback )

Class OnImageSentCallback implements StreamOnWriteFileCallback
Method OnWriteFileCallback:Void(file:File)
Print "Congratulations! File "+file.Name+" was successfully sent!"
End
End [/monkeycode]
This would mean I could have a class that handles operations for network, resources, and other stuff, myown and monkey intermixed:

[monkeycode]' Global is some file
Global Network:NetworkManager = new NetworkManager

'In Network Code
MonkeyTCPStream.Write( new DataBuffer("Test"), Network ) 'TCPStream.OnSendCallback
MonkeyTCPStream.Read( Network ) 'TCPStream.OnReceiveCallback

'In App Specific Network code
MyTCPStream.Send( new MyPacket("Hello world!"), Network )
MyTCPStream.ListenToMyOwnPacket( Network ) ' OnMyOwnPacketReceivedCallback

Class NetworkManager implements TCPStreamOnSend, TCPStreamOnReceive, MyOwnPacketOnSent, MyOwnPacketOnReceived

Method OnTCPStreamSend:Void(data:DataBuffer) 'TCPStreamOnSend Interface
Print "We sent "+data.Length+" bytes!"
End

Method OnTCPStreamReceive :Void(data:DataBuffer) 'TCPStreamOnReceive Interface
Print "We received "+data.Length+" bytes!"
End

Method OnMyOwnPacketSent:Void(pack:MyOwnPacket) ' MyOwnPacketOnSent Interface
Print "Congratulations! File "+file.Name+" was successfully sent!"
End

Method OnMyOwnPacketReceived:Void(pack:MyOwnPacket) 'MyOwnPacketOnReceived Interface
Print "Received "+pack.ID+" from "+pack.FromName+" containing: "+pack.Data
End
End [/monkeycode]

Any class can be used to implement the Interface, even the App class.


Samah(Posted 2012) [#19]
I implemented threading and coroutines in a subproject of Diddy, but not for HTML5, Flash, or XNA (I need to read up on threading in .NET)

http://code.google.com/p/diddy/source/browse/#svn%2Ftrunk%2Fsrc%2Fthreading


slenkar(Posted 2012) [#20]
perhaps the system should guarantee this is only called when your app is idle, ie: has returned from OnUpdate/OnRender etc


sounds reasonable


Xaron(Posted 2012) [#21]
Omg, Marc I love you. Lol. TCP streams! Any ETA on that? I can concentrate on Bluetooth in that case for my MNet module.


marksibly(Posted 2012) [#22]
Hi,

> An interface is also a simple and a more versatile approach.

Yes, definitely more flexible and I think it's the right way to go, but simple...?

I had a quick hack at rewriting this simple HTTPGet asynchronously...

Function Main()
	Local stream:=New TCPStream
	
	If stream.Connect( "www.blitzbasic.com",80 )
		Print "Connected!"
		stream.WriteLine "GET / HTTP/1.0"
		stream.WriteLine "Host: www.blitzbasic.com"
		stream.WriteLine ""
		Local n:=0
		While Not stream.Eof()
			Local line:=stream.ReadLine()
			Print line
		Wend
		stream.Close
		Print "BYE!!!!"
	Else
		Print "Failed to connect!"
	Endif
End


...and it's pretty heavy going. Reading is easy when you just want to read everything until EOF, but if you want to read X bytes but only Y bytes have arrived, or you need to detect EOL or something, things start getting a bit hairy. You have to start buffering up unread bytes and joining buffers together etc.

But it possibly just requires a bit of practice - I think I was starting to get the hang of it. I'm definitely gonna leave all the current synchronous stream stuff in there for the sake of general sanity though (and databuffer/file IO etc), and it'll be a while before I can have a good crack at this anyway. Should be interesting!


Skn3(Posted 2012) [#23]
Awesome sounds like this is gonna happen :D

Could you manipulate the data buffer manually within the interface callback?

Eg
Class MyStreamManager implements StreamListener
	Method OnDataAvailable:Buffer(data:Buffer)
		'check for end of line
		if data contains eol
			line = slice up Until EOL
			
			data = slice after EOL
			
			Return data
		Else
			Return data
		EndIf
	End
End


The default action would be to return nothing which assumes the data has been dealt with.


Tibit(Posted 2012) [#24]
Async do makes code more complex, but it can be still be only slightly so :)

Mark can you share how your async implementation looked like? And perhaps an simple non-async example with the read/write problem that becomes complex in async mode? Btw, nice browser! :)

Here is my example on the same browser using async. I used this mockup class and interfaces and to test and run:
[monkeycode]
Interface IOnConnect
Method OnConnect:Void()
End
Interface IOnConnectFail
Method OnConnectFail:Void()
End
Interface IOnClose
Method OnClose:Void()
End
Interface IOnReadLine
Method OnReadLine:Void(lineRead:String)
End
Interface IOnWriteLine
Method OnWriteLine:Void(lineWritten:String)
End

'Mockup Class for Testing purposes
Class TCPStream
' Conned Success only on port = 80
Method Connect:Bool(url:string, port:Int, onConnect:IOnConnect = null, onConnectFail:IOnConnectFail = null)
If port = 80
If onConnect Then onConnect.OnConnect()
Return true
Else
If onConnectFail Then onConnectFail.OnConnectFail()
Return false
End
End

Method WriteLine:Void(text:String, onWriteLine:IOnWriteLine = null)
If onWriteLine Then onWriteLine.OnWriteLine(text)
End

Method ReadLine:String(onReadLine:IOnReadLine = null)
testingCounter -= 1
Local mockData:String = "Reading Line " + testingCounter
If onReadLine Then onReadLine.OnReadLine(mockData)
return mockData
End

Method Eof:Bool()
Return testingCounter <= 0
End

Method Close:Void(onCloseCallback:IOnClose = null)
If onCloseCallback Then onCloseCallback.OnClose
End

Private
Field testingCounter:Int = 5 ' Only for testing
End
[/monkeycode] I wrote the code above only so I could make sure I got the same result running the sync browser above and mine async browser below.

Example Async:
[monkeycode]
Function Main()
New WebBrowser
End

Class WebBrowser implements IOnConnect, IOnConnectFail, IOnClose, IOnReadLine', IOnWriteLine

Field stream:= New TCPStream

Method New()
stream.Connect("www.blitzbasic.com", 80, Self, Self)
End

Method OnConnect:Void()
Print "Connected!"
stream.WriteLine "GET / HTTP/1.0"
stream.WriteLine "Host: www.blitzbasic.com"
stream.WriteLine ""
Local n:= 0
While Not stream.Eof()
Local line:= stream.ReadLine(Self)
Wend
stream.Close(Self)
End

Method OnReadLine:Void(line:String)
Print line
End

Method OnConnectFail:Void()
Print "Failed to connect!"
End

Method OnClose:Void()
Print "BYE!!!!"
End
End
[/monkeycode]

I do agree it is more complex, I find it still quite simple, slightly elegant, perhaps easier to test and it is optional. Or maybe I'm missing the truly painful threaded scenario in this simple web browser?


marksibly(Posted 2012) [#25]
Hi,

Well, it's a start, but it's not really asynchronous. The Connect is, but after that you go into a synchronous while loop.

Just to clarify what I think what we're trying to do here: asynchronous (aka non-blocking) IO without the use of threads - something html5/flash etc friendly. If you're assuming threads, then it's all much simpler - just do everything synchronously on a thread! Without threads, all potentially blocking Reads/Writes/Connects etc need to return immediately and use an OnComplete handler to signal completion.

[edit]Note that when I say 'without threading', I mean without threading the Monkey app - some of this stuff will have to be implemented with threads behind the scenes. The Monkey app will continue to behave as if it's single threaded.[/edit]

Anyway, your example works because your ReadLine/WriteLine are synchronously executing the callbacks. In real network code, ReadLine/WriteLine will return immediately and the callback will happen later - possibly much later.

This means a gazillion ReadLines inside the while loop may have executed before the next block of data arrives. Even if these are queued up, something will probably have to block at some point. And the While Eof() means that whatever happens inside the loop, it wont exit until the entire page has been read anyway. If something blocks in the process, everything blocks.

To execute a sequential series of reads without blocking, each read needs to be triggered by the completion of the previous read, so the while loop might end up like...

Interface IOnReadPageComplete
	Method OnReadPageComplete:Void( page:String )
End

Class WebBrowser Implements OnReadLineComplete

	'Can block, so we use a completion hander...
	Method ReadPage:Void( url:String,onComplete:IOnReadPageComplete )
		...
	End

	...eventually executes initial ReadLine to kickstart things...

	'Our new while loop!
	Method OnReadLineComplete:Void( line:String )
		If line 'empty line=Eof.
			'Received another line
			_buf.Push line	   		'push line on a StringStack
			_stream.ReadLine Self   'start reading next line.
		Else
			'Eof!
			Local page:=_buf.Join("")
			_onComplete.OnGetPageComplete( page )
		Endif
	End

	...
	
End


Note that this is a pretty easy example! HTTP1.1 is more interesting - pages are sent length/data style and the stream stays open, ie: there's no EOF.

And then there's how to implement ReadLine in the first place. Given an abstract stream class designed to be extended by native streams, say...

Interface IOnReadComplete
	Method OnReadComplete:Void( buf:DataBuffer,offset:Int,count:Int )
End

Class Stream	'Extended by native TCPStream...
	Method Read:Void( buf:DataBuffer,offset:Int,count:Int,onComplete:IOnReadComplete ) Abstract
End


...writing a StreamBuffer or something that wraps this and provides ReadInt, WriteInt, ReadLine, WriteLine etc is pretty intense.

Finally, some interesting reading:

http://www.2ality.com/2012/06/continuation-passing-style.html

It's JS specific, but covers many of the ideas behind converting sync code to async code.


marksibly(Posted 2012) [#26]
Hi,

I made some seriously good progress on this today!

For starters, I decided to put off the 'automatic' delivery of async completion events for another day - instead, for now you'll have to 'poll' via an Update() method that triggers all the callbacks. This simplified things immensely though.

Then, with the addition of a quick 'n' nasty *internal use only* thread module and several hours of head-scratching I ended up with this:



This provides a very minimal threading wrapper around the native TCPStream, that works without any data copying or buffer allocation etc - ie: all reads/writes come/go directly from/to the buffers you provide. Yes, it's very low level and might not everyone's cup of tea, but it really just represents the core of a system that can be built on.

You can queue up to 256 connect/read/write ops and they all execute in sequence in the background, notifying you each time the next one completes. It'd be easy to have different threads handling reads/writes too but I don't know if there's any point...?

The only catch is that you need to call Update() to invoke the callbacks. This theoretically introduces a bit of lag, as data may have arrived between OnUpdates but you wont be notified until the next OnUpdate when you get to call socket.Update.

But it works amazingly well! Although I've only tested on PC GLFW and Android so far...

An AsyncStreamBuffer style wrapper that does slicing and dicing of buffers would be useful and I'll probably get around to one eventually, but I really need a clearer idea of what people are planning to do with this stuff first. I don't know if there's much point in async ReadInt, WriteInt methods with this. On the other hand, ReadLine, WriteLine, ReadUntilEOF are probably useful?

In general, I think all async libs that use async components will need to be async themselves too. But this isn't such a big deal, as the complexity of dealing with the lower level components is handled by the higher level components, and the completion callbacks get simpler as they bubble up.

For example, here's a new AsyncHTTPGetter...



This works as smooth as a baby's bottom, with no glitches or hiccups unlike the old version!

Getting rid of the need for Update() will happen eventually, but it's a reasonably complex problem as it interacts with Mojo at quite a low level. For now, I think Update() is an acceptable compromise. In fact, it's quite nice to be able to control exactly when the callbacks happen.


Xaron(Posted 2012) [#27]
Awesome stuff Mark! Will that be in the next update?


Samah(Posted 2012) [#28]
Getting rid of the need for Update() will happen eventually, but it's a reasonably complex problem as it interacts with Mojo at quite a low level.

Is it possible to put as much of this as possible into the public domain? It sounds like it's very mojo-specific, which means I can't include it in monkey-ext. :(


marksibly(Posted 2012) [#29]
Hi,

Well, if you're willing to fund my lavish lifestyle, I'd be happy to give the whole thing away!

It'll be a while before I tackle this anyway. The whole app architecture needs an overhaul to make it easier to create custom targets etc and this'll probably be part of that. But I have no idea what final form it'll all take.


Tibit(Posted 2012) [#30]
This looks stellar! :D

100% support the "not threading the monkey app" since there is so much sanity to be solved in that.

Unless writing a high-performance server is async callbacks in read/write really needed? I can't directly imagine a serious use case for read/write Async callbacks, those 1/Framerate milliseconds can matter for the truly hardcore but only on high performance action games that probably won't work over high latency 3G,4G and wireless internet connections that mobile devices have today anyway! :)

I like the Update approach, it gives you the power and it is easily understood imo. I did not get how or why it should be removed? I think it is a very acceptable solution.

I assume above is threaded async, and that thread is syncronized with the main thread before OnUpdate in mojo and when I then later on call networkThing.Update() I simply get the messages in main thread, and except for a almost unnoticable latency loss of maybe 10ms everyone is happy? :)


Tibit(Posted 2012) [#31]
For use cases in a mobile game.

Http can be used to save and load data from a server - settings,analytics,ad services, and more. TCP I intend to use for sending gameplay information about the game state. Depending on game, but 10 times per second is reasonable with a TCP packet size below 500bytes.

For debugging networking it is nice to be able to get (maybe from the tcpStream) local IP,Port and Remote IP,Port. Also Monkey should where possible set nagle = false (TCP NODELAY), I think most realtime apps wants that?

Server side of things:
Maybe I should ask if an Accept() is planned as well in the future? I have found that having the server and client in the same language can be very practical, tough not needed.


marksibly(Posted 2012) [#32]
Hi,

> Unless writing a high-performance server is async callbacks in read/write really needed?

Depends what sort of quality you're after. Reading even a single byte will cause blocking if there's a network glitch, resulting in freezing/choppy gameplay. This may or may not be a problem depending on the quality of the network/server. I doubt any AAA games do any network IO synchronously though.

I'm also a bit worried about keeping things sweet for the app store too - the ios developer docs stress that you should either use UINetwork (or whatever it's called), or BSD sockets on a separate thread. I couldn't find anything in the official guidelines about networking, but I wouldn't be surprised if the testing process involves hammering the network with a bit of noise!

Also, if we want WebSockets, we don't have a choice - everything will have to be asynchronous there.

The queuing system means you basically don't have to worry about writes, so it's only reads that get a bit tricky, which makes sense since this is all about a different way of returning data.

Here's the callbacks of the httpgetter:



Not too heavy?

Basically, the read bit is just doing a ReadAll. This is something that could be placed in a wrapper class (along with IOnReadAllComplete) as could a bunch of no hassle Write methods that could have optional callbacks.

I'm sure there are more 'higher level' read ops that could reused like this - eg: HTTP1.1 chunked transfers - but I don't really know much about tcp usage beyond http and, despite asking around, no one's given me anything else to go on!

If what you're after is just plain a synchronous TCPStream, but with an async connect, this is doable. In fact, this is where I started out this morning! It works but, on my PC anyway, although the connect was instant, there was a noticeable hiccup before any data arrived. Things are probably sweet after that, but async version is 100% delay free.

> I assume above is threaded async, and that thread is syncronized with the main thread before OnUpdate in mojo

It's threaded, but it runs 'freely' - ie: there's no syncing, it's all done with a FIFO queue and 3 cursors. Basically, the Connect/Read/Write methods are adding AsyncOp objects to the queue, which is being greedily plowed through by the thread. Update simply catches up with the thread, dispatching OnComplete's as it goes. It's actually the coolest code I've written in a while!


marksibly(Posted 2012) [#33]
Hi,

> TCP I intend to use for sending gameplay information about the game state. Depending on game, but 10 times per second is reasonable with a TCP packet size below 500bytes.

What format is this data in (roughly)?

Here's a thought - would a JSON reader help?

In fact, with reflection it should be possible to do an async object loader!


Tibit(Posted 2012) [#34]
> It's actually the coolest code I've written in a while!

It actually sounds awesome as well!

> In fact, with reflection it should be possible to do an async object loader!
Wow! Yeah serialization is probably where "most time is spent" in networking. So that is a really good idea.

> What format is this data in (roughly)?
Json is very readable, and it would certainly work, but I'd lean more towards going binary in this case.

When bandwidth is of importance and I would say most web or mobile games qualify then I tend to go with reading & writing bytes/shorts/ints/strings directly to the stream using a simple packet protocol.

I was thinking in order:
1. 2bytes = short, size of packet
2. 1byte, packet type - my code knows if it is a new game event, a position update, a onFire event, and so on.
3. data - We know how to read this because of type in step 2

I naivly assume here all packets broadcast to all players - since that is enough for my current project. For games with more than 8 players in a game one might want the server to have logic for only sending to relevant receivers.

UDP already comes with Size included so there I would simply lose the first 2 bytes. And In UDP I might even "gather up" multiple packages sent the same frame in one and use a PacketType for that. In TCP I'd just push everything onto the stream.

This can easily be built upon the TCPStream, TCPSocket and also work with future UDP support, also easy to implement on a target like NodeJS.

However I think the question of using JSON for game data is a good one, for smaller games it certainly should work fine, and it might very well work in my current project! If it can be used interchangeably with a binary format (for example if using Interfaces again) it would probably help a lot in debugging.


Tibit(Posted 2012) [#35]
Note: When I talk about bandwidth it is the server I'm concerned about, not the client!


marksibly(Posted 2012) [#36]
Hi,

Actually, for game state blobs I think the best approach is probably just to use a ReadBlock/IOnReadBlockComplete (that reads an int, followed by that many bytes) and just read/write the whole game state in one hit.

You can then go through it with a DataStream synchronously.

This means your code for serializing gamestate/actors can remain pretty much the same, eg:

Class Actor
   Method Read:Void( stream:Stream )
      stream.Read x,y,state etc
   End

   Method Write:Void( stream:Stream )
      stream.Write x,y,state etc
   End
End


In fact, I think 2 new async methods solves pretty much everything, eg:

Interface IOnReadAllComplete
   Method OnReadAllComplete:Void( buffer:DataBuffer,offset:Int,count:Int... )
End

Interface IOnReadBlockComplete
   Method OnReadBlockComplete:Void( buffer:DataBuffer,offset:Int,count:Int... )
End

'new methods in AsyncStream

   'read until EOF
   Method ReadAll:Void( buffer:DataBuffer,offset:Int,onComplete:IOnRealAllComplete ) 

   'read length/databuffer
   Method ReadBlock:Void( buffer:DataBuffer,offset:Int,onComplete:IOnReadBlockComplete )



(think I prefer ReadChunk...)

Even with JSON, it's probably easier/cleaner/faster to do this, as long as the data isn't too huge.

As a bonus, the whole think can be compressed in one hit.

Or is that too easy...?


Tibit(Posted 2012) [#37]
With TCP I do not know if the entire packet has been received. So I do not want to start reading a packet until awail tells me I have the number of bytes I seek.

[This applies even to an Async TCP game server in C#]
1. Start Read 2 Bytes (not blocking)
2. (later on in callback) On 2 Bytes Read --> From stream I read a short, these 2 bytes are my virtual packet's size
3. In same callback I Start a Read of "Size" bytes.
4. (later on in callback) On "Size" bytes read --> I parse the data using stream commands

So when I do a Read I can't do a ReadAll (not for continuous gameplay traffic), I need a Read that specifiyBytes, or I can poll Awail.

And with Actors, I need to have them ordered on all clients or use an ID to know which actor an update relates to, before I read that specific package if I'm not missing something obvious :)

This should work without Reflection:
[ monkeycode ]
[monkeycode]Interface UnitData
Method ID:Int() Property
Method X:Float() Property
Method X:Float() Property
Method Direction:Float() Property
End

' Unit implements UnitData
For Local unit:Unit = Eachin UnitList
Send( unit)
Next[/monkeycode][ /monkeycode ]

Using reflection I might send all variables that does not start with _ or something.

[monkeycode]Class LocalUnit
Field EntityID:Int
Field X#, Y#
Field _test:Int
Field Important:String = "Abc"
Field _localThing:String = "test"
End
Class RemoteUnit
Field EntityID:Int
Field X#, Y#
Field Important:String = "Abc"
End[/monkeycode]

In Update I'd check all my local units (or units I can control) for any values being changed since last frame, I then send them into the stream starting with the EntityID.

OnReceive at the other client I packincoming up and using reflection I find the RemoteUnit with this EntityID and populate the data, or create if it does note exist.

That would be a simple approach, but with dead-reconing and area of interest management I'm not sure it would be that simple. Location based objects needs to be handled differently from raw data like chat. Also the most important thing would likley be delayed events.

The benefit of reflection is that for each NEW packet I only need to create an interface, or I only need to add that variable.

Ex:
Player1 clicks button.
Player1 sees a button loading animation, button is not activated yet
Player2 then gets a message Player1 pressed button, and a timestamp
Player2 calucated when this event should happen, and once that frame comes Player2 simulates a buttonPress event
Player1 assumes Player2 got the message and after a pre-determined time the button activates for Player1
And unless Player2 disconnected, player2 sees the button activated at the the same time

That is a very common scenario. With a framework around doing above should be quite simple for the end user.


marksibly(Posted 2012) [#38]
Hi,

> With TCP I do not know if the entire packet has been received. So I do not want to start reading a packet until awail tells me I have the number of bytes I seek.

You never start reading a packet though - the thread does.

Currently, the thread just does a single 'read' and then notifies you - and yes, currently it could return less than you asked for.

But it could just as easily do a full read - eg: if you ask for 16 bytes, it loops until 16 bytes are read. It doesn't matter if it blocks 'coz it's on the thread. This is probably the preferred behaviour.

Also, you'd never have to even read the length header - the ReadChunk method will do this for you internally - ie: it will handle the callback that contains the length. ReadAll is only for situations like HTTP1.0 where the connection is closed to signify 'end of data'. But in a game we want the connection open all the time so we use ReadChunk instead.

So the main game recv 'loop' might look like this:

Method OnReadChunkComplete:Void( data:DataBuffer,offset,length )
   'OK, data contains an entire message from the server!
   ProcessServerMessage data,length
   'Start receiving next message
   stream.ReadChunk data,0   'OK, next message please!
End

Method ProcessServerMessage( data:DataBuffer,length )
   If data.PeekInt(0)=GAMESTATE_UPDATE
      ...apply gamestate changes...
   ElseIf data.PeekInt(0)=PLAYER_CHAT
      ...etc...
   Endif
End


It just needs to be kicked off in OnCreate (or somewhere) with an initial ReadChunk - ie: you don't wait for data and then read it, you start reading before it arrives. After that, server messages keep getting sent to the callback because it keeps asking for them.

This is pretty much the async equivalent of

Repeat
  Local data:=ReadServerMessage()
  ProcessServerMessage data
Forever


Sending data is easier - you just wack an entire message into a DataBuffer and use SendChunk, which handles doing the WriteInt(length)/Write(data) for you so the server can read it with ReadChunk.

And...now I see why there needs to be separate read/write threads...! Otherwise, if the recv thread is waiting for data, it'll hold up any send data.

> Using reflection I might send all variables that does not start with _ or something.

IMO, you'd want to send them to a DataStream though, not the socket. Only once you've got an *entire* client message together would you send it with a single WriteChunk.

I guess there is the issue of extra lag introduced by only sending stuff once after you've packaged it up, in which case you'd have to break things down into smaller chunks and things would get a bit state-machiney. But in a lot of cases just one mega message would work fine I think.


Gerry Quinn(Posted 2012) [#39]
This sounds good to me: a nice simple model that everyone can understand.


Tibit(Posted 2012) [#40]
Ah ok Read/Write Chunk sounds exactly what I was after :)

What exactly does WriteChunk/ReadChunk do?

If I want to talk to my NodeJS game server then how do I handle the packets on that side? Or will Chunks only work if the server is in Monkey as well? (only need Accept or similar for that).

> if you ask for 16 bytes, it loops until 16 bytes are read. This is probably the preferred behaviour.

It seems either way would work pretty well since it seems I don't have to do a copy on the bytes after the read - I can just leave them in the stream and read them later? I would expect if specifying a size it loops until size recv, else I get OnReadComplete every frame there is at least 1 byte in the buffer?

There are three Scenarios.

If I expect 100 bytes and buffer has 100 bytes, np.

If I expect 100 bytes and buffer has 99 bytes, wait til next update.

If buffer has 101 bytes when I want to read 100 bytes? Can I just "leave" the 1byte alone in the stream and they will automatically be part of the next OnReadComplete? Or do I need to save them so I can process them when the remaining 99bytes of the second package arrives?

If my buffer has 100 bytes and I want to read 10 bytes 10 times in a row? Does OnReadComplete return immediately so I can handle "as much as possible" that update then?

If my buffer has 101 bytes and I want to read 10 bytes 10 times in a row and save 1 byte for next update?

DataStream <-- I think the data stream will make it very simple to handle data, but if I want my game server to run in non monkey, like nodeJS, php, plain C#? Since monkey translates easily, maybe it would be trivial to generate code for this?


Tibit(Posted 2012) [#41]
And about JSON. Here is a nice video html5 multiplayer: http://www.youtube.com/watch?v=zj1qTrpuXJ8

If a game JSON Packet looks like this (Assumed FPS Action Game) and we are sending one Actor.
{
     z: 1,
     id: 1234590,
     s: {
       x: 5,
       y: 34,
       v: 3,
       a: 0.46
     } 
}

The size would be: 1 + 4 + 12 + 3 + 4 + 5 + 4 + 5 + 2 =~ 40bytes
Extra Actors would mean: ~20 bytes

In Binary assuming ints and floats are 4 bytes: 4*6 = 24bytes
Extra Actors would mean 4*4 = 12 Bytes

A lot of games would work with Json, but it scales less nicely, so both options are useful.


Skn3(Posted 2012) [#42]
I just read all of your conversations and there is not really much I could add but the implementation is sounding really nice! I would say that having json functions as part of monkey is a must! It is a widely used standard and it makes dealing with sending objects very simple across the different targets. The amount of overhead for sending basic game packets is minimal and if there is a gzip option in there then no worries!

A possible request which could be worked in, is there some way we can feed in an encryption worker object or just have some encryption methods built in so packets can automatically be encrypted before sent. Perhaps the same could apply for compression, set compression on and then all "packets" are compressed.

I assume UDP will be an option for the platforms that support it?

All very exciting to see monkey growing up, keep up the good work!


Tibit(Posted 2012) [#43]
While on topic wanted to share this for html5/flash support. Maybe using http://socket.io/ would be useful? If not only for browser support but for heartbeats, timeouts and disconnection.


Why not just call it `WebSocket` if the actual WebSocket is not present and mimick its API?
Socket.IO does more than WebSocket, even if WebSocket is selected as the transport and the user is browsing your website with an ultra modern browser. Certain features like heartbeats, timeouts and disconnection support are vital to realtime applications but are not provided by the WebSocket API out of the box.

This is akin to jQuery's decision of creating a feature-rich and simple $.ajax API as opposed to normalizing XMLHttpRequest.



EDIT - Never mind, this is for NodeJS servers only. However heartbeats, timeouts and disconnects are always nice :)


vbnz(Posted 2012) [#44]
I have wrote html5 and flash native sockets!(tcp)
Also io operations (html5).
Mark,i can send you source code if you wish!
If yes,please give me your email.

Sincerely, vbnz.


marksibly(Posted 2012) [#45]
Hi,

> What exactly does WriteChunk/ReadChunk do?

They read/write a length value, then read/write that much data. The reading/writing is done on a thread, so it doesn't block the main app. Once length/data have been read/written, OnComplete is called. Whether length is represented as a byte, short or int is an issue. This could be specified in a flag when you call ReadChunk/WriteChunk, or there could be multiple versions of ReadChunk/WriteChunk. It'll probably be in flag though - I suspect there'll be quite a few flags!

It's not doing anything magical though - just prefixing blocks with a length value. You could do it yourself in client code state-machine style, it's just nice to have this built-in to AsyncStreams, which can do this sort of thing easily.

> If I want to talk to my NodeJS game server then how do I handle the packets on that side?

Just read the length byte/short/int then read the actual data. Just like 'normal' code!

Perhaps the best way of thinking about how AsyncStreams should work is to think of all the possible ways you can read blocks of data. I can think of:

* Fixed size blocks - ie: read N bytes. This is what Read/Write currently do.

* Chunked blocks - ie: read N, then read N bytes of data.

* Terminated blocks - ie: read data until the value X has been read. Useful for ReadLine (X=10), ReadCString (X=0) etc.

...any more?

If these are all built-into AsyncStream via Read(), ReadChunk(), ReadUntil() methods it should make life easier for everyone. Note that these will all have OnComplete handlers.

Also, async ReadAll is probably not gonna happen - there's no way to make it safe since there's no guarantee the read data will fit into the databuffer you provide. It could return how much data was read before the buffer filled up, but then it'd be doing effectively the same thing as Read()!

As for JSON/compression etc, yes this stuff is important, but I don't think JSON/compression modules need to (or should) know anything about AsyncStreams - they can continue to work with plain sync streams and databuffers.

You can still send/recv JSON objects asynchronously easily though. For example, if you've got a JSON module with functions like, say:

'Note: these use plain sync streams!
Function ReadObjectFromStream:JSONObject( stream:Stream )
Function WriteObjectToStream:Void( obj:JSONObject,stream:Stream )

You can then 'send' a JSONObject async using code like:

Function SendJSONObject:Void( obj:JSONObject,stream:AsyncStream )
   Local data:=New DataBuffer( MAX_JSON_SIZE )
   Local dstream:=New DataStream( data )
   WriteObjectToStream( obj,dstream )
   Local length:=dstream.Position
   'possibly compress databuffer here...
   stream.WriteChunk data,length
End


And to receive a JSON object...

Method OnReadChunkComplete:Void( data:DataBuffer,offset:int,length:int,stream:AsyncStream )
   'possibly decompress databuffer here...
   Local dstream:=new DataStream( data,0,length )
   Local obj:=ReadObjectFromStream( dstream )
   stream.ReadChunk data,data.Length   'kick off the next read...
   '
   'OK, an object has arrived! Do something with it...
   '
End


Decompression would go right at the top.

Hmmm...the SendObject raises an interesting point - Write functions will have to be careful not to overwrite the databuffer used by a previous write that may still be queued...will think about this.

But sending individual objects is probably not the way to go - you want to send entire messages that may contain many objects, extra control data, message type etc. I guess what I'm trying to say here is that you should probably do 99.9% of your network reading/writing using a synchronous DataStream, and then once you have an entire message ready to send to the client/server/peer, you use ReadChunk/WriteChunk to transfer the underlying DataBuffer. This way, 99.9 of your code can use synchronous streams and not have to worry about callbacks.

> Mark,i can send you source code if you wish!

Yes please! email is in my profile...


dopeyrulz(Posted 2012) [#46]
Mark,
One of the nice things in .net now is the standardized identification of async methods and functions. If you planning to offer both synchronous and asynchronous methods appending Async to the name might offer a very clear and easy way to differentiate.


Tibit(Posted 2012) [#47]
I really like the concept of AsyncStreams and how the networking part is turning out! That can't be said too many times :)

> you should probably do 99.9% of your network reading/writing using a synchronous DataStream
Yeah I agree, any other form of abstraction is better to happen at a higher level.

> ...any more?

Variable Sized blocks. That would mean we read until the first/last bit of a byte is 1. So and int of value 0 to 127 would take 1 byte, 127 to 16383 would take 2 bytes, 16384 to 2097152 would take up 3 bytes and so on.

Not saying they are needed, just saying they exist and can be quite handy since "numbers" can just be pushed onto the stream and are automatically optimized. However the bitlogic req to implementing them is the downside.

Might also be a efficient way to send unicode strings?

> flags

I also think flags will do very well for that :)


vbnz(Posted 2012) [#48]
Mark,I sent you a email!