Creating the Ultimate Monkey Performance Test?
Monkey Forums/Monkey Programming/Creating the Ultimate Monkey Performance Test?
| ||
| I feel that at times even the simplest of games can has low FPS in Monkey, or more importantly "spikes" or some kind of sloppiness. Notice I used the word "feel". Might to a big degree be html5 related. I want to have facts that answers WHY. If there is anyone who manage to create awesome looking high performance 2D games in on targets that Monkey translate too then should not Monkey come very close in terms of performance? There are things we cannot change like Hardware & Browers, but we can still measure and compare - the factor we can change is the programming. Because Monkey Translates code I can just not buy the "excuse" that Monkey code is worse off - should it not be just as good (or close to) as native code? If the "Error" lies in the programmer then I'd love to hear what should be done and what should not be done. I.e. What factors should we look at. Creating Performance Tests I suggest that to find the answer we can create a suit of performance tests. Is there a good series of tests can be created to test important performance factors? Performance usually goes hand-in-hand with quality - so there needs to be tests for common effects as well. (The only one I can think of is gradual png transparency) Creating tests is not as simple as it sounds - we need to find and test factors that matters to gameplay - and be able to measure the point of slowdown or the point when spikes start to occur. Threads & Links Example of a Test: The Bunny Test (html5) Source Code of Bunny Test on Flash here More info at his Blog. Awesome thread on performance on html5 here Nice looking Html5 game with awesome performance: X Type More info in this monkey thread. They used this engine The main question is, how could such test look like? * Is the bunny test good? * What does it measure? * Any Monkey specific issues? * Any Target specific issues to test or deal with? * Is it "all" about graphics? * What about garbage collection? |
| ||
| I think look at some of the existing bench marking tools, they dont just throw inert objects at the screen they run full physics on a lot of them, just throwing art about is not that big a deal but having objects interact in some way is in terms of cpu time. so yeah. make those bunnies bounce off each other, or make them bounce off of something else like a wall or ball the user can move, or that moves around on its own. |
| ||
| I've had no problems with Monkey performance on Android (using Monkey ver 42 - using canvas not GL) as long as the app is written well...I've written a bunch of games for both flash and android with Monkey and performance seems fine. Using a good approach to managing object creation and removal seems to be the key, as well as the algorithms you choose. EDIT - what would be the key I think.... use Arrays of objects rather than lists...choose your algorithms wisely - design well at the outset, ?? |
| ||
| do lists really have that much more overhead and if so it might be an idea the write a fake list routine that actually uses arrays while giving you the functionality of a list. |
| ||
| do lists really have that much more overhead For Android - Yes! if so it might be an idea the write a fake list routine Check out Diddy's ArrayList. For "fun" I quickly converted the Bunny demo to Monkey: [monkeycode]Strict Import mojo Function Main:Int() New MyGame() Return 0 End Class MyGame Extends App Field numBunnies:Int = 3000 Field gravity:Float = 3 Field bunnies:List<Bunny> Field maxX:Int = 640 Field minX:Int = 0 Field maxY:Int = 480 Field minY:Int = 0 Field bitmap:Image Field fpsRate:Int = 30 Method OnCreate:Int() SetUpdateRate(fpsRate) bitmap = LoadImage("wabbit_alpha.png") bunnies = New List<Bunny> Local bunny:Bunny For Local i:Int = 0 Until numBunnies bunny = New Bunny bunny.image = bitmap bunny.speedX = Rnd() * 10 bunny.speedY = (Rnd() * 10) - 5 bunnies.AddLast(bunny) Next Return 0 End Method OnUpdate:Int() For Local bunny:Bunny = Eachin bunnies bunny.x += bunny.speedX bunny.y += bunny.speedY bunny.speedY += gravity If bunny.x > maxX bunny.speedX *= -1 bunny.x = maxX Else If (bunny.x < minX) bunny.speedX *= -1 bunny.x = minX End If bunny.y > maxY bunny.speedY *= -0.8 bunny.y = maxY If Rnd() > 0.5 bunny.speedY -= Rnd() * 12 End Else If (bunny.y < minY) bunny.speedY = 0 bunny.y = minY End bunny.posX = bunny.x bunny.posY = bunny.y + bunny.z Next If KeyHit(KEY_LEFT) fpsRate-=5 SetUpdateRate(fpsRate) End If KeyHit(KEY_RIGHT) fpsRate+=5 SetUpdateRate(fpsRate) End Return 0 End Method OnRender:Int() FPSCounter.Update() Cls For Local b:Bunny = Eachin bunnies DrawImage(b.image, b.posX, b.posY) Next FPSCounter.Draw(0,0) DrawText("FPS Rate: "+fpsRate, 0, 10) Return 0 End End Class Bunny Field speedX:Float = 0 Field speedY:Float = 0 Field speedZ:Float = 0 Field image:Image Field x:Float = 0 Field y:Float = 0 Field z:Float = 0 Field posX:Float = 0 Field posY:Float = 0 Field angle:Float = 0 Field speed:Float = 0 End Class FPSCounter Abstract Global fpsCount:Int Global startTime:Int Global totalFPS:Int Function Update:Void() If Millisecs() - startTime >= 1000 totalFPS = fpsCount fpsCount = 0 startTime = Millisecs() Else fpsCount+=1 End End Function Draw:Void(x% = 0, y% = 0, ax# = 0, ay# = 0) DrawText("FPS: " + totalFPS, x, y, ax, ay) End End[/monkeycode] HTML5 Demo: http://www.therevillsgames.com/monkey/bunnies/bunniesHTML.html Flash Demo: http://www.therevillsgames.com/monkey/bunnies/bunnyFlash.html Press Left/Right to change the FPS Rate. Using Chrome: FPS Rate at 30: HTML5 - 20FPS Flash - 29FPS FPS Rate at 60: HTML5 - 19FPS Flash - 51FPS |
| ||
| 28 fps @ 60 rate on html and 60 fps @ 60 rate on flahs thats the first time I have seen something run faster on flash than html. |
| ||
| Strange I always see Flash out perform HTML5... |
| ||
| Bunny monkey html5 example runs 2 fps on ipad1. 1 fps on new Chrome browser for ipad1. 1 fps on Galaxy tab 7. Bunny original runs 5 fps on chrome browser. |
| ||
| Opps I had 6000 bunnies, there should have only been 3000 (changed now)... HTML5 Demo: http://www.therevillsgames.com/monkey/bunnies/bunniesHTML.html Flash Demo: http://www.therevillsgames.com/monkey/bunnies/bunnyFlash.html (Press Left/Right to change the FPS Rate) Using Chrome: FPS Rate at 30: HTML5 - 30FPS Flash - 30FPS FPS Rate at 60: HTML5 - 42FPS Flash - 59FPS Bunny original runs around 50FPS... it hops around a lot (45-52FPS) ;) |
| ||
| Also try adding the (X or $00) trick to speed it up a bit. |
| ||
| Sorry, what trick? |
| ||
| http://www.monkeycoder.co.nz/Community/posts.php?topic=1164 |
| ||
| The trick is this: Replace on line ~85 the DrawImage part with this: [monkeycode]DrawImage(b.image, ~~(0.5+b.posX), ~~(0.5+b.posY))[/monkeycode] It practically converts the position to an int (and is much faster that using Int(number) ) The results are amazing (all fps are maximum achieved fps) Safari before hack: 39 fps Safari with hack: 86 fps Firefox without hack: 16 fps Firefox with hack: 51 fps Interestingly Chrome doesn't seem to care. It stays at 39 fps. |
| ||
| Yep, just tested it myself - I added this to Mojo's DrawSurface and DrawSurface2: [monkeycode]x = (x + .5) | 0; y = (y + .5) | 0;[/monkeycode] Results:
Browser | Version | FPS Rate | FPS
--------|----------+----------+-----
IE9 | Mojo | 30 | 23
| | 60 | 20
| Hack | 30 | 30
| | 60 | 50
--------|----------+----------+-----
Chrome | Mojo | 30 | 29
| | 60 | 42
| Hack | 30 | 29
| | 60 | 41
--------|----------+----------+-----
FF(13) | Mojo | 30 | 29
| | 60 | 58
| Hack | 30 | 30
| | 60 | 59
Mojo Version: http://www.therevillsgames.com/monkey/bunnies/bunniesHTML.html Hack Version: http://www.therevillsgames.com/monkey/bunnies/bunnyHTML5.html From my results the "hack" really helps IE. |
| ||
| It kind of feels like the only think I go on about on these forums. But if the app you are creating creates garbage, there will be performance issues. I found this out when dealing with the Xbox, but I've recently started messing around with Android and while not to the same extent, it is still an issue. It seems to me that lists and maps that are iterated through during game time (as in during the level, not during the init) are really not suitable for most targets (because of the garbage creation). On PC it isn't a problem because the GC is so quick but everywhere else, it seems to be a problem. Originally I had assumed that if this functionality was available it meant it had been suitably tested. But really I don't think it has (yet). That is: I don't think there's any suitable advice on when you can use such things. I stick to arrays (and will be giving ArrayLists a proper look now!), because it's the only way I can iterate without junk. DrawImage(b.image, ~~(0.5+b.posX), ~~(0.5+b.posY)) This is amazing :) I use Int() for practically every draw command so this I hope will really help. Just to be clear though, I love Monkey and am really thankful to have it! |
| ||
| @raz Also try therevills version (post #14) as it may even be faster. But I think they should be about the same speed, but more importantly they are much faster than casting to int. |
| ||
| Raz, Yeah I have even passed on your garbage message to some people! You could say I am a disciple? hah! Have you tried iterating over lists manually instead of foreach? (blitzmax: http://www.blitzbasic.com/Community/posts.php?topic=97352 ) I converted to this method in Objecty half way through development and it has definitely improved performance! |
| ||
| Performance statements are useless without clear definition of platform and timings. Casting to Int in Monkey converts to "((float_val)|0);" in javascript. Is that somehow massively slower than "~~"? Even if it is, how would that relate to the C++ translation, C#, Java or AS3? How does any of it actually relate to real usage in a game? Do you seriously think that the issue revealed by the bunnies test is about the cost of converting from a float to an int and not about the underlying rendering engine? |
| ||
| Html for now, can't use the gpu for floats and that is the reason why ints are faster (from what I've understood). For all other targets you are right, it might be useless. As soon as you have a gpu that can handle floats the advantage of the above method is gone. |
| ||
| You're missing the point. The performance advantage has nothing to do with the cost of converting to an Int, it's just to do with passing an Int to the renderer. Int(float_val+0.5) or your ~~(float_val+0.5) is irrelevant. |
| ||
| muddy_shoes : I would say it isn't the most substantial issue but clearly it has an impact to Monkey as it is now. Is it acceptable though? Not really. I agree that the "fix" needs to be Monkey base code, not our code. |
| ||
| Chris, I could disagree with people in this thread all day, but if you're not going to replace your int casts on the basis of what's been said then my work is done. |
| ||
| Skn3 : It's all about the garbage :D Muddy_shoes : While as a general fix for all targets it wouldn't be worth it. Does the above example not prove that it's worth having when targeting HTML5 (for Monkey as it is now)? |
| ||
| Which point of view would you rather take? If you expect Monkey to protect you from all external forces, then yes. Except it doesn't and couldn't. If you want Monkey to support low-level games performance coding then it can't and doesn't. In the specific example we're talking about, the difference is due to Javascript engine programming, not Monkey. |
| ||
| I'm with muddy_shoes here. Also, optimizing in order to get better performance on a given browser is not a good idea as all related to html5 is still being heavily optimized on browsers. |
| ||
| I agree too in the long run, it would be nice to have a perfect system that requires only one way of doing things. When it comes to anything running directly in a browser sometimes you are forced to do browser specific optimisation! In the 5 years I did web design/development professionally there was not one site/system I worked on that didn't require some type of browser specific code. Just something to bear in mind. |
| ||
| was the flash example compiled with mojo image filtering disabled? tested on firefox on linux: original bunny example no pixel snapping 1FPS pixel snapping 4FPS monkeymojo version 1FPS monkeyhack html5 version 2FPS monkey flash version 8-10FPS |
| ||
| I think that ANY method of speeding up FPS and performance over all can never be irrelevant, even if it is only one a specific target, or browser, it's still a performance boost and something that does in some situations have value, it would not be the first time I have seen a flash game, or html 5 game that asks nicely , please use FireFox, or please use IE because I assume they have done something similar, and have some enhancement in that browser. I think I agree as well that with HTML5 there is such a wide range of results between the different browsers, it will take a little time for it all to settle down and become more standard, which I believe it will in a few months or a years time. So. basically, keep this discussion going, keep talking about ways to speed up the performance of our code, because it can only help the community in the long run. |
| ||
| Any time I look I find Flash crushes HTML5 for speed at the moment. As for Monkey, it has an added level of indirection from translating to the target languages, so it can be expected to be a little bit slower than code written in the targets. On the bright side, I suspect that the overhead in translating to Java or Actionscript in particular is very low. It is higher for C++, but then the compiler makes C++ into blindingly fast code, so the end result outpaces anything else. For Javascript I don't know - I reckon this will always be the slowest but there are so many things making HTML5 slow that this is just one extra. |
| ||
| I don't know if HTML5 will ever be ideal for games. Has the sound issue been cleared up yet? And the performance is vastly different on mobile since they are not tapping the GPU (except in iPhone5). NaCl would be the next step: https://developers.google.com/native-client/ |
| ||
| Awesome discussion :) @therevills - Nice! Well the Bunny Test Runs really good for me. So I'll recap a little. Performance Factors: * Creating objects at runtime - results in garbage which on some platforms may cause lag -- seems this is a big issue on Android, Xbox and html5 in particular * To avoid Garbage - DO NOT use any Monkey standard collections like, List, Map or iterate using EachIn - instead use Arrays or DiddyCollections * Precreate Shots/particles - Objects can be created at level start/game start and then be reused using a push/pop system * There are a few hacks that can increase performance on html5 in particular, required mojo modification - I assume this will be added to mojo in an future update. See below: [monkeycode]'Add this to Mojo's DrawSurface and DrawSurface2: x = (x + .5) | 0; y = (y + .5) | 0;[/monkeycode]Thanks to therevills for testing! * Mobile CPU Limits - Obvious performance tip is to make sure Physics, Collision, Logic, AI and such are optimized based on your specific game play. Ex: Do not check all bullets against all bullets if you do not have too! (However in this regard Monkey should be no different from any other language or engine) * Prioritize optimization - Some optimization factors only apply to one or a few targets, only do such optimizations on targets you plan to release too. Also depending on your game and gameplay, optimizing certain factors such as object creation might be a waste of time. Questions Remaining * Image-maps - How big effect does image-maps have? Do we need them? When do we need them? Performance Difference? What Monkey options are there for image-maps atm? What are there pros/cons? * Loading times - Short loading times is important on both web and mobile. What can be done to reduce them? (Monkey req you to load a lot at App start for example) * Casting - Should we avoid Casting extensively in runtime? * Finding more factors.. - What more to consider? Target specific? Monkey specific? |
| ||
| I'm with muddy_shoes here. Also, optimizing in order to get better performance on a given browser is not a good idea as all related to html5 is still being heavily optimized on browsers. To be clear, I agree we shouldn't target specific browsers, but I don't see what's so bad about implementing something that shows it improves monkey's performance on most browsers and at the very least doesn't reduce it on any. |
| ||
| Image-maps Using a sprite altas is so much faster as the sprite(s) only share the one image. Casting Casting should be okay, its boxing you should be careful with. Muddy did an excellent blog post of this: http://pointlessdiversions.blogspot.co.nz/2012/03/monkey-tip-be-careful-around-unboxing.html |
| ||
| I don't see what's so bad about implementing something that shows it improves monkey's performance on most browsers and at the very least doesn't reduce it on any I didn't say it was bad, although it's an open question whether it is "harmless". Do fractional pixel positions result in smoothing? If so then is it certain that no game would ever want that feature? What I was reacting to was the usual result of people searching for performance "tricks" which is that they end up grabbing at easy answers that are wrong. Casting to Int isn't a performance boost on other platforms and ~~ rather than Int() _is_ irrelevant when the issue is actually the rendering speed resulting from using integer positions. It's even more irrelevant when the fact is that Int() is faster than ~~. To pluck the idea from this that you should change all the Int casts in your code is a classic example of how people waste their time on "optimisations" that have no basis in reality. |
| ||
| Ahh right ok I see. I got the impression from this thread that all draw commands to HTML5 were rounded to Int coordinates anyway. It's even more irrelevant when the fact is that Int() is faster than ~~. I assumed from the above that ~~ is shown to be faster than Int() for HTML5 though? I suppose I really should check these things out for myself. |
| ||
| I would suggest against this 'hack' going in the official monkey version because, as muddy_shoes mentioned, what if someone wants to have pixel smoothing? @muddy_shoes It's even more irrelevant when the fact is that Int() is faster than ~~. My test's resulted that when I used Int() conversion the framerate stayed the same as with floats. But I think I know what you are up to. People tend to believe that some 'magic code' will somehow improve performance not really knowing what happens behind the curtains and using absurd test cases. Still, at this moment, if you don't need pixel smoothing, I see no reason why not to use this little trick on HTML5. It would also be interesting to know if this hack scales, so when adding other components typically found in a game, if the version with the hack will still be that much faster (in %). |
| ||
| I think you should put the int conversion in your own code rather than in mojo. it seems to me that there are quite probable situations when you would want to use it on some images and not others. For example, spaceship sprites might prefer to use ints for various reasons (not just speed but lack of blurring), but if they are moving on a scaled background, the background might well look better with floats. |
| ||
| the ~~ trick should be official, enabled with mojo image filtering disabled it has exactly the same graphical effect as the flash version |
| ||
| @muddy_shoes - Awesome article on Boxing! I can strongly recommend this article for those working on performance and reducing the number of objects created: Monkey Tips Be Careful Around Unboxing |
| ||
| From muddy_shoes Blog: On iOS that doesn't matter because you pay for every object you have live. What that means is that object pooling or caching strategies will benefit you on Android and actively work against you on iOS That explains a lot! iOS Lag has less to do with garbage from NEW objects and a lot more to do with Monkey's garbage checking process for live objects! So on iOS it seems one needs to reduce the number of objects TOTAL, not the number of objects created in runtime. Any more suggestions on how to work with that? |
| ||
| The first question when considering performance issues is "Do you actually have a performance issue?". Fast enough is fast enough. Many games are not going to use large numbers of objects and will never see the iOS GC causing a problem. If your framerate isn't lower than you want then just get on with writing your game and stop worrying about optimisation. If there is a performance issue then you move onto the second question: "Where is the time going?". If you don't know how to use a profiler on your target platform then now is a good time to learn. In this case we can assume that you've seen the gc_mark process eating a lot of time so we know that is the culprit. As I wrote in the blog post, your choices are now to either reduce your object count or to change the GC. It might be that you're in the fortunate position where nearly all your objects are in just a few sub-systems: particles, gamemap etc. You could then maybe change those systems so that they are based on arrays of primitives rather than objects. It's possible that this will be faster across the board (haven't tested so I don't know if it actually plays out that way) and resolve your iOS GC problems. On the other hand, your objects might be more spread throughout your code and not so easy to change. In that case warming up your C++ muscles and tinkering with the GC implementation is probably the better option. |
| ||
| In that case warming up your C++ muscles... hahaha very funny. |
| ||
| On iOS that doesn't matter because you pay for every object you have live. What that means is that object pooling or caching strategies will benefit you on Android and actively work against you on iOS is this true for both methods of GC? (the methods that you can choose in the config file) |
| ||
| I did not know you could choose GC actually. Any more info on that? Is it in the docs? |
| ||
| its on the bottom of the page here: http://blitz-wiki.appspot.com/Monkey_Config_settings |
| ||
| I non-incremental GC is worse because it tries to deal with all the objects on every frame. The incremental version spreads the marking phase over several frames. |
| ||
| Hi there. This is a very old post but I used it to create my own performance test code. I just added some flags to enable: alpha, setcolor and rotation. My result: - Drawing images with the simple prototype "drawimage image,x,y" is much faster than the extended prototype "drawimage, image,x,y, rotation, scalex,scaley,frame". Even if rotation and scale is 0,1,1... - On Mac OS X, Safari is slow and Firefox is awful. Chrome is amazing. - SetColor is forbidden with HTML5 (for those who did not know :)) - Something amazing with Safari: use of an alpha < 1 is faster (almost x2 !). So, using SetAlpha(0.99) will double the performances compared to SetAlpha(1)... Weird!! I'm going to test this app on a Smart TV soon, that's why I'm preparing the code, wait and see... |
| ||
| I'm going to test this app on a Smart TV soon, that's why I'm preparing the code, wait and see... SmartTVs are VERY slow. My LG Smart TV 2012 is only able to clean canvas 1280x720 at 2 fps:http://imgur.com/XOyVt9x |
| ||
| Argggg! |
| ||
| @David Casual Box Will you share the source code of your benchmark? |
   |