BlitzMax vs C# speed comparison

BlitzMax Forums/BlitzMax Programming/BlitzMax vs C# speed comparison

JoshK(Posted 2010) [#1]
Does anyone have any benchmarks that compare these two? Just curious.


ziggy(Posted 2010) [#2]
Small test.
This is the C# code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text; 

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            int x = 0, sum = 0;
            DateTime curtime = DateTime.Now;
            while (x < 1000000)
            {
                MyClass instance = new MyClass();
                instance.IncValue();
                sum += instance.Value;
                x++;
            }
            //int result = DateTime.Now.Millisecond - curtime
            Console.WriteLine("Result = " + sum);
            Console.WriteLine((DateTime.Now - curtime).Milliseconds );
            Console.In.ReadLine();

        }
    }
    class MyClass
    {
        public int Value;
        public void IncValue()
        {
            Value++;
        }
    }

}


And this is the BlitzMax port of it:
SuperStrict
	Program.Main()
	Type Program
		Function Main()
			Local x:Int = 0, sum:Int = 0
			Local Time:Int = MilliSecs()
			While x < 1000000
				Local instance:MyClass = New MyClass
				instance.IncValue()
				sum:+instance.Value;
				x:+1
			Wend
			Print "Result = " + sum;
			Print MilliSecs() - Time
		End Function
	End Type
	Type MyClass
		Field Value:Int
		Method IncValue()
			Value:+1;
		End Method
	End Type


C# code executes about two times faster than BlitzMax in my machine, both working in release mode. C# can be even faster when install time compilation is performed using NGen tho, so not really a very significative comparison.

C# = 25 millisecs
BlitzMax = 56 millisecs


JoshK(Posted 2010) [#3]
I was under the impression C# was slower than ST BMX because it collected circular references.


BlitzSupport(Posted 2010) [#4]
It really will come down to what code you're testing. As far as I can see, this test only creates an object and adds one to a variable, one million times! Hardly real-world stuff.

Although I'd guess C# will be faster at a lot of stuff in extreme benchmarking, I'd guess it's the object creation which is really taking the time here -- but creating 1 million objects in 56 milliseconds? When are you going to do that? Where would that actually matter?


Mahan(Posted 2010) [#5]
Actually there are (rare) situations when you want to create/destroy millions of small objects, and that's the reason as an similar example Java uses a "short time GC list" in addition to the normal GC reference list. This allows the JVM (java virtual machine) to reuse the object-spaces in memory without actually go down to the OS-level to malloc() and free() the storage when objects of the same type (class type) are created and destroyed at a very fast pace. I'm using Java as an example because I got more experience in it/knowledge about it, but I'm pretty sure C# got a similar strategy that we see in the results above.

But as I said above I agreed that the situations where you would need this in a real world application are rare, and would that situation actually occur it's not rocket science for a BMX-programmer to simulate this kind of instance-caching at that particular moment for a certain type (class).


ziggy(Posted 2010) [#6]
The idea was to compare object allocation and Garbage collection, but it's just a small silly test. If someone wants to write a better one... I'm sure it'll be much consrtuctive. :D


Mahan(Posted 2010) [#7]
@ziggy: Just wanted to clarify that my post was in no way meant as any criticism on your test code, but merely a possible explanation to the speed difference we see from it :-)


BlitzSupport(Posted 2010) [#8]
@Mahan: no doubt there are extreme cases where this *would* matter. But in general games development, intended to run on a wide range of PCs, and even just including the 'high end'? I don't see it being a problem.

@ziggy: no direct criticism really intended! It just comes down to what you're actually trying to do in any given situation. I can't imagine it would really matter on any system capable of running the (rather lovely) Leadwerks Engine at an acceptable speed in the first place -- that really must be GPU-bound rather than CPU-bound for most use-cases.


ziggy(Posted 2010) [#9]
Hey, I did not get it as criticism or anything! I was just trying to aim people to write more benchmarks. I think it could be interesting.

Anyways, the real speed monster of C# comes to reallity when the applicatoin is strong named and compiled using NGEN, so the JIT compilation is removed. I've seen speed increases of 20x or more when doing this on BLIde. Some code performs faster than Visual Studio C++ code when pre-compiled using NGen.

A real comparison should be done in this scenario. We'll see what happens to the BM in lots of platforms, bur if it is developed using LLVM,. it theorically could beneffit from several compile-time optimizations + runtime optimizations and also install-time optimizations. We'll see (hopefully).


Mahan(Posted 2010) [#10]
@BlitzSupport: I fully agree with you and I'm still often impressed at the raw speed of BMX applications. Tbh, considering the strategy and the algorithms used to solve a problem is much more often important (imo) than the speed of the language anyways, I have yet to see a situation where I'm forced to switch to a lower level language (asm/c etc) to get acceptable results. (but then I'm no BMX guru :-)

EDIT: ... and also there are few things that can get a developer more engaged in an argument, than challenging the speed of one of his his "favorite programming languages(tm)" which is pretty fun in retrospective (talking 'bout myself) ;-)


JoshK(Posted 2010) [#11]
Why would removing the JIT compilation have any effect on execution speed? It would only decrease the delay before the program started running.


Robert Cummings(Posted 2010) [#12]
Coming from coding iphone games, I just make it work, I don't care how much cpu time I waste, I just use it all floats, whatever and it runs at 60fps.

Its unlikely you'll slow down your blitzmax games like ever. If you are doing some kind of server framework thats really important for speed, it will still actually come down to the method you use, not the language, that ultimately decides if its effective.

It wouldn't suprise me if all variants of C were faster than Blitzmax after all they are mantained by up to thousands of people on an ongoing basis.

Blitzmax makes great games in basic though... lets not forget.


RexRhino(Posted 2010) [#13]
Given that no one here is a AAA game developer, I would say the speed of the language is irrelevant. The biggest problem that indie developers is developing a decent game, not squeezing out performance.

I use C# for Xbox360 development, and I use BlitzMax for development on other platforms. I don't really notice any difference whatsoever in terms of speed... both engines can handle all I throw at them. XNA has a lot nicer features in terms of shaders and stuff that make the C# games look a little nicer on the console, than BlitzMax and miniB3D who even though use openGL have features closer to the DirectX 7 era.

If you *ARE* using C# for game development, please don't use XNA. From my experience, XNA sucks on the PC. It rocks on the 360, but it is a pain in the ass on the PC. If you must use C#, just use DirectX or openGL directly, not XNA.


ziggy(Posted 2010) [#14]
Why would removing the JIT compilation have any effect on execution speed? It would only decrease the delay before the program started running.
Becouse JIT compilation also happens while the program is running, not only before it runs.


Brucey(Posted 2010) [#15]
Becouse JIT compilation also happens while the program is running

You mean it is compiled... just-in-time ? :-)


ziggy(Posted 2010) [#16]
Whenever a class is requested, the .net JIT compiler compiles the class if it has not been compiled before. At program start, only the first class to be loaded is compiled, and if at runtime it is requesting more classes functionality (due to inheritance or other classes instances creation), then the additional classes are compiled at first use. That usually makes .net applications slower to start when compared to native ones, except if the .net application stores a pre-compiled version of itself in the .net assembly cache.


slenkar(Posted 2010) [#17]
Can A* pathfinding be a good benchmark?

I noticed that with blitz3d compared to blitzmax
blitzmax was a lot faster


Czar Flavius(Posted 2010) [#18]
Any realistic benchmark would need a few images and things in there as well, and a mouse pointer for clicking buttons. If it's not smooth enough to click the buttons, it fails the realistic benchmark test :D


Kryzon(Posted 2010) [#19]
XNA has a lot nicer features in terms of shaders and stuff that make the C# games look a little nicer on the console, than BlitzMax and miniB3D who even though use openGL have features closer to the DirectX 7 era.

I think it's a bit unfair to compare an engine supported by Microsoft against another one created by one guy =)

After all, MiniB3D is an OpenGL interface [EDIT] with a game engine framework. It's open to receive whatever advanced features the GL (and your videocard) supports.


Dreamora(Posted 2010) [#20]
XNA is no engine, its a "nicified & simplified DX"

The "looks nicer" part is partially good but it must not be forgotten that XNA has Shader Model 2 as minimum requirement. Its not a "feature"


Gabriel(Posted 2010) [#21]
I think it's a bit unfair to compare an engine supported by Microsoft against another one created by one guy =)

Equally, it's unfair to compare a maths teacher who studied at Oxford with some guy who had to leave school at fourteen to support his sick father and never had the money to resume his studies, as the former has clearly had many advantages over the latter, but I know who I'd want teaching my child. My point being, if you make all your decisions based on what's fair, you're going to make a lot of stupid decisions.


JoshK(Posted 2010) [#22]
XNA has a lot nicer features in terms of shaders and stuff that make the C# games look a little nicer on the console, than BlitzMax and miniB3D who even though use openGL have features closer to the DirectX 7 era.

I don't think people generally get BlitzMax because they want to make games. Most people like it because it is a general-purpose cross-platform language with some nice basic libraries included. Selling it as a 2D game maker is a mistake.


Czar Flavius(Posted 2010) [#23]
Are you sure? That's the whole reason I got it.


Robert Cummings(Posted 2010) [#24]
Given that no one here is a AAA game developer, I would say the speed of the language is irrelevant. The biggest problem that indie developers is developing a decent game, not squeezing out performance.


AAA game developers do not optimise for speed or optimise at all. They make very heavy use of hardware specific calls, ready made API and slow scripting engines.

AAA game developers work on consoles. They are easily powerful enough to do the job and fixed hardware. They know they can use this xbox core for that task.

There is very little optimisation going on there, the biggest optimisation is in the methods they use. A case study would be Bioware, who used Unreal Engine 3 for both mass effect and mass effect 2.

There was no change in code or engine between the two versions yet Mass Effect 2 was able to deliver more on-screen content than the previous game.

It was as simple as changing their methods used, rather than optimising. They in their own words "read the documentation for it".

And designed levels using the props methodology recommended by Epic.


The morale of the story is: big or small, what you do is more important than optimisation.


Brucey(Posted 2010) [#25]
Most people like it because it is a general-purpose cross-platform language

That's why I got it...


shinkiro1(Posted 2010) [#26]
I have Win7 and Ubuntu 9.04 on my Notebook here.
When I tested my game, Ubuntu was able to render ~6000 game objects,
while Win7 rendered just around ~4500 (60 FPS).

First of all, I don't want to bash Windows -.-
I just discovered this and it is really interesting.
Maybe if you would change driver settings it would be the same,
I am not an expert, I just wanted to inform you about this.
(I also noticed this speed difference in my other projects)

Generally, as long as the Game runs, you don't have to worry about
optimisation. After all, it's about the games and not how fast they would be able to run :D


*(Posted 2010) [#27]
as long as you get a steady 60fps no matter what the hell you do it dont matter.


Gabriel(Posted 2010) [#28]
When I tested my game, Ubuntu was able to render ~6000 game objects,
while Win7 rendered just around ~4500 (60 FPS).

That's insufficient information to tell us anything though. What graphic drivers were you using?


JoshK(Posted 2010) [#29]
When I tested my game, Ubuntu was able to render ~6000 game objects,
while Win7 rendered just around ~4500 (60 FPS).

In very rough terms, that's the kind of performance loss I have observed with Windows Vista/7 versus XP. In my tests, NVidia Linux drivers run a bit faster than XP, and XP is about 20% faster than Windows 7.


Dreamora(Posted 2010) [#30]
@Espada: And your game in both cases only used the OpenGL driver?
Cause otherwise the comparision can only fail due to the defaulting on the platforms.


Who was John Galt?(Posted 2010) [#31]
Win7/Vista is chock full of all sorts of DRM 'features', such as internal communications between functions being encrypted, tilt switches, etc. This garbage will cause a performance hit, no matter how well it's programmed.


slenkar(Posted 2010) [#32]
A*pathfinding and raycasting are real-world applications that can be used to test language speed


Mahan(Posted 2010) [#33]
Some testing by me. I tried to convert the small test benchmark program in the samples-folder of BlitzMax to C# (mono 2.4.4 on my Ubuntu Linux machine):

C# code (Main.cs):


BMX code (bench.bmx):




Results:

C# (release):

SIEVE OF ERATOSTHENES - 10000 iterations
10000 iterations took 00:00:02.2620590
Primes: 1899

BMX (release):

SIEVE OF ERATOSTHENES - 10000 iterations
10000 iterations took 1.59899998 seconds.
Primes: 1899


NOTE: Feel free to optimize the codes if you want. =)


JoshK(Posted 2010) [#34]
Is 00:00:02.2620590 2.26 seconds?


Mahan(Posted 2010) [#35]
Yep. That's probably the default .toString()-rendering of a TimeSpan in C#.


ziggy(Posted 2010) [#36]
My results compiled using Visual C# compiler 2008 from MicroSoft (and run outside visual studio, this is very important to remove any debugging sync between the IDE and the program)

SIEVE OF ERATOSTHENES - 10000 iterations
10000 iterations took 00:00:00.4992009
Primes: 1899



My results compiled with BlitzMax latest relase:

SIEVE OF ERATOSTHENES - 10000 iterations
10000 iterations took 0.426999986 seconds.
Primes: 1899



So BlitzMax a bit faster regarding this benchmark, but I would preffer to see a banchmark that includes the usage of garbage collection, multithreading, etc. This would be an even more realistic benchmark.


xlsior(Posted 2010) [#37]
Given the short runtime of that example, it probably should be increased to 100,000 iterations or something to get a more accurate result.


impixi(Posted 2010) [#38]
I performed some extensive height map generation comparison benchmarks a couple of years ago, in plain C, C++ and BlitzMax. C won by a considerable margin for most of the algorithms. C# is a slightly different beast again, and if I can spare the time and effort I'll adapt some of my C code. If anyone else wants to have a crack at it, my worklog contains a version of my C-coded 'heightmap toolbox' algorithms...


Mahan(Posted 2010) [#39]
@xlsior: up:ed the ITERATIONS-constant to 100000 and rerun.

This time on my Windows 7 laptop:

Main.cs with Microsoft (R) Visual C# 2008 Compiler version 3.5.30729.4926
results:

SIEVE OF ERATOSTHENES - 100000 iterations
100000 iterations took 00:00:06.6093780
Primes: 1899

bench.bmx (lastest BlitzMax):

SIEVE OF ERATOSTHENES - 100000 iterations
100000 iterations took 4.48899984 seconds.
Primes: 1899

@ziggy: I agree that a bigger test that don't only rely on calculations might be more overall useful, but that was not my point with this test.

My point is: C# and BlitzMax are at least in the same ballpark speedwize. It's quite realistic to expect that you could write a game/database/math-lib or almost whatever with very similar results in C# vs BlitzMax.

It's not like ruby, python or another scripting language where you just have to admit that some things are not possible to do as the same speed.

And that's what I wanted to show :)


Ian Thompson(Posted 2010) [#40]
JIT code is always distributed as bytecode, it can be interpreted or compiled at runtime. Personally I feel this dynamic compilation technique is not as quite efficient as a full multipass compiler with plenty of time to optimise the code across the width of the whole program. It can be thought of as a halfway between an interpreter and a compiler. So comparing it to BMax is not really fair since Bmax is not another JIT compiler.


taumel(Posted 2010) [#41]
In scenarios like a fractal calculation i found BlitzMax slightly faster than C# (JIT-Compiled).


Mahan(Posted 2010) [#42]
@Ian Thompson:

Examples in this thread has shown both that C# can be faster than BMX and vice versa in two separate tests/tasks.

The original question to this thread was:

Does anyone have any benchmarks that compare these two? Just curious.



Thus my interpretation of the original question is "are the languages of comparable speed?". The two tests so far have shown that yes: these languages are in the same ballpark.

I don't know how "fairness" comes into the picture at all, care to enlighten me?

Why I think fairness is not important here: Let's imagine I was writing a c compiler in ruby and it became the slowest compiler anyone has ever heard of and about as useful as a pimple on ones backside, and then I just said: "Hey, it's not fair to compare my c compiler written in ruby, with other c compilers written in C because ruby is a script language".

Relevant? No. And neither is pre-compilation vs. JIT/VM when we benchmark these two languages imho. We just want to know which one is faster in different situations.


ziggy(Posted 2010) [#43]
JIT code is always distributed as bytecode, it can be interpreted or compiled at runtime. Personally I feel this dynamic compilation technique is not as quite efficient as a full multipass compiler with plenty of time to optimise the code across the width of the whole program. It can be thought of as a halfway between an interpreter and a compiler. So comparing it to BMax is not really fair since Bmax is not another JIT compiler.


you can compile the byte-code to native code at install time, and you get a non JIT binary. That works as an ultra optimized native compilation that takes into acount, not only the architecture, but also the amount of ram, os version, etc. of the host machine. That's the most optimized way of deploying an application. Thinks like NGen and the .net Assembly Cache for strong named binary executables are great additions regarding this respect.