Cower.TCC - Compile C at runtime

BlitzMax Forums/BlitzMax Programming/Cower.TCC - Compile C at runtime

N(Posted 2007) [#1]
TinyCC (aka TCC) is a small but hyper fast C compiler. Unlike other C compilers, it is meant to be self-relying: you do not need an external assembler or linker because TCC does that for you.

...

With libtcc, you can use TCC as a backend for dynamic code generation (see section 7. The libtcc library).

(Tiny C Compiler Reference)

What that means, in general, is that you can compile C during runtime with this module. You do not have to output an EXE, DLL, OBJ, etc. -- you just simply compile it and you can use what was compiled. I've included a ported version of libtcc's own test code to give you an idea of how this is done, as well.

The point of this all is that you don't have to write code in Lua and such, and it erases some of the annoying glue issues surrounding scripting languages (no intermediary functions to call to convert script values and such to what you need, instead you just call the function directly). It's not a scripting language.

Who's this for? Aside from me, anyone who is interested in avoiding the annoyance of using scripting languages and writing glue functions and the like. Really, it's very tedious to do. It's also for C programmers, like me, who would much rather do their work in C than another language. Lastly, it's for nutjobs who want to mess around with weird stuff that few people have a use for.

So, without further ado, have at it: DOWNLOAD

Edit: Oh, and since I forgot, here's the link to the TCC website.


Michael Reitzenstein(Posted 2007) [#2]
That's really quite cool. I might hook it up to my lisp interpreter to start supporting compiled functions.


JoshK(Posted 2007) [#3]
Whoa, cool. I wonder if this is how Quake CC works?


N(Posted 2007) [#4]
No, Quake C is interpreted last I checked. However, in order to be sure, I'd have to go through my copy of the source, which I don't really feel like doing.


xlsior(Posted 2007) [#5]
By the way: Welcome back, Noel!


morszeck(Posted 2007) [#6]
Is this only for windows? I am unfortunate :-(

Building libtcc_test
Compiling:libtcc_test.bmx
Linking:libtcc_test.debug
/usr/bin/ld: archive: /Applications/BlitzMax/mod/cower.mod/tcc.mod/libtcc.dll.a has no table of contents, add one with ranlib(1) (can't load from it)
collect2: ld returned 1 exit status
Build Error: Failed to link /Applications/BlitzMax/mod/cower.mod/tcc.mod/libtcc_test.debug.app/Contents/MacOS/libtcc_test.debug
Process complete



kronholm(Posted 2007) [#7]
Welcome back Noel :)


N(Posted 2007) [#8]
morszeck: From the readme..

I can't provide support for Linux or Macs, as I
do not own a Mac and I do not care to go through
the trouble of installing Linux again. If you
want to help by working on a Linux or Mac version,
be my guest, I'll gladly incorporate your work
into the module.


Basically, unless someone provides me with a Mac, I can't really do much. You can compile TCC libs yourself for the Mac, as I have provided the source code for the library (including Doug Currie's modifications to make the Windows DLL work). You will likely be able to adapt it easily if you know your C and how the Mac OS X stuff works.


kronholm(Posted 2007) [#9]
If you can afford an AppleTV, you should be able to compile on it if you "hack" it and install MacOSX. Just saying, if you didn't know, it's the cheapest solution there is, next to "hack-installing" MacOSX on your own PC, which will most likely fail :)


N(Posted 2007) [#10]
I don't get paid for this stuff, so don't count on it. >_>


Michael Reitzenstein(Posted 2007) [#11]
Unfortunately, I think TCC is x86 only which is kind of a shame. So while it'll work on Intel Macs, it won't work on anything G4/G5.


Who was John Galt?(Posted 2007) [#12]
Good to see you back, Cower!


Kev(Posted 2007) [#13]
Intresting i will take a look at this sometime later. thanks Noel.


You do not have to output an EXE, DLL, OBJ, etc.



without looking is it even possable to build an .exe using TinyCC this way?

kev


N(Posted 2007) [#14]
Yes.


Nigel Brown(Posted 2007) [#15]
Noel,
Thanks this looks really usefull. Are you planning more examples?

Just found tcc_add_file( s, "MyProg.c" ) :-)


GW(Posted 2007) [#16]
HELP!
I'm having a problem with this module blowing up after a number of compilations.
Everything seems to be cleaning up after itself after each compilation and run but i'm consistently getting a memory exception at the same iteration number [509].

Can anyone help me with this issue? Noel?

Here is a small example:
Import Cower.TCC

Function add(a%, b%)
    Return a + b
End Function

Global MyProgram$ = "int fib(int n)~n"+..
"{~n"+..
"    if (n <= 2)~n"+..
"        return 1;~n"+..
"    else~n"+..
"        return fib(n-1) + fib(n-2);~n"+..
"}~n"+..
"~n"+..
"int foo(int n)~n"+..
"{~n"+..
"    //printf(~qHello World!\n~q);~n"+..
"   // printf(~qfib(%d) = %d\n~q, n, fib(n));~n"+..
"    //printf(~qadd(%d, %d) = %d\n~q, n, 2 * n, add(n, 2 * n));~n"+..
"    return 0;~n"+..
"}~n"

Global Count%
For Local I = 0 To 1000
	'Local m = MilliSecs()
	Local val%
	Local func%(a%)
	
	Local s@ Ptr = tcc_new()
	Print "~t " + tcc_set_output_type( s, TCC_OUTPUT_MEMORY )
	Print "~t" + tcc_compile_string( s, MyProgram )
	Print "~t" + tcc_add_symbol( s, "add", Int(Byte Ptr(add)) )
	Print "~t" + tcc_relocate( s )	'<- Error here
	Print "~t" + tcc_get_symbol( s, val, "foo" )
	func = Byte Ptr(val)
	Print "~t" + func(32)
	Print "~t" + tcc_delete(s)
	s = Null
	
	GCCollect
	'Print MilliSecs()-m
	Print GCMemAlloced()
	Print "done.. " + count + "~n~n"
	count:+1
Next



N(Posted 2007) [#17]
No errors here. What versions of BlitzMax, MinGW, etc. are you using? Have you modified the module? Have you done anything that can explain why you've even gotten errors before this?


GW(Posted 2007) [#18]
I was using 1.20 and just updated to 1.26 + syncmods.
I sure I'm using the older minGW (3.4.2) as I have not updated it, but I haven't done any kind of module rebuild and I'm using your module exactly as it was downloaded.
Im still getting the error on the 509th iteration regardless of the size or type of C function I am calling.
I put the contents of the DLL directory from your download into my Bmax\tmp dir.

Since I've just updated to 1.26 and your able to test the sample and run 1000+ iterations, what do you recommend?

[edit]I should add that for me its always bombed on me at the 509th iteration. So its not a situation that something has changed


N(Posted 2007) [#19]
I recommend you update your copy of MinGW.


GW(Posted 2007) [#20]
I did that. also I rebuilt all the modules.

Can you confirm that you can create run and destroy tcc memory programs over 1000 times? I first need to know if its just a problem with my setup or everyone has it.

Do you see anything in the sample program i posted that would indicate why the 'relocate' function would fail after so many iterations?

TCC is absolutely perfect for a current project I'm working on. I'm very desperate to get it working and would appreciate any assistance.


N(Posted 2007) [#21]
Considering I've run it about a dozen times now with no errors, yes, it works fine here. The module has not been changed since I created it (considering TCC hasn't changed either, and probably won't). My best guess is you're doing something that your machine can't handle. Since you haven't actually given me any debugging output for your application, your machine specifications, etc., there's very little I can do to assist you.

In short, unless you can give me something useful to work with, I can't do anything for you. My best guess here is that the code you posted above and the code you're actually using are different, and somewhere in your own code is a stupid mistake.


GW(Posted 2007) [#22]
Everything works fine until i get 'Unhandled Memory Exception Error'
My specs are: winxpPro sp2 AMD 64X2 dualcore 4400 and 2gb ram [300gb free disk].
I'm not postulating that your module is to blame for the error. Mainly i'm looking for assistance from someone who has more experience with C and this lib than I do. I've been working on this error non-stop for close to 2 days now.
Here are a few of my observations:
If im calling a C function the error consistantly fails at the 509th iteration on the function 'tcc_relocate'.
If the c program just has a 'main' function and i'm calling 'tcc_run' the program fails on that command after the same number of iterations.
Using 'printf' from the c function has no effect on the error.
the return type of the C function has no effect on the error.

Here is another example program. i'm bypassing the mod and the lib file altogether. Also the DLL should not be remaining in memory after each iteration. So my conjecture is that something outside of the GC is being accumulated, but I'm not sure if its in the DLL or in Bmax.

Is there something related to ZStrings that need to expunged manually??
Is there something about the 'tcc_relocate' command thats leaking memory?

Strict
Framework brl.basic
Import pub.win32
'''Import Cower.TCC

SeedRnd(MilliSecs())


Const TCC_OUTPUT_MEMORY% = 0
Const TCC_OUTPUT_EXE% = 1
Const TCC_OUTPUT_DLL% = 2
Const TCC_OUTPUT_OBJ% = 3

' Output formats
Const TCC_FORMAT_ELF% = 0
Const TCC_FORMAT_BINARY% = 1
Const TCC_FORMAT_COFF% = 2


Global count% 

Global  MyProgram$

MyProgram$ = "float arr(float a)~n"+..
"{~n"+..
"	printf(~qhello\n~q);~n"+..
"	return a*2.0+1.0;~n"+..
"}~n"


For Local I = 0 To 10000
	Test
	GCCollect
Next	
	
	
Function Test()
	Local Handle = loadlibrarya("E:\Dev\BlitzMax\tmp\libtcc.dll")
		
		Local tcc_new@ Ptr() = getprocaddress(handle,"tcc_new")
		Local tcc_delete(state@ Ptr) = getprocaddress(handle,"tcc_delete")
		'Local tcc_enable_debug(state@ Ptr) = getprocaddress(handle,"tcc_enable_debug") '// This function does not exist in the dll//
		Local tcc_set_error_func(state@ Ptr, opaque@ Ptr, error_func(opaque@ Ptr, msg@ Ptr)) = getprocaddress(handle,"tcc_set_error_func")
		Local tcc_set_warning%(state@ Ptr, name$z, value%) = getprocaddress(handle,"tcc_set_warning")
		Local tcc_add_include_path%(state@ Ptr, pathname$z) = getprocaddress(handle,"tcc_add_include_path")
		Local tcc_add_sysinclude_path%(state@ Ptr, pathname$z) = getprocaddress(handle,"tcc_add_sysinclude_path")
		Local tcc_define_symbol(state@ Ptr, pathname$z) = getprocaddress(handle,"tcc_define_symbol")
		Local tcc_undefine_symbol(state@ Ptr, pathname$z) = getprocaddress(handle,"tcc_undefine_symbol")
		Local tcc_add_file%(state@ Ptr, filename$z) = getprocaddress(handle,"tcc_add_file")
		Local tcc_compile_string%(state@ Ptr, buf$z) = getprocaddress(handle,"tcc_compile_string")
		Local tcc_set_output_type%(state@ Ptr, output_type%) = getprocaddress(handle,"tcc_set_output_type")
		Local tcc_add_library_path%(state@ Ptr, path$z) = getprocaddress(handle,"tcc_add_library_path")
		Local tcc_add_library%(state@ Ptr, libraryname$z) = getprocaddress(handle,"tcc_add_library")
		Local tcc_add_symbol%(state@ Ptr, name$z, value%) = getprocaddress(handle,"tcc_add_symbol")
		Local tcc_output_file%(state@ Ptr, filename$z) = getprocaddress(handle,"tcc_output_file")
		Local tcc_run%(state@ Ptr, argc%, argv@ Ptr) = getprocaddress(handle,"tcc_run")
		Local tcc_relocate%(state@ Ptr) = getprocaddress(handle,"tcc_relocate")
		Local tcc_get_symbol%(state@ Ptr, value% Var, name$z) = getprocaddress(handle,"tcc_get_symbol")
	
		Local val%
		Local func#(a#)
		
		Local s@ Ptr = tcc_new()	
			
		tcc_set_output_type( s, TCC_OUTPUT_MEMORY )
		
		Print "CS> " + tcc_compile_string( s, MyProgram )
		
		'// Comment out these next 5 lines and no error occurs//
			
			'// randomly call relocate and the function//
			'// everytime relocate is called it accumulates the error //
			'// now instead of failing on 509th iteration, it is now roughly 900 to 1100
			If Rand(1,2) > 1 Then tcc_relocate( s )

			tcc_get_symbol( s, val, "arr" )	'// Val is 0 if relocate was not called //
			
			Print val + "  <func"
			
			func = Byte Ptr(val)
			
			If val > 0 Then Print "arr= " +  func(32.1)
		'//
		
		tcc_delete(s)
		s = Null
		func = Null
		Handle = 0
		GCCollect

		Print "count= " + count + "~n"
		count:+1
End Function 	



N(Posted 2007) [#23]
No idea what's wrong yet, but at least I've been able to reproduce it now. You're apparently pretty stupid for doing the random relocate thing though, since that makes it harder to pinpoint problems by reducing the chance of getting a single environment where the error can be reproduced.


GW(Posted 2007) [#24]
The conditional relocate was only to illustrate that the error is accumulating. I've written at least 500 variations of this testbed program in the last 2 days. What I posted above was just one example. Just remove the 2 if statements.

As a side note, I was finally able get it working in a way.
I had to import 'tcc.c' directly into my bmax program and make some small tweaks to it and config.h

it now works without blowing up, but there seems to be some weirdness.
when passing a bmax float array to a c function, the values seem to offset by 6 places.
EX: if BMaxArray# = [1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, 9.1]
after passing to the c Function the array argument is now:
[0,0,0,0,0,0, 1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, 9.1]
I'm not sure why that is. Does BMax arrays have a hidden header or something?? Is it better to use Banks or something?

Also, if your C program uses a function like 'sqrt()' TCC will complain that its an unidentified symbol. Adding an include to math.h has no effect, but if you write your own 'sqrt' function in the same script TCC will complain about a duplicate reference. The only way I was able to get it working is to define sqrt in Bmax and add the symbol to the C program. Its a workaround, but better would be to find out why its complaining. however my knowledge of C is not that extensive.

Let me know how goes your testing with the Dll:Relocate().


N(Posted 2007) [#25]
How are you passing the array? What code are you using? You're aware that arrays are objects and not just blocks of memory containing only the array's values, right?

Also, if your C program uses a function like 'sqrt()' TCC will complain that its an unidentified symbol. Adding an include to math.h has no effect, but if you write your own 'sqrt' function in the same script TCC will complain about a duplicate reference. The only way I was able to get it working is to define sqrt in Bmax and add the symbol to the C program. Its a workaround, but better would be to find out why its complaining. however my knowledge of C is not that extensive.
This is probably because you're not running your application in an area that TCC can access its required header and object files.


GW(Posted 2007) [#26]
You're aware that arrays are objects and not just blocks of memory containing only the array's values, right

I suspected that might be the trouble. its not as if the bmax docs mention it anywhere.
you're not running your application in an area that TCC can access its required header and object files

the lib and include directories are alongside the '.a' file and the exe.
Ive also used the commands "tcc_add_include_path" and "tcc_add_sysinclude_path" with no effect.
I belive its getting the include file because otherwise you get an error about the file not being found.

whats strange is that adding the line:
" printf(~q%f~q,sqrt(n));~n"+..
to the foo function of your original example does not give an error, however the result is always 0.0 for any argument.


Dreamora(Posted 2007) [#27]
Don't mention it, correct.
But as they are referenced and not by value it is obvious ... or if you ever have tried :Array :)


AnniXa(Posted 2011) [#28]
is there any chance to get this download again? :(


BlitzSupport(Posted 2011) [#29]
It's still on Noel's site, here:

http://www.spifftastic.net/mod/


N(Posted 2011) [#30]
.

Last edited 2012