Build system

Community Forums/Monkey2 Talk/Build system

marksibly(Posted 2015) [#1]
Monkey1's 'build everything all the time' system has to go - translation time isn't so bad, but build times with a monolithic source code file can be terrible, esp. with mingw!

So monkey2 will translate modules to individual source files, which will only be rebuilt if they have changed, ala blitzmax. The entire program will probably still need to be parsed, but this process should be even faster in monkey2 than it is in monkey1.

Monkey2 will also include some kind of variant on blitzmax's ultra flexible file import system, where you can just import source/object/library files in-code, and they will be magically compiled/linked for you.


Danilo(Posted 2015) [#2]
marksibly wrote:
esp. with mingw!

I recommend to let MinGW go and include LLVM Clang compiler (compatible to GNU C++).

http://llvm.org/
Clang is an "LLVM native" C/C++/Objective-C compiler, which aims to deliver amazingly fast compiles (e.g. about 3x faster than GCC when compiling Objective-C code in a debug configuration), extremely useful error and warning messages and to provide a platform for building great source level tools. The Clang Static Analyzer is a tool that automatically finds bugs in your code, and is a great example of the sort of tool that can be built using the Clang frontend as a library to parse C/C++ code.

Clang is included in the main LLVM package: http://llvm.org/releases/download.html#3.6.0

About Clang:
http://clang.llvm.org/
Clang is considered to be a production quality C, Objective-C, C++ and Objective-C++ compiler when targeting X86-32, X86-64, and ARM (other targets may have caveats, but are usually easy to fix). If you are looking for source analysis or source-to-source transformation tools, clang is probably a great solution for you. Clang supports C++11, please see the C++ status page for more information.

Clang supports C++11 and C++14, and parts of C++1z:
http://clang.llvm.org/cxx_status.html

http://clang.llvm.org/docs/UsersManual.html#command-line-options

Producing 32bit/64bit assembly files:
// compile 32bit:
// clang -S -m32 test.c -o test-32bit.S
// compile 64bit:
// clang -S test.c -o test-64bit.S

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

int main() {
    uint64_t a = 0ULL, b = 0ULL;
    scanf( "%lld %lld", &a, &b );
    printf( "64-bit division is %lld\n", a / b );
    return EXIT_SUCCESS;
}



marksibly(Posted 2015) [#3]
Wow, clang looks like it's come a long way!

I'll give it a shot and if I have any problems I'll switch back to mingw.


Danilo(Posted 2015) [#4]
Using precompiled headers (in addition to precompiled .obj/.lib) with the MX2 library system (Mojo, Windows.h, GLFW.hpp etc.) should also speed-up C++ compilation.


EdzUp(Posted 2015) [#5]
Also can we have the ability like max to make modules so we dont have to release source code with stuff ala monkeyX


Pharmhaus(Posted 2015) [#6]
The thing that gets quite complicated in Bmax is that no cross references for modules are allowed.
This was really a joy moment to me using monkey when cross importing everything without any problems just worked.


Derron(Posted 2015) [#7]
The thing that gets quite complicated in Bmax is that no cross references for modules are allowed.


Because you add circular dependencies then ...
a requires c
b requires a
c requires b
a imports c, which imports b which tries to import a which already tries to import c...

To avoid such things, you could put all the code lines into one big file - so none of them gets "precompiled" without the other. I assume Monkey is doing such a thing - "Include" instead of "import".

Maybe the compiler-chain would be able to "try" to recognize such circular dependencies and "include" them in that case - so in the worst case your whole 100+ files project will become a single big (virtual) file. Just to assist the developer in his lazyness of creating base classes:

a requires cBase
b requires aBase
c requires bBase
aBase, bBase and cBase do not require other files

There are surely other approaches to fix that problem, maybe it is better to have such things "automated" but I am not sure about this.

I assume: for "Monkey 2" the BlitzMax-behaviour would come back.

bye
Ron


Samah(Posted 2015) [#8]
@Derron: Because you add circular dependencies then ...

There are no circular dependences in Monkey, because all of the required source code is parsed before it even begins to build up identifier references.
There's no reason Monkey 2 couldn't do the same thing.


degac(Posted 2015) [#9]

a requires c
b requires a
c requires b



I still don't understand why the compiler simply doesn't take note (in a list,map etc) of all 'import statement' and check it. It seems a simple work for a pre-compiler. (or maybe I missed something quite obviuos!)


Skn3(Posted 2015) [#10]
Wooohooo! It has always been a massive ballache in monkey trying to work around the order of compilation. Separating everything and declaring dependencies is a massive win! What is the limitation of this? Currently in some of my modules (unreleased ones I think mostly) in native code I have to hack around the fact that everything is dumped into one file. What are you plans for native code in the build process? If I write some module that requires the internal image object, in native code what will the process be? The same question for 3rd party modules?


Pharmhaus(Posted 2015) [#11]

Wooohooo! It has always been a massive ballache in monkey trying to work around the order of compilation.


Indeed interfacing with C++ is a pain in Monkey.




declaring dependencies is a massive win!


Only when Interdependencies are allowed. Once the code base is big enough one will realize how constraining Bmax no-interdependency module system is. Even with a multilayer architecture once the magic number of lines is reached thinks get ugly. Like really ugly.


ziggy(Posted 2015) [#12]
@Pharmhaus: While I agree that allowing cross referencing between modules is very nice, take into account that having dependency injection in Monkey based on interfaces can help a lot mitigating this situation.


Samah(Posted 2015) [#13]
@Pharmhaus: Indeed interfacing with C++ is a pain in Monkey any language.

Fixed that for you.


Derron(Posted 2015) [#14]
There are no circular dependences in Monkey, because all of the required source code is parsed before it even begins to build up identifier references.
There's no reason Monkey 2 couldn't do the same thing.


I did not say such a thing (maybe my English isnt the best...). I replied to "Bmax not allowing cross references" - and why BlitzMax does not allow that - while Monkey seems to enable such a coding style. I also explained that it might be the case because Monkey "includes" everything, instead of precompiling individual files (like BlitzMax does it).


bye
Ron


nullterm(Posted 2015) [#15]
Would it be possible to also get the ability to run custom scripts with-in Monkey code? Like you can with run script build phase in Xcode or makefile projects in Visual Studio.

#SCRIPT "python CollectAssetsToData.py"


I have a few projects that have scripts I run manually to copy/convert data files to game ready format, life would be so much easier if it was part of the F5 to Run / F7 to Build process.


ziggy(Posted 2015) [#16]
@nullterm: This can be done in Jungle Ide (and in some other IDEs). I don't think it is specifically a language feature but more of a build toolchain setting.


degac(Posted 2015) [#17]
@nullterm
interesting, but it seems more an IDE features than a language feature.
And the second question is: what type of 'language-scripts'?
What happens if I copy/use your source code (for example) and there's a #SCRIPT in a language I don't have installed?
Adding another level of dependency brings to another complication and MX2 seems to aim to 'simplify/reducing the complexity.

ps: of course Mark could decide that MX2 setup installs - for example - python or whatelse as 'required' component for using MX2.
Brucey did something similar with his own BMK version based on LUA to manage the various configuration (I think!).


marksibly(Posted 2015) [#18]
How about inline '>' to execute a shell command? eg:

>g++ -o ${EXE_PATH} ${SRC_PATH}
#if CONFIG="debug"
>gdb ${EXE_PATH}
#endif

...it's up to the user to make sure g++, gdb (or python or whatever) is installed.

I think it would be nice to have sort of control over the build process inline, for pre-build/post-build steps etc.


nullterm(Posted 2015) [#19]
@degan
It's not IDE though, to me. IDE is a tool for editing and launching build process & app.
Build scripts are a part of the build process itself, the same way the compiler parses code and spits out an executable. It takes raw assets and converts them to game ready format.
One example I've used for iOS games is a .py script that packaged up all my .lua and other game config scripts into one .zip like file. So the game just needed one quick load at the start, and then pull out script from memory.

@marksibly
I'm not picky about syntax. > would work too.
Pre-build could also allow for script generated .monkey code. Like reading an event definition .txt or .json file, and then spitting out all the necessary struct/class code.

Anytime I can integrate makefile and/or python into a build process, there's no end of cool stuff you can do. And make your own process alot more efficient too.


dawlane(Posted 2015) [#20]
@marksibly: I like the idea of being able to pass parameters directly to the compiler tool chain. Maybe the translater can parse the source files for compiler directives to build it's own make file to suite the situation instead of having a generic make file that has to be edited manually.


degac(Posted 2015) [#21]
@nullterm

I have no doubt that launching scripts for trivial and boring works to speed up building, compiling etc is very useful and I think needed: there are so many steps to 'make' an application (converting assets, compiling, signing, packing etc) that everything is helpful.
But - as Mark stated - why just dont' consider to have one 'default' (or supported/preferred) script system for MX2? So it could be used - safely (as all the users should have it installed) - to manage MX2 compilation/building/update etc. (just thinking on Mojo and the problems with different sound/gfx format
Of course this doesn't mean that single user can have other script languages for their own use, accessed with > or whatelse.


Skn3(Posted 2015) [#22]
How about inline '>' to execute a shell command? eg:

>g++ -o ${EXE_PATH} ${SRC_PATH}
#if CONFIG="debug"
>gdb ${EXE_PATH}
#endif

...it's up to the user to make sure g++, gdb (or python or whatever) is installed.

I think it would be nice to have sort of control over the build process inline, for pre-build/post-build steps etc.


Could be risky unless you build in some kind of warning /security system into the compiler. Someone could create malicious source files otherwise.

I like the idea but not so sure about using a >. Will there be a proper preprocessing? Eg maybe it would be better to make something like #exec("gcc ${blah}") so we just introduce new preprocess commands instead of a whole new symbol..


ziggy(Posted 2015) [#23]
Being honest, using > for even more things seems horrible to me. It's a comparison operator, and also a generics identifier. I think it's better to be as consistent as possible and give a single responsibility to symbols. Add to this that there are lots of symbols we're not currently using, such as {, |, \, etc.
In a worst case situation, we could define a digraph such as >>, <<, >_ or anything else instead of using a single >
If you ask me, a keyword would be much more clear here, something as Exec, Run, Shell or whatever. Much more readable and easy to follow when you haven't still read the manual to find out that a comparison operator at the begining of a line is a shell command.

A "function" would be much better as you will be wanting to get a bit of control on commands successful completion (check "errorlevel"), and then things will start to look uggly:
#If >gcc -E main.c > main.i <> 0 then #Error "Can't compile"


Much better
#If Shell("gcc -E main.c > main.i") <> 0 Then #Error "Can't compile"


But I still think that having this at language level is not very clean.


nullterm(Posted 2015) [#24]
@Skn3, Monkey already has os.Execute that can run external processes at runtime. How would an external process at build time be different?
Risk, sure? But the same risk/trust that most every language and 3rd party libraries (Monkey, C++, Python, etc) on the planet.


ziggy(Posted 2015) [#25]
@nullterm: A runtime execute command needs the program to be compiled. AFAIK, we're talking about a compile-time execution command, something that is evaluated by the compiler, and executed only during compilation. As instance, an external tool to convert images, a texturepacker call, a preprocessor built in another language, etc. Seems nice, but I honestly do not see the point of including what I think a pre-build commands into the build toolchain. this all should be executed first, and does not need to be defined at language level. Brucey did very well at BlitzMax by introducing Lua scripting in the build tool, and it did not require any blitzMax language change, because they're different things! In my opinion, this will make language syntax a bit dirty, for no good reasons.


nullterm(Posted 2015) [#26]
@ziggy, if there was a .monkey-script (as an example, not proposing that) that got execute as a pre-build, that would be just as awesome.
It doesn't need to be part of the syntax, but I do like the way Monkey is now with build options (like controlling the GC) in the app's .monkey file. It also means optional or different steps depending on configuration, like #if debug.


Nobuyuki(Posted 2015) [#27]
agreed with both ziggy and nullterm. Perhaps the scripts used along the trans-compilation build chain can be "exposed" more. We have things like the android /templates/ folder, we have config.monkey, and (for jungle IDE users), a way to pre-execute some commands before building. But they're not all consolidated into one easily-exposed place when creating a build. Perhaps all of these things can be consolidated into an easy-to-access pre-build script? Something which specifies what template modifications are to be used, what executes before compilation, and what executes after compilation. This would allow a lot more things, not just texture packing, but different build configurations, automatic resource editing (for platform-specific icons), package injection and etc. Trans builds a default version of the script from its built-in target templates, and can be changed a similar way we change config.monkey.

note: I often specify preprocessor directives at the top of my main source file rather than in config.monkey because I will occasionally delete build-specific folders. It would be nice if build scripts and their associated templates lived in a place that wasn't subjected to being deleted upon a clean build. Maybe something like ./[ENTRYPOINT].config[VERSION]/[TARGET]/ ?


marksibly(Posted 2015) [#28]
> #If Shell("gcc -E main.c > main.i") <> 0 Then #Error "Can't compile"

Nice! Being able to #If is important. But I'm starting to think script code in '#' is not nice! You can do pre-build stuff with it, but not post build, and it'll slow down the parser/semanter when it only needs to slow down build.

> introducing Lua scripting in the build tool

I'd like to avoid additional languages if possible. At the least, I think it should be possible to do a lot without resorting to LUA etc...

Things are actually reaching an interesting point here as I just successfully built/run my first 'hello world'! Here's the output c++ code:

// HEADER FILE test.h
namespace mojo{
  static void main();
}

// SOURCE FILE test.cpp
#include <cstdio>
#include "test.h"
namespace mojo{
  void main(){
    puts("Hello World!");
  }
}
int main(){
  mojo::main();
  return  0;
}


The next step will be to tackle import etc, so the build process is gonna become a top priority.

Here's my basic idea of what the build process could look like - pretty much just what BMK does, reduced down to a sane chunk of code:

Global REBUILD_ALL:Bool
Global OUTPUT_FILE:String
Global SOURCE_FILES:String[]
Global LIBRARY_FILES:String[]
Global COMPILER_OPTS:String[] 
Global LINKER_OPTS:String[]
Global PRE_BUILD:String[]
Global POST_BUILD:String[]

Function Build:Void()

	For Local pre:=Eachin PRE_BUILD
		Execute pre
	Next

	Local obj_files:=New StringStack

	For Local src:=Eachin SOURCE_FILES
	
		Local obj:=MungSourceFileToObjFileSomehow( src )	'generates a 'hidden' .o file for building
		
		If REBUILD_ALL Or FileTime( src )>FileTime( obj )
		
			Execute "g++ -c "+COMPILER_OPTS.Join( " " )+" -o "+obj+" "+src
		
		Endif
		
		obj_files.Push obj
		
	Next
	
	Execute "g++ "+LINKER_OPTS.Join( " " )+" -o "+OUTPUT_FILE+" "+obj_files.Join( " " )+" "+LIBRARY_FILES.Join( " " )
	
	For Local post:=Eachin POST_BUILD
		Execute post
	Next

End


I think just using monkey1's #BLAH+= system would provide a lot of flexibility here - but would it be enough? One thing it's probably lacking is the ability to set CC_OPTS on a 'per .cpp file' or per module level...

#Import could 'deduce' what you're += to, eg:

#Import "test.cpp" 'same as: SOURCE_FILES+="${CURENT_DIR}/test.cpp"
#Import "Include/*.h" 'same as: COMPILER_OPTS+=" -I${CURRENT_DIR}/include"

I do think Import when used with extern files should have a '#' prefix, as it's part of the build system, not part of the language.

Thoughts?


Samah(Posted 2015) [#29]
@marksibly: At the least, I think it should be possible to do a lot without resorting to LUA etc...

Lua is a word, not an acronym. However I agree that bringing in another language is a bad idea.


therevills(Posted 2015) [#30]
@Samah: Lua is a word, not an acronym.


You should add that to your signature ;)

http://www.lua.org/about.html
"Lua" (pronounced LOO-ah) means "Moon" in Portuguese. As such, it is neither an acronym nor an abbreviation, but a noun. More specifically, "Lua" is a name, the name of the Earth's moon and the name of the language. Like most names, it should be written in lower case with an initial capital, that is, "Lua". Please do not write it as "LUA", which is both ugly and confusing, because then it becomes an acronym with different meanings for different people. So, please, write "Lua" right!



Nobuyuki(Posted 2015) [#31]

I think just using monkey1's #BLAH+= system would provide a lot of flexibility here - but would it be enough? One thing it's probably lacking is the ability to set CC_OPTS on a 'per .cpp file' or per module level...

#Import could 'deduce' what you're += to, eg:

#Import "test.cpp" 'same as: SOURCE_FILES+="${CURENT_DIR}/test.cpp"
#Import "Include/*.h" 'same as: COMPILER_OPTS+=" -I${CURRENT_DIR}/include"

I do think Import when used with extern files should have a '#' prefix, as it's part of the build system, not part of the language.

Thoughts?



I dig all of it. With the += and import stuff, one shouldn't have to dig into the build script in many cases, but having the flexibility available is fantastic.


marksibly(Posted 2015) [#32]
> one shouldn't have to dig into the build script in many cases

Trouble is, it isn't a script - it's monkey code that needs to be built into something itself.

Perhaps something simple like the above could be built into trans2, but you could run trans2 in a special mode that generated .cpp/.h files, but instead of building just dumped all the #VARs to a text file. This way, 3rd party IDE's and tools could provide their own alternative to the above.


dawlane(Posted 2015) [#33]
Personally I don't think that trans shouldn't do all the work and calling scripts should be left out of monkey code. I think that the build system needs to be more modular. Give trans the task of just converting the code and a control application that checks to see if there is a pretrans_HOST-SHELL-SCRIPT/postrans_HOST-SHELL-SCRIPT/configfile within a directory called CONTROL that resides in the projects own directory. The control application can then use the pretrans_HOST-SHELL-SCRIPT to create the build environment for the host ready for trans that will use the configfile to control the translation before passing it over to the compiler tools. The posttrans_HOST-SHELL-SCRIPT does the final clean up tasks. One thing that must be taken into account is if a script returns a value, so if the pretrans_HOST-SHELL-SCRIPT creates it's own compiler options to be passed on.


Skn3(Posted 2015) [#34]
I think just using monkey1's #BLAH+= system would provide a lot of flexibility here - but would it be enough? One thing it's probably lacking is the ability to set CC_OPTS on a 'per .cpp file' or per module level...


I think for most cases the += system was suitable however in monkey1 support for LIBS for example was limited to certain targets. This needs to be fleshed out.

There was also the issue of importing library/framework files from your module into the built project. For example I write a steam module and want to copy across headers/dlls/resources(images/sounds/data/etc) at build time. Currently support for this was limited in monkey1.

There is also another point where the += preprocessor doesn't really suffice, or at least not currently:
with iOS there are a ton of features we can extend into the app runtime but require access to the target's main app classes (viewcontroller/appdelegate). The same with Android. Monkey needs a way that a module can define injection/override code somehow into this process. Lets say for example a module needs to handle events on the appdelegate, but this isn't currently managed by Monkey2. Is there a smart way that Monkey2 can inject code in this scenario so Module developers can extend?


Ferdi(Posted 2015) [#35]
Sorry if this kinda derail what you all are discussing. But there were 2 build problems I had in Monkey X. Well more annoyance than problems.

1. Adding framework into xcode

The only way to add any “file.framework” is to open the xcode project file and put the framework in there.

This is the thread: http://www.monkey-x.com/Community/posts.php?topic=8326

I did some research, but this is a while back and it seems xcode command line is very limited. Anyway if we can add file.framework from Monkey code that will be great.

2. Adding a target requires a recompile of transcc

So my target was only for WINNT. So I can’t use the GLFW target. I had to modify the method IsValid and MakeTarget such that it is only for WINNT and to only use Visual C. I also had to modify builders.monkey to add my new target.

If transcc can detect there is a new target without recompiling, that will be great.


Skn3(Posted 2015) [#36]
Another thought to consider is what if we were able to modify the data/content import lists via processor vars. E.g.

This would prevent all images importing
#IMAGE_DATA_ITEMS = ""


This would add an item
#IMAGE_DATA_ITEMS += "c:\myimage.png;"


This would add an image import into a specified build subfolder
#IMAGE_DATA_ITEMS += "c:\myimage.png=subfolder\blah.png;"



Once monkey has processed these vars, it would split them into key/value pairs and process them one by one. The images/data that exists within the data folder of the project, would be added at the start of building. E.g.:
IMAGE_DATA_ITEMS = "
${SRC}\data\image1.png=image1.png;
${SRC}\data\image2.png=image2.png;
${SRC}\data\image3.png=image3.png;
${SRC}\data\image4.png=image4.png;
"


These data list vars would be different to the current monkey1 data filters. They would still exist, but their job would just be to select which data gets added before build as demonstrated above.

You could even then have content pipeline commands executed on each of the lists. These could be generic open source command line tools(texture packing, sound sample, encrypting) or even allow the content processors to be written in monkey2 code via preprocessor.

#PRE
'each pre block is executed in its own context so we can import module code?
Import skn3.atlas

'set the image processing var
IMAGE_DATA_PROCESS = PackTextures()

'function Monkey2 will call as specified in IMAGE_DATA_PROCESS
Function PackTextures:String(images:String)
	Local items := images.Split(";")

	'iterate over all content imports
	Local atlas := new Atlas
	For Local item := eachin items
		Local pieces := item.Split("=")
		atlas.AddImage(pieces[0])
	Next
	
	'pack the ones we want into textures
	atlas.Save("${TEMP}\atlas.png")
	
	'return the content imports that we still want to copy across to built project
	'Monkey2 will replace the list of imports with whatever was returned here
	'this means that if we have packed images, and dont now want them imported, we can skip them by leaving them out of the return
	return "${TEMP}\atlas.png=atlas.png;"
End
#END


Content pipeline processor execution should also be run on the entire DATA_ITEMS var This way for example you could automatically pack all images into a texture.

I like the XNA/mono content pipeline in that it lets you process your content automatically. Monkey/mojo has a very basic version of this already with the way it precompiles image dimensions. It would be awesome to expand this to developers for Monkey2.

The inline preprocessor script shown above would be nice, no idea if its practical though? I think monkey could definitely still benefit on letting preprocessor vars alter the import data lists though...