We need to change how the preprocessor works

Monkey Forums/Monkey Programming/We need to change how the preprocessor works

ImmutableOctet(SKNG)

(Posted 2014) [#1]

EDIT: Thanks to Mark, as of V82(A), the preprocessor now works the way I was describing in this thread.

So, I remember bringing this up in the past, but now it's getting to be a bigger issue. Basically, the preprocessor deals with the current module from start to finish before even considering its imports. This is counterintuitive, and it has been an annoying drawback to Monkey for some time now.

The way I see it, the preprocessor could easily (At least from a compatibility standpoint) be changed to deal with modules as the compiler sees them. I can't think of any code, nor can I find any code that would be broken with this kind of change. In fact, I'm pretty sure most Monkey users already thought it worked like this (Which it does, but module imports aren't considered currently). Everyone's (As far as I've seen or worked with) preprocessor directives are at the top of their files or expecting the import to be taken into account, so we shouldn't have an issue. Even the 'mojo' module itself technically wouldn't have an issue. Plus, even if they aren't, they really should be at the top of the file. And it's not like the original idea of configuration would be broken in any way by this change.

I've been writing my own public modules for a while now, and they need to be able to check for optional compatibility with each other. I can't expect every single person who downloads one of my modules to download them all, so I use the preprocessor to check for these things.

I pretty much need the preprocessor to process modules as they get imported, not after the current module has been processed. This would work exactly like the standard C and C++ preprocessors, and it would make my life, as well as many others' a lot easier using the preprocessor.

My only real way of working around this is a very convoluted multi-import hack (Which will only work in rare situations, and end up making my code incredibly messy).

Well, either that, or a simple 'MODULEAVAILABLE' thing for imports (I'm pretty sure this doesn't exist), but that doesn't fix everything, just the main issue at the moment.

I'm already expecting the "This was by design" line, and I'm already aware that it is. Just because you couldn't see this far ahead when designing Monkey, that doesn't mean we can't change how things work. At least consider the idea. I'm not even asking you to change this, just your thoughts on it. If you're okay with this, but don't have time to implement it, I'll likely end up modifying 'Trans' and making a pull request. But I'm not going to go out of my way modifying Monkey again without a response first.

marksibly

(Posted 2014) [#2]

> I can't expect every single person who downloads one of my modules to download them all, so I use the preprocessor to check for these things.

Can you provide an example of what you're having problems with and what you'd like to achieve? I can't quite see how modifying the preprocessor achieves the same as something like:

#If ModuleExists( brl.filesystem )
#Endif

(assuming this 'effect' is what you're after).

Changing the preprocessor would introduce subtle issue with cyclic imports (ie: where a imports b imports c imports a etc) and is not something I'd want to attempt without good reason. ModuleExists() on the other hand would be easier to add, and have less chance of weird side effects.

Finally, the original goal of Monkey was for each module to be compilable in isolation, ala Java, hence the preprocessor approach. I ran out of time on this front (and the preprocessor has morphed into something a bit uglier along the way) but I still do not like the idea of just embracing C's 'spaghetti' preprocessor approach.

EdzUp

(Posted 2014) [#3]

I think the way that all other BRL languages do imports and includes would be OK, although a moduleexists command would be welcomed as it would make code a little less easier to break :)

ImmutableOctet(SKNG)

(Posted 2014) [#4]

Modules are only preprocessed once already, so I don't see how cyclic imports would be an issue. The standard C and C++ preprocessor(s) do this with "#define NAMEHERE_H" and "#pragma once", something Monkey just assumes from what I can see. Now, the real issue comes from the preprocessor going into another file before the current one is finished. Then, if that module we're importing decides it wants the main functionality of the first module, it would cause a loop. The answer to this is to not preprocess what has already been done, then continue where it left off on the next import, then follow these rules recursively until finished. That should theoretically work. It's probably best for me to look into this idea further and modify the 'trans' source to try and get this running.

In fact, technically, generating multiple versions of the module based on the preprocessor at the time of importing would be the most favorable; but that's insanely ridiculous, not to mention slow and bulky if not done exactly right. (I don't recommend this approach)

Since Monkey doesn't care if a module doesn't exist from an import standpoint, if it were to preprocess the modules as they're imported, I could just check 'INSERMODULENAMEHERE_IMPLEMENTED' (And any other related preprocessor variables) and call it a day. We should probably look into this theory further, but the 'ModuleExists' thing might be a good idea regardless.

I think you're a bit off with your idea of modular programs in a more realistic sense. The point is not to be able to compile or write everything separately, it's to be able to break the program into modules that can easily be put together as minimally as possible (Imports or not). If it were about everything compiling on its own, we'd reinvent the wheel each time. The entire reason I want this change is so I don't have to reinvent the wheel when my modules are imported in the wrong order.

I'm not asking for the design of the C/C++ preprocessor, I'm asking for standard behavior akin to it; i.e. modules get processed in the order in the same way a program would call a function.

Here's the basic idea of what I want to do:

module01.monkey (Build file):

Strict

Public

#Rem
	PLEASE READ THE COMMENTS OF ALL THREE FILES TO FULLY UNDERSTAND THE ISSUE.
#End

' Preprocessor related:
#MODULE_01_IMPLEMENTED = True

' Imports:
Import module02

' Functions:
Function Main:Int()
	Print("'module01':")
	OutputPreprocessorState()
	
	Print("'module02':")
	module02.OutputPreprocessorState()
	
	Print("'module03':")
	module03.OutputPreprocessorState()
	
	' Return the default response.
	Return 0
End

Function OutputPreprocessorState:Void()
	#If MODULE_01_IMPLEMENTED
		Print("MODULE_01_IMPLEMENTED: True")
	#Else
		#Rem
			This will only happen if that preprocessor-variable at the top of the file is not defined.
			(Which it will always be in this case)
		#End
		
		Print("MODULE_01_IMPLEMENTED: False")
	#End
	
	#If MODULE_02_IMPLEMENTED
		Print("MODULE_02_IMPLEMENTED: True")
	#Else
		' This will happen with the current preprocessor.
		Print("MODULE_02_IMPLEMENTED: False")
	#End
	
	#If MODULE_03_IMPLEMENTED
		Print("MODULE_03_IMPLEMENTED: True")
	#Else
		' This will happen with the current preprocessor.
		Print("MODULE_03_IMPLEMENTED: False")
	#End
	
	Return
End

module02.monkey:

Strict

Public

' Preprocessor related:
#MODULE_02_IMPLEMENTED = True

' Imports:

' This won't change the state of 'MODULE_01_IMPLEMENTED' as it won't be preprocessed until after this module.
' That, and in this example, it's already defined. For this reason, it will be shown as 'True'.
' If this module ('module02') was the first module, 'MODULE_01_IMPLEMENTED' would not be defined.
Import module01

' This won't change the state of 'MODULE_03_IMPLEMENTED' as it won't be preprocessed until after this module.
' That, and in this example, it's already defined. For this reason, it will be shown as 'True'.
' If this module ('module02') was the first module, 'MODULE_03_IMPLEMENTED' would not be defined.
Import module03

' Functions:

' This is the same as the 'module01' module's version, but I can't exactly use
' that one for this example (At least with the current behavior of the preprocessor):
Function OutputPreprocessorState:Void()
	#If MODULE_01_IMPLEMENTED
		Print("MODULE_01_IMPLEMENTED: True")
	#Else
		#Rem
			This will only happen if this is the primary file, otherwise, this will be defined.
			(This has nothing to do with the import, which is my main problem)
		#End
		
		Print("MODULE_01_IMPLEMENTED: False")
	#End
	
	#If MODULE_02_IMPLEMENTED
		Print("MODULE_02_IMPLEMENTED: True")
	#Else
		#Rem
			This will only happen if that preprocessor-variable at the top of the file is not defined.
			(Which it will always be in this case)
		#End
		
		Print("MODULE_02_IMPLEMENTED: False")
	#End
	
	#If MODULE_03_IMPLEMENTED
		Print("MODULE_03_IMPLEMENTED: True")
	#Else
		#Rem
			This will happen with the current preprocessor.
			
			If my idea was applied, this would only happen if the
			'module03' module's directive wasn't there. (Which means it'd preprocess the imports as they come along)
		#End
		
		Print("MODULE_03_IMPLEMENTED: False")
	#End
	
	Return
End

module03.monkey:

Strict

Public

' Preprocessor related:
#MODULE_03_IMPLEMENTED = True

' Imports:

' This won't change the state of 'MODULE_01_IMPLEMENTED' as it won't be preprocessed until after this module.
' That, and in this example, it's already defined. For this reason, it will be shown as 'True'.
' If this module ('module03') was the first module, 'MODULE_01_IMPLEMENTED' would not be defined.
Import module01

' This won't change the state of 'MODULE_02_IMPLEMENTED' as it won't be preprocessed until after this module.
' That, and in this example, it's already defined. For this reason, it will be shown as 'True'.
' If this module ('module03') was the first module, 'MODULE_02_IMPLEMENTED' would not be defined.
Import module02

' Functions:

' This is the same as the versions in 'module01' and 'module02', but I can't exactly use
' either of those for this example (At least with the current behavior of the preprocessor):
Function OutputPreprocessorState:Void()
	#If MODULE_01_IMPLEMENTED
		Print("MODULE_01_IMPLEMENTED: True")
	#Else
		#Rem
			This will only happen if that variable at the top is not defined.
			(Which it will always be in this case)
		#End
		
		Print("MODULE_01_IMPLEMENTED: False")
	#End
	
	#If MODULE_02_IMPLEMENTED
		Print("MODULE_02_IMPLEMENTED: True")
	#Else
		#Rem
			If you haven't modified anything, this will never happen (In this example).
			
			This isn't due to the imports like one would expect, this is because we've already processed the other modules.
			Which is basically a coincidence at this point (Purely because 'module01' was made into the primary module).
			If this was the primary module, 'MODULE_02_IMPLEMENTED' and 'MODULE_01_IMPLEMENTED' wouldn't be defined.
		#End
		
		Print("MODULE_02_IMPLEMENTED: False")
	#End
	
	#If MODULE_03_IMPLEMENTED
		Print("MODULE_03_IMPLEMENTED: True")
	#Else
		#Rem
			This will only happen if that preprocessor-variable at the top of the file is not defined.
			(Which it will always be in this case)
		#End
		
		Print("MODULE_03_IMPLEMENTED: False")
	#End
	
	Return
End

This will output the following:

'module01':
MODULE_01_IMPLEMENTED: True
MODULE_02_IMPLEMENTED: False
MODULE_03_IMPLEMENTED: False
'module02':
MODULE_01_IMPLEMENTED: True
MODULE_02_IMPLEMENTED: True
MODULE_03_IMPLEMENTED: False
'module03':
MODULE_01_IMPLEMENTED: True
MODULE_02_IMPLEMENTED: True
MODULE_03_IMPLEMENTED: True

If my idea was applied, these would all be set to 'True', and I honestly think that's more logical.

As this example will show, the 'MODULE_02_IMPLEMENTED' and 'MODULE_03_IMPLEMENTED' preprocessor variables won't even be defined until after they're needed. This means some really awful side-effects in my modules. Take for example my 'imagedimensions' module, which doesn't need 'autostream', but it can use it for extra functionality (Which the user should be expecting if they have the module). Also keep in mind that this situations a bit of an odd case, and it really should be formatted differently if it were following these new rules. You can also see something like this with my 'vector' module, only manual (Not really my choice, I'd rather have it auto-detect with a possible manual override to disable it). In this case, we could fix things with 'ModuleExists', but this is only one specific situation. The way to get around this is to import the optional module first, but then you get to the point of having the modules requiring each other in a somewhat "cyclic" manner. This leads to the user being unable to order the modules in two (Or more) places at once. And that brings us back to my issue at the very beginning; "We need to change how the preprocessor works". Even the "_IMPLEMENTED" preprocessor variables you define don't technically work in this situation. If I were to import 'brl.stream', it wouldn't have the 'BRL_STREAM_IMPLEMENTED' variable defined in this context, and this is because of the current preprocessing setup.

And it's not like we can preprocess the imports before the current module, otherwise we'd be unable to configure them to begin with.

So basically, the only logical conclusion I've been able to come up with is changing the preprocessor to care about imports. It's not like it isn't already top-to-bottom to begin with.

I hope this long post is enough to convince you this is an issue we need to deal with.

marksibly

(Posted 2014) [#5]

I had a quick look at this today and I don't think it'll be toooo tricky.

Will have a closer look this week sometime.

ImmutableOctet(SKNG)

(Posted 2014) [#6]

EDIT: See my first post for details on the current state of the preprocessor.

Yeah, last night I had time and decided to see how the preprocessor was integrated. It's really a matter of how the source code for the module is loaded. It simply filters beforehand while also doing the main preprocessor routine. For my idea to work (Without major restructuring), either the preprocessor would need to be integrated into the 'ParseMain' method of 'Parser', or the 'Import' functionality would need to be put into the preprocessor. The latter ('Preprocess' function) being pretty convoluted, not to mention messy, and the former ('ParseMain' method) being a lot nicer if done well.

Though I didn't test it very much, I did technically get my idea working at one point. I hacked in an alternate form of 'Import', "#Import" into the preprocessor (Which was a pretty bad idea for testing this, in hindsight). The hack I did was rather awful, but that's just because I wasn't really trying to do anything but get the preprocessor to work this way as a test.

Anyway, thanks for the response, Mark. Here's hoping this change works out. (Or at least 'ModuleExists')