Module rearrangement ideas?

Monkey Forums/Monkey Programming/Module rearrangement ideas?

marksibly(Posted 2013) [#1]
Hi,

I think the current module layout is a bit messy and am considering tidying it up a bit.

How about:

* A new 'frameworks' dir is added alongside 'modules' that would contain things like Mojo, iGnition, Flixel etc. and added to MODPATH - ie: Import mojo would still work.

* All modules in 'brl' are moved to a new 'std' module, the idea being that people can add their own general purpose stuff here. I prefer this to having arbitrarily named 'brl' and 'pub' modules. In bmx this made more sense, because pub modules were wrappers around 'public' 3rd party native code, while brl modules were the bmx implementations. In monkey, I think having brl vs pub would just be confusing.

* os gets moved to std.

* dom, reflection, opengl and trans can stay where they are.

* Things like box2d, guis, font engines etc can probably just get added to modules, although I guess it's a fine line between frameworks and modules. What is diddy?!?

* In the cases where modules get moved, 'stub' modules can redirect imports while things settle down.

* I'd also like to move most of monkey to std. Stubs and auto-importing can handle existing code, but I think I'd like to start encouraging people to Import std.list in their code.

It's a big change though, although I think it can all be done in a way that wont affect (most) existing code...what does everyone think?

One thing I'd like to add to the language soon is better package importing, eg: currently this:

Import mojo.graphics

...only actually imports a 'graphics' symbol. So this...

mojo.graphics.DrawImage

...bombs out.

I'd like to fix this, BUT it really would 'lock' modules in place so I think it's worth discussing whether modules are in the *right* place first!


Samah(Posted 2013) [#2]
It's too hard to define "module" and "framework", and then label things as such.
The current system is fine.


therevills(Posted 2013) [#3]
The current system is fine.

I agree.


marksibly(Posted 2013) [#4]
> It's too hard to define "module" and "framework", and then label things as such.

Ok, fair enough, although I'd argue things like Mojo, iGnition and Flixel are clearly frameworks...other modules, not so clear...

What about the brl->std issue?

I'm just thinking about future modules like 'xml' or 'json' or something - should they just get lumped into modules? Or into something like 'pub'?

I'd kind of prefer 'std.xml', 'std.json', 'std.os', 'std.stream' etc. as that makes it clear they're part of the 'default' system.


muddy_shoes(Posted 2013) [#5]
I'm not sure what you're trying to achieve with the framework/module distinction. I dislike the idea of breaking up import directories by some concept of purpose. They're all modules.

The only reason I separate module directories is to keep my/3rd party stuff clear of the install directory. That's mostly because of the lack of a proper installer that cleans up after itself. If the update/install process was more sophisticated and maybe incorporated a module manager I'd probably not bother with that.

"brl" vs "std" is unimportant to me. They both imply that the sub-modules are part of the official distribution.


Samah(Posted 2013) [#6]
@muddy_shoes: The only reason I separate module directories is to keep my/3rd party stuff clear of the install directory.

Yep, that's how I use it too. I have multiple entries in MODPATH so that I can point it directly to my Diddy Hg clone without having to copy over source files.

Also, thank you for finally adding MODPATH. I'm glad I don't have to keep hacking it into trans myself. XD


AdamRedwoods(Posted 2013) [#7]
everything is a module, keep it simple.

if something comes standard with monkey, why not just say "import monkey.list"? is there a reason for std?

the way it is now, skn3 makes an xml so we import "skn3.xml" vs "monkey.xml" (if monkey had one) or "diddy.xml". i think it tells us where it came from, who the owner is rather than lumping things together in "std". especially if i build a collada importer and i tell people to get the skn3.xml version, i need to make sure they import the correct one.

i guess i don't see the motivation for the changes, so i may be confused.


Nobuyuki(Posted 2013) [#8]
moving brl to std is a good idea, if/when the method signatures for all brl interfaces are standardized to the point where it's reasonable to assume they won't change anymore. Additions or changes to std should bear in mind the pitfalls of increasing the initial learning curve for users should the modules in a "std" become outdated.

Also, please consider this: Importing std should never error out trans when compiling to the monkeyPro default targets. Limitations in the target platforms to implementing certain features in the brl module should be worked around using precompiler directives, and probably throw a warning, but still compile, and fail and/or return nothing/sane defaults on unsupported targets (Eg: FileStreams in html5, tcpstream on xna) that can be caught by the coder and dealt with without forcing him or her to resort to precompiler directives himself all around the import areas and his own functions.


marksibly(Posted 2013) [#9]
> if something comes standard with monkey, why not just say "import monkey.list"? is there a reason for std?

Not particularly, but I think it'd be nice to reserve 'monkey' for stuff that's tightly bound to the language, and as minimal as possible. As such, I'd rather see list & co. in brl (or std), and monkey perhaps just left with 'lang' and 'math', ie: the absolute minimum you need for Monkey to work.

> the way it is now, skn3 makes an xml so we import "skn3.xml" vs "monkey.xml" (if monkey had one) or "diddy.xml". i think it tells us where it came from, who the owner is rather than lumping things together in "std"

[edit]Just reread that, and I all I'm really saying here is std.xml IS monkey.xml - it's the standard xml module that comes with monkey. But yes, I agree that standard xml should be scoped somewhere, ie: not in the root modules dir. You can disregard much of the following...[/edit]

But wouldn't it be nice to have a 'standard' xml module that comes with monkey, even if it can't do everything? Like python et-al have? If monkey is to grow, don't we want *lots* of standard modules? And if so, where do they go? Or should standard monkey come with multiple xml readers by different authors?

A while back, there was a discussion semi-related to this about adding a set of 'pub' modules to standard monkey that would consist of 3rd party utility stuff like xml readers etc. I think this is a good idea, but don't really see the sense in a brl/pub division. Aren't they all just standard modules? And IMO, brl is badly named for this once it contains a bunch of 'language standard' modules written by a bunch of different people.

Perhaps 3rd party modules could just go in the root namespace - but this does pollute the global namespace and increase the chances of name collisions, esp. if their called 'xml', 'json' etc - I think std.xml, std.json is tidier. Or perhaps they could go under author name, eg: 'skn3' etc, but again just sticking it all in std seems simpler to me.

It's not really a big deal and I'm certainly not planning to do anything in the near future (I probably shouldn't have posted this publicly in the first place), and I'm actually kind of glad everyone seems to be pretty happy with things the way they are, but IMO it could be tidier.


therevills(Posted 2013) [#10]
I like the fact that the official modules are in a folder all by themselves (brl), its clean and tidy IMHO.


ziggy(Posted 2013) [#11]
I think the current simple folder based system works pretty well. Not sure what this framework/modules implies other than reorganizing things on different folders... ?
the way I see it, taking into account the great dead code removal that Trans has, things like diddy wich are sort of a framework and a big usability library can exist and you can import it all and be sure only the bits that you're actually using will be added to the application. That defeats a bit the need of very big modularization of things.
What I would say, go for a better organization of the modules tree folder and that's it. You could also add a framework folder that is by default added to ModPath and expect people to place their "frameworks" there, but if this is by default included in every zip based update, it's kind of useless as you'll be having to copy and paste everything everytime and modpath allows you explictly to avoid this.... ?

Just my 2 cents!


marksibly(Posted 2013) [#12]
> I like the fact that the official modules are in a folder all by themselves (brl), its clean and tidy IMHO.

I agree, I just think it should be renamed (or migrated) to 'std' (ie: the standard monkey modules) and include a whole heap more 3rd party utility modules.

I'd also like to see more non-brl modules in there too (eg: Flixel, JungleGui and co), and I wouldn't consider these any less official once they're part of the repos. IMO, the entire blitz-research/monkey repos *is* the official Monkey release (as official as one branch can be anyway). Brl made sense when I came up with it because it was for my modules only. But now that monkey's on Github, I really want to start thinking of them as monkey's modules instead!


muddy_shoes(Posted 2013) [#13]
If you're looking at Python for an example then it pretty much does just dump all the standard libs in the root directory. However it has a standard directory for user/3rd-party packages/modules too.

I'd be cautious about getting too inclusive with the concept of a standard library in the core repository. Promoting the likes of skn3's xml module to the standard library (or writing another one or whatever) makes sense as it's a feature that people expect in a batteries included language. Including Box2D or Flixel makes far less sense to me as these are more than a little beyond nuts and bolts functionality. They'll bloat the repo and require rather pointless effort to keep in sync.

I do think it's a good idea to make these larger packages more easily available to users but not by cramming them all into the standard lib or the repository. The easiest way would be by including the latest releases as part of the distribution zip or as an extra-packages zip. A better long term solution would be to give some official backing to the idea of a module manager.


Skn3(Posted 2013) [#14]
I always thought the naming convention for the module scopes was a bit out of wack. 'std' is much better.

If you are in a developing mood for this, then perhaps something like maximus could be developed or backed so that 3rd party modules can easily be imported and centralised?

http://maximus.htbaa.com/

The issue with enforcing all 3rd party modules be centralised on one github repo is that there are numerous version control systems and numerous ways people commit to them. It needs some kind of intermediate layer that simplifies the installation process but doesn't dictate to developers.

That may be way off your radar, but if it was more suitable to do the inclusion of 3rd party modules as core modules then that would be a good as well. It might end up becoming quite a swamp of modules some way down the line? What happens when a dev abandons a module but monkey users still need it, would you pick up the development slack?


Playniax(Posted 2013) [#15]
For me the current module system works fine now you have added MODPATH. This way I can have monkey or more versions of monkey in one folder and keep other 3rd party stuff in another. For every update to monkey I just need to copy the MODPATH line in config.winnt.txt and it’s done. Simple and effective! For the other stuff I don’t really have an opinion (yet) ;) To me a module is a module and I don’t care if it’s a framework or whatever it is called.


devolonter(Posted 2013) [#16]
I fully support the idea to rename the brl to std. This will make things more semantic. I also support the idea of adding modules like xml, json and etc. to std. It will be really useful to have this functionality out of box. This saves us from the need to have modules such as pub.

As for frameworks like Box2D, Flixel, etc. I think it would be useful to have a command line tool, which would help to install modules (automatically download, install the module into the necessary folder, and build docs).

Something like flixel -install or flixel -update, etc. Something similar to http://maximus.htbaa.com/ but easier. I can try to implement that as I see it, if that makes sense.

Also, this utility could upgrade the Monkey core. For example, it could take an archive by version tag from GitHub and automatically install it. I think it would be quite useful.

In general, it's nice to see that you want to improve the situation with modules. I often have to tell Monkey newbies which modules to use for xml, json, etc. And it really causes a headache.


marksibly(Posted 2013) [#17]
> Also, please consider this: Importing std should never error out trans when compiling to the monkeyPro default targets.

I've tried to do this in the top level brl module, which conditionally imports submodules depending on target.

But I think it makes less sense for the submodules to protect themselves, eg: I'd rather get a 'brl.tcpstream cannot be used on the html5 target' error than 'TcpStream not found'.

Is that what you mean?

> If you're looking at Python for an example then it pretty much does just dump all the standard libs in the root directory.

The more I think about this, the less dangerous it sounds.

Sure, you get a bunch of globals like 'xml' and 'json' in your app scope, making them unusable for global/class names, but good apps don't use lots of globals(!) and classes name are generally more descriptive than module names (and usually capitalized/prefixed). I doubt there'd be many clashes at app scope, and there's still Alias if there are any.

> Including Box2D or Flixel makes far less sense to me as these are more than a little beyond nuts and bolts functionality.

Yes, I was getting a bit carried away there, although I would like to see more stuff like this in the release.

> As for frameworks like Box2D, Flixel, etc. I think it would be useful to have a command line tool, which would help to install modules (automatically download, install the module into the necessary folder, and build docs).

I tried a 'module manager' once, and it sucked - anyone remember syncmods?!? There were issues with proxies, versioning paradoxes and getting stuff from my machine to the system in the first place.

I'm sure if you know what your doing with Git or some other magical web based system it could be done better than my hack php based attempt - but I'm not experienced at this sort of stuff and wont be attempting another module manager in the near future.

And even if there was such a system, I'd still like to provide full, 'clean slate' releases anyway. They give people a fresh start if an update has somehow messed up their installation, and also allow you to easily have multiple versions of monkey installed at once, something I think is kind of important.

> Something like flixel -install or flixel -update, etc. Something similar to http://maximus.htbaa.com/ but easier. I can try to implement that as I see it, if that makes sense.

By all means, if you come up with something useful I'd be happy to add it to the repos and/or provide it here.


Paul - Taiphoz(Posted 2013) [#18]
Making things cleaner is never a bad thing, but I don't think you should rule out a module manager.

Have you not considered a simpler option, for example, brl could host a module list, which simply lists repository urls, TED could then look at this, and list each repo to the user, allowing them to tick a box to pull that module into their local modules folder.

We already have to as developers install sdks, I dont think it would be much of ask to say, hay if you wana use this module list thing, you need to install the following repository commandline tools.

Could also have a simple add repo option that would add a new repo locally to the client ted, and an option to send any local additions to the master list, if you then check the addition and its valid it gets added to the master for all to use.

it would give us in essence a module list, that is integrated, easy to use, easy to work with, and still decentralized to each of the individual module authors own repositories so no need to extra work on anyone's part. other than the initial creation of the ted code.

As for the module folders, I really don't care, as long as it works, its easy to understand and makes sense then I am all for it./


Nobuyuki(Posted 2013) [#19]
Edit: I came to a more succinct (and different) conclusion after writing this post; if you don't want to read a whole bunch of words, you can safely skip directly to my next post.

>I've tried to do this in the top level brl module, which conditionally imports submodules depending on target.

>But I think it makes less sense for the submodules to protect themselves, eg: I'd rather get a 'brl.tcpstream cannot be used on the html5 target' error than 'TcpStream not found'.
>Is that what you mean?

How is this an improvement over the #Error messages in the individual submodules? If someone tries to use TcpStream, FileStream or something else in those submodules when specifying simply "Import brl", it just changes the type of error they get, right? To something like an "x not found" error? I think might be even more cryptic than forcing an error.

I'm not sure how the full functionality of the lib can to be leveraged if there's a chance that some devs may be unaware that importing it will limit the official targets which the compiler can successfully output to, whether explicitly (via #Error directive) or indirectly (ie: when compilation fails on some targets, but with a less-direct error message due to missing submodules). Please correct me if I'm mistaken about whether this problem would occur under the most common usage patterns.

I mean, I wouldn't like to see a runtime error when attempting to use some of these modules, either. The functionality I'm proposing is to dummy out the functions or their return data so that they fail "gracefully" without a runtime error -- returning some constant which should already be expected, but slightly different in order to indicate the specific loss of functionality. Something the compiler will WARN us about, but not try to stop us from compiling and running the project.

For example, BBTcpStream can has a non-extern version for platforms where TcpStream is not supported, returning default values for all methods and doing nothing in the method body. A new precompiler directive (such as #Warning) which doesn't prevent compilation, but gives a warning in the output would take the place of #Error, and the application would behave at runtime the same way as if no connection were available.

FileStreams would be a bit more tumultuous with how to handle externs to Open, but only because the current functionality gives a runtime error instead of a catchable one in the ctor. If the programmer always expected ctor failures to throw a catchable error or Null, then it would already be handled in their code.

Again, with dummy methods, the individual submodules can be imported without having to worry about writing a wrapper around it, or a bajillion TARGET directives where the offending functions would normally be called (if importing a submodule directly). The developer wouldn't need to delve into the precompiler to deal with the separate targets, because it would be handled on the module side.


Finally, if you'll excuse me for being long-winded, an anecdote -- I used brl for the first time when writing Monkey-UTF8, because I needed FileStreams in order to load a utf8-formatted text file directly on most platforms where LoadString() would've eaten the file. FileStream's functionality was not incorporated for HTML5 (HTML5 file buffers aren't standardized yet? I dunno why the target's not supported, but that's a topic for another conversation), so every place where I'd normally use one of FileStream's methods (including importing it), I had to write precompiler code defining alternate behavior for the html5 target, just to allow the UTF8 module to compile and not "pass the buck" to the developer who would be using my module.

Now, arguably, this would have to be done anyway, but I could've handled it at runtime instead of at compile time by checking for Null, and killed two birds with one stone in the process should FileStream be unable to return a handle on any target for any reason. Having a #Warning directive informing about the reduced functionality would also have given me an opportunity to add a #Warning of my own to end-developers -- unlike the other targets, html5 could load extended unicode chars using LoadString() (negating the need to use FileStream directly), but only as UCS2 surrogate pairs, not UTF8. Direct handling of raw string data would've been different on that platform.


Anyway, the point here is that I feel like my usage pattern for brl is not unique, and that the changes I'm asking to be considered might facilitate writing library/module code which is a bit more robust and needs "less hacks" to compile on all of the official targets. If I were new to Monkey, I would like to see something called "std" support compilation to all the official targets in full, and would prefer not to have target-dependent behavior at design-time preventing compilation.

I'm presuming that consistent behavior across platforms was one of the original goals of keeping Monkey's default modules so simple, originally, anyway. You gotta ask yourself if the inevitable creep of target-dependent functionality should be expected more at runtime (where unintentional differences like the kind I mentioned above with LoadString() already exist), or if we should force devs to handle them at design-time before letting them compile. We're starting to deliberately add this target-dependent functionality to the standard modules. I'm saying, I think since we already should expect (and have to handle) differences in the platforms at runtime, that deliberate t-d functionality should be handled at runtime, too, and to prevent the hazards which go along with that from being uncaught, we should have the compiler shoot a warning to the output buffer so that the end-developer knows about this. Otherwise, importing std could incur a different type of hazard -- inability to compile to some of the officially supported targets.


Nobuyuki(Posted 2013) [#20]
On further consideration, maybe I'm trying to bend over backwards too much for the sake of supporting a refactor of brl to "std". Maybe this shouldn't be called std -- after all, it's mostly just streams and buffers, right? Maybe it should be called "io", or split up ?

Anyway, here's what I've concluded: There needs to be some way to keep the target-dependent functionality away from common usage scenarios. Maybe we should discourage use of brl/brl.monkey, or get rid of it entirely. Forcing users to add the components separately is a better way to make it explicit to the naive user that modules with t-d functionality only get added with full awareness of the platforms that are/aren't supported. In addition, this prevents code examples from utilizing "import brl" -- a practice which could lead to a lot of new Monkey coders scratching their heads when their example won't compile on some targets, or when they try to expand on an example with other functionality contained in that module that is t-d.


marksibly(Posted 2013) [#21]
> If someone tries to use TcpStream, FileStream or something else in those submodules when specifying simply "Import brl", it just changes the type of error they get, right?

Right, but IMO the problem is really that people shouldn't really be lazily importing stuff via brl, they should be importing individual modules instead. But I do consider this more of a 'crap error' issue than a fundamental problem.

I do agree this is an issue that is going to become more problematic as more stuff is added, but I'm not (yet) convinced that a bunch of NOP versions of everything is the right solution. Such classes etc are seldom likely to be usable/useful on their own, so people will still have to #If around the fact they do nothing anyway wont they?

I'd be interested to see your UTF8 module and how you dealt with lack of FIleStream etc - is it available somewhere or can you send it to me?


Nobuyuki(Posted 2013) [#22]
>I'd be interested to see your UTF8 module and how you dealt with lack of FIleStream etc - is it available somewhere or can you send it to me?

yes of course! I dealt with lack of FileStream that the way I expect most users would eventually deal with it; Precompiler directives. There is some html5 native buffer/stream stuff out there, but I ended up not having to deal with it thanks to a lucky break with LoadString() functionality. It's still missing some string manip methods, but it does work: https://github.com/nobuyukinyuu/monkey-utf8 (click the ZIP button, or clone it)

I still think it would be nice to have a #Warning directive for this specific type of code (3rd party modules with non-blocking t-d functionality), although I understand that I can use #Print to achieve the same thing. In the future, perhaps it could be useful for 3rd party IDEs to catch...


marksibly(Posted 2013) [#23]
Hmm...I don't really think there's any clever way to improve on what you've done there!

You could probably refactor it a bit to get rid of some of the #Ifs, but even if there was a dummy FileStream class you'd probably still end up with roughly the same code to handle the fact the dummy FileStream didn't actually do anything.

But I think the bigger point you're (possibly?) making is completely valid: "Import std" should import the same decls regardless of the target you're using. If a module can't work at least minimally usefully everywhere, it shouldn't be in std.

In fact, I'd suggest *no* module should conditionally import/declare stuff. Either the module works the same way (and supplies the same decls) on a range of targets, or it doesn't. Modules can still 'auto import' stuff, they just have to either auto import the same things regardless of target, or crap out with a 'not on this target big boy' error. brl is actually the only module that doesn't do this I think - will double check.


Nobuyuki(Posted 2013) [#24]
> You could probably refactor it a bit to get rid of some of the #Ifs, but even if there was a dummy FileStream class you'd probably still end up with roughly the same code to handle the fact the dummy FileStream didn't actually do anything.

You're correct. That was one of the reasons I reconsidered suggesting dummying-out the return values -- even if it would get rid of some #Ifs, as a 3rd-party module-writer, I'd still have to add them back in anyway so that the t-d functionality isn't dumped on the end-dev. For most practical purposes, all platforms behave the same with my module, but in reality, LoadRaw() is not the same on all platforms and the end-dev would benefit from a warning.

That being said, if I was the end-dev and not the module writer... say I was porting over Monkey code intended for one platform to another platform (maybe we did our own netcode or IAP?) that contains t-d submodules from brl. My original long post advocating dummy values was mainly for their benefit -- they could still make test compilations on the platform with the dummy values, to test other unrelated things, and not have to immediately focus their attention on the t-d parts just to get it to compile.

The relative benefits and pitfalls of that seemed to me at the time to support dummy values, but after thinking about it, it's better (and cleaner) to block compilation the way you just suggested -- if and only if the entire module is consistent in its (lack of) platform support and the end-dev is fully aware of it before importing. In that case, I still think brl might benefit from being refactored into separate modules -- either at the individual submodule level (this may cause Import soup), or grouped by something like "io" and "buffer" -- to better localize the t-d functionality and maximize the number of platforms each of those refactored modules can compile to without end-dev #If workarounds.