More sytnax candy?

BlitzMax Forums/BlitzMax Programming/More sytnax candy?

Pineapple

(Posted 2015) [#1]

I love BlitzMax and I've been using it for so long, I have so many random libraries and snippets that I have to rewrite any time I do something substantial in another language. But I feel like it has some serious lackings as far as usability goes. Operator and function overriding being a big one. Also conditional compiling, multiple inheritance, lambdas, macros, explicit function inlining, inlined C, and lots more. All of these could surely be accomplished without too much fuss just by writing a fairly simple preprocessor to translate the syntax candy into regular BlitzMax code.

And so I'm very tempted to write such a thing myself, but first: Has anyone done this already? It'd be much more convenient that way.

Brucey

(Posted 2015) [#2]

I was considering applying interfaces, generics and overloading to bmx_ng at some point (since Monkey has them).

LT	(Posted 2015) [#3]

True first-order functions would be nice, too.

EDIT: Make that first-class...

Yasha

(Posted 2015) [#4]

just by writing a fairly simple preprocessor

My feeling is that a "preprocessor" that could handle anything as complicated as operator overloading or multiple inheritance would be a far more powerful piece of technology than the existing compiler anyway. The existing backend is completely redundant in that situation - if you're writing a full compiler (which you are, since it has to recognise the complete syntax of Max and do sophisticated type checking), you may as well target a more useful backend, like C or LLVM.

@LT you mean first-class/higher order? (first-order is more or less what it already has)

Did anyone (*cough* Brucey) obtain or reverse-engineer a proper grammar? (The last time I looked at it the backported Monkey compiler wasn't rule-based ...) Community language tools would be much more plausible if there was any kind of concrete, non ad-hoc information about the language itself.

LT	(Posted 2015) [#5]

@Yasha - Yeah, first-class is what I meant.

As much as I love BlitzMax, I am finding myself running into a wall on some of the limitations.

Yahfree

(Posted 2015) [#6]

I'm currently busy with some other obligations but I have the start of a formal grammar for bmax I put together over a couple hours. If anyone is interested here it is (*ISON / BNF), this is Jison, a JavaScript parser generator:

%{
  /* Injected Code */
  /* These are the ASTNode classes that are dependencies of
     this parser since these nodes are constructed on semantic
     actions to build the Abstract Syntax Tree */
  var FunctionCall = require('./ast/FunctionCall');
  var MethodCall = require('./ast/MethodCall');
  var FormalParam = require('./ast/FormalParam');
  var FormalParamList = require('./ast/FormalParamList');
  var Identifier = require('./ast/Identifier');
  var Type = require('./ast/Type');
  var ExpList = require('./ast/ExpList');
  var TypeCompositeList = require('./ast/TypeCompositeList');
  var TypeComposite = require('./ast/TypeComposite');
%}

/* Precedence rules to remove SR conflicts in expressions */
%left RPAREN LPAREN
%left AND OR
%nonassoc LESSTHAN GREATERTHAN EQUAL
%left PERIOD
%left RBRACE LBRACE

%left PLUS MINUS
%left TIMES DIVIDE
%left KARAT

%%

/* 
   BlitzMax Grammar
   Tokens (terminals) are all caps
   and are defined in the scanner/lex.jison file.
 */


TopLevel
    : SUPERSTRICT Program
      { $$ = $2; }
    ;

Program
    : Stmt Program
      { $$ = $2; $$.stmts.add($1); }
    | ClassDecl Program
      { $$ = $2; $$.classes.add($1); }
    | FunctionDecl Program
      { $$ = $2; $$.functions.add($1); }
    | 
      { $$ = new Program(yylineno); }
    ;

ClassDecl
    : TYPE Identifier ClassBody ENDTYPE
      { $$ = new ClassDeclSimple($1, $2, yylineno); }
    ;

ClassBody
    : Stmt ClassBody
      { $$ = $2; $$.stmts.add($1); }
    | FieldDecl ClassBody
      { $$ = $2; $$.fields.add($1); }
    | MethodDecl ClassBody
      { $$ = $2; $$.methods.add($1); }
    | 
      { $$ = new ClassBody(yylineno); }
    ;

ElseIfList
    : ELSEIF Exp StmtList ElseIfList
      { $$ = $4; $$.elseIfList.add(new If($2, $3, null, yylineno)); }
    | ELSE StmtList ElseIfList
      { $$ = $3; $$.else = $2; }
    |
      { $$ = new ElseIfList(yylineno); }
    ;

Stmt
    : IF Exp StmtList ElseIfList ENDIF
      { $$ = new If($2, $3, $4, yylineno); }
    | LOCAL Identifier COLON Type
      { $$ = new LocalVarDecl($2, $4, yylineno); }
    | GLOBAL Identifier COLON Type
      { $$ = new GlobalVarDecl($2, $3); }
    | Identifier EQUAL Exp
      { $$ = new Assign($1, $3); }
    | LOCAL Identifier COLON Type EQUAL Exp
      { $$ = new LocalVarDeclAssign($2, $4. $6, yylineno); }
    | GLOBAL Identifier COLON Type EQUAL Exp
      { $$ = new LocalVarDeclAssign($2, $4. $6, yylineno); }
    ;

StmtList
    : Stmt StmtList
      { $$ = $2; $$.add($1); }
    |
      { $$ = new StmtList(yylineno); }
    ;

FormalParam
    : Identifier COLON Type
      { $$ = new FormalParam($1, $2); }
    ;

FormalParamList
    : FormalParam COMMA FormalParamList
      { $$ = $3; $$.add($1); }
    | FormalParam
      { $$ = new FormalParamList(yylineno); $$.add($1); }
    |
      { $$ = new FormalParamList(yylineno); }
    ;

Identifier
    : IDENTIFIER
      { $$ = new Identifier(yytext, yylineno); }
    ;

FunctionDecl
    : FUNCTION Identifier COLON Type LPAREN FormalParamList RPAREN StmtList ENDFUNCTION
      { $$ = new FunctionDecl($2, $4, $6, yylineno); }
    | FUNCTION Identifier COLON Type LPAREN FormalParamList RPAREN StmtList END
      { $$ = new FunctionDecl($2, $4, $6, $8, yylineno); }
    ;

TypeCompositeList
    : PTR TypeCompositeList
      { $$ = $2; $$.add(new TypeComposite("Ptr")); }
    |
      { $$ = new TypeCompositeList(yylineno); }
    ;

Type
    : FLOAT TypeCompositeList
      { $$ = new Type('Float', $2, yylineno); }
    | BYTE TypeCompositeList
      { $$ = new Type('Byte', $2, yylineno); }
    | INT TypeCompositeList
      { $$ = new Type('Int', $2, yylineno); }
    | LONG TypeCompositeList
      { $$ = new Type('Long', $2, yylineno); }
    | SHORT TypeCompositeList
      { $$ = new Type('Short', $2, yylineno); }
    | STRING TypeCompositeList
      { $$ = new Type('String', $2, yylineno); }
    | DOUBLE TypeCompositeList
      { $$ = new Type('Double', $2, yylineno); }
    | Object TypeCompositeList
      { $$ = new Type('Object', $2, yylineno); }
    | Identifier TypeCompositeList
      { $$ = new Type($1.name, $2, yylineno); }
    ;

Exp 
    : Exp TIMES Exp
      { $$ = new Times($1, $3, yylineno); }
    | Exp MINUS Exp
      { $$ = new Minus($1, $3, yylineno); }
    | Exp PLUS Exp
      { $$ = new Plus($1, $3, yylineno); }
    | Exp LESSTHAN Exp
      { $$ = new LessThan($1, $3, yylineno); }
    | Exp LESSTHAN EQUAL Exp
      { $$ = new LessThanEqual($1, $4, yylineno); }
    | Exp LESSTHAN GREATERTHAN Exp
      { $$ = new NotEqual($1, $4, yylineno); }
    | Exp GREATERTHAN Exp
      { $$ = new GreaterThan($1, $3, yylineno); }
    | Exp GREATERTHAN EQUAL Exp
      { $$ = new GreaterThanEqual($1, $4, yylineno); }
    | Exp OR Exp
      { $$ = new Or($1, $3, yylineno); }
    | Exp AND Exp
      { $$ = new And($1, $3, yylineno); }
    | Exp LBRACE Exp RBRACE
      { $$ = new ArrayLookup($1, $3, yylineno); }
    | Identifier LPAREN ExpressionList RPAREN
      { $$ = new FunctionCall($1, $2); }
    | Exp PERIOD Identifier
      { $$ = new FieldAccess($1, $3, yylineno); }
    | Exp PERIOD Identifier LPAREN ExpressionList RPAREN
      { $$ = new MethodCall($1, $3, $5, yylineno); }
    | LPAREN Exp RPAREN
      { $$ = $2; }
    ;

And here's more or less a working lexer:

%{
	/* Injected Code */
	var Token = require('./Token');
%}

%%

/* Language tokenizations */

/* Comments and whitespace */
\s+                   /* skip whitespace */
\'.*\n                   /* skip line comments */

/* Language keywords */
"Strict"		return new Token('STRICT');
"SuperStrict"		return new Token('SUPERSTRICT');
"Rem"		return new Token('REM');
"EndRem"		return new Token('ENDREM');
"End Rem"		return new Token('ENDREM');
"True"		return new Token('TRUE');
"False"		return new Token('FALSE');
"Pi"		return new Token('PI');
"Null"		return new Token('NULL');
"Byte"		return new Token('BYTE');
"Short"		return new Token('SHORT');
"Int"		return new Token('INT');
"Long"		return new Token('LONG');
"Float"		return new Token('FLOAT');
"Double"		return new Token('DOUBLE');
"String"		return new Token('STRING');
"Object"		return new Token('OBJECT');
"Var"		return new Token('VAR');
"Ptr"		return new Token('PTR');
"If"		return new Token('IF');
"Then"		/* Then is not important to the Parser */
"Else"		return new Token('ELSE');
"ElseIf"		return new Token('ELSEIF');
"EndIf"		return new Token('ENDIF');
"End If"		return new Token('ENDIF');
"For"		return new Token('FOR');
"To"		return new Token('TO');
"Step"		return new Token('STEP');
"Next"		return new Token('NEXT');
"EachIn"		return new Token('EACHIN');
"While"		return new Token('WHILE');
"Wend"		return new Token('WEND');
"EndWhile"		return new Token('ENDWHILE');
"Repeat"		return new Token('REPEAT');
"Until"		return new Token('UNTIL');
"Forever"		return new Token('FOREVER');
"Select"		return new Token('SELECT');
"EndSelect"		return new Token('ENDSELECT');
"Case"		return new Token('CASE');
"Default"		return new Token('DEFAULT');
"Exit"		return new Token('EXIT');
"Continue"		return new Token('CONTINUE');
"Const"		return new Token('CONST');
"Local"		return new Token('LOCAL');
"Global"		return new Token('GLOBAL');
"Field"		return new Token('FIELD');
"Function"		return new Token('FUNCTION');
"EndFunction"		return new Token('ENDFUNCTION');
"End Function"		return new Token('ENDFUNCTION');
"Return"		return new Token('RETURN');
"Type"		return new Token('TYPE');
"EndType"		return new Token('ENDTYPE');
"End Type"		return new Token('ENDTYPE');
"Extends"		return new Token('EXTENDS');
"Method"		return new Token('METHOD');
"EndMethod"		return new Token('ENDMETHOD');
"End Method"		return new Token('ENDMETHOD');
"Abstract"		return new Token('ABSTRACT');
"Final"		return new Token('FINAL');
"New"		return new Token('NEW');
"Self"		return new Token('SELF');
"Super"		return new Token('SUPER');
"Delete"		return new Token('DELETE');
"Release"		return new Token('RELEASE');
"Public"		return new Token('PUBLIC');
"Private"		return new Token('PRIVATE');
"Extern"		return new Token('EXTERN');
"EndExtern"		return new Token('ENDEXTERN');
"End Extern"		return new Token('ENDEXTERN');
"Module"		return new Token('MODULE');
"ModuleInfo"		return new Token('MODULEINFO');
"Incbin"		return new Token('INCBIN');
"IncbinPtr"		return new Token('INCBINPTR');
"IncbinLen"		return new Token('INCBINLEN');
"Include"		return new Token('INCLUDE');
"Framework"		return new Token('FRAMEWORK');
"Import"		return new Token('IMPORT');
"Assert"		return new Token('ASSERT');
"Goto"		return new Token('GOTO');
"Try"		return new Token('TRY');
"EndTry"		return new Token('ENDTRY');
"End Try"		return new Token('ENDTRY');
"Catch"		return new Token('CATCH');
"Throw"		return new Token('THROW');
"DefData"		return new Token('DEFDATA');
"ReadData"		return new Token('READDATA');
"RestoreData"		return new Token('RESTOREDATA');
"And"		return new Token('AND');
"Or"		return new Token('OR');
"Not"		return new Token('NOT');
"Shl"		return new Token('SHL');
"Shr"		return new Token('SHR');
"Sar"		return new Token('SAR');
"Len"		return new Token('LEN');
"Abs"		return new Token('ABS');
"Mod"		return new Token('MOD');
"Sgn"		return new Token('SGN');
"Min"		return new Token('MIN');
"Max"		return new Token('MAX');
"Varptr"		return new Token('VARPTR');
"SizeOf"		return new Token('SIZEOF');
"Asc"		return new Token('ASC');
"Chr"		return new Token('CHR');
"Print"		return new Token('PRINT');
"Input"		return new Token('INPUT');
"End"		return new Token('END');

/* End of the file */
<<EOF>>		return new Token('EOF');

/* Special characters and ops */
"("		return new Token('LPAREN');
")"		return new Token('RPAREN');
"["		return new Token('LBRACE');
"]"		return new Token('RBRACE');

"="		return new Token('EQUAL');
"^"		return new Token('KARAT');
"<"		return new Token('LESSTHAN');
">"		return new Token('GREATERTHAN');
"+"		return new Token('PLUS');
"-"		return new Token('MINUS');
"/"		return new Token('DIVIDE');
"*"		return new Token('TIMES');
"."		return new Token('DOT');
","		return new Token('COMMA');
":"		return new Token('COLON');

/* Identifiers */
[a-zA-Z][a-zA-Z_0-9]*		return new Token('IDENTIFIER', yytext);

/* Literals */
"\""[^"\""]*"\""	return new Token('STRINGLITERAL', yytext);
[0-9]+"."[0-9]+		return new Token('FLOATLITERAL', yytext);
[0-9]+		return new Token('INTEGERLITERAL', yytext);

%%

Yasha

(Posted 2015) [#7]

That's really cool, thanks.

I made an initial attempt at a Max translation of the lexer:

' BlitzMax Lexer in BlitzMax

SuperStrict

Import "TMeta.bmx"	'Get this here: <a href="http://www.blitzbasic.com/codearcs/codearcs.php?code=3090" target="_blank">http://www.blitzbasic.com/codearcs/codearcs.php?code=3090</a>

Type MaxLexer Final
	Function Get:TLexer()
		Global Store(_:TLexer) = TLexAction.Store, Discard(_:TLexer) = TLexAction.Discard, Mode(_:TLexer) = TLexAction.Mode
		
		Global l:TLexer = TLexer.withRules([..
			R("[0-9]+", Store, "intlit", "0123456789"),..	'Numbers
			R("$[0-9a-fA-F]+", Store, "hexlit"),..
			R("[0-9]+\.[0-9]+([eE]-?[0-9]+)?", Store, "floatlit", "0123456789"),..
			R("[0-9]+[eE]-?[0-9]+", Store, "floatlit", "0123456789"),..
		..
			R("~q[^\n~q]*~q", Store, "strlit"),..	'Strings
		..
			R("'[^\n]*", Discard),..			'Line comment
			R("", "Rem", Mode, "COMMENT"),..		'Long comment
			M("COMMENT", "End[ \t]*Rem", Mode, ""),..
			M("COMMENT", "[^[:space:]]", Discard),..
			M("COMMENT", "[ \n\r\t]", Discard),..
		..
			R("include[ \t]*~q[^\n~q]*~q", TLexAction.Incl, "~q"),..
		..
			S("strict", Store),..
			S("superstrict", Store),..
			S("true", Store),..
			S("false", Store),..
			S("pi", Store),..
			S("null", Store),..
			S("byte", Store),..
			S("short", Store),..
			S("int", Store),..
			S("long", Store),..
			S("float", Store),..
			S("double", Store),..
			S("string", Store),..
			S("object", Store),..
			S("var", Store),..
			S("ptr", Store),..
			S("if", Store),..
			S("then", Store),..
			S("else", Store),..
			S("else[ \t]*if", Store, "elseif"),..
			S("end[ \t]*if", Store, "endif"),..
			S("for", Store),..
			S("to", Store),..
			S("step", Store),..
			S("next", Store),..
			S("eachin", Store),..
			S("while", Store),..
			S("wend", Store),..
			S("end[ \t]*while", Store, "wend"),..
			S("repeat", Store),..
			S("until", Store),..
			S("forever", Store),..
			S("select", Store),..
			S("end[ \t]*select", Store, "endselect"),..
			S("case", Store),..
			S("default", Store),..
			S("exit", Store),..
			S("continue", Store),..
			S("const", Store),..
			S("local", Store),..
			S("global", Store),..
			S("field", Store),..
			S("function", Store),..
			S("end[ \t]*function", Store, "endfunction"),..
			S("return", Store),..
			S("type", Store),..
			S("end[ \t]*type", Store, "endtype"),..
			S("extends", Store),..
			S("method", Store),..
			S("end[ \t]*method", Store, "endmethod"),..
			S("abstract", Store),..
			S("final", Store),..
			S("new", Store),..
			S("self", Store),..
			S("super", Store),..
			S("delete", Store),..
			S("release", Store),..
			S("public", Store),..
			S("private", Store),..
			S("extern", Store),..
			S("end[ \t]*extern", Store, "endextern"),..
			S("module", Store),..
			S("moduleinfo", Store),..
			S("incbin", Store),..
			S("incbinptr", Store),..
			S("incbinlen", Store),..
			S("framework", Store),..
			S("import", Store),..
			S("assert", Store),..
			S("goto", Store),..
			S("try", Store),..
			S("end[ \t]*try", Store, "endtry"),..
			S("catch", Store),..
			S("throw", Store),..
			S("defdata", Store),..
			S("readdata", Store),..
			S("restoredata", Store),..
			S("and", Store),..
			S("or", Store),..
			S("not", Store),..
			S("shl", Store),..
			S("shr", Store),..
			S("sar", Store),..
			S("len", Store),..
			S("abs", Store),..
			S("mod", Store),..
			S("sgn", Store),..
			S("min", Store),..
			S("max", Store),..
			S("varptr", Store),..
			S("sizeof", Store),..
			S("asc", Store),..
			S("chr", Store),..
			S("end", Store),..
		..
			S("(", Store, "lparen"),..		'Special characters
			S(")", Store, "rparen"),..
			S("{", Store, "lcurly"),..
			S("}", Store, "rcurly"),..
			S("[", Store, "lbrace"),..
			S("]", Store, "rbrace"),..
			S("#", Store, "hash"),..
			S("%", Store, "intsigil"),..
			S("!", Store, "doublesigil"),..
			S("$", Store, "stringsigil"),..
			S("@", Store, "bytesigil"),..
			S(";", Store, "semicolon"),..
			S(":", Store, "colon"),..
			S(",", Store, "comma"),..
			S(".", Store, "dot"),..
			S("..", Store, "dotdot"),..
		..
			S("+",  Store, "plus"),..		'Operators
			S("-", Store, "minus"),..
			S("*", Store, "mul"),..
			S("/", Store, "div"),..
			S("^", Store, "pow"),..
			S("<>", Store, "neq"),..
			S("<=", Store, "leq"),..
			S(">=", Store, "geq"),..
			S("<", Store, "lt"),..
			S(">", Store, "gt"),..
			S("=", Store, "equals"),..
			S("&", Store, "bitand"),..
			S("|", Store, "bitor"),..
			S("~", Store, "bitnot"),..
		..
			R("[a-zA-Z_][a-zA-Z0-9_]*", Store, "name", "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_"),..
		..
			R("[^[:space:]]", TLexAction.Error)..		'Raise an error over any other printable character
		])
		l.SetCaseSensitivity False
		l.SetGuardMode True
		
		Return l
		Function R:TLexRule(r:String, a(l:TLexer), res:String = "", ic:String = "")
			Return TLexRule.Create(r, a, res, "", ic)
		End Function
		Function S:TLexRule(r:String, a(l:TLexer), res:String = "")
			Return TLexRule.CreateSimple(r, a, res)
		End Function
		Function M:TLexRule(m:String, r:String, a(l:TLexer), res:String = "")
			Return TLexRule.Create(r, a, res, m)
	End Function
End Type

I can't directly translate the parser quite as easily, as TMeta.bmx is LL/top down. Am willing to try to follow that up later if anyone is interested though.

Yahfree

(Posted 2015) [#8]

The parser isn't complete yet unfortunately. :\

It's difficult reverse engineering the language, since there are a lot of cases where you can do things multiple ways to do things, Like:

Graphics 200, 200
Local test = 10

Graphics(200, 200)
test = 10

But definitively possible. I was going to target SuperStrict mode first.

But the key is to get it into a parser generator language like Bison, because then you can generate a parser for basically any language, and any kind of parser that is powerful enough as well (LL, LALR(1), you name it).

I chose to do it in JavaScript because after I'm done writing it I wanted to put the parser in the browser so people can test their code with it and verify it produces a sensible abstract syntax tree.

If getting BlitzMax to parse BlitzMax is what you're after, I would still recommend finishing the Bison implementation, since you can generate a parser for C++ and can most definitively interop it with BlitzMax.

Yasha

(Posted 2015) [#9]

Bison is the way forward for a serious implementation, yes.

TMeta is just something I came up with mainly because the two-stage build process of parser-generator -> source -> program was annoying when exploring a design space. It is combinator-based. It isn't very efficient, but I haven't seen any actual problems with performance yet. Unfortunately there's no syntactically elegant way to do combinators in BlitzMax... *sigh*.