Docmods Import Tweak

BlitzMax Forums/BlitzMax Module Tweaks/Docmods Import Tweak

Brucey

(Posted 2006) [#1]

Greetings :-)

I've hacked docmods to be able to process imports.
What do I mean?

Imagine you have a module with the following :

Import "moredocumentedtypes.bmx"

The current docmods simply skips over the import and you subsequently don't get to see the documentation which resides in that file.

My hack causes the parser to pick up "...bmx" imports, parse those, and add the documentation to the module docs.
Useful if you are fed up with 4000 line .bmx files that you want to split apart yet be able to document properly.

It's only 2 lines to bmxtoker.bmx and about 40 lines to docparser.bmx

So, the question is... what is the best way to distribute it? (should anyone want it...)

kfprimm

(Posted 2006) [#2]

post whatever files you modified in this thread.

Brucey

(Posted 2006) [#3]

UPDATE : Now also parses included files
Here we are then.
Assuming that Mark doesn't mind the source for these being posted...

The changes made allow docmods to parse .bmx file imports, rather than just process the main .bmx for a module.
This means that you can split the module into separate .bmx files and have it fully documented at the same time.

bmxtoker.bmx

Strict

Import BRL.Map
Import BRL.Stream

'----- Simple BMX Parser -----

Const T_EOF=		-1
Const T_EOL=		10

Const T_IDENT=		$10001
Const T_INTLIT=		$10002
Const T_FLOATLIT=	$10003
Const T_STRINGLIT=	$10004
Const T_STRINGERR=	$10005

Const T_REM=		$20001
Const T_ENDREM=		$20002
Const T_FUNCTION=	$20003
Const T_ENDFUNCTION=$20004
Const T_TYPE=		$20005
Const T_ENDTYPE=	$20006
Const T_CONST=		$20007
Const T_METHOD=		$20008
Const T_ENDMETHOD=	$20009
Const T_GLOBAL=		$2000a
Const T_INCLUDE=		$2000b ' Added by BaH - 01/06/2006
Const T_IMPORT=		$2000d
Const T_FIELD=		$2000e
Const T_EXTENDS=	$2000f
Const T_ABSTRACT=	$20010
Const T_FINAL=		$20011
Const T_MODULE=		$2000c
Const T_MODULEINFO=	$20012

Const T_DOTDOT=		$30001
Const T_ARRAYDECL=	$30002

Const T_KEYWORD=	$40000

Type TIntValue
	Field value:Int
End Type

Function IntValue:TIntValue( value )
	Local t:TIntValue=New TIntValue
	t.value=value
	Return t
End Function

Function IsSpace( char )
	Return char<=Asc(" ") And char<>Asc("~n")
End Function

Function IsAlpha( char )
	Return ( char>=Asc("a") And char<=Asc("z") ) Or ( char>=Asc("A") And char<=Asc("Z") )
End Function

Function IsNumeric( char )
	Return char>=Asc("0") And char<=Asc("9")
End Function

Function IsAlphaNumeric( char )
	Return IsAlpha( char ) Or IsNumeric( char )
End Function

Type TBmxToker

	Field _filename:String ' Added by BaH - 25/05/2006

	Field _spc
	Field _pos,_line$
	Field _toke,_text$
	Field _stream:TStream
	
	Global _keywords:TMap
	
	Method Delete()
		Close
	End Method

	Method Bump()
	
		If _toke=T_EOF Return _toke
	
		If _pos=_line.length
			_pos=0
			If _stream.Eof()
				_toke=T_EOF
				_text=""
				Return
			EndIf
			_line=_stream.ReadLine().Trim()+"~n"
		EndIf
		
		Local from=_pos
		While _pos<_line.length And IsSpace( _line[_pos] )
			_pos:+1
		Wend
		
		If _spc And _pos<>from
			_text=_line[from.._pos]
			_toke=Asc(" ")
			Return _toke
		EndIf
		
		from=_pos
		Local char=_line[_pos]
		_pos:+1
		
		If char=Asc("'")
			_pos=_line.length
			_toke=T_EOL
		Else If char=Asc("~n")
			_toke=T_EOL
		Else If isAlpha(char) Or char=Asc("_")
			While IsAlphaNumeric(_line[_pos]) Or _line[_pos]=Asc("_")
				_pos:+1
			Wend
			_toke=T_IDENT
			Local id$=_line[from.._pos].ToLower()
			If id="end" And _line[_pos]=Asc(" ") And IsAlpha(_line[_pos+1])
				Local t_pos=_pos+2
				While IsAlphaNumeric(_line[t_pos]) Or _line[t_pos]=Asc("_")
					t_pos:+1
				Wend
				Local id$="end"+_line[_pos+1..t_pos].ToLower()
				Local v:TIntValue=TIntValue( _keywords.ValueForKey( id ) )
				If v
					_pos=t_pos
					_toke=v.value
				EndIf
			EndIf
			If _toke=T_IDENT
				Local v:TIntValue=TIntValue( _keywords.ValueForKey( id ) )
				If v _toke=v.value
				If _toke=T_REM
					_text=""
					Repeat
						If _stream.Eof()
							_pos=0
							_line=""
							Return _toke
						EndIf
						_line=_stream.ReadLine().Trim()+"~n"
						If _line[..6].ToLower()="endrem" Or _line[..7].ToLower()="end rem" Exit
						_text:+_line
					Forever
					_pos=_line.length
					Return _toke
				EndIf
			EndIf
		Else If IsNumeric(char)
			While IsNumeric(_line[_pos])
				_pos:+1
			Wend
			_toke=T_INTLIT
			If _line[_pos]=Asc(".")
				_pos:+1
				While IsNumeric(_line[_pos])
					_pos:+1
				Wend
				_toke=T_FLOATLIT
			EndIf
		Else If char=Asc("~q")
			While _line[_pos]<>Asc("~q") And _line[_pos]<>Asc("~n")
				_pos:+1
			Wend
			If _line[_pos]=Asc("~q")
				_pos:+1
				_toke=T_STRINGLIT
			Else
				_toke=T_STRINGERR
			EndIf
		Else
			_toke=char
		EndIf
		
		_text=_line[from.._pos]
		
		Return _toke
		
	End Method
	
	Method Curr()
		Return _toke
	End Method
	
	Method Text$()
		Return _text
	End Method
	
	Method Line$()
		Return _line
	End Method
	
	Method Parse$( toke )
		If Curr()<>toke Throw "Unexpected token"
		Local t$=Text()
		Bump
		Return t
	End Method
	
	Method CParse$( toke )
		If Curr()<>toke Return
		Local t$=Text()
		Bump
		Return t
	End Method
	
	Method ParseUntil$( toke )
		Local t$
		While Curr()<>toke
			If Curr()=T_EOF Throw "Unexpected EOF"
			t:+Text()
			Bump
		Wend
		Bump
		Return t
	End Method
	
	Method Spaces( enable )
		If enable
			_spc:+1
		Else
			_spc:-1
		EndIf
	End Method
	
	Method Close()
		If _stream _stream.Close
		_stream=Null
	End Method
	
	Function CreateKeywords()
		If _keywords Return
		Function kw( id$,toke )
			_keywords.insert id,IntValue(toke)
		End Function
		_keywords=New TMap
		kw "rem",T_REM
		kw "endrem",T_ENDREM
		kw "function",T_FUNCTION
		kw "endfunction",T_ENDFUNCTION
		kw "method",T_METHOD
		kw "endmethod",T_ENDMETHOD
		kw "const",T_CONST
		kw "global",T_GLOBAL
		kw "include",T_INCLUDE ' Added by BaH - 01/06/2006
		kw "import",T_IMPORT
		kw "field",T_FIELD
		kw "type",T_TYPE
		kw "endtype",T_ENDTYPE
		kw "extends",T_EXTENDS
		kw "abstract",T_ABSTRACT
		kw "final",T_FINAL
		kw "module",T_MODULE
		kw "moduleinfo",T_MODULEINFO
	End Function	
	
	Function Create:TBmxToker( url:Object )
		CreateKeywords
		Local stream:TStream=ReadStream( url )
		If Not stream Throw "Unable to read stream: "+url.ToString()
		Local t:TBmxToker=New TBmxToker
		t._filename = url.ToString() ' Added by BaH - 25/05/2006
		t._stream=stream
		t.Bump
		Return t
	End Function

End Type

docparser.bmx

Strict

Import BRL.StandardIO
Import BRL.MaxUtil

Import "bmxtoker.bmx"

Function FindIdent( pre$,text$,i Var,e Var )

	i=text.Find( pre,i )
	
	If i=-1 Return False
	
	If i>0 And text[i-1]>32 Return False
	
	e=i+1
	While e<text.length
		Local c=text[e]
		If c=Asc(".") And e+1<text.length And IsAlpha( text[e+1] )
			e:+2
			Continue
		EndIf
		If c<>Asc("_") And Not IsAlphaNumeric(c) Exit
		e:+1
	Wend
	Return True
End Function

Function FormatText( t$ Var )

	If t.length And IsAlphaNumeric( t[t.length-1] ) t:+"."

	Local i,e

	i=0
	While FindIdent( "#",t,i,e )
	
		While e<t.length-1 And t[e]=Asc(".") And IsAlpha( t[e+1] )
			e:+2
			While e<t.length And IsAlpha(t[e])
				e:+1
			Wend
		Wend
		
		Local id$=t[i+1..e],q=id.Find("."),ln$

		If q=-1
			ln="<a href=#"+id+">"+id+"</a>"
		Else
			ln="<a href=../../../"
			Repeat
				ln:+id[..q]+".mod/"
				id=id[q+1..]
				q=id.Find(".")
			Until q=-1
			ln:+"doc/commands.html#"+id+">"+id+"</a>"
		EndIf
			
		t=t[..i]+ln+t[e..]
		i:+ln.length
	Wend

	i=0
	While FindIdent( "@",t,i,e )
		Local p$="<b>"+t[i+1..e]+"</b>"
		t=t[..i]+p+t[e..]
		i:+p.length
	Wend

End Function

Type TDocs

	Field kind		'T_CONST, T_GLOBAL etc
	Field ident$	'identifier
	Field proto$	'Global/Function etc text
	Field bbdoc$	'rem/endrem bbdoc: text
	Field infos:TList=New TList	'ModuleInfo's
	Field imports:TList = New TList ' Added by BaH - 25/05/2006
	Field parent:TDocs ' Added by BaH - 25/05/2006
	
	Field kids:TList=New TList
	
	Method Sort()
		kids.Sort
		For Local t:TDocs=EachIn kids
			t.Sort
		Next
	End Method
	
	' Added by BaH - 25/05/2006
	' Checks for extra references to the same imports...
	Method containsImport:Int(_file:String)
		If imports.contains(_file) Then
			Return True
		End If
		If parent <> Null Then
			Return parent.containsImport(_file)
		End If
		
		Return False
	End Method
	
	Method Compare( with:Object )
		Local t:TDocs=TDocs(with)

		If kind=t.kind
			If kind=T_MODULE 
				Return bbdoc.ToLower().Compare( t.bbdoc.ToLower() )
			EndIf
			Return ident.ToLower().Compare( t.ident.ToLower() )
		EndIf

		If kind=T_MODULE Return -1
		If t.kind=T_MODULE Return 1
		
		If kind=T_CONST Return -1
		If t.kind=T_CONST Return 1

		If kind=T_GLOBAL Return -1
		If t.kind=T_GLOBAL Return 1
		
		If kind=T_FIELD Return -1
		If t.kind=T_FIELD Return 1

		If kind=T_METHOD Return -1
		If t.kind=T_METHOD Return 1

		If kind=T_FUNCTION Return -1
		If t.kind=T_FUNCTION Return 1
		
		If kind=T_TYPE Return -1
		If t.kind=T_TYPE Return 1
		
		If kind=T_EOL Return -1
		If t.kind=T_EOL Return 1
		
		Throw "OOps!"
		
	End Method
	
	Method EmitHtml$()
	
		Local returns$
		Local keyword$
		Local shortdesc$
		Local longdesc$
		Local example$
		
		Local d$=bbdoc
		
		Local i=d.Find("~n")
		If i=-1 i=d.length
		shortdesc=d[6..i].Trim()
		d=d[i+1..].Trim()
		If kind<>T_MODULE 
			FormatText shortdesc
		EndIf
		
		If d.Find( "returns:" )=0
			Local i=d.Find("~n")
			If i=-1 i=d.length
			returns=d[8..i].Trim()
			d=d[i+1..].Trim()
			FormatText returns
		Else If d.Find( "keyword:" )=0
			Local i=d.Find("~n")
			If i=-1 i=d.length
			keyword=d[8..i].Trim()
			d=d[i+1..].Trim()
			FormatText keyword
		EndIf
		If d.Find( "about:" )=0
			longdesc=d[6..].Trim()
			FormatText longdesc
		EndIf
		Local stream:TStream=ReadStream( ident+".bmx" )
		If stream
			While Not stream.Eof()
				example:+stream.ReadLine()+"~n"
			Wend
			stream.Close
			example=example.Trim()
		EndIf
		
		Select kind
		Case 0
		Case T_MODULE
			Print "<html><head><title>"+shortdesc+" reference</title>"
			Print "<link rel=stylesheet Type=text/css href='../../../../doc/bmxstyle.css'>"
			Print "</head><body>"
			Local n_consts
			Local n_globals
			Local n_functions
			Local n_types
			For Local t:TDocs=EachIn kids
				Select t.kind
				Case T_CONST n_consts:+1
				Case T_GLOBAL n_globals:+1
				Case T_FUNCTION n_functions:+1
				Case T_TYPE n_types:+1
				End Select
			Next
			
			Print "<table width=100% cellspacing=0><tr align=center><td class=small>&nbsp;</td>"
'			Print "<td class=small width=1%><b>"+ident+" :</b></td>"
			Print "<td class=small width=1%><b>"+ident+":</b></td>"
			If n_consts Print "<td class=small width=1%><a href=#consts class=small>Constants</a></td>"
			If n_globals Print "<td class=small width=1%><a href=#globals class=small>Globals</a></td>"
			If n_functions Print "<td class=small width=1%><a href=#functions class=small>Functions</a></td>"
			If n_types Print "<td class=small width=1%><a href=#types class=small>Types</a></td>"
			
			If Not infos.IsEmpty()
				Print "<td class=small width=1%><a href=#modinfo class=small>Modinfo</a></td>"
			EndIf
			
			Local t$=ModuleSource( ident.ToLower() )
			Local i=t.Find( "/mod/" )
			If i<>-1
				t="../../../.."+t[i..]
				Print "<td class=small width=1%><a href='"+t+"' class=small>Source</a></td>"
			EndIf
			Print "<td class=small>&nbsp;</td></tr></table>"
			
			Local stream:TStream=ReadStream( "intro.bbdoc" )
			
			If stream
				Print "<h1>"+shortdesc+"</h1>"
				If longdesc Print longdesc.Trim()
			Else
				stream=ReadStream( "intro.html" )
				If Not stream
					Print "<h1>"+shortdesc+"</h1>"
					If longdesc Print longdesc.Trim()
				EndIf
			EndIf
			
			If stream
				Local intro$
				While Not stream.Eof()
					intro:+stream.ReadLine()+"~n"
				Wend
				Local i=intro.Find("<body>")
				If i<>-1
					intro=intro[i+6..]
					i=intro.find("</body>")
					If i<>-1 intro=intro[..i]
				EndIf
				stream.close
				intro=intro.Trim()
				FormatText intro
				Print intro
			EndIf

		Default
			Print "<table class=doc width=100% cellspacing=3 id="+ident+">"
			Print "<tr><td class=doctop colspan=2>"+proto+"</td></tr>"
			If returns
				Print "<tr><td class=docleft width=1%>Returns</td><td class=docright>"+returns+"</td></tr>"
			EndIf
			If shortdesc
				Print "<tr><td class=docleft width=1%>Description</td><td class=docright>"+shortdesc+"</td></tr>"
			EndIf
			If longdesc
				Print "<tr><td class=docleft width=1%>Information</td><td class=docright>"+longdesc+"</td></tr>"
			EndIf
			If example
				Print "<tr><td class=docleft width=1%><a href="+ident+".bmx class=small>Example</a></td><td class=docright><pre>"+example+"</pre></td></tr>"
			EndIf
			Print "</table>"
		End Select
		
		If Not kids.IsEmpty()
			Local tkind=0
			For Local doc:TDocs=EachIn kids
				If kind<>T_TYPE And doc.kind<>tkind
					Print "<h2"
					Select doc.kind
					Case T_CONST Print " id=consts>Constants"
					Case T_GLOBAL Print " id=globals>Globals"
					Case T_FUNCTION Print " id=functions>Functions"
					Case T_TYPE Print " id=types>Types"
					Case T_FIELD Print ">Fields"
					Case T_METHOD Print ">Methods"
					Case T_EOL Print ">Keywords"
					End Select
					Print "</h2>"
					tkind=doc.kind
				EndIf
				doc.EmitHtml
				If kind<>T_TYPE Print "<br>"
			Next
		EndIf
		
		Select kind
		Case T_MODULE
			If Not infos.IsEmpty()
				Print "<h2 id=modinfo>Module Information</h2>"
				Print "<table width=100%>"
				For Local t$=EachIn infos
					t=t[1..t.length-1]
					If Not t Continue
					Local i=t.Find(":")
					If i=-1 Continue
					Local key$=t[..i].Trim()
					Local val$=t[i+1..].Trim()
					Print "<tr><th width=1%>"+key+"</th><td>"+val+"</td></tr>"
				Next
			EndIf
			Print "</body></html>"
		End Select
		
	End Method

End Type

Type TDocParser

	Field toker:TBMXToker
	
	Method Close()
		If Not toker Return
		toker.Close
		toker=Null
	End Method

	' Added by BaH - 25/05/2006
	Method processImport(parent:TDocs, txt:String)
	
		If txt.find(".bmx") > 0 Then
	
			Local importFile:String = ExtractDir(toker._filename) + "/" + txt[1..txt.length-1]
			Local _file:String = StripDir(importFile).toLower()
			
			If Not parent.containsImport(_file) Then
				If FileType(importFile) = 1 Then ' exists !
					parent.imports.AddLast(_file)
					Local _toker:TBMXToker=TBMXToker.Create(importFile)
					Local _parser:TDocParser=TDocParser.WithToker(_toker)
					_parser.Parse parent
					_parser.Close
				End If
			End If
		End If
	
	End Method

	Method Parse( parent:TDocs )
	
		If Not toker Throw "closed"
	
		While toker.Curr()<>T_EOF

			If toker.Curr()=T_ENDTYPE 
				If parent.kind=T_TYPE Return
				toker.Bump
				Continue
			EndIf

			' Added by BaH - 25/05/2006, modified 01/06/2006
			If toker.Curr()=T_IMPORT Or toker.Curr()=T_INCLUDE Then
				If parent.kind=T_MODULE
					If toker.Bump()=T_STRINGLIT
						processImport(parent, Toker.Text())
					End If
				End If
			End If
			
			If toker.Curr()=T_MODULEINFO
				If parent.kind=T_MODULE 
					If toker.Bump()=T_STRINGLIT
						parent.infos.AddLast toker.Text()
					EndIf
				EndIf
				toker.Bump
				Continue
			EndIf
		
			If toker.Curr()<>T_REM 
				toker.Bump
				Continue
			EndIf
			
			If toker.Text()[..6]<>"bbdoc:"
				toker.Bump
				Continue
			EndIf
			Local bbdoc$=toker.Text()
			
			Local kind=toker.Bump()
			If kind<>T_CONST And kind<>T_GLOBAL And kind<>T_FUNCTION And kind<>T_METHOD..
			And kind<>T_MODULE And kind<>T_TYPE And kind<>T_FIELD And kind<>T_EOL
				toker.Bump
				Continue
			EndIf
			Local ident$,proto$
			If kind=T_EOL
				Local i=bbdoc.Find( "keyword:" )
				If i=-1 Continue
				ident=bbdoc[i+8..].Replace( "~q","" )
				proto="Keyword "+ident.Trim()
			Else
				proto=toker.Line()
				If toker.Bump()<>T_IDENT Continue
				ident=toker.Text()
				If toker.Bump()=Asc(".") And toker.Bump()=T_IDENT
					ident:+"."+toker.Text()
				EndIf
				toker.Bump
				Select kind
				Case T_CONST,T_GLOBAL
					Local i=proto.Find( "=" )
					If i<>-1 proto=proto[..i]
				Case T_FUNCTION,T_METHOD
					Local i=proto.Find( "=" )
					While i<>-1 And proto.Find( ")",i+1 )<>-1
						i=proto.Find( "=",i+1 )
					Wend
					If i<>-1 proto=proto[..i]
					
					
					i=proto.toLower().Find( "nodebug" )
					While i<>-1 And proto.Find( ")",i+1 )<>-1
						i=proto.toLower().Find( "nodebug",i+1 )
					Wend
					If i<>-1 proto=proto[..i]
					
				End Select
			EndIf
			
			'got a valid Doc!
			Local docs:TDocs=New TDocs
			docs.kind=kind
			docs.ident=ident.Trim()
			docs.proto=proto.Trim()
			docs.bbdoc=bbdoc.Trim()
			docs.parent = parent ' Added by BaH - 25/05/2006

			parent.kids.AddLast docs
			
			If kind=T_MODULE Or kind=T_TYPE
				Parse docs
			EndIf
			
		Wend
	
	End Method
	
	Function WithToker:TDocParser( toker:TBmxToker )
	
		Local t:TDocParser=New TDocParser
		t.toker=toker
		Return t

	End Function
	
End Type

Note: the source code for docmods is included with Blitzmax and can be found in the BlitzMax/src folder.

Enjoy :-)

Dreamora

(Posted 2006) [#4]

Nice :-)

Hopefully the "damned" miss of parsing of includes is fixed in the near future as well

Brucey

(Posted 2006) [#5]

I didn't realise people still used includes.... isn't that just for old blitz backwards compatibility? :-p

Dreamora

(Posted 2006) [#6]

No
Its the only way to come around BMs incapability to separate declaration and implementation.

If you have a class a that uses parts of class b and a class b that has a "back reference" (user - owner like) to class a you MUST include them, import will not work due to circle import references.

There is not much you can do about that core design issue ... They better had stolen more from java and C# (package design based not stupid idiotic import approach of C++)

Brucey

(Posted 2006) [#7]

I've updated the above code to take into account "include" now.

So now you can put documentation in imported and included source for your modules and it will be included into the module docs...

:o)

Dreamora

(Posted 2006) [#8]

Great addition
Thank you :-)

Now I only need to fix up the missing type docmod headers and all should work

dmaz	(Posted 2006) [#9]

ok... before I go through the trouble ;) of installing this change I was wondering if I could document my source files from projects? I was thinking I could create a mod that just imported all the files I want to doc.... does that sound like a good or bad idea?

Dreamora

(Posted 2006) [#10]

There is no reason not to do it actually ^^

Warren

(Posted 2006) [#11]

Just wanted to say thanks for doing this, Brucey! Works like a champ!