The 5 minute "I'm bored at work" tokenizer

BlitzMax Forums/BlitzMax Programming/The 5 minute "I'm bored at work" tokenizer

ozak	(Posted 2006) [#1]

Well. I wanted to make syntax highlighting for my LUA based script language so I wrote this in 5 mins.

It parses a line at a time, but stores the entire token list + seperator list inside the type.

My idea is to only parse the line the user is editing and then color code it accordingly.
I think it will work fine for my simple scripting language.
Anyhoo, here it is, shared with the world :)

I'll keep you guys posted and post the code for a simply LUA highlighting text component as soon as it's done.

' Simple line based tokenizer
Type TLineTokenizer

	' Token list
	Field tokens:TList = New TList
	
	' Seperator list
	Field seps:TList = New TList
	
	' Seperator search list
	Field sepSearchList:String = " "

	' Init tokenizer
	Method Init(sepList:String)
	
		Reset()
		sepSearchList = sepList
	
	End Method
	
	' Reset tokenizer
	Method Reset()
	
		tokens = New TList
		seps = New TList

	End Method
	
	' Tokenize
	Method Tokenize(line:String)
	
		Local lastSepPos:Int = 0
		For Local i:Int = 0 To line.length-1
		
			If (IsSep(Chr(line[i])))
			
				Local temp:String = line[lastSepPos..i]
				temp = StripIllegalChars(temp)
				If (temp.length > 0)
					tokens.addLast(temp)
				End If
				
				If (seps.Contains(Chr(line[i])) = False And Chr(line[i]) <> " ")				
					seps.addLast(Chr(line[i]))
				End If
				lastSepPos = i+1
			
			End If
					
		Next
		
		' Any leftovers?
		If (lastSepPos < line.length)
		
			tokens.addLast(line[lastSepPos..line.length])
		
		End If
	
	End Method	
	
	' Get tokens
	Method GetTokens:TList()
		
		Return tokens
	
	End Method
	
	' Get non-space seperators
	Method GetSeperators:TList()
	
		Return seps
		
	End Method		
	
	' Private check for seperator function
	Method IsSep:Int(ch:String)
	
		If (sepSearchList.find(ch) <> -1)
		
			Return True		
				
		End If
		
		Return False
	
	End Method
	
	' Private strip illegal chars function
	Method StripIllegalChars:String(in:String)

		Local ret:String = in.replace("~t", "")
		Return ret	
			
	End Method


End Type

' Test
myTokenizer:TLineTokenizer = New TLineTokenizer

myTokenizer.Init(",= ()")
myTokenizer.Tokenize("for i = 0, 10 do")
myTokenizer.Tokenize("~tprint(Dice(6)")
myTokenizer.Tokenize("end")

tokens:TList = myTokenizer.GetTokens()
seps:Tlist = myTokenizer.GetSeperators()

For Local ct:String = EachIn tokens

	Print("Token: " + ct)

Next

Print("~n")

For Local cs:String = EachIn seps

	Print("Seps: " + cs)

Next