Weird crashes with TMap.Values()

BlitzMax Forums/BlitzMax Programming/Weird crashes with TMap.Values()

grable

(Posted 2015) [#1]

Im having a bit of trouble with when enumerating over TMap.Values(), for some reason a Null value creeps into the tree and crashes the enumerator on TNode.NextNode() (it gets it when tracing the nodes parents).

But this only happens in Release mode, in Debug mode everything runs as expected :(

And the weirder part is that if i enumerate over TNodes instead, it also works as expected, even in Release mode!

Also this only happens in NON-Threaded mode, so might have something to do with refcounting.

Im not doing anything special either, just using generated strings as keys...

Anyway, could someone try this piece of code and see if they get the same problem? Atleast then id know if its something with my setup or otherwise.

typemap.bmx

Framework BRL.StandardIO

SuperStrict

Import BRL.Map

Const IR_T_NONE:Int = 0
Const IR_T_BOOL:Int = 1
Const IR_T_INT:Int = 2
Const IR_T_FLOAT:Int = 3
Const IR_T_STRING:Int = 4

Function IrTypeToString:String( tag:Int)
	Assert tag >= IR_T_NONE And tag <= IR_T_STRING Else "invalid type tag: " + tag
	Select tag
		Case IR_T_NONE		Return ""
		Case IR_T_BOOL		Return "Bool"
		Case IR_T_INT		Return "Int"
		Case IR_T_FLOAT	Return "Float"
		Case IR_T_STRING	Return "String"
	EndSelect
EndFunction

Function IrTypeToString_short:String( tag:Int)
	Assert tag >= IR_T_NONE And tag <= IR_T_STRING Else "invalid type tag: " + tag
	Select tag
		Case IR_T_NONE		Return "?"
		Case IR_T_BOOL		Return "B"
		Case IR_T_INT		Return "I"
		Case IR_T_FLOAT	Return "F"
		Case IR_T_STRING	Return "S"
	EndSelect
EndFunction

Function IrTypeFromString:Int( name:String)
	If Not name Then Return IR_T_NONE
	Select name.ToLower()
		Case "bool"		Return IR_T_BOOL
		Case "int"		Return IR_T_INT
		Case "float"	Return IR_T_FLOAT
		Case "string"	Return IR_T_STRING
	EndSelect
	Return IR_T_NONE
EndFunction


Type TIrOpMapping
	Field Name:String
	Field Value:String
	Field IsBinary:Int
	Field Op:Int
	
	Method Compare:Int( other:Object)
		Local m:TIrOpMapping = TIrOpMapping(other)
		If Not m Then Return 1
		Return Name.Compare(m.Name)
	EndMethod
	
	Method ToString:String()
		Return Name
	EndMethod
EndType

Type TIrTypeMapping
	Field Name:String
	Field Op:TIrOpMapping
	Field LType:Int
	Field RType:Int
	Field DType:Int
	Field Code:String
	
	Method ToName:String()
		If Name Then Return Name
		Name = IrTypeToString_short(DType) + "_" + IrTypeToString_short(LType) + "_" + Op.Name + "_" + IrTypeToString_short(RType)
		Return Name
	EndMethod
	
	Method ToString:String()
		Local s:String = IrTypeToString(DType) + " := "
		If LType <> IR_T_NONE Then s :+ IrTypeToString(LType) + " "
		If Op Then s :+ Op.Name
		If RType <> IR_T_NONE Then s :+ " " + IrTypeToString(RType)
		Return s
	EndMethod
EndType

Type TIrTypeMap
	Field OpMaps:TMap = New TMap
	Field TypeMaps:TMap = New TMap
	Field AssignOp:TIrOpMapping
	
	Method New()
		AssignOp = New TIrOpMapping
		AssignOp.Name = "ASSIGN"
		AssignOp.Value = "="
		AssignOp.IsBinary = False
		AssignOp.Op = 0
		OpMaps.Insert( AssignOp.Name, AssignOp)
	EndMethod
	
	Method LookupOp:TIrOpMapping( name:String)
		Assert name Else "attempted to Lookup Null op name"
		Return TIrOpMapping( OpMaps.ValueForKey(name))
	EndMethod
	
	Method InsertOp:TIrOpMapping( map:TIrOpMapping)
		Assert map Else "attempted to Insert Null op mapping"
		Assert Not TIrOpMapping( OpMaps.ValueForKey(map.Name)) Else "op mapping already defined: " + map.ToString()
		OpMaps.Insert( map.Name, map)
	EndMethod

	Method LookupMap:TIrTypeMapping( map:TIrTypeMapping)
		Assert map Else "attempted to Lookup Null type mapping"
		Return TIrTypeMapping( TypeMaps.ValueForKey(map.ToName()))
	EndMethod
	
	Method InsertMap:TIrTypeMapping( map:TIrTypeMapping)
		Assert map Else "attempted to Insert Null type mapping"
		Assert Not TIrTypeMapping( TypeMaps.ValueForKey(map.ToName())) Else "type mapping already defined: " + map.ToString()
		TypeMaps.Insert( map.ToName(), map)
	EndMethod
	
	Method ReadFromFile:Int( url:Object)
		Function nextspace:Int( s:String)
			If Not s Then Return -1
			For Local i:Int = 0 Until s.Length
				If s[i] = Asc(" ") Or s[i] = Asc("~t") Then Return i
			Next
			Return -1
		EndFunction
		
		Function stripafter:String( s:String, after:String)
			If Not s Then Return Null
			Local i:Int = s.Find(after)
			If i = -1 Then Return s
			s = s[.. i]
			If s Then Return s.Trim()
		EndFunction
		
		Function swapends:String( s:String, middle:String)
			If Not s Then Return Null
			Local a:String[] = s.Split(middle)
			If a.Length <> 2 Then Return s
			Return a[1].Trim() + " " + middle + " " + a[0].Trim()
		EndFunction
		
		
		Local file:String = String(url)
		Local stream:TStream = ReadStream(url)
		If Not stream Then Return 0
		
		Local line:Int = 0
		Local count:Int = 0		
		Local opmode:Int = 0 ' 0=none 1=unary 2=binary 
		Local numops:Int = 1 ' +1 for ASSIGN
		
		While Not stream.Eof()
			Local l:String = stream.ReadLine()
			Local s:String = stripafter( l.Trim(), "//")
			line :+ 1
			If Not s Then Continue
			
			If s.StartsWith("%") Then
				If s = "%%" Then
					opmode = 0
				ElseIf s = "%unary" Then
					opmode = 1
				ElseIf s = "%binary" Then
					opmode = 2					
				EndIf
				Continue
			EndIf
			
			If opmode > 0 Then
				Local i:Int = nextspace(s)
				If i = -1 Then
					DebugLog "TirTypeMap.ReadFromFile(" + file + "): invalid %op syntax at line " + line + ", expected: T T..."
					Continue
				EndIf
				Local opmap:TIrOpMapping = New TIrOpMapping
				opmap.Name = s[.. i].Trim()
				opmap.Value = s[i + 1 ..].Trim()
				opmap.Op = numops
				opmap.IsBinary = (opmode = 2)
				InsertOp(opmap)
				numops :+ 1
				Continue
			EndIf			

			Local a:String[] = s.Split(":=")
			If a.Length <> 2 Then
				DebugLog "TirTypeMap.ReadFromFile(" + file + "): invalid type syntax at line " + line + ", expected: T := T..."
				Continue
			EndIf

			Local dtype:Int = IR_T_NONE
			Local ltype:Int = IR_T_NONE
			Local rtype:Int = IR_T_NONE
			Local ops:String[] = Null
			Local code:String = Null
			
			dtype = IrTypeFromString(a[0].Trim())
			If dtype = IR_T_NONE Then
				DebugLog "TirTypeMap.ReadFromFile(" + file + "): invalid type '" + a[0] + "' at line " + line
				Continue
			EndIf

			s = a[1].Trim()
			a = s.Split(Null)
			If a.Length < 1 Then
				DebugLog "TIrTypeMap.ReadFromFile(" + file + "): invalid syntax at line " + line + ", expected: T..."
				Continue			
			EndIf

			Local mode:Int = 0 ' 0=types 1=ops
			Local i:Int = 0
			While i < a.Length
				s = a[i].Trim()
				If Not s Then
					i :+ 1
					Continue
				EndIf
				If s.StartsWith("(") Then ' enter ops mode
					If s.EndsWith(")") And s.Length > 2 Then
						ops :+ [ s[1.. s.Length - 1].Trim() ]
						mode = 0
						i :+ 1
						Continue
					ElseIf s.Length > 1 Then
						ops :+ [ s[1 ..].Trim() ]
					EndIf
					mode = 1
				ElseIf s.EndsWith(")") Then ' enter type mode
					If s.Length > 1 Then ops :+ [ s[.. s.Length - 1] ]
					mode = 0
				ElseIf s = "->" Then ' read code and exit
					code = " ".Join(a[i + 1..])
					Exit
				ElseIf mode = 0 Then
					If ops.Length = 0 Then
						If ltype = IR_T_NONE Then
							ltype = IrTypeFromString(s)
						Else
							DebugLog "TIrTypeMap.ReadFromFile(" + file + "): left-type already specified " + line
							Exit
						EndIf
					ElseIf ops.Length > 0 Then
						If ltype = IR_T_NONE And rtype = IR_T_NONE Then
							rtype = IrTypeFromString(s)
						ElseIf ltype <> IR_T_NONE And rtype = IR_T_NONE Then
							rtype = IrTypeFromString(s)
						Else
							DebugLog "TIrTypeMap.ReadFromFile(" + file + "): right-type already specified " + line
							Exit
						EndIf
					EndIf
				ElseIf mode = 1 Then
					ops :+ [ s ]
				EndIf
				i :+ 1
			Wend
			
			If ops.Length = 0 Then
				If ltype = IR_T_NONE Then
					DebugLog "TirTypeMap.ReadFromFile(" + file + "): missing left-type at line " + line
					Continue
				EndIf
				Local typemap:TIrTypeMapping = New TIrTypeMapping
				typemap.Op = AssignOp
				typemap.RType = ltype
				typemap.DType = dtype
				typemap.Code = code
				InsertMap(typemap)
				Continue
			EndIf
			
			For Local op:String = EachIn ops
				Local opmap:TIrOpMapping = LookupOp(op)
				If Not opmap Then
					DebugLog "TirTypeMap.ReadFromFile(" + file + "): invalid op '" + op + "' at line " + line
					Continue
				EndIf
				
				If opmap.IsBinary Then
					If (ltype = IR_T_NONE Or rtype = IR_T_NONE) Then
						DebugLog "TirTypeMap.ReadFromFile(" + file + "): binary op '" + opmap.Name + "' missing types at line " + line
						Continue
					EndIf
				Else
					If (ltype <> IR_T_NONE Or rtype = IR_T_NONE) Then
						DebugLog "TirTypeMap.ReadFromFile(" + file + "): unary op '" + opmap.Name + "' missing types at line " + line
						Continue
					EndIf
				EndIf
				
				Local typemap:TIrTypeMapping = New TIrTypeMapping
				typemap.Op = opmap
				typemap.LType = ltype
				typemap.RType = rtype
				typemap.DType = dtype
				typemap.Code = code
				InsertMap(typemap)
				count :+ 1
				
				' also add with swapped types
				If opmap.IsBinary And ltype <> IR_T_NONE And rtype <> IR_T_NONE And ltype <> rtype Then
					typemap = New TIrTypeMapping
					typemap.Op = opmap
					typemap.LType = rtype
					typemap.RType = ltype
					typemap.DType = dtype
					typemap.Code = swapends( code, "$O").Replace( "$L", "$_").Replace( "$R", "$L"). Replace( "$_", "$R")
					InsertMap(typemap)
					count :+ 1
				EndIf
			Next
		Wend
		stream.Close()
		Return count
	EndMethod
	
	Method CanAssign:Int( dst:Int, src:Int)
		If dst = IR_T_NONE Or src = IR_T_NONE Then Return False
		If dst = src Then Return True
		Local typemap:TIrTypeMapping = New TIrTypeMapping
		typemap.DType = dst
		typemap.Op = AssignOp
		typemap.RType = src
		typemap = TIrTypeMapping( TypeMaps.ValueForKey(typemap.ToName()))
		If Not typemap Then Return IR_T_NONE
		Return typemap.DType = dst
	EndMethod
	
	Method GetBinaryResult:Int( ltype:Int, op:String, rtype:Int)
		Local typemap:TIrTypeMapping = New TIrTypeMapping
		typemap.LType = ltype
		typemap.Op = LookupOp(op)
		typemap.RType = rtype
		typemap = TIrTypeMapping( TypeMaps.ValueForKey(typemap.ToName()))
		If Not typemap Then Return IR_T_NONE
		Return typemap.DType
	EndMethod
	
	Method GetUnaryResult:Int( op:String, rtype:Int)
		Local typemap:TIrTypeMapping = New TIrTypeMapping
		typemap.Op = LookupOp(op)
		typemap.RType = rtype
		typemap = TIrTypeMapping( TypeMaps.ValueForKey(typemap.ToName()))
		If Not typemap Then Return IR_T_NONE
		Return typemap.DType
	EndMethod
EndType

'
' TEST
'
Local map:TIrTypeMap = New TIrTypeMap
map.ReadFromFile("typemap.txt")
Print "---------------------------------------------------------"
' swap the next line with the 2 below it for non crash behaviour
For Local m:TIrTypeMapping = EachIn map.TypeMaps.Values()
'For Local n:TNode = EachIn map.TypeMaps
'	Local m:TIrTypeMapping = TIrTypeMapping(n._value)
	Local s:String = m.ToName()[.. 15] + m.ToString()
	WriteStdout s[.. 42]
	If m.Code Then
		WriteStdout " -> "
		WriteStdout m.Code
	EndIf
	WriteStdout "~n"
Next

typemap.txt

// operator mappings
%binary
	ADD		+
	SUB		-
	MUL		*
	DIV		/
	MOD		Mod
	BAND	&
	BOR		|
	BXOR	~
	BSHL	Shl
	BSHR	Shr
	LAND	And
	LOR		Or
	EQ		=
	NEQ		<>
	LE			<
	LEQ		<=
	GR		>
	GRQ		>=
%unary
	NEG		-
	BNOT	~
	LNOT	Not
	ISZ		0 =
	ISNZ		0 <>
%%

// type assign mappings
Bool := Int
Bool := Float
Bool := String
Int := Float
Float := Int
String := Bool
String := Int
String := Float

// matching type mappings
Bool := Bool (LAND LOR EQ NEQ) Bool
Bool := (LNOT ISZ ISNZ) Bool

Int := Int (ADD SUB MUL DIV MOD BAND BOR BXOR BSHL BSHR) Int
Int := (NEG BNOT) Int																		
Bool := Int (LAND LOR EQ NEQ LE LEQ GR GRQ) Int
Bool := (LNOT ISZ ISNZ) Int

Float := Float (ADD SUB MUL DIV MOD) Float
Float := (NEG) Float
Bool := Float (LAND LOR) Float							-> $L:Bool $O $R:Bool
Bool := Float (EQ NEQ LE LEQ GR GRQ) Float
Bool := (LNOT) Float											-> $O $R:Bool
Bool := (ISZ ISNZ) Float

String := String (ADD) String
Bool := String (LAND LOR) String						-> $L:Bool $O $R:Bool
Bool := String (EQ NEQ LE LEQ GR GRQ) String
Bool := (LNOT ISZ ISNZ) String							-> $O $R:Bool

// mixed type mappings
Bool := Bool (LAND LOR EQ NEQ LE LEQ GR GRQ) Int		-> $L:Int $O $R
Bool := Bool (LAND LOR EQ NEQ) String							-> $L $O $R:Bool

Float := Int (ADD SUB MUL DIV MOD) Float						-> $L:Float $O $R
Bool := Int (LAND LOR EQ NEQ LE LEQ GR GRQ) Float		-> $L:Float $O $R

String := Bool (ADD) String			-> $L:String $O $R
String := Int (ADD) String				-> $L:String $O $R
String := Float (ADD) String			-> $L:String $O $R
Bool := Int (EQ NEQ) String			-> $L:String $O $R
Bool := Float (EQ NEQ) String		-> $L:String $O $R
Bool := Int (LAND LOR) String		-> $L:Bool $O $R:Bool
Bool := Float (LAND LOR) String		-> $L:Bool $O $R:Bool

Derron

(Posted 2015) [#2]

I had the "null" problem on a "NextNode()" last friday - when I freshly installed BlitzMax 1.50 on a Linux Mint (15 or 16?) 32Bit. The error did not happen when compiling it on my desktop (1.50, Mint 17,64Bit).

On the desktop I use the "maxmod"-variant of TMap. According to the changes it does not do change things (but it adds "concurrent modification"-checks).

But: I did not have issues on the 32bit-linux-laptop when directly working with "values()". It only segfaulted when using "themap.copy().Values()"

So if the reason is the same, some people wont see your problem while others might experience it.

Your example did not segfault here on 64bit Mint17. Tried release, release+threaded, debug.

bye
Ron

grable

(Posted 2015) [#3]

Hmm so this might be something relating to 32-bit OS then, as im on Windows 7 32-bit right now.
Never ran into this before on my main 64-bit rig.

And i should probably add that a GCCollect anywhere crashes too,
so it seems more and more to do with the single threaded GC really.

Brucey

(Posted 2015) [#4]

Assuming nothing has changed with BlitzMax, I wonder why it would suddenly start to behave erratically...

grable

(Posted 2015) [#5]

Yeah it is kinda weird, and i use TMap all the time so should have noticed something like this sooner.

I even tried it on the OpenSource version to no avail.

Only thing ive done is make it run on Mingw 5.1.0..
But il try a recompile on this machine, as i just copied the precompiled stuff over. Maybe gcc did something funny, who knows :p

Derron

(Posted 2015) [#6]

@grable

do you use prebuild modules or ones you recompiled? Vanilla "fasm" or an updated one?

@Brucey
I think so too ... but if it crashes, it crashes and most of us are left "puzzled" then.

bye
Ron

grable

(Posted 2015) [#7]

@Derron

I always recompile when i update to a new Mingw, so they should be fine.
And the version of fasm i use is 1.71.39.

grable

(Posted 2015) [#8]

Well that took its sweet time. 1 hour for a recompile, and that doesnt include thread mode!

Sadly it did nothing :(

Derron

(Posted 2015) [#9]

You should only need to recompile the modules you are using... only wxmax is what takes forever here.

Does your example crash with a "vanilla blitzmax 1.50"-compilation ?

bye
Ron

Floyd

(Posted 2015) [#10]

Does your example crash with a "vanilla blitzmax 1.50"-compilation ?

It crashes with my vanilla BlitzMax 1.50, and with 1.48 as well.

Both use FASM 1.69.14 and I have not changed/rebuilt anything since they were installed.

Brucey

(Posted 2015) [#11]

Crashes on OS X too.

Runs fine in NG though :-)

GW	(Posted 2015) [#12]

Not sure if it helps but inserting this

Local _count% = 0
For Local m:TIrTypeMapping = EachIn map.TypeMaps.Values()
	_count:+ 1
Next
'// continue to the offending loop
'// unchanged
For Local m:TIrTypeMapping = EachIn map.TypeMaps.Values()
'For Local n:TNode = EachIn map.TypeMaps
'	Local m:TIrTypeMapping = TIrTypeMapping(n._value)
	Local s:String = m.ToName()[.. 15] + m.ToString()
	WriteStdout s[.. 42]
	If m.Code Then
		WriteStdout " -> "
		WriteStdout m.Code
	EndIf
	WriteStdout "~n"
Next

just above your value iteration (~ln:360) stops the crashing.

strange, I've never had Map blow up on me before.

Derron

(Posted 2015) [#13]

This sounds as if the iterator is creating some trouble.

NextNode() fails (in my case "_node" was null then in the debug tree) .

bye
Ron

grable

(Posted 2015) [#14]

Ive done some more tests, and have more or less figured out why it happens. Exactly where though i have no idea :/

Things that does NOT crash:

1. GCSuspend before the loop, and GCResume/GCCollect afterwards.
2. Making the container type Global instead of Local.
3. Using container type as an argument to a function.
4. Adding a field to TNodeEnumerator that contains the TMap itself.

This leads me to believe that its a combination of the Single-Threaded GC and For-EachIn enumeration that frees the container type in the middle of the loop.

The easiest fix without changing the compiler is to just add a field to the enumerator to make sure its map can never go out of scope early.

But a proper fix would be better...