NewsForumsCodeLogsGallerySpecsSearch

SqLite Release vs Debug

BlitzMax Forums/Brucey's Modules/SqLite Release vs Debug

Cocopino(Posted 2017) [#1]
Hi,

I am using Brucey's sqlite mod which was working perfectly fine, now I am completely lost. The piece of code that used to work, but now does not anymore in release mode, is similar to this:


Local sql:String = "SELECT layer1,layer2,layer3 FROM map WHERE id=" + mapID
Local query:TDatabaseQuery = db.GetQuery(sql)

If query.nextRow()

	Local record:TQueryRecord = query.rowRecord()
	For Local i:Int = 0 To 2
		Local layer:String = record.getField(i).value.getString()
		Notify layer
	Next
	
EndIf


In debug mode, the messageboxes will read and show the contents of the database just fine. In release mode, only the first field of the record is shown with its correct contents, the other 2 fields are empty...?

It used to work in release mode, until now. I am using vanilla Blitzmax with Blide (I just pick debug or release from the listbox) and have never tinkered with the compiler or any of its settings.

Any ideas? Thanks


Cocopino(Posted 2017) [#2]
I "solved" it for now by removing record.getField from the for loop and storing the values in an array first.
Apparently, the record object is getting changed when calling record.getField(i), but only when compiled in release mode... ?

Blitzmax NG luckily does not have this behavior so it might be time to take the plunge.


Derron(Posted 2017) [#3]
Make sure to use the latest dbsqlite.mod and database.mod

Current code:
https://github.com/maxmods/bah.mod/tree/master/dbsqlite.mod


The "getField()" does not change anything, it just returns the entry of an array.

This is why you could replace
Local layer:String = record.getField(i).value.getString()
with (another getter...)
Local layer:String = TDBString(record.value(i)).value
or (raw access - no array boundaries check)
Local layer:String = TDBString(record.values[i]).value



@ other fields are empty
Do they just not get "Notify"-exposed or what do you exactly mean?


@ db.GetQuery()
As this is not a known Method to me I assume you extended the DB-Type to contain a new method - what is the content of "GetQuery()" ?




I created a sample code (out of "test_01.bmx" which creates a fake-data-db for your layer-stuff and fetches the data as it should - and it works for release and debug.
(I also replaced db.GetQuery() with db.executeQuery())
SuperStrict

Framework BaH.DBSQLite
Import BRL.filesystem
Import BRL.Retro

DeleteFile("maptest.db")

Local db:TDBConnection = LoadDatabase("SQLITE", "maptest.db")

If Not db Then
	DebugLog("Didn't work...")
	End
End If

Local rows:String[][] = [ ..
	[ "100", "layer1_A", "layer1_B", "layer1_C" ],   ..
	[ "200", "layer2_A", "layer2_B", "layer2_C" ], ..
	[ "300", "layer3_A", "layer3_B", "layer3_C" ] ]

If db.isOpen() Then

	Local s:String = "CREATE TABLE map (id integer primary key AUTOINCREMENT, " + ..
	  " mapID integer," + ..
	  " layer1 varchar(30)," + ..
	  " layer2 varchar(30)," + ..
	  " layer3 varchar(30))"

	db.executeQuery(s)

	If db.hasError() Then errorAndClose(db)



	For Local i:Int = 0 Until rows.length
		db.executeQuery("INSERT INTO map values (NULL, "+rows[i][0]+", '" + rows[i][1] + "', '" + rows[i][2] + "', '" + rows[i][3] + "')")
		If db.hasError() Then errorAndClose(db)
	Next



	'try also 200 or 300
	local mapID:int = 100

	Local sql:String = "SELECT layer1,layer2,layer3 FROM map WHERE mapID=" + mapID
	Local query:TDatabaseQuery = db.executeQuery(sql)

	If query.nextRow()
		Local record:TQueryRecord = query.rowRecord()
		For Local i:Int = 0 To 2
			Local layer:String = record.getField(i).value.getString()
			print layer
			'Notify layer
		Next
		
	EndIf

			
	db.close()

End If

Function errorAndClose(db:TDBConnection)
	DebugLog(db.error().toString())
	db.close()
	End
End Function



bye
Ron


Henri(Posted 2017) [#4]
@Ron
Actually, it might not.

I modified your example a bit:

SuperStrict

Framework BaH.DBSQLite
Import BRL.filesystem
Import BRL.Retro
Import BRL.System

DeleteFile("maptest.db")

Local db:TDBConnection = LoadDatabase("SQLITE", "maptest.db")

If Not db Then
	DebugLog("Didn't work...")
	End
End If

Local rows:String[][] = [ ..
	[ "100", "layer1_A", "layer1_B", "layer1_C" ],   ..
	[ "200", "layer2_A", "layer2_B", "layer2_C" ], ..
	[ "300", "layer3_A", "layer3_B", "layer3_C" ] ]

If db.isOpen() Then

	Local s:String = "CREATE TABLE map (id integer primary key AUTOINCREMENT, " + ..
	  " mapID integer," + ..
	  " layer1 varchar(30)," + ..
	  " layer2 varchar(30)," + ..
	  " layer3 varchar(30))"

	db.executeQuery(s)

	If db.hasError() Then errorAndClose(db)



	For Local i:Int = 0 Until rows.length
		db.executeQuery("INSERT INTO map values (NULL, "+rows[i][0]+", '" + rows[i][1] + "', '" + rows[i][2] + "', '" + rows[i][3] + "')")
		If db.hasError() Then errorAndClose(db)
	Next

	'try also 200 or 300
	Local mapID:Int = 100

	Local sql:String = "SELECT layer1,layer2,layer3 FROM map WHERE mapID=" + mapID
	Local query:TDatabaseQuery = db.executeQuery(sql)

	If query.nextRow()
		Local record:TQueryRecord = query.rowRecord()
		For Local i:Int = 0 To 2
			Local layer:String = record.getField(i).value.getString()
			
			If Not layer Then Notify("WHAT! NO LAYER???", True)
			AWastOfTime()
		Next
		
	EndIf

	db.close()
End If

Function errorAndClose(db:TDBConnection)
	DebugLog(db.error().toString())
	db.close()
	End
End Function

Function AWastOfTime()
	For Local i:Int = 0 Until 10000
		Print i + ".."
	Next
EndFunction


-Henri


Derron(Posted 2017) [#5]
Hmm, so what does that mean?

printing 10.000 times and "getfield()" returning nothing anymore...

Something does not make sure that its content is staying in memory (not referencing it anymore - eg. just a pointer to memory).


PS: 
Function AWastOfTime()
	For Local i:Int = 0 Until 10000
		local v:string = Rand(0,10000)
'		Print i + ".."
	Next
EndFunction


Results in the same. I cannot use "notify" here as this is not supported on Linux - so having "print" between 10k of prints is not nice ;-)



Edit:
Maybe One of the field's parents (TQueryRecord or so) gets garbage collected / "Delete()" - which would result in the fields getting emptied too.

bye
Ron


Cocopino(Posted 2017) [#6]
Once I saw this happening, the first thing I did was to download and build the latest dbsqlite.mod and database.mod, that did not change anything unfortunately.


@ other fields are empty
Do they just not get "Notify"-exposed or what do you exactly mean?



A messagebox is shown, but the message is empty after the first iteration. So somewhere inside the loop the fields of the record get altered or destroyed but I am not referencing it anywhere else in the loop.

'Fields have correct values in both debug and release
Local record:TQueryRecord = query.rowRecord()
Local layer0:String = record.getField(0).value.getString()
Local layer1:String = record.getField(1).value.getString()
Local layer2:String = record.getField(2).value.getString()

'Works in debug but only the first iteration has a value in release
Local record:TQueryRecord = query.rowRecord()
For Local i:Int = 0 To 2
	Local layer:String = record.getField(i).value.getString()
Next


Thanks for your example, it does work, but so did my code, once. What's really bothering me is that the same code can yield different results based on a compiler flag. So I am not convinced this is a problem in the sqlite mod... Garbage collection differences maybe?


Cocopino(Posted 2017) [#7]
I did not update the page in a while so I missed both your posts.
The example of Henri indeed shows a difference in debug and release...


Derron(Posted 2017) [#8]
The problem is that there are references freed which free the "fields" too ... (ClearBindValues()). So we have to find out _why_ they are freed instead of being kept.

Somehow "TDatabaseQuery.Delete()" is called "randomly" (when garbage collected). In DebugMode the GC does not see a need for "deletion" in that moment (but it might...) so it still works.

I think using a "prefetch" is just working with a huge amount of luck. If there is a hickup in the application execution (AV tools kicking in) then GC might run somewhen and you will still run into trouble.



bye
Ron


Brucey(Posted 2017) [#9]
Hallo :-)

Okay, this is what appears to be happening in the example code :
...
	Local query:TDatabaseQuery = db.executeQuery(sql)

	If query.nextRow()
		Local record:TQueryRecord = query.rowRecord()
		For Local i:Int = 0 To 2
			Local layer:String = record.getField(i).value.getString()
			
			If Not layer Then Notify("WHAT! NO LAYER???", True)
			AWastOfTime()
		Next
		
	EndIf
...

In the legacy/asm-generated BlitzMax(my version is OSX 1.50), the variable "query" is being garbage-collected while the code is in AWastOfTime().
That's obviously a bug with the generated ASM, as the local variable is still in scope until the end of the parent If statement.
It is not an issue in debug because of the way the generated code handles the references it needs to provide the debugger with variable information.
It's also not an issue in NG, because I'm making sure the GC will not collect such variables, even though they are in fact never directly used again in the local scope - let's call the problem, over-enthusiastic optimisation. (It had been an issue in early versions of NG when -O3 was used, but it's guaranteed now that the compiler won't over optimise them out.)

A workaround, for legacy BlitzMax is to reference "query" at some point later. For example, after the If statement, adding :
query.IsActive()

prevents the early collection.


Derron(Posted 2017) [#10]
And I crawled through the sources adding prints to narrow it down ... maybe file an official issue here:
https://github.com/blitz-research/blitzmax


bye
Ron


Cocopino(Posted 2017) [#11]
Thanks for clearing this up!
Had me boggled for a while.


Derron(Posted 2017) [#12]
Created an issue:
https://github.com/blitz-research/blitzmax/issues/8


Maybe someone has some tinker time with it.


bye
Ron


Derron(Posted March) [#13]
Mark closed that issue.

Bye
Ron