Reading and Writing Stream problems
BlitzMax Forums/BlitzMax Programming/Reading and Writing Stream problems
| ||
| This one's been bugging me for a few days now, and I can't seem to locate the error. The program is essentially a simple relational database wherein all the actual data is stored in a Bank field in the Table type, with two other types (FieldGroup and Field) used to hold offset information and fieldnames, etc. However, my code to save and load the data to the hard drive for further use isn't working properly - the data is garbled somewhere along the line and thus becomes unusable. Whether or not it works depends on the data being used, and if it does work, saving after adding further data to the database will bring up the same error after loading. Sorry if it's not too clear. The save function is:
'Data Saving
Function SaveData()
Local FileName:String = Input("Enter Filename: ")
Local file:TStream = WriteStream(FileName)
'save totals
WriteInt(file, TotalTables)
WriteInt(file, TotalFieldGroups)
WriteInt(file, TotalFields)
'save tables, then field groups, then fields
For Local tbl:DBTable = EachIn DBTableList
WriteLine(file, tbl.Name)
WriteInt(file, tbl.Id)
WriteInt(file, tbl.Records)
WriteInt(file, tbl.MaxRecords)
WriteInt(file, tbl.MaxSize)
WriteInt(file, tbl.Mode)
WriteInt(file, tbl.RecordSize)
For Local a = 0 To tbl.MaxRecords - 1
WriteInt(file, tbl.Offsets[a])
Next
'Bank Sizes
WriteInt(file, BankSize(tbl.DataBank))
WriteInt(file, BankSize(tbl.RecordFlags))
WriteBank(tbl.DataBank, file, 0, BankSize(tbl.DataBank))
WriteBank(tbl.RecordFlags, file, 0, BankSize(tbl.RecordFlags))
Next
For Local grp:DBFieldGroup = EachIn DBFieldGroupList
WriteLine(file, grp.Name)
WriteInt(file, grp.Id)
WriteInt(file, grp.Size)
WriteInt(file, grp.Table)
WriteInt(file, grp.Offset)
Next
For Local fld:DBField = EachIn DBFieldList
WriteLine(file, fld.Name)
WriteLine(file, fld.StrValue)
WriteLine(file, fld.DefaultStr)
WriteInt(file, fld.Id)
WriteInt(file, fld.DataType)
WriteInt(file, fld.Size)
WriteInt(file, fld.BitSize)
WriteInt(file, fld.Offset)
WriteInt(file, fld.Group)
WriteInt(file, fld.Table)
WriteInt(file, fld.IntValue)
WriteInt(file, fld.DefaultInt)
WriteInt(file, fld.Record)
Next
Print "Saved!"
CloseStream file
End Function
The load function is
Function LoadData()
For Local tbl:DBTable = EachIn DBTableList
ListRemove(DBTableList, tbl)
Next
For Local grp:DBFieldGroup = EachIn DBFieldGroupList
ListRemove(DBFieldGroupList, grp)
Next
For Local fld:DBField = EachIn DBFieldList
ListRemove(DBFieldList, fld)
Next
Print "Existing Data is wiped!"
Local FileName:String = Input("Enter Filename: ")
Local file:TStream = ReadStream(FileName)
'load in totals
TotalTables = Readint(file)
TotalFieldGroups = Readint(file)
TotalFields = Readint(file)
'load tables, then field groups, then fields
For Local count = 1 To TotalTables
Local tbl:DBTable = New DBTable
tbl.Name = ReadLine(file)
tbl.Id = Readint(file)
tbl.Records = Readint(file)
tbl.MaxRecords = Readint(file)
tbl.MaxSize = Readint(file)
tbl.Mode = Readint(file)
tbl.RecordSize = Readint(file)
tbl.Offsets = New Int[tbl.MaxRecords]
For Local a = 0 To tbl.MaxRecords - 1
tbl.Offsets[a] = Readint(file)
Next
Local DataBankSize = Readint(file)
Local RecordFlagSize = Readint(file)
tbl.DataBank = CreateBank(DataBankSize)
tbl.RecordFlags = CreateBank(RecordFlagSize)
ReadBank(tbl.DataBank, file, 0, DataBankSize)
ReadBank(tbl.RecordFlags, file, 0, RecordFlagSize)
ListAddLast(DBTableList, tbl)
Next
For Local count = 1 To TotalFieldGroups
Local grp:DBFieldGroup = New DBFieldGroup
grp.Name = ReadLine(file)
grp.Id = Readint(file)
grp.Size = Readint(file)
grp.Table = Readint(file)
grp.Offset = Readint(file)
ListAddLast(DBFieldGroupList, grp)
Next
For Local count = 1 To TotalFields
Local fld:DBField = New DBField
fld.Name = ReadLine(file)
fld.StrValue = ReadLine(file)
fld.DefaultStr = ReadLine(file)
fld.Id = Readint(file)
fld.DataType = Readint(file)
fld.Size = Readint(file)
fld.BitSize = Readint(file)
fld.Offset = Readint(file)
fld.Group = Readint(file)
fld.Table = Readint(file)
fld.IntValue = Readint(file)
fld.DefaultInt = Readint(file)
fld.Record = Readint(file)
ListAddLast(DBFieldList, fld)
Next
Print "Loaded!"
CloseStream file
End Function
I'm wondering if I'm doing something wrong with the way I'm saving and loading the data. I've scoured the code for hours trying to find if I'm forgetting to save an important piece of data, or if I'm correctly creating the Table, FieldGroup and Field types and adding them to their list (although I'm convinced that's not the problem, as the data read in is garbled before I add the object to the list). Note that I've mainly only encountered a problem with the "Field" data - never with the FieldGroup or Table data (indeed, the banks are being read and stored correctly). The field data ends up with offsets in the tens of thousands, bitsizes far too big and results in garbled data. On one occasion, however, I did get a error reading a databank (trying to access a memory location out of range), but never had enough time to find the exact cause - it's likely part of the field offset problem. Any help is greatly, greatly appreciated. |
| ||
| I don't see anything wrong right off the bat. You might try adding a number of debuglog statements to make sure the values you are writing are really what you think they are. Another possible issue is that you are mixing binary and text mode commands. Don't use WriteLine and ReadLine, use WriteInt to write the length of the string data and then WriteString to write the char data. The when reading use ReadInt and ReadString. I'm not sure but the use of Write/Read Line might be throwing you off somewhere. Does the problem alway happen? What data causes it to happen? What data doesn't cause it to happen? |
| ||
| I'll certainly give the WriteInt/WriteString combo a whirl, I might get lucky but you never know. The problem doesn't always happen, but I've yet to properly isolate the cases in which it does occur. I have stepped through the code using the debugger, and the values I'm writing are the ones that should be written. When read, they're screwed up - but only for the Field types - the preceeding Tables and FieldGroups are read fine. One case in which it definitely happens is as follows: There are two objects of type Table, two of type FieldGroup, and four of type Field (meaning each table has one field group, with two fields in it). When created in one sitting, this saves and loads perfectly well. When I then add another Table, with one Field Group and two Fields, then save and reload that (including when starting the program from scratch) the problem occurs - when I select an option to view the contents, the fields themselves are incorrectly read due to the screwed field data. |
| ||
| In that case we need to see the rest of your code. The entire type definitions for Table, FieldGroup and Field. I suspect you have something messed up in your list handling. |
| ||
Ok, the lists are set up as follows:Global DBTableList:TList Global DBFieldGroupList:TList Global DBFieldList:TList and then initialised upon the running of the code with 'Initiate System Function InitialiseDBOne() DBTableList = New TList DBFieldGroupList = New TList DBFieldList = New TList End Function Each type includes methods, including one to create an object and add it to the relevant list Table: Type DBTable Field Name:String 'name of the table Field Id:Int 'Id code for the table Field Records:Int 'number of records Field MaxRecords:Int 'maximum number of records Field MaxSize:Int 'maximum size of area (for dynamic tables) Field Mode:Int 'dynamic or static mode Field RecordSize:Int 'size of a record (static only) Field DataBank:TBank 'bank used to store all data Field RecordFlags:TBank 'bank used to store flags for record existence Field Offsets:Int[] 'array used to store offsets of records (dynamic). May become bank 'Function to create a new table Function CreateTable:DBTable(Name:String, Id:Int, MaxRecords:Int, Mode:Int, MaxSize:Int) Local this:DBTable = New DBTable this.Name = Name this.Id = Id this.Records = 0 this.MaxRecords = MaxRecords this.Mode = Mode this.RecordSize = -1 this.Offsets = New Int[MaxRecords] TotalTables = TotalTables + 1 ListAddLast(DBTableList, this) Return this End Function End Type FieldGroup Type DBFieldGroup Field Name:String 'name of the field group Field Id:Int 'Id code for the group Field Size:Int 'size of field group in bytes (depends on component fields) Field Table:Int 'ID of parent table Field Offset:Int 'offset of field group from start of record Function CreateFieldGroup:DBFieldGroup(Name:String, Id:Int, Table:Int) Local this:DBFieldGroup = New DBFieldGroup this.Name = Name this.Id = Id this.Size = -1 this.Table = Table this.Offset = -1 TotalFieldGroups = TotalFieldGroups + 1 ListAddLast(DBFieldGroupList, this) Return this End Function End Type Field Type DBField Field Name:String 'name of the field Field Id:Int 'ID code for the field Field DataType:Int 'type of data (String, int, BitPattern etc.) Field Size:Int 'size of data (e.g. characters, max value) Field BitSize:Int 'size of data in bits Field Offset:Int 'offset in bits of field from group Field Group:Int 'ID of parent group Field Table:Int 'ID of parent table 'Values Field StrValue:String 'string value (if string) Field IntValue:Int 'int value (if int) Field DefaultStr:String 'default string value Field DefaultInt:Int 'default int value Field Record:Int 'current record stored Function CreateField:DBField(Name:String, Id:Int, DataType:Int, Size:Int, Group:Int, Table:Int, Str:String = "", Integer:Int = 0) Local this:DBField = New DBField this.Name = Name this.Id = Id this.DataType = DataType this.Size = Size this.Offset = -1 this.Group = Group this.Table = Table this.DefaultInt = Integer this.DefaultStr = Str If this.DataType = 0 Then this.BitSize = this.Size * 8 Else If this.DataType = 2 Then this.BitSize = this.Size Else this.BitSize = GetBitSize(this.Size) End If TotalFields = TotalFields + 1 ListAddLast(DBFieldList, this) Return this End Function Method ReadValue(rec:Int) Record = rec If DataType = 0 Then ThisReadStr() Else ThisReadInteger() End Method Method WriteValue(rec:Int) Record = rec If DataType = 0 Then ThisWriteStr() Else ThisWriteInteger() End Method Method ThisReadStr() 'get record offset Local RecordOff 'Get table for databank For Local tbl:DBTable = EachIn DBTableList If tbl.id = Table Then RecordOff = tbl.RecordSize * record StrValue = ReadStr(tbl.DataBank, RecordOff + (Offset / 8), Size) End If Next End Method Method ThisWriteStr() 'get record offset Local RecordOff 'get table for databank For Local tbl:DBTable = EachIn DBTableList If tbl.id = Table Then RecordOff = tbl.RecordSize * record WriteStr(tbl.DataBank, RecordOff + (Offset / 8), StrValue) End If Next End Method Method ThisReadInteger() 'get record offset Local RecordOff 'get table for databank For Local tbl:DBTable = EachIn DBTableList If tbl.id = Table Then RecordOff = tbl.RecordSize * record IntValue = ReadInteger(tbl.DataBank, (RecordOff * 8) + Offset, BitSize) End If Next End Method Method ThisWriteInteger() 'get record offset Local RecordOff 'get table for databank For Local tbl:DBTable = EachIn DBTableList If tbl.id = Table Then RecordOff = tbl.RecordSize * record WriteInteger(tbl.DataBank, (RecordOff * 8) + Offset, BitSize, IntValue) End If Next End Method Method StoreString(str:String) If str.length <= Size Then StrValue = str Else StrValue = str[0..(size-1)] End If End Method Method StoreInt(integer:Int) If integer > Size Then integer = 0 End If IntValue = integer End Method End Type Throughout the code, the objects are only accessed via those methods, and directly during the Save and Load functions. |
| ||
| I don't suppose you can send me a zip file with the program and source I would like to see it running in front of me. Check my profile for an email addr. |
| ||
| I've sent the two source files, haven't got access to the built files right now as I'm posting from work (no net access at home at the moment, sadly). Thanks |
| ||
| Okay I got the files, I'm having a hard time making it screw up. Test 1: Started a new database Added a new table "test" added a Group "group1" added field "string" type String Max chars 10 Default 0123456789 added field "int" type Int Max value 255 Default 0 Saved db1 reloaded no problem Test 2: Load db1 Add 3 records to table "test" abcdefghij,1 bcdefghijk,2 cdefghijkl,3 view records, okay save db2 quit run load db2 view records, okay Test 3: Added a new table "second" added a Group "group2" added field "string" type String Max chars 10 Default 0123456789 added field "int" type Int Max value 255 Default 0 Saved db3 quit run load db3 view records, okay Test 4: Add 1 records to table "second" abcdefghij,1 view records, okay save db4 quit run load db4 view records, okay Test 5: Added a new table "third" added a Group "group3" added field "string" type String Max chars 10 Default 0123456789 added field "int" type Int Max value 255 Default 0 Saved db4 quit run load db4 view records, okay EDIT: just notice that if you enter a string in a string field that has a max chars of 10 and the string is longer than 10 it truncates the string to 9 characters. Still saves and load okay though. |
| ||
| I haven't been able to lock it down yet but there is something wrong with tables that have more than one int field. EDIT: okay the problem shows when you have two or more tables where one or more of them have multiple int fields. You don't even have to save/load the database. EDIT: well you've got me it appears to be totally random when it fails, except that I've been able to make it fail with tables that use multiple int fields. |
| ||
| Another possible problem, create a single table with a single group with a string and an int field. enter a single record then try to view the record. the program doesn't list the value of the int field only the string field and it seems to hang until you hit enter. Here is the out of the session where i did this. DatabaseOne Main Menu 1. Load Database 2. Save Database 3. Create Table 4. Edit Table 5. Data Entry 6. View Records 7. View Tables 8. Exit 9. DEBUG TABLE BANKS Select option: 3 Table Name: table1 Maximum Records: 10 Mode (Static/Dynamic): static Field Group Name: group1 Number of Fields: 2 Field Name: string1 Data Type (String/Int/Bit): string Size (Characters/Max Value/Bits): 10 Default: empty Field Name: int1 Data Type (String/Int/Bit): int Size (Characters/Max Value/Bits): 255 Default: 0 Another Field Group (y/n)? n Now Packing... Memory Used: 6730 DatabaseOne Main Menu 1. Load Database 2. Save Database 3. Create Table 4. Edit Table 5. Data Entry 6. View Records 7. View Tables 8. Exit 9. DEBUG TABLE BANKS Select option: 7 0. table1 (STATIC, 0/10) View any table, -1 to exit: 0 Table: table1 Group: group1 0, string1 (STRING, 10 Chars) 1, int1 (INT, Max 255) Link Field (-1 to exit): -1 Memory Used: 6730 DatabaseOne Main Menu 1. Load Database 2. Save Database 3. Create Table 4. Edit Table 5. Data Entry 6. View Records 7. View Tables 8. Exit 9. DEBUG TABLE BANKS Select option: 5 0, table1 (0/10) Enter data for which table? 0 Group: group1 string1: Scott int1: 1 Memory Used: 6752 DatabaseOne Main Menu 1. Load Database 2. Save Database 3. Create Table 4. Edit Table 5. Data Entry 6. View Records 7. View Tables 8. Exit 9. DEBUG TABLE BANKS Select option: 6 0, table1 (1/10) View data for which table? 0 0, Scott Group: group1 string1: Scott Memory Used: 6762 DatabaseOne Main Menu 1. Load Database 2. Save Database 3. Create Table 4. Edit Table 5. Data Entry 6. View Records 7. View Tables 8. Exit 9. DEBUG TABLE BANKS Select option: 7 0. table1 (STATIC, 1/10) View any table, -1 to exit: 0 Table: table1 Group: group1 0, string1 (STRING, 10 Chars) 1, int1 (INT, Max 255) Link Field (-1 to exit): Process terminated |
| ||
| Thanks for your help so far Scott, I'll get a good look at those issues tonight. I think I may have solved any direct saving/loading issue. The way I was writing strings to the databank, I neglected to fill the remainder of the allocated space (e.g. writing a four-character string to a ten-character space, I'd need to fill the last six with spaces) which then caused problems. This space would have been a list of null characters (byte value #00). This is fine, until you read the value from the bank - instead of getting, say, "TEST", you'd get "TEST" followed by 16 null values, which completely screws up the saving and subsequent loading. This explains why it only occurs in the Field type - as it is only fields that read and write data to Banks. By filling the remainder with space characters, and using Trim() after reading them, the problem seems to have disappeared. I'm getting a few other problems which may be related to what you've pointed out, so I'll give those a whirl. It's also good to note that sometimes the Output screen of BlitzMax fails to display properly - it hangs as you have indicated. It should work fine in a normal command window when running the EXE directly. If not, it's definitely a bug (probably a bug anyway, but I'm baffled by it). |
   |