File System

Monkey Forums/Monkey Code/File System

GfK	(Posted 2011) [#1]

Here's a basic pseudo file system for Monkey; tested and working on Android devices (LG Optimus One), iOS emulator, HTML5 and Flash targets.

Self explanatory for the most part and I've only added the features I need for myself right now. Basically all the Write<whatever> functions add data into a string corresponding to the filename opened with WriteFile. Data is not persistent until you call filesystem.SaveAll().

[edit 13-Sep-11]

Updated to fix a couple of issues as well as now avoiding having ASCII 0 in the string (which caused a load of problems) - Thanks muddy_shoes for the String/Int conversion.

Also I've added in Little Endian string/int conversion code in the conversion class. These are useful if you've converted something from Blitzmax to Monkey, as you can load your level data with LoadString, and give the conversion method a four-byte string which it will then convert to an integer.

test.monkey

Strict

Import filesystem

Function Main:Int()
	New testApp
	Return 0	
End Function

Class testApp Extends App
	Field fileHandler:FileSystem

		Method OnCreate:Int()
			Self.fileHandler = New FileSystem
			
			Local stream:FileStream
			Local n:int
			stream = Self.fileHandler.WriteFile("test/test.bin")
			stream.WriteString("Hello")
			stream.WriteInt(1234536343)
			stream.WriteString("Bye!")
			
			Self.fileHandler.SaveAll()
			Self.fileHandler.ListDir()
			
			stream = Self.fileHandler.ReadFile("test/test.bin")
			if stream
				Print stream.ReadString()
				Print stream.ReadInt()
				Print stream.ReadString()
			EndIf
			Return 0
		End Method
End Class

filesystem.monkey

Strict
Import mojo

Class FileSystem Extends DataConversion
Private
	Field _header:String = "MKYDATA"
	Field fileData:String
	Field index:StringMap<FileStream>
Public
	
	Method New()
		Self.LoadAll()
	End
	
	Method WriteFile:FileStream(filename:String)
		Local f:FileStream = new FileStream
		f.filename = filename.ToLower()
		f.fileptr = 0
		Self.index.Insert(f.filename.ToLower(),f)
		Return f	
	End
	
	Method ReadFile:FileStream(filename:String)
		filename = filename.ToLower()
		Local f:FileStream
		f = Self.index.ValueForKey(filename)
		f.fileptr = 0
		Return f
	End
	
	Method FileExists:Bool(filename:String)
		filename = filename.ToLower()
		if Self.index.Contains(filename)
			Return True
		Else
			Return False
		End
		Return False
	End
	
	Method ListDir:Void()
		Local filename:String
		Local stream:FileStream
		Print "Directory Listing:"
		For filename = EachIn Self.index.Keys()
			stream = Self.index.ValueForKey(filename)
			Print filename + "   " + stream.data.Length()+" byte(s)."
		Next
	End
	
	Method DeleteFile:Void(filename:String)
		filename = filename.ToLower()
		if Self.index.Contains(filename)
			Self.index.Remove(filename)
		End
	End
	
	Method SaveAll:Void()
		Local f:FileStream
		Self.fileData = Self._header'header
		self.fileData+= Self.IntToString(Self.index.Count())'number of files in index
		if Self.index.Count() > 0
			For f = EachIn Self.index.Values()
				'store filename
				Self.fileData+= Self.IntToString(f.filename.Length())
				if f.filename.Length() > 0
					Self.fileData+= f.filename
				End
				'store data
				Self.fileData+= Self.IntToString(f.data.Length())
				if f.data.Length() > 0
					Self.fileData+= f.data
				End
			Next
		End
		SaveState(Self.fileData)
	End
	
	Method LoadAll:Void()
		Local numFiles:Int
		Local stream:FileStream
		Local len:Int
		Local ptr:Int
		Self.fileData = LoadState()
		self.index = New StringMap<FileStream>
		if Self.fileData.Length() > 0
			if Self.fileData.StartsWith(Self._header)
				Self.index.Clear()
				ptr+=Self._header.Length()
				numFiles = Self.StringToInt(Self.fileData[ptr..ptr+3])
				ptr+=3
				if numFiles > 0				
					For Local n:Int = 1 to numFiles
						stream = New FileStream
						'filename
						len = Self.StringToInt(Self.fileData[ptr..ptr+3])
						ptr+=3
						if len > 0
							stream.filename = Self.fileData[ptr..ptr+len]
							ptr+=len
						End
						'data
						len = Self.StringToInt(Self.fileData[ptr..ptr+3])
						ptr+=3
						if len > 0
							stream.data = Self.fileData[ptr..ptr+len]
							ptr+=len
						End
						Self.index.Insert(stream.filename,stream)
					Next
				End
			End
		Else
			SaveState("")'save empty file and indicate no files stored
		End
	End
End



Class FileStream Extends DataConversion
	Field filename:String
	Field fileptr:Int
Private
	Field data:String
Public
	
	Method ReadInt:Int()
		Local result:string
		result = Self.data[Self.fileptr..self.fileptr+3]
		Self.fileptr+=3
		Return Self.StringToInt(result)
	End
	
	Method WriteInt:Void(val:Int)
		Self.data+=Self.IntToString(val)
	End
	
	Method ReadString:String()
		Local result:String
		Local strLen:Int = self.StringToInt(self.data[self.fileptr..self.fileptr+3])
		Self.fileptr+=3
		if strLen > 0
			result = Self.data[Self.fileptr..self.fileptr+strLen]
			Self.fileptr+=strLen
		End
		Return result
	End
	
	Method WriteString:Void(val:String)
		Self.data+=Self.IntToString(val.Length())
		if val.Length() > 0
			Self.data+=val
		End
	End
	
	Method ReadFloat:Float()
		Local result:float
		Local s:String
		Local strLen:Int = self.StringToInt(self.data[self.fileptr..self.fileptr+3])
		Self.fileptr+=3
		s = Self.data[Self.fileptr..self.fileptr+strLen]
		result = Self.StringToFloat(s)
		Self.fileptr+=strLen
		Return result
	End
	
	Method WriteFloat:Void(val:Float)
		Local s:String = self.FloatToString(val)
		Self.data+=Self.IntToString(s.Length())
		Self.data+=s
	End
	
	Method ReadBool:Bool()
		Local result:Bool
		result = Bool(Self.data[Self.fileptr])
		Self.fileptr+=1
		Return result
	End Method
	
	Method WriteBool:Void(val:Bool)
		Self.data+=String.FromChar(val)
	End Method	
End

Class DataConversion
	Method LittleEndianIntToString:String(val:Int)
		Local result:String
		result = String.FromChar((val) & $FF)
		result+= String.FromChar((val Shr 8) & $FF)
		result+= String.FromChar((val Shr 16) & $FF)
		result+= String.FromChar((val Shr 24) & $FF)
		Return result
	End

	Method StringToLittleEndianInt:Int(val:String)
		Local result:Int
		result = (val[0])
		result|= (val[1] Shl 8)
		result|= (val[2] Shl 16)
		result|= (val[3] Shl 24)
		Return result
	End	
	
    Method IntToString:String(val:Int)
        Local result:String
        result = String.FromChar($F000 | ((val Shr 20) & $0FFF) )
        result += String.FromChar($F000 | ((val Shr 8) & $0FFF))
        result += String.FromChar($F000 | (val & $00FF))
        Return result
    End
        
    Method StringToInt:Int(val:String)
        Return ((val[0]&$0FFF) Shl 20) | ((val[1]&$0FFF) Shl 8) |(val[2]&$00FF)
    End


	Method FloatToString:String(val:Float)
		Return String(val)
	End		
	
	Method StringToFloat:Float(val:String)
		Return Float(val)
	End		
End

c.k.	(Posted 2011) [#2]

Awesome! Somebody owes you a beverage of your choice! :)

Qcat	(Posted 2011) [#3]

Fantastic! i think we all owe you a beverage :)

therevills

(Posted 2011) [#4]

Cool... can I make one suggestion though, stick to a standard in your coding, you've sometimes got PascalCase methods and sometimes camelCase methods - and since Monkey is case-sensitive it makes sense to keep everything the same.

For the docs:

Monkey naming convention

The standard Monkey modules use a simple naming convention:

All-caps case (eg: 'ALLCAPS' ):
* Constants

Pascal case (eg: 'PascalCase' ):
* Classes
* Globals
* Functions, methods and properties.

Camel case (eg: 'camelCase' ):
* Fields
* Locals and function parameters

Also would mind if I stick this in Diddy? ;)

muddy_shoes

(Posted 2011) [#5]

As I pointed out in your other thread, your int conversion is using twice as many characters as necessary. More efficient versions are at the bottom of the float conversion code I posted.

GfK	(Posted 2011) [#6]

Cool... can I make one suggestion though, stick to a standard in your coding, you've sometimes got PascalCase methods and sometimes camelCase

I normally use camelCase! For this, though, I used the PascalCase for the duplicates of the Blitzmax commands (because that's how they are in Blitzmax!) so I guess I did it that way for familiarity. Admittedly there are a couple of rogue ones in there though.

Also would mind if I stick this in Diddy? ;)

Don't mind at all.

therevills

(Posted 2011) [#7]

Ta, just committed it to Diddy (after a quick tidy up):

http://code.google.com/p/diddy/source/browse/trunk/src/diddy/filesystem.monkey

Have you tested muddy_shoes suggestions?

GfK	(Posted 2011) [#8]

Not yet but feel free.

[Edit] it says its 4 or 5 times slower so i think I'll pass. up to you for diddy tho.

therevills

(Posted 2011) [#9]

Cheers for testing it Dave... 4 or 5 times slower is a hugh amount... think Diddy will stay with your version..

muddy_shoes

(Posted 2011) [#10]

Cheers for testing it Dave

I don't get the impression any testing was done. He's just noting what I wrote about my testing and seems to have misinterpreted. The float conversion is "up to" 4-5 times slower. In other words, it's the worst case from my tests, the average case is not that bad as shown by the Android figures that I gave. The trade off is the reduction in string size which seemed to be the concern being voiced about using SaveState.

The integer conversion is no slower, in fact I'd expect it to be faster as well as more space efficient as there are fewer operations involved in the packing and unpacking.

GfK	(Posted 2011) [#11]

No I didn't test it - I was just going on what you said.

I won't get around to it for a few hours tho, if at all today, cos I'm busy on something right now.

If anybody wants to update and test the code in the meantime with something they think might be more efficient, knock yourselves out. That's what I put it here for!

muddy_shoes

(Posted 2011) [#12]

Okay, so here are some test results using the Diddy module as a base. The numbers after the operation descriptions are in ms. Note that the byte values are what the module prints out based on the assumption that 1 char = 1 byte. That assumption is actually incorrect but the number is okay for comparing storage requirements.

Starting with writing and reading 20,000 floats and 20,000 ints on Flash, which appears to have the worst performance of the PC-based targets for me.

Current code:

Write floats: 165
Directory Listing:
test.bin 464780 byte(s).
Read floats: 32
Write ints: 30
Directory Listing:
test.bin 80000 byte(s).
Read ints: 7

Alternative:

Write floats: 204
Directory Listing:
test.bin 141565 byte(s).
Read floats: 89
Write ints: 18
Directory Listing:
test.bin 40000 byte(s).
Read ints: 1

My conversion code is slightly slower when writing floats and 3-4 times as slow when reading. However, the actual difference that the user would experience when reading a level or similar would be unnoticeable and the storage requirements are about a third of the current implementation. As I expected, the integer read and write times are about halved along with the storage used.

Now on Android, using 1000 floats and ints. Current code:

Write floats: 14914
Directory Listing:
test.bin 14544 byte(s).
Read floats: 149
Write ints: 1803
Directory Listing:
test.bin 4000 byte(s).
Read ints: 66

Alternative code:

Write floats: 4945
Directory Listing:
test.bin 5011 byte(s).
Read floats: 491
Write ints: 917
Directory Listing:
test.bin 2000 byte(s).
Read ints: 3

The reduction in overall string length means that writing is much faster with the alternative methods. As with Flash the read cost for floats is 3-4 times higher.

My altered version of the Diddy file is below. There are still a number of tweaks that could improve matters further and I'm also wondering if a Monkey StringBuilder/StringBuffer type class backed by native implementations would make a big difference.

Strict
Import mojo

Class FileSystem Extends DataConversion
Private
	Field _header:String = "MKYDATA"
	Field fileData:String
	Field index:StringMap<FileStream>
Public
	
	Method New()
		Self.LoadAll()
	End
	
	Method WriteFile:FileStream(filename:String)
		Local f:FileStream = new FileStream
		f.filename = filename
		f.fileptr = 0
		Self.index.Set(f.filename.ToLower(),f)
		Return f	
	End
	
	Method ReadFile:FileStream(filename:String)
		filename = filename.ToLower()
		Local f:FileStream
		f = Self.index.ValueForKey(filename)
		f.fileptr = 0
		Return f
	End
	
	Method FileExists:Bool(filename:String)
		filename = filename.ToLower()
		if Self.index.Contains(filename)
			Return True
		Else
			Return False
		End
	End
	
	Method ListDir:Void()
		Local filename:String
		Local stream:FileStream
		Print "Directory Listing:"
		For filename = EachIn Self.index.Keys()
			stream = Self.index.ValueForKey(filename)
			Print filename + "   " + stream.data.Length()+" byte(s)."
		Next
	End
	
	Method DeleteFile:Void(filename:String)
		filename = filename.ToLower()
		if Self.index.Contains(filename)
			Self.index.Remove(filename)
		End
	End
    
    Method DeleteAll:Void()
        For Local filename := EachIn Self.index.Keys()
			Self.index.Remove(filename)
		Next
    End
	
	Method SaveAll:Void()
		Local f:FileStream
		Self.fileData = Self._header'header
		self.fileData+= Self.IntToString(Self.index.Count())'number of files in index
		if Self.index.Count() > 0
			For f = EachIn Self.index.Values()
				'store filename
				Self.fileData+= Self.IntToString(f.filename.Length())
				if f.filename.Length() > 0
					Self.fileData+= f.filename
				End
				'store data
				Self.fileData+= Self.IntToString(f.data.Length())
				if f.data.Length() > 0
					Self.fileData+= f.data
				End
			Next
		End
		SaveState(Self.fileData)
	End
	
	Method LoadAll:Void()
		Local numFiles:Int
		Local stream:FileStream
		Local len:Int
		Local ptr:Int
		Self.fileData = LoadState()
		self.index = New StringMap<FileStream>
		if Self.fileData.Length() > 0
			if Self.fileData.StartsWith(Self._header)
				Self.index.Clear()
				ptr+=Self._header.Length()
				numFiles = Self.StringToInt(Self.fileData,ptr)
				ptr+=CharsPerInt
				if numFiles > 0
					For Local n:Int = 1 to numFiles
						stream = New FileStream
						'filename
						len = Self.StringToInt(Self.fileData,ptr)
						ptr+=CharsPerInt
						if len > 0
							stream.filename = Self.fileData[ptr..ptr+len]
							ptr+=len
						End
						'data
						len = Self.StringToInt(Self.fileData,ptr)
						ptr+=CharsPerInt
						if len > 0
							stream.data = Self.fileData[ptr..ptr+len]
							ptr+=len
						End
						Self.index.Set(stream.filename,stream)
					Next
				End
			End
		Else
			SaveState(Self._header+Self.IntToString(0))'save empty file and indicate no files stored
		End
	End
End



Class FileStream Extends DataConversion
	Field filename:String
	Field fileptr:Int
Private
	Field data:String
Public
	
	Method ReadInt:Int()
	    Self.fileptr+=CharsPerInt
        Return StringToInt(Self.data,Self.fileptr-CharsPerInt)
	End
	
	Method WriteInt:Void(val:Int)
		Self.data+=Self.IntToString(val)
	End
	
	Method ReadString:String()
		Local result:String
		Local strLen:Int = self.StringToInt(Self.data,Self.fileptr)
		Self.fileptr+=CharsPerInt
		if strLen > 0
			result = Self.data[Self.fileptr..self.fileptr+strLen]
			Self.fileptr+=strLen
			Return result
		End
	End
	
	Method WriteString:Void(val:String)
		Self.data+=Self.IntToString(val.Length())
		if val.Length() > 0
			Self.data+=val
		End
	End
	
	Method ReadFloat:Float()
		Local result:float
		Local strLen:Int = self.StringToInt(Self.data,Self.fileptr)
		Self.fileptr+=CharsPerInt
		
        'swap these two calls along with the DataConversion implementations
        result = Self.StringToFloat(Self.data,Self.fileptr,strLen)
        'result = Self.StringToFloat(Self.data[Self.fileptr..Self.fileptr+strLen])
		
        Self.fileptr+=strLen
		Return result
	End
	
	Method WriteFloat:Void(val:Float)
		Local s:String = self.FloatToString(val)
		Self.data+=Self.IntToString(s.Length())
		Self.data+=s
	End
End
'#rem
Class DataConversion
    Global floatChars := ["","0","1","2","3","4","5","6","7","8","9","-",".","e"] '4-bit
    Const CharsPerInt:Int = 2
    
    Method FloatToString:String(val:Float)
        Local fString:String = String(val)
        Local output:String = ""
        Local currChar:Int = 0
        Local currShift:Int = 0 
        Local i:Int = fString.Length
        
        While i > 0
            i -= 1
            Local ch := fString[i]
            If ch >47 And ch<58
                currChar |= (ch-47) Shl currShift
            ElseIf ch = 45
                currChar |= 11 Shl currShift
            ElseIf ch = 46
                currChar |= 12 Shl currShift
            ElseIf ch = 69 Or ch = 101
                currChar |= 13 Shl currShift
            End
            
            currShift += 4
            If currShift = 16
                currShift = 0
                output = String.FromChar(currChar) + output
                currChar = 0
            End
        End
        
        If currShift <> 0
            output = String.FromChar(currChar) + output
        End
        
        Return output
    End		
        
    Method StringToFloat:Float(val:String,ind:Int,len:Int)
        Local fString:String = ""
        Local i:Int = ind
        While i < ind+len 
            Local charCode := val[i]
            For Local shift:Int = 12 To 0 Step -4
                Local index:Int = (charCode Shr shift) & $F 
                If index > 0
                    fString += floatChars[index]
                End
            End
            i += 1
        End
        
        Return Float(fString)
    End
    
    Method IntToString:String(val:Int)
        Local result:String
        result = String.FromChar((val Shr 16) & $FFFF )
        result+= String.FromChar(val & $FFFF)
        Return result
    End
        
    Method StringToInt:Int(val:String,ind:Int)
        Return (val[ind] Shl 16)|val[ind+1]
    End
End
'#end
#rem

Class DataConversion
    Const CharsPerInt:Int = 4
    
    Method IntToString:String(val:Int)
		Local result:String
		result = String.FromChar((val Shr 24) & $FF)
		result+= String.FromChar((val Shr 16) & $FF)
		result+= String.FromChar((val Shr 8) & $FF)
		result+= String.FromChar(val & $FF)
		Return result
	End

	Method FloatToString:String(val:Float)
		Return String(val)
	End		
	
	Method StringToInt:Int(val:String,ind:Int)
		Local result:Int
		result = (val[ind] Shl 24)
		result|= (val[ind+1] Shl 16)
		result|= (val[ind+2] Shl 8)
		result|= val[ind+3]
		Return result
	End

	Method StringToFloat:Float(val:String)
		Return Float(val)
	End		
End
#end

GfK	(Posted 2011) [#13]

Well... finished what I was doing but its Sunday and I've been on the sauce so I'm not quite sure what planet I'm on right now... I'll have to check this stuff out tomorrow when I arrive back on Earth.

therevills

(Posted 2011) [#14]

Sorry I misread what Dave put...

Can you post your test app muddy?

muddy_shoes

(Posted 2011) [#15]

Sure. It's probably been changed a little from the runs above, but it was pretty much this:

Strict

Import diddy

Function Main:Int()
	New TestApp
End Function

Class TestApp Extends App
	Field fileHandler:FileSystem

	Method OnCreate:Int()
		Self.fileHandler = New FileSystem
        Self.fileHandler.DeleteAll()
		Local stream:FileStream
        Const TESTS:Int = 20000
        Local floats:= New Float[TESTS]
        Local ints:= New Int[TESTS]
        
        For Local i:Int = 0 Until TESTS
            floats[i] = (Rnd(2.0)-1.0) * Rnd(100000000.0)
            ints[i] = Int((Rnd(1000)-500) * Rnd(100000))
        End
    
        
		stream = Self.fileHandler.WriteFile("test.bin")
        Local t := app.Millisecs()
        For Local i:Int = 0 Until TESTS
            stream.WriteFloat(floats[i])
        End
        t = app.Millisecs() - t
        Print "Write floats: " + t
        
        t = app.Millisecs()
        Self.fileHandler.SaveAll()
		t = app.Millisecs() - t
        Print "Save floats: " + t
        Self.fileHandler.ListDir()
		
		stream = Self.fileHandler.ReadFile("test.bin")
        
        t = app.Millisecs()
        For Local i:Int = 0 Until TESTS
            Local f := stream.ReadFloat()
'accuracy check
'            Local diff:Float = f - floats[i]
'            If diff < -0.00000001 Or diff > 0.00000001
'                Print "Fail: " + f + " - " + floats[i]
'            End
        End
        t = app.Millisecs() - t
        Print "Read floats: " + t
        
        Self.fileHandler.DeleteFile("test.bin")
        Self.fileHandler.SaveAll()
		stream = Self.fileHandler.WriteFile("test.bin")
		
        t = app.Millisecs()
        For Local i:Int = 0 Until TESTS
            stream.WriteInt(ints[i])
        End
        t = app.Millisecs() - t
        Print "Write ints: " + t
        
        t = app.Millisecs()
        Self.fileHandler.SaveAll()
		t = app.Millisecs() - t
        Print "Save ints: " + t
        
        Self.fileHandler.ListDir()
		
		stream = Self.fileHandler.ReadFile("test.bin")
        
        t = app.Millisecs()
        For Local i:Int = 0 Until TESTS
            Local f := stream.ReadInt()
            If f <> ints[i]
                Print "Fail: " + f
            End
        End
        t = app.Millisecs() - t
        Print "Read ints: " + t
        ExitApp
	End
End

GfK	(Posted 2011) [#16]

Just having a look at your int/string conversion code and I'm not sure it will work all the time. When I was doing my code I did some test or other and could not get FromChar() to return anything higher than 60599 - potentially you'd want anything up to 65535 to be returned. This is why I ended up doing single byte conversions.

Maybe I got it wrong though. Is your code tested to the maximum range?

GfK	(Posted 2011) [#17]

Added Boolean support. Add this code into the FileStream class:

	Method ReadBool:Bool()
		Local result:Bool
		result = Bool(Self.data[Self.fileptr])
		Self.fileptr+=1
		Return result
	End Method
	
	Method WriteBool:Void(val:Bool)
		Self.data+=String.FromChar(val)
	End Method

muddy_shoes

(Posted 2011) [#18]

It's rather difficult to define "maximum range" for a Monkey Integer or Float as the eventual type varies across the targets. All the tests I ran showed that charcodes are 16-bit values and the methods I posted work for 32-bit signed integer ranges.

I don't have any iOS or MacOS devices, so they are untested. It's certainly possible that some environments could have smaller chars, but it would be a little odd considering the need to support unicode.

maverick69

(Posted 2012) [#19]

I think this doesn't work on iOS Devices.

I have a file created with BlitzMax, when I try to load the Ressource with LoadString I always get an empty result.

I think it's because as it's a binary file the following line doesn't work:

NSString *str=[NSString stringWithContentsOfURL:url usedEncoding:&enc  error:nil];

Do you have any suggestions on how I get this working without converting the binary file into a text-based format?

When I read out the error message it links to the following error code:

NSFileReadUnknownStringEncodingError = 264,         // Read error (string encoding of file contents could not be determined)

maverick69

(Posted 2012) [#20]

Ok, here is a small fix to make everything work on ios devices. Instead of LoadString, use the following function:

String LoadBinaryString(String fpath) {
	NSString *rpath=pathForResource( fpath );
	if( !rpath ) return "";

    const char *filename = [rpath UTF8String];
    FILE *fp = fopen(filename, "r");    
    String ret = String::Load(fp);
    fclose(fp);
    return ret;
}