Filesystem for Monkey
Monkey Forums/Monkey Programming/Filesystem for Monkey
| ||
| Just having a play around with a few ideas I mentioned here yesterday but I'm running into all manner of problems. The targets I have in mind are iOS and Android so the OS module is not an option. My idea was to have a filesystem class which emulates all of Blitzmax's Read/Write operations to file streams. In concept its simple enough but I seem to be running into brick wall after brick wall with it. First off I've discovered that its going to be impossible to store an Integer in any sensible form. All my tests indicate that by fudging it (writing the ASCII code of characters), the highest Int I can store is 60599, which is not going to be good enough. The only other way I can think of is to store the Integer as a string, but then an Int which would normally take four bytes (1234567890 for example), would take up ten bytes, making saved data much larger than it really ought to be. Even then, I'd have to waste another byte to tell the file parser how many bytes to read in order to retrieve the string, then convert it to an Int. So, am I right in thinking that this needs to be done at a lower level? At the moment this is a real show-stopper for me as all my other code is just about converted to Monkey, except for this. The way I'm doing it just doesn't seem practical and I don't really know where to go from here. Thoughts? |
| ||
| I wrote a base converter some months ago. You could pack integers from base 10 to 36 and unpack it when you need it. http://www.monkeycoder.co.nz/Community/posts.php?topic=945 |
| ||
| but then an Int which would normally take four bytes (1234567890 for example), would take up ten bytes, making saved data much larger than it really ought to be. Even then, I'd have to waste another byte to tell the file parser how many bytes to read in order to retrieve the string, then convert it to an Int. Are you making things hard for yourself to save a few KB? |
| ||
| Are you making things hard for yourself to save a few KB? Yes, as SaveState/LoadState are size-limited to varying degrees on different platforms. |
| ||
Think I've sussed how to convert an Int to a 4-byte string:Method intToString:String(val:Int) Local result:String result = String.FromChar((val Shr 24) & $FF) result+= String.FromChar((val Shr 16) & $FF) result+= String.FromChar((val Shr 8) & $FF) result+= String.FromChar(val & $FF) Return result End Method Don't bother using Print to see the result though - unprintable characters are just that - they don't have any visual representation, so although result is four bytes long, it might only show as one or two characters using Print. ...and back again: Method stringToInt:Int(val:String) Local result:Int result = (val[0] Shl 24) result|= (val[1] Shl 16) result|= (val[2] Shl 8) result|= val[3] Return result End Method ...this might be doable after all... |
| ||
| I HAVE A RUDDY FILESYSTEM WORKING!!!!!!!!!!!!!!!!! :D |
| ||
| I've got strings and Ints reading/writing to perfectly now! I'm having a little bother doing the same with Floats, though. Anybody know what's going on here? I'm assuming Floats are four bytes too, right? Though I also tried 8 bytes and that didn't work either. Basically if you feed 3.14 to floatToString(), and give the result of that back to stringToFloat(), I should be getting 3.14 back. [edit] Never mind, had a lil typo that was mucking my filepointer up. But now I get 3 returned instead of 3.14....?? [edit again] The plot thickens... tried the same code in Blitzmax and I get an error "Bitwise operators can only be used with Integers"... Method floatToString:String(val:Float) Local result:String result = String.FromChar((val Shr 24) & $FF) result+= String.FromChar((val Shr 16) & $FF) result+= String.FromChar((val Shr 8) & $FF) result+= String.FromChar(val & $FF) Return result End Method Method stringToFloat:Float(val:String) Local result:Float result = (val[0] Shl 24) result|= (val[1] Shl 16) result|= (val[2] Shl 8) result|= val[3] Return result End Method |
| ||
| If memory serves me a float (or any floating point number) is stored in memory in mantissa format so I think it would be logical convert a float to a string, get the decimal place store it then remove it from the string, store the number without is decimal point. The only problem would be dealing with numbers less than 1.0 like 0.01 as you would have to store the zeros some how. It would be a good idea to brush up on mantissa's and the IEEE floating point standard. (I definitely need to) |
| ||
| I think that removing the decimal point would cause an overflow in some cases. plus id need to know where id removed it from when i put it back later. |
| ||
| Hey, it's my first ever Monkey post. And I haven't learned the language, so take this with a grain of salt. It sounds impossible to me. There are no pointers, hence no way to get at the memory ( raw bits ) representing a float. You have access only to the numerical value that memory represents when interpreted as a float. An expression like ( val Shr 24 ) makes sense if val is an integer. But it is meaningless for a float. That's why BlitzMax doesn't allow it. The fact that it works in Monkey indicates the a float val is being silently converted to integer before any bitwise operations are performed. So when you thought you were operating on 3.14 it was really being turned into a 3. Is there any way to mix C/C++ code with Monkey? If so you could have two very simple functions to convert a float to/from the integer represented by the same memory. Then use your Monkey code for integers. If this works it still may not be feasible for multiple platforms. Can you assume a float is four bytes? And you have to consider byte order. Is everything like Intel? |
| ||
| I think you may be barking up the wrong tree a bit here. You're trying to create something that requires low-level access and understanding of the primitive types on each target from Monkey itself. As Monkey is an abstraction from those targets by nature, you're likely to keep tripping over compatibility issues. For a start, Monkey Floats end up as single precision in C,CS and Java and double in JS and Flash. Edit: If I were doing this myself I'd probably start by looking at the feasibility of implementing string compression in Monkey or creating a simple wrapper around calls to appropriate libraries on each target. Serialise to JSON -> compress the string -> save. |
| ||
| I probably am barking up the wrong tree but unless somebody else can provide a better way (beyond just theory) its all i have to go on. i don't store many floats anyway so will probably just do a straight string conversion. |
| ||
| Err... okay, but I'm pretty sure that neither JSON (or string serialisation in general) or compression are just theoretical. I'm also pretty sure that what I wrote above will work, would avoid most cross-target compatibility issues and would probably be comparable in space efficiency and speed at a user level. If you really insist on going down the route you're on then you might want to know that the Monkey concept of a character code appears to be consistently mapped to an unsigned 16-bit integer. This means that your integer conversions above are using twice as many characters as necessary. |
| ||
| Its not a case of insisting. Your way may well work but in all honesty I have no idea where to start with that stuff. So unless somebody who knows can put fingers to keyboard and get something working, I don't have much alternative. |
| ||
| What about saving your strings in Base64 format or Hex? |
| ||
| Dunno. ten years of blitz has left me not having to worry about this kind of thing. I'm almost ready to pay somebody to fix this up. |
| ||
| Well Diddy has base64 encoding and decoding ;) |
| ||
| How's base 64 help tho? I'm not saying it doesn't i just don't know. |
| ||
| Think I've sussed how to convert an Int to a 4-byte string: Depending on how much you're reading/writing, that's a massive amount of string manipulation. On Android (and possibly other garbage-collected languages), that's going to be a killer right there. |
| ||
| I stand by my point about looking at more general compression algorithms but, as I'm not about to implement DEFLATE in a spare ten minutes, here's some hacky code to pack floats into strings based on your apparent desired approach. It's relatively untested and I've made no attempt to optimise it but it should move you along the road you're on. Basically, it lets Monkey do the string conversion and then packs that string based on the assumption that there's a reduced character set. I don't know if e-notation ever gets spat out of such conversions, so maybe that can be removed, but it wouldn't save any space anyway. The improvement in space usage will vary depending on the value being stored, but it should be somewhere between 50% and 25% of the original. Performance is obviously compromised by any extra processing and currently the functions are up to 4-5 times slower than just using Monkey's conversions alone. Whether this is an issue or not for your purposes, I don't know. At the user level, my Android phone takes 750ms or so to pack and unpack 1000 values, but just using Monkey's conversions takes nearly 400ms so loading and saving large amounts of string-encoded data is going to be an issue on Android whichever way you do it. The other targets are much faster, at least on my middling laptop, with the next poorest performer being Flash taking 150ms or so to pack and unpack 10,000 values. |
| ||
| What about writting file/database/something wrapper for each platform? Im sure if you are only targetting 1 or 2 platforms to start with it wouldn't be too hard to wrap with a little help from web tutorials? Thats just theory, but for example: http://www.raywenderlich.com/1914/how-to-save-your-app-data-with-nscoding-and-nsfilemanager |
| ||
| Depending on how much you're reading/writing, that's a massive amount of string manipulation. On Android (and possibly other garbage-collected languages), that's going to be a killer right there. Just benchmarked my method vs Diddy's EncodeBase64() on Android (LG Optimus One). My method: 390ms Diddy: 968ms Strict Import diddy Import mojo Function Main:Int() New myClass Return 0 End Function Class myClass extends App Field time:Int Method OnCreate:Int() Self.time = Millisecs() For Local n:Int = 1 to 1000 Self.intToString(23462469) Next Self.time = Millisecs()-Self.time SetUpdateRate(30) Return 0 End Method Method intToString:String(val:Int) 'Return EncodeBase64(val) Local result:String result = String.FromChar((val Shr 24) & $FF) result+= String.FromChar((val Shr 16) & $FF) result+= String.FromChar((val Shr 8) & $FF) result+= String.FromChar(val & $FF) Return result End Method Method OnUpdate:Int() End Method Method OnRender:Int() Cls DrawText(Self.time,50,50) Return 0 End Method End Class I also noted that the resulting output from EncodeBase64() took up more space than if I'd just stored the Float as a String in the first place (by three bytes per float). What about writting file/database/something wrapper for each platform? Im sure if you are only targetting 1 or 2 platforms to start with it wouldn't be too hard to wrap with a little help from web tutorials? I have absolutely no grasp of C whatsoever. But yeah, I'm really only interested in iOS and Android at the moment. Moreso iOS. |
| ||
| Well.... I think I'm done. It all works now; multiple file support with read/write of ints, floats and strings, plus saveAll() which concatenates and saves data to the device and a few helper functions such as fileExists, listDir and whatnot. OK it would be much better to dig into lower level file handling per platform but given the tools and the knowledge I have, I think this is as good as its going to get. The important point is that it works pretty well considering how hacky some parts have been. I still need to test it out on iOS but its working fine in the emulator, and on my Android phone too, HTML5 and Flash targets. |
| ||
| Nice job! |
| ||
| @Gfk How's base 64 help tho? I'm not saying it doesn't i just don't know. Base64 is a method of storing binary data as printable characters. It is/was mostly used for Usenet and emails, but it's also a handy way to store binary data in XML files. |
   |