DataBuffer and ARM unaligned access

Monkey Forums/Monkey Programming/DataBuffer and ARM unaligned access

AdamRedwoods

(Posted 2012) [#1]

from here:
http://monkeycoder.co.nz/Community/post.php?topic=2212&post=26299

This may be an ARM unaligned memory access.
https://brewx.qualcomm.com/bws/content/gi/common/appseng/en/knowledgebase/docs/kb95.html

Any ideas on how we should get around this? Change to using memcpy() instead of casts?
Or should I handle it on my end with ints and a lot of bitshifting?

Thanks.

JIM	(Posted 2012) [#2]

Is the buffer made up of floats alone? Or mixed?

Ints and bitshifting is not a very good idea.
I know there's #pragma pack for MSVC, but don't know of the xcode equivallent.

Does the error happen on the first read? If so, it's kind of a monkey bug.
If you have pushed one byte and then a float and you're trying to read the float then it makes sense.

Make sure you either pack them better (never let floats of ints have offset != 4) or use ints and floats and nothing else.
It is highly recommended to make sure your data is properly aligned, because even if it works, reading from an unaligned address in memory can be very slow (depending on number of reads).

AdamRedwoods

(Posted 2012) [#3]

If you have pushed one byte and then a float and you're trying to read the float then it makes sense.

yes.
the problem is with reading in Base64 data from B3D files, so I have no control over the incoming stream and chances are it's packed bytewise.

But it looks like I solved it by reading in bytes and using another 4-byte buffer for int/float conversion:

Local i2f:DataBuffer = Databuffer.Create(4)
i2f.PokeByte(0,data.PeekByte(pos-4))
i2f.PokeByte(1,data.PeekByte(pos-3))
i2f.PokeByte(2,data.PeekByte(pos-2))
i2f.PokeByte(3,data.PeekByte(pos-1))
Local result:Float = i2f.PeekFloat(0)

This seems like an adequate solution, since it's on initialization only.

I think Mark said HTML5 has a similar problem.

JIM	(Posted 2012) [#4]

I made my own implementation for base64 that seems to work on HTML5 as well, so I suppose it might work in your case as well.

Let me find the code...

Strict

Global MIME$="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
Global Dime:Int[]

Function B64:String(x:Int)
	if ((x < 0) Or (x > 63))
		Print "[B64]: Out of 64 range!"
	EndIf
	
	Return String.FromChar(MIME[x])
End

Function RB64:Int(c:Int)
	If (Not Dime)
		Dime = New Int[256]
		For Local i:Int = 0 To 63
			Dime[MIME[i]] = i
		Next
	End
	
	Return Dime[c]
End

Function StringToIntArray:Int[](s:String)
	Local arrSize:Int = Ceil(s.Length() / 4.0)
	Local arr:Int[] = New Int[arrSize]
	
	For Local i:Int = 0 Until s.Length()
		Select (i Mod 4)
			Case 0
				arr[Floor(i / 4.0)] |= (s[i] & 127) Shl 0
			Case 1
				arr[Floor(i / 4.0)] |= (s[i] & 127) Shl 8
			Case 2
				arr[Floor(i / 4.0)] |= (s[i] & 127) Shl 16
			Case 3
				arr[Floor(i / 4.0)] |= (s[i] & 127) Shl 24
		End
	Next
	
	Return arr
End

Function IntArrayToString:String(d:Int[])
	Local s:String = ""
	For Local i:Int = 0 Until d.Length()
		s += String.FromChar((d[i] Shr 0) & 127)
		s += String.FromChar((d[i] Shr 8) & 127)
		s += String.FromChar((d[i] Shr 16) & 127)
		s += String.FromChar((d[i] Shr 24) & 127)
	Next
	Return s
End

Function Base64Encode:String(Data:Int[])
	Local TotalBytes:Int = Data.Length() * 4
	Local ProcessedBytes:Int = 0
	
	Local b:Int[3]
	Local b6:Int[4]
	
	Local idx:Int = 0		'index
	Local sidx:Int = 0		'sub-index
	
	Local encodeStr:String
	
	While (ProcessedBytes < TotalBytes)
		idx = Floor(ProcessedBytes / 4.0)
		sidx = ProcessedBytes Mod 4
		
		If ((TotalBytes - ProcessedBytes) = 2)
			'handle finishes
			b[0] = (Data[idx + Floor((sidx + 0) / 4.0)] Shr (((sidx + 0) Mod 4) * 8)) & 127
			b[1] = (Data[idx + Floor((sidx + 1) / 4.0)] Shr (((sidx + 1) Mod 4) * 8)) & 127
			
			b6[0] = b[0] Shr 2
			b6[1] = ((b[0] & 3) Shl 4) | (b[1] Shr 4)
			b6[2] = ((b[1] & 15) Shl 2)
			
			encodeStr += B64(b6[0])
			encodeStr += B64(b6[1])
			encodeStr += B64(b6[2])
			encodeStr += "="
			
			ProcessedBytes += 2
		ElseIf ((TotalBytes - ProcessedBytes) = 1)
			b[0] = (Data[idx + Floor((sidx + 0) / 4.0)] Shr (((sidx + 0) Mod 4) * 8)) & 127
			
			b6[0] = b[0] Shr 2
			b6[1] = ((b[0] & 3) Shl 4)
			
			encodeStr += B64(b6[0])
			encodeStr += B64(b6[1])
			encodeStr += "=="
			
			ProcessedBytes += 1
		Else
			b[0] = (Data[idx + Floor((sidx + 0) / 4.0)] Shr (((sidx + 0) Mod 4) * 8)) & 127
			b[1] = (Data[idx + Floor((sidx + 1) / 4.0)] Shr (((sidx + 1) Mod 4) * 8)) & 127
			b[2] = (Data[idx + Floor((sidx + 2) / 4.0)] Shr (((sidx + 2) Mod 4) * 8)) & 127
			
			b6[0] = b[0] Shr 2
			b6[1] = ((b[0] & 3) Shl 4) | (b[1] Shr 4)
			b6[2] = ((b[1] & 15) Shl 2) | (b[2] Shr 6)
			b6[3] =  b[2] & 63
			
			encodeStr += B64(b6[0])
			encodeStr += B64(b6[1])
			encodeStr += B64(b6[2])
			encodeStr += B64(b6[3])
			
			'advance
			ProcessedBytes += 3
		EndIf
	Wend
	
	Return encodeStr
End

Function Base64Decode:Int[](s:String)
	Local b6:Int[4]
	Local b:Int[3]
	Local data:Int[]
	
	Local TotalBytes:Int = s.Length() * 3 / 4
	Local ProcessedBytes:Int = 0
	Local NrBlocks:Int = s.Length() / 4
	
	data = New Int[TotalBytes / 4]
	
	Local crtPos:Int = 0
	
	For Local i:Int = 0 Until NrBlocks
		b6[0] = RB64(s[i*4])
		b6[1] = RB64(s[i*4+1])
		b6[2] = RB64(s[i*4+2])
		b6[3] = RB64(s[i*4+3])
			
		b[0] = ((b6[0] Shl 2) | (b6[1] Shr 4)) & 255
		b[1] = ((b6[1] Shl 4) | (b6[2] Shr 2)) & 255
		b[2] = ((b6[2] Shl 6) | b6[3]) & 255
		
		Select (crtPos Mod 4)
			Case 0
				data[Floor(crtPos / 4.0)] |= b[0] Shl 0
				data[Floor(crtPos / 4.0)] |= b[1] Shl 8
				data[Floor(crtPos / 4.0)] |= b[2] Shl 16
			Case 1
				data[Floor(crtPos / 4.0)] |= b[0] Shl 8
				data[Floor(crtPos / 4.0)] |= b[1] Shl 16
				data[Floor(crtPos / 4.0)] |= b[2] Shl 24
			Case 2
				data[Floor(crtPos / 4.0)] |= b[0] Shl 16
				data[Floor(crtPos / 4.0)] |= b[1] Shl 24
				If (b[2]) data[Floor(crtPos / 4.0)+1] |= b[2] Shl 0
			Case 3
				data[Floor(crtPos / 4.0)] |= b[0] Shl 24
				If (b[1]) data[Floor(crtPos / 4.0)+1] |= b[1] Shl 0
				If (b[2]) data[Floor(crtPos / 4.0)+1] |= b[2] Shl 8
		End
		
		crtPos += 3
	Next
	
	Return data
End

You will probably need 2 more functions in there: Int2Float and Float2Int.

You can use a temp databuffer (size 4) and poke/peek from that.

The code above uses Int arrays, but those arrays actually contain bytes. Probably another function for that: CompressArray:Int[](intarray:Int[]) to make a size / 4 array with each int containing 4 bytes.

NoOdle

(Posted 2012) [#5]

Why is Base64 useful? What advantages does it have with Monkey? I'm not really sure of its application, but it looks fun to program!

AdamRedwoods

(Posted 2012) [#6]

Why is Base64 useful?

useful for binary files to be converted into common character codes. javascript doesn't allow binary files to be read, so this format allows that conversion to happen. it's an easy conversion and online tools exist to convert binary to base64.

because monkey is multi-platform, base64 allows one file to be shared with all targets.

ascii85 is another type of encoding.

impixi

(Posted 2012) [#7]

Another option is to encode your binary files as PNG images (which also potentially compresses your data) and then unencode via Monkey's built-in DataBuffer commands (opengl module). Unfortunately Monkey's HTML implementation has limitations... To overcome them, I've coded a canvas-based version, however I'm stuck on peeking/poking of floats (see here). Personally, to get around floats, I use my own fixed point decimal number format, however for completeness sake a floating point solution would be nice.

Back on topic, that base64 implementation looks good, though as with all such binary->text encodings, the data gets 'inflated'...

visionastral

(Posted 2012) [#8]

Sorry if it's a noob question but, what's the problem with encoding a binary in 8 bits characters? (and so making it a pseudo text file)
I make this all the time for loading/saving ini files and the like and never had a problem...
I don't really understand the need for base64.

Right now I'm encoding booleans, bytes, words/shorts and ints in byte sequences and it works perfectly.

AdamRedwoods

(Posted 2012) [#9]

It's a valid question. The answer is that you can fit Base64 into an XML document or into Javscript, without it interfering with the code.

Another reason I chose it is because it's popular enough on the internet to convert files online.

visionastral

(Posted 2012) [#10]

Ha! ok, it's for compatibility and portability then.
Good to know :)

DruggedBunny

(Posted 2012) [#11]

It's the same reason email uses base64 encoding for attachments -- you'll notice if you send a large file, it's even larger at the other end (or in your Sent Items folder).