Asian Text in a file

Blitz3D Forums/Blitz3D Programming/Asian Text in a file

dna	(Posted 2017) [#1]

Hello

I was using some code to encrypt a text file. Using only XOR, the resulting text became all Asian text characters and the file took much more time to open.

I'm sure that many of you have come across this problem in the past.

What causes it?

xlsior

(Posted 2017) [#2]

What causes it?

You told it to?

Most likely your resulting XOR text ended up resembling unicode encoding since you would have ended up using a lot of normally unprintable ASCII codes (>128). It happened to be displayed as some Asian lettering, but could have been anything. (Asian characters are stored in unicode, where instead of each byte representing a single character like in ASCII, it takes three or four bytes represent a character. If the first three/four XORed bytes of output file happened to be a valid match for some Asian unicode character, that's what it would show in a text editor)

If you want the resulting file to look 'normal', then don't save any ascii codes over 127, and especially don't use chr$(0) and chr$(255).

Dan	(Posted 2017) [#3]

Please tell us, with which program have you opened the resulting text.

There is a difference between notepad, wordpad, Word and other text editing tools.
If the tool is able to open unicode, then the resulting text may be displayed as such.

Crimson Editor is able to switch the encoding type - between ascii, Unicode litle endian, Unicode big endian, UTF-8 with BOM or utf-8 without BOM.

If you are trying to open encrypted text, then the result is correct.

here is a simple xor test program, with this you can see how xor is working:

Screen(320,200)

;Graphics 640,480,32,2   ;uncomment this and next if you do not want to use screen() function !
;Graphics 320,200,32,3

SetBuffer BackBuffer()

Repeat
Cls
x=Rand(0,255)
y=Rand(0,255)

a=1
For z=10 To 80 Step 8
a=-a : If a = 1 Then Color $0,$50,0 Else Color $0,$0,$0

Rect z,31,z+8,42,1
Next ; z

Color $ff,$ff,$ff
Text 10,0,LSet("x",3)+"  -  "+LSet("y",3)
Text 10,11,LSet(x,3)+" Xor "+LSet(y,3)+" = "+(x Xor y)
Text 10,31,bn$(x)+" - x"
Text 10,41,bn$(y)+" - y"
Text 10,51,"---------"
Text 10,61,bn$((x Xor y))+" - xor result"
Text 10,160,"Press space for next result"
Text 10,170,"Hold esc to quit"
value=0
While Not value 
value=GetKey() 
Wend 
Delay 1
Flip
Until KeyDown(1)

WaitKey()

Function bn$(x)
Return Right$(Bin$(x),8)
End Function

Function Screen(x,y,full=0)
;fill <0 = Fulldesktop, borderless window
;full =0 = Fulldesktop 
;full =1 = original x,y
;full >1-5 = x*full,y*full size
;full >5 = 0
	DeskX=api_GetSystemMetrics(0)
	DeskY=api_GetSystemMetrics(1)
	If x>DeskX Then x=DeskX
    If x<64 Then x=64
	If y>DeskY Then y=DeskY
    If y<64 Then y=64
    bits=api_GetDeviceCaps(api_GetDC( api_GetDesktopWindow()),12)
	Graphics x,y,bits,6
	Graphics x,y,bits,7
	If full<=0 Or full>5
		If full<0 Then api_SetWindowLong(api_GetActiveWindow(), -16, $10000000)
		api_MoveWindow(api_GetActiveWindow(),0,0,DeskX,DeskY,True)
    EndIf
	If full>1 And full<=5 Then api_MoveWindow(api_GetActiveWindow(),0,0,x*full,y*full,True)
End Function

Actually, the next example shows how the encoding with xor really works:

(same as above example, but now the password (y) does not change)

Screen(320,200)

;Graphics 640,480,32,2   ;uncomment this and next if you do not want to use screen() function !
;Graphics 320,200,32,3

SetBuffer BackBuffer()

b=1

Repeat
	Cls
	
	If b=1
		x=%11001010
		y=%00100111
	Else
		x=x Xor y
	EndIf
	b=-b
	
	a=1
	For z=10 To 80 Step 8
		a=-a : If a = 1 Then Color $0,$50,0 Else Color $0,$0,$0
		
		Rect z,31,z+8,42,1
	Next ; z
	
	Color $ff,$ff,$ff
	Text 10,0,LSet("x",3)+"  -  "+LSet("y",3)
	Text 10,11,LSet(x,3)+" Xor "+LSet(y,3)+" = "+(x Xor y)
	Text 10,31,bn$(x)+" - x"
	Text 10,41,bn$(y)+" - y"
	Text 10,51,"---------"
	Text 10,61,bn$((x Xor y))+" - xor result"
	Text 10,160,"Press space for next result"
	Text 10,170,"Hold esc to quit"
	
	value=0
	While Not value 
		value=GetKey() 
	Wend 
	
	Delay 1
	
	Flip
Until KeyDown(1)

WaitKey()

Function bn$(x)
	Return Right$(Bin$(x),8)
End Function

Function Screen(x,y,full=0)
;fill <0 = Fulldesktop, borderless window
;full =0 = Fulldesktop 
;full =1 = original x,y
;full >1-5 = x*full,y*full size
;full >5 = 0
	DeskX=api_GetSystemMetrics(0)
	DeskY=api_GetSystemMetrics(1)
	If x>DeskX Then x=DeskX
    If x<64 Then x=64
	If y>DeskY Then y=DeskY
    If y<64 Then y=64
    bits=api_GetDeviceCaps(api_GetDC( api_GetDesktopWindow()),12)
	Graphics x,y,bits,6
	Graphics x,y,bits,7
	If full<=0 Or full>5
		If full<0 Then api_SetWindowLong(api_GetActiveWindow(), -16, $10000000)
		api_MoveWindow(api_GetActiveWindow(),0,0,DeskX,DeskY,True)
    EndIf
	If full>1 And full<=5 Then api_MoveWindow(api_GetActiveWindow(),0,0,x*full,y*full,True)
End Function

see 3 functions Timer_Sec Timer_mSec and Screen for the missing userlib definitions (for the screen function)

dna	(Posted 2017) [#4]

dAN

What id the api_GetSystemMetrics() function?

I get a compiler error

Bobysait

(Posted 2017) [#5]

It's a function coming from windows dll (user32)

Just add it to your userlib links :

* get the decls file from the code archive and copy it to your userlib folder

or

* create a file in your blitz3d/userlibs folder (name it 'user32.decls')
* open it in your text editor
* add this to the file :

.lib "user32.dll"
api_GetSystemMetrics%(param%) : "GetSystemMetrics"

Then restart blitz3d. The new function should work.

Dan	(Posted 2017) [#6]

or simply delete the function screen() and the first line ( screen(320,200) )
and uncomment the two graphic calls.

(btw: i wrote it in the code)

and at the end:

see 3 functions Timer_Sec Timer_mSec and Screen for the missing userlib definitions (for the screen function)

There i'v wrote you an answer on how to save the userlib definitions (arround 1 year ago) :p .

        DeskX=api_GetSystemMetrics(0)
	DeskY=api_GetSystemMetrics(1)

it gets the desktop screen resolution, which is not available in the pure blitz3d.

dna	(Posted 2017) [#7]

@Dan

That is Clever that resolution. I never BB could do that

I also never knew you could display Hex with a Dollar sign in front of it. I always used 255