.ZNG image file format

BlitzMax Forums/BlitzMax Programming/.ZNG image file format

Chroma

(Posted 2008) [#1]

See latest post in this thread for the final .zng code. Basically it's a zlib zipped and encrypted image format so you can protect your art resources.

Chroma

(Posted 2008) [#2]

...

Filax

(Posted 2008) [#3]

Zng = Zip + Pgn ? :)

PantsOn

(Posted 2008) [#4]

Hi

I might be doing something wrong but in each case the PNG is smaller than the ZNG.

example image if you want to test
JPG = 16kb
ZNG = 129kb
PNG = 93kb

Chroma

(Posted 2008) [#5]

Nm have to work on it a bit more.

^^^^

:p

Chroma

(Posted 2008) [#6]

I opted to just load the image into a bank, compress the bank, then write the contents of the bank to a file. Then reverse it for loading. You might thinks it's dumb, whatever. It does make em smaller 90% of the time and gives a light layer of security.

And yep Filax. ZNG stands for zipped network graphic. Or zip + png. The future goal for this is after the image is loaded into the bank, scramble the data with a key, then compress it and write it to a file. Then load it, decompress it, descramble it etc etc.

ZNG_Converter.bmx:

Local inFile$ = AppArgs[1]
Local outFile$ = StripExt(inFile)
Local ibank:TBank = LoadBank(inFile)
Local cbank:TBank = CompressBank(ibank)
Local out:TStream = WriteFile(outFile + ".zng")
out.writebytes(cbank.buf(), cbank.size())
CloseStream(out)

End

Load ZNG files:

Function LoadImageZNG:TImage(zngFile$, flags%=-1)
	Local cbank:TBank = LoadBank(zngFile)
	Local ubank:TBank = UncompressBank(cbank)
	Return LoadImage(ubank, flags)
End Function

Function LoadPixmapZNG:TPixmap(zngFile$)
	Local cbank:TBank = LoadBank(zngFile)
	Local ubank:TBank = UncompressBank(cbank)
	Return LoadPixmap(ubank)
End Function

Filax

(Posted 2008) [#7]

Good idea Chroma! ;) But maybe the Koriolis ZipStream module
can be usefull for you ?

http://www.blitzbasic.com/Community/posts.php?topic=71734

It's a great piece of code! You can incbin a zip file
(password accepted) and read all media files directly,
without convertion.Take a look if you want :)

See ya

PantsOn

(Posted 2008) [#8]

great idea...
good to work on, but it does seem like re-inventing the wheel.

(not having a dig, as I have re-invented the wheel many times to see how things work)

good luck with the project

Chroma

(Posted 2008) [#9]

I just added bank encryption. Now the .zng file is encrypted before it's compressed and written to the file. I think it's pretty secure. At least enough for my needs. Feel free to use it if you want to protect your game art.

Please test and lemme know if you have any problems.

Here's the code to encrypt a bank:

Function XorBank:TBank(bank:TBank, key$)
	'read bank data to buffer string
	Local buffer$
	For i = 0 To bank.size()-1
		Local b:Byte = PeekByte(bank,i)
		buffer :+ Chr$(b)
	Next
	
	buffer = xor(buffer,key$)

	Local rbank:TBank = CreateBank(buffer.length)
	For i = 0 To rbank.size()-1
		PokeByte(rbank,i,Asc(Mid$(buffer,i+1,1)))
	Next
	Return rbank
End Function

Function xor:String(str$, key$)
	Local ml%, pl%, i%, result$
	ml% = Len(str)
	pl% = Len(key)
	For i = 0 Until ml
		Local c:Byte = str$[i]
		Local k = key$[i Mod pl]
		c = c ~ k
		result$ :+ Chr(c)
	Next 
	Return result$
EndFunction

And here's the ZNG_Converter again with the encryption option:

'Zipped Network Graphic [.zng] - Image File Format Converter

'Encryption Supported

'Build into an .exe and drag and drop your image files to convert them to .zng

'by Noel Cole (Chroma)

Local ZNG_Key$ ="abc123"

Local inFile$ = AppArgs[1]
Local outFile$ = StripExt(inFile)

Local ibank:TBank = LoadBank(inFile)
If ZNG_Key Then ibank = XorBank(ibank,ZNG_Key)
Local cbank:TBank = CompressBank(ibank)

Local out:TStream = WriteFile(outFile + ".zng")
out.writebytes(cbank.buf(), cbank.size())
CloseStream(out)

End

Function XorBank:TBank(bank:TBank, key$)
	'read bank data to buffer string
	Local buffer$
	For i = 0 To bank.size()-1
		Local b:Byte = PeekByte(bank,i)
		buffer :+ Chr$(b)
	Next
	
	buffer = xor(buffer,key$)

	Local rbank:TBank = CreateBank(buffer.length)
	For i = 0 To rbank.size()-1
		PokeByte(rbank,i,Asc(Mid$(buffer,i+1,1)))
	Next
	Return rbank
End Function

Function xor:String(str$, key$)
	Local ml%, pl%, i%, result$
	ml% = Len(str)
	pl% = Len(key)
	For i = 0 Until ml
		Local c:Byte = str$[i]
		Local k = key$[i Mod pl]
		c = c ~ k
		result$ :+ Chr(c)
	Next 
	Return result$
EndFunction

Function CompressBank:TBank( bank:TBank )
	Local size=bank.Size()
	Local out_size=size+size/10+32
	Local out:TBank=TBank.Create( out_size )
	compress out.Buf()+4,out_size,bank.Buf(),size
	out.PokeByte 0,size
	out.PokeByte 1,size Shr 8
	out.PokeByte 2,size Shr 16
	out.PokeByte 3,size Shr 24
	out.Resize out_size+4
	Return out
End Function

Function UncompressBank:TBank( bank:TBank )
	Local out_size
	out_size:|bank.PeekByte(0)
	out_size:|bank.PeekByte(1) Shl 8
	out_size:|bank.PeekByte(2) Shl 16
	out_size:|bank.PeekByte(3) Shl 24
	Local out:TBank=TBank.Create( out_size )
	uncompress out.Buf(),out_size,bank.Buf()+4,bank.Size()-4
	Return out
End Function

And the Loader:

'.ZNG ImageLoader

'Author: Noel Cole (Chroma)
'Email:  noel@...

Global ZNG_Key$

Function LoadImageZNG:TImage(zngFile$, flags%=-1)
	Local cbank:TBank = LoadBank(zngFile)
	Local ubank:TBank = UncompressBank(cbank)
	If ZNG_Key Then ubank = XorBank(ubank,ZNG_Key)
	Return LoadImage(ubank, flags)
End Function

Function LoadPixmapZNG:TPixmap(zngFile$)
	Local cbank:TBank = LoadBank(zngFile)
	Local ubank:TBank = UncompressBank(cbank)
	Return LoadPixmap(ubank)
End Function

Function XorBank:TBank(bank:TBank, key$)
	Local buffer$
	For i = 0 To bank.size()-1
		Local b:Byte = PeekByte(bank,i)
		buffer :+ Chr$(b)
	Next
	buffer = xor(buffer,key$)

	Local rbank:TBank = CreateBank(buffer.length)
	For i = 0 To rbank.size()-1
		PokeByte(rbank,i,Asc(Mid$(buffer,i+1,1)))
	Next
	Return rbank
End Function

Function xor:String(str$, key$)
	Local ml%, pl%, i%, result$
	ml% = Len(str)
	pl% = Len(key)
	For i = 0 Until ml
		Local c:Byte = str$[i]
		Local k = key$[i Mod pl]
		c = c ~ k
		result$ :+ Chr(c)
	Next 
	Return result$
EndFunction

Function CompressBank:TBank( bank:TBank )
	Local size=bank.Size()
	Local out_size=size+size/10+32
	Local out:TBank=TBank.Create( out_size )
	compress out.Buf()+4,out_size,bank.Buf(),size
	out.PokeByte 0,size
	out.PokeByte 1,size Shr 8
	out.PokeByte 2,size Shr 16
	out.PokeByte 3,size Shr 24
	out.Resize out_size+4
	Return out
End Function

Function UncompressBank:TBank( bank:TBank )
	Local out_size
	out_size:|bank.PeekByte(0)
	out_size:|bank.PeekByte(1) Shl 8
	out_size:|bank.PeekByte(2) Shl 16
	out_size:|bank.PeekByte(3) Shl 24
	Local out:TBank=TBank.Create( out_size )
	uncompress out.Buf(),out_size,bank.Buf()+4,bank.Size()-4
	Return out
End Function

Function SetZNGKey(key$)
	ZNG_Key = key
End Function

Instructions:

1. Set the ZNG_Key$ in the ZNG_Converter.bmx source to your desired password
1. Build the ZNG_Converter.bmx into an .exe
2. Drag an image file onto it to make a .zng
3. Include the LoadZNG.bmx in your source
4. Use the command SetZNGKey("password") to set the password to the same one you used in the converter
5. Use the command LoadImageZNG("image.zng") to load your protected image files.

Include "LoadZNG.bmx"

SetZNGKey("abc123")

Graphics 800,600

Local img:TImage = LoadImageZNG("myimage.zng")

SetBlend ALPHABLEND

SetClsColor 100,100,100

While Not KeyHit(KEY_ESCAPE)
Cls

DrawImage img,100,100


Flip
Wend
End

Chroma

(Posted 2008) [#10]

I don't think I'm interacting with the banks very efficiently by reading them into a long string. When I encrypt an image that's around 2048x2048 it just hangs for a couple minutes and never finishes writing the file.

Any ideas?

xlsior

(Posted 2008) [#11]

Another thing to consider: encrypting *before* compressing will likely end up with a much larger file than encrypting *after* compressing:

If you have an image with a lot of 'similar' bits, you'll have a lot of repetition in the bitmap, and it can be compressed down significantly. If you encrypt it first, then your similar bits look semi-randomized, and you're going to get a much worse compression.

(Put simply:

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -> compressed = '80xA' -> encrypted = a9hB

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -> encrypted ->
h#^kairJ289aGGhjsadf1akdsjadsfdaf$(*1fa,mvakdfkjq3487139847fakakfj@#$JKMAdAsfj34hakfhkjhaf -> compressed = something a whole lot longer than the 4 characters in the previous example

Filax

(Posted 2008) [#12]

i have try this code and you are right, the program loop forever. The function xor seem to be the cause of the problem.

I see that you XOR a string with a key, but a png file is
a binary file? Maybe you must xor with bytes and not with
string.

Example :
'read bank data to buffer string
Local buffer$
For i = 0 To bank.size()-1
Local b:Byte = PeekByte(bank,i)
buffer :+ Chr$(b)
Next

Must be IMO:

'read bank data
For i = 0 To bank.size()-1
Local b:Byte = PeekByte(bank,i)

' Xor the byte with one char of the key
' then write the char generated by the xor
Next

xlsior : You are right, Chroma you need to compress the file before encrypting.


'.ZNG ImageLoader

'Author: Noel Cole (Chroma)
'Email:  noel@...

Global ZNG_Key$
ZNG_Key$ ="abc123"

SetZNGKey("abc123")

Local inFile$ = "Fond Deviant Art_00009.png"
Local outFile$ = StripExt(inFile)

Local ibank:TBank = LoadBank(inFile)

' De activate this function made loop forever?
'If ZNG_Key Then ibank = XorBank(ibank,ZNG_Key)
Local cbank:TBank = CompressBank(ibank)

Local out:TStream = WriteFile(outFile + ".zng")
out.writebytes(cbank.buf(), cbank.size())
CloseStream(out)
End



Graphics 800,600

Local img:TImage = LoadImageZNG("Fond Deviant Art_00009.zng")

SetBlend ALPHABLEND

SetClsColor 100,100,100

While Not KeyHit(KEY_ESCAPE)
Cls

DrawImage img,100,100


Flip
Wend
End




'And here's the ZNG_Converter again with the encryption option:

'Zipped Network Graphic [.zng] - Image File Format Converter

'Encryption Supported

'Build into an .exe and drag and drop your image files to convert them to .zng

'by Noel Cole (Chroma)

Function XorBank:TBank(bank:TBank, key$)
	'read bank data to buffer string
	Local buffer$
	For i = 0 To bank.size()-1
		Local b:Byte = PeekByte(bank,i)
		buffer :+ Chr$(b)
	Next
	
	buffer = xor(buffer,key$)

	Local rbank:TBank = CreateBank(buffer.length)
	For i = 0 To rbank.size()-1
		PokeByte(rbank,i,Asc(Mid$(buffer,i+1,1)))
	Next
	Return rbank
End Function

Function xor:String(str$, key$)
	Local ml%, pl%, i%, result$
	ml% = Len(str)
	pl% = Len(key)
	For i = 0 Until ml
		Local c:Byte = str$[i]
		Local k = key$[i Mod pl]
		c = c ~ k
		result$ :+ Chr(c)
	Next 
	Return result$
EndFunction

Function CompressBank:TBank( bank:TBank )
	Local size=bank.Size()
	Local out_size=size+size/10+32
	Local out:TBank=TBank.Create( out_size )
	compress out.Buf()+4,out_size,bank.Buf(),size
	out.PokeByte 0,size
	out.PokeByte 1,size Shr 8
	out.PokeByte 2,size Shr 16
	out.PokeByte 3,size Shr 24
	out.Resize out_size+4
	Return out
End Function




Function LoadImageZNG:TImage(zngFile$, flags%=-1)
	Local cbank:TBank = LoadBank(zngFile)
	Local ubank:TBank = UncompressBank(cbank)
	If ZNG_Key Then ubank = XorBank(ubank,ZNG_Key)
	Return LoadImage(ubank, flags)
End Function

Function LoadPixmapZNG:TPixmap(zngFile$)
	Local cbank:TBank = LoadBank(zngFile)
	Local ubank:TBank = UncompressBank(cbank)
	Return LoadPixmap(ubank)
End Function




Function UncompressBank:TBank( bank:TBank )
	Local out_size
	out_size:|bank.PeekByte(0)
	out_size:|bank.PeekByte(1) Shl 8
	out_size:|bank.PeekByte(2) Shl 16
	out_size:|bank.PeekByte(3) Shl 24
	Local out:TBank=TBank.Create( out_size )
	uncompress out.Buf(),out_size,bank.Buf()+4,bank.Size()-4
	Return out
End Function

Function SetZNGKey(key$)
	ZNG_Key = key
End Function

Chroma

(Posted 2008) [#13]

you need to compress the file before encrypting

xlsior: EDIT: Yep you're right i was encrypting before compression. I switched it around thanks.

filax: I can't encrypt the byte itself because it will return a 1 to 3 character string which some could be string characters which not not fit back into the bank index.

For example. I retrieve a byte from a bank and it's 119. I encrypt that and it gives me back something like "(2^". I then can not just poke that back into the bank. What I'm doing is reading the byte, turning it into a single Chr$(), encrypting it to a different single char, changing it back into byte form with asc(). Any other ideas? :(

ImaginaryHuman

(Posted 2008) [#14]

Is ZNG your own image format? Does it actually have any relation to PNG at all?

I like xlsior's idea of encrypting after compressing otherwise you aren't giving your compression `ideal data` to compress.

Also I wondered, are you applying any of the `filters` which PNG uses to make the data more compressable? At the very least, I would convert the data to the *difference in value between the previous pixel and the next pixel* rather than absolute pixel values. Since colors don't change that rapidly usually this results in a much narrower range of values with much more repeatability, and compresses better. You might also look as some of PNG's other filter methods - although I'm assuming you're not already doing that.

I would avoid using strings ever at all to deal with data, you don't need to be using strings for anything other than handling text. Use pointers for fastest access, and write using things like Bank.WriteBytes or whatever it is.

Using zip to do your compression will probably be fairly good lossless compression, but you really can help it along by making reversable changes to the data before compressing.

How long does it take to compress and decompress? Does it decompress/load faster than a PNG?

Is your encryption basically to do a modulo based on a key?

PantsOn

(Posted 2008) [#15]

From what I understand now is that its a container for any image format..

Chroma

(Posted 2008) [#16]

It started off as my own format but migrated to a zipped and encrypted data file. It still serves my purpose quite well and loads very fast.

The encryption is the xor_crypt type encryption from the archives. And I'm using zlib for the compression via Mark's CompressBank and UncompressBank functions.

Yep I'm encrypting after compression now which makes much more sense. No there are no image filters etc. Like I said above it's really just a compressed and secure mirror of the file whether it's a jpg, png, bmp, doc, xls, b3d file it' doesn't matter. It just compresses, encrypts and writes it in byte form. I've actually altered the routine now to save the file extension in the encrypted file so now it can out put the original file.

Snarkbait

(Posted 2008) [#17]

Dont use a string for your key, use a 32bit or higher (powers of 2 only though) integer.... XOR that byte by byte. Straight xor encryption is lousy, however... in fact, post a .zng file that began as a bmp or png and I bet I can crack it in under an hour.

Chroma

(Posted 2008) [#18]

Ok here's the zng file. Btw, it started out as a .png. Clock is ticking. ;)

http://www.zamagames.com/downloads/crackme.zng

Snarkbait

(Posted 2008) [#19]

Dammit, I gotta go to work... will try this tonight!

Chroma

(Posted 2008) [#20]

Hehe cool. I opened the png and the zng in notepade and started doing string searches and a lot of png strings showed up in the zng. That's because by loading the entire bank into a string was bogging down the whole xor function so I ended up just encrypting a small part of the start, middle and end of the file. But by doing xor on each bank byte I can compress/encrypt an enormous bank of 32Mb in about 16 sec. Thanks for that suggestion Snark. After I used this method there are no matches at all so the entire contents are scrambled very thoroughly.

So now the flow is read the png into a bank. Compress the bank with zlib, encrypt the bank byte per byte with the 32bit encryption key, then savebank to a zng. That's pretty freakin secure. I also have a LoadImageZNG function that loads the zng into a bank, decrypts it, uncompresses it and returns a LoadImage(bank). I couldn't ask for more.

Oh and the 32Mb file was only 27Mb after being converted to ZNG. :-)

Oh here's the new method with your suggestion:

Function XorBank:TBank(bank:TBank, key$)
	For i = 0 To bank.size()-1
		Local c$ = Chr$(PeekByte(bank,i))
		PokeByte(bank,i,Asc(XorCrypt(c,key))) 
	Next
	Return bank
End Function

Function XorCrypt:String(str$, key$)
	Local ml%, pl%, i%, result$
	ml% = Len(str)
	pl% = Len(key)
	For i = 0 Until ml
		Local c:Byte = str$[i]
		Local k = key$[i Mod pl]
		c = c ~ k
		result$ :+ Chr(c)
	Next 
	Return result$
EndFunction

ImaginaryHuman

(Posted 2008) [#21]

Where will the key be stored? In your executable? Hackers can search your exe and try everthing as a potential key until they crack it.

Dreamora

(Posted 2008) [#22]

you can store it as SHA1 or use Kevs security modules ... both gives it enough security.

Media can never be protected against serious crackers as they are sent to GPU in any case and can be read at that moment.

Chroma

(Posted 2008) [#23]

Atm, the key is stored in an .ini file so the Loader and the Converter can access it and so it's easy to change during the dev process. All I was gonna do originally was put the key in the exe. I'll research the SHA1 stuff thanks for the tip Dreamora.

Chroma

(Posted 2008) [#24]

Snark, I just found a major problem with that way of implementing the encryption.

By cycling through the bank and applying the xor to each byte, only the first number is used to encrypt it. What I mean is even if you set the key to "198490378", the person only needs the first character of the key to be able to decrypt it.

If I put the key as "87264" then I just put an "8" as the key it still decrypts it. I think thats because xor works best on big strings. So now the problem again is to apply the xor to chunks of chr bytes. Which will slow it down immensely.

I could convert the bank into a stream and maybe xor the entire stream possibly. No idea how to actually do that. Any here have any ideas on how to encrypt an entire bank or stream as one long string?

ImaginaryHuman

(Posted 2008) [#25]

I don't see why you ever need to be dealing with strings at all, you should be using a pointer and reading the content of the bank directly ie mybankbaseptr[offset]. Strings are for dealing with text, you don't need the extra overhead of trying to think of stuff as characters unless it realyl is text, and even then you can work at a byte level.

I think Dreamora has a good point about being able to grab your media when it's drawn to the card. I guess you could say that whenever your game `unpacks` or `decrypts` the data for its own use, it is making it available to being stolen, even if it's at runtime and even if it's within the executable. As soon as you open up that file and make it interpretable for your own uses it's now open for access by anyone.

I remember playing around on the Amiga making a screen grabber in AMOS Basic - basically would display the contents of chip memory (graphics memory) as an image on the screen and you could move around in memory, change the size of the window/line modulo etc until you would see the graphics for open programs stored in memory - GUI's, screens for applications, game graphics (unless they totally shut out multitasking which they don't these days). It was really easy to access whatever graphics you wanted as soon as it was loaded into memory.

Chroma

(Posted 2008) [#26]

I don't care about people being able to grab media as it's drawn to the card or people that manipulate the actual memory to get your media. Most people have no clue how to do that so I'm not worried about it.

What I am trying to do here is:

bank = LoadBank("myfile.doc")
bank = CompressBank(bank)
bank = EncryptBank(bank)
SaveBank(bank,"myfile.doc.zng")

That's it. I have it all done except for how to effectivey encrypt a bank. Not a string but a bank. It's probably a lot more simple than it looks. I'm just not there yet. Maybe Xor type sucks and I need to use RC4 or z7 who knows.

My current method sucks. I'm doing this:

str$ = LoadString(bank)
str = XorCrypt(str,key)
bstream.writebytes(str,str.length)

Any file size over a few K bytes takes a really long time to encrypt. Now that you know exactly what I'm trying to do:

How would you encrypt a bank of say 128 bytes?

Chroma

(Posted 2008) [#27]

It was the blowfish encryption. I changed to a different type and now I'm all good.

EDIT: Yep, all good here now. I do actually use a string but that's the only way i know how to make sure it get's thorougly encrypted. 32Mb file encrypted in about 15 sec. Not bad at all.

On to the next project.

Chroma

(Posted 2008) [#28]

Actually, what's the max string length for a bmax string?

grable

(Posted 2008) [#29]

what's the max string length for a bmax string?

I would guess 2147483647 bytes, if your os can find a hole big enough ;)