Embedding files question

Blitz3D Forums/Blitz3D Programming/Embedding files question

Sir Gak(Posted 2014) [#1]
Is there a way to embed a sound or graphic file in a Blitz3D program so that there is no need to load it separately from the executable?

For instance, if I have a beep.wav file, can it somehow be embedded in the code itself, as DATA statements or something, so that the WAV file doesn't have to be loaded at all separately, but automatically is a part of the executable?


Yasha(Posted 2014) [#2]
Not in a way that will work with any built-in Load* commands. You will need to use the data to re-create the object in question using construction functions. (These functions aren't guaranteed to exist for all things, e.g. there's no builtin way to adjust or even create an MD2 object; they are only intended to be loaded in.)

There are "packer" tools that can embed your regular application in a kind of virtual filesystem and run it in that environment... don't know of any that are free offhand, but it might be a simpler avenue to investigate. That way you could develop the main application without worrying about it and pack it up later.


Guy Fawkes(Posted 2014) [#3]
SmartPacker Pro is not free, but it's DEFINITELY worth the money! It can also be used to encrypt your program with RJindael as well as Blowfish!


Rick Nasher(Posted 2014) [#4]
Maybe PakMaker by Terabit(which is no longer avail), see here:
http://www.blitzbasic.com/Community/posts.php?topic=85356

From the convo there I'm getting the impression it's free(dunno for sure).

But perhaps if you ask nicely, I know who has a copy ;-)
http://www.blitzbasic.com/Community/posts.php?topic=101776#1214462


This is why we need a Blitz file repository.. Pitty there's no such thing on this site, cos files come and go.


Rroff(Posted 2014) [#5]
One of the things I'm considering doing now the B3D source has been released is adding the ability to use banks to load data for sounds, meshes, etc. which would massively increase flexibility for stuff like this.


Guy Fawkes(Posted 2014) [#6]
Yes.. GOD yes..


Kryzon(Posted 2014) [#7]
That's a very good idea Rroff, it shouldn't be to hard to modify and the simple capability to load something from a bank covers all eccentric needs (loading from Data statements, loading from a pack, downloading from the Internet etc.)


virtlands(Posted 2014) [#8]
@yasha ::
There are packer tools that can embed your regular application in a kind of virtual filesystem...

Here are some virtual filesystem programs :::: [ Most of these are free. ]

Enigma Virtual Box :: http://enigmaprotector.com/en/aboutvb.html
EVALaze :: http://www.evalaze.de/en/home/
P-Apps :: http://portable-app.com/
MoleBox Virtualization :: http://www.molebox.com/molebox-vs.shtml
JauntPE :: http://jauntepe.sourceforge.net/index.html

BoxedApp Virtualization { NOT FREE } :: http://boxedapp.com/boxedapppacker/#.U-lCWGN5E0w
iCore Virtual Account :: http://icoresoftware.com/


Bobysait(Posted 2014) [#9]
You have the ability to store mesh, music, textures etc ... as data.
So, when you come to reload them from the binary, just export as a temporary file and load the real data

for ex :
Function genBinary(file$, outBB$, label$)
	Local in = ReadFile(file);
	Local out = WriteFile(outBB);
	Local count = FileSize(file);
	Local bank = CreateBank(count);
	ReadBytes(bank, in, 0, count);
	Local o = 0;
	Local n = 0;
	WriteLine (out, "."+label);
	WriteLine (out, "Data "+count );
	Local l$="";
	For o = 0 To count -1
		If n>0 Then l = l +",";
		l = l + PeekByte(bank, o);
		n = n + 1;
		If n=50
			WriteLine (out, "Data "+l);
			n = 0;
			l = "";
		EndIf;
	Next
	If Len(l)
		WriteLine (out, "Data "+l);
	EndIf
	
	WriteLine ( out, "Function readBinary_"+label+"$()" );
	WriteLine ( out, "  Restore "+label );
	WriteLine ( out, "  Local seed = RndSeed() : SeedRnd(MilliSecs())" );
	WriteLine ( out, "  local randName$ = str(rand(1,2^31)) + str(rand(1,2^31)) + "+Chr(34)+".tmp"+Chr(34) );
	WriteLine ( out, "  SeedRnd(seed)" );
	WriteLine ( out, "  Local tmp = writefile(randName)" );
	WriteLine ( out, "  Local size = 0 : Read size" );
	WriteLine ( out, "  local i%, d")
	WriteLine ( out, "  For i = 1 to size" );
	WriteLine ( out, "     read d : writebyte(tmp, d)" );
	WriteLine ( out, "  Next" );
	WriteLine ( out, "  closefile(tmp)" );
	WriteLine ( out, "  return randName" );
	WriteLine ( out, "End Function" );
	
	CloseFile(in);
	CloseFile(out);
End Function


Then all you have to do is :
export as ".bb" binary datas using 'genBinary("MyFileToExport.ext", "MyDataBBFileToCreate.bb","MyLabelForData")'

Finally, to load the medias from binaries, just use the generated function (it returns the name of the temporary file containing the datas and don't forget to delete the temporary file after it's been loaded)


Of course, it's not the best method, but it might be a quick and dirty solution.


mrniceguy(Posted 2014) [#10]
Check out Smart Packer Pro (X). Maybe it suits your needs.

More info...


virtlands(Posted 2014) [#11]
I like BobySait's experiment.
--------------------------------------------------------
Here is yet another way to access files (embedded) inside an EXE ::

Pros ::
(a) This method computes very fast, (no DATA statements needed).
(b) Doesn't require virtualization.

Cons ::
(a) It's complex to set up & requires a hex editor.
----------------------

Here's my demo code { attachfiles1.BB } -->
http://uploadingit.com/file/kfbjz5c5busi1rga/attachfiles1.zip

It requires freeimage --> http://freeimage.sourceforge.net/

This method involves using the binary copy command to append files to an EXE:

COPY /B /Y attachfiles1.exe +nightshade.jpg +poisonivy.jpg +lantana.jpg +laurel.jpg attachfiles1.exe

(...takes the first 5 files and joins them together into the last file)
( Reference :: https://en.wikipedia.org/wiki/Copy_%28command%29 )
(...Later on, the EXE file reads itself, and extracts media files from itself.)

[A:] This simple program takes a few JPGs and does a binary append to the EXE file.
[B:] Your mission, is to take a hex editor and find the offsets into the EXE file
where the media files (JPGs or otherwise) occur.

[C:] Further instructions are included in "Attachfiles1.BB" .

Sample program output :

-------------------------------------------------------------
Later, I realized that an even better method is to append all the media files together (using binary copy), then find each of those offsets
within that large chunk (for each media file).

[ Obviously, the first file shall start at offset zero. ]
Those offsets shall not change (often). Then take that large chunk and append that to your EXE.

Then, you'll only need to find the very 1st offset within the EXE, ...the rest of the offsets can be calculated mathematically,
(because they are relative to each other).



Blitzplotter(Posted 2015) [#12]
Excuse my question, please could you hi-light how to....

@VirtLands - could you hi-light the definition for the FreeImage_Unload function? Cheers....


Function FiUnload(dib)
 ;Frees a FreeImage bitmap, a wrapper function
 ;Returns nothing

 FreeImage_Unload(dib)

End Function



I've downloaded the FreeImage directory, was kinda hoping the dll would've sufficed....

found this within the FreeImage header file:-


// Allocate / Clone / Unload routines ---------------------------------------

DLL_API FIBITMAP *DLL_CALLCONV FreeImage_Allocate(int width, int height, int bpp, unsigned red_mask FI_DEFAULT(0), unsigned green_mask FI_DEFAULT(0), unsigned blue_mask FI_DEFAULT(0));
DLL_API FIBITMAP *DLL_CALLCONV FreeImage_AllocateT(FREE_IMAGE_TYPE type, int width, int height, int bpp FI_DEFAULT(8), unsigned red_mask FI_DEFAULT(0), unsigned green_mask FI_DEFAULT(0), unsigned blue_mask FI_DEFAULT(0));
DLL_API FIBITMAP * DLL_CALLCONV FreeImage_Clone(FIBITMAP *dib);
DLL_API void DLL_CALLCONV FreeImage_Unload(FIBITMAP *dib);



So... do I have to compile FreeImage.c to make a FreeImage.o that I can reference within the Attachfiles1.bb ? I thought depositing the FreeImage.dll within my B3D userlibs directory might've sufficed.

Incidentally, the attachfiles1.exe that comes with the download runs fine ;)


Blitzplotter(Posted 2015) [#13]
hmm, might need to d0 this-
http://www.blitzbasic.com/Community/posts.php?topic=93118#1063439


Guy Fawkes(Posted 2015) [#14]
HI, VIRTLANDS! =D


Blitzplotter(Posted 2015) [#15]
Got it - I needed a decls file declared a while back.....


Guy Fawkes(Posted 2015) [#16]
Cool! :)


virtlands(Posted 2015) [#17]
So... do I have to compile FreeImage.c to make a FreeImage.o that I can reference within the Attachfiles1.bb ?

Hi there, there's no need to re-compile the freeimage project,
you may use the FreeImage.DLL as it already is.

--------------------------------------------------------------------------

Here is the FreeImage.DECLS as of 11-29-2008 ::

--> https://www.dropbox.com/s/xg8hhw4nautbq5c/freeimage.decls?dl=0

( I replied to your email. )

--------------------------------------------------------------------------

However, if you do wish to search for some advanced freeimage code, then here are some links ::

FreeImage (Original) :: http://freeimage.sourceforge.net/sourcecode.html

Kanma/FreeImage :: https://github.com/Kanma/FreeImage


Blitzplotter(Posted 2015) [#18]
Hi Virtlands - I intend to finish what I started later today - thankyou for the additional feedback, it is really appreciated.

Really nice code you've put together with respect to FreeImage ;)


virtlands(Posted 2015) [#19]
I like the way you said "glueing/appending" files.

Well, I suppose the process can be automated a little bit.

Suppose you have a chunk of files to append to an EXE, perhaps 10 of them,

(a) You can first do a binary concatenation of those 10, into Chunk A,
(b) then later append Chunk A to the EXE.

(c) Once you know the magic offset (into the EXE) of the 1st file,
then you can do simple math to figure out the offsets of the remaining 9.
{ These other offsets would be "file sizes". }
--------------------------------------------------------------------------

The binary copy command is as follows :: source : http://support.microsoft.com/kb/71161


copy /b source1 + source2 + source3 + (....) targetfile

You can also combine several files into one by using wildcards. For example:

copy /b *.exe combin.exe


Blitzplotter(Posted 2015) [#20]
@VirtLands, this is good stuff.

[edit] just clocked this bit:

the EXE, ...the rest of the offsets can be calculated mathematically,
(because they are relative to each other).


yep, I'll get there......kinda sums up the rest of my post here:-

As you've suggested at 'C':



(c) Once you know the magic offset (into the EXE) of the 1st file,
then you can do simple math to figure out the offsets of the remaining 9.
{ These other offsets would be "file sizes". }




I suppose, in tandem with your instructions - you need to record the filesize of each individual file you are appending (if indeed they have different sizes), and populate an array with this in order for the sourcefilesize for each appended file?

I fail to see how your AttachFiles1.bb was able to know where each appended file 'ended' - is there a EOF in effect in binary?



If I'm interpreting what the sourcefilesize attribute is meant to be.....
(i.e. the size of each individually appended file...)




Hmmm, I think I might be asking a little much of the FreeImage.dll to try and load/manipulate .b3d files into memory banks alongside .tga files.....

I'll revert to using 4 image files as you have in the example rather than interspersing the image files with .b3d models.

[edit] As you've suggested, I might also try simple hex arithmetic to ascertain individual file sizes.

Cheers, BP.


virtlands(Posted 2015) [#21]
...interspersing the image files with .b3d models.

Maybe embed the B3D file into the EXE, self-extract it to disk, and then read it naturally.

Just in case you want to keep your media files a secret, then please do encrypt them before embedding them into the EXE

, otherwise people can hack your program apart with programs such as these

(a) Binary Inspector :: http://tinyurl.com/l5s79jt (FREE Online App)
(b) Allmedia Grabber :: http://tinyurl.com/25ek3d ($54.90)
(c) Dragon Unpacker :: http://tinyurl.com/pbuwlhe (Free)
(d) Free Resource Extractor :: http://tinyurl.com/mpb7mjc (Free)
(e) MultiExtractor :: http://tinyurl.com/oj6d2g6 (Free?)
---------------------------------------------------------------------------------


As an example, I took some game data file, (res2.dat), dragged it into Binary Inspector;
It found 371 images, and 105 music files. That's awesome.,




Blitzplotter(Posted 2015) [#22]
@VirtLands, thanks for the feedback - managed to add a fifth image to your AttachFiles.exe and display it ;) onwards and upwards:--



Binary Inspector looks like an interesting utility. I need to look into how to self-extract .b3d file to disk.... my aspiration was to rip it straight from the memory bank in question into the relevant entity variable at run time therefore removing the necessity to self-extract the .b3d file to disk. I might need a little more sleep for my grey matter to digest further where I'm going with this - for now though- pleased to get my test5th image added to your initial example ;)

Hmmm, I fear I'm trying to achieve the impossible with this approach, receving a MAV around the attempt to load the b3d from the memory bank.

; Get Freeimage bitmaps for each JPG bank
For z=1 To 5
    bitmap[z] = FiLoadFromBank( bank[z] )  ;; Save into a Freeimage bitmap
    image[z] = FiRead(bitmap[z])    
Next 

Print "getting this far (3)... press any key..."

WaitKey

; Get a b3d entity

b3dentityraw = FiLoadFromBank( bank[6] )  ;; Now, is this attempting to put a b3d entity into a bitmap? Could cause issues

b3dentityraw = FiLoadFromBank( bank[7] )


Print "getting this far (4)... press any key..."

WaitKey

;Local b3dentityone = LoadAnimMesh(Read(bank[6]))

;Local b3dentitytwo = LoadAnimMesh(Read(bank[7]))

b3dentityone = LoadAnimMesh(b3dentityrawone)

b3dentitytwo = LoadAnimMesh(b3dentityrawtwo)

Print "getting this far (5)... press any key..."

WaitKey




Blitzplotter(Posted 2015) [#23]
I've managed to identify where the offset for the first and last byte for a particular memory bank, which has been appended to a file. So, somehow, I need to:--

1. Write out just these bytes to a new file upon each execution

2. Load the AnimMesh from that file once I have written it to disk

3. Delete said file after loading it from disk.

I've tried analysing Boby Saits code with a view to achieving this goal - however brainache has set in.

Every time it loads - it'd be nice to simply copy the memory bank in question into the LoadAnimMesh command - is there any syntax available to do this?