How to write massive amounts of hard data to a file?
BlitzPlus Forums/BlitzPlus Programming/How to write massive amounts of hard data to a file?
| ||
| I have a great little program going that lets you select a directory, then it generates code for an installer. The program has the directory structure and files hard-coded in CreateDir() and WriteByte() files. The problem is that the 16-mb program I am trying to make the installer for results in a 150 mb .bb file, since every byte generates a WriteByte command. WriteInt wouldstill result in something like a 30-mb .bb file. Source code that large does not compile. Isn't there are way to write large amounts to a file in some other kind of data, that would let me hard-code the bytes and generate a smaller amount code? Binary or something? I don't know about this kind of thing because I never use it. |
| ||
| Sure, you could use "Data" and "Read". That would mean that for each byte or int of data (depending on whether you use WriteInt or WriteByte), you would just have to have a comma and an occasional "Data " and newline (depending on how long Data statements can be). You would also only need one Write statement in a loop. This would generate much smaller source than using a Write command for each chunk of data. But I guess I don't see why you want the intermediate step of writing sourcecode to generate a file when you could just generate the file from the get-go... If you wrote all this data to a binary file first, you wouldn't need to write it INTO the installer code. You could just have the installer program read the binary data and then do whatever with it... I guess. |
| ||
| Then the installer wouldn't be a single program. It would be a program which copied files from one directory to another. In my experience, users are often to dumb to unzip a file. I guess the question is, how big a source can Blitz compile? It will be at about twice as big as the files you are writing the installer for. |
| ||
| soja is correct. You might be able to compact the data a bit more by squashing it into long Strings in the data statements. |
| ||
| If you use strings you risk losing data. No, you WILL lose data. I am using Data() now, and successfully made an installer for about 2 megs of data. The resulting .exe was 15 megs, but I UPX'ed it down to 2. I ran the program, and it installed all the files perfectly, and very fast. It is really nice to have an installer that just does its job without taking like 30 mouse clicks. I have a 60-mb source code I am going to try to compile now. This might be the biggest Blitz source ever. |
| ||
| Halo, Does the 2MB limit apply across Include files (ie. can you have say 50 lots of 2MB files?) Alternatively, write the data to a text file, and use the various utitlities available for Blitz to add the text file into the exe, which extracts and reads it at runtime. |
| ||
| Appending the data to the .exe itself is probably a much better way to do this. I have been compiling a 15 mb source for the last 25 minutes. Somehow, I don't think Blitz was designed for this. |
| ||
| Okay...how do you tell where the end of an executable is? I guess I could code the file size of the .exe into the program itself, but that is kind of complicated. |
| ||
| I suggest you put in a special byte sequence at the start of the data which is lengthy and complex enough to ensure that it won't appear elsewhere by accident. Your program can then search for that, and read on from there. The next section being a header which contains data about the files, and the next being the files themselves. The char equivilent of "HALOSINSTALLER" for example :) |
| ||
To write sig:
;Open Test Executable
fileout = OpenFile("test.exe")
SeekFile fileout,FileSize("test.exe")
;Write Signature for locating start of data
WriteByte fileout,Asc("R")
WriteByte fileout,Asc("O")
WriteByte fileout,Asc("B")
CloseFile fileout
End
To locate sig:
file = OpenFile("test.exe")
While Not Eof(file)
If ReadByte(file)=Asc("R")
;Found first
If ReadByte(file)=Asc("O")
;Found second
If ReadByte(file)=Asc("B")
;Found third
found=FilePos(file)
EndIf
EndIf
EndIf
Wend
While Not KeyHit(1)
Text 20,20,Found
Wend
End
I've got more complete code if you want. |
| ||
| I think is better to simply to locate the end of the EXE, then store this value. Put all the data on the EXE, and then put the value you stored before. Then, to access the data you only have to read the last Int in the EXE to find the start of the data, and read everything from that position to the end of the file, except the last Int. |
| ||
| Didn't think of that - it would certainly be a lot faster. There is a delay of about 2000ms whilst it locates the .sig as I posted above. |
| ||
| Yup that's the way DataPak does it, remember the size of the .exe and then write it as the last four bytes of the .exe after your appended data. Just read those last four bytes and you have your starting offset to your data. |
   |