File writing (stumped!?)

Blitz3D Forums/Blitz3D Beginners Area/File writing (stumped!?)

BlackD

(Posted 2004) [#1]

Why does this not work? :) Trying to write an integer over an entire file.

	timer=MilliSecs()
	
	fileout=OpenFile("test.txt")
	Print "opened!"
	SeekFile fileout,0
	While Not Eof(fileout)
	WriteInt fileout,0
;	SeekFile fileout,(FilePos(fileout)+4)
	Wend
	CloseFile fileout
	
	timeelapsed=MilliSecs()-timer
	Print timeelapsed/1000
	Print "done!"
	Delay 200
	End

Uncommenting the seekfile line doesn't work either.

The following method DOES work, but isn't suitable due to any file larger than a certain size resulting in a negative value for filesize()

	timer=MilliSecs()
	
	fileout=OpenFile("test.txt")
	Print "opened!"
         for currentint = 0 to filesize("test.txt") step 4
	SeekFile fileout,currentint
	WriteInt fileout,0
	Next
	CloseFile fileout
	
	timeelapsed=MilliSecs()-timer
	Print timeelapsed/1000
	Print "done!"
	Delay 200
	End

Any ideas why EOF isn't working here? In fact, it doesn't actually write ANY bytes in the first example. ?? No idea why not. :( It's doing pretty much the same as the second one, except the second one relies on a counter to seek.

+BlackD

Graythe

(Posted 2004) [#2]

Well, looking at the first code example - The first valid file location is 1 - not zero. You are trying to seek a location that does not exist. The second does the same.

[edit]Whoops! - Graythe wipes egg from chin[/edit]

BlackD

(Posted 2004) [#3]

No, seeking works via offset from start of file. To get the first byte, the offset is 0. As I said, the second example works. HOWEVER, changing this to a 1 makes no difference to it working or not working. In both cases, the first example doesn't work, but the second does. I'm not trying to get the second example working, I'm trying to get the first example working. I need file writing using EOF, but it doesn't seem to be working. >_<

VIP3R

(Posted 2004) [#4]

In the first block of code you posted, the SeekFile is after the WriteInt. Also, after you write the integer, you're trying to seek from the current file position (end of file) then adding 4 bytes (past the end of file). The opposite to the second block of code.

I'm confused, if all you want to achieve is writing an integer over an entire file, why not just use WriteFile instead of OpenFile?

BlackD

(Posted 2004) [#5]

1. If SeekFile goes past the end of the file, it doesn't error out, it simply returns a negative number. The EOF should then be detected before the next WRITEINT in the loop. It isn't being detected, and this is irrelevant, as it's not writing any data at all for some reason?
2. I want to "overwrite" the current data. It's a file shredder. I've got 27 file passes to write. It's working, with files of limited file size (as with the second example), but over 500 megs or so - it just errors out as filesize() returns a negative value (because it's a large, signed int). The passes I'm writing:

	.passdata
	Data 27
	Data 1,0,1,1431655765,"01010101 01010101 01010101","0x55"
	Data 1,0,1,-1431655766,"10101010 10101010 10101010","0xAA"
	Data 0,1,1,-1843115630,"10010010 01001001 00100100","0x92 0x49 0x24"
	Data 0,1,1,1234314313,"01001001 00100100 10010010","0x49 0x24 0x92"
	Data 0,1,1,608801316,"00100100 10010010 01001001","0x24 0x92 0x49"
	Data 1,1,0,0,"00000000 00000000 00000000","0x00 0x00 0x00"
	Data 1,0,0,286331153,"00010001 00010001 00010001","0x11"
	Data 1,0,0,572662306,"00100010 00100010 00100010","0x22"
	Data 1,1,0,858993459,"00110011 00110011 00110011","0x33"
	Data 1,0,0,1145324612,"01000100 01000100 01000100","0x44"
	Data 1,0,1,1431655765,"01010101 01010101 01010101","0x55"
	Data 1,1,0,1717986918,"01100110 01100110 01100110","0x66"
	Data 1,0,0,2004318071,"01110111 01110111 01110111","0x77"
	Data 1,0,0,-2004318072,"10001000 10001000 10001000","0x88"
	Data 1,1,0,-1717986919,"10011001 10011001 10011001","0x99"
	Data 1,0,1,-1431655766,"10101010 10101010 10101010","0xAA"
	Data 1,0,0,-1145324613,"10111011 10111011 10111011","0xBB"
	Data 1,1,0,-858993460,"11001100 11001100 11001100","0xCC"
	Data 1,0,0,-572662307,"11011101 11011101 11011101","0xDD"
	Data 1,0,0,-286331154,"11101110 11101110 11101110","0xEE"
	Data 1,1,0,-1,"11111111 11111111 11111111","0xFF"
	Data 0,1,1,-1843115630,"10010010 01001001 00100100","0x92 0x49 0x24"
	Data 0,1,1,1234314313,"01001001 00100100 10010010","0x49 0x24 0x92"
	Data 0,1,1,608801316,"00100100 10010010 01001001","0x24 0x92 0x49"
	Data 0,1,0,1843115629,"01101101 10110110 11011011","0x6D 0xB6 0xDB"
	Data 0,1,0,-1234314314,"10110110 11011011 01101101","0xB6 0xDB 0x6D"
	Data 0,1,0,-608801317,"11011011 01101101 10110110","0xDB 0x6D 0xB6"

From left to right:
1/0 - Destroy RLL1 magnetic scanning (Scanning Probe Microscopy)?
1/0 - Destroy RLL2 magnetic scanning (Magnetic Force Scanning Tunnelling Microscopy)?
1/0 - Destroy MFM magnetic scanning (Magnetic Force Microscopy)?
INT - Int to write to file, resulting in actual bit mask.
"0100100etc" - Binary representation of the bit mask
"0xCCetc" - Hexidecimal representation of the bit mask

Most of the "file-shredding" programs out there, use "DoD level file destruction!" as if it's some great thing. At only 7-9 levels of overwriting, unfortunately, this only stops SPM. The proclaimed "DoD Standard" is a decade old. There are two newer scanning functions out there which walk all over programs like "Evidence Eliminator", etc. My program destroys data beyond any currently known recovery methods except.. sigh.. it isn't work on files over 500 megs because of this bug I can't work out. :)

The README if anyone's interested in helping sort this error out:

============================================================
                         FileBLITZer 1.0
             Copyright (c) 2004, Kinnear Penman Jnr
============================================================

    This document contains important usage and licensing 
    information that you'll need to read to get the most
    out of the software, and to ensure your files are
    properly shredded.

    Please read in its entirety, then click the I AGREE
    button below.

    CONTENTS
     - Introduction
     - What is involved in shredding
     - How does it work
     - What you need to know
     - Licensing

------------------------------------------------------------

    INTRODUCTION

    FileBLITZer is a free file-shredding utility created
    in the BlitzPlus programming language. BlitzPlus is a
    BASIC syntaxed Windows API programming language
    created by Blitz Research Ltd.

    FileBLITZer provides a level of file destruction that 
    exceeds even many commercial file shredding software 
    such as Evidence Eliminator. While most file shredding
    programs provide a level of data destruction claimed 
    to be at "US DoD" standards - this standard is by far
    out of date, and is dangerous to be relied upon.

------------------------------------------------------------

    WHAT IS INVOVLED IN FILE SHREDDING

    Regular "DoD" file shredding invoked by most free and
    commercial data destruction software involves between 
    7-9 bit writing passes over the length of the file.
    This guarantees the data cannot be recovered by any
    software means, and also rules out recovry via electron
    microscopes with the old Scanning Probe Microscopy
    method. 

    However, times have changed. Magnetic Force Microscopy 
    (MFM) techniques currently in use OUTDATE the "DoD"
    effective file-shredding methods. And don't get lured
    into thinking "But I'm not doing anything illegal, so
    I don't need NSA-level shredding, they're never going
    to be accessing my stuff". Wrong! MFM readers can be
    built for under $2000. Every data recovery company has
    one. A snooping flatmate or employer can have all your
    deleted (and even DOD-Shredded) data for a mere $100
    down at the local data recovery co.

    Futhermore, as hard-drive manufacturers pack more and
    more data onto high-density platters, they start using
    methods of using neighbouring bits to record additional
    data. Even if your file is completely shredded, the
    information can be read from these neighbouring bits
    using Magnetic Force Scanning Tunneling Microscopy 
    (STM). Now we ARE talking NSA-level. :)

    Furthermore, the file entry in the FAT table can give
    many clues to investigations. Some shredding programs
    merely delete the file after it is "shredded", leaving
    the original file name and size information intact.

    Yet, FileBLITZer provides hope. :)

------------------------------------------------------------

    HOW DOES IT WORK

    FileBLITZer performs (up to) 35 individual passes on
    the file. Not 7. Not 12. 35! And I'm not talking mere
    random data or 0000s and 1111s.

    To truly delete a file forever, you need to align the
    magnetic field of the file and neighbouring bits, back
    to nothing. Neither 0 or 1. Of course every bit has to 
    be either a 0 or 1 as we are writing (not using a 
    degausser) so this instead has to be done using a 
    careful sequential series of bit masks which "pull" the
    magnetic in both directions to equilibrium. The 
    particular  series of bit-masks required is known as
    the Gutmann-Method, after Peter Gutmann, Department of 
    Computer Science, Auckland University, New Zealand.

    FileBLITZer is broken down into the following steps.
     - You select which file(s) you wish to "blitz".
     - The total byte-length is calculated and shredding
       begins.
     - 4 passes of random data are written initally. This
       is merely a precaution.
     - 27 (at most, depending on what shredding level you
       have selected) bit-mask passes completely overwrite
       the data beyond any recovery means known to modern
       science.
     - 4 passes of random data are written at the end of
       the alignment process to provide "filler" data.
     - The filenames are overwritten 35 times with the
       Gutmann method so even their NAMES cannot be
       recovered.
     - The files size entry in the FAT table is rewritten
       50 times with random sizes.
     - The data in memory which housed the information of
       what files are being shredded in RAM is also
       shredded by bitmask. RAM can store information even
       after your computer is closed down. This removes
       any chances of that happening. I know of no other
       program which takes this precaution.

    Now, the file is unrecoverable even by the most
    advanced magnetic force electron scanning methods.
    Nobody even knows the file name or file size. Even the
    system RAM in use has no idea what just happened. :)

------------------------------------------------------------

    WHAT YOU NEED TO KNOW

    Firstly - FileBLITZer is not fast. In fact, it's
    painfully slow, due to the way Blitz accesses the disk.
    But it's secure.

    If you're running any automatic data-backup software in
    the background, such as Windows System Restore, you need
    to disable it. What's the point in shredding a file if
    you've got another program making backups of all your
    data beforehand to make sure you don't delete anything
    important?

    If you ARE using a software recovery program, then not
    only will you need to disable it, but you'll need to
    "BLITZ" the backup files. Simply disabling it means the
    backup files will just stop being used. They're still
    there and data can still be recovered.

    If you are using Norton Unerase or Norton RecycleBin
    Protection, it is highly recommended you disable these
    utilities. There is no reason why having them active
    should cause any problems, but maybe they've put in
    some new-fangled thing to make protected backups of
    files that look like they're being "overwritten" by a
    virus. I don't know - but just to be safe. :)

------------------------------------------------------------

    LICENSING

    FileBLITZer is free for everyone. However, it is only a
    proof-of-concept program. It should not be taken
    seriously, and is used at your own risk. If it deletes
    all your data, fries your hardware, sets fire to your
    cat, or any other unforseen mishaps - I can't be held
    responsible. You're under no obligation to use this
    software, and doing so is your own choice.

    You have been warned.

    The source-code is provided, ONLY for review purposes.
    You may not re-use, re-package or re-distribute the 
    source code in any other form. You cannot alter it.
    You cannot sell it. You cannot post it to IRC. You
    get the point.

    If you want to port FileBLITZer to another (faster)
    language, then please get permission first. Likewise,
    with comments, requests, email me:
                        black3d@...

------------------------------------------------------------

+BlackD

VIP3R

(Posted 2004) [#6]

Sorry BlackD, I was misunderstanding what you meant :)

Ok, try this code instead:

timer=MilliSecs()
	
fileout=OpenFile("test.txt")
Print "opened!"

While Not Eof(fileout)

SeekFile fileout,FilePos(fileout)
WriteInt fileout,0

Wend

CloseFile fileout
	
timeelapsed=MilliSecs()-timer
Print timeelapsed/1000
Print "done!"
Delay 200
End

In your first example the '+4' isn't necessary, as this would write 4 bytes (0-3) then you were jumping 4 bytes on top of the current file position which is now 3. So in effect you were writing 4 bytes (0-3) then missing 4 bytes (4-7) then writing 4 bytes (8-11) and so on...

Also, the 'SeekFile fileout,0' isn't necessary as the file position will already be set to zero when you open the file with OpenFile.

The SeekFile *MUST* be before the WriteInt to work correctly, I'm not sure why, maybe something to do with checking the EOF after using the SeekFile command?

I'm not sure what the maximum file size this code will work on though, you need to test it and see, but it does work.

BlackD

(Posted 2004) [#7]

Aha! I had this working before as you've posted it above, but for some reason it kept on infinitely writing (as your code does, I kept stopping it at around 50 megs) and it was never reaching EOF. I couldn't figure out why, and my other method (the first one) didn't write at all. However, something in the way you just explained it made me realise why..

Of course, unless the file is evenly divisible by 4 (the length of the INT), then once it writes the last INT, its no longer at the EOF, so it writes another one, etc etc ad nauseum. :) To fix it, I wrote another EOF check which EXITs the loop if the following byte is past the end of the file.

Thanks immensely for your help. :)

timer=MilliSecs()
	
fileout=OpenFile("test.txt")
Print "opened!"

Repeat
SeekFile fileout,FilePos(fileout)
WriteInt fileout,-1843115630
SeekFile fileout,FilePos(fileout)+1
If Eof(fileout) Then Exit
SeekFile fileout,FilePos(fileout)-2
Forever
CloseFile fileout
	
timeelapsed=MilliSecs()-timer
Print timeelapsed/1000
Print "done!  "
Delay 200
End

Oh, the -2 is for my own use. I'm writing INTs (for speed) but actually only want to write 3 bytes, not 4, for the bit-masking pattern. This is a crap way of doing it, so now that I've got the seeking sorted out, I'll get it to dynamically create memory banks up to 5 megs, containing the bitmask to write, to drastically speed up large-file shredding. At the moment, with the code above, it takes about 10 seconds per meg. But considering I'm writing 35 passes (27+8 random data), 4.5 minutes per meg is a little on the slow side. ;) Writing 5 meg chunks should reduce this to about 2 seconds per meg. :)

+BlackD

VIP3R

(Posted 2004) [#8]

Glad to help BlackD :)

I've done a lot of file stuff recently when working on a file encryption application (which I can't release thanks to DTI export controls). I was using banks for that too, for the same reason to speed things up, which it does dramatically! ;)