Large 2GB+ file stream

BlitzMax Forums/BlitzMax Programming/Large 2GB+ file stream

Azathoth (Posted 2010) [#1]

This is my first attempt at creating a file stream that exceeds the current 32bit limit. I haven't tested it much, it does however work writing to a file past the 32bit barrier. You should use the PosEx, SizeEx, and SeekEx to handle the larger 64bit positions and size. It's Win32 only. Strict Import Pub.Win32 Const INVALID_HANDLE_VALUE = -1 Const GENERIC_READ = $80000000 Const GENERIC_WRITE = $40000000 Const OPEN_EXISTING = 3 Extern "win32" Function CreateFileW(lpFileName:Byte Ptr,dwDesiredAccess,dwShareMode,lpSecurityAttributes:Byte Ptr,dwCreationDisposition,dwFlagsAndAttributes,hTemplateFile) Function GetFileSizeEx(hFile, lpFileSize:Byte Ptr) Function SetFilePointerEx(hFile, liDistanceToMove:Long, lpNewFilePointer:Byte Ptr, dwMoveMethod) Function ReadFile(hFile, lpBuffer:Byte Ptr, nNumberOfBytesToRead, lpNumberOfBytesRead:Byte Ptr, lpOverlapped:Byte Ptr) Function WriteFile(hFile, lpBuffer:Byte Ptr, nNumberOfBytesToWrite, lpNumberOfBytesWritten:Byte Ptr, lpOverlapped:Byte Ptr) Function FlushFileBuffers(hFile) Function CloseHandle(hObject) Function GetLastError() EndExtern Function FileSizeEx:Long(path:String) Local size:Long, handle:Int handle=CreateFileW(path.ToWString(),GENERIC_READ,0,Null,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0) If handle<>INVALID_HANDLE_VALUE If GetFileSizeEx(handle, Varptr size)=0 Then size=-1 CloseHandle(handle) EndIf Return size EndFunction Type TFileStream Extends TStream Field _pos:Long, _size:Long, _handle=INVALID_HANDLE_VALUE, _mode Method PosEx:Long() Return _pos EndMethod Method SizeEx:Long() Return _size EndMethod Method Eof() Return (PosEx()=SizeEx()) EndMethod Method SeekEx:Long(pos:Long) Assert _handle<>INVALID_HANDLE_VALUE Else "Attempt to seek closed stream" SetFilePointerEx(_handle,pos,Varptr _pos,0) Return _pos EndMethod Method Read(buf:Byte Ptr, count) Assert _handle<>INVALID_HANDLE_VALUE Else "Attempt to read from closed stream" Assert _mode & GENERIC_READ Else "Attempt to read from write-only stream" If ReadFile(_handle,buf,count, Varptr count, Null)=0 Then Return 0 _pos:+count Return count EndMethod Method Write( buf:Byte Ptr,count ) Assert _handle<>INVALID_HANDLE_VALUE Else "Attempt to write to closed stream" Assert _mode & GENERIC_WRITE Else "Attempt to write to read-only stream" If WriteFile(_handle,buf,count,Varptr count, Null )=0 Then Return 0 _pos:+count If _pos>_size _size=_pos Return count End Method Method Pos() Return Int(_pos) EndMethod Method Size() Return Int(_size) EndMethod Method Seek(pos) Return SeekEx(pos) EndMethod Method Flush() If _handle<>INVALID_HANDLE_VALUE Then FlushFileBuffers(_handle) EndMethod Method Close() If _handle=INVALID_HANDLE_VALUE Then Return Flush() CloseHandle(_handle) _handle=INVALID_HANDLE_VALUE _pos=0 _size=0 EndMethod Function OpenFile:TFileStream(path:String,readable,writeable ) Local m, handle If readable m:|GENERIC_READ If writeable m:|GENERIC_WRITE handle=CreateFileW(path.ToWString(),m,0,Null,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,0) If handle<>INVALID_HANDLE_VALUE Local stream:TFileStream=New TFileStream stream._handle=handle SetFilePointerEx(handle,0,Varptr stream._pos,1) SetFilePointerEx(handle,0,Varptr stream._size,2) SetFilePointerEx(handle,stream._pos,Null,0) stream._mode=m Return stream EndIf EndFunction EndType

beanage (Posted 2010) [#2]

Couldnt you also use file mapping for this? I use file mapping to manipulate large database files, which is, afaik, what it's intended to be used for [in addition to shared memory].

Azathoth (Posted 2010) [#3]

Actually this is just meant to be an alternative file stream that can read and write files larger than 2GB which the standard module doesn't support.

beanage (Posted 2010) [#4]

Oh I propably behaved a little ignorant, sry, I realy like this solution! It was just a thought, that immidiately came to my mind when I saw your solution.

* (Posted 2010) [#5]

fraggin heck that would be one hell of a level to have a file thats 2Gb :s

Czar Flavius (Posted 2010) [#6]

What can I say, I like my whole world defined to the last blade of grass ;)

Sub_Zero (Posted 2013) [#7]

I've encountered a problem with this 2gb limit... A couple of questions: How do I do this in linux? How do i do file mapping?

Brucey (Posted 2013) [#8]

How do I do this in linux? Use Wine ? :-)

Sub_Zero (Posted 2013) [#9]

Ah thanks Brucey, but i'm looking for another approach :)

Brucey (Posted 2013) [#10]

Ah! Well, there is a module called BaH.SStream, which could do with some proper testing. It's basically a port of BRL.Stream with large file support. Type names have an extra "S". So, TSStream instead of TStream. And of course, most things return Long instead of Int. It's in SVN.

Sub_Zero (Posted 2013) [#11]

thanks

Sub_Zero (Posted 2013) [#12]

I get this: Compile error: Duplicate identifier 'WriteStream' in modules 'brl.stream' and 'bah.sstream' I tried to comment out "import brl.stream", but to no luck So i edited sstream.bmx, just changing some of the function names, and then rebuilt the module. Then building my program gave me this error: /home/BlitzMax/mod/bah.mod/sstream.mod/sstream.release.linux.x86.a(glue.c.release.linux.x86.o): In function `ftell64_': glue.c:(.text+0x4b): undefined reference to `ftell64' So i replaced line 33 of glue.c with: *pos = ftello64(stream); And now it works in linux :) ps. and it's really fast too :)