Leadwerks Abstract File System

BlitzMax Forums/BlitzMax Programming/Leadwerks Abstract File System

JoshK

(Posted 2007) [#1]

I was coincidentally dealing with package formats when Noel put his PhysFS module up. I took some ideas from that and wrote a file system and package reader using ZipEngine.

http://www.leadwerks.com/post/AFS.zip

You start by registering the file path, returning a "Virtual Directory":

vd:TVirtualDir=RegisterPath(AppDir)

You can then go through the files and folders recursively:

For vf:TVirtualFile=EachIn vd
	print vf.url
Next
For sd:TVirtualDir=EachIn vd
	'Do something...
Next

This will treat zip file contents just as if they were in the same directory as the zip file. It will also handle zip directory structures. For example, a "materials" folder in the mypak.zip package is considered the same virtual folder as the "materials" folder on the hard drive.

To read a virtual file:

If vf stream:TStream=vf.Read()

This will either load the real file, or extract and load a temp file from its package file. You don't care where it comes from, you just want the file. Files with the same names in different virtual directories are accounted for, of course.

I think the best aspect of this is you can add new package formats by extending the TPackageReader type, as I did to make the zip reader. Nem's tools has a HL2 package lib, and it would be nice to see that working.

Picklesworth

(Posted 2007) [#2]

This sounds a bit like GnomeVFS (which I've been recently learning about) (if you know what that is).

Do you have plans for transparent access to files over the Internet and the like?

JoshK

(Posted 2007) [#3]

Hell no.

N	(Posted 2007) [#4]

Will take a look at this once I manage to fully wake up. Should be pretty good though.

JoshK

(Posted 2007) [#5]

This will display the contents of the Q3A directory. Double-click a file to extract and launch it.

'===============================================================
'Abstract File System by Leadwerks Software
'===============================================================

window:TGadget=CreateWindow("File System Browser",200,200,640,480)
treeview:TGadget=CreateTreeView(0,0,window.clientwidth(),window.clientheight(),window)
SetGadgetLayout treeview,1,1,1,1

'Clean up temp dir
DeleteDir AppDir+"/temp",True

'vd:TVirtualDir=RegisterPath(AppDir)
vd:TVirtualDir=RegisterPath("C:\Program Files\Quake III Arena\baseq3")

do vd,TreeViewRoot(treeview)

ActivateGadget window

While WaitEvent()
	Select EventID()
		Case EVENT_GADGETACTION
			For vf:TVirtualFile=EachIn TVirtualFile.list
				If HandleToObject(vf.userdata)=SelectedTreeViewNode(treeview)
					OpenURL vf.RealURL()
					Exit
				EndIf
			Next
		Case EVENT_WINDOWCLOSE
			End
	End Select
Wend

Function do(vd:TVirtualDir,node:TGadget)
	For sd:TVirtualDir=EachIn vd.kids
		snode:TGadget=AddTreeViewNode(StripDir(sd.url),node)
		do(sd,snode)
	Next
	For vf:TVirtualFile=EachIn vd.files
		snode:TGadget=AddTreeViewNode(StripDir(vf.url),node)
		vf.userdata=HandleFromObject(snode)
	Next
EndFunction

Function RegisterPath:TVirtualDir(url$,parentdir:TVirtualDir=Null)
	url=RealPath(url)
	d=ReadDir(url)
	If Not d Return
	vd:TVirtualDir=TVirtualDir.find(url,parentdir)
	Repeat
		file$=NextFile(d)
		If file="" Exit
		Select FileType(url+"/"+file)
			Case 1
				pr:TPackageReader=TPackageReader.ExtensionSupported(Lower(ExtractExt(file)))
				If pr
					pr.Read(url+"/"+file,vd)
				Else
					TVirtualFile.Register(url+"/"+file,vd)
				EndIf
			Case 2
				If file<>"." And file<>".."
					RegisterPath(url+"/"+file,vd)
				EndIf
		EndSelect
	Forever
	CloseDir d
	Return vd
EndFunction

Type TVirtualFile
	Global list:TList=New TList

	Field url$
	Field lurl$
	Field tempurl$
	Field rpath$=""
	Field source
	Field userdata:Int
	Field package:TPackage

	Method New()
		list.addlast(Self)
	EndMethod
	
	Method Read:TStream()
		If package=Null
			tempurl=url
			Return ReadFile(url)
		Else
			Return package.readfile(Self)
		EndIf
	EndMethod

	Method RealURL$()
		If package=Null
			Return url
		Else
			package.extract(Self)
			Return tempurl
		EndIf
	EndMethod

	Function Find:TVirtualFile(url$)
		url=RealPath(url)
		lurl$=Lower(url)
		For vf:TVirtualFile=EachIn list
			If vf.lurl=lurl Return vf
		Next
	EndFunction
	
	Function Register:TVirtualFile(url$,vd:TVirtualDir,package:TPackage=Null,rpath$="")
		If Find(url) Return Null
		vf:TVirtualFile=New TVirtualFile
		vf.url=RealPath(url)
		vf.lurl$=Lower(url)
		vf.rpath=rpath
		vf.package=package
		vd.files.addlast(vf)
		Return vf
	EndFunction
	
EndType

Type TPackage
	Field url$
	Field files:TList=New TList
	Field vf:TVirtualDir
	Field parent:TPackage
	Field kids:TList=New TList
	Field reader:TPackageReader
	
	Function Load:TPackage(url$,vd:TVirtualDir)
		url=RealPath(url)
		ext$=Lower(ExtractExt(url))
		For pr:TPackageReader=EachIn TPackageReader.list
			If pr.extensions.contains(ext)
				package:TPackage=pr.Read(url,vd)
			EndIf
		Next
		Return package
	EndFunction
	
	Method Extract:Int(vf:TVirtualFile)
		Return reader.extract(vf)
	EndMethod
	
	Method ReadFile:TStream(vf:TVirtualFile)
		Return reader.readfile(vf)
	EndMethod
	
EndType

Type TPackageReader
	Global list:TList=New TList
	
	Field extensions:TList=New TList
	
	Method New()
		list.addfirst(Self)
	EndMethod

	Method Read:TPackage(url$,vd:TVirtualDir) Abstract	
	Method Extract:Int(vf:TVirtualFile) Abstract
	Method ReadFile:TStream(vf:TVirtualFile) Abstract
	
	Function ExtensionSupported:TPackageReader(ext$)
		For pr:TPackageReader=EachIn TPackageReader.list
			If pr.extensions.contains(ext)
				Return pr
			EndIf
		Next
		Return Null
	EndFunction

EndType

Type TZipPackageReader Extends TPackageReader
	Global pr:TPackageReader=New TZipPackageReader
	
	Field zrObject:ZipReader
	
	Method New()
		extensions.addlast("zip")
		extensions.addlast("pk3")
	EndMethod

	Method GenerateExtractPath$(url$)
		Local index=0
		file$=StripAll(url)
		ext$=ExtractExt(url)
		path$=AppDir+"/temp/"+file+"."+ext
		While FileType(path)=1
			path$=AppDir+"/temp/"+file+"_"+index+"."+ext
			index:+1
		Wend
		Return path
	EndMethod
	
	Method Extract:Int(vf:TVirtualFile)
		If Not zrObject.OpenZip(vf.package.url) Return False
		If FileType(AppDir+"/temp")<>2 CreateDir(AppDir+"/temp")
		If FileType(AppDir+"/temp")<>2 Return Null
		vf.tempurl=GenerateExtractPath$(vf.url)
		zrObject.ExtractFileToDisk(vf.rpath,vf.tempurl,False,"")
		zrObject.closezip()
		Return True
	EndMethod
	
	Method ReadFile:TStream(vf:TVirtualFile)
		If Not Extract(vf) Return Null
		stream:TStream=ReadStream(vf.tempurl)
		Return stream
	EndMethod

	Method Read:TPackage(url$,vd:TVirtualDir)
		zrObject=New ZipReader
		If Not zrObject.OpenZip(url) Return Null
		package:TPackage=New TPackage
		package.url=url
		package.reader=Self
		package.vf=vd
		For Local i:Int=0 To zrObject.getFileCount()-1
			file$=zrObject.getFileInfo(i).zipFileName
			ext$=Lower(ExtractExt(file))
			Select ext
				Case ""
					dirname$=ExtractDir(url)+"/"+file
					vfd:TVirtualDir=TVirtualDir.Find(dirname,Null,True)
				Default
					If Instr(file,"/")
						dirname$=ExtractDir(url)+"/"+ExtractDir(file)
						vfd:TVirtualDir=TVirtualDir.Find(dirname,Null,True)
					Else
						vfd=vd
					EndIf
					realfile$=vfd.url+"/"+StripDir(file)
					TVirtualFile.Register realfile,vfd,package,file
			EndSelect
		Next
		zrObject.CloseZip()
		Return package
	EndMethod
	
EndType

Type TVirtualDir
	Global list:TList=New TList
	Field parent:TVirtualDir
	Field kids:TList=New TList
	Field url$,lurl$
	Field files:TList=New TList
	
	Method New()
		list.addlast(Self)
	EndMethod
	
	Function Find:TVirtualDir(url$,parent:TVirtualDir=Null,fix=False)
		url=RealPath(url)
		lurl$=Lower(url)
		For vf:TVirtualDir=EachIn list
			If vf.lurl=lurl Return vf
		Next
		vf=New TVirtualDir
		vf.url=url
		vf.lurl=lurl
		If parent=Null
			If fix
				dirname$=ExtractDir(url)
				parent=Find(dirname)
			EndIf
		EndIf
		If parent parent.kids.addlast(vf)
		Return vf
	EndFunction
	
	Method Display(ws$="",test=False)
		For f:TVirtualFile=EachIn files
			Print ws+"File: "+f.url
			If test
				stream:TStream=f.read()
				If stream
					stream.close()
				Else
					Notify "Error reading virtual file "+Chr(34)+f.url+Chr(34)
				EndIf
			EndIf
		Next
		For d:TVirtualDir=EachIn kids
			Print ws+"Folder: "+d.url
			d.display ws+"	",test
		Next
	EndMethod
	
EndType