Reading/Writing File formats

BlitzMax Forums/BlitzMax Programming/Reading/Writing File formats

gellyware

(Posted 2006) [#1]

Does anyone have example(s) of how to read/write bmp, jpg, png files?

I'm looking at the wotsit.org site and although intrigued at all of the usefull file formats, I am not sure what to do with them.

Any help of reading/writing byte by byte information would be great.

Thanks.

ImaginaryHuman

(Posted 2006) [#2]

See in the manual,..

LoadPixmap
LoadImage
SavePixmapPNG

Max automatically can load jpeg png and bmp and can save png

ImaginaryHuman

(Posted 2006) [#3]

As to reading and writing files that's a different matter. See streams.

gellyware

(Posted 2006) [#4]

I believe there is a miscommunication.

I am not talking about the normal blitz way of loading images... I am talking about reading and writing raw (bytes) in the jpg,png (etc) format.

N	(Posted 2006) [#5]

http://www.wotsit.org/

Have fun.

Robert Cummings

(Posted 2006) [#6]

File : TStream commands in the manual. ReadFile, readbyte etc.

gellyware

(Posted 2006) [#7]

I am looking for code that reads in byte by byte information. I know of the blitz commands and the wotsit site.... what I am asking for is if anyone has written a png, jpg, bmp, reading/writing program(using bytes). I would like to see examples of what someone has done with info found at the wotsit site.

Examples Examples Examples.

Thanks.

Scott Shaver

(Posted 2006) [#8]

here is some java code to read a bmp file

/**
 * Handles dealing with windows bitmap files. This class doesn't handle palettized files.
 * +--------------------------------------+
 * | Bitmap File Header                   |
 * +--------------------------------------+
 * | Bitmap Information Header            |
 * +--------------------------------------+
 * | Palette Data (only in 8 bit files)   |
 * +--------------------------------------+
 * | Bitmap Data                          |
 * +--------------------------------------+
 *
 * @author Scott Shaver
 */
public class WindowsBitmapFile implements com.xith3d.imaging.ImageFile {

  private int    FHsize            =    0;
  private short  FHreserved1       =    0;
  private short  FHreserved2       =    0;
  private int    FHoffsetBits      =    0;

  private int    IHsize            =    0;
  private int    IHwidth           =    0;
  private int    IHheight          =    0;
  private short  IHplanes          =    0;
  private short  IHbitCount        =    0;
  private int    IHcompression     =    0;
  private int    IHsizeImage       =    0;
  private long   IHxpelsPerMeter   =    0;
  private long   IHypelsPerMeter   =    0;
  private int    IHcolorsUsed      =    0;
  private int    IHcolorsImportant =    0;
  private int    filePointer       =    0;
  private byte[] fileContents      = null;

  private byte[] data = null;

    public WindowsBitmapFile() {
    }

    public byte[] getData() {
        return data;
    }

    public int getWidth() {
        return IHwidth;
    }

    public int getHeight() {
        return IHheight;
    }

    public int getBPP() {
        return IHbitCount;
    }

    public int getDataLength() {
        return data.length;
    }

    public static BufferedImage getBufferedImage(String filename){
      WindowsBitmapFile loader = new WindowsBitmapFile();
      loader.load(filename);

      int width  = loader.getWidth(),

          height = loader.getHeight();

      BufferedImage bufferedImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);

      byte[] imageData = loader.getData();
      for(int j = height - 1; j >= 0; j--)
        for(int i = 0; i < width; i++) {
          int index = ((height - 1 - j)* width + i) * 3,
              color= (        255          & 0xFF) << 24|
 (imageData[index + 2] & 0xFF) << 16|
 (imageData[index + 1] & 0xFF) <<  8|
 (imageData[index + 0] & 0xFF);
          bufferedImage.setRGB(i,j, color);
        }
      return bufferedImage;
	}

    public void printHeaders() {
        System.out.println("-----------------------------------");
        System.out.println("File Header");
        System.out.println("-----------------------------------");
        System.out.println("            File Size:"+FHsize);
        System.out.println("           Reserved 1:"+FHreserved1);
        System.out.println("           Reserved 2:"+FHreserved2);
        System.out.println("          Data offset:"+FHoffsetBits);
        System.out.println("-----------------------------------");
        System.out.println("Info Header");
        System.out.println("-----------------------------------");
        System.out.println("     Info Header Size:"+IHsize);
        System.out.println("                Width:"+IHwidth);
        System.out.println("               Height:"+IHheight);
        System.out.println("               Planes:"+IHplanes);
        System.out.println("                  BPP:"+IHbitCount);
        System.out.println("          Compression:"+IHcompression);
        System.out.println("           Image size:"+IHsizeImage);
        System.out.println("     Pels Per Meter X:"+IHxpelsPerMeter);
        System.out.println("     Pels Per Meter Y:"+IHypelsPerMeter);
        System.out.println("     # of Colors Used:"+IHcolorsUsed);
        System.out.println("# of Important Colors:"+IHcolorsImportant);
    }

    public void load(String filename) {
        // reset everything
        FHsize            = 0;
        FHreserved1       = 0;
        FHreserved2       = 0;
        FHoffsetBits      = 0;
        IHsize            = 0;
        IHwidth           = 0;
        IHheight          = 0;
        IHplanes          = 0;
        IHbitCount        = 0;
        IHcompression     = 0;
        IHsizeImage       = 0;
        IHxpelsPerMeter   = 0;
        IHypelsPerMeter   = 0;
        IHcolorsUsed      = 0;
        IHcolorsImportant = 0;
        filePointer       = 0;

        InputStream  dis = ClassLoader.getSystemResourceAsStream(filename);

        try
        {
            if( dis == null)
              dis = new FileInputStream(filename);


            fileContents = new byte[dis.available()];
            dis.read(fileContents);
            try{dis.close();}catch(Exception x){}

            short magicNumber = readShort();

         //   FHtype
            // make sure it's windows bitmap file
            if(magicNumber != 19778)
            {
                fileContents = null;
                return;
            }

            // read the file header
            FHsize       = readInt();
            FHreserved1  = readShort();
            FHreserved2  = readShort();
            FHoffsetBits = readInt();

            // read the info header
            IHsize            = readInt();
            IHwidth           = readInt();
            IHheight          = readInt();
            IHplanes          = readShort();
            IHbitCount        = readShort();
            IHcompression     = readInt();
            IHsizeImage       = readInt();
            IHxpelsPerMeter   = readInt();
            IHypelsPerMeter   = readInt();
            IHcolorsUsed      = readInt();
            IHcolorsImportant = readInt();

            // allocate memory for the pixel data
            data = new byte[IHsizeImage];

            System.arraycopy(fileContents, FHoffsetBits, data, 0, IHsizeImage);
            fileContents = null;

            // swap the R and B values to get RGB, bitmap color format is BGR
            for(int loop=0;loop<IHsizeImage;loop+=3)
            {
                byte btemp = data[loop];
                data[loop] = data[loop+2];
                data[loop+2] = btemp;
            }
        }
        catch(Exception x)
        {
            x.printStackTrace();
            System.out.println(x.getMessage());
        }
    }

  private short readShort(){
    int s1 = (fileContents[filePointer++] & 0xFF),
        s2 = (fileContents[filePointer++] & 0xFF) << 8;
    return ((short)(s1 | s2));
  }

  private int readInt(){
    return (fileContents[filePointer++] & 0xFF)      |
 (fileContents[filePointer++] & 0xFF) <<  8|
 (fileContents[filePointer++] & 0xFF) << 16|
 (fileContents[filePointer++] & 0xFF) << 24;
  }


    public static void main(String[] args) {
        WindowsBitmapFile bf = new WindowsBitmapFile();
        bf.load(args[0]);
        bf.printHeaders();
    }
}

here is a little snippet of blitz code reading a binary tilemap file.

	Function Load:LayeredTileMap(filename:String)
		Local majorVer=1
		Local MinorVer=0
		Local newmap:LayeredTileMap = Null
		
		Local in:TStream = ReadStream(filename)
		If Not in RuntimeError "Failed to open a ReadStream to file "+filename
		Local mfs:TStream = LittleEndianStream(in)
			
		' read the file header
		Local fileid:String = Null
		fileid = ReadString(mfs,6)
		If fileid <> "SASMEF" RuntimeError filename+" is not an SAS Map Editor map file."
		majorVer = ReadInt(mfs)
		minorVer = ReadInt(mfs)

		' read version 1.0 of the map format
		If majorVer=1 And minorVer=0
			' read the map header
			Local mapid:String = Null
			mapid = ReadString(mfs,6)
			If mapid<>"SASMHD" RuntimeError filename+" is corrupt, unable to locate the SASMHD block."
			Local titleLen:Int = ReadInt(mfs)
			Local title:String = ReadString(mfs,titleLen)
			Local editLen:Int = ReadInt(mfs)
			Local edit:String = ReadString(mfs,editLen)
			Local creatorLen:Int = ReadInt(mfs)
			Local creator:String = ReadString(mfs,creatorLen)
			Local layerCount:Int = ReadInt(mfs)
			Local mcw:Int = ReadInt(mfs)
			Local mch:Int = ReadInt(mfs)
			
			' Read Tile Image
			Local nameLen:Int = ReadInt(mfs)
			Local name:String = ReadString(mfs,nameLen)
			Local frames:Int = ReadInt(mfs)
			Local cpw:Int = ReadInt(mfs)
			Local cph:Int = ReadInt(mfs)
		
			Local index:Int = filename.FindLast("/")
			Local path:String = ""
			If index<>-1 Then	path=filename[..index]
			Local TileImage:TImage = LoadAnimImage(path+"/"+name,cpw,cph,0,frames,FILTEREDIMAGE|MASKEDIMAGE)
		
			' create the new map object
			newmap = LayeredTileMap.Create(TileImage,frames,layerCount,mcw,mch,cpw,cph)

			newmap.imageName = name
			newmap.mapTitle = title
			newmap.mapCreator = creator
			newmap.mapLastEdit = edit
			newmap.imageTileCount = frames
		
			' read in layers
			For Local l=0 Until layerCount
				' read layer header
				Local layerid:String = Null
				layerid = ReadString(mfs,6)
				If layerid<>"SASLHD" RuntimeError filename+" is corrupt, unable to locate the SASLHD block for layer "+l+"."
				
				Local px:Int = ReadInt(mfs)
				Local py:Int = ReadInt(mfs)
				newmap.CreateLayer(l,px,py)
			
				' this is a speed optimization, istead of reading each int one at a time
				' we read them in in blocks, much faster but we have to deal with the endian stuff
				' ourselves
				Local temp:Int[] = New Int[mcw]
				Local bptr:Byte Ptr = Varptr temp[0]
				' cell tile index data
				For Local y:Int=0 Until mch
					'FlushMem
					mfs.ReadBytes( bptr,mcw*4 )
					For Local x:Int=0 Until mcw
?BigEndian
						Local t ' only do the byte swapping if on a big endian system, file is little endian
						t=bptr[(x*4)];bptr[(x*4)]=bptr[(x*4)+3];bptr[(x*4)+3]=t
						t=bptr[1];bptr[(x*4)+1]=bptr[(x*4)+2];bptr[(x*4)+2]=t
?
						newmap.layers[l].tiles[x,y]=temp[x]
					Next
				Next
				
				Local btemp:Byte[] = New Byte[mcw]
				bptr = Varptr btemp[0]
				' cell alpha data
				For Local y:Int=0 Until mch
					'FlushMem
					mfs.ReadBytes( bptr,mcw )
					For Local x:Int=0 Until mcw
						newmap.layers[l].alpha[x,y]=btemp[x]
					Next
				Next
			Next
		ElseIf majorVer=1 And minorVer=1
			' read the map header
			Local mapid:String = Null
			mapid = ReadString(mfs,6)
			If mapid<>"SASMHD" RuntimeError filename+" is corrupt, unable to locate the SASMHD block."
			Local titleLen:Int = ReadInt(mfs)
			Local title:String = ReadString(mfs,titleLen)
			Local editLen:Int = ReadInt(mfs)
			Local edit:String = ReadString(mfs,editLen)
			Local creatorLen:Int = ReadInt(mfs)
			Local creator:String = ReadString(mfs,creatorLen)
			Local layerCount:Int = ReadInt(mfs)
			Local mcw:Int = ReadInt(mfs)
			Local mch:Int = ReadInt(mfs)
			
			' Read Tile Image
			Local nameLen:Int = ReadInt(mfs)
			Local name:String = ReadString(mfs,nameLen)
			Local frames:Int = ReadInt(mfs)
			Local cpw:Int = ReadInt(mfs)
			Local cph:Int = ReadInt(mfs)
			Local transRed:Int = ReadInt(mfs)
			Local transGreen:Int = ReadInt(mfs)
			Local transBlue:Int = ReadInt(mfs)
		
			Local index:Int = filename.FindLast("/")
			Local path:String = ""
			If index<>-1 Then	path=filename[..index]
			SetMaskColor(transRed,transGreen,transBlue)
			Local TileImage:TImage = LoadAnimImage(path+"/"+name,cpw,cph,0,frames,FILTEREDIMAGE|MASKEDIMAGE)
		
			' create the new map object
			newmap = LayeredTileMap.Create(TileImage,frames,layerCount,mcw,mch,cpw,cph)

			newmap.imageName = name
			newmap.mapTitle = title
			newmap.mapCreator = creator
			newmap.mapLastEdit = edit
			newmap.imageTileCount = frames
			newmap.imageTransRed = transRed
			newmap.imageTransGreen = transGreen
			newmap.imageTransBlue = transBlue
		
			' read in layers
			For Local l=0 Until layerCount
				' read layer header
				Local layerid:String = Null
				layerid = ReadString(mfs,6)
				If layerid<>"SASLHD" RuntimeError filename+" is corrupt, unable to locate the SASLHD block for layer "+l+"."
				
				Local px:Int = ReadInt(mfs)
				Local py:Int = ReadInt(mfs)
				newmap.CreateLayer(l,px,py)
			
				' this is a speed optimization, istead of reading each int one at a time
				' we read them in in blocks, much faster but we have to deal with the endian stuff
				' ourselves
				Local temp:Int[] = New Int[mcw]
				Local bptr:Byte Ptr = Varptr temp[0]
				' cell tile index data
				For Local y:Int=0 Until mch
					'FlushMem
					mfs.ReadBytes( bptr,mcw*4 )
					For Local x:Int=0 Until mcw
?BigEndian
						Local t ' only do the byte swapping if on a big endian system, file is little endian
						t=bptr[(x*4)];bptr[(x*4)]=bptr[(x*4)+3];bptr[(x*4)+3]=t
						t=bptr[1];bptr[(x*4)+1]=bptr[(x*4)+2];bptr[(x*4)+2]=t
?
						newmap.layers[l].tiles[x,y]=temp[x]
					Next
				Next
					
				Local btemp:Byte[] = New Byte[mcw]
				bptr = Varptr btemp[0]
				' cell alpha data
				For Local y:Int=0 Until mch
					'FlushMem
					mfs.ReadBytes( bptr,mcw )
					For Local x:Int=0 Until mcw
						newmap.layers[l].alpha[x,y]=btemp[x]
					Next
				Next
			Next
		Else
			 RuntimeError filename+" has an unknown version of "+majorVer+"."+minorVer
		EndIf
					
		CloseStream in
		Return newmap
	End Function

together they should give you an idea of how to go about it.

gellyware

(Posted 2006) [#9]

Thanks Scott :) I'm going to go through these now.

Any other examples anyone? I like a lot of examples.

N	(Posted 2006) [#10]

The file format references are all you should need to read these formats like those.

Beaker

(Posted 2006) [#11]

You might want to take a look at the bmax source code for loading images:
C:\Program Files\BlitzMax\mod\brl.mod\pngloader.mod\pngloader.bmx
C:\Program Files\BlitzMax\mod\brl.mod\bmploader.mod\bmploader.bmx
etc

ImaginaryHuman

(Posted 2006) [#12]

Probably you'll have to write it yourself. As far as I know the jpeg and png loaders are based on external C code. I'd be surprised if anyone has yet written a jpeg or png loader/saver in purely blitzmax code. I'm going to try doing it at some point for my project but not right now. It can be pretty complicated depending on the file format.

xlsior

(Posted 2006) [#13]

JPG and GIF are hideously complex formats, mostly thanks to the bizarre compression routines & hash tables.

I did make a 256 color PCX writer for PowerBASIC 3.x (DOS) 8 years ago, based on the fileformat description from Wotsit. But PCX compression is much easier than JPG, since all it uses is a basic RLE (Run-Length encoding) algorithm, similar to BMP.

Probably won't be of much use in today's true-color world, but just in case:

' PB-PCX for PowerBASIC 3.x
' (C) February 1998 | Freeware, by Marc van den Dikkenberg
' <a href="http://www.xlsior.org" target="_blank">http://www.xlsior.org</a>
$LIB ALL -
$DYNAMIC
DEFINT A - Z

TYPE Pcxheader
	Mfg AS BYTE
	Ver AS BYTE
	Enc AS BYTE
	Bpp AS BYTE
	Xmin AS INTEGER
	Ymin AS INTEGER
	Xmax AS INTEGER
	Ymax AS INTEGER
	Hres AS INTEGER
	Vres AS INTEGER
	Pal AS STRING * 48
	Resrv AS BYTE
	Colpl AS BYTE
	Bpl AS INTEGER
	Paltyp AS INTEGER
	Filler AS STRING * 58
END TYPE
DIM Header AS SHARED Pcxheader
DIM Times AS SHARED BYTE
DIM Chunksize AS SHARED WORD
DIM Maxscrnptr AS SHARED WORD
DIM Textofst AS SHARED WORD
DIM Handle AS SHARED INTEGER
DIM Buffer AS SHARED STRING * 32256
Chunksize = 32256

' Screen 13 (320 x 200 x 256)
   ! MOV AX,&H13
	! INT &H10
	DEF SEG=&HA000

' Put some lines on the screen
	for t=0 to 199 step 2
	   poke$ t*320,string$(160,t)
	next t
	Sound 420,.5
	a$=input$(1)

SUB PCXSave(F$)
	header.mfg=10
	header.ver=5
	header.enc=1
	header.bpp=8
	header.xmin=0
	header.ymin=0
	header.xmax=319
	header.ymax=199
	header.hres=300
	header.vres=300
	header.pal=string$(48,0)
	header.Resrv=0
	header.colpl=1
	header.bpl=320
	header.paltyp=1
	header.filler=string$(58,0)

   filhandle=freefile
	open f$ for binary as #filhandle
	put$ #filhandle,header

	DEF SEG=&HA000
	for t=header.YMin to Header.YMax
	   temp$=peek$(t*320+header.XMin,Header.bpl)

' ****************
      poke$ t*320+header.xmin,string$(header.bpl,0)
'     Erase the current line from the screen
' ****************

	   count=0
	   while len(temp$)>0
		   CheckFor$=left$(temp$,1)
		   while left$(temp$,1)=CheckFor$ and count<63
			   count=count+1
	         temp$=mid$(temp$,2)
		   wend
	      if count=1 then
	         if CheckFor$>chr$(191) then
	            put$ #filhandle,chr$(val("&B11000001"))+CheckFor$
	         else
   	         put$ #filhandle,CheckFor$
	         end if
   	   else
      	   temp2$=bin$(count)
	         while len(temp2$)<6
	            temp2$="0"+temp2$
	         wend
		      put$ #filhandle,chr$(val("&B11"+temp2$))+CheckFor$
	      end if
      	count=0
   	wend
	next t
	put$ #filhandle,chr$(12)'						Start of Palette Marker
	OUT &H3C7, 0
	FOR T = 0 TO 255
		R = (INP(&H3C9) * 65280) \ 16128
		G = (INP(&H3C9) * 65280) \ 16128
		B = (INP(&H3C9) * 65280) \ 16128
	   PUT$ #filhandle,chr$(R)
	   PUT$ #filhandle,chr$(G)
	   PUT$ #filhandle,chr$(B)
	NEXT T
	close #filhandle
END SUB

END

Anyway, if you're looking for some source material that you could possibly adapt, I suggest searching around a bit for the old ABC releases ("All Basic Code"). If I recall correctly, it had a bunch of image file readers & writers over the years, mostly written in QuickBASIC. It might be easier to adapt to Blitz than some of the Java/C code?

ImaginaryHuman

(Posted 2006) [#14]

An IFF/ILBM loader/saver is fairly easy to do, if you want something to practice on.

I agree that some of these formats are hideously complex, I don't think I would even be able to understand all the math involved in jpeg, for example, or to even `get` what all the articles about it are trying to say. Also PNG and GIF are pretty complicated and highly detailed. I still don't understand their compression either.

That's why people sometimes opt for simple ones like PCX, BMP, IFF, etc

Or do a direct raw graphics dump ;-D

xlsior

(Posted 2006) [#15]

That's why people sometimes opt for simple ones like PCX, BMP, IFF, etc

Yup - When I last looked into it, PCX and BMP were the only 'mainstream' formats that looked easy enough to understand for me to tackle...
Just looking at the JPG file format description made my head hurt.

N	(Posted 2006) [#16]

I prefer custom formats. Keeps most people from messing with my stuff.

xlsior

(Posted 2006) [#17]

Custom formats are great when you want to keep your info locked up, (like game graphics, etc.) but they won't do you a whole lot of good when you are creating a paint program for example, where you need to be able to read/write other common file formats to interface with the outside world. Not being able to do so would severely limit the usefulness of such an application.

N	(Posted 2006) [#18]

Custom formats are great when you want to keep your info locked up, (like game graphics, etc.) but they won't do you a whole lot of good when you are creating a paint program for example

Well, this is a game programming language, so it's a fair assumption that one is making a game using it. As for me, I like to make random modules that're neat (mostly so I'm not bored) and then share them.

Anyhow, that's a different subject.

ImaginaryHuman

(Posted 2006) [#19]

A custom file format can be a good idea if your application has features which you want to store that wouldn't be supported by any other formats, such as photoshop saving a file that preserves all the layers and information etc. Also if your application intends to be successful and widely used, a custom format is a good idea as it will then be adopted by other people to support files of that kind. After all, before photoshop there were no photoshop files.

I don't think Blitz is necessarily just a game programming language, I think that's just what it is geared towards and is mostly used for. If it were impossible to write anything other than games in it, I would call it a game language. But it's not exclusive. I am also working on something along the lines of a paint program in BlitzMax.

You could maybe look into using the existing C code of those file loaders and savers, but change them in some way to get access to the data in the way that you want to. Also maybe look around for existing libraries of file-format support that you could translate.

Dreamora

(Posted 2006) [#20]

BM in its core or GUI working is for 3D at least and not 2D because all drawing etc is in 3D. And have requirements with XXMB 3D cards aren't that good for 2D programs normally ;-)
You would need to implement own drivers for GDI+ / WinFX and the Linux / OSX parts to make it really 2D ...

I would call Blitz (not only BM) a multifunctional media language ... there is more in 3D than just games after all and the best commercial showcase apps for b3d for example weren't games at all ...

Hotcakes

(Posted 2006) [#21]

An IFF/ILBM loader/saver is fairly easy to do, if you want something to practice on.

Yeh but that would be reinventing the wheel, now =]

@lucid: Check out the brl.bmploader module. That's the best basic example of byte-reading a picture file in that there could possibly be.