A faster way to read Files?
BlitzMax Forums/BlitzMax Beginners Area/A faster way to read Files?
| ||
| Mhhh, loading Data that way: Local file=OpenFile(GeoDatpath$) While Not Eof(file) LineofData$=ReadLine(file) Wend CloseStream File took me around 2-3 minutes. If i open the File with an MS Editor it tooks around 3 secs (!). As more lines BMax reads, as slower it gets. How would you get acces to a large chunk of Data? Data are simple x,y,z coords -1.000000,1.000000,2.00000 -1.000000,1.000000,2.00000 Not sure how ReadLine works. Would it be faster to read it from memory (which means that porting the file on HD to a memory bank should be faster then 3 minutes)? Any suggestions? |
| ||
| how big is the file you are reading? this takes 60ms for me:
Local file=OpenFile("c:\test.txt")
start=MilliSecs()
While Not Eof(file)
Local LineofData$ = ReadLine(file)
'Print lineofData$
FlushMem
Wend
Print "Time taken: " + (MilliSecs()-start) + "ms"
CloseFile file
or about 700ms when I uncomment print Sample text: *EDIT* it takes 700ms when run from the IDE using print, it takes 260ms when run from the command prompt. |
| ||
| slightly different test: using 11872 lines of : -1.000000,1.000000,2.00000 (4.1 seconds at command prompt) |
| ||
| Tried your code and 104341 lines took around 102 seconds. My CPU is a good part slower then yours, so at least i know there isnt something wrong with my PC or that BMax behaves different here. Thank you. But 102 seconds? 10 Files like that and you got a 17 minutes loading screen;-) BlitzEditor needs around 1 second to read and print that file... |
| ||
| This takes about 400-500 ms less time (using a bankStream) |
| ||
| But 102 seconds? 10 Files like that and you got a 17 minutes loading screen;-) You wouldn't normally print the value of each vertice to the screen though would you? It takes 123ms when not writing the values out (this includes conversion from string to float), so 10 files would take 1 second to load like that (on my PC). You would probably be better converting the files to a binary format then reading them with ReadFloat. |
| ||
| 6ms at command prompt when reading floats vs 123ms with strings |
| ||
| Great, thnx to your code i found it. It was a missing "Flushmem" in the readline() loop. Now speed is fine. But what data releases Flushmem here? Does every Readline command create a copy of the already red file which should be then released? Think i nearly understand what Flushmem does, but its a little hard to use if you need to understand how an other command exactly works. |
| ||
| Take a look here for a decent explanation of Flushmem (take special notice of the NOTE: section). |
| ||
| Glad i am asked here..didnt know Flushmem has local scope. Thnx:-) |
| ||
| Local file=OpenFile(GeoDatpath$) While Not Eof(file) LineofData$=ReadLine(file) Wend CloseStream File Each time the Readline command is executed, it reads a line from your datafile, puts that line in a string-object and assigns the pointer of that string-object (it's memory-address) to the variable LineofData$, overwriting the previous memory-location. So eventually you'll end up using megabytes of memory, because BMax creates a new string each loop, without deleting/overwriting the old one. The previous line still exists in memory, but the LineofData$ variable is pointing to the new string, leaving the old one in memory, without a reference to it. That's why FlushMem fixes it, it deletes any object (including string-objects) that doesn't have a reference to it. Hope this all makes sense (and that it is correct). |
| ||
| I think he got it already. :) |
| ||
| Just trying to help, as he stated that he "nearly" understands what FlushMem does. |
| ||
| Other poeple will read it too and will find it helpfull. Thnx to both;-) |
   |