Need a faster way of doing this..
Monkey Forums/Monkey Programming/Need a faster way of doing this..
| ||
| So basically the folder I am looking at gets files logged to it, one for every run, as you can imagine the total number of logs does not take long to swell to really big numbers and when this happens my code hang's or appears to hang and most users dont give it a chance to finish so they just alt&F4 and then complain to me that its crashing :) The full filename's contain the date and time they were created and the channel name their logging chat into it, so as an example it might look like this "TheCitadel_20170225_102542.txt" I'm only interested in the most recent file of the given name so as per bellow I loop through the directory listing in reverse and return the first occurrence of the file I am looking for, works a treat but as I said struggles when the folder is packed. Method FindIntel:String(_channelname:String = "TheCitadel") 'DebugStop Local Dir:String[] = LoadDir(Options.logPath) Local total:Int = (Dir.Length - 1) For Local index:Int = total To 0 Step - 1 Local str:String = Dir[index] If str[0 .. _channelname.Length()] = _channelname Then Return Dir[index] Next Return "false" End |
| ||
| Well, you should be able to speed things up straight away by using String.StartsWith() instead of slicing and equality testing. Though I can't be certain that the string slicing is the problem - if LoadDir() is too slow you might want to look at having subfolders (e.g. a separate folder for each channel). You could also look at adding a progress bar or equivalent, so your users will be more patient. Even if you speed things up by a factor of ten, you'll still have problems when the folder is ten times bigger! |
| ||
| As Gerry Quinn suggested, perhaps use sub folders to split things up abit. Perhaps sort your log files into sub folders according to date then each file is sorted in chronological order, you could then use a binary search to traverse through the list. You could look into the Boyer-Moore string search algorithm. |
| ||
| Is there any reason you can't log to a single file? Usually you could just append the content. Then you can open the file and pick the last entry. |
| ||
| Basically what Shinkiro1 said. The fastest method would be to use an index file system when dealing with large amounts of data, but you would have to have a limit to it's size, say a 60 day limit for each entry. The alternative is to go native and use the target systems own file handling system and look for time stamps. |
| ||
| This tool is an intel map for EvE Online an MMO space game full of nerds that like killing each other, the game makes the logs so I have no control over that other than prompting the user to clear his logs if there are to many in the dir. Within this game players have chat channels specifically for giving intel, the game makes a fresh log file per login per channel saving logs uniquely via the time/date as part of the log filename, so I parse the dir, find the first occurrence of say SomeIntelChannel_##timeDate##.txt and then start parsing its lines for new intel, as the game feeds new lines of text into the log I grab it and update a map. https://taiphoz.itch.io/ember-intel more info here if any of you are curious. String.StartsWith(), does this not just slice anyway ? will take a look at it later. thanks for the suggestions guys, its not a massive issue but it is a bit annoying. |
| ||
| I can't be sure of every target, but on desktop StartsWith() will call an efficient C library function that doesn't do any memory allocation, just compares the characters of the two strings directly. |
| ||
| not tried it yet but will deff give it a go , any speedup will be good. |
| ||
| The code translates to: So you won't get much faster than that whatever you do. |
   |