Proposed AddGadgetItems API (batch add items)

BlitzMax Forums/MaxGUI Module/Proposed AddGadgetItems API (batch add items)

skn3	(Posted 2011) [#1]

I have been playing around with maxgui recently and discovered that it really doesn't perform very well with adding moderatly high amounts of data to gadgets.

I have a standard list adding 1000 items and the delay is pretty bad!

I had a poke around and boiled it down to a maxgui issue. If you look at the win32maxguiex.bmx file you will see:

	Method InsertListItem(index,Text$,tip$,icon,tag:Object)
		
		Local it:LVITEMW
		it=New LVITEMW
		it.mask=LVIF_TEXT|LVIF_DI_SETITEM
		it.iItem=index
		it.pszText=Text.toWString()
		
		'If icon>=0 Then
			it.mask:|LVIF_IMAGE
			it.iImage=icon
		'EndIf
		
		DeSensitize()
		SendMessageW _hwnd,LVM_INSERTITEMW,0,Int Byte Ptr(it)
		SendMessageW _hwnd,LVM_SETCOLUMNWIDTH,0,-2
		If Not IsSingleSelect() Then SelectionChanged()
		Sensitize()
		MemFree it.pszText
		
	EndMethod

Simply by commenting out the lines:

		'DeSensitize()
		SendMessageW _hwnd,LVM_INSERTITEMW,0,Int Byte Ptr(it)
		'SendMessageW _hwnd,LVM_SETCOLUMNWIDTH,0,-2
		'If Not IsSingleSelect() Then SelectionChanged()
		'Sensitize()

It was possible to drastically increase the speed of the list manipulation. Obviously maxgui needs to do its own bits and call those functions which is why I am suggesting that maybe a AddGadgetItems API could be added.

AddGadgetItems would prepare the gadget once; add all of the items in a batch; finalise the gadget. This would avoid all those repeated calls to system/driver that have been commented out above.

skn3	(Posted 2011) [#2]

I have written a stand alone function that seems to work on windows:

Type AddListItemsBatch
	Field items:TList = CreateList()
	Field total:Int
	
	'api
	Method Go(gadget_:TGadget)
		' --- dump all of the items into the gadget ---
		Local link:TLink
		Local Item:TGadgetItem
		
		'do os speciffic bits to setup the operation and confirm we have a valid gadget
		?Win32
			Local it:LVITEMW = New LVITEMW
			
			'convert to windows list box
			Local gadget:TWindowsListBox = TWindowsListBox(gadget_)
			If gadget = Null Return
			
			'get a hwnd
			Local hwnd:Int = QueryGadget(gadget,QUERY_HWND)
		?Not Win32
			'fall back to maxgui behaviour by adding each object individually
			For Item = EachIn items
				AddGadgetItem(gadget_,Item.text,Item.Flags,Item.icon,Item.tip,Item.extra)
			Next
			total = 0
			items.Clear()
			Return
		?
		
		'only proceed if it was infact a valid gadget
		'resize the items array
		Local index:Int = gadget.items.Length
		gadget.items = gadget.items[..index+total]
		
		'do os speciffic gadget preperation
		?Win32
			gadget.DeSensitize()
			SendMessageW(hwnd,WM_SETREDRAW,False,0)
		?
		
		'insert the items into the gadgets items array
		link = items.FirstLink()
		While link
			'move to next link
			Item = TGadgetItem(link.value())
			link = link.NextLink()
			
			'add item to maxgui gadget
			gadget.items[index] = Item
			
			'add item to os
			?Win32
				it.Mask = LVIF_TEXT|LVIF_DI_SETITEM
				it.iItem = index
				it.pszText = Item.text.toWString()
				
				'If item.icon>=0 Then
					it.mask:|LVIF_IMAGE
					it.iImage = Item.icon
				'EndIf
				
				SendMessageW hwnd,LVM_INSERTITEMW,0,Int Byte Ptr(it)
				MemFree it.pszText
			?
			
			'increase index for next item
			index :+ 1
		Wend
		
		'do os speciffic finalisation
		?Win32
			SendMessageW hwnd,LVM_SETCOLUMNWIDTH,0,-2
			SendMessageW(hwnd,WM_SETREDRAW,True,0)
			If Not gadget.IsSingleSelect() Then gadget.SelectionChanged()
			gadget.Sensitize()
		?
		
		'reset the batch
		items.Clear()
		total = 0
	End Method
	
	Method Add(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
		' --- create a gadget item so we use it to add later ---
		'create the new item
		Local Item:TGadgetItem = New TGadgetItem
		Item.Set(text,tip,icon,extra,Flags)
		
		'add it to the batch list
		items.AddLast(Item)
		
		'increase the count
		total :+ 1
	End Method
End Type

I dunno if this breaks anything but it all seems to work fine on my tests. To use it you do:

Local batch:AddListItemsBatch = New AddListItemsBatch
batch.Add("item 1")
batch.Add("item 2")
batch.Add("item 3")
batch.Add("item 4")
batch.Go(listbox)

It only supports the listbox at the moment as I don't need anything else but as a test case here is a comparison.

Maxgui adding 10000 items: took 23.8309994 sec
Batch adding 10000 items: 1.70700002 sec

Obviously there is the argument that your design is broken if your adding 10000 items, but that is just to highlight the drastic difference.

Last edited 2011

Last edited 2011

Grisu

(Posted 2011) [#3]

Hi, thanks for sharing this code.

Can't one just program some sort of "Locklistbox()" function in order to speed up the progress? - From my experience the constant redrawing is what slows the listbox down.

SebHoll

(Posted 2011) [#4]

Love it! :-)

skn3	(Posted 2011) [#5]

I always thought there was a lockgadget up until few days ago, b+?? hmm

LockGadget() would be ideal!

Grisu

(Posted 2011) [#6]

@Seb: Is it possible to do this multi-platform (gui enhancement)?

@skn3: I'm poking for such a function for over 5 years now. :O

Quote myself: http://www.blitzbasic.com/Community/posts.php?topic=60869#679298

Grisu

(Posted 2011) [#7]

Well, I have been using these functions in my little radio player for some weeks now. All works fine so far (Win7 64Bit).

I also see a visual speed benefit while using ~1.200 items: http://188.165.211.46/~knot/prp/prp_win32.zip

skn3	(Posted 2011) [#8]

Excellent news! Can also confirm things working on daily development/usage for past weeks! Hurrah for batch :)

Grisu

(Posted 2011) [#9]

Has anyone tested this under Linux / Mac yet?

skn3	(Posted 2011) [#10]

The code should work fine with mac / linux but if you look in the pre conditional compiling, it is just reverting to the default behaviour...

If you/anyone do a mac one please let me know and i'll update the thread.

Grisu

(Posted 2011) [#11]

I found a tiny "glitch" under Windows:
The listbox the items are added to displays a horizontal slider for a second when the gadget is updated. Even if the items don't need one (= item length is fine). - I remember Seb had that glitch in the original MaxGui as well. Not sure how he fixed this.

skn3	(Posted 2011) [#12]

Hmmmm I havnt noticed this? I'll have a look tomorrow when back on pc. What version of windows?

Grisu

(Posted 2011) [#13]

System Windows 7 (64Bit) and XP (32Bit)

I have put up a simple example program:

SuperStrict 
Import MaxGui.Drivers

Const MAINWINDOW_W:Int = 278 
Const MAINWINDOW_H:Int = 154
Const MAXITEMS:Int=2000
Global window:TGadget=CreateWindow("My Lisboxwindow",30,20,MAINWINDOW_W,MAINWINDOW_H+100,Null,WINDOW_TITLEBAR|WINDOW_CENTER)
Global listbox:TGadget=CreateListBox(2,2,MAINWINDOW_W-6,MAINWINDOW_H,window)
Global Button1:TGadget=CreateButton("Normal Add Items "+MAXITEMS,2,MAINWINDOW_H+2,MAINWINDOW_W-6,20,window,BUTTON_OK)
Global Button2:TGadget=CreateButton("Batch Add Items "+MAXITEMS,2,MAINWINDOW_H+25,MAINWINDOW_W-6,20,window,BUTTON_OK)
Global ListboxStrip:TIconStrip  = LoadIconStrip(LoadPixmapPNG("temp.png")) 

While WaitEvent()
	Select EventID()
		Case EVENT_WINDOWCLOSE
			End
		Case EVENT_GADGETACTION
   	 	      Select EventSource() 
                 Case BUTTON1
			          Add_Listboxitems()
                 Case BUTTON2
			          Add_Listboxitems_Batch()
                 End Select 
	End Select
Wend

Function Add_Listboxitems_Batch()
Local batch:AddListItemsBatch = New AddListItemsBatch
Local i:Int    
 
ClearGadgetItems (listbox)

For i:Int=0 To MAXITEMS 
batch.Add "item "+i,GADGETITEM_NORMAL
Next 
batch.Go(listbox)

End Function 


Function Add_Listboxitems()
Local i:Int     
ClearGadgetItems (listbox)
SetGadgetIconStrip(listbox, Null)
For i:Int=0 To MAXITEMS
 AddGadgetItem listbox,"item "+i,GADGETITEM_NORMAL,-1
Next 
End Function 

' New AddListItem type too speed up the process! EXPERIMENTAL!
Type AddListItemsBatch
	Field items:TList = CreateList()
	Field total:Int
	
	'api
	Method Go(gadget_:TGadget)
		' --- dump all of the items into the gadget ---
		Local link:TLink
		Local Item:TGadgetItem
		
		'do os speciffic bits to setup the operation and confirm we have a valid gadget
		?Win32
			Local it:LVITEMW = New LVITEMW
			
			'convert to windows list box
			Local gadget:TWindowsListBox = TWindowsListBox(gadget_)
			If gadget = Null Return
			
			'get a hwnd
			Local hwnd:Int = QueryGadget(gadget,QUERY_HWND)
		?Not Win32
			'fall back to maxgui behaviour by adding each object individually
			For Item = EachIn items
				AddGadgetItem(gadget_,Item.text,Item.Flags,Item.icon,Item.tip,Item.extra)
			Next
			total = 0
			items.Clear()
			Return
		?
		
		'only proceed if it was infact a valid gadget
		'resize the items array
		Local index:Int = gadget.items.Length
		gadget.items = gadget.items[..index+total]
		
		'do os speciffic gadget preperation
		?Win32
			gadget.DeSensitize()
		?
		
		'insert the items into the gadgets items array
		link = items.FirstLink()
		While link
			'move to next link
			Item = TGadgetItem(link.value())
			link = link.NextLink()
			
			'add item to maxgui gadget
			gadget.items[index] = Item
			
			'add item to os
			?Win32
				it.Mask = LVIF_TEXT|LVIF_DI_SETITEM
				it.iItem = index
				it.pszText = Item.text.toWString()
				
				'If item.icon>=0 Then
					it.mask:|LVIF_IMAGE
					it.iImage = Item.icon
				'EndIf
				
				SendMessageW hwnd,LVM_INSERTITEMW,0,Int Byte Ptr(it)
				MemFree it.pszText
			?
			
			'increase index for next item
			index :+ 1
		Wend
		
		'do os speciffic finalisation
		?Win32
			SendMessageW hwnd,LVM_SETCOLUMNWIDTH,0,-2
			If Not gadget.IsSingleSelect() Then gadget.SelectionChanged()
			gadget.Sensitize()
		?
		
		'reset the batch
		items.Clear()
		total = 0
	End Method
	
	Method Add(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
		' --- create a gadget item so we use it to add later ---
		'create the new item
		Local Item:TGadgetItem = New TGadgetItem
		Item.Set(text,tip,icon,extra,Flags)
		
		'add it to the batch list
		items.AddLast(Item)
		
		'increase the count
		total :+ 1
	End Method
End Type

1, Run Program
2. When you press the button add batch items you should see the issue.
But only the first time! After that it works fine. At least in this example.

Thanks for your help!

skn3	(Posted 2011) [#14]

OH AWESOME! After years of using win api calls I never knew windows had a built in ability to disable redraw for a hwnd!

SuperStrict 
Import MaxGui.Drivers

Const MAINWINDOW_W:Int = 278 
Const MAINWINDOW_H:Int = 154
Const MAXITEMS:Int=2000
Global window:TGadget=CreateWindow("My Lisboxwindow",30,20,MAINWINDOW_W,MAINWINDOW_H+100,Null,WINDOW_TITLEBAR|WINDOW_CENTER)
Global listbox:TGadget=CreateListBox(2,2,MAINWINDOW_W-6,MAINWINDOW_H,window)
Global Button1:TGadget=CreateButton("Normal Add Items "+MAXITEMS,2,MAINWINDOW_H+2,MAINWINDOW_W-6,20,window,BUTTON_OK)
Global Button2:TGadget=CreateButton("Batch Add Items "+MAXITEMS,2,MAINWINDOW_H+25,MAINWINDOW_W-6,20,window,BUTTON_OK)
Global ListboxStrip:TIconStrip  = LoadIconStrip(LoadPixmapPNG("temp.png")) 

While WaitEvent()
	Select EventID()
		Case EVENT_WINDOWCLOSE
			End
		Case EVENT_GADGETACTION
   	 	      Select EventSource() 
                 Case BUTTON1
			          Add_Listboxitems()
                 Case BUTTON2
			          Add_Listboxitems_Batch()
                 End Select 
	End Select
Wend

Function Add_Listboxitems_Batch()
Local batch:AddListItemsBatch = New AddListItemsBatch
Local i:Int    
 
ClearGadgetItems (listbox)

For i:Int=0 To MAXITEMS 
batch.Add "item "+i,GADGETITEM_NORMAL
Next 
batch.Go(listbox)

End Function 


Function Add_Listboxitems()
Local i:Int     
ClearGadgetItems (listbox)
SetGadgetIconStrip(listbox, Null)
For i:Int=0 To MAXITEMS
 AddGadgetItem listbox,"item "+i,GADGETITEM_NORMAL,-1
Next 
End Function 

' New AddListItem type too speed up the process! EXPERIMENTAL!
Type AddListItemsBatch
	Field items:TList = CreateList()
	Field total:Int
	
	'api
	Method Go(gadget_:TGadget)
		' --- dump all of the items into the gadget ---
		Local link:TLink
		Local Item:TGadgetItem
		
		'do os speciffic bits to setup the operation and confirm we have a valid gadget
		?Win32
			Local it:LVITEMW = New LVITEMW
			
			'convert to windows list box
			Local gadget:TWindowsListBox = TWindowsListBox(gadget_)
			If gadget = Null Return
			
			'get a hwnd
			Local hwnd:Int = QueryGadget(gadget,QUERY_HWND)
		?Not Win32
			'fall back to maxgui behaviour by adding each object individually
			For Item = EachIn items
				AddGadgetItem(gadget_,Item.text,Item.Flags,Item.icon,Item.tip,Item.extra)
			Next
			total = 0
			items.Clear()
			Return
		?
		
		'only proceed if it was infact a valid gadget
		'resize the items array
		Local index:Int = gadget.items.Length
		gadget.items = gadget.items[..index+total]
		
		'do os speciffic gadget preperation
		?Win32
			gadget.DeSensitize()
			SendMessageW(hwnd,WM_SETREDRAW,False,0)
		?
		
		'insert the items into the gadgets items array
		link = items.FirstLink()
		While link
			'move to next link
			Item = TGadgetItem(link.value())
			link = link.NextLink()
			
			'add item to maxgui gadget
			gadget.items[index] = Item
			
			'add item to os
			?Win32
				it.Mask = LVIF_TEXT|LVIF_DI_SETITEM
				it.iItem = index
				it.pszText = Item.text.toWString()
				
				'If item.icon>=0 Then
					it.mask:|LVIF_IMAGE
					it.iImage = Item.icon
				'EndIf
				
				SendMessageW hwnd,LVM_INSERTITEMW,0,Int Byte Ptr(it)
				MemFree it.pszText
			?
			
			'increase index for next item
			index :+ 1
		Wend
		
		'do os speciffic finalisation
		?Win32
			SendMessageW hwnd,LVM_SETCOLUMNWIDTH,0,-2
			SendMessageW(hwnd,WM_SETREDRAW,True,0)
			If Not gadget.IsSingleSelect() Then gadget.SelectionChanged()
			gadget.Sensitize()
		?
		
		'reset the batch
		items.Clear()
		total = 0
	End Method
	
	Method Add(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
		' --- create a gadget item so we use it to add later ---
		'create the new item
		Local Item:TGadgetItem = New TGadgetItem
		Item.Set(text,tip,icon,extra,Flags)
		
		'add it to the batch list
		items.AddLast(Item)
		
		'increase the count
		total :+ 1
	End Method
End Type

See lines 89 and 124 for the changes. That should solve it, I will update the original post at the top too.

Last edited 2011

Last edited 2011

Last edited 2011

Grisu

(Posted 2011) [#15]

Works fine under xp and win7 now.

Great job!

P.S. Is Seb still working on "MaxGui2"?

skn3	(Posted 2011) [#16]

No idea about the maxgui2 thing! It would be cool if he was but Id imagine his time is fairly limited!

NoOdle

(Posted 2012) [#17]

great work, your modifications from this post and others should be added to the MaxGUI release. Very handy!

[edit] couldn't find anything else on Mac OSX disable redrawing, only your question on Stackoverflow

Last edited 2012

skn3	(Posted 2012) [#18]

"only your question on Stackoverflow" haha yes indeed ;)

Grisu

(Posted 2014) [#19]

Warning!!! Zombie thread approaching... :)

I modified the code add bit, so I can create a list with the items sorted "upwards" or "downwards":

	Method AddLast(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
		' --- create a gadget item so we use it to add later ---
		'create the new item
		Local Item:TGadgetItem = New TGadgetItem
		Item.Set(text,tip,icon,extra,Flags)
		
		'add it to the batch at the very end 
		items.AddFirst(Item)
		
		'increase the count
		total :+ 1
	End Method

	Method AddFirst(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
		' --- create a gadget item so we use it to add later ---
		'create the new item
		Local Item:TGadgetItem = New TGadgetItem
		Item.Set(text,tip,icon,extra,Flags)
		
		'add it to the batch list at the very start
		items.AddFirst(Item)
		
		'increase the count
		total :+ 1
	End Method

I'm looking for a fast way to generate a random list of items. I.e. each item that is added will be placed at a random spot of list.

I could use Modifygadgetitem() after the list has been created and switching each item of the list afterwards (= ugly and slow).

Does someone know how to do this while the list is generated and using the functions above? I need it to work with larger lists (2.000+ items with icons and tooltips).

Henri

(Posted 2014) [#20]

Hello Grisu,

here is a one way of doing it:

SuperStrict

Rem
Random list insertation (almost)
EndRem

Global list:TList = New TList

SeedRnd MilliSecs()

'Simulate x number of insertations to listbox
For Local i:Int = 1 To 100
	Addrandom(String(i))
Next

'Main function
Function AddRandom(obj:Object)

	Local link:TLink = list.firstlink()
	If Not link Then
		list.addlast(obj)
		Return
	EndIf
	
	Local count:Int = Rand(0,list.count())
	Local notEven:Int = count Mod 3
	For Local i:Int = 0 Until count
		If link.nextlink() Then
			link = link.nextlink()
		Else
			Exit
		EndIf
	Next
	
	If Not notEven Then
		list.insertbeforelink(obj,link)
	Else
		list.insertafterlink(obj,link)
	EndIf
EndFunction

' Display results
For Local s:String = EachIn list
	Print s
Next

EDIT: Or more in context (not tested):

EDIT2: NOTE: Need to randomize seed before first entry ( SeedRnd MilliSecs() )

Method AddRandom(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
	' --- create a gadget item so we use it to add later ---
	'create the new item
	Local Item:TGadgetItem = New TGadgetItem
	Item.Set(text,tip,icon,extra,Flags)
	
	'add it to the batch list
	Local link:TLink = items.firstlink()
	If Not link Then
		items.addlast(item)
		total :+ 1
		Return
	EndIf
	Local count:Int = Rand(0,items.count())
	Local notEven:Int = count Mod 3
	For Local i:Int = 0 Until count
		If link.nextlink() Then
			link = link.nextlink()
		Else
			Exit
		EndIf
	Next
	If Not notEven Then
		items.insertbeforelink(item,link)
	Else
		items.insertafterlink(item,link)
	EndIf
	
	'increase the count
	total :+ 1
End Method

-Henri

Grisu

(Posted 2014) [#21]

Hey,
thanks for that demonstration code.

As I needed to get this to work with Skn3's batch functions above, I chose a different approach in the end:

1. I create a temporary TList and add all items into it.
2. I pick a random item from that list and remove it afterwards.
3. I look up the random item from my database and add it to the listbox.
4. I repeat steps 2 + 3 until all items have been randomized.
5. I call GoBatch() to make the new listbox visible to the user.

Ideas to speed the process up are welcome.

Here comes my ugly example code: (from the main app)

 	     Local Random_list:TList=CreateList()
 	     Local it:trecord
	     Local tmp_str:String
	     Local counter:Int
	
		 ' Create a temp list with all station filenames present, so we can select a random one out of it later
         For it:trecord=EachIn MapValues(trecord.map_Station)
             ListAddLast Random_list, Upper(it.filename)
         Next  

         Local Max_items:Int=CountList(Random_list)
         Local Rnd_item:Int

         While Max_items>0 
 
         Rnd_item:Int=Rand(1,Max_items)
		 counter:Int=1

  		For tmp_str:String = EachIn Random_list
            If counter=Rnd_item Then 
             ListRemove( Random_list:TList, tmp_str ) 
	         Max_items:Int=Max_items-1
             'Print Tmp_str+" found! "+counter+" Max: "+Max_items
             Exit 
            EndIf                         
			counter=counter+1          
    	Next
                 
		it:trecord=trecord(MapValueForKey(trecord.map_Station,tmp_str)) ' Grab Record according to filename
            If it Then
?Not Linux	     
          		If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,ConvertGenreToString(it.genre), it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),ConvertGenreToString(it.genre), it.filename
?
?Linux	     
          		If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,ConvertGenreToString(it.genre)+" ("+Get_Rating_text(it.rating)+")", it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),ConvertGenreToString(it.genre)+" ("+Get_Rating_text(it.rating)+")", it.filename
?
		      it=Null

            EndIf

        Wend ' Until no items left in random_list

Henri

(Posted 2014) [#22]

Does your solution work at satisfactory speed ? It's hard to test when you are relying on other things not visible here.

-Henri

Grisu

(Posted 2014) [#23]

It works, but for me it's quite slow (~30ms) and I'm on a high end PC. I have no idea how it will run on lower end systems. So I'm worried users may be disappointed.

I created a standalone app. So the results can be checked. Please select "Randomise List" to generate a new list and post your results (non-debugmode).

Download Source: http://www.mediafire.com/download/lsr5n28rsbb20yg/RandomTestV3.zip

|Edit| Degac's changes from below are now included!

degac

(Posted 2014) [#24]

Just tested your source code on my computer (AMD PhenomII X4 955)
At startup
debug on: 38-54ms
debug off: 13ms

When clicked on random mode

debug off: 132ms - 190ms

degac

(Posted 2014) [#25]

Some questions about your source code.

1. why do you use ReadFile? It seems useless as you use LoadText(). If you want to check if file is present, just use FileType()

Local k:String[] = LoadText("prp.dat").split("~n")

2. your record is already 'fixed size'... just replace with this

	If l<>""
	cols=L.split("|")
	cols=cols[..5]
	trecord.Add(cols[0],cols[1],Byte(cols[2]),Byte(cols[3]),Byte(cols[4]))
		
	End If

in release mode now I get 11 ms (min was 8ms!) (at loading!)

Same in function ConvertGenreToString: just use an array string... the Select..Case statement is not needed here (and no call to a function also if Index are bounded at loading time.

genre:String[]=["","50s, 60s & 70s","80s & 90s","Alternative","Ambient & Chill","Anime & Asian"

Grisu

(Posted 2014) [#26]

Hi Christian,

thanks a lot for the input! Will test the changes.

Please select "Randomised List" from the Combobox. This creates the random list and should be slower. I get ~10ms for the "Normal list" generation.

P.S. Ahm, for the readfile, don't I need this or Openfile() in order to use LoadText()?

degac

(Posted 2014) [#27]

Looking at your source code I think it's better to use LINK.Remove() instead of ListRemove()

 Local Max_items:Int=CountList(Random_list)
 Local Rnd_item:Int
 local link_item:TLink=Random_list.FirstLink() 

 While Max_items>0 
    Rnd_item:Int=Rand(1,Max_items)
    counter:Int=1
       For tmp_str:String = EachIn Random_list
     	 If counter=Rnd_item Then 
         If link_item=Null link_item=random_list.lastlink()
	 'ListRemove( Random_list:TList, tmp_str ) 
	 link_item.remove()
	 Max_items:Int=Max_items-1
         Exit 
        EndIf                         
	counter=counter+1          
    Next

Now (in release) I got 107-109ms every 'random' call

degac

(Posted 2014) [#28]

Hi
yes I posted and then edited my results! Sorry.
I realized that you mean the 'reload time' when you choose 'randomised list' afted I posted!

About LoadText()

If you look at the source code you'll see that LoadText 'open' a stream... so it's a double work.

Grisu

(Posted 2014) [#29]

Added all your changes and reuploaded the source code (see initial post).

The Link.remove is much faster than the old code! My delay dropped from ~30ms to ~20ms.

Btw: Is "(Casuale)" the right translation for "(Random)" in Italian?. I need this new phrase for the selection box. :)

degac

(Posted 2014) [#30]

Yes, the translation is correct.

Grisu

(Posted 2014) [#31]

Is there a way to access the temporary item index of the listbox before it's passed to the station ListBox itself via BatchGo()?

I think it would be faster to:
1. Go through the whole station map item by item and...
- Count the temorary list batch of items
- Insert the new item to a random place (1..max present)
3. Call BatchGo() to move the item array to the station list.

This way there is no need for the additional temporary TList (overhead).

Grisu

(Posted 2014) [#32]

Ok, well, the code creates double entries when the list gets randomised... :/

Still trying to figure out why...

degac

(Posted 2014) [#33]

Just found this http://www.blitzbasic.com/Community/posts.php?topic=69137 about scrambling an array... this is source code changed

SuperStrict 
' Little TestApp to measure the time taken to randomise a list. 

Framework Maxgui.Drivers
	Import maxgui.xpmanifest
	Import Maxgui.ProxyGadgets
    Import BRL.Filesystem
	Import Brl.EventQueue
	Import Brl.Timer     
	Import Brl.Pixmap  
	Import BRL.Retro
	Import BRL.Random
	Import BRL.D3D9Max2D
	Import BRL.GLGraphics
	Import BRL.GLMax2D

	
Const MAINWINDOW_W:Int = 278 
Const MAINWINDOW_H:Int = 254
Const Buttonsize: Int = 32
Const BUTTON_ACTION:Int = EVENT_GADGETACTION

Global batch:AddListItemsBatch = New AddListItemsBatch ' for newAddgadgetitems functions

Global genre:String[]=["","50s, 60s & 70s","80s & 90s","Alternative","Ambient & Chill","Anime & Asian","Black, Rap & HipHop","Blues & Love","Classical","Club","Children","Country","Dance & Pop", "Electronic","Folk & Latino","Gothic","Hits","Jazz & Soul","Reggae","Retro","Rock & Metal","Talk","Schlager","Soundtrack","Season"]

Local Style:Int = WINDOW_TITLEBAR | WINDOW_CLIENTCOORDS | WINDOW_CENTER | WINDOW_ACCEPTFILES | WINDOW_RESIZABLE | WINDOW_HIDDEN

Global MainWindow:TGadget  = CreateWindow("Radomise TestApp V3" , 0 , 0 , MAINWINDOW_W , MAINWINDOW_H , Null , Style)
SetMinWindowSize(MainWindow,MAINWINDOW_W,MAINWINDOW_H)
SetMaxWindowSize(MainWindow,MAINWINDOW_W,DesktopHeight())

Global StationPanel:TGadget = CreatePanel(0, 0, MAINWINDOW_W, MAINWINDOW_H, MainWindow, PANEL_ACTIVE)', PANEL_ACTIVE)
SetGadgetLayout StationPanel, EDGE_ALIGNED, EDGE_ALIGNED, EDGE_ALIGNED, EDGE_ALIGNED

Global StationListbox:TGadget  = CreateListBox(-1,-1,MAINWINDOW_W+2, MAINWINDOW_H-BUTTONSIZE-3, StationPanel) 
SetGadgetLayout StationListbox, EDGE_ALIGNED, EDGE_ALIGNED, EDGE_ALIGNED, EDGE_ALIGNED

Global Station_Filterbox:TGadget=CreateComboBox(MAINWINDOW_W-1-BUTTONSIZE*8-4,MAINWINDOW_H-30,118+30+4,28, StationPanel,COMBOBOX_EDITABLE) 
SetGadgetLayout Station_Filterbox, Null, Null, Null, EDGE_ALIGNED

AddGadgetItem station_filterbox,"Normal List", GADGETITEM_NORMAL, 0
AddGadgetItem station_filterbox,"Randomised List", GADGETITEM_NORMAL, 0
AddGadgetItem station_filterbox,"Randomised 2", GADGETITEM_NORMAL, 0


Global station_genre_filter:Int=0 'select all
SelectGadgetItem station_filterbox,0

Global StationListbox_Genre:Int=0

' load data
Load_DB_Indexfile()
Local cnt:Int
For Local k:String=EachIn MapKeys(trecord.map_station)
	cnt:+1
Next

Print "Items: "+trecord.array_station.length+" K: "+cnt


' fill default list
PopulateList(StationListbox_Genre)

SeedRnd MilliSecs() ' get a random seed...

ShowGadget (StationPanel)
ShowGadget (MainWindow)

Repeat ' Main Loop
	Select WaitEvent()
	Case EVENT_WINDOWCLOSE, EVENT_APPTERMINATE
			End

	
	Case EVENT_GADGETSELECT
					Select EventSource()
						Case stationlistbox
							Print "ListBox: "+EventData()
							If EventData()>-1
								
								Local sx:String=String(GadgetItemExtra(StationListBox,EventData()))
								
								Print "SX: "+sx
							
							
							End If
					
					
					End Select



     End Select 
 



	Select EventSource()
    Case Station_Filterbox 
	 Select EventID()
 	            Case EVENT_GADGETACTION
                  StationListbox_Genre:Int=SelectedGadgetItem(station_filterbox)
                  If EventData() > -1 Then ' Genre has been selected from list
                      PopulateList(StationListbox_Genre)
                  EndIf 

       End Select
     End Select 

Forever
End ' of the world ;)


' New AddListItem type too speed up the process! EXPERIMENTAL!
Type AddListItemsBatch
	Field items:TList = CreateList()
	Field total:Int
	'api
	Method Go(gadget_:TGadget)
		' --- dump all of the items into the gadget ---
		Local link:TLink
		Local Item:TGadgetItem
		
		'do os speciffic bits to setup the operation and confirm we have a valid gadget
		?Not Win32
			'fall back to maxgui behaviour by adding each object individually
			For Item = EachIn items
				AddGadgetItem(gadget_,Item.text,Item.Flags,Item.icon,Item.tip,Item.extra)
			Next
			total = 0
			items.Clear()
			Return
		?
		?Win32
			Local it:LVITEMW = New LVITEMW
			
			'convert to windows list box
			Local gadget:TWindowsListBox = TWindowsListBox(gadget_)
			If gadget = Null Return
			
			'get a hwnd
			Local hwnd:Int = QueryGadget(gadget,QUERY_HWND)
		?
				
		'do os speciffic gadget preperation
		?Win32
			'only proceed if it was infact a valid gadget
			'resize the items array
			Local index:Int = gadget.items.Length
			gadget.items = gadget.items[..index+total]

			gadget.DeSensitize()
			SendMessageW(hwnd,WM_SETREDRAW,False,0)



			'insert the items into the gadgets items array
			link = items.FirstLink()
			While link
				'move to next link
				Item = TGadgetItem(link.value())
				link = link.NextLink()
				
				'add item to maxgui gadget
				gadget.items[index] = Item
				
				'add item to os
'				?Win32
					it.Mask = LVIF_TEXT|LVIF_DI_SETITEM
					it.iItem = index
					it.pszText = Item.text.toWString()
					
					'If item.icon>=0 Then
						it.mask:|LVIF_IMAGE
						it.iImage = Item.icon
					'EndIf
					
					SendMessageW hwnd,LVM_INSERTITEMW,0,Int Byte Ptr(it)
					MemFree it.pszText
'				?
				
				'increase index for next item
				index :+ 1
			Wend

		?

		
		'do os speciffic finalisation
		?Win32
			SendMessageW hwnd,LVM_SETCOLUMNWIDTH,0,-2
			SendMessageW(hwnd,WM_SETREDRAW,True,0)
			If Not gadget.IsSingleSelect() Then gadget.SelectionChanged()
			gadget.Sensitize()
			
		'reset the batch
		items.Clear()
		total = 0		
		?
	
	End Method
	
	Method Add(text:String,Flags:Int=0,icon:Int=-1,tip:String="",extra:Object=Null)
		' --- create a gadget item so we use it to add later ---
		'create the new item
		Local Item:TGadgetItem = New TGadgetItem
		Item.Set(text,tip,icon,extra,Flags)
		
		'add it to the batch list
		items.AddLast(Item)
		'items.AddFirst(Item)
		
		'increase the count
		total :+ 1
	End Method

End Type

Function Load_DB_Indexfile()
Local starttime:Int=MilliSecs()
Local Endtime:Int

Local pos:Int
'Local file:TStream=OpenFile("prp.dat")
Local l:String

Local n:String 
Local sta:String
Local g:Int
Local r:Int
Local fav:Int 

	Print "Loading in Indexfile..."
	Local line_counter:Int=0
	Local k:String[] = LoadText("prp.dat").split("~n")
    Local cols:String[] 
  
	For l:String = EachIn k
      If l <> "" Then
  
		cols=L.split("|")
		cols=cols[..5]
		trecord.Add(cols[0],cols[1],Byte(cols[2]),Byte(cols[3]),Byte(cols[4]))
	
     'Print n$ + "|" + sta$ + "|" + g + "|" + r + "|" + fav
     'trecord.Add(n$ , sta$ , g , r , fav)

   EndIf 
     
Next 

Endtime:Int = MilliSecs()
'Print "Start = " + Starttime
'Print "End   = " + Endtime 
Print "Delta at Start= " + (Endtime - Starttime)+ " ms"

End Function

Type trecord
	Global map_station:tmap=CreateMap() 'it store each station information -> search by name


'	Field name$,filename$,genre:Int,rating:Int,fav:Int,recent:Int,url:String,max_server:Int,logo:TRamstream,server_list:String[]
	Field name$,filename$,genre:Byte,rating:Byte,fav:Byte', url:String,max_server:Int',server_list:String[]
	
	Global array_Station:Trecord[]
	Global ID_counter:Int
	
	
	Function Add:trecord(_na$,_fna$,_ge:Byte=0,_ra:Byte=0,_fav:Byte=0)
'	Function Add:trecord(_na$,_fna$,_ge:Int=0,_ra:Int=0,_fav:Int=0,_rec:Int=0,_url:String="",_max_server:Int,_server_list:String[])
'	Function Add:trecord(_na$,_fna$,_ge:Int=0,_ra:Int=0,_fav:Int=0,_rec:Int=0,_url:String="",_max_server:Int,_logo:TRamstream,_server_list:String[])
          ' genre (one of 24 so far) 
	      ' rating (0-5) -> |||||
	      ' tag (0 or 1) -> fav or not	
	      ' recent station?                
		Local st:trecord,st1:Trecord
		If _fna="" Return Null 'No name, no station to store	
		
  	    st=trecord(MapValueForKey(map_Station,Upper(_fna)))'the key is the Station's filename! no duplicates allowed
		If st=Null'create a new one
			st=New trecord
			st.name=_na
            st.filename=_fna
			st.genre=_ge
			st.rating=_ra
			st.fav=_fav
			'st.recent=_rec
 			'st.url=_url
			'st.max_server=_max_server
                 'st.logo=_logo
			'st.server_list=_server_list			
			MapInsert map_Station,Upper(_fna),st

			array_station=array_station[..ID_counter+1]
			array_station[ID_counter]=st
			ID_counter:+1
			
			Return st			
		End If
		
	End Function
	
	Function Clear:trecord()
		If Not MapIsEmpty( map_Station:TMap ) Then ClearMap (map_Station)
	End Function    

End Type


Function PopulateList(mode:Int=0)
Local EndTime:Int 
Local StartTime:Int=MilliSecs() 
Print "--------------------------------------"
Print "Start: "+starttime 

ClearGadgetItems (StationListbox)

Select mode  

	Case 0 ' All Genre      
        For Local it:trecord=EachIn MapValues(trecord.map_Station)
?Not Linux
          If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,Genre[it.genre], it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),Genre[it.genre], it.filename
?
?Linux
          If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,Genre[it.genre]+" ("+Get_Rating_text(it.rating)+")", it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),Genre[it.genre]+" ("+Get_Rating_text(it.rating)+")", it.filename
?
        Next 

	Case 2	'random with array

		Shuffle(trecord.array_station)
		
		For Local it:Trecord=EachIn trecord.array_station
			If it
?Not Linux	     
          		If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,Genre[it.genre], it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),Genre[it.genre], it.filename
?
?Linux	     
          		If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,Genre[it.genre]+" ("+Get_Rating_text(it.rating)+")", it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),Genre[it.genre]+" ("+Get_Rating_text(it.rating)+")", it.filename
?

			End If
		Next
		

		
		

	Case 1 ' by Random *NEW*

 	     Local Random_list:TList=CreateList()
 	     Local it:trecord
	     Local tmp_str:String
	     Local counter:Int
	
		 ' Create a temp list with all station filenames present, so we can select a random one out of it later
         For it:trecord=EachIn MapValues(trecord.map_Station)
             ListAddLast Random_list, Upper(it.filename)
         Next  

 Local Max_items:Int=CountList(Random_list)
 Local Rnd_item:Int
 Local link_item:TLink=Random_list.FirstLink() 

 While Max_items>0 
    Rnd_item:Int=Rand(1,Max_items)
    counter:Int=1
       For tmp_str:String = EachIn Random_list
     	 If counter=Rnd_item Then 
         If link_item=Null link_item=random_list.lastlink()
		 'ListRemove( Random_list:TList, tmp_str ) 
	 	link_item.remove()
		 Max_items:Int=Max_items-1
         Exit 
        EndIf                         
	counter=counter+1          
    Next
                 
		it:trecord=trecord(MapValueForKey(trecord.map_Station,tmp_str)) ' Grab Record according to filename
            If it Then
?Not Linux	     
          		If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,Genre[it.genre], it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),Genre[it.genre], it.filename
?
?Linux	     
          		If it.fav=0 Then batch.Add it.name,GADGETITEM_NORMAL,it.rating,Genre[it.genre]+" ("+Get_Rating_text(it.rating)+")", it.filename Else batch.Add it.name,GADGETITEM_NORMAL,(it.rating+6),Genre[it.genre]+" ("+Get_Rating_text(it.rating)+")", it.filename
?
		      it=Null

            EndIf

        Wend ' Until no items left in random_list
'        Print Max_items         

End Select ' = mode

batch.Go(StationListbox)

ShowGadget (StationListbox)
ActivateGadget StationListbox

EndTime:Int=MilliSecs() 
Print "End: "+starttime 
Print "Time Taken: "+(EndTime-starttime)+" ms" 
Print "--------------------------------------"

End Function



Function Shuffle(arr:Object[] Var, topindex:Int = -1, fragmentation:Float = 1.0)

	SeedRnd MilliSecs()

	If (topindex < 0) Or (topindex > arr.length - 1) Then topindex = arr.length - 1
		
	For Local n:Int = 1 To Int(topindex * fragmentation)
		
		Local i1:Int = Rand(0, topindex)
		Local i2:Int = Rand(0, topindex)
		
		Local temp:Object = arr[i1]
		arr[i1] = arr[i2]
		arr[i2] = temp
	
	Next
	
EndFunction

The 'core' is that function

Function Shuffle(arr:Object[] Var, topindex:Int = -1, fragmentation:Float = 1.0)

	SeedRnd MilliSecs()

	If (topindex < 0) Or (topindex > arr.length - 1) Then topindex = arr.length - 1
		
	For Local n:Int = 1 To Int(topindex * fragmentation)
		
		Local i1:Int = Rand(0, topindex)
		Local i2:Int = Rand(0, topindex)
		
		Local temp:Object = arr[i1]
		arr[i1] = arr[i2]
		arr[i2] = temp
	
	Next
	
EndFunction

It's damned faster!

Grisu

(Posted 2014) [#34]

I searched the forums for code before posting here, but I never used the term "shuffle". ;)

Apart from the little overhead, this is nearly as fast as the normal list creation for me.

To test the code under real-life conditions, I created a PRP Build: (deleted)

Works fine so far. Let me know, if you see any screen tearing. Grazie!

degac

(Posted 2014) [#35]

Well... on my dad's computer (PhenomX6 WinXP) debug off, original random 27ms, new one 6ms. To me seems quite fast :P

Just downloaded your latest PRP. Everything seems to work perfectly.
No tearing, no interruptions of sort.

Grisu

(Posted 2014) [#36]

Thanks for the feedback.

After using the test version for several hours. I found a bug. :)

1. Extract the zip to a fresh folder.
2. Start the app and select the favourite station list.
3. Select "Random" list.
4. Delete one station from this station list. -> "Right mouse click" -> "Delete station".
5. Select the "Random" list again.
6. There will be more (double) items inside the station list then before.

Will get some sleep and start fresh tomorrow.

degac

(Posted 2014) [#37]

If you are using my latest code with array I suppose I found the problem.
When you delete a station you need to
. delete the array station
. rebuild it again

Otherwise that station will be present.
And putting the array entry to null I don't know what other problems can raise

Grisu

(Posted 2014) [#38]

This "should" work now... (deleted)

I was think, instead of rebuilding the array station, can't I just remove the one item that got deleted? This should be much faster. Can I shrink the array by the one item that was deleted?

Example:

			
' Remove a station
MapRemove map_Station,Upper(_fna),st
			
'Not sure how to delete an item here
array_station=array_station[..ID_counter-1 
array_station[ID_counter]=st

ID_counter:-1

			
' Add a station
MapInsert map_Station,Upper(_fna),st
			
array_station=array_station[..ID_counter+1]
array_station[ID_counter]=st
ID_counter:+1

degac

(Posted 2014) [#39]

Sure... but 1st example I think is wrong. You should do something like this

SuperStrict
Local arr:String[20]
For Local k:Int=0 Until 20
	arr[k]="Item "+k
Next
'remove 10th item (it's 'Item 9')
arr=arr[..9]+arr[10..]
For Local k:Int=0 Until arr.length
	Print arr[k]
Next

Grisu

(Posted 2014) [#40]

Thanks again!

I found another method. At least it should be a bit faster than the old code.

Last update for the weekend: http://www.mediafire.com/download/v8w91drqm96h53f/PRP_Test4.zip - I'm going to stress test the new build more next week.