HTMLView - Save Web Page

BlitzMax Forums/MaxGUI Module/HTMLView - Save Web Page

Chalky

(Posted 2015) [#1]

I am trying to capture and save to disk the HTML source of a webpage displayed in an HTMLView. I searched the forum and found this:

Strict 

Import maxgui.win32maxguiex

Local window:TGadget
Local htmlview:TGadget

window=CreateWindow("My Window",30,20,600,440,,15|WINDOW_ACCEPTFILES)

htmlview=CreateHTMLView(0,0,ClientWidth(window),ClientHeight(window),window,HTMLVIEW_NONAVIGATE)
SetGadgetLayout htmlview,1,1,1,1 

HtmlViewGo htmlview,"www.blitzmax.com"

While WaitEvent()
	Print CurrentEvent.ToString()
	Select EventID()
		Case EVENT_WINDOWCLOSE
			End
		Case EVENT_GADGETACTION
			DebugLog EventText()
		Case EVENT_GADGETDONE
			HtmlViewRun(htmlview,"window.location=~q/body=~q+document.body.outerHTML")
	End Select
Wend

which was posted by skidracer 7 years ago (I changed Notify to Debuglog) in a response to a thread with the title "HTMLView - how to get source". However, when I run the code, I get this:

GadgetDone: data=0, mods=0, x=0, y=0, extra=""
GadgetAction: data=0, mods=0, x=0, y=0, extra="about:/body=<BODY></BODY>"
GadgetAction: data=0, mods=0, x=0, y=0, extra="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fwww.facebook.com/BlitzResearchLtd&layout=button_count&show_faces=true&width=160&action=like&font=arial&colorscheme=light&height=21"

which is not the page HTML source.

The online manual for HTMLViewRun has no information about the "window." or "document." parameters - can anyone tell me where I can find details about their format and possible usage?

markcw

(Posted 2015) [#2]

What you can do is get the path to the file with HtmlViewCurrentUrl. Then you use brl.filesystem and brl.stream commands to parse the source for what you want and save. Saving stuff is very simple, look at the WriteFile example in the docs. For reading stuff look at the OpenFile, etc commands. You should use OpenFile when parsing Http otherwise OpenStream. You can use ReadString to get the whole file contents (with FileSize) or ReadLine for one line at a time until Eof.

Chalky

(Posted 2015) [#3]

Thanks for taking the time to reply munch. I already know how to read/write files (I'm a programmer by profession) and am aware of the need to use OpenFile (rather than OpenStream).

I'm not sure how HtmlViewCurrentUrl will help as it surely returns the website url (running it displayed "http://www.blitzmax.com/" - which I knew anyway as I supplied it via the HtmlViewGo statement)?

The reason I need to be able to access the HTML source code is that no remote file exists at the time of access as the page is "auto-generated" at the remote end depending on parameters submitted via a form. My intention was therefore to use an HtmlView to access the page, then save the source so that I could parse the saved file.

I was hoping that there was some documentation somewhere on how to use the "window.xxxxx" and "document.xxxxx" parameters with an HtmlView. Maybe none exists (other than that I found on the MDN site, which I am unsure how to utlilise within BlitzMax)?

grable

(Posted 2015) [#4]

In short, you cant. There is no way to get anything out of the htmlview without resorting to hacks (like putting it in the location bar).

That is unless you access the raw control itself...

?Win32
Extern
Type IHTMLElement Extends IDispatch
	Method setAttribute:Int()
	Method getAttribute:Int()
	Method removeAttribute:Int()
	Method put_className:Int()
	Method get_className:Int()
	Method put_id:Int()
	Method get_id:Int()
	Method get_tagName:Int()
	Method get_parentElement:Int()
	Method get_style:Int()
	Method put_onhelp:Int()
	Method get_onhelp:Int()
	Method put_onclick:Int()
	Method get_onclick:Int()
	Method put_ondblclick:Int()
	Method get_ondblclick:Int()
	Method put_onkeydown:Int()
	Method get_onkeydown:Int()
	Method put_onkeyup:Int()
	Method get_onkeyup:Int()
	Method put_onkeypress:Int()
	Method get_onkeypress:Int()
	Method put_onmouseout:Int()
	Method get_onmouseout:Int()
	Method put_onmouseover:Int()
	Method get_onmouseover:Int()
	Method put_onmousemove:Int()
	Method get_onmousemove:Int()
	Method put_onmousedown:Int()
	Method get_onmousedown:Int()
	Method put_onmouseup:Int()
	Method get_onmouseup:Int()
	Method get_document:Int()
	Method put_title:Int()
	Method get_title:Int()
	Method put_language:Int()
	Method get_language:Int()
	Method put_onselectstart:Int() 
	Method get_onselectstart:Int()
	Method scrollIntoView:Int()
	Method contains:Int()
	Method get_source3Index:Int()
	Method get_recordNumber:Int()
	Method put_lang:Int()
	Method get_lang:Int()
	Method get_offsetLeft:Int()
	Method get_offsetTop:Int()
	Method get_offsetWidth:Int()
	Method get_offsetHeight:Int()
	Method get_offsetParent:Int()
	Method put_innerHTML:Int( bstr:Short Ptr Ptr)
	Method get_innerHTML:Int( bstr:Short Ptr Ptr)
	Method put_innerText:Int( bstr:Short Ptr Ptr)
	Method get_innerText:Int( bstr:Short Ptr Ptr)
	Method put_outerHTML:Int( bstr:Short Ptr Ptr)
	Method get_outerHTML:Int( bstr:Short Ptr Ptr)
	Method put_outerText:Int( bstr:Short Ptr Ptr)
	Method get_outerText:Int( bstr:Short Ptr Ptr)
	Method insertAdjacentHTML:Int()
	Method insertAdjacentText:Int()
	Method get_parentTextEdit:Int()
	Method isTextEdit:Int()
	Method click:Int()
	Method get_filters:Int()
	Method put_ondragstart:Int()
	Method get_ondragstart:Int()
	Method toString:Int() 
	Method put_onbeforeupdate:Int()
	Method get_onbeforeupdate:Int()
	Method put_onafterupdate:Int()
	Method get_onafterupdate:Int()
	Method put_onerrorupdate:Int()
	Method get_onerrorupdate:Int()
	Method put_onrowexit:Int()
	Method get_onrowexit:Int()
	Method put_onrowenter:Int()
	Method get_onrowenter:Int()
	Method put_ondatasetchanged:Int()
	Method get_ondatasetchanged:Int()
	Method put_ondataavailable:Int()
	Method get_ondataavailable:Int()
	Method put_ondatasetcinokete:Int()
	Method get_ondatasetcinokete:Int()
	Method put_onfilterchange:Int()
	Method get_onfilterchange:Int()
	Method get_children:Int()
	Method get_all:Int()
EndType
EndExtern
?

Function GetHTMLViewBody:String( g:TGadget, full:Int = False)
?Win32
	Local htmlview:TWindowsHTMLView = TWindowsHTMLView(g)
	If Not htmlview Then RuntimeError "not a html-view"
	
	Local disp:IDispatch
	Local doc:IHTMLDOCUMENT2
	
	Assert htmlview.browser
	If htmlview.browser.lfget_Document( Varptr disp) Then RuntimeError "no document interface"
	If disp.QueryInterface( htmlview.IID_IHTMLDocument2, Varptr doc) Then RuntimeError "no document2 interface"

	Local bstr:Short Ptr
	Local e:IHTMLElement
	Local p:IHTMLElement
		
	doc.get_body( Varptr e)
	If Not e Then Return Null	
	
	If full Then
		e.get_parentElement( Varptr p)
		If p Then
			p.get_outerHTML(Varptr bstr)
			p.Release_()
		EndIf
	EndIf
	
	If Not bstr Then
		e.get_outerHTML(Varptr bstr)
	EndIf
	e.Release_()
	
	If bstr Then
		Local s:String = String.FromWString(bstr)
		SysFreeString(bstr)
		Return s
	EndIf
?
	Return Null	
EndFunction

Note: that IHTMLElement is incomplete, since only outerHTML and get_parentElement is used here.

edit: I noticed that getting the actual HTML element (not just the BODY) doesnt work correctly..