Readline and streams

BlitzMax Forums/BlitzMax Programming/Readline and streams

Rambo_Bill

(Posted 2005) [#1]

I can make readline never return depending on what I'm reading. Is this a bug or is there someway to check to see if a line is available?
I've put the code below, the part thats iffy is the part labeled with the big @@@ comment. Note: I have found work arounds, so for me it's not a big deal.

Global WebSock:tSocket ' our lovely socket
Global WebStream:tSocketStream ' our lovely stream
Global TheURL:String=""
Global EndRequest:String="0"

Global SessionCookie:String="" ' cookie for saving web session

Local TXT:String,TXT2:String=""
Connect("www.google.com")
Request("")

TXT = PollData(1000)
While TXT<>""
	TXT2=TXT2+ TXT
	TXT = PollData(1000)
Wend
Print "Closing"
Close()
Print "Done"

	
' connect to server
	Function Connect(sURL:String,iTimeOut:Int=5000)
		TheURL=sURL
		WebSock=CreateTCPSocket()
		ConnectSocket(WebSock,HostIp(sURL),80)
		iTimeOut = iTimeOut + MilliSecs()
		While Not SocketConnected(WebSock) 
			If MilliSecs()>iTimeOut Then
				Return 0 ' failed due to timeout
			End If
		Wend
     ' We are Connected, create our streamer
		WebStream=CreateSocketStream(WebSock,False)
		Return 1 ' success
	End Function
	
' Requests data from web server
	Function Request(sPage:String)
		Local TXT:String
		Local NL:String= Chr(13) + Chr(10)
		EndRequest=""
		' Form a HTTP 1.1 Get Request
		TXT="GET /" + sPage + " HTTP/1.1" + NL
		TXT=TXT + "Host: " + TheURL + NL
		TXT=TXT + "User-Agent: BlitzBrowser" + NL
		TXT=TXT + "Accept: */*" + NL
		'TXT=TXT + "Content-Length: 0" + NL
		If SessionCookie<>"" Then
			TXT = TXT + SessionCookie + NL
		End If
		WriteLine WebStream, TXT
		FlushStream(WebStream)
	End Function
' Poll for incoming data	
	Function PollData:String(sTimeOut:Int=1)
		Local TXT:String="",TXT2:String=""
		sTimeOut =sTimeOut+MilliSecs()
		While sTimeOut>MilliSecs() 
			'==================================
			' @@@ IS THIS A BUG -- SHOULD THIS HANG ?????
			'==================================
			Print "Calling Readline"
			TXT = ReadLine(WebStream)
			Print "Return"
			If Len(TXT)>0 Then TXT2=TXT2 + TXT + Chr(13) + Chr(10)
			' if no cookie 
			If SessionCookie="" Then
					If Instr(TXT,"Set-Cookie",1)>0 Then
						If Not Instr(TXT,"n/a")>0 Then
							SessionCookie=Replace(TXT,"Set-Cookie","Cookie")
						End If
					End If
			End If
		Wend
		Return TXT2
	End Function
' Close Connection
	Function Close()
		SessionCookie=""
		CloseStream(WebStream)
		CloseSocket(WebSock)
	End Function

Tibit

(Posted 2005) [#2]

It might me that no end-of-line is found before the TCP stream is out-of-bytes? Try ReadString so that you can control the length more precise.

Also you can use SocketReadAvail() which returns the number of bytes that can be read. Then read that amount.

Ex:

Global WebSock:tSocket ' our lovely socket
Global WebStream:tSocketStream ' our lovely stream
Global TheURL:String=""
Global EndRequest:String="0"

Global SessionCookie:String="" ' cookie for saving web session

Local TXT:String,TXT2:String=""
Connect("www.google.com")
Request("")

TXT = PollData(1000)
While TXT<>""
	TXT2=TXT2+ TXT
	TXT = PollData(1000)
Wend
Print "Closing"
Close()
Print "Done"

	
' connect to server
	Function Connect(sURL:String,iTimeOut:Int=5000)
		TheURL=sURL
		WebSock=CreateTCPSocket()
		ConnectSocket(WebSock,HostIp(sURL),80)
		iTimeOut = iTimeOut + MilliSecs()
		While Not SocketConnected(WebSock) 
			If MilliSecs()>iTimeOut Then
				Return 0 ' failed due to timeout
			End If
		Wend
     ' We are Connected, create our streamer
		WebStream=CreateSocketStream(WebSock,False)
		Return 1 ' success
	End Function
	
' Requests data from web server
	Function Request(sPage:String)
		Local TXT:String
		Local NL:String= Chr(13) + Chr(10)
		EndRequest=""
		' Form a HTTP 1.1 Get Request
		TXT="GET /" + sPage + " HTTP/1.1" + NL
		TXT=TXT + "Host: " + TheURL + NL
		TXT=TXT + "User-Agent: BlitzBrowser" + NL
		TXT=TXT + "Accept: */*" + NL
		'TXT=TXT + "Content-Length: 0" + NL
		If SessionCookie<>"" Then
			TXT = TXT + SessionCookie + NL
		End If
		WriteLine WebStream, TXT
		FlushStream(WebStream)
	End Function
' Poll for incoming data	
	Function PollData:String(sTimeOut:Int=1)
		Local TXT:String="",TXT2:String=""
		sTimeOut =sTimeOut+MilliSecs()
		While sTimeOut>MilliSecs() 
			'==================================
			' @@@ IS THIS A BUG -- SHOULD THIS HANG ?????
			'==================================

			Length = SocketReadAvail(WebSock)
			If Length > 0
				Print "Calling Readline"
				TXT = ReadString(WebStream,Length)'ReadLine(WebStream)
				
				If Len(TXT)>0 Then TXT2=TXT2 + TXT + Chr(13) + Chr(10)
				' if no cookie 
				If SessionCookie="" Then
						If Instr(TXT,"Set-Cookie",1)>0 Then
							If Not Instr(TXT,"n/a")>0 Then
								SessionCookie=Replace(TXT,"Set-Cookie","Cookie")
						End If
						End If
				End If
			EndIf	
		Wend
		Print "Exit PollData()"
		Return TXT2
	End Function
' Close Connection
	Function Close()
		SessionCookie=""
		CloseStream(WebStream)
		CloseSocket(WebSock)
	End Function

Do you have to use readline? I'm not sure what the result should be, but it does read something from google =) So it works that way.

A code TIP:
TXT = TXT + "Some String"
Can be written as:
TXT:+ "Some String"

Rambo_Bill

(Posted 2005) [#3]

SocketReadAvail() sounds very promising, try it tonight when I get home. Thanks

Rambo_Bill

(Posted 2005) [#4]

SocketReadAvail helped alot. Here is my newest code,but only works with sites that use content-length (Appears most do not)

Global WebSock:tSocket ' our lovely socket
Global WebStream:tSocketStream ' our lovely stream
Global TheURL:String=""
Global SessionCookie:String="" ' cookie for saving web session

Local TXT:String

HTTPConnect("www.asitethatsupportscontentlength.com")
TXT=HTTPRequest("")
Print "Body--------"
Print TXT

Print "Closing"

HTTPClose()
Print "Done"

	
' connect to server
	Function HTTPConnect(sURL:String,iTimeOut:Int=5000)
		TheURL=sURL
		WebSock=CreateTCPSocket()
		ConnectSocket(WebSock,HostIp(sURL),80)
		iTimeOut = iTimeOut + MilliSecs()
		While Not SocketConnected(WebSock) 
			If MilliSecs()>iTimeOut Then
				Return 0 ' failed due to timeout
			End If
		Wend
     ' We are Connected, create our streamer
		WebStream=CreateSocketStream(WebSock,False)
		Return 1 ' success
	End Function
	
' Requests data from web server
	Function HTTPRequest:String(sPage:String,iTimeOut:Int=5000)
		Local TXT:String,A:Int
		Local tBody:String="x" ' body of html
		Local tHeader:String="x" ' header of html
		Local NL:String= Chr(13) + Chr(10)
		Local ContentLength:Int
		' Form a HTTP 1.1 Get Request
		TXT="GET /" + sPage + " HTTP/1.1" + NL
		TXT=TXT + "Host: " + TheURL + NL
		TXT=TXT + "User-Agent: BlitzBrowser" + NL
		TXT=TXT + "Accept: */*" + NL
		'TXT=TXT + "Content-Length: 0" + NL
		If SessionCookie<>"" Then
			TXT = TXT + SessionCookie 
		End If
		WriteLine WebStream, TXT
		FlushStream(WebStream)
		' now read in header grabbing length
		iTimeout=iTimeOut + MilliSecs()
		TXT=""
		Repeat 
			If MilliSecs()>iTimeOut And SocketReadAvail(WebSock)=0 Then Exit
			If SocketReadAvail(WebSock)>0 Then
				If tHeader="x" Then
					' header not complete yet
					TXT=TXT + ReadString(WebStream,SocketReadAvail(WebSock))
					' if end of header has been reached
					A=Instr(TXT,NL+NL,1)
					If A>1 Then 
						 ' grab header, remainder is body
						tHeader=Mid(TXT,1,A+2)
						tBody=Mid(TXT,Len(tHeader)+1)
						' save content length
						A=Instr(tHeader,"Content-Length:",1)
						TXT=Mid(tHeader,A+Len("Content-Length:"))
						TXT=Mid(TXT,1,Instr(TXT,NL,1))
						ContentLength=Int(TXT)
						TXT=""
						' save ASP session cookie
						If Instr(tHeader,"Set-Cookie: ASP",1)
							A=Instr(tHeader,"Set-Cookie: ASP",1)
							TXT=Mid(tHeader,A+Len("Set-Cookie:"))
							TXT=Mid(TXT,1,Instr(TXT,NL,1)-1)
							SessionCookie="Cookie:" + TXT + NL
							TXT=""
						End If
						' in case we get the whole thing in one shot check body length
						If Len(tbody)>=ContentLength Then Exit
					End If
				Else
					tBody=tBody + ReadString(WebStream,SocketReadAvail(WebSock))
					If Len(tbody)>=ContentLength Then Exit
				End If				
			End If
		Forever
		Print "Header-----"
		Print tHeader
		
		'Print "Body-------"
		'Print tBody
		Return tBody
		
	End Function
	
	' Close Connection
	Function HTTPClose()
		SessionCookie=""
		CloseStream(WebStream)
		CloseSocket(WebSock)
	End Function

Tibit

(Posted 2005) [#5]

Cool! Really cool! So now I can see how my browser sees it. Are you doing a real browser or did you have something else in mind suchs as highscore/stats online?

I can't get it to work on sub sites, www.yahoo.com/mail
while yahoo.com works and also sub domains like movies.yahoo.com.

Tibit

(Posted 2005) [#6]

Wait I don't get the full body on yahoo.. guess that's what not works.

Is the code public? Can I use it in TNet for html:highscore?

Rambo_Bill

(Posted 2005) [#7]

Yep, its public domain code, otherwise I wouldn't post it. I am working on an encryption scheme that works with it which I won't share though.

To make it work with yahoo/google or any site that uses the chunk method, you would have to look for chunk-encoding (or something like that) in the header. It looks like when a server is using chunking that it gives you the length of each chunk at the beginning of the body. I don't need to support that for my site, but feel free to modify the code to support it, shouldn't be too hard. I'm going to use this for two way communications between my programs and my server. That's why it was important to maintain a session id. It shall work nicely I think.