Download a file with redirected URL?

BlitzMax Forums/BlitzMax Programming/Download a file with redirected URL?

Grisu

(Posted 2014) [#1]

Hi everyone!

The download function I have been using for years is broken. I'd like to download files directly from Dropbox, but the original download url is redirected and I so only download the error message below:

302 Found
The resource was found at https://dl.dropboxusercontent.com/u/98035398/prp/prp_stationlist.zip;
you should be redirected automatically.

If I change the url to "https" the download isn't possible as well. But when I copy the exact link into my browser window it works fine?!?
I put up a simple example that shows the issue.

Example:

Const  PRP_UPD_LIST:String          = "prp_stationlist" 
Global ini_update_server:String     = "http://dl.dropboxusercontent.com/u/98035398/prp/" ' <- Default URL TO UPDATE SERVER
'Global ini_update_server:String     = "https://dl.dropboxusercontent.com/u/98035398/prp/" ' <- Default URL TO UPDATE SERVER

If HTTPGetFile:Int(ini_update_server+PRP_UPD_LIST+".zip", PRP_UPD_LIST+".zip") = -1 Then Print "File not found!" Else Print "File downloaded!"

Function HTTPGetFile:Int(url$, filePath$)
	Url$ = url$.Replace("http://", "http::")
	Local bank:TBank=LoadBank(url)
	If bank <> Null Then 
	?macos
        SaveBank bank,CurrentDir()+"/"+filePath$
	?
	?Win32
	   SaveBank bank,CurrentDir()+"\"+filePath$
	?
	?Linux
	   SaveBank bank,CurrentDir()+"/"+filePath$
	?
         Return 1
      Else 
         Return -1
      EndIf
End Function

Thanks in advance.
Grisu

Derron

(Posted 2014) [#2]

You will have to check the response header.

It contains things like "302" - redirects.

In that case you just do another request to the url you read out of the former response.

Think when using something like "curl" this should be done automatically.

bye
Ron

Brucey

(Posted 2014) [#3]

Indeed, libcurl can handle redirects transparently.

Aren't you already using libcurl in PRP ?

BlitzSupport

(Posted 2014) [#4]

I think this code handled redirects properly -- haven't tried it for a while!

' SuperStrict

Function BlitzGet:TBank (url:String)

	If Not url Then Return Null
	
	Local success:Int = False	' File downloaded OK...
	Local done:Int = False		' Exit download loop (for retries, etc)...
	
	Local host:String
	Local file:String

	Local bytestoread:Int
	Local date:String
	Local server:String
	Local contenttype:String
	Local location:String
	Local pos:Int

	Local bank:TBank
	
	Print ""
	Print "Downloading..."
	Print ""
	
	Repeat
	
		If Left (url, 7) = "http://"
			url= Right (url, Len (url) - 7)
		EndIf
		
		Local slash:Int = Instr (url, "/")
	
		If slash
			host = Left (url, slash - 1)
			file = Right (url, Len (url) - slash + 1)
		Else
			host = url
			file = "/"
		EndIf
	
		Local http:TSocket = CreateTCPSocket ()
		
		If http
		
			If ConnectSocket (http, HostIp (host), 80)
				
				Local www:TStream = CreateSocketStream (http)
				
				WriteLine www, "GET " + file + " HTTP/1.1" ' "GET /" gets default page...
				WriteLine www, "Host: " + host
				WriteLine www, "User-Agent: BlitzGet Deluxe"
				WriteLine www, "Accept: */*"
				WriteLine www, ""
	
				Local response:String = ReadLine (www)
		
				Print "Server response: " + response
				
				Local replycode:String
				
				If Left (response, 5) = "HTTP/"
					pos = Instr (response, " ")
					replycode = Mid (response, pos + 1, 3)
				EndIf
	
				Local header:String
	
				Repeat
			
					header = ReadLine (www)
			
					Local reply:String = ""
	
					pos = Instr (header, ": ")
	
					If pos
						reply = Left (header, pos + 1)
					EndIf
			
					Select Lower (reply)
						Case "content-length: "
							bytestoread = Int (Right (header, Len (header) - Len (reply)))
						Case "location: "
							location = Right (header, Len (header) - Len (reply))
					End Select
	
					If header Then Print header ' Skip blank line (if header = "" then nothing is printed)...
									
				Until header$ = "" Or (Eof (www))
	
				Select replycode$
		
					Case "200" ' File found...
		
						Print "Downloading file..."
						
						bank = CreateBank (bytestoread)
						Local save:TStream = CreateBankStream (bank)
	
						If save
					
							Local readwebfile:Int

							' Crude download routine!
							
							CopyBytes www, save, bytestoread
					
							' Fully downloaded, ie. same size?
							
							If FileSize (savepath + savefile) = bytestoread
								success = True
							EndIf
							
							done = True
						
						Else
							Print "Failed to create local file!"
						EndIf
	
					Case "404" ' File Not found...
					
						Print "File not found"
						done = True
					
					Case "301" ' File permanently moved...
					
						url = location

					Case "302" ' File temporarily moved...
			
						url = location
		
					Case "303" ' File moved...
		
						url = location
		
					Case "307" ' Naughty...
		
						url = location
						
				End Select
	
			EndIf
			
			CloseSocket http
			
		EndIf

		' If 'done' is still false, go back to the start with new URL (from '30*' responses)...
		
	Until done
		
	Return bank
	
End Function

Graphics 640, 480

' This URL has worked for years...

Local download:String = "http://www.google.com/images/title_homepage4.gif"			' Google homepage logo

' Test error 404, file not found...
' download:String	 = "http://www.hi-toro.com/mp3/diffusion.mp3"					' Old music (now gone)...

' Test error 303, file moved example (works as of 28 Aug 2009)...
' download:String = "http://www.rentnet.com/" ' Redirects to <a href="http://www.move.com/apartments/main.aspx" target="_blank">http://www.move.com/apartments/main.aspx</a>

Local download_path:String = CurrentDir ()

Local imagebank:TBank = BlitzGet (download)

If imagebank
	
	Print ""
	Print "File downloaded successfully!"
	
	Local image:TImage = LoadImage (imagebank)
	
	Repeat
	
		Cls
		DrawImage image, MouseX (), MouseY ()
		Flip
		
	Until KeyHit (KEY_ESCAPE)
	
	End
	
Else
	Print ""
	Print "Download failed!"
EndIf

Grisu

(Posted 2014) [#5]

Well, the code above has the same issues. It doesn't download the file and "hangs" without giving a response.

Server response: HTTP/1.1 302 FOUND
cache-control: no-cache
Content-Type: text/html; charset=utf-8
Date: Sat, 02 Aug 2014 18:03:56 GMT
location: https://dl.dropboxusercontent.com/u/98035398/prp/prp_stationlist.zip
pragma: no-cache
Server: nginx
set-cookie: flash=; Domain=dropbox.com; expires=Sat, 02 Aug 2014 18:03:56 GMT; Path=/; httponly
set-cookie: bang=; Domain=dropbox.com; expires=Sat, 02 Aug 2014 18:03:56 GMT; Path=/; httponly
set-cookie: uc_session=yyK0PVy4FhRj4b813QNGc5IaKIrepQP06jVPLb5SaDEKQQs8hcI1z0XkMjOQpW7N; Domain=dropboxusercontent.com; Path=/; secure; httponly
Content-Length: 400
Connection: keep-alive

Grisu

(Posted 2014) [#6]

@Brucey: Yes, PRP uses libcurl, but only for the multifile downloads so far.

I tried one of the examples. Also with no luck.

Executing:ex_05.exe
success
*********************************************
Timeout was reached

So my guess is that I need to look for another file webspace... :/

Brucey

(Posted 2014) [#7]

So my guess is that I need to look for another file webspace

There's nothing wrong with dropbox.

That example works fine for me, replacing with your URL above - assuming you are using libcurlssl of course (since it is an https url, which requires SSL support).

I doubt James' example support https.

BlitzSupport

(Posted 2014) [#8]

Yeah, sorry, didn't notice it was https! It works for non-https redirects though.

Grisu

(Posted 2014) [#9]

Do I need both modules libcurlssl and libcurl? Is libssh2 a requirement as well?

I got a "libeay32.dll" error on my system. Installed http://slproweb.com/products/Win32OpenSSL.html to get around it. - Do all users need to install the dll manually? Or will the compiled exe be enough?

Is there any simple example for downloading a file with Libcurlssl? I'm fishing in the dark here.

Brucey

(Posted 2014) [#10]

Do I need both modules libcurlssl and libcurl?

They are the same thing - libcurlssl adds support for SSL certificates and related content. The reason that there are two modules is that SSL support adds extra requirements, such as OpenSSL which lots of people don't need for many uses of curl (like if they only want support for http and ftp, etc).

Is libssh2 a requirement as well?

Only if there's an import for it in the module (which there appears to be). Probably for SFTP support, which is FTP over an SSH tunnel.

Do all users need to install the dll manually?

It's a dll, just include it in the same dir as your exe, as you would with any other third-party DLL. Make sure you include all the required DLLs from that distribution.
Windows apps look first in the local directory for DLLs.

Is there any simple example for downloading a file with Libcurlssl

The examples are essentially the same for both modules. There are some extra examples for libcurlssl, which show usage of SSL-related things.