wget not getting whole files sometimes

Community Forums/General Help/wget not getting whole files sometimes

Matty(Posted 2013) [#1]
Hi all,

Does anyone know why this would be happening?

The wget command gets a full complete file from an external website using wget -o filepath urlpath some of the time but not all of the time.

Basically we are downloading a series of 4kb files from a weather service to our linux server via a cronjob and sometimes this file is truncated to 1kb and is missing most of the file.

Is there a problem with wget or am I missing something? Is it a security feature?


Derron(Posted 2013) [#2]
Did you consider the remote site hanging up to early ?

This may happen eg. if the site is under heavy load and connections tend to timeout then.
Also the data may get created corrupted already.


Also of interest: do you use "--content-disposition" ? this is a really "new" feature.
What other params do you use?


bye
Ron


Matty(Posted 2013) [#3]
Hi Ron,

The parameters we use are simply

wget -o myfile.extension url

How can we tell the remote site is hanging up early?
We run this by a batch process every few hours during the day.

from Matt


Derron(Posted 2013) [#4]
You can tell by getting corrupt data...

Seriously: during file downloads some servers send the content length (file length) - this is most times the case when you just request an existing file.
When files are getting send to you by a script (script reads bytes, sends bytes...) then you may not get that length info.

You recognize such things when downloading... and the download just says: "x mb transfered" without mentioning the total amount.


Depending on the info you get - you can find out whether it happens on server side or not.


PS: Check other available wget parameters:
-v (more information)
-t X (try to download X times if it failed somehow)


bye
Ron


skidracer(Posted 2013) [#5]
I would try switching from wget to curl.