wget not getting whole files sometimes
Community Forums/General Help/wget not getting whole files sometimes
| ||
Hi all, Does anyone know why this would be happening? The wget command gets a full complete file from an external website using wget -o filepath urlpath some of the time but not all of the time. Basically we are downloading a series of 4kb files from a weather service to our linux server via a cronjob and sometimes this file is truncated to 1kb and is missing most of the file. Is there a problem with wget or am I missing something? Is it a security feature? |
| ||
Did you consider the remote site hanging up to early ? This may happen eg. if the site is under heavy load and connections tend to timeout then. Also the data may get created corrupted already. Also of interest: do you use "--content-disposition" ? this is a really "new" feature. What other params do you use? bye Ron |
| ||
Hi Ron, The parameters we use are simply wget -o myfile.extension url How can we tell the remote site is hanging up early? We run this by a batch process every few hours during the day. from Matt |
| ||
You can tell by getting corrupt data... Seriously: during file downloads some servers send the content length (file length) - this is most times the case when you just request an existing file. When files are getting send to you by a script (script reads bytes, sends bytes...) then you may not get that length info. You recognize such things when downloading... and the download just says: "x mb transfered" without mentioning the total amount. Depending on the info you get - you can find out whether it happens on server side or not. PS: Check other available wget parameters: -v (more information) -t X (try to download X times if it failed somehow) bye Ron |
| ||
I would try switching from wget to curl. |