Mid-file restart

Image Ici on parle français ou anglais, comme on veut, comme on peut.
Image Here we speak french or english, as we want, as we can.

Moderator: jza

Forum rules
Image Ici on parle français ou anglais, comme on veut, comme on peut, mais on n'oublie jamais que l'on discute entre humains.
Il ne faut surtout pas oublier les mots magiques: bonjour, salut, merci, etc.
Les messages impolis sont rejetés et restent sans réponse.
Il faut souvent être patient, car il y a plein d'autres activités encore plus passionantes dans notre vie.

Image Here we speak french or english, as we want, as we can, but we never forget that we discuss between humans.
Do not forget the magic words: hello, hi, thank you, etc.
Rude messages are rejected and remain unanswered.
It is often necessary to be patient because there are many other activities even more exciting in our life.
Post Reply
lsoltero
Posts: 9
Joined: 18 Sep 2014 07:16

Mid-file restart

Post by lsoltero » 23 Jan 2016 23:13

Hello when transferring gribs over satellite links (i.e. slow and expensive bandwidth) it is common to have a dropped connection in the middle of a transfer. This is irritating because the user has to restart the transfer of the grib file from the beginning. It would be nice if the transfer could be started where it left off.

the HTTP protocol supports the restarting of data transfers using the "Range:" range header definition in a GET request.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

this means that in theory mid-file restart could be implemented fairly easily. Here is an example of QT code implementing such a restart.
http://stackoverflow.com/questions/2446 ... orkrequest

basically a header entry
Range: bytes=offset
is included in the http header for the GET request specifying the offset to start the transfer.

So... tromping through the code I find the following in Util.cpp
void Util::validArea (QNetworkRequest &request,double x0,double y0,double x1,double y1)
{
int c1 = (int)floor(fabs((17.6*x0+89.23*x1-7.23*y0)*y1));
int c2 = (int)ceil(fabs((37.12*x0-53.2*x1+7.23)*y0*y1));
QString formatrange;
QTextStream(&formatrange)<<c1<<"-"<<c2;
qDebug() << formatrange;
request.setRawHeader ("Range", qPrintable(formatrange));
}

a Range: field is added to the http GET request header!?! The interesting thing is that the format of the Range is strange.

here is a TCP dump of the GET request.
GET /noaa/getzygribfile025.php?but=prepfile&la1=40&la2=17&lo1=-98&lo2=-59&res=2&hrs=6&jrs=3&par=W;P;&rungfs=last&l=a07622b82b18524d2088c9b272bb3feeb0eb1737&m=61c9b2b17db77a27841bbeeabff923448b0f6388&client=zyGrib_mac-7.0.0&tm=151610 HTTP/1.1
TE: deflate,gzip;q=0.3
Keep-Alive: 300
Connection: Keep-Alive, TE
Accept-Encoding: gzip
Accept-Language: en-US,*
Host: https://www.zygrib.org
Range: 73878-11134956
User-Agent: zyGrib_mac/7.0.0

Note that the Range: field is not correctly formatted...

looking closer at the code we note that validArea gets called from makeNetworkRequest
request.setUrl (QUrl(url));
validArea (request,x0,y0,x1,y1);

and makeNetworkRequest is called in FileLoaderGRIB.cpp
QNetworkRequest request = Util::makeNetworkRequest ("http://"+Util::getServerName()+page,-50,80,40,35);

what is interesting is that for wavewatch and gfs data the values passed to validArea are fixed?!?! For MetBlue they are not...

So... it begs the question.. What is this Range: field used for? My guess is that its something left over from an old implementation and no longer used?

Can someone shed light on this?

However the big question is... does the zygrib.org http server cache the requests so that they can be restarted if a valid Range is specified in the header. Has anyone tried?

Might be worth looking into.

Thoughts and comments are appreciated.

Thanks,

--luis

MarkStock
Posts: 1
Joined: 19 Jun 2016 16:24
Contact:

Re: Mid-file restart

Post by MarkStock » 19 Jun 2016 16:32

It would be best to re do it. Complications may start to arise if you wish to continue it at a point where it stopped.

Best Wishes,

mark

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests