[Rivet] [Fastjet] FastJet "robot" downloads blocked

[Frank Siegert]

I see. I guess it's up to you whether to change it on your server. I
have modified the Rivet bootstrap script such that it always sends the
Accept header now, so for us at least this shouldn't be a problem
anymore.

Thanks for the very quick reply and solution of this issue!

Cheers,
Frank


On 19 April 2013 15:30, Matteo Cacciari <cacciari at lpthe.jussieu.fr> wrote:
> It does indeed work for me too.
>
> The request for the Accept header comes from the security module as an
> extreme caution, I think. Here's what it says:
>
> #
> # Some common HTTP usage patterns are indicative of attacks but may also be
> used by no
> n-browsers for legitimate uses.
> #
> # Do not accept requests without common headers.
> # All normal web browsers include Host, User-Agent and Accept headers.
> # Implies either an attacker or a legitimate automation client.
>
> We may consider deactivating it of course (it wasn't us who put it there,
> but our system administrator. His competence being limited, I think he just
> threw everything he had at it)
>
> M.
>
>
> On 19/04/2013 15:22, Frank Siegert wrote:
>> Hi Gregory,
>>
>> thanks, I can confirm that this is also working for me with this change.
>>
>> I wonder though why the Accept header is necessary: Shouldn't the
>> server assume the client takes any format if it doesn't specify an
>> accept header?
>>
>> Cheers,
>> Frank
>>
>>
>> On 19 April 2013 15:17, Gregory Soyez <soyez at lpthe.jussieu.fr> wrote:
>>> Hi all,
>>>
>>> The solution seems to be to replace
>>>
>>>   hreq = urllib2.urlopen(url)
>>>
>>> by
>>>
>>>   request = urllib2.Request(url, headers={"Accept" : "text/html"})
>>>   hreq = urllib2.urlopen(request)
>>>
>>> as it is indicated e.g. here
>>>
>>> http://stackoverflow.com/questions/385262/how-do-i-send-a-custom-header-with-urllib2-in-a-http-request
>>>
>>> At the very least, it works at my end.
>>>
>>> Cheers,
>>> Gregory
>>>
>> _______________________________________________
>> Fastjet mailing list
>> Fastjet at projects.hepforge.org
>> http://www.hepforge.org/lists/listinfo/fastjet
>>
>