[Rivet] [Fastjet] FastJet "robot" downloads blocked

[Matteo Cacciari]

Thank you Frank.

Here you are:

[Fri Apr 19 14:58:51 2013] [error] [client 131.169.104.214] ModSecurity:
Access denied with code 403 (phase 2). Operator EQ matched 0 at
REQUEST_HEADERS. [file
"/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"]
[line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept
Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.6"] [maturity "9"]
[accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"]
[tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname
"fastjet.fr"] [uri "/repo/fastjet-3.0.3.tar.gz"] [unique_id
"UXE-i4adCgEAAAFi99IAAAAP"]

It's indeed our security module blocking it. We can discuss later what the
best solution may be.

Matteo



On 19/04/2013 15:01, Frank Siegert wrote:
> Hi again,
> 
> I'm attaching a short script that demonstrates what our bootstrap
> script is doing. I have just "successfully" tested it -- i.e. I got a
> 403 error from the server:
> 
> $ ./testfj.py
> Traceback (most recent call last):
>   File "./testfj.py", line 8, in <module>
>     hreq = urllib2.urlopen(url)
>   File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
>     return _opener.open(url, data, timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 406, in open
>     response = meth(req, response)
>   File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
>     'http', request, response, code, msg, hdrs)
>   File "/usr/lib/python2.7/urllib2.py", line 444, in error
>     return self._call_chain(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
>     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
> urllib2.HTTPError: HTTP Error 403: Forbidden
> 
> My IP address in this case was 131.169.104.214.
> 
> Hope that helps,
> Frank
> 
> 
> On 19 April 2013 14:13, Matteo Cacciari <cacciari at lpthe.jussieu.fr> wrote:
>> Follow up: can you please give me the IP from which you are running your
>> script, so that I can look into the logs?
>>
>> Matteo
>>
>>
>> On 19/04/2013 14:03, Matteo Cacciari wrote:
>>> Hi Andy.
>>>
>>> As far as I remember we are not doing eny explicit blocking. I don't even
>>> think we have a robots.txt file there.
>>>
>>> We'll look into it asap. Thank you for the mail.
>>>
>>> Matteo
>>>
>>>
>>> On 19/04/2013 12:42, Andy Buckley wrote:
>>>> Hi Gregory, Gavin, et al,
>>>>
>>>> We've noticed recently that the FastJet website blocks "robot" downloads
>>>> of the FastJet tarball, e.g. http://fastjet.fr/repo/fastjet-3.0.3.tar.gz
>>>>
>>>> Unfortunately this means that the Rivet bootstrap script can fail if it
>>>> tries to download and build FastJet, rather than using the LCG installed
>>>> copy from AFS. We're using Python's urllib2 to do the fetching... is
>>>> there anything weor you can do to not fall foul of this blocking? (I'm
>>>> not sure if urllib2 automatically respects robots.txt files, but if you
>>>> want to give us a special unblocked User-Agent name to use, I'm sure we
>>>> can manage to update our script accordingly)
>>>>
>>>> Thanks!
>>>> Andy & co
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Fastjet mailing list
>>> Fastjet at projects.hepforge.org
>>> http://www.hepforge.org/lists/listinfo/fastjet
>>>
>>
>>
>> _______________________________________________
>> Rivet mailing list
>> Rivet at projects.hepforge.org
>> http://www.hepforge.org/lists/listinfo/rivet

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2945 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://www.hepforge.org/lists-archive/rivet/attachments/20130419/56929170/attachment.bin>