[Rivet] ruvet and osx

[David Grellscheid]

You've diagnosed it right. All programs in /usr/bin are considered protected, and will not be passed any DYLD environment. A python in /usr/local or /opt will work fine

Also, making the script executable does not help. The first line with the #! still calls either /usr/bin/env or /usr/bin/python with exactly the same issue

  David (on house move teabreak)




On 30 July 2018 16:17:18 BST, James William Monk <james.william.monk at cern.ch> wrote:
>Hi everyone,
>
>Did we ever come across the problem that the mac System Integrity
>Protection (SIP) does not pass on the DYLD_LIBRARY_PATH to a subprocess
>if it thinks that subprocess is a “protected” process? I believe that
>is the problem here, and is preventing the library path being set
>correctly during the Rivet build because it spawns a python subprocess
>that does not inherit the correct environment
>
>It seems that if I do a python script to simply print the environment,
>i.e.
>
>	import os
>	for key in os.environ.keys():
>	    print key
>
>then execute
>
>	foo=“bar” python test
>(where test contains the script above)
>
>Then I see that foo is detected in the environment.  However, if I do
>
>	DYLD_LIBRARY_PATH=“bar” python test
>
>Then DYLD_LIBRARY_PATH does not show up! Presumably SIP has decided
>that the python subprocess must be protected from malicious setting of
>the environment.  I can see why they might think that is important,
>since changing the path would potentially cause it to load a bad
>version of some library.  I can disable SIP to confirm this, but that
>requires a reboot of my mac and is not something we should ask users to
>do.
>
>I think that the fix for this is that the mk-analysis-html script has
>to be executable.  That way it will inherit from the build environment,
>not from the python subprocess whose environment has been wiped clean. 
>
>
>cheers,
>
>James
>
>
>
>
>
>
>> On 30 Jul 2018, at 16:36, James William Monk
><james.william.monk at cern.ch> wrote:
>> 
>> Hi Sabah,
>> 
>> There is a bug in the build system for the docs in Rivet 2.6 on mac. 
>This issue has occurred before, but I only just remembered it.  It
>looks like David Grellscheid has tried to fix it in the past, but when
>I use the updated Makefile.am from the doc dir, I still get the same
>problem.  I knew there was a reason I was still on Rivet 2.5.4 :)
>> 
>> It looks to me like there is a mistake in the setup for the
>environment needed for the python script mk-analysis-html.  At least
>one error is that it seems to have the PYTHONPATH pointing to the
>installation prefix, but there is nothing there of course because Rivet
>hasn’t been installed yet.  Another error is that DYLD_LIBRARY_PATH is
>not set.  I tested this by modifying doc/mk-analysis-html by adding
>> 
>> print "Hello from mk-analysis-html.  I think $PYTHONPATH = " +
>os.environ['PYTHONPATH']
>> print "environ keys are"
>> for key in os.environ.keys():
>>   print key
>> 
>> This problem is only related to the building of the documentation,
>but disabling doxygen doesn’t seem to help.  I *think* if you plough on
>regardless and do make install, you will get Rivet installed in your
>prefix, just without the docs and the rivetenv.sh script.
>> 
>> cheers,
>> 
>> James
>> 
>>> On 28 Jul 2018, at 13:11, Sabah Salih <sabah.salih at manchester.ac.uk>
>wrote:
>>> 
>>> Dear James,
>>>          I am very grateful for your kind help
>>> I tried to apply rivet-bootstrap content line by line via command
>line
>>> 
>>> YODA, HEPMC,... are fine
>>> When I reached Rivet I get the following:
>>> I attached the file which have the complete steps for RIVET
>>> 
>>> Many Thanks, Sabah.
>>> 
>>> 
>>> 
>>> File
>"/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py",
>line 12, in <module>
>>>    from rivet.core import *
>>> ImportError:
>dlopen(/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so,
>10): Library not loaded: /tmp/rivet/local/lib/libRivet.dylib
>>>  Referenced from:
>/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>>  Reason: image not found
>>> make[1]: *** [analyses.json] Error 1
>>> make[1]: *** Waiting for unfinished jobs....
>>> Using output file name 'analyses.html' and directory 'analyses'
>>> Traceback (most recent call last):
>>>  File "mk-analysis-html", line 25, in <module>
>>>    import rivet
>>>  File
>"/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py",
>line 12, in <module>
>>>    from rivet.core import *
>>> ImportError:
>dlopen(/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so,
>10): Library not loaded: /tmp/rivet/local/lib/libRivet.dylib
>>>  Referenced from:
>/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>>  Reason: image not found
>>> make[1]: *** [analyses.html] Error 1
>>> make: *** [all-recursive] Error 1
>>> 
>>> ls -lh
>/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/
>>> total 312
>>> -rw-r--r--  1 admin  wheel   351B  3 Oct  2017 __init__.py
>>> -rw-r--r--  1 admin  wheel   598B 28 Jul 11:26 __init__.pyc
>>> -rw-r--r--  1 admin  wheel   3.0K  6 Nov  2017 aopaths.py
>>> -rwxr-xr-x  1 admin  wheel   121K 28 Jul 11:15 core.so
>>> -rw-r--r--  1 admin  wheel   7.8K  3 Oct  2017 plotinfo.py
>>> -rw-r--r--  1 admin  wheel   2.5K  3 Oct  2017 spiresbib.py
>>> -rw-r--r--  1 admin  wheel   6.9K 20 Nov  2017 util.py
>>> 
>>> 
>>> Many Thanks, Sabah.
>>> 
>>> From Sabah Salih
>>> Particle Physics Group,
>>> The School of Physics and Astronomy,
>>> The University of Manchester,
>>> Schuster Laboratory,
>>> Brunswick Street,
>>> Manchester M13 9PL.
>>> Tel: +44 1612754171 or  x4171
>>> E-mail: sabah.salih at manchester.ac.uk
>>> 
>>> ________________________________________
>>> From: James William Monk [james.william.monk at cern.ch]
>>> Sent: 25 July 2018 22:46
>>> To: Sabah Salih
>>> Cc: James William Monk; Andy Buckley; rivet at projects.hepforge.org
>>> Subject: Re: [Rivet] ruvet and osx
>>> 
>>> Hi Sabah,
>>> 
>>> That error is just confirming that the Rivet lib is not present, I
>presume because the build and/or installation failed.  Do you see any
>other error that looks like a build error?  I’m not very familiar with
>the bootstrap, so I can’t say whether it produces any build logs, or
>where they would be, but we need to see the build failure message to
>know why the libraries weren't built properly.
>>> 
>>> cheers,
>>> 
>>> James
>>> 
>>> 
>>> 
>>>> On 25 Jul 2018, at 19:09, Sabah Salih
><sabah.salih at manchester.ac.uk> wrote:
>>>> 
>>>> Dear James,
>>>>      Thank you so much.
>>>>   Following your advice I did the following:
>>>> 
>>>> edit rivet-bootstrap
>>>> 
>>>> and changed
>>>> 
>>>> function conf { ./configure --prefix=$INSTALL_PREFIX "$@"; }
>>>> to
>>>> function conf { ./configure CC=/usr/bin/clang CXX=/usr/bin/clang++
>--prefix=$INSTALL_PREFIX "$@"; }
>>>> 
>>>> and
>>>> ./configure  --prefix=$INSTALL_PREFIX
>>>> to
>>>> ./configure CC=/usr/bin/clang CXX=/usr/bin/clang++
>--prefix=$INSTALL_PREFIX
>>>> 
>>>> When I type
>>>> ./rivet-bootstrap
>>>> 
>>>> I get this
>>>> File
>"/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py",
>line 12, in <module>
>>>>   from rivet.core import *
>>>> ImportError:
>dlopen(/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so,
>10): Library not loaded:
>/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/local/lib/libRivet.dylib
>>>> Referenced from:
>/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>>> Reason: image not found
>>>> make[1]: *** [analyses.json] Error 1
>>>> make[1]: *** Waiting for unfinished jobs....
>>>> Using output file name 'analyses.html' and directory 'analyses'
>>>> Traceback (most recent call last):
>>>> File "mk-analysis-html", line 25, in <module>
>>>>   import rivet
>>>> File
>"/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py",
>line 12, in <module>
>>>>   from rivet.core import *
>>>> ImportError:
>dlopen(/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so,
>10): Library not loaded:
>/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/local/lib/libRivet.dylib
>>>> Referenced from:
>/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>>> Reason: image not found
>>>> make[1]: *** [analyses.html] Error 1
>>>> make: *** [all-recursive] Error 1
>>>> 
>>>> ls local/lib/libRivet.dylib
>>>> ls: local/lib/libRivet.dylib: No such file or directory
>>>> 
>>>> Many Thanks, Sabah.
>>>> 
>>>> 
>>>> From Sabah Salih
>>>> Particle Physics Group,
>>>> The School of Physics and Astronomy,
>>>> The University of Manchester,
>>>> Schuster Laboratory,
>>>> Brunswick Street,
>>>> Manchester M13 9PL.
>>>> Tel: +44 1612754171 or  x4171
>>>> E-mail: sabah.salih at manchester.ac.uk
>>>> 
>>>> ________________________________________
>>>> From: James William Monk [james.william.monk at cern.ch]
>>>> Sent: 25 July 2018 13:57
>>>> To: Sabah Salih
>>>> Cc: James William Monk; Andy Buckley; rivet at projects.hepforge.org
>>>> Subject: Re: [Rivet] ruvet and osx
>>>> 
>>>> Hi Sabah,
>>>> 
>>>> No, there’s no need to cross compile here, we just need to make
>sure it uses compilers and libs all from the same toolchain and doesn’t
>mix and match system tools with homebrew tools.  For example, to try to
>use the system compiler you could try something like configuring Rivet
>and its dependencies with the extra config option
>>>> 
>>>>       ./configure <usual config options> CC=/usr/bin/clang
>CXX=/usr/bin/clang++
>>>> 
>>>> It looks like you prefer to use homebrew’s tools though.  I don’t
>know where they are located on your system, but the equivalent
>configure command would be
>>>> 
>>>>       ./configure <usual config opts> CC=/path/to/brew/gcc
>CXX=/path/to/brew/g++
>>>> 
>>>> where you fill in the paths to the brew installation versions of
>gcc and g++.  Of course, any Rivet dependencies (FastJet, GSL,
>HepMC...) will also need to be configured in the same way first.
>>>> 
>>>> If you just want to run Rivet, as opposed to develop for it, I’d
>also suggest taking a look at the Docker distribution for mac:
>>>> 
>>>>       https://rivet.hepforge.org/trac/wiki/Docker
>>>> 
>>>> cheers,
>>>> 
>>>> James
>>>> 
>>>> 
>>>>> On 25 Jul 2018, at 08:37, Sabah Salih
><sabah.salih at manchester.ac.uk> wrote:
>>>>> 
>>>>> Dear James,
>>>>>       It is so nice to hear from you.
>>>>> Thank you so much for your kind reply.
>>>>> 
>>>>> I just done
>>>>> 
>>>>> brew tap osx-cross/arm
>>>>> brew tap osx-cross/avr
>>>>> 
>>>>> brew install arm-gcc-bin
>>>>> 
>>>>> brew install avr-gcc
>>>>> This look like installing gcc8
>>>>> 
>>>>> Is this what you meant?
>>>>> 
>>>>> Many Tanks, Sabah.
>>>>> 
>>>>> From Sabah Salih
>>>>> Particle Physics Group,
>>>>> The School of Physics and Astronomy,
>>>>> The University of Manchester,
>>>>> Schuster Laboratory,
>>>>> Brunswick Street,
>>>>> Manchester M13 9PL.
>>>>> Tel: +44 1612754171 or  x4171
>>>>> E-mail: sabah.salih at manchester.ac.uk
>>>>> 
>>>>> ________________________________________
>>>>> From: James William Monk [james.william.monk at cern.ch]
>>>>> Sent: 25 July 2018 01:10
>>>>> To: Andy Buckley; Sabah Salih
>>>>> Cc: rivet at projects.hepforge.org
>>>>> Subject: Re: [Rivet] ruvet and osx
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> From a bit of nurdling about on various fora, that error appears
>to refer to an option that is present for clang, but not gcc.  Clang is
>the supplied default on mac.  You have installed a separate gcc
>toolchain via homebrew.  What I can imagine may be happening is that
>autotools is detecting the system clang and adding options
>appropriately, but then the g++ command is now calling the gcc from
>homebrew.  For example, on my own mac I see
>>>>> 
>>>>>      g++ --version
>>>>>      Configured with:
>--prefix=/Applications/Xcode.app/Contents/Developer/usr
>--with-gxx-include-dir=/usr/include/c++/4.2.1
>>>>>      Apple LLVM version 9.1.0 (clang-902.0.39.2)
>>>>>      Target: x86_64-apple-darwin17.7.0
>>>>> 
>>>>> It’s possible you may be able to cure this with some appropriate
>environment settings like
>>>>> 
>>>>>      export CC=/usr/bin/clang
>>>>>      export CXX=/usr/bin/clang++
>>>>> 
>>>>> Which would try to force it to use the system clang compiler
>consistently.  Alternatively, try to point it to your homebrew compiler
>location.  You may also like to consider removing homebrew as another
>solution - I don’t know what else it adds for you, but personally I
>have never found those kinds of packaging systems helpful on a mac
>precisely because they bring a second toolchain into play that is not
>always entirely consistent with the system toolchain.
>>>>> 
>>>>> cheers,
>>>>> 
>>>>> James
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 25 Jul 2018, at 00:48, Andy Buckley <andy.buckley at cern.ch>
>wrote:
>>>>>> 
>>>>>> I'm afraid I don't use Macs, so am not sure what is best. Using
>an earlier compiler might work; I'm a little surprised that autoreconf
>did not help. Hopefully someone else on the Rivet team has a bit more
>Mac experience and can help out.
>>>>>> 
>>>>>> Andy
>>>>>> 
>>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research
>Fellow
>>>>>> Particle Physics Experiment Group, University of Glasgow
>>>>>> On Jul 24 2018, at 4:52 pm, Sabah Salih
><sabah.salih at manchester.ac.uk> wrote:
>>>>>> 
>>>>>> 
>>>>>> Dear Andy,
>>>>>>        Thank you ever so much. The problem look like it is in
>YODA.
>>>>>> I did
>>>>>> 
>>>>>> cd YODA-1.7.0
>>>>>> 
>>>>>> autoreconf -i
>>>>>> make
>>>>>> 
>>>>>> that did not fix it
>>>>>> 
>>>>>> cd ..
>>>>>> this is where rivet-bootstrap
>>>>>> 
>>>>>> and rerun
>>>>>> 
>>>>>> ./rivet-bootstrap
>>>>>> 
>>>>>> This did not fix it neither.
>>>>>> 
>>>>>> What is the best way to get rivet to work in a mac please.?
>>>>>> It does not have to be gcc8
>>>>>> 
>>>>>> Many thanks, Sabah
>>>>>> 
>>>>>> 
>>>>>> From Sabah Salih
>>>>>> Particle Physics Group,
>>>>>> The School of Physics and Astronomy,
>>>>>> The University of Manchester,
>>>>>> Schuster Laboratory,
>>>>>> Brunswick Street,
>>>>>> Manchester M13 9PL.
>>>>>> Tel: +44 1612754171 or  x4171
>>>>>> E-mail: sabah.salih at manchester.ac.uk
>>>>>> From: Andy Buckley [andy.buckley at cern.ch]
>>>>>> Sent: 19 July 2018 14:41
>>>>>> To: Sabah Salih
>>>>>> Cc: rivet at projects.hepforge.org
>>>>>> Subject: Re: [Rivet] ruvet and osx
>>>>>> 
>>>>>> Hi Sabah,
>>>>>> I have no idea about compilation with GCC 8 -- it looks like our
>"autotools" build system assumes that a compiler flag will work, but
>GCC 8 has removed it. I suspect that new versions of autotools would
>fix it, but of course our existing tarball is built with an older
>version.
>>>>>> 
>>>>>> You could try building the (final) Rivet step by hand, first
>rebuilding the autotools bits with a call to "autoreconf -i". It may
>even be possible to re-run the bootstrap script after doing that, and
>let it finish the build with the updated Makefiles.
>>>>>> 
>>>>>> Andy
>>>>>> 
>>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research
>Fellow
>>>>>> Particle Physics Experiment Group, University of Glasgow
>>>>>> On Jul 19 2018, at 9:15 am, Sabah Salih
><sabah.salih at manchester.ac.uk> wrote:
>>>>>> 
>>>>>> 
>>>>>> Dear All,
>>>>>>     I am trying to install rivet in a max osx 10.13.6
>>>>>> I have
>>>>>> 
>>>>>> gcc --version
>>>>>> gcc (Homebrew GCC 8.1.0) 8.1.0
>>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>>>>> This is free software; see the source for copying conditions. 
>There is NO
>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A
>PARTICULAR PURPOSE.
>>>>>> 
>>>>>> gfortran --version
>>>>>> GNU Fortran (Homebrew GCC 8.1.0) 8.1.0
>>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>>>>> This is free software; see the source for copying conditions. 
>There is NO
>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A
>PARTICULAR PURPOSE.
>>>>>> 
>>>>>> g++ --version
>>>>>> g++ (Homebrew GCC 8.1.0) 8.1.0
>>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>>>>> This is free software; see the source for copying conditions. 
>There is NO
>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A
>PARTICULAR PURPOSE.
>>>>>> 
>>>>>> 
>>>>>> I got rivet from the following url
>>>>>> 
>>>>>> wget
>http://rivet.hepforge.org/hg/bootstrap/raw-file/2.6.0/rivet-bootstrap
>>>>>> chmod +x rivet-bootstrap
>>>>>> 
>>>>>> After I run the following
>>>>>> ./rivet-bootstrap
>>>>>> 
>>>>>> I get the following in the end
>>>>>> 
>>>>>> 
>>>>>> g++ -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe
>-fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX
>-DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g
>-fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -DNDEBUG
>-I/usr/include -I/usr/include -pipe
>-I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/include
>-I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/include
>-I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda
>-I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda
>-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
>-c
>/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda/util.cpp
>-o
>/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/build/temp.macosx-10.13-intel-2.7/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda/util.o
>-O3 -Wno-unused-but-set-variable -Wno-sign-compare
>>>>>> g++: error: unrecognized command line option '-Wshorten-64-to-32'
>>>>>> error: command 'g++' failed with exit status 1
>>>>>> make[2]: *** [all-local] Error 1
>>>>>> make[1]: *** [all-recursive] Error 1
>>>>>> make: *** [all-recursive] Error 1
>>>>>> 
>>>>>> 
>>>>>> What do I have to do to fix it please?
>>>>>> 
>>>>>> 
>>>>>> Maany Thanks, Sabah.
>>>>>> 
>>>>>> From Sabah Salih
>>>>>> Particle Physics Group,
>>>>>> The School of Physics and Astronomy,
>>>>>> The University of Manchester,
>>>>>> Schuster Laboratory,
>>>>>> Brunswick Street,
>>>>>> Manchester M13 9PL.
>>>>>> Tel: +44 1612754171 or  x4171
>>>>>> E-mail: sabah.salih at manchester.ac.uk
>>>>>> _______________________________________________
>>>>>> Rivet mailing list
>>>>>> Rivet at projects.hepforge.org
>>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>>>> _______________________________________________
>>>>>> Rivet mailing list
>>>>>> Rivet at projects.hepforge.org
>>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>>> 
>>>>> _______________________________________________
>>>>> Rivet mailing list
>>>>> Rivet at projects.hepforge.org
>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>> 
>>>> _______________________________________________
>>>> Rivet mailing list
>>>> Rivet at projects.hepforge.org
>>>> https://www.hepforge.org/lists/listinfo/rivet
>>> 
>>> <rivet-install.txt>_______________________________________________
>>> Rivet mailing list
>>> Rivet at projects.hepforge.org
>>> https://www.hepforge.org/lists/listinfo/rivet
>>