[Rivet] ruvet and osx

James William Monk james.william.monk at cern.ch
Mon Jul 30 16:17:18 BST 2018
Previous message: [Rivet] ruvet and osx
Next message: [Rivet] ruvet and osx
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi everyone,

Did we ever come across the problem that the mac System Integrity Protection (SIP) does not pass on the DYLD_LIBRARY_PATH to a subprocess if it thinks that subprocess is a “protected” process? I believe that is the problem here, and is preventing the library path being set correctly during the Rivet build because it spawns a python subprocess that does not inherit the correct environment

 It seems that if I do a python script to simply print the environment, i.e.

	import os
	for key in os.environ.keys():
	    print key

then execute

	foo=“bar” python test
(where test contains the script above)

Then I see that foo is detected in the environment.  However, if I do

	DYLD_LIBRARY_PATH=“bar” python test

Then DYLD_LIBRARY_PATH does not show up! Presumably SIP has decided that the python subprocess must be protected from malicious setting of the environment.  I can see why they might think that is important, since changing the path would potentially cause it to load a bad version of some library.  I can disable SIP to confirm this, but that requires a reboot of my mac and is not something we should ask users to do.

I think that the fix for this is that the mk-analysis-html script has to be executable.  That way it will inherit from the build environment, not from the python subprocess whose environment has been wiped clean.  

cheers,

James






> On 30 Jul 2018, at 16:36, James William Monk <james.william.monk at cern.ch> wrote:
> 
> Hi Sabah,
> 
> There is a bug in the build system for the docs in Rivet 2.6 on mac.  This issue has occurred before, but I only just remembered it.  It looks like David Grellscheid has tried to fix it in the past, but when I use the updated Makefile.am from the doc dir, I still get the same problem.  I knew there was a reason I was still on Rivet 2.5.4 :)
> 
> It looks to me like there is a mistake in the setup for the environment needed for the python script mk-analysis-html.  At least one error is that it seems to have the PYTHONPATH pointing to the installation prefix, but there is nothing there of course because Rivet hasn’t been installed yet.  Another error is that DYLD_LIBRARY_PATH is not set.  I tested this by modifying doc/mk-analysis-html by adding
> 
> print "Hello from mk-analysis-html.  I think $PYTHONPATH = " + os.environ['PYTHONPATH']
> print "environ keys are"
> for key in os.environ.keys():
>   print key
> 
> This problem is only related to the building of the documentation, but disabling doxygen doesn’t seem to help.  I *think* if you plough on regardless and do make install, you will get Rivet installed in your prefix, just without the docs and the rivetenv.sh script.
> 
> cheers,
> 
> James
> 
>> On 28 Jul 2018, at 13:11, Sabah Salih <sabah.salih at manchester.ac.uk> wrote:
>> 
>> Dear James,
>>          I am very grateful for your kind help
>> I tried to apply rivet-bootstrap content line by line via command line
>> 
>> YODA, HEPMC,... are fine
>> When I reached Rivet I get the following:
>> I attached the file which have the complete steps for RIVET
>> 
>> Many Thanks, Sabah.
>> 
>> 
>> 
>> File "/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py", line 12, in <module>
>>    from rivet.core import *
>> ImportError: dlopen(/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so, 10): Library not loaded: /tmp/rivet/local/lib/libRivet.dylib
>>  Referenced from: /private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>  Reason: image not found
>> make[1]: *** [analyses.json] Error 1
>> make[1]: *** Waiting for unfinished jobs....
>> Using output file name 'analyses.html' and directory 'analyses'
>> Traceback (most recent call last):
>>  File "mk-analysis-html", line 25, in <module>
>>    import rivet
>>  File "/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py", line 12, in <module>
>>    from rivet.core import *
>> ImportError: dlopen(/private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so, 10): Library not loaded: /tmp/rivet/local/lib/libRivet.dylib
>>  Referenced from: /private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>  Reason: image not found
>> make[1]: *** [analyses.html] Error 1
>> make: *** [all-recursive] Error 1
>> 
>> ls -lh /private/tmp/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/
>> total 312
>> -rw-r--r--  1 admin  wheel   351B  3 Oct  2017 __init__.py
>> -rw-r--r--  1 admin  wheel   598B 28 Jul 11:26 __init__.pyc
>> -rw-r--r--  1 admin  wheel   3.0K  6 Nov  2017 aopaths.py
>> -rwxr-xr-x  1 admin  wheel   121K 28 Jul 11:15 core.so
>> -rw-r--r--  1 admin  wheel   7.8K  3 Oct  2017 plotinfo.py
>> -rw-r--r--  1 admin  wheel   2.5K  3 Oct  2017 spiresbib.py
>> -rw-r--r--  1 admin  wheel   6.9K 20 Nov  2017 util.py
>> 
>> 
>> Many Thanks, Sabah.
>> 
>> From Sabah Salih
>> Particle Physics Group,
>> The School of Physics and Astronomy,
>> The University of Manchester,
>> Schuster Laboratory,
>> Brunswick Street,
>> Manchester M13 9PL.
>> Tel: +44 1612754171 or  x4171
>> E-mail: sabah.salih at manchester.ac.uk
>> 
>> ________________________________________
>> From: James William Monk [james.william.monk at cern.ch]
>> Sent: 25 July 2018 22:46
>> To: Sabah Salih
>> Cc: James William Monk; Andy Buckley; rivet at projects.hepforge.org
>> Subject: Re: [Rivet] ruvet and osx
>> 
>> Hi Sabah,
>> 
>> That error is just confirming that the Rivet lib is not present, I presume because the build and/or installation failed.  Do you see any other error that looks like a build error?  I’m not very familiar with the bootstrap, so I can’t say whether it produces any build logs, or where they would be, but we need to see the build failure message to know why the libraries weren't built properly.
>> 
>> cheers,
>> 
>> James
>> 
>> 
>> 
>>> On 25 Jul 2018, at 19:09, Sabah Salih <sabah.salih at manchester.ac.uk> wrote:
>>> 
>>> Dear James,
>>>      Thank you so much.
>>>   Following your advice I did the following:
>>> 
>>> edit rivet-bootstrap
>>> 
>>> and changed
>>> 
>>> function conf { ./configure --prefix=$INSTALL_PREFIX "$@"; }
>>> to
>>> function conf { ./configure CC=/usr/bin/clang CXX=/usr/bin/clang++ --prefix=$INSTALL_PREFIX "$@"; }
>>> 
>>> and
>>> ./configure  --prefix=$INSTALL_PREFIX
>>> to
>>> ./configure CC=/usr/bin/clang CXX=/usr/bin/clang++ --prefix=$INSTALL_PREFIX
>>> 
>>> When I type
>>> ./rivet-bootstrap
>>> 
>>> I get this
>>> File "/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py", line 12, in <module>
>>>   from rivet.core import *
>>> ImportError: dlopen(/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so, 10): Library not loaded: /Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/local/lib/libRivet.dylib
>>> Referenced from: /Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>> Reason: image not found
>>> make[1]: *** [analyses.json] Error 1
>>> make[1]: *** Waiting for unfinished jobs....
>>> Using output file name 'analyses.html' and directory 'analyses'
>>> Traceback (most recent call last):
>>> File "mk-analysis-html", line 25, in <module>
>>>   import rivet
>>> File "/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/__init__.py", line 12, in <module>
>>>   from rivet.core import *
>>> ImportError: dlopen(/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so, 10): Library not loaded: /Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/local/lib/libRivet.dylib
>>> Referenced from: /Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/Rivet-2.6.0/pyext/build/lib.macosx-10.13-intel-2.7/rivet/core.so
>>> Reason: image not found
>>> make[1]: *** [analyses.html] Error 1
>>> make: *** [all-recursive] Error 1
>>> 
>>> ls local/lib/libRivet.dylib
>>> ls: local/lib/libRivet.dylib: No such file or directory
>>> 
>>> Many Thanks, Sabah.
>>> 
>>> 
>>> From Sabah Salih
>>> Particle Physics Group,
>>> The School of Physics and Astronomy,
>>> The University of Manchester,
>>> Schuster Laboratory,
>>> Brunswick Street,
>>> Manchester M13 9PL.
>>> Tel: +44 1612754171 or  x4171
>>> E-mail: sabah.salih at manchester.ac.uk
>>> 
>>> ________________________________________
>>> From: James William Monk [james.william.monk at cern.ch]
>>> Sent: 25 July 2018 13:57
>>> To: Sabah Salih
>>> Cc: James William Monk; Andy Buckley; rivet at projects.hepforge.org
>>> Subject: Re: [Rivet] ruvet and osx
>>> 
>>> Hi Sabah,
>>> 
>>> No, there’s no need to cross compile here, we just need to make sure it uses compilers and libs all from the same toolchain and doesn’t mix and match system tools with homebrew tools.  For example, to try to use the system compiler you could try something like configuring Rivet and its dependencies with the extra config option
>>> 
>>>       ./configure <usual config options> CC=/usr/bin/clang CXX=/usr/bin/clang++
>>> 
>>> It looks like you prefer to use homebrew’s tools though.  I don’t know where they are located on your system, but the equivalent configure command would be
>>> 
>>>       ./configure <usual config opts> CC=/path/to/brew/gcc CXX=/path/to/brew/g++
>>> 
>>> where you fill in the paths to the brew installation versions of gcc and g++.  Of course, any Rivet dependencies (FastJet, GSL, HepMC...) will also need to be configured in the same way first.
>>> 
>>> If you just want to run Rivet, as opposed to develop for it, I’d also suggest taking a look at the Docker distribution for mac:
>>> 
>>>       https://rivet.hepforge.org/trac/wiki/Docker
>>> 
>>> cheers,
>>> 
>>> James
>>> 
>>> 
>>>> On 25 Jul 2018, at 08:37, Sabah Salih <sabah.salih at manchester.ac.uk> wrote:
>>>> 
>>>> Dear James,
>>>>       It is so nice to hear from you.
>>>> Thank you so much for your kind reply.
>>>> 
>>>> I just done
>>>> 
>>>> brew tap osx-cross/arm
>>>> brew tap osx-cross/avr
>>>> 
>>>> brew install arm-gcc-bin
>>>> 
>>>> brew install avr-gcc
>>>> This look like installing gcc8
>>>> 
>>>> Is this what you meant?
>>>> 
>>>> Many Tanks, Sabah.
>>>> 
>>>> From Sabah Salih
>>>> Particle Physics Group,
>>>> The School of Physics and Astronomy,
>>>> The University of Manchester,
>>>> Schuster Laboratory,
>>>> Brunswick Street,
>>>> Manchester M13 9PL.
>>>> Tel: +44 1612754171 or  x4171
>>>> E-mail: sabah.salih at manchester.ac.uk
>>>> 
>>>> ________________________________________
>>>> From: James William Monk [james.william.monk at cern.ch]
>>>> Sent: 25 July 2018 01:10
>>>> To: Andy Buckley; Sabah Salih
>>>> Cc: rivet at projects.hepforge.org
>>>> Subject: Re: [Rivet] ruvet and osx
>>>> 
>>>> Hi,
>>>> 
>>>> From a bit of nurdling about on various fora, that error appears to refer to an option that is present for clang, but not gcc.  Clang is the supplied default on mac.  You have installed a separate gcc toolchain via homebrew.  What I can imagine may be happening is that autotools is detecting the system clang and adding options appropriately, but then the g++ command is now calling the gcc from homebrew.  For example, on my own mac I see
>>>> 
>>>>      g++ --version
>>>>      Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
>>>>      Apple LLVM version 9.1.0 (clang-902.0.39.2)
>>>>      Target: x86_64-apple-darwin17.7.0
>>>> 
>>>> It’s possible you may be able to cure this with some appropriate environment settings like
>>>> 
>>>>      export CC=/usr/bin/clang
>>>>      export CXX=/usr/bin/clang++
>>>> 
>>>> Which would try to force it to use the system clang compiler consistently.  Alternatively, try to point it to your homebrew compiler location.  You may also like to consider removing homebrew as another solution - I don’t know what else it adds for you, but personally I have never found those kinds of packaging systems helpful on a mac precisely because they bring a second toolchain into play that is not always entirely consistent with the system toolchain.
>>>> 
>>>> cheers,
>>>> 
>>>> James
>>>> 
>>>> 
>>>> 
>>>>> On 25 Jul 2018, at 00:48, Andy Buckley <andy.buckley at cern.ch> wrote:
>>>>> 
>>>>> I'm afraid I don't use Macs, so am not sure what is best. Using an earlier compiler might work; I'm a little surprised that autoreconf did not help. Hopefully someone else on the Rivet team has a bit more Mac experience and can help out.
>>>>> 
>>>>> Andy
>>>>> 
>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>>> Particle Physics Experiment Group, University of Glasgow
>>>>> On Jul 24 2018, at 4:52 pm, Sabah Salih <sabah.salih at manchester.ac.uk> wrote:
>>>>> 
>>>>> 
>>>>> Dear Andy,
>>>>>        Thank you ever so much. The problem look like it is in YODA.
>>>>> I did
>>>>> 
>>>>> cd YODA-1.7.0
>>>>> 
>>>>> autoreconf -i
>>>>> make
>>>>> 
>>>>> that did not fix it
>>>>> 
>>>>> cd ..
>>>>> this is where rivet-bootstrap
>>>>> 
>>>>> and rerun
>>>>> 
>>>>> ./rivet-bootstrap
>>>>> 
>>>>> This did not fix it neither.
>>>>> 
>>>>> What is the best way to get rivet to work in a mac please.?
>>>>> It does not have to be gcc8
>>>>> 
>>>>> Many thanks, Sabah
>>>>> 
>>>>> 
>>>>> From Sabah Salih
>>>>> Particle Physics Group,
>>>>> The School of Physics and Astronomy,
>>>>> The University of Manchester,
>>>>> Schuster Laboratory,
>>>>> Brunswick Street,
>>>>> Manchester M13 9PL.
>>>>> Tel: +44 1612754171 or  x4171
>>>>> E-mail: sabah.salih at manchester.ac.uk
>>>>> From: Andy Buckley [andy.buckley at cern.ch]
>>>>> Sent: 19 July 2018 14:41
>>>>> To: Sabah Salih
>>>>> Cc: rivet at projects.hepforge.org
>>>>> Subject: Re: [Rivet] ruvet and osx
>>>>> 
>>>>> Hi Sabah,
>>>>> I have no idea about compilation with GCC 8 -- it looks like our "autotools" build system assumes that a compiler flag will work, but GCC 8 has removed it. I suspect that new versions of autotools would fix it, but of course our existing tarball is built with an older version.
>>>>> 
>>>>> You could try building the (final) Rivet step by hand, first rebuilding the autotools bits with a call to "autoreconf -i". It may even be possible to re-run the bootstrap script after doing that, and let it finish the build with the updated Makefiles.
>>>>> 
>>>>> Andy
>>>>> 
>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>>> Particle Physics Experiment Group, University of Glasgow
>>>>> On Jul 19 2018, at 9:15 am, Sabah Salih <sabah.salih at manchester.ac.uk> wrote:
>>>>> 
>>>>> 
>>>>> Dear All,
>>>>>     I am trying to install rivet in a max osx 10.13.6
>>>>> I have
>>>>> 
>>>>> gcc --version
>>>>> gcc (Homebrew GCC 8.1.0) 8.1.0
>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>>>> This is free software; see the source for copying conditions.  There is NO
>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>>>> 
>>>>> gfortran --version
>>>>> GNU Fortran (Homebrew GCC 8.1.0) 8.1.0
>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>>>> This is free software; see the source for copying conditions.  There is NO
>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>>>> 
>>>>> g++ --version
>>>>> g++ (Homebrew GCC 8.1.0) 8.1.0
>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>>>>> This is free software; see the source for copying conditions.  There is NO
>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>>>> 
>>>>> 
>>>>> I got rivet from the following url
>>>>> 
>>>>> wget http://rivet.hepforge.org/hg/bootstrap/raw-file/2.6.0/rivet-bootstrap
>>>>> chmod +x rivet-bootstrap
>>>>> 
>>>>> After I run the following
>>>>> ./rivet-bootstrap
>>>>> 
>>>>> I get the following in the end
>>>>> 
>>>>> 
>>>>> g++ -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -DNDEBUG -I/usr/include -I/usr/include -pipe -I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/include -I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/include -I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda -I/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c /Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda/util.cpp -o /Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/build/temp.macosx-10.13-intel-2.7/Users/admin/madgraph/MG5_aMC_v2_6_3/rivet/YODA-1.7.0/pyext/yoda/util.o -O3 -Wno-unused-but-set-variable -Wno-sign-compare
>>>>> g++: error: unrecognized command line option '-Wshorten-64-to-32'
>>>>> error: command 'g++' failed with exit status 1
>>>>> make[2]: *** [all-local] Error 1
>>>>> make[1]: *** [all-recursive] Error 1
>>>>> make: *** [all-recursive] Error 1
>>>>> 
>>>>> 
>>>>> What do I have to do to fix it please?
>>>>> 
>>>>> 
>>>>> Maany Thanks, Sabah.
>>>>> 
>>>>> From Sabah Salih
>>>>> Particle Physics Group,
>>>>> The School of Physics and Astronomy,
>>>>> The University of Manchester,
>>>>> Schuster Laboratory,
>>>>> Brunswick Street,
>>>>> Manchester M13 9PL.
>>>>> Tel: +44 1612754171 or  x4171
>>>>> E-mail: sabah.salih at manchester.ac.uk
>>>>> _______________________________________________
>>>>> Rivet mailing list
>>>>> Rivet at projects.hepforge.org
>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>>> _______________________________________________
>>>>> Rivet mailing list
>>>>> Rivet at projects.hepforge.org
>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>> 
>>>> _______________________________________________
>>>> Rivet mailing list
>>>> Rivet at projects.hepforge.org
>>>> https://www.hepforge.org/lists/listinfo/rivet
>>> 
>>> _______________________________________________
>>> Rivet mailing list
>>> Rivet at projects.hepforge.org
>>> https://www.hepforge.org/lists/listinfo/rivet
>> 
>> <rivet-install.txt>_______________________________________________
>> Rivet mailing list
>> Rivet at projects.hepforge.org
>> https://www.hepforge.org/lists/listinfo/rivet
>
Previous message: [Rivet] ruvet and osx
Next message: [Rivet] ruvet and osx
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Rivet mailing list