[Rivet] possibility to add (hadd) yoda files, and running rivet on lxplus batch jobs

[David Grellscheid]

Hi Raghav,

I assume the shell is bash or zsh, which keeps an internal cache of 
available commands on PATH. Try 'hash -r' after all the sourcing to 
rebuild that cache.

   David


On 09/06/2016 15:03, Raghav Kunnawalkam Elayavalli wrote:
> Hi Andy,
>
> very good on the yodamerge. that'll come in handy :).
>
> Regarding the lxplus batch and python environment. Im just sourcing the following scripts:
> source /afs/cern.ch/sw/lcg/contrib/gcc/4.7.2/x86_64-slc6/setup.sh<http://cern.ch/sw/lcg/contrib/gcc/4.7.2/x86_64-slc6/setup.sh>
> source /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/rivetenv.sh<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/rivetenv.sh>
> export RIVET_ANALYSIS_PATH=/afs/cern.ch/work/r/rkunnawa/JEWEL/ANALYSIS/:$RIVET_ANALYSIS_PATH<http://cern.ch/work/r/rkunnawa/JEWEL/ANALYSIS/:$RIVET_ANALYSIS_PATH>
>
> after sourcing this, if i ask for which python, i still get
> /usr/bin/python
> when locally on lxplus if i source the following scripts and ask for which python i get /cvmfs/sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/bin/python<http://sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/bin/python>
>
> and then i try to run my jobs and this is what i get:
>
> 'import site' failed; use -v for traceback
> Traceback (most recent call last):
>    File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet>", line 20, in <module>
>      import os, sys
> ImportError: No module named os
> 'import site' failed; use -v for traceback
> Traceback (most recent call last):
>    File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet-mkhtml<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet-mkhtml>", line 17, in <module>
>      import rivet, sys, os
>    File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/lib/python2.7/site-packages/rivet/__init__.py<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/lib/python2.7/site-packages/rivet/__init__.py>", line 7, in <module>
>      import dl
> ImportError: No module named dl
>
> So im not sure what is happening also here. Im also asking the CERN service desk and in the meanwhile if anyone has successfully submitted batch jobs from lxplus running rivet, please let me know if Im missing something.
>
> Cheers
> Raghav
> On Jun 9, 2016, at 2:16 PM, Andy Buckley <a.g.buckley at gmail.com<mailto:a.g.buckley at gmail.com>> wrote:
>
> On 09/06/16 13:07, Raghav Kunnawalkam Elayavalli wrote:
> Hi Rivet Experts,
>
> My current analysis has exploded into quite the complicated one
> leading to a large run time ~25 seconds per 100 events. It would be
> very useful if i can submit jobs to run this in parallel say 100 job
> running over 1000 events each. In leu of this requirement, i would
> need to add the output yoda files together similar to the hadd script
> that does root files.
>
> This is no problem: that's what the yodamerge script does.
>
> Im also having several difficulties with the proper python
> environment at the node where my batch jobs end up (for rivet -a
> ANALYSIS input.hepmc). this is what i used in my rivet installation
> on lxplus in the rivet-bootstrap-lcg SETUPCMD_PYTHON="source
> $LCGDIR/Python/2.7.10/x86_64-slc6-gcc48-opt/Python-env.sh" and this
> is what i set in my job source
> /cvmfs/sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/Python-env.sh<http://sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/Python-env.sh>
>
> when i submit a job with this python and rivet setup, i get a
>> following error
>
> @(#)CERN job starter $Date: 2010/06/23 14:22:16 $ Working directory
> is </pool/lsf/rkunnawa/809932193> on <b64f2daace.cern.ch<http://b64f2daace.cern.ch>> Started
> running the job
>
> this error is for sourcing that python env.
>
> 'import site' failed; use -v for traceback Traceback (most recent
> call last): File
> "/cvmfs/sft.cern.ch/lcg/releases/LCG_84/lcgenv/1.3.2.1/x86_64-centos7-gcc49-dbg/lcgenv<http://sft.cern.ch/lcg/releases/LCG_84/lcgenv/1.3.2.1/x86_64-centos7-gcc49-dbg/lcgenv>",
> line 11, in <module> import os ImportError: No module named os
>
> # output of which python: /usr/bin/python
>
> these are for the following error messages when it comes to the rivet
> -a 'import site' failed; use -v for traceback Traceback (most recent
> call last): File
> "/afs/cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet<http://cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet>", line 20, in
> <module> import os, sys ImportError: No module named os 'import site'
> failed; use -v for traceback Traceback (most recent call last): File
> "/afs/cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet-mkhtml<http://cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet-mkhtml>", line 17,
> in <module> import rivet, sys, os ImportError: No module named rivet
>
> Hmm, I don't really know what's going on. So you set up the same Python environment for your job as used in the build? If you just set up that Python without setting up the Rivet environment, can you open a Python interpreter, import os, etc. without getting these errors?
>
> This isn't really a Rivet problem, though -- the environment is always a pain on lxplus and other SLC6 systems, since the system compilers and Python are not the ones that are actually supported by LCG.
>
> Andy
>
> --
> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
> Particle Physics Expt Group, University of Glasgow
>
>
>
>
> _______________________________________________
> Rivet mailing list
> Rivet at projects.hepforge.org
> https://www.hepforge.org/lists/listinfo/rivet
>