[Rivet] possibility to add (hadd) yoda files, and running rivet on lxplus batch jobs

[Raghav Kunnawalkam Elayavalli]

Hi Andy,

very good on the yodamerge. that'll come in handy :).

Regarding the lxplus batch and python environment. Im just sourcing the following scripts:
source /afs/cern.ch/sw/lcg/contrib/gcc/4.7.2/x86_64-slc6/setup.sh<http://cern.ch/sw/lcg/contrib/gcc/4.7.2/x86_64-slc6/setup.sh>
source /afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/rivetenv.sh<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/rivetenv.sh>
export RIVET_ANALYSIS_PATH=/afs/cern.ch/work/r/rkunnawa/JEWEL/ANALYSIS/:$RIVET_ANALYSIS_PATH<http://cern.ch/work/r/rkunnawa/JEWEL/ANALYSIS/:$RIVET_ANALYSIS_PATH>

after sourcing this, if i ask for which python, i still get
/usr/bin/python
when locally on lxplus if i source the following scripts and ask for which python i get /cvmfs/sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/bin/python<http://sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/bin/python>

and then i try to run my jobs and this is what i get:

'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet>", line 20, in <module>
    import os, sys
ImportError: No module named os
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet-mkhtml<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/bin/rivet-mkhtml>", line 17, in <module>
    import rivet, sys, os
  File "/afs/cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/lib/python2.7/site-packages/rivet/__init__.py<http://cern.ch/sw/lcg/external/MCGenerators_lcgcmt67c/rivet/2.4.1/x86_64-slc6-gcc47-opt/lib/python2.7/site-packages/rivet/__init__.py>", line 7, in <module>
    import dl
ImportError: No module named dl

So im not sure what is happening also here. Im also asking the CERN service desk and in the meanwhile if anyone has successfully submitted batch jobs from lxplus running rivet, please let me know if Im missing something.

Cheers
Raghav
On Jun 9, 2016, at 2:16 PM, Andy Buckley <a.g.buckley at gmail.com<mailto:a.g.buckley at gmail.com>> wrote:

On 09/06/16 13:07, Raghav Kunnawalkam Elayavalli wrote:
Hi Rivet Experts,

My current analysis has exploded into quite the complicated one
leading to a large run time ~25 seconds per 100 events. It would be
very useful if i can submit jobs to run this in parallel say 100 job
running over 1000 events each. In leu of this requirement, i would
need to add the output yoda files together similar to the hadd script
that does root files.

This is no problem: that's what the yodamerge script does.

Im also having several difficulties with the proper python
environment at the node where my batch jobs end up (for rivet -a
ANALYSIS input.hepmc). this is what i used in my rivet installation
on lxplus in the rivet-bootstrap-lcg SETUPCMD_PYTHON="source
$LCGDIR/Python/2.7.10/x86_64-slc6-gcc48-opt/Python-env.sh" and this
is what i set in my job source
/cvmfs/sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/Python-env.sh<http://sft.cern.ch/lcg/releases/LCG_84/Python/2.7.10/x86_64-slc6-gcc48-opt/Python-env.sh>

when i submit a job with this python and rivet setup, i get a
> following error

@(#)CERN job starter $Date: 2010/06/23 14:22:16 $ Working directory
is </pool/lsf/rkunnawa/809932193> on <b64f2daace.cern.ch<http://b64f2daace.cern.ch>> Started
running the job

this error is for sourcing that python env.

'import site' failed; use -v for traceback Traceback (most recent
call last): File
"/cvmfs/sft.cern.ch/lcg/releases/LCG_84/lcgenv/1.3.2.1/x86_64-centos7-gcc49-dbg/lcgenv<http://sft.cern.ch/lcg/releases/LCG_84/lcgenv/1.3.2.1/x86_64-centos7-gcc49-dbg/lcgenv>",
line 11, in <module> import os ImportError: No module named os

# output of which python: /usr/bin/python

these are for the following error messages when it comes to the rivet
-a 'import site' failed; use -v for traceback Traceback (most recent
call last): File
"/afs/cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet<http://cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet>", line 20, in
<module> import os, sys ImportError: No module named os 'import site'
failed; use -v for traceback Traceback (most recent call last): File
"/afs/cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet-mkhtml<http://cern.ch/work/r/rkunnawa/JEWEL/local/bin/rivet-mkhtml>", line 17,
in <module> import rivet, sys, os ImportError: No module named rivet

Hmm, I don't really know what's going on. So you set up the same Python environment for your job as used in the build? If you just set up that Python without setting up the Rivet environment, can you open a Python interpreter, import os, etc. without getting these errors?

This isn't really a Rivet problem, though -- the environment is always a pain on lxplus and other SLC6 systems, since the system compilers and Python are not the ones that are actually supported by LCG.

Andy

--
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.hepforge.org/lists-archive/rivet/attachments/20160609/240ac3b6/attachment.html>