[Rivet] installation

[Frank Siegert]

Ah, and one thing that I forgot: probably not a conceptual problem,
but a bug, which basically stopped Frank's Rivet installation efforts
in the end yesterday:

When we do
  configure; make; make install;
then the make step will also go into doc/ and try to use
rivet-mkanalyses-html. As far as I understand the Makefile and also
Frank's error from yesterday, this assumes that libRivet.so is already
installed into --prefix, which clearly isn't the case in the make
step. Am I misunderstanding this? Could we move the complete doc/
building into the --enable-pdfmanual conditional (and disable it by
default), to avoid such simple compile failures?

Cheers,
Frank



On 27 January 2014 13:43, Frank Siegert <frank.siegert at cern.ch> wrote:
> Hi Andy,
>
> I think there is one simple issue explaining the difference in
> perception between you and Frank et al.:
> One actually needs to install *Rivet-dev* to get current analyses.
>
> And installing Rivet-dev is the annoying bit, in particular for Rivet2
> vs. Rivet1 due to:
>
> 1. Cython
> It's simply not possible to install cython properly into the install
> prefix automatically in a bootstrap script, which is why we have
> worked around by setting the PATH and PYTHONPATH to find it during the
> bootstrap. But that "solution" fails in case you want to actually
> update your Rivet later (new analyses!) and compile again, since those
> variables aren't set anymore. Swig was much easier in that respect.
>
> 2. hg
> While SVN was basically available on all systems that I have worked
> on, I often had to install hg myself.
>
> 3. YODA from hg (release sometimes isn't new enough for Rivet-dev)
> + its own dependencies.
>
> 4. autotools
> I have seen two independent clusters where the autotools from SL6 (or
> SL5, not 100% sure) lead to very hard to understand linker problems,
> which went away with recent autotools.
>
> 5. yaml_cpp
> This needs cmake for installation, i.e. yet another dependency. So if
> that's bundled in the future, it will already make quite a difference.
>
>
> Bottom line: no, it's not just about the Boost version in YODA.
>
> Cheers,
> Frank
>
>
> On 27 January 2014 13:03, Andy Buckley <andy.buckley at cern.ch> wrote:
>> On 27/01/14 11:17, Frank Krauss wrote:
>>> Hi Andy,
>>> well, that's great.  I found out about the dependencies the hard way,
>>> but I still did not manage to produce an installation.  And since you
>>> guys appear to claim zero time of developer time it has about zero
>>> usability at the moment for anyone who cannot rely on it being already
>>> installed on lxplus.  I guess however that Rivet was also meant for the
>>> wider MC community, including theorists not sitting at CERN.  For those
>>> it is pretty much useless, and, the way I see it, because the developers
>>> mainly aimed at proving how smart they are rather than at having
>>> something useful.  So, up to now, you just showed me that you guys do
>>> not give a toss about this part of the community.  Good to know.  I
>>> sincerely hope this attitude of producing monumental software for the
>>> nerds is not the new trend in HEP computing.
>>
>> I'll try to reply nicely... but not exactly the style of feedback that
>> makes me want to be Mr Helpful, Frank.
>>
>> First off, of course we care about MC/pheno theorists. But as far as I
>> was aware the part of the community that Rivet was started for (i.e. you
>> guys) were more-or-less happily using it without significant trouble. So
>> the recent work (shall I moan about manpower again?) has been aimed at
>> getting the experiment uptake to the same level as in the generator
>> community. Understandable, no? So it looks like time for the balance to
>> swing a bit back toward the non-LCG users again.
>>
>> I _really_ hope we have better ways to look (be!) clever than introduce
>> new pain-in-the-ass library dependencies. It's more the other way
>> around: when there's a better implementation somewhere else, which we
>> have no chance/time to reimplement internally, we had to be stupid and
>> acquire the dependency. Most of this happened > 5 years ago. Where there
>> have been changes they have been to *reduce* build system problems (cf.
>> the previous way of bundling yaml-cpp). I didn't think that much had
>> changed with the release of version 2, but clearly some changes have had
>> painful impact: it would be good to know exactly what these are. As I
>> mentioned in another email, I suspect that YODA's Boost version
>> requirement is too high -- is that right?
>>
>> The dependencies that we have were chosen (I thought fairly carefully)
>> for compatibility with the "canonical" HEP system, i.e. SLC6 and the LCG
>> libraries. Frank's new script should handle tarball builds on systems
>> where LCG isn't present, which is great. Installation on personal
>> laptops _should_ be far easier because so many dependencies can come
>> direct from system packages... but you're right that that's assuming a
>> degree of nerdity in that you're already administrating your own machine
>> so can presumably choose what to install from where. If not, Frank's new
>> "install everything" script should help... but be aware that if you
>> build one library with a system Boost/GSL/whatever and Rivet with this
>> all-in-one bootstrap then the fact that it fails to run isn't really a
>> Rivet problem! Finally, building an unreleased development version is of
>> course a bit harder: heck, *I* don't try to build the developer version
>> on lxplus direct from the repository: I make a tarball on my laptop,
>> copy it over, and run the standard bootstrap.
>>
>> Ok, enough. The feedback and the more general bootstrap script are
>> really good to have, and I wish we'd known the strength of feeling about
>> Rivet much earlier... but if no-one reports it then we don't know.  Let
>> us know (nicely :-P) if there are more problems -- particularly,
>> concrete help like Frank S' is hugely appreciated.
>>
>> Andy
>>
>> PS. Current manpower, since there was incredulity: I would guess < 1
>> day/month on average for me, plus something similar or less for David G,
>> and occasional (very welcome!) contributions from others like Frank S.
>> Much of that time goes on analysis integration and (slow) feature
>> development. Does it look like more from outside?
>>
>> PPS. When yaml-cpp is bundled back inside Rivet, the tarball
>> dependencies will be exactly the same as Rivet 1, except that the
>> histogramming now has to be built separately. So I think the 1 vs. 2
>> dichotomy is a false one, and the issue is more to with the change to
>> bootstrap scripts. You'll probably still need to have cmake installed,
>> unless we bundle that as well... and why not an entire operating system?
>> But while I dislike that cmake requirement, it's the same for a *lot* of
>> other HEP packages now since LCG standardised on it. There will be a lot
>> of pressure for HepMC3 to go that way, I think.
>>
>>
>>> On 26/01/14 22:59, Andy Buckley wrote:
>>>> Hi Franks,
>>>>
>>>> Well, it's usable in proportion to the amount of developer time that we
>>>> have had to spend on making it nice... the main users what we have had
>>>> to win over are the experiment ones who access it through experiment
>>>> frameworks or from the Genser-built copy. So certainly that's who I've
>>>> been aiming my limited build system dev time at, and to be honest I
>>>> still stand by that choice although I'd rather not have *anyone* annoyed
>>>> by our code.
>>>>
>>>> For a tarball build I think the dependencies are as follows:
>>>>
>>>> YODA requires:
>>>>   * Boost (headers only, but currently asks for quite a new version...
>>>> can maybe be relaxed: any testers?)
>>>>
>>>> Rivet requires:
>>>>   * HepMC, FastJet (unavoidable, I think)
>>>>   * YODA (yes, could have been built-in, but it has wider utility)
>>>>   * yaml-cpp (for 2.1.0 I'd like to use a built-in copy, cf. the latest
>>>> LHAPDF6)
>>>>   * GNU Scientific Library (volunteers to eliminate that dependency? But
>>>> is typically an easily-met requirement, I think)
>>>>   * Boost (headers only, fairly easy version requirement, I think)
>>>>
>>>> If we eliminate yaml-cpp, how much is there to object to? Anyway, yes
>>>> Frank I'd be very happy for you to make a new rivet2-bootstrap script
>>>> which builds everything from scratch... and/or attempts to work out what
>>>> is already on the system? At some level users should make their own
>>>> minds up about what to install from packaging systems and what by
>>>> hand... but by saying that I probably give away what side of Frank's
>>>> nerd fence I'm sitting on ;-)
>>>>
>>>> Anyway, in short I don't see so much room for cutting things down, but
>>>> always think that smoothing the entry paths (with scripts or otherwise)
>>>> is a good thing as long as it's kept manageable for the developers, too.
>>>> Since that's translated as "next to no manpower" for the last... two
>>>> years?, I'm afraid the situation isn't so surprising. But we'll do what
>>>> we can -- certainly I want to get rid of the yaml-cpp explicity
>>>> requirement, and hopefully we can get a couple of new part-time
>>>> developers (re)activated in the coming months.
>>>>
>>>> Andy
>>>>
>>>>
>>>> On 26/01/14 19:47, Frank Siegert wrote:
>>>>> I hate to admit that I agree with Frank's rant here, and I have had
>>>>> multiple users recently who made the same point, though admittedly
>>>>> more politely. The bootstrap script currently is close to useless to
>>>>> everybody who isn't installing on lxplus. I have it on my agenda to
>>>>> replace it with a working version, but haven't got around to it yet.
>>>>> And to be honest, all the changes and external dependencies that came
>>>>> with Rivet2 have left me as a second-class developer (if even),
>>>>> because there is a lot of stuff in the codebase which is beyond my
>>>>> capabilities to fix -- I'm normally quite annoyed myself with even
>>>>> installing Rivet on a new cluster, where
>>>>> cython/yaml/yoda/boost/cmake/hg/fastjet/hepmc... isn't available. Some
>>>>> of those are only necessary for the development version, but it's
>>>>> still rather annoying.
>>>>> This goes far beyond the switch to YODA, which I agree with as
>>>>> necessary, though tbh I originally thought would be Rivet-in-house as
>>>>> well.
>>>>>
>>>>> Anyway, as I'm clearly not the majority of Rivet developers, and
>>>>> mostly everybody else on the team seems to be happy about this usage
>>>>> of fancy new technologies, separation and dependencies, there is no
>>>>> point in discussing that in itself. But we definitely need a usable
>>>>> bootstrap script much more urgently than we needed it for Rivet1.
>>>>>
>>>>> Andy, what about renaming the current rivet2 bootstrap script to
>>>>> rivet-2-bootstrap-lcg, and (me) creating a separate one which actually
>>>>> does all the installations on its own (but without support for Rivet1
>>>>> of course, to avoid it becoming too bloated as the old one did)?
>>>>>
>>>>> Cheers,
>>>>> Frank
>>>>>
>>>>>
>>>>> On 26 January 2014 17:13, Frank Krauss <frank.krauss at durham.ac.uk>
>>>>> wrote:
>>>>>> Dear Rivet Authors,
>>>>>> I am just towards the end of my second hour running trying to
>>>>>> install Rivet 2.0.  Admittedly, I am not half as geeky and nerdy as
>>>>>> you are, so I accepted/anticipated that this would take me an hour.
>>>>>> But, please, can you explain the rationale why you use
>>>>>> cython, yaml, yoda, boost, fastjet & hepmc (I am sure i missed some!)
>>>>>> in a non-trivial way with non-trivial version specifications without
>>>>>> providing a meaningful installation guide?  You certainly do not
>>>>>> want the common people to use your code - is that it?  If you think
>>>>>> this helps the user base you are mightily mistaken.  If you think I
>>>>>> will continue to advertise your code, you're delusional - this is
>>>>>> next to uninstallable as it stands now, with a pretty superficial
>>>>>> installation help.
>>>>>> Best wishes
>>>>>>      Frank
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Rivet mailing list
>>>>>> Rivet at projects.hepforge.org
>>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>> _______________________________________________
>>>>> Rivet mailing list
>>>>> Rivet at projects.hepforge.org
>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>>
>>>>
>>>
>>
>>
>> --
>> Dr Andy Buckley, Royal Society University Research Fellow
>> Particle Physics Expt Group, University of Glasgow / PH Dept, CERN