[Rivet] installation

[Andy Buckley]

Turning off the PDF manual build by default is possible, and perhaps
would be welcome: what do others think?

The build system / scripts in the doc directory try to bootstrap their
environment to only use the uninstalled libraries from inside the Rivet
build dir. There were some bugs in that which I fixed "recently" and
greatly tidied things up, but I forget whether they're in version 2.0.0
or just the dev head. So I'm surprised that it didn't work, but if
disabling the default manual build would avoid a whole bunch of common
problems then I think we should do it: for released versions it will be
obtainable via the arXiv anyway.

Cheers,
Andy


On 27/01/14 13:47, Frank Siegert wrote:
> Ah, and one thing that I forgot: probably not a conceptual problem,
> but a bug, which basically stopped Frank's Rivet installation efforts
> in the end yesterday:
> 
> When we do
>   configure; make; make install;
> then the make step will also go into doc/ and try to use
> rivet-mkanalyses-html. As far as I understand the Makefile and also
> Frank's error from yesterday, this assumes that libRivet.so is already
> installed into --prefix, which clearly isn't the case in the make
> step. Am I misunderstanding this? Could we move the complete doc/
> building into the --enable-pdfmanual conditional (and disable it by
> default), to avoid such simple compile failures?
> 
> Cheers,
> Frank
> 
> 
> 
> On 27 January 2014 13:43, Frank Siegert <frank.siegert at cern.ch> wrote:
>> Hi Andy,
>>
>> I think there is one simple issue explaining the difference in
>> perception between you and Frank et al.:
>> One actually needs to install *Rivet-dev* to get current analyses.
>>
>> And installing Rivet-dev is the annoying bit, in particular for Rivet2
>> vs. Rivet1 due to:
>>
>> 1. Cython
>> It's simply not possible to install cython properly into the install
>> prefix automatically in a bootstrap script, which is why we have
>> worked around by setting the PATH and PYTHONPATH to find it during the
>> bootstrap. But that "solution" fails in case you want to actually
>> update your Rivet later (new analyses!) and compile again, since those
>> variables aren't set anymore. Swig was much easier in that respect.
>>
>> 2. hg
>> While SVN was basically available on all systems that I have worked
>> on, I often had to install hg myself.
>>
>> 3. YODA from hg (release sometimes isn't new enough for Rivet-dev)
>> + its own dependencies.
>>
>> 4. autotools
>> I have seen two independent clusters where the autotools from SL6 (or
>> SL5, not 100% sure) lead to very hard to understand linker problems,
>> which went away with recent autotools.
>>
>> 5. yaml_cpp
>> This needs cmake for installation, i.e. yet another dependency. So if
>> that's bundled in the future, it will already make quite a difference.
>>
>>
>> Bottom line: no, it's not just about the Boost version in YODA.
>>
>> Cheers,
>> Frank
>>
>>
>> On 27 January 2014 13:03, Andy Buckley <andy.buckley at cern.ch> wrote:
>>> On 27/01/14 11:17, Frank Krauss wrote:
>>>> Hi Andy,
>>>> well, that's great.  I found out about the dependencies the hard way,
>>>> but I still did not manage to produce an installation.  And since you
>>>> guys appear to claim zero time of developer time it has about zero
>>>> usability at the moment for anyone who cannot rely on it being already
>>>> installed on lxplus.  I guess however that Rivet was also meant for the
>>>> wider MC community, including theorists not sitting at CERN.  For those
>>>> it is pretty much useless, and, the way I see it, because the developers
>>>> mainly aimed at proving how smart they are rather than at having
>>>> something useful.  So, up to now, you just showed me that you guys do
>>>> not give a toss about this part of the community.  Good to know.  I
>>>> sincerely hope this attitude of producing monumental software for the
>>>> nerds is not the new trend in HEP computing.
>>>
>>> I'll try to reply nicely... but not exactly the style of feedback that
>>> makes me want to be Mr Helpful, Frank.
>>>
>>> First off, of course we care about MC/pheno theorists. But as far as I
>>> was aware the part of the community that Rivet was started for (i.e. you
>>> guys) were more-or-less happily using it without significant trouble. So
>>> the recent work (shall I moan about manpower again?) has been aimed at
>>> getting the experiment uptake to the same level as in the generator
>>> community. Understandable, no? So it looks like time for the balance to
>>> swing a bit back toward the non-LCG users again.
>>>
>>> I _really_ hope we have better ways to look (be!) clever than introduce
>>> new pain-in-the-ass library dependencies. It's more the other way
>>> around: when there's a better implementation somewhere else, which we
>>> have no chance/time to reimplement internally, we had to be stupid and
>>> acquire the dependency. Most of this happened > 5 years ago. Where there
>>> have been changes they have been to *reduce* build system problems (cf.
>>> the previous way of bundling yaml-cpp). I didn't think that much had
>>> changed with the release of version 2, but clearly some changes have had
>>> painful impact: it would be good to know exactly what these are. As I
>>> mentioned in another email, I suspect that YODA's Boost version
>>> requirement is too high -- is that right?
>>>
>>> The dependencies that we have were chosen (I thought fairly carefully)
>>> for compatibility with the "canonical" HEP system, i.e. SLC6 and the LCG
>>> libraries. Frank's new script should handle tarball builds on systems
>>> where LCG isn't present, which is great. Installation on personal
>>> laptops _should_ be far easier because so many dependencies can come
>>> direct from system packages... but you're right that that's assuming a
>>> degree of nerdity in that you're already administrating your own machine
>>> so can presumably choose what to install from where. If not, Frank's new
>>> "install everything" script should help... but be aware that if you
>>> build one library with a system Boost/GSL/whatever and Rivet with this
>>> all-in-one bootstrap then the fact that it fails to run isn't really a
>>> Rivet problem! Finally, building an unreleased development version is of
>>> course a bit harder: heck, *I* don't try to build the developer version
>>> on lxplus direct from the repository: I make a tarball on my laptop,
>>> copy it over, and run the standard bootstrap.
>>>
>>> Ok, enough. The feedback and the more general bootstrap script are
>>> really good to have, and I wish we'd known the strength of feeling about
>>> Rivet much earlier... but if no-one reports it then we don't know.  Let
>>> us know (nicely :-P) if there are more problems -- particularly,
>>> concrete help like Frank S' is hugely appreciated.
>>>
>>> Andy
>>>
>>> PS. Current manpower, since there was incredulity: I would guess < 1
>>> day/month on average for me, plus something similar or less for David G,
>>> and occasional (very welcome!) contributions from others like Frank S.
>>> Much of that time goes on analysis integration and (slow) feature
>>> development. Does it look like more from outside?
>>>
>>> PPS. When yaml-cpp is bundled back inside Rivet, the tarball
>>> dependencies will be exactly the same as Rivet 1, except that the
>>> histogramming now has to be built separately. So I think the 1 vs. 2
>>> dichotomy is a false one, and the issue is more to with the change to
>>> bootstrap scripts. You'll probably still need to have cmake installed,
>>> unless we bundle that as well... and why not an entire operating system?
>>> But while I dislike that cmake requirement, it's the same for a *lot* of
>>> other HEP packages now since LCG standardised on it. There will be a lot
>>> of pressure for HepMC3 to go that way, I think.
>>>
>>>
>>>> On 26/01/14 22:59, Andy Buckley wrote:
>>>>> Hi Franks,
>>>>>
>>>>> Well, it's usable in proportion to the amount of developer time that we
>>>>> have had to spend on making it nice... the main users what we have had
>>>>> to win over are the experiment ones who access it through experiment
>>>>> frameworks or from the Genser-built copy. So certainly that's who I've
>>>>> been aiming my limited build system dev time at, and to be honest I
>>>>> still stand by that choice although I'd rather not have *anyone* annoyed
>>>>> by our code.
>>>>>
>>>>> For a tarball build I think the dependencies are as follows:
>>>>>
>>>>> YODA requires:
>>>>>   * Boost (headers only, but currently asks for quite a new version...
>>>>> can maybe be relaxed: any testers?)
>>>>>
>>>>> Rivet requires:
>>>>>   * HepMC, FastJet (unavoidable, I think)
>>>>>   * YODA (yes, could have been built-in, but it has wider utility)
>>>>>   * yaml-cpp (for 2.1.0 I'd like to use a built-in copy, cf. the latest
>>>>> LHAPDF6)
>>>>>   * GNU Scientific Library (volunteers to eliminate that dependency? But
>>>>> is typically an easily-met requirement, I think)
>>>>>   * Boost (headers only, fairly easy version requirement, I think)
>>>>>
>>>>> If we eliminate yaml-cpp, how much is there to object to? Anyway, yes
>>>>> Frank I'd be very happy for you to make a new rivet2-bootstrap script
>>>>> which builds everything from scratch... and/or attempts to work out what
>>>>> is already on the system? At some level users should make their own
>>>>> minds up about what to install from packaging systems and what by
>>>>> hand... but by saying that I probably give away what side of Frank's
>>>>> nerd fence I'm sitting on ;-)
>>>>>
>>>>> Anyway, in short I don't see so much room for cutting things down, but
>>>>> always think that smoothing the entry paths (with scripts or otherwise)
>>>>> is a good thing as long as it's kept manageable for the developers, too.
>>>>> Since that's translated as "next to no manpower" for the last... two
>>>>> years?, I'm afraid the situation isn't so surprising. But we'll do what
>>>>> we can -- certainly I want to get rid of the yaml-cpp explicity
>>>>> requirement, and hopefully we can get a couple of new part-time
>>>>> developers (re)activated in the coming months.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>> On 26/01/14 19:47, Frank Siegert wrote:
>>>>>> I hate to admit that I agree with Frank's rant here, and I have had
>>>>>> multiple users recently who made the same point, though admittedly
>>>>>> more politely. The bootstrap script currently is close to useless to
>>>>>> everybody who isn't installing on lxplus. I have it on my agenda to
>>>>>> replace it with a working version, but haven't got around to it yet.
>>>>>> And to be honest, all the changes and external dependencies that came
>>>>>> with Rivet2 have left me as a second-class developer (if even),
>>>>>> because there is a lot of stuff in the codebase which is beyond my
>>>>>> capabilities to fix -- I'm normally quite annoyed myself with even
>>>>>> installing Rivet on a new cluster, where
>>>>>> cython/yaml/yoda/boost/cmake/hg/fastjet/hepmc... isn't available. Some
>>>>>> of those are only necessary for the development version, but it's
>>>>>> still rather annoying.
>>>>>> This goes far beyond the switch to YODA, which I agree with as
>>>>>> necessary, though tbh I originally thought would be Rivet-in-house as
>>>>>> well.
>>>>>>
>>>>>> Anyway, as I'm clearly not the majority of Rivet developers, and
>>>>>> mostly everybody else on the team seems to be happy about this usage
>>>>>> of fancy new technologies, separation and dependencies, there is no
>>>>>> point in discussing that in itself. But we definitely need a usable
>>>>>> bootstrap script much more urgently than we needed it for Rivet1.
>>>>>>
>>>>>> Andy, what about renaming the current rivet2 bootstrap script to
>>>>>> rivet-2-bootstrap-lcg, and (me) creating a separate one which actually
>>>>>> does all the installations on its own (but without support for Rivet1
>>>>>> of course, to avoid it becoming too bloated as the old one did)?
>>>>>>
>>>>>> Cheers,
>>>>>> Frank
>>>>>>
>>>>>>
>>>>>> On 26 January 2014 17:13, Frank Krauss <frank.krauss at durham.ac.uk>
>>>>>> wrote:
>>>>>>> Dear Rivet Authors,
>>>>>>> I am just towards the end of my second hour running trying to
>>>>>>> install Rivet 2.0.  Admittedly, I am not half as geeky and nerdy as
>>>>>>> you are, so I accepted/anticipated that this would take me an hour.
>>>>>>> But, please, can you explain the rationale why you use
>>>>>>> cython, yaml, yoda, boost, fastjet & hepmc (I am sure i missed some!)
>>>>>>> in a non-trivial way with non-trivial version specifications without
>>>>>>> providing a meaningful installation guide?  You certainly do not
>>>>>>> want the common people to use your code - is that it?  If you think
>>>>>>> this helps the user base you are mightily mistaken.  If you think I
>>>>>>> will continue to advertise your code, you're delusional - this is
>>>>>>> next to uninstallable as it stands now, with a pretty superficial
>>>>>>> installation help.
>>>>>>> Best wishes
>>>>>>>      Frank
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Rivet mailing list
>>>>>>> Rivet at projects.hepforge.org
>>>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>>> _______________________________________________
>>>>>> Rivet mailing list
>>>>>> Rivet at projects.hepforge.org
>>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Dr Andy Buckley, Royal Society University Research Fellow
>>> Particle Physics Expt Group, University of Glasgow / PH Dept, CERN


-- 
Dr Andy Buckley, Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow / PH Dept, CERN