[Rivet] installation

[Andy Buckley]

On 27/01/14 14:09, Frank Krauss wrote:
> Look, Andy, it is quite simple: I burnt three hours yesterday to no
> avail.  Without Frank S' help producing a bootstrap I would have
> stopped.  I do know that others stopped.  Your first, and to some degree
> also the second answer, did not convince me really that you are aware of
> this.

Exactly, I wasn't aware of this at all!

> Even worse, I now have the strong perception of a "so what, you
> cannot do it - tough luck on you" attitude.  So, my experiences with
> Rivet 2 up to now will guarantee that my Rivet advertisement will sound
> like
> 
> "It could be a nice tool, but unfortunately in its latest versions it
> comes with so many dependencies that I cannot recommend to use it now. 
> It is just not matured enough and the feedback by the core developer was
> less than helpful."
> 
> I am really annoyed by this, asking around here also told me that I am
> not the first to complain.  So, thanks for helping me to an enjoyable
> afternoon.

I've genuinely no idea what these "other complaints" were -- there's a
lot of incoming email, and maybe I'm forgetting them... in which case
apologies to those involved. But we *really* try to be helpful if there
are specific problems, and to gradually do what we can about "big
picture" feedback like "too many dependencies". Most feedback that I get
is positive, and private rumbles of discontent that never make it into a
feedback email may as well have not happened. At least that's my best
guess at what's happened, and I wish we could have done something better
to avoid getting to this point.

Fixes to the bootstrap scripts and other tweaks to the build system have
largely happened in response to bug reports, which is great -- and we
*fixed* those, usually quickly. I really don't recall any general
complaint about the dependencies having got much worse... because,
certainly for tarball users, they _haven't_ changed much. Most of the
dependencies you mentioned have been there, virtually without complaint,
for many years: they are the reason for there being a bootstrap script
in the first place, i.e. since about 2006. I'd *like* to have fewer
dependencies, but it's not obvious how to go about that.

As Frank S just made clear, the big problem seems to be for people who
want to install the development trunk in order to get new, _unreleased_
analyses. I didn't even know that people were doing that: the
experimenters I'm in touch with certainly aren't. It reopens (difficult)
questions about whether we can make analysis releases more timely, but I
think you need to appreciate that building an unreleased Rivet branch
from version control isn't something that we advise anyone to do, nor
has any particular effort been spent on making it easy. It specifically
requires several non-trivial developer tools which aren't needed at all
for the releases, so for people who take the "standard route" to use
Rivet the issue never arises. Apologies for any "touch luck" subtext
that made it into my reply, but that's a bit like me checking out the
Sherpa 2 trunk a year ago and then complaining to you that your
development code is unstable and unprofessional!

So I'm very disappointed if you'd now like to tell people that Rivet
support is crap because of a bad experience with the development branch:
most of the time, for supporting released versions, I think we're super
helpful. Sorry again if my responses were a little terse, but check the
style of the incoming emails and maybe that is understandable. I
definitely care and want to help, but the way to do that is to make the
releases match what you need, not to encourage people to use the dev
branch. That's why I support Frank S' script to help non-LCG users get
the tarballs installed on odd systems, but *not* a public bootstrap of
the next release branch so that people can run unreleased code.

I see David just expressed the same thing in fewer words, but will send
anyway since there's more history/context stuff above in case you care,
and I want you to know that I care deeply about making Rivet work for
all our users, but the interaction has to happen in the right way.

Andy





> On 27/01/14 12:03, Andy Buckley wrote:
>> On 27/01/14 11:17, Frank Krauss wrote:
>>> Hi Andy,
>>> well, that's great.  I found out about the dependencies the hard way,
>>> but I still did not manage to produce an installation.  And since you
>>> guys appear to claim zero time of developer time it has about zero
>>> usability at the moment for anyone who cannot rely on it being already
>>> installed on lxplus.  I guess however that Rivet was also meant for the
>>> wider MC community, including theorists not sitting at CERN.  For those
>>> it is pretty much useless, and, the way I see it, because the developers
>>> mainly aimed at proving how smart they are rather than at having
>>> something useful.  So, up to now, you just showed me that you guys do
>>> not give a toss about this part of the community.  Good to know.  I
>>> sincerely hope this attitude of producing monumental software for the
>>> nerds is not the new trend in HEP computing.
>> I'll try to reply nicely... but not exactly the style of feedback that
>> makes me want to be Mr Helpful, Frank.
>>
>> First off, of course we care about MC/pheno theorists. But as far as I
>> was aware the part of the community that Rivet was started for (i.e. you
>> guys) were more-or-less happily using it without significant trouble. So
>> the recent work (shall I moan about manpower again?) has been aimed at
>> getting the experiment uptake to the same level as in the generator
>> community. Understandable, no? So it looks like time for the balance to
>> swing a bit back toward the non-LCG users again.
>>
>> I _really_ hope we have better ways to look (be!) clever than introduce
>> new pain-in-the-ass library dependencies. It's more the other way
>> around: when there's a better implementation somewhere else, which we
>> have no chance/time to reimplement internally, we had to be stupid and
>> acquire the dependency. Most of this happened > 5 years ago. Where there
>> have been changes they have been to *reduce* build system problems (cf.
>> the previous way of bundling yaml-cpp). I didn't think that much had
>> changed with the release of version 2, but clearly some changes have had
>> painful impact: it would be good to know exactly what these are. As I
>> mentioned in another email, I suspect that YODA's Boost version
>> requirement is too high -- is that right?
>>
>> The dependencies that we have were chosen (I thought fairly carefully)
>> for compatibility with the "canonical" HEP system, i.e. SLC6 and the LCG
>> libraries. Frank's new script should handle tarball builds on systems
>> where LCG isn't present, which is great. Installation on personal
>> laptops _should_ be far easier because so many dependencies can come
>> direct from system packages... but you're right that that's assuming a
>> degree of nerdity in that you're already administrating your own machine
>> so can presumably choose what to install from where. If not, Frank's new
>> "install everything" script should help... but be aware that if you
>> build one library with a system Boost/GSL/whatever and Rivet with this
>> all-in-one bootstrap then the fact that it fails to run isn't really a
>> Rivet problem! Finally, building an unreleased development version is of
>> course a bit harder: heck, *I* don't try to build the developer version
>> on lxplus direct from the repository: I make a tarball on my laptop,
>> copy it over, and run the standard bootstrap.
>>
>> Ok, enough. The feedback and the more general bootstrap script are
>> really good to have, and I wish we'd known the strength of feeling about
>> Rivet much earlier... but if no-one reports it then we don't know.  Let
>> us know (nicely :-P) if there are more problems -- particularly,
>> concrete help like Frank S' is hugely appreciated.
>>
>> Andy
>>
>> PS. Current manpower, since there was incredulity: I would guess < 1
>> day/month on average for me, plus something similar or less for David G,
>> and occasional (very welcome!) contributions from others like Frank S.
>> Much of that time goes on analysis integration and (slow) feature
>> development. Does it look like more from outside?
>>
>> PPS. When yaml-cpp is bundled back inside Rivet, the tarball
>> dependencies will be exactly the same as Rivet 1, except that the
>> histogramming now has to be built separately. So I think the 1 vs. 2
>> dichotomy is a false one, and the issue is more to with the change to
>> bootstrap scripts. You'll probably still need to have cmake installed,
>> unless we bundle that as well... and why not an entire operating system?
>> But while I dislike that cmake requirement, it's the same for a *lot* of
>> other HEP packages now since LCG standardised on it. There will be a lot
>> of pressure for HepMC3 to go that way, I think.
>>
>>
>>> On 26/01/14 22:59, Andy Buckley wrote:
>>>> Hi Franks,
>>>>
>>>> Well, it's usable in proportion to the amount of developer time that we
>>>> have had to spend on making it nice... the main users what we have had
>>>> to win over are the experiment ones who access it through experiment
>>>> frameworks or from the Genser-built copy. So certainly that's who I've
>>>> been aiming my limited build system dev time at, and to be honest I
>>>> still stand by that choice although I'd rather not have *anyone*
>>>> annoyed
>>>> by our code.
>>>>
>>>> For a tarball build I think the dependencies are as follows:
>>>>
>>>> YODA requires:
>>>>    * Boost (headers only, but currently asks for quite a new version...
>>>> can maybe be relaxed: any testers?)
>>>>
>>>> Rivet requires:
>>>>    * HepMC, FastJet (unavoidable, I think)
>>>>    * YODA (yes, could have been built-in, but it has wider utility)
>>>>    * yaml-cpp (for 2.1.0 I'd like to use a built-in copy, cf. the
>>>> latest
>>>> LHAPDF6)
>>>>    * GNU Scientific Library (volunteers to eliminate that
>>>> dependency? But
>>>> is typically an easily-met requirement, I think)
>>>>    * Boost (headers only, fairly easy version requirement, I think)
>>>>
>>>> If we eliminate yaml-cpp, how much is there to object to? Anyway, yes
>>>> Frank I'd be very happy for you to make a new rivet2-bootstrap script
>>>> which builds everything from scratch... and/or attempts to work out
>>>> what
>>>> is already on the system? At some level users should make their own
>>>> minds up about what to install from packaging systems and what by
>>>> hand... but by saying that I probably give away what side of Frank's
>>>> nerd fence I'm sitting on ;-)
>>>>
>>>> Anyway, in short I don't see so much room for cutting things down, but
>>>> always think that smoothing the entry paths (with scripts or otherwise)
>>>> is a good thing as long as it's kept manageable for the developers,
>>>> too.
>>>> Since that's translated as "next to no manpower" for the last... two
>>>> years?, I'm afraid the situation isn't so surprising. But we'll do what
>>>> we can -- certainly I want to get rid of the yaml-cpp explicity
>>>> requirement, and hopefully we can get a couple of new part-time
>>>> developers (re)activated in the coming months.
>>>>
>>>> Andy
>>>>
>>>>
>>>> On 26/01/14 19:47, Frank Siegert wrote:
>>>>> I hate to admit that I agree with Frank's rant here, and I have had
>>>>> multiple users recently who made the same point, though admittedly
>>>>> more politely. The bootstrap script currently is close to useless to
>>>>> everybody who isn't installing on lxplus. I have it on my agenda to
>>>>> replace it with a working version, but haven't got around to it yet.
>>>>> And to be honest, all the changes and external dependencies that came
>>>>> with Rivet2 have left me as a second-class developer (if even),
>>>>> because there is a lot of stuff in the codebase which is beyond my
>>>>> capabilities to fix -- I'm normally quite annoyed myself with even
>>>>> installing Rivet on a new cluster, where
>>>>> cython/yaml/yoda/boost/cmake/hg/fastjet/hepmc... isn't available. Some
>>>>> of those are only necessary for the development version, but it's
>>>>> still rather annoying.
>>>>> This goes far beyond the switch to YODA, which I agree with as
>>>>> necessary, though tbh I originally thought would be Rivet-in-house as
>>>>> well.
>>>>>
>>>>> Anyway, as I'm clearly not the majority of Rivet developers, and
>>>>> mostly everybody else on the team seems to be happy about this usage
>>>>> of fancy new technologies, separation and dependencies, there is no
>>>>> point in discussing that in itself. But we definitely need a usable
>>>>> bootstrap script much more urgently than we needed it for Rivet1.
>>>>>
>>>>> Andy, what about renaming the current rivet2 bootstrap script to
>>>>> rivet-2-bootstrap-lcg, and (me) creating a separate one which actually
>>>>> does all the installations on its own (but without support for Rivet1
>>>>> of course, to avoid it becoming too bloated as the old one did)?
>>>>>
>>>>> Cheers,
>>>>> Frank
>>>>>
>>>>>
>>>>> On 26 January 2014 17:13, Frank Krauss <frank.krauss at durham.ac.uk>
>>>>> wrote:
>>>>>> Dear Rivet Authors,
>>>>>> I am just towards the end of my second hour running trying to
>>>>>> install Rivet 2.0.  Admittedly, I am not half as geeky and nerdy as
>>>>>> you are, so I accepted/anticipated that this would take me an hour.
>>>>>> But, please, can you explain the rationale why you use
>>>>>> cython, yaml, yoda, boost, fastjet & hepmc (I am sure i missed some!)
>>>>>> in a non-trivial way with non-trivial version specifications without
>>>>>> providing a meaningful installation guide?  You certainly do not
>>>>>> want the common people to use your code - is that it?  If you think
>>>>>> this helps the user base you are mightily mistaken.  If you think I
>>>>>> will continue to advertise your code, you're delusional - this is
>>>>>> next to uninstallable as it stands now, with a pretty superficial
>>>>>> installation help.
>>>>>> Best wishes
>>>>>>       Frank
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Rivet mailing list
>>>>>> Rivet at projects.hepforge.org
>>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>> _______________________________________________
>>>>> Rivet mailing list
>>>>> Rivet at projects.hepforge.org
>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>>
>>
> 


-- 
Dr Andy Buckley, Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow / PH Dept, CERN