[Rivet] C++11 plan for Rivet

[Andy Buckley]

On 15/08/16 11:46, Frank Siegert wrote:
> Hi Andy,
>
> On 14 August 2016 at 00:24, Andy Buckley <andy.buckley at cern.ch> wrote:
>> On 11/08/16 10:43, Frank Siegert wrote:
>>>
>>> On the same note, I realised that we should document on the website
>>> which gcc versions we support (so far this is neither in the 2.5.0
>>> release announcement, nor on the Getting Started page, or did I miss
>>> it?). For example: are we happy with >=4.7 as still used in
>>> experiments a lot, or do we need the features introduced in 4.8
>>> (https://gcc.gnu.org/projects/cxx-status.html#cxx11). Since we are one
>>> of the first projects in the MC landscape to switch to a requirement
>>> of C++11, we should provide the users with some guidance, and also
>>> mention that they have to export CXXFLAGS=-std=c++11 when compiling
>>> with an older compiler.
>>
>> Yes, we should explicitly mention this in the docs. I think we need 4.8 --
>> isn't this the version used by the experiments?
>
> ATLAS evgen happens with gcc 4.7 (Athena release 19.2.5.x) at the moment.

Ok. But Rivet isn't run in production (right? there were some plans to 
use Rivet in Grid production monitoring via the JEM system, but I'm not 
sure what happened to that... and the latest versions are not 
essential), and there are Athena r20.* releases runnable on the Grid 
into which the C++11 Rivet+YODA can be (and is being) installed.

I don't want to dismiss this, but I think there's an easily workable 
standard way in ATLAS to run the new versions, without us having to jump 
through compatibility hoops.

>> I've not tested with
>> anything earlier. The compiler flags are automatically passed to analysis
>> plugin building and the output of rivet-config: is it sufficient to advise
>> using the latter to get the appropriate extra flags for building code units
>> that use the Rivet API?
>
> I have tested building with gcc 4.7.1, but this already fails during
> configure due the "test_variadic_templates" in
> m4/ax_cxx_compile_stdcxx_11.m4

Hmm, we're not using that feature, but the testing macro is looking for 
*full* compliance... and I side with David on preferring that to the 
mess of establishing whether compilers support the appropriate (and 
potentially changing) subset of C++11. I'm tempted to just say GCC >= 
4.8 for Rivet >= 2.5 and YODA >= 1.6, to keep it simple.

> To see whether we really use gcc 4.8 functions, I disabled the
> configure checks, and the only place where compilation fails with 4.7
> is the map::emplace in Logging.cc and the special normalize functions
> in MC_MET.cc. Don't know whether it's possible/worth replacing these
> two and run it through the test suite. Thus we would have a Rivet
> version which still builds with 4.7 in the short term while that's
> still used in the community?
> This is of course "only" a social argument, which runs contrary to
> David's strong opinion about not caring about backwards compatibility,
> which I can understand from a programmer's point of view as well.

Thanks for doing the digging. Useful to know that it only fails on 
those... they are obviously work-aroundable.

I'm surprised that it fails on those "special normalize" functions, 
since they're just using initializer list syntax, not variadic templates 
or anything similarly funky. Hmm. Maybe that version needs an extra set 
of braces to convince the compiler that it knows what's going on.

But that serves as an illustration of why this is difficult... once we 
start supporting partial implementations of C++11, we have to be 
super-careful that anything we add is not just valid C++11, but valid 
within the partial support and bugginess of GCC 4.7. Yikes.

So we could do this, and I'm sure we *will* do it if necessary, but my 
gut feeling is that we're better off making a clean break and fully 
using all (nice) new language features, specifically because the 
experiments can still use Rivet via Grid-distributed standard releases, 
just not the same ones as used in sample production. Given that people 
don't use the sample production releases for analysis, either, this 
doesn't strike me as a major obstacle, but maybe I'm biased.

Cheers,
Andy


>>> On 24 March 2016 at 11:55, Andy Buckley <andy.buckley at cern.ch> wrote:
>>>>
>>>> Yes, I agree. This should be an opportunity to clean up and make our API
>>>> more expressive, not to introduce more fallbacks and complexity.
>>>>
>>>> Andy
>>>>
>>>>
>>>>
>>>> On 24/03/16 10:27, David Grellscheid wrote:
>>>>>
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> I'm all for the transition to C++11. But if we do it, I vote for a hard
>>>>> switchover. No backwards compat flags, #ifdefs falling back to Boost,
>>>>> etc. Once a function is converted, it stays that way. We should assume a
>>>>> fully-compliant C++11 compiler and not have to worry about use of
>>>>> individual features, and when they started being supported by which
>>>>> version of gcc.
>>>>>
>>>>> See you,
>>>>>
>>>>>    David
>>>>>
>>>>>
>>>>>
>>>>> On 23/03/2016 22:19, Andy Buckley wrote:
>>>>>>
>>>>>>
>>>>>> On 17/03/16 10:27, Holger Schulz wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17/03/16 09:19, Andy Buckley wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm currently hacking together a prototype physics object smearing
>>>>>>>> system for Rivet 2.5 or 2.6, to be beta-tested with the BSM
>>>>>>>> experimental groups and re-casters. No capitulation: we will continue
>>>>>>>> to require unfolded "proper" measurements, but there is BSM demand
>>>>>>>> for
>>>>>>>> Rivet and they will not bite unless we can provide at least some
>>>>>>>> simple machinery for efficiency curves.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I completely agree now that the ATOM guys have essentially decoupled
>>>>>>> their efforts from Rivet.
>>>>>>> I did something like that in my ATLAS analysis for random removal of
>>>>>>> tracks based on track
>>>>>>> reco efficiency and I know that there is demand to have that in rivet,
>>>>>>> too.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Somehow I was interested enough in this to spend a bit of time hacking
>>>>>> a
>>>>>> start together. Here it is:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://rivet.hepforge.org/hg/rivet/file/tip/include/Rivet/Projections/SmearedParticles.hh
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://rivet.hepforge.org/hg/rivet/file/tip/include/Rivet/Projections/SmearedJets.hh
>>>>>>
>>>>>>
>>>>>>
>>>>>> (Connoisseurs of borderline-evil code may like to note how the function
>>>>>> hash for projection comparison is being done ;-) But no worse than what
>>>>>> you need to do to use POSIX dlsym... aka the evil_cast in AGILe's
>>>>>> module
>>>>>> loader.)
>>>>>>
>>>>>> And here's how the SmearedJets looks in action in an analysis:
>>>>>>
>>>>>> FastJets fj(FinalState(Cuts::abseta < 5), FastJets::ANTIKT, 0.4);
>>>>>> SmearedJets sj(fj, JET_EFF_ONE, JET_SMEAR_IDENTITY);
>>>>>>
>>>>>> Ok, those are do-nothing standard perfect efficiency and smearing
>>>>>> functions, but the principle works. I'm populating a file with a few
>>>>>> real efficiency and smearing functions... but they will strongly
>>>>>> benefit
>>>>>> from going to mandatory C++11 so we can use the new random generator
>>>>>> machinery. And here's another C++11 killer feature for this use-case --
>>>>>> inline lambda functions:
>>>>>>
>>>>>> SmearedJets sj(fj,
>>>>>>                 [](const Jet& j) { return 1 - exp(-j.pT()/(10*GeV)); },
>>>>>>                 [](const Jet& j) { return j; });
>>>>>>
>>>>>> I think this stuff makes a pretty good case for going to C++11 with the
>>>>>> addition of this feature... and then gradually converting lots of our
>>>>>> internal Boost and hackery to use the much nicer new language features
>>>>>> instead.
>>>>>>
>>>>>>>> The way I'm designing it can all be done with Boost features, but
>>>>>>>> once
>>>>>>>> again it's stuff that is part of the core language in C++11 and using
>>>>>>>> the less-standard Boost implementations feels like going in the wrong
>>>>>>>> direction. Also, we are starting to see submitted analyses which use
>>>>>>>> C++11 features, and it both feels unnecessarily restrictive on our
>>>>>>>> "clients" and is extra work for us to have to revert that code to
>>>>>>>> C++98.
>>>>>>>>
>>>>>>>> So I would like us to make the switch to mandatory C++11 building of
>>>>>>>> Rivet in the next couple of 2.x releases. This would also help us to
>>>>>>>> reduce the currently huge number of "paradigm shifts" scheduled (for
>>>>>>>> lack of imagination) on v3.0.
>>>>>>>>
>>>>>>>> There is one major sticking point, in the form of FastJet. While
>>>>>>>> C++11
>>>>>>>> compatible, it makes use of auto_ptr and exposes that in its public
>>>>>>>> headers, meaning that anyone compiling a Rivet analysis in C++11 mode
>>>>>>>> gets a terminal output dominated by FJ auto_ptr deprecation warnings.
>>>>>>>> There seems to be no way in GCC to disable these warnings -- or does
>>>>>>>> someone know of one? So I think we need to put a bit of pressure on
>>>>>>>> FastJet to make a non-complaining release; I already did this ~6
>>>>>>>> months ago and was told that they are working on a major new
>>>>>>>> development, but it has not appeared and the issue is more urgent now
>>>>>>>> (I don't know what the experiments are doing re. this). So I'll prod
>>>>>>>> them again and hopefully we'll be able to make this switch in Rivet
>>>>>>>> 2.5 or 2.6.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We can just ask again, maybe point them to this forum:
>>>>>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59325
>>>>>>
>>>>>>
>>>>>>
>>>>>> Did that: thanks for the link :-) And the timing was good: FastJet
>>>>>> 3.2.0
>>>>>> is out now and at the last minute they removed auto_ptr from fjcore and
>>>>>> added a configure-time option to disable the auto_ptrs in the main FJ
>>>>>> interface.
>>>>>>
>>>>>> And it looks like LCG can be persuaded to build their FJ 3.2.0
>>>>>> installations using that flag, so once there is such a bundle to play
>>>>>> with, we can make a mandatory C++11 release. Which I would like to use
>>>>>> for beta-testing the new smearing features with interested parties.
>>>>>>
>>>>>> Cheers,
>>>>>> Andy
>>>>>>


-- 
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow