[Rivet] Rivet boost::BOOST_FOREACH

[Andy Buckley]

Hi David,

Yes, I think this bump is the best idea. I don't know how and why this 
inconsistency happens, especially as we're using the recipe recommended 
by Boost for that version... oh well, we just take the easiest route to 
have a working system, acknowledging that some past versions of Boost 
have been (in retrospect) buggy.

For Rivet 3.0 (which we were hoping to produce before year-end) I am 
very inclined to make the leap to full C++11 and do what we can to ease 
running on vanilla SL6 systems...

Andy


On 11/08/15 10:19, David Bjergaard wrote:
> Hi Andy, Andrii,
>
> I just ran into this very same problem with 1.53.  Installing 1.56 as per the
> bootstrap prescription fixed it.  I've always used 1.55 previously so maybe its
> worth bumping the configure script to expect 1.55 or higher? This doesn't change
> the bootstrap prescription and it avoids a seemingly broken build.  Once
> configure finishes, the rest of the build should "just work".
>
> Cheers,
>
>      Dave
>
> Andy Buckley <andy.buckley at cern.ch> writes:
>
>> Hi again Andrii,
>>
>> This myriad of compilers and Boost versions that are incompatible with
>> Boost's own "pretty foreach" recipe is very surprising to me. We've
>> tested with a variety of common systems for every release since at
>> least 2010, and always made sure that it was usable... modulo the
>> requirement to install a more recent Boost than the ancient SL5/6
>> system copies.
>>
>> But we didn't check *all* versions of Boost for compatibility, because
>> Boost's own documentation didn't suggest that there would be any
>> issue. They are doing complicated tricks internally, and "unbuggy"
>> Boost releases have frequently become buggy as new, stricter compilers
>> have come into use. I'm very surprised that you report Boost 1.53 as
>> incompatible, since that version's documentation specifically
>> recommends the recipe that we use, but to the best of my knowledge I
>> have not personally tried that version.
>>
>> It's good to know that there is this problem, but I think our response
>> just has to be to increase the required Boost version. We have never
>> been able to use the SL system version, so this is no change... and if
>> you use our bootstrap script it will make a sufficient Boost
>> installation automatically (without needing to compile it, since we
>> use only headers).
>>
>> Sorry for the inconvenience, but glad you found a workaround for your
>> system.
>>
>> Andy
>>
>>
>>
>> On 05/08/15 12:13, Andrii Verbytskyi wrote:
>>> Dear Andy,
>>> 1)
>>>>>>
>>>>>> http://www.boost.org/doc/libs/1_58_0/doc/html/foreach.html#foreach.introduction.making__literal_boost_foreach__literal__prettier
>>>>>
>>>>> But that is not 1.55 you are using and might not (and it doesn't!) fit
>>>>> Rivet >1.48 requirement. Quite confusing.
>>>>
>>>> I just picked the first result that came up in Google. That instruction
>>>> has been in place since version 1.52 in 2012, because of problems
>>>> introduced by clashes with the internal use of a "foreach" identifier.
>>>> Sounds like we should update the Rivet Boost version requirement, if
>>>> 1.48 is causing trouble -- we're not testing on a platform with that
>>>> version so it's good to know there's a problem.
>>>
>>> The >1.47 requirement is in the Rivet *now*. I don't know why it is
>>> there and if it causes problems.
>>> Maybe the workaround is present there since 1.52, but it clearly fails
>>> with 1.53.
>>> Also, it looks SL7 will stay with 1.53 and making 1.55 a minimal
>>> requirement will make Rivet incompatible with the default and most used
>>> HEP OS of the next 5 years.
>>>
>>>
>>> 2)
>>>>>> I have a feeling that this bug relates to the version of Boost you're
>>>>>> using, in conjunction with more recent compilers.
>>>>>
>>>>> Well, I try to stick to the default systems, I do not want to introduce
>>>>> my custom versions for boost, gcc, glibc and so on. Now my system is
>>>>> vanilla CentOS7.
>>>>
>>>> Maybe someone else on this list can confirm, but it sounds like the
>>>> default GCC + Boost versions on CentOS 7 might be a problematic combination.
>>>
>>> It also fails with clang.  I don't have other compilers and time to try.
>>> I cannot try other boost as recompilation of boost requires too much
>>> external libraries and there is no binaries(rpms).
>>>
>>>
>>>
>>>
>>>
>>> 3)
>>>>
>>>>>> I recall that an
>>>>>> updated boost.m4 script that we introduced in the recent version 2.3.0
>>>>>> addresses the problem...
>>>>>
>>>>> No, I've tried 2.2 and 2.1 as well.
>>>>
>>>> I mean that versions before 2.3 might have the problem while 2.3 is ok.
>>>> Or does 2.3.0 also get through your configure step and then fail at
>>>> compile time?
>>>
>>> Sure.
>>>
>>>
>>> 4)
>>>>>> or rather it detects that there will be a
>>>>>> problem and requires a consistent GCC+Boost installation.
>>>>> It requires boost>1.48 or so and nothing more to solve r/lvalue problem.
>>>>>
>>>>>
>>>>>> The *real* solution will be to require and use the C++11 range-for
>>>>>> construct instead of Boost's macro. This will also work a lot better for
>>>>>> e.g. iterating over std::maps, which don't work well with the macro
>>>>>> approach. And of course we can replace a lot (but not all, I think)
>>>>>> Boost functionality in Rivet with C++11 standard features
>>>>> As I see, foreach appears in quite simple constructions for which even
>>>>> C++0x will work fine. I mean "for (std::<blabla>::iterator it=...)"
>>>>> one will need  one or zero more lines. In total, that would require
>>>>> ~200-300 lines.
>>>>
>>>> Yes, but Rivet's whole design philosophy is aligned with the "make
>>>> simple things simple quoted on the Boost foreach web page". We've been
>>>> using this functionality for more than 5 years! We're not going to take
>>>> a backward step to use iterators unnecessarily when a) the current use
>>>> of BOOST_FOREACH works on all the systems we've tested, including LCG
>>>> platforms, and b) the properly language-integrated version of that macro
>>>> is so close to being usable.
>>>
>>> Sure, BOOST_FOREACH looks nice and in the Rivet it also works on some
>>> "specially prepared" systems. But "so close" implies "is not", which
>>> means it will fail on other, "ordinary" systems.
>>>
>>>
>>>
>>> 5)
>>>>>
>>>>>> , so it's very
>>>>>> appealing to many of us. But we don't feel that non-experiment HEP
>>>>>> computing resources are yet in a state where we can make that C++11
>>>>>> requirement without causing pain to many theory users.
>>>>>
>>>>> Actually the experementalists are even less flexible with the software.
>>>>> In many cases it is _very_ hard and inconvenient  to install somewhere a
>>>>> custom version of compiler, boost and so on in the userspace. Some
>>>>> programs will not work with custom versions at all. And custom versions
>>>>> for every single library is a direct way to dependency hell. For  many
>>>>> physics software packages I would prefer to have less features but
>>>>> easier installation and compatibility.
>>>>
>>>> What I meant is that the SL *system* versions of these packages are
>>>> often 5+ years out of date and not sufficient, but that experiment
>>>> software environments usually use much newer versions of all our
>>>> dependencies. We could move to requiring C++11 and the experiment/LCG
>>>> environments would have no problem, but the system compilers would not
>>>> be sufficiently capable.
>>>
>>> I would not blame compilers or old software:
>>>
>>>
>>> https://gcc.gnu.org/gcc-4.8/
>>> GCC 4.8.3 May 22, 2014
>>> http://www.boost.org/users/history/
>>> Version 1.53.0 February 4th, 2013 18:29 GMT
>>> http://llvm.org/
>>> 3.4.2: June 2014
>>>
>>> 2013-10-25  Andy Buckley  <andy.buckley at cern.ch>	
>>> * 2.0.0 release!!!!!!
>>> 2014-02-06  Andy Buckley  <andy.buckley at cern.ch>
>>> * 2.1.0 release!
>>> 2014-09-30  Andy Buckley  <andy.buckley at cern.ch>
>>> * 2.2.0 release!
>>> 2015-06-30  Andy Buckley  <andy.buckley at cern.ch>
>>> * Bump version to 2.3.0 and require YODA > 1.4.0 (current head at
>>>
>>>
>>>
>>>
>>> 6)
>>>>>> For what it's worth, I'm currently using GCC 4.9.2 and Boost 1.55 and it
>>>>>> works fine. In the pre-release testing that we do on CERN lxplus we use
>>>>>> the system compiler but install our own 1.55 version of Boost.
>>>>>
>>>>> As far as I understand other versions don't  work.
>>>>> GCC 4.9.2 and Boost 1.55 are not default on the lxplus.cern.ch,
>>>>> bastion.desy.de, gate.rzg.mpg.de, (fnal?) and many others. If it doesn't
>>>>> work there, one should use custom version.
>>>>
>>>> As I said, the defaults on HEP systems are very often not sufficient.
>>>> That's why the experiments override them with full up-to-date
>>>> environments.
>>>
>>> For obvious reasons that will not work for any of past (i.e. non-LHC)
>>> experiments (significant fraction of Rivet analyses). The machines and
>>> software are either "frozen" or don't exist anymore (implies using
>>> public computing resources, mostly with default versions of compilers
>>> and libraries). In some cases the old software will not work with
>>> up-to-date software.
>>>
>>>
>>>
>>> 7)
>>>>> Anyway,
>>>>> it looks it works (compiles!) for me with a trivial patch (see
>>>>> attached).
>>>>
>>>> Which is exactly what we used to have, and reversed because of the
>>>> problems that it can have and the updated Boost solution. I'm glad it
>>>> works for your package/compiler combination, but we won't apply it upstream.
>>>
>>> Sure. The proper solution would be to replace BOOST_FOREACH or to have
>>> different versions of that part of code for different boost versions.
>>> (Sorry, no time to prepare that patch.)
>>>
>>>
>>>
>>> Best regards,
>>> Andrii
>>>
>>>
>>>> Andy
>>>>
>>>>
>>>>>> On 04/08/15 13:05, Andrii Verbytskyi wrote:
>>>>>>> Dear Andy,
>>>>>>> I just tried to compile 2.* versions of Rivet on CentOS
>>>>>>> 7/gcc4.8.3/boost1.53.
>>>>>>> All those attempts failed, the compiler doesn't accept
>>>>>>> the Rivet code:
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>>       CXX      libRivetCore_la-Run.lo
>>>>>>> In file included from ../../include/Rivet/Run.hh:5:0,
>>>>>>>                      from Run.cc:2:
>>>>>>> ../../include/Rivet/Tools/RivetBoost.hh:16:36: error: declaration of
>>>>>>> namespace 'boost::BOOST_FOREACH' conflicts with
>>>>>>>        namespace BOOST_FOREACH = foreach;
>>>>>>>                                         ^
>>>>>>> In file included from /usr/include/boost/foreach.hpp:89:0,
>>>>>>>                      from ../../include/Rivet/Tools/RivetBoost.hh:12,
>>>>>>>                      from ../../include/Rivet/Run.hh:5,
>>>>>>>                      from Run.cc:2:
>>>>>>> /usr/include/boost/foreach_fwd.hpp:56:1: error: previous declaration of
>>>>>>> namespace 'boost::BOOST_FOREACH' here
>>>>>>>      {
>>>>>>>      ^
>>>>>>> In file included from ../../include/Rivet/Math/MathUtils.hh:5:0,
>>>>>>>                      from ../../include/Rivet/Math/Math.hh:5,
>>>>>>>                      from ../../include/Rivet/Tools/Utils.hh:5,
>>>>>>>                      from ../../include/Rivet/Config/RivetCommon.hh:26,
>>>>>>>                      from ../../include/Rivet/AnalysisHandler.hh:5,
>>>>>>>                      from Run.cc:3:
>>>>>>> ../../include/Rivet/Math/MathHeader.hh:45:16: warning: 'Rivet::MAXINT'
>>>>>>> defined but not used [-Wunused-variable]
>>>>>>>        const double MAXINT = std::numeric_limits<int>::max();
>>>>>>> ----------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> It looks the reason is a well known (since 4 years!) BOOST bug.
>>>>>>> 1) Which compilers do you use for tests, i.e. with which compiler Rivet
>>>>>>> will work?
>>>>>>> 2) Has anyone tried to compile Rivet with (new)gcc?
>>>>>>> 3) Is it possible to drop that construction from Rivet? It looks it
>>>>>>> doesn't appear too often in the code and when it does the replacement is
>>>>>>> obvious.
>>>>>>>
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Andrii
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 2014-12-05 at 14:10 +0100, Andrii Verbytskyi wrote:
>>>>>>>> Hi,
>>>>>>>> yes, environment variables are working, however passing arguments via
>>>>>>>> environment variables is implicit way and implicit way is worse that
>>>>>>>> explicit. Also, in many cases environment variables will make things
>>>>>>>> less convenient or even defunct. A good example is using rivet
>>>>>>>> from a Makefile. One can say the same about using rivet in system() and
>>>>>>>> so on. Also, the way of setting environment variables depends on shell
>>>>>>>> --> less convenient usage of rivet even from shell.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Andrii
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, 2014-12-05 at 13:00 +0000, Andy Buckley wrote:
>>>>>>>>> Hi Andrii,
>>>>>>>>>
>>>>>>>>> Don't the environment variables work for these? --analysis-path is just
>>>>>>>>> syntactic sugar for those, and to be honest I was considering removing
>>>>>>>>> it to clean up the command-line interface!
>>>>>>>>>
>>>>>>>>> Andy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/12/14 12:44, Andrii Verbytskyi wrote:
>>>>>>>>>> Dear Andy,
>>>>>>>>>> here are some (hopefully useful) improvements for Rivet.
>>>>>>>>>>
>>>>>>>>>> In addition to --analysis-path I think it would be useful to have
>>>>>>>>>>
>>>>>>>>>> --ref-path, --info-path, --plot-path.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So here are the files with the changes. Please note that rivet should be
>>>>>>>>>> generated from rivet.in with a substitution of install paths.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Andrii
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>


-- 
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow