[Rivet] Fwd: Re: Problem with yodamerge

[Andy Buckley]

Oops, I didn't realise I'd taken the discussion with Hannes off-list.

Here's the status: looks like he was experiencing a side-effect of a 
hack around merge failures for completely empty histos. We fixed that in 
the latest YODA, so hopefully the hack can also be removed and all will 
be well...

Andy


-------- Forwarded Message --------
Subject: 	Re: [Rivet] Problem with yodamerge
Date: 	Tue, 13 Oct 2015 15:42:57 +0200
From: 	Hannes Jung <hannes.jung at desy.de>
To: 	andy.buckley at cern.ch



Hi Andy

ahhh.... I put by hand a normalization factor very small (so that
1/factor is very large),
so that when the histo is normalized with this factor it becomes zero...
but perhaps
this screws up when unpacking the normalized histos in yodamerge.

The problem occurred in previous versions when I had a histo with no
entires,
then yodamerge had problems with the normalization... this is why I
introduced this hack....  perhaps I should check again whether it works now
if I have sometimes a histo without entires and without normalization
factor...

Cheers
Hannes



> On 13.10.2015, at 15:33, Andy Buckley <andy.buckley at cern.ch
> <mailto:andy.buckley at cern.ch>> wrote:
>
> Hi Hannes,
>
> Yes, those ScaledBy values are quite extreme. They are created by the
> calls to scale() or normalize() in Rivet, and the value just
> corresponds to the scale/normalization factor. How are you managing to
> generate factors of nearly 10^100?!
>
> In the latest version of YODA, the --assume-normalized flag for
> yodamerge doesn't do anything. We changed the default logic since it
> seemed that lots of people were using that flag "blindly" to work
> around a class of problem for which it wasn't really the right answer.
> But since it's now a do-nothing flag it shouldn't be related to your
> ScaledBy problems.
>
> Andy
>
>
> On 13/10/15 11:05, Hannes Jung wrote:
>> Hi Andy, Chris
>>
>> thanks  a lot for your reply.
>> Yes, I am trying to merge normalised Histograms.
>> ahh, and yes, I have some strange Scaledby values: 5.1279402177630292e+97
>> I guess this is what causes the problem.
>> Where are those set, is this an issue of statistics ?
>>
>> The cpmmand I use is
>>
>> yodamerge --assume-normalized yoda-files
>>
>> Thanks again
>> Cheers
>> Hannes
>>
>>
>>
>>> On 13.10.2015, at 11:23, Andy Buckley <andy.buckley at cern.ch
>>> <mailto:andy.buckley at cern.ch>
>>> <mailto:andy.buckley at cern.ch>> wrote:
>>>
>>> Hi Hannes,
>>>
>>> What sort of data object are you merging here? From the y-axis it
>>> looks like a normalized histogram rather than {unnormalized histo,
>>> profile, ratio}.
>>>
>>> If that's right, yodamerge will look for the ScaledBy attribute of
>>> that plot to reverse the normalisation, add all the unnormalised
>>> histos together, then rescale by the weighted mean ScaledBy. It
>>> *shouldn't* be sensitive to one run having much smaller stats than the
>>> others, certainly not in a way that would show up like this.
>>>
>>> If you can provide a bit more detail of what this object is, and how
>>> you're trying to merge the runs, we can try to help. You can also plot
>>> or take a look inside the .yoda files to see if the individual runs
>>> look sane, i.e. have reasonable fill counts and ScaledBy values.
>>>
>>> Andy
>>>
>>>
>>> On 13/10/15 08:22, Hannes Jung wrote:
>>>> Dear Rivet experts
>>>>
>>>> I am using yodamerge to merge several yoda files (actually it is
>>>> 1000 or
>>>> so),
>>>> to get reasonable statistics in a reasonable computing time.
>>>> Sometimes I obtain strange results, like in the plot below,
>>>> where the normalization is completely off.
>>>> I am using Rivet 2.4.0, but a similar problem I had with Rivet 2.2.0.
>>>>
>>>> Any idea what this could be ?
>>>>
>>>> Since I am merging 1000 yoda files, it's hard to tell whether it comes
>>>> from one
>>>> specific file.... however, all jobs finished without error...
>>>>
>>>> Thanks a lot
>>>> Cheers
>>>> Hannes
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ***********************************************************************
>>>> Hannes Jung
>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>> mobile :+49 40 8998 93741
>>>> http://www.desy.de/~jung
>>>> Tel: +49 (0) 40 8998 3741
>>>> Fax: +49 (0) 40 8994 3741
>>>> DESY, CMS 01B/02.213
>>>> Notkestr.85, 22603 Hamburg, FRG
>>>> ***********************************************************************
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Rivet mailing list
>>>> Rivet at projects.hepforge.org
>>>> <mailto:Rivet at projects.hepforge.org><mailto:Rivet at projects.hepforge.org>
>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>>
>>>
>>>
>>> --
>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>> Particle Physics Expt Group, University of Glasgow
>>
>> ***********************************************************************
>> Hannes Jung
>> Email:Hannes.Jung at desy.de
>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>> mobile :+49 40 8998 93741
>> http://www.desy.de/~jung
>> Tel: +49 (0) 40 8998 3741
>> Fax: +49 (0) 40 8994 3741
>> DESY, CMS 01B/02.213
>> Notkestr.85, 22603 Hamburg, FRG
>> ***********************************************************************
>>
>>
>>
>
>
> --
> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
> Particle Physics Expt Group, University of Glasgow

***********************************************************************
Hannes Jung
Email: Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
mobile :+49 40 8998 93741
http://www.desy.de/~jung
Tel: +49 (0) 40 8998 3741
Fax: +49 (0) 40 8994 3741
DESY, CMS 01B/02.213
Notkestr.85, 22603 Hamburg, FRG
***********************************************************************