[Rivet] yodamerge and normalisation

[Andy Buckley]

Hi Hannes,

I think this sort of case is better dealt with by *not* normalizing the 
DY+jet histos, or normalizing them to an area of 1. Then use yodamerge 
to combine both the Z+jet and DY+jet pt plots and write a (very) little 
Python script to do the last bit:

import yoda
hs = yoda.read("merged.yoda")
norm = hs["/MYANA/DYpt"].integral()
hs["/MYANA/DYpt"].normalize(norm)
yoda.write(hs, "rescaled.yoda")

Not tested, but that's the basic idea. Let us know if that works for you.

Andy


On 14/10/15 18:23, Hannes Jung wrote:
> Hi Andy et al
>
> the new yodamerge works much better now, this is really great.
>
> But I have now another problem (which I tried to solve with the hack
> from before):
>
> I fill histograms and normalize them to some xsection.
> Now it can happen, that the histo is filled, but the normalization is zero,
> for example when I plot the pt of DY+jet normalized to Z+jet, then it
> can happen that
> I have no Z+jet events, while the histo for DY+jet is filled, if the DY
> mass is small.
>
> Now, I want to run many jobs in parallel, so that I have enough
> statistics, and then
> I want to merge all histos.
> In the case above, I just artificially put a very small xsection, so
> that I still can
> normalize the histo, and I was hoping, that by merging this would work.
> But apparently, yodamerge does not really like value of 1E99=1/xsect where
> xsect is put to a small number, but not zero.
>
> At the moment I am a bit stuck with this, just don't know, how I can
> tell the
> merging at the end to take into account a very small normalization
> factor for some runs.
>
> Any idea on this would be very welcome
>
> thanks a lot
> Cheers
> Hannes
>
>
>
>
>
>> On 13.10.2015, at 17:38, Andy Buckley <andy.buckley at cern.ch
>> <mailto:andy.buckley at cern.ch>> wrote:
>>
>> Oops, I didn't realise I'd taken the discussion with Hannes off-list.
>>
>> Here's the status: looks like he was experiencing a side-effect of a
>> hack around merge failures for completely empty histos. We fixed that
>> in the latest YODA, so hopefully the hack can also be removed and all
>> will be well...
>>
>> Andy
>>
>>
>> -------- Forwarded Message --------
>> Subject:Re: [Rivet] Problem with yodamerge
>> Date:Tue, 13 Oct 2015 15:42:57 +0200
>> From:Hannes Jung <hannes.jung at desy.de <mailto:hannes.jung at desy.de>>
>> To:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>
>>
>>
>>
>> Hi Andy
>>
>> ahhh.... I put by hand a normalization factor very small (so that
>> 1/factor is very large),
>> so that when the histo is normalized with this factor it becomes zero...
>> but perhaps
>> this screws up when unpacking the normalized histos in yodamerge.
>>
>> The problem occurred in previous versions when I had a histo with no
>> entires,
>> then yodamerge had problems with the normalization... this is why I
>> introduced this hack....  perhaps I should check again whether it
>> works now
>> if I have sometimes a histo without entires and without normalization
>> factor...
>>
>> Cheers
>> Hannes
>>
>>
>>
>>> On 13.10.2015, at 15:33, Andy Buckley <andy.buckley at cern.ch
>>> <mailto:andy.buckley at cern.ch>
>>> <mailto:andy.buckley at cern.ch>> wrote:
>>>
>>> Hi Hannes,
>>>
>>> Yes, those ScaledBy values are quite extreme. They are created by the
>>> calls to scale() or normalize() in Rivet, and the value just
>>> corresponds to the scale/normalization factor. How are you managing to
>>> generate factors of nearly 10^100?!
>>>
>>> In the latest version of YODA, the --assume-normalized flag for
>>> yodamerge doesn't do anything. We changed the default logic since it
>>> seemed that lots of people were using that flag "blindly" to work
>>> around a class of problem for which it wasn't really the right answer.
>>> But since it's now a do-nothing flag it shouldn't be related to your
>>> ScaledBy problems.
>>>
>>> Andy
>>>
>>>
>>> On 13/10/15 11:05, Hannes Jung wrote:
>>>> Hi Andy, Chris
>>>>
>>>> thanks  a lot for your reply.
>>>> Yes, I am trying to merge normalised Histograms.
>>>> ahh, and yes, I have some strange Scaledby values:
>>>> 5.1279402177630292e+97
>>>> I guess this is what causes the problem.
>>>> Where are those set, is this an issue of statistics ?
>>>>
>>>> The cpmmand I use is
>>>>
>>>> yodamerge --assume-normalized yoda-files
>>>>
>>>> Thanks again
>>>> Cheers
>>>> Hannes
>>>>
>>>>
>>>>
>>>>> On 13.10.2015, at 11:23, Andy Buckley <andy.buckley at cern.ch
>>>>> <mailto:andy.buckley at cern.ch>
>>>>> <mailto:andy.buckley at cern.ch>
>>>>> <mailto:andy.buckley at cern.ch>> wrote:
>>>>>
>>>>> Hi Hannes,
>>>>>
>>>>> What sort of data object are you merging here? From the y-axis it
>>>>> looks like a normalized histogram rather than {unnormalized histo,
>>>>> profile, ratio}.
>>>>>
>>>>> If that's right, yodamerge will look for the ScaledBy attribute of
>>>>> that plot to reverse the normalisation, add all the unnormalised
>>>>> histos together, then rescale by the weighted mean ScaledBy. It
>>>>> *shouldn't* be sensitive to one run having much smaller stats than the
>>>>> others, certainly not in a way that would show up like this.
>>>>>
>>>>> If you can provide a bit more detail of what this object is, and how
>>>>> you're trying to merge the runs, we can try to help. You can also plot
>>>>> or take a look inside the .yoda files to see if the individual runs
>>>>> look sane, i.e. have reasonable fill counts and ScaledBy values.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>> On 13/10/15 08:22, Hannes Jung wrote:
>>>>>> Dear Rivet experts
>>>>>>
>>>>>> I am using yodamerge to merge several yoda files (actually it is
>>>>>> 1000 or
>>>>>> so),
>>>>>> to get reasonable statistics in a reasonable computing time.
>>>>>> Sometimes I obtain strange results, like in the plot below,
>>>>>> where the normalization is completely off.
>>>>>> I am using Rivet 2.4.0, but a similar problem I had with Rivet 2.2.0.
>>>>>>
>>>>>> Any idea what this could be ?
>>>>>>
>>>>>> Since I am merging 1000 yoda files, it's hard to tell whether it comes
>>>>>> from one
>>>>>> specific file.... however, all jobs finished without error...
>>>>>>
>>>>>> Thanks a lot
>>>>>> Cheers
>>>>>> Hannes
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ***********************************************************************
>>>>>> Hannes Jung
>>>>>> Email:Hannes.Jung at desy.de
>>>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>>>> mobile :+49 40 8998 93741
>>>>>> http://www.desy.de/~jung
>>>>>> Tel: +49 (0) 40 8998 3741
>>>>>> Fax: +49 (0) 40 8994 3741
>>>>>> DESY, CMS 01B/02.213
>>>>>> Notkestr.85, 22603 Hamburg, FRG
>>>>>> ***********************************************************************
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Rivet mailing list
>>>>>> Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>
>>>>>> <mailto:Rivet at projects.hepforge.org><mailto:Rivet at projects.hepforge.org>
>>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>>> Particle Physics Expt Group, University of Glasgow
>>>>
>>>> ***********************************************************************
>>>> Hannes Jung
>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>> mobile :+49 40 8998 93741
>>>> http://www.desy.de/~jung
>>>> Tel: +49 (0) 40 8998 3741
>>>> Fax: +49 (0) 40 8994 3741
>>>> DESY, CMS 01B/02.213
>>>> Notkestr.85, 22603 Hamburg, FRG
>>>> ***********************************************************************
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>> Particle Physics Expt Group, University of Glasgow
>>
>> ***********************************************************************
>> Hannes Jung
>> Email:Hannes.Jung at desy.de
>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>> mobile :+49 40 8998 93741
>> http://www.desy.de/~jung
>> Tel: +49 (0) 40 8998 3741
>> Fax: +49 (0) 40 8994 3741
>> DESY, CMS 01B/02.213
>> Notkestr.85, 22603 Hamburg, FRG
>> ***********************************************************************
>>
>>
>>
>>
>>
>> _______________________________________________
>> Rivet mailing list
>> Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>
>> https://www.hepforge.org/lists/listinfo/rivet
>
> ***********************************************************************
> Hannes Jung
> Email: Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
> mobile :+49 40 8998 93741
> http://www.desy.de/~jung
> Tel: +49 (0) 40 8998 3741
> Fax: +49 (0) 40 8994 3741
> DESY, CMS 01B/02.213
> Notkestr.85, 22603 Hamburg, FRG
> ***********************************************************************
>
>
>


-- 
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow