[Rivet] yodamerge and normalisation

[Hannes Jung]

Hi Andy

thanks a lot for your mail and the suggestion.

Why do you say, it would be better outside Rivet?

Maybe it is not clear to me, how the "un-normalising" in yodamerge is done.
Is perhaps the problem the large (or small) number which I took to replace infty or zero ?

The point is, that we have the measurements in such a way, and it would be just
fine, to have a Rivet plugin, which would fill the histos such that it can be
plotted easily with rivet-mkhtml.

Thanks again

Best
Hannes

> On 14.10.2015, at 23:39, Andy Buckley <andy.buckley at cern.ch> wrote:
> 
> Hi Hannes,
> 
> I think this sort of case is better dealt with by *not* normalizing the DY+jet histos, or normalizing them to an area of 1. Then use yodamerge to combine both the Z+jet and DY+jet pt plots and write a (very) little Python script to do the last bit:
> 
> import yoda
> hs = yoda.read("merged.yoda")
> norm = hs["/MYANA/DYpt"].integral()
> hs["/MYANA/DYpt"].normalize(norm)
> yoda.write(hs, "rescaled.yoda")
> 
> Not tested, but that's the basic idea. Let us know if that works for you.
> 
> Andy
> 
> 
> On 14/10/15 18:23, Hannes Jung wrote:
>> Hi Andy et al
>> 
>> the new yodamerge works much better now, this is really great.
>> 
>> But I have now another problem (which I tried to solve with the hack
>> from before):
>> 
>> I fill histograms and normalize them to some xsection.
>> Now it can happen, that the histo is filled, but the normalization is zero,
>> for example when I plot the pt of DY+jet normalized to Z+jet, then it
>> can happen that
>> I have no Z+jet events, while the histo for DY+jet is filled, if the DY
>> mass is small.
>> 
>> Now, I want to run many jobs in parallel, so that I have enough
>> statistics, and then
>> I want to merge all histos.
>> In the case above, I just artificially put a very small xsection, so
>> that I still can
>> normalize the histo, and I was hoping, that by merging this would work.
>> But apparently, yodamerge does not really like value of 1E99=1/xsect where
>> xsect is put to a small number, but not zero.
>> 
>> At the moment I am a bit stuck with this, just don't know, how I can
>> tell the
>> merging at the end to take into account a very small normalization
>> factor for some runs.
>> 
>> Any idea on this would be very welcome
>> 
>> thanks a lot
>> Cheers
>> Hannes
>> 
>> 
>> 
>> 
>> 
>>> On 13.10.2015, at 17:38, Andy Buckley <andy.buckley at cern.ch
>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>> wrote:
>>> 
>>> Oops, I didn't realise I'd taken the discussion with Hannes off-list.
>>> 
>>> Here's the status: looks like he was experiencing a side-effect of a
>>> hack around merge failures for completely empty histos. We fixed that
>>> in the latest YODA, so hopefully the hack can also be removed and all
>>> will be well...
>>> 
>>> Andy
>>> 
>>> 
>>> -------- Forwarded Message --------
>>> Subject:Re: [Rivet] Problem with yodamerge
>>> Date:Tue, 13 Oct 2015 15:42:57 +0200
>>> From:Hannes Jung <hannes.jung at desy.de <mailto:hannes.jung at desy.de> <mailto:hannes.jung at desy.de <mailto:hannes.jung at desy.de>>>
>>> To:andy.buckley at cern.ch <http://cern.ch/> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>
>>> 
>>> 
>>> 
>>> Hi Andy
>>> 
>>> ahhh.... I put by hand a normalization factor very small (so that
>>> 1/factor is very large),
>>> so that when the histo is normalized with this factor it becomes zero...
>>> but perhaps
>>> this screws up when unpacking the normalized histos in yodamerge.
>>> 
>>> The problem occurred in previous versions when I had a histo with no
>>> entires,
>>> then yodamerge had problems with the normalization... this is why I
>>> introduced this hack....  perhaps I should check again whether it
>>> works now
>>> if I have sometimes a histo without entires and without normalization
>>> factor...
>>> 
>>> Cheers
>>> Hannes
>>> 
>>> 
>>> 
>>>> On 13.10.2015, at 15:33, Andy Buckley <andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>
>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>
>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>> wrote:
>>>> 
>>>> Hi Hannes,
>>>> 
>>>> Yes, those ScaledBy values are quite extreme. They are created by the
>>>> calls to scale() or normalize() in Rivet, and the value just
>>>> corresponds to the scale/normalization factor. How are you managing to
>>>> generate factors of nearly 10^100?!
>>>> 
>>>> In the latest version of YODA, the --assume-normalized flag for
>>>> yodamerge doesn't do anything. We changed the default logic since it
>>>> seemed that lots of people were using that flag "blindly" to work
>>>> around a class of problem for which it wasn't really the right answer.
>>>> But since it's now a do-nothing flag it shouldn't be related to your
>>>> ScaledBy problems.
>>>> 
>>>> Andy
>>>> 
>>>> 
>>>> On 13/10/15 11:05, Hannes Jung wrote:
>>>>> Hi Andy, Chris
>>>>> 
>>>>> thanks  a lot for your reply.
>>>>> Yes, I am trying to merge normalised Histograms.
>>>>> ahh, and yes, I have some strange Scaledby values:
>>>>> 5.1279402177630292e+97
>>>>> I guess this is what causes the problem.
>>>>> Where are those set, is this an issue of statistics ?
>>>>> 
>>>>> The cpmmand I use is
>>>>> 
>>>>> yodamerge --assume-normalized yoda-files
>>>>> 
>>>>> Thanks again
>>>>> Cheers
>>>>> Hannes
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 13.10.2015, at 11:23, Andy Buckley <andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>
>>>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>
>>>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>
>>>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>> wrote:
>>>>>> 
>>>>>> Hi Hannes,
>>>>>> 
>>>>>> What sort of data object are you merging here? From the y-axis it
>>>>>> looks like a normalized histogram rather than {unnormalized histo,
>>>>>> profile, ratio}.
>>>>>> 
>>>>>> If that's right, yodamerge will look for the ScaledBy attribute of
>>>>>> that plot to reverse the normalisation, add all the unnormalised
>>>>>> histos together, then rescale by the weighted mean ScaledBy. It
>>>>>> *shouldn't* be sensitive to one run having much smaller stats than the
>>>>>> others, certainly not in a way that would show up like this.
>>>>>> 
>>>>>> If you can provide a bit more detail of what this object is, and how
>>>>>> you're trying to merge the runs, we can try to help. You can also plot
>>>>>> or take a look inside the .yoda files to see if the individual runs
>>>>>> look sane, i.e. have reasonable fill counts and ScaledBy values.
>>>>>> 
>>>>>> Andy
>>>>>> 
>>>>>> 
>>>>>> On 13/10/15 08:22, Hannes Jung wrote:
>>>>>>> Dear Rivet experts
>>>>>>> 
>>>>>>> I am using yodamerge to merge several yoda files (actually it is
>>>>>>> 1000 or
>>>>>>> so),
>>>>>>> to get reasonable statistics in a reasonable computing time.
>>>>>>> Sometimes I obtain strange results, like in the plot below,
>>>>>>> where the normalization is completely off.
>>>>>>> I am using Rivet 2.4.0, but a similar problem I had with Rivet 2.2.0.
>>>>>>> 
>>>>>>> Any idea what this could be ?
>>>>>>> 
>>>>>>> Since I am merging 1000 yoda files, it's hard to tell whether it comes
>>>>>>> from one
>>>>>>> specific file.... however, all jobs finished without error...
>>>>>>> 
>>>>>>> Thanks a lot
>>>>>>> Cheers
>>>>>>> Hannes
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> ***********************************************************************
>>>>>>> Hannes Jung
>>>>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>>>>>> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>><mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>>>>>> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>><mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>>>>>> mobile :+49 40 8998 93741
>>>>>>> http://www.desy.de/~jung <http://www.desy.de/~jung>
>>>>>>> Tel: +49 (0) 40 8998 3741
>>>>>>> Fax: +49 (0) 40 8994 3741
>>>>>>> DESY, CMS 01B/02.213
>>>>>>> Notkestr.85, 22603 Hamburg, FRG
>>>>>>> ***********************************************************************
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Rivet mailing list
>>>>>>> Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org> <mailto:Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>>
>>>>>>> <mailto:Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>><mailto:Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>>
>>>>>>> https://www.hepforge.org/lists/listinfo/rivet <https://www.hepforge.org/lists/listinfo/rivet>
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>>>> Particle Physics Expt Group, University of Glasgow
>>>>> 
>>>>> ***********************************************************************
>>>>> Hannes Jung
>>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>>>> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>><mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>>>> mobile :+49 40 8998 93741
>>>>> http://www.desy.de/~jung <http://www.desy.de/~jung>
>>>>> Tel: +49 (0) 40 8998 3741
>>>>> Fax: +49 (0) 40 8994 3741
>>>>> DESY, CMS 01B/02.213
>>>>> Notkestr.85, 22603 Hamburg, FRG
>>>>> ***********************************************************************
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>> Particle Physics Expt Group, University of Glasgow
>>> 
>>> ***********************************************************************
>>> Hannes Jung
>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>><mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>> mobile :+49 40 8998 93741
>>> http://www.desy.de/~jung <http://www.desy.de/~jung>
>>> Tel: +49 (0) 40 8998 3741
>>> Fax: +49 (0) 40 8994 3741
>>> DESY, CMS 01B/02.213
>>> Notkestr.85, 22603 Hamburg, FRG
>>> ***********************************************************************
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Rivet mailing list
>>> Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org> <mailto:Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>>
>>> https://www.hepforge.org/lists/listinfo/rivet <https://www.hepforge.org/lists/listinfo/rivet>
>> 
>> ***********************************************************************
>> Hannes Jung
>> Email: Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>> mobile :+49 40 8998 93741
>> http://www.desy.de/~jung <http://www.desy.de/~jung>
>> Tel: +49 (0) 40 8998 3741
>> Fax: +49 (0) 40 8994 3741
>> DESY, CMS 01B/02.213
>> Notkestr.85, 22603 Hamburg, FRG
>> ***********************************************************************
>> 
>> 
>> 
> 
> 
> -- 
> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
> Particle Physics Expt Group, University of Glasgow

***********************************************************************
Hannes Jung 
Email: Hannes.Jung at desy.de 
mobile :+49 40 8998 93741
http://www.desy.de/~jung                                  
Tel: +49 (0) 40 8998 3741         
Fax: +49 (0) 40 8994 3741
DESY, CMS 01B/02.213
Notkestr.85, 22603 Hamburg, FRG   
***********************************************************************



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.hepforge.org/lists-archive/rivet/attachments/20151015/724595b5/attachment.html>