[Rivet] Compatibility of YODA reference data from Rivet with HEPData

[David Grellscheid]

Hi Graeme,

thanks for your email, we have started discussing it. Just one technical
point, YODA comes with a yodacmp script already, which addresses many of
your technical issues with comparisons. Maybe you can use that
internally in your script?

See you,

 David


On 21/05/2018 20:16, Graeme Watt wrote:
> Dear Rivet developers,
> 
> I wrote a Python script to compare a YODA reference data file, intended
> for inclusion in Rivet, with the corresponding YODA file downloaded from
> HEPData:
> 
> https://github.com/HEPData/miscellaneous/blob/master/scripts/yoda_compare_hepdata.py
> 
>   Example usage:  ./yoda_compare_hepdata.py ATLAS_2017_I1614149.yoda -i
> 1614149 -a
> 
> This means: compare a local YODA file "ATLAS_2017_I1614149.yoda" with a
> YODA file downloaded from the HEPData record with INSPIRE ID "1614149"
> and also compare YODA annotations "-a".  Since the HEPData YODA file
> might contain additional analysis objects compared to the Rivet YODA
> file, and since there might be inconsequential rounding errors or
> differences in number formats, comparison using a simple "diff" of .yoda
> files is not always adequate.
> 
> I had a few problems with the YODA 1.7.0 software when writing the
> Python script, which could perhaps be improved in future:
> 
> * Calling dump() on Scatter objects does not output the same format as
> in the input .yoda files, e.g. dumping a Scatter2D gives "HISTO1D" in
> the output and the central value of the x bin is not output.  This might
> be due to deficiencies in
> https://yoda.hepforge.org/trac/browser/src/WriterFLAT.cc .
> * I expected to be able to check (fuzzy) equality of two Scatter objects
> using "s == s1", which seems to be implemented in C++ but not in Python.
> * Checking (fuzzy) equality of two Point2D objects using "p == p1" is
> implemented in Python, but it only compares the x axes and not the y
> axes.  Similarly, for Point3D objects, the (fuzzy) equality operator
> "==" only compares the x and y axes, but not the z axes.
> * In the end, I just copied the definition of "fuzzyEquals" from the C++
> code into my script and did my own comparisons, without relying on the
> "==" operator for Point or Scatter objects.
> 
> Recall that Holger Schulz made some similar comparisons in 2016 between
> YODA reference data files from Rivet and from the old HepData, where he
> found significant inconsistencies:
> 
> https://www.hepforge.org/lists-archive/rivet/2016-October/007318.html
> https://rivet.hepforge.org/trac/browser/contrib/devscripts/HepDataConsistency
> 
> 
> While fixing all Rivet/HEPData inconsistencies is probably unrealistic
> for now, we could at least start by ensuring that analyses added to new
> Rivet releases include a YODA file that's compatible with the YODA
> download given by HEPData.  My new script should be useful for these
> purposes.  (This work was prompted by a conversation with Peter Skands
> and Jon Butterworth last month in Durham.)  You're welcome to modify my
> script as you need and include it in a future Rivet release.  The script
> could be run by the experiment contact persons before they upload new
> analyses to the Rivet "contrib" area.  It could also be run (perhaps in
> a more automated way) by the Rivet developers when moving analyses from
> the "contrib" area to a new Rivet release.  I just ran the script for
> all the new analyses in the current Rivet "contrib" area (
> https://www.hepforge.org/archive/rivet/contrib/ ) and it already turned
> up some useful information:
> 
> * ATLAS_2014_I1310835, ATLAS_2017_I1614149, and ATLAS_2017_I1624693, are
> all compatible with HEPData.
> * ATLAS_2016_I1502620 has multiple .yoda files which can't be handled by
> my script.
> * ATLAS_2017_I1625109 showed some apparent inconsistencies but the
> problem looks to be on the HEPData side.  The dataset index starts at 0
> (instead of 1) due to a bug in the hepdata-converter package (which I'll
> fix) and the year in the analysis name is 2018 (instead of 2017) taken
> from the journal publication.
> * ALICE_2017_I1620477 is incompatible with HEPData.
> * CMS_2016_I1487277 is compatible with HEPData.
> * CMS_2012_I1111014 and CMS_2016_I1491950 are incompatible with HEPData.
> * CMS_2017_I1499471, CMS_2017_I1635889, CMS_2018_I1662081 and
> CMS_2018_I1663958 are all missing from HEPData.
> * CMS_2017_I1605749 has a HEPData record, but the YODA conversion fails
> due to a problem with the original HEPData submission.
> * LHCF_2015_I1351909 is compatible with HEPData, but the annotations
> differ.
> * LHCF_2016_I1385877 is incompatible with HEPData.
> 
> I hope this gives you something to discuss at this week's Rivet
> workshop! :-)
> 
> Best regards,
> Graeme Watt (HEPData)
> 
> 
> 
> 
> _______________________________________________
> Rivet mailing list
> Rivet at projects.hepforge.org
> https://www.hepforge.org/lists/listinfo/rivet
>