[Rivet] make-plots batch render and -> TikZ?

[Andy Buckley]

On 14/10/16 08:42, Frank Siegert wrote:
> Hi Andy,
>
> On 13 October 2016 at 21:59, Andy Buckley <andy.buckley at cern.ch> wrote:
>> Having been too busy to do real work recently, yesterday I tried following
>> David's clever trick with batch compilation and tried to do a similar thing
>> in make-plots: rather than multi-processing a different LaTeX doc per plot,
>> I made and compiled one big TeX document.
>>
>> It works, after a bit of fiddling due to pstricks not playing well with the
>> standalone class, and does speed things up a bit: 70 plots rendered in 45
>> seconds over 4 cores on my laptop with the current make-plots, while it took
>> just over 30 seconds with the one-big-doc approach. And of course a hybrid
>> could be tried: split into Ncore biggish docs.
>>
>> It's not quite the speed gain I was hoping for: graphics are still
>> expensive, so no factor x10 gains. But worth pursuing? I have to admit that
>> I also found it disconcerting to not have the "N plots remaining" feedback
>> to the terminal!
>
> Thanks, sounds interesting. Does that mean with the one-big-doc
> approach make-plots would only use a single core and still be 1/3
> quicker than in the old approach when using 4 cores? Sounds
> surprisingly good, though practically (>4 cores available on many
> machines) it wouldn't be a path I would pursue.

Yes, in this ~unscientific test that's the sort of result I got. I'd 
hoped that the LaTeX start-up and package parsing would be a larger 
fraction, i.e. driving down compile times to a few seconds rather than 
tens of them, but it is an improvement and a multi-process/large-doc 
hybrid could probably squeeze out significantly more performance.

>> I'm not sure what the effect on speed would be, but this experiment also got
>> me wondering about whether a future version should switch internally from
>> pstricks to TikZ for rendering. That's definitely where the action is in TeX
>> graphics these days, and being able to simplify the processing chain from
>> TeX -> DVI -> PS -> PDF to just TeX -> PDF would make things a lot easier
>> and less fragile. (I would like to not both with PS and EPS anymore... does
>> anyone have a good reason to not use pdflatex?) We could maybe stop having
>> to bundle advanced pstricks packages with Rivet, too. It would impact users
>> of the SPECIAL plot sections, though, although maybe TikZ wrappers for \rput
>> etc. could be provided. There are simple TikZ equivalents, of course. Any
>> opinions?
>
> Absolutely in favour, unless it makes it significantly slower!

Ok, good! I thought that you and other Sherpas would be the most likely 
to object, since you've done more plot customisation than most. We 
should archive the current version as make-plots-pstricks or similar, so 
old .dat files can continue to be built if needed. I don't think 
there'll be a lot of migration required for the "official" Rivet 
analysis .plot files.

Moving to TikZ will greatly simplify (and _maybe_ speed up) the 
processing chain, but another reason to like it is that pstricks' 
conflicts with standalone means that I currently have to put the margins 
specification into a call to the pdfcrop script. With a fully PDF-based 
chain, it's a lot neater. Ok, I'll look into it...

> By the way, maybe that was discussed at the dev workshop, but was the
> idea of eventually rewriting make-plots in matplotlib scratched? I
> know that we haven't gotten past some technical feasibility studies
> yet (at least I, maybe you have got further?), but I don't remember
> any final decision against it.

No, there's no final decision like that. I only did this a) to see if I 
could, and b) because plotting is always so far down the priority/reward 
list that I suspect in practice make-plots is going to be around in some 
form for at least another year or two.

I would love to see it superseded by a programmatic system with 
high-quality output, but so far my efforts with matplotlib have been no 
faster than 100% TeX-based rendering, the output isn't as nice (Hendrik 
did a great job on the aesthetics), and the implementation code has to 
perform all sorts of gymnastics to e.g. get the appropriate consistent 
TeX fonts used for tick labels, axis labels, titles, etc. Yuck. So while 
others (yourself, Johannes, Karl, ...) try to gradually get somewhere 
with mpl, I'll carry on seeing if better speed and programmability can 
be retro-fitted to make-plots.

One place where I think useful progress can be made quite easily is to 
provide "convenience" functions in yoda.plotting, for relatively 
low-level operations like returning tuples of bin edges, values/errors, 
envelope edges, etc. from YODA analysis objects. Those "atomic" 
functionalities can be used to implement a push-button YODA replacement 
for make-plots, but would also be a big help to anyone wanting to do 
their own custom rendering of YODA data. That's also been on my list for 
many months but I've not had time to do it... so if anyone else wants to 
play with plotting YODA stuff, then let me know and we can divide out 
some tasks ;-)

Andy

-- 
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow