[Rivet] mcplots

[Peter Skands]

Hi Andy,

>> Our production system is
>> up and running, but we are not able to combine the outputs of several
>> runs yet. That's essentially the limiting factor, since we need to keep
>> the running time of each job "reasonable".
>
> Why? I know that you're using BOINC (for *all* generation?), but if the
> VM can be suspended and resumed arbitrarily then is there a problem if
> an MC job takes a couple of days?

We're doing initial runs on lxplus, 100k then 1M (since some of the 1M 
ones crash or never finish, we at least have a minimum of 100k for 
everything this way), and we are considering the possibility of doing a 
few 10M ones as well. Concerning BOINC, there has been a *huge* amount 
of development, which I cannot briefly summarize. Some of it will be 
described in an upcoming paper. Just trust me that there are some 
limitations, at least for the present, and some other constraints we 
choose to impose ourselves for the sake of simplicity, again at least 
for the present.

> Actually, I don't know of *any* analysis system which can deal with this
> problem exactly -- the really big problem is that in general the
> finalising step of a Rivet analysis might do *anything*. So if you batch
> the event loops to increase statistics, then in general you would need
> to then gather the interim data members of each analysis class, somehow
> know how to appropriately combine them, and then let the arbitrary
> histogram combination/normalising/mangling take place. That's a really
> hard problem to solve without making simple things incredibly hard.

Don't worry, we will gladly accept anything you do provide, which is 
already incredibly much. As far as mcplots is concerned, I know how I 
want to solve the batching problem, at least to my own satisfaction, but 
it will not be in the way you want. Basically, within the more narrow 
confines of the mcplots context, we can do things that would be a lot 
more ugly in a more general Rivet setting. The problem has been to find 
manpower for it, after Holger ended his mcnet studentship, but I am 
still hopeful we will end with something that actually does work, at 
least for mcplots. That pilot project should then also help show us 
whether it is, indeed, hopeless to do something similar in the more 
general Rivet context. I am not as pessimistic as you on that point, but 
agree it is not easy to see a simple solution that would work out of the 
box for anything.

Peter