[Rivet] rivet-mkhtml recursively removes ${pwd}/plots without warning

[David Bjergaard]

Hi,

Just because unix does something doesn't mean that makes it user friendly or
sane. I haven't used rivet-mkhtml until today because I prefer dumping the *.dat
files with rivet-cmphistos and then calling make-plots.  As a result, I had
plots/dat and plots/pdf in ${PWD} and I would call rivet-cmphistos with -o
plots/dat in order to avoid cluttering the ${PWD}.

Given that /I/ made ${PWD}/plots and the files contained therein, I was
surprised to find that the whole directory structure had been clobbered and
replaced. 

When rivet replaces a file it created (.yoda) when I re-run the command, that's
an expected side effect.  My objection is that command shouldn't be allowed to
appropriate and replaces a directory that the user made for another purpose.

David's suggestion that use put a hidden zero size file (whatever the python
equivalent of "touch ${PWD}/plots/.mkhtml" is) and check for it would be the
best compromise. That would keep the core behavior for people who expect their
output to be over-written and it prevents people from clobbering plots that they
might otherwise need.

If you don't do anything, please at least change this:
"You can overwrite an existing output directory."
to
"WARNING: This script automatically overwrites the existing output directory
(default: ./plots)."

That is a much more clear description of whats going to happen when reading the
usage and options of the rivet-mkhtml script.

Best,

    David


David Grellscheid <david.grellscheid at durham.ac.uk> writes:

> Hi!
>
> Please keep the core behaviour as it is now. Andy has mentioned most of
> the arguments already. We've had it the other way round before, and that
> was a real pain to work with. I much prefer the silent convenience of
> overwriting the existing output.
>
> The issue is more in the naming I think. A directory name that has Rivet
> explicitly in the name is less likely to have pre-existed. Or we check
> for a hidden 0-size file in the directory before deleting, to make sure
> the directory is one of ours.
>
>   David
>
>
>
> On 08/10/15 12:03, Andy Buckley wrote:
>> Hi David,
>> 
>> I'm not really sure why it would be a surprise. We equally don't warn or 
>> stop users from running rivet if the output .yoda file already exists, 
>> nor re. overwriting of the files and directories from rivet-cmp histos.
>> 
>> The core Unix shell tools also don't complain when they would overwrite 
>> an existing file -- unless you specifically tell them to do so (for 
>> those which have such an option). Personally I prefer the convenience of 
>> the tools not objecting to overwrites. A priori if I run any command 
>> that would logically write into the same dir as a previous run, I should 
>> not expect my existing data to remain intact -- that's not just 
>> restricted to Rivet & friends!
>> 
>> I was going to add the printed warning that you suggested, then changed 
>> my mind -- by the time they've seen the message the damage is already 
>> done, and we'd "need" to add similar messages to every tool in the Rivet 
>> (and YODA) suite. It just seems like extra noise to me. But maybe others 
>> disagree?
>> 
>> Andy
>> 
>> 
>> On 08/10/15 11:48, David Bjergaard wrote:
>>> Hi All,
>>>
>>> Luckily I didn't lose any plots that I was attached to, but I did have a ./plots
>>> folder that was using to store intermediate results.  Removing any files without
>>> warning the user is kind of scary which is what I think this line does:
>>>      > if os.path.exists(opts.OUTPUTDIR) and not os.path.realpath(opts.OUTPUTDIR)==os.getcwd():
>>>      >     import shutil
>>>      >     shutil.rmtree(opts.OUTPUTDIR)
>>> Why can't you throw an exception and say "OUTPUTDIR not empty, please use -f to
>>> force overwriting", or less ideal but still better "OUTPUTDIR is not empty,
>>> removing" so that its less of a surprise when all the subdirectories in
>>> OUTPUTDIR have mysteriously vanished.
>>>
>>>      David
>>> _______________________________________________
>>> Rivet mailing list
>>> Rivet at projects.hepforge.org
>>> https://www.hepforge.org/lists/listinfo/rivet
>>>
>> 
>>