Tree Filling Failure

[ershockley]

There is a rare error during processing where branches of the tree are not filled properly, but the processing continues and finishes. Up until now we have only caught the error when trying to open the final root file (gives an R__unzip error). We should be able to catch in the processing chain even if the issue can't be easily solved at the pax level. An example trace of the error is shown below.

NB as you can see below there is also a very strange OSError: [Errno 28] No space left on device error, which suggests maybe this isn't a pax issue but something weird at the OSG site. The weird thing is the processing continues even after displaying this error, so it seems very unlikely that we did in fact fill up the allocated disk space for that job.

Run 16169, event 12972.

Event:  72%|#######2  | 72/100 [07:31<01:44,  3.74s/it]ROOT.TFile.WriteBuffer MainProcess L98 ERROR error writing all requested bytes to file output/180119_2051.root, wrote 16194 of 17824
ROOT.TBranchElement.Fill MainProcess L98 ERROR Failed filling branch:peaks.range_area_decile[11], nbytes=-1
--- Logging error ---
Traceback (most recent call last):
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/logging/__init__.py", line 982, in emit
    self.flush()
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/logging/__init__.py", line 962, in flush
    self.stream.flush()
OSError: [Errno 28] No space left on device
Call stack:
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/bin/cax-process", line 11, in <module>
    load_entry_point('cax==5.0.5', 'console_scripts', 'cax-process')()
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/cax-5.0.5-py3.4.egg/cax/tasks/process.py", line 455, in main
    _process(*sys.argv[1:])
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/cax-5.0.5-py3.4.egg/cax/tasks/process.py", line 163, in _process
    core.Processor(**pax_kwargs).run()
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/pax-6.8.0-py3.4.egg/pax/core.py", line 315, in run
    self.process_event(event)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/pax-6.8.0-py3.4.egg/pax/core.py", line 276, in process_event
    event = plugin.process_event(event)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/pax-6.8.0-py3.4.egg/pax/plugin.py", line 91, in process_event
    event = self._process_event(event)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/pax-6.8.0-py3.4.egg/pax/plugin.py", line 138, in _process_event
    result = self.write_event(event)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/pax-6.8.0-py3.4.egg/pax/plugins/io/ROOTClass.py", line 272, in write_event
    self.event_tree.Fill()
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/rootpy/logger/roothandler.py", line 98, in python_logging_error_handler
    log.log(lvl, msg)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/logging/__init__.py", line 1346, in log
    self._log(level, msg, args, **kwargs)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/logging/__init__.py", line 1414, in _log
    self.handle(record)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/logging/__init__.py", line 1424, in handle
    self.callHandlers(record)
  File "/cvmfs/xenon.opensciencegrid.org/releases/anaconda/2.4/envs/pax_v6.8.0_OSG/lib/python3.4/site-packages/rootpy/logger/extended_logger.py", line 216, in callHandlers
    result = LoggerClass.callHandlers(self, record)
Message: 'Failed filling branch:peaks.range_area_decile[11], nbytes=-1'
Arguments: ()
ROOT.TBranchElement.Fill MainProcess L98 ERROR Failed filling branch:peaks.peaks.range_area_decile[11], nbytes=-1
ROOT.TBranchElement.Fill MainProcess L98 ERROR Failed filling branch:events.peaks, nbytes=-1
ROOT.TTree.Fill MainProcess L98 ERROR Failed filling branch:tree.events, nbytes=-1, entry=73
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)

Event:  73%|#######3  | 73/100 [07:38<02:08,  4.76s/it]

[JelleAalbers]

Thanks for reporting, could this be the same as #536 ?

[ershockley]

Yep looks the same - I'll close this one. Thanks!