PS1_IPP_Czarlog_20141110 – Pan-STARRS IPP

Context Navigation

Version 19 (modified by eugene, 12 years ago) ( diff )
--

11:25 CZW: started pantasks server on j001b09 to do raw data shuffle to backup nodes.

5:00 HAF registration is stuck on o6972g0517o -- usual bag o tricks didn't work. I ended up setting a quality of 42 on that stupid burntool. update rawImfile set quality = 42, fault = 0, state = 'full' where exp_name = 'o6972g0517o' and class_id = 'XY14';, but not after spending an hour trying to figure out what was going on...
6:20 HAF registration is still stick on teh same exp, after moving on.. I redid the update since it seems to have been reverted... registration is moving again but I don't trust it
6:24 Chris investigated, regtool -updateprocessedimfile -burntool_state -14 -exp_id 816417 -class_id XY14 -set_ignored -set_state full is the proper way to tell the database to ignore an xy

12:00 MEH: mops wondering where data is, stdsci looks to have been down since ~10am? did someone shutdown?
- restarting .. -- lanl stdlocal off until nightly finished (wasnt doing anything anyways).. putting ippsXX nodes into stdsci to finish up asap..
12:40 MEH: ipp035 unresponsive.. neb-host down until back up, out of processing --
12:50 MEH: ipp035 up, putting back into repair -- ipp029 now unresponsive.. seriously..
- ipp029 from neb-host up to down -- out of processing
- ipp029 up (kernel panic), putting neb-host repair (normally up)

13:40 MEH: when nightly finished, ippsXX off in stdsci and stdlocal for MD/CNP processing, stdlocal on again and update labels back into stdsci
15:30 EAM : stdlocal is going slowly, so i'm stopping stacks in prep for restarting
- good was inbetween runs, had set the ippsXX off in stdlocal.. setting back to off again
17:30 HAF : restarted registration, it had crashed (? i assumed?)

06:40 EAM : I am stopping stdlocal for now in anticipation of launching relphot on the compute nodes. Also note that nightly science is proceeding slowly - is this the db again?