== PS1 IPP Czar Logs for the week 2011.11.07 - 2011.11.13 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2011.11.07 === Mark is czar * 09:00 ipp036 removed from nebulous so Cindy can powerdown and replace CPU+heatsink * 09:30 remaining night data having 503 error in summitcopy (c5872g0011o--c5872g0014o). finished @10:00 * 11:30 MD07-y nightly stack sample running in normal stack pantasks to verify skycell coverage with the new center shift for exposures since 4/2009. * 13:15 ipp036 back up, putting back into nebulous fully and slowly adding back into processing * 13:30 ipp053 slowly putting back into processing, taken out Friday for over weekend due to mount issues and many tasks wanting to run on it. * 13:45 ipp036 export not up, ran exportfs -f * 14:00 ippc13 slowly add back processing since watching ipp036,053 too. was taken out from processing because of second crash w/in hour over weekend. * 14:45 ippc10 also back online, adding back in for processing * 15:30 running MD07.reftest.20111107 sample of different number of input warps in skycell.055 for all filters with the deepstack pantasks (compute2 group). * 16:30 cleaning LAP -- needed to do more often, now have 1500 stacks to run.. {{{ -- troubled update fault=2, find update log neb://ipp020.0/gpc1/LAP.ThreePi.20110809/2011/10/09/o5463g0339o.229905/o5463g0339o.229905.ch.317278.XY31.log.update and problem with with neb://ipp044.0/gpc1/20100924/o5463g0339o/o5463g0339o.ota31.fits 0 NON-EXISTANT file:///data/ippb02.1/nebulous/10/c3/967714255.gpc1:20100924:o5463g0339o:o5463g0339o.ota31.fits 0 NON-EXISTANT file:///data/ippb02.1/nebulous/10/c3/967715079.gpc1:20100924:o5463g0339o:o5463g0339o.ota31.fits chiptool -updateprocessedimfile -set_state full -chip_id 317278 -class_id XY31 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 317278 -class_id XY31 -dbname gpc1 -- more update trouble chip_id=339071, drop from multiple runs - laptool -updateexp -lap_id 1705 -exp_id 229905 -set_data_state drop -dbname gpc1 laptool -updateexp -lap_id 1706 -exp_id 229905 -set_data_state drop -dbname gpc1 laptool -updateexp -lap_id 1709 -exp_id 229905 -set_data_state drop -dbname gpc1 laptool -updateexp -lap_id 1710 -exp_id 229905 -set_data_state drop -dbname gpc1 -- warp (update?) trouble for exp_id=260145, drop from runs laptool -updateexp -lap_id 1699 -exp_id 260145 -set_data_state drop -dbname gpc1 laptool -updateexp -lap_id 1700 -exp_id 260145 -set_data_state drop -dbname gpc1 laptool -updateexp -lap_id 1703 -exp_id 260145 -set_data_state drop -dbname gpc1 -- warp fault due to chip file problem Reading FITS file /data/ipp033.0/nebulous/a6/76/1544401387.gpc1:LAP.ThreePi.20110809:2011:11:07:o5523g0098o.260145:o5523g0098o.260145.ch.339377.XY52.ch.fits failed. perl ~ipp/src/ipp-20110622/tools/runchipimfile.pl --chip_id 339377 --class_id XY52 --redirect-output }}} * 16:40 info note: running lap_science.pl --monitor_mode --lap_id 1730 reported {{{ QUALITY: 223457 has bad warp quality: 16 / 77 STATUS: LAP_ID 1730 STATUS: DEFINED_QSTACK: 0 STATUS: HAVE_QSTACK: 0 STATUS: DEFINED_FSTACK: 0 STATUS: HAVE_FSTACK: 0 STATUS: NEEDS_REMADE: 0 STATUS: NEEDS PRIVATIZE: 0 STATUS: ARE WARPED: 102 STATUS: ARE MAGICKED: 50 STATUS: CAN_DIFF: 101 STATUS: HAVE_DIFF: 54 STATUS: TOTAL_EXPOSURES: 113 means LAP kicking out that exposure (currently done if >20% warps have bad quality). }}} * 17:30 ippc13 okay so far with the 4x stdscience, 2x stack so adding back distribution,pstamp,update,publish * 18:00 ippc13 wend down, so seems can handle the stdscience+stack load but not distribution. will try some combos to see what is stable. Chris also recommended maybe adding a compute_broken group in ~ipp/ippconfig/pantasks_hosts.input that could also include ippc11 (seems to handle 3 processes okay). * 19:00 ippc13 looks like crashed when the 1x update started while running 2x stdscience, 2x stack. drop down to 2x stdscience, 1x stack and group with ippc11 in hosts_poor_compute * 20:30 another LAP chip lost {{{ Nebulous key neb://ipp043.0/gpc1/20100619/o5366g0539o/o5366g0539o.ota25.fits is consistent. 0 NON-EXISTANT file:///data/ippb02.0/nebulous/4c/7e/893019408.gpc1:20100619:o5366g0539o:o5366g0539o.ota25.fits 0 NON-EXISTANT file:///data/ippb02.2/nebulous/4c/7e/893026715.gpc1:20100619:o5366g0539o:o5366g0539o.ota25.fits }}} * 21:00 ippc11 donw. cannot run 1x stack + 2x stdscience so remove from processing completely again. rebooting, but not coming back up. leaving off. * 23:30 ippc11 powered up. * 23:40 LAP clearing all of sunday's (11/6) runs now and morving forward. * MD07.refstack.20111106 z-band etc running under the deepstack pantasks. MD07.GR0 sample nightly stacks added to the main stack pantasks as they run quicker and at a lower priority that can be interrupted by LAP. === Tuesday : 2011.11.08 === * 11:50 Mark: stare nodes been quiet for a while, adding back into processing. distribution uses 4 nodes on each of the stare nodes, maybe 1-2 of these each could be added to stack since distribution seems to be keeping up. === Wednesday : YYYY.MM.DD === === Thursday : YYYY.MM.DD === === Friday : YYYY.MM.DD === === Saturday : YYYY.MM.DD === === Sunday : YYYY.MM.DD ===