== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.06.23 ===
  * 20:30(EDT) MEH: restarting pantasks, been running a while and need some stamps quickly
=== Tuesday : 2014.06.24 ===
 * 10:00(EDT) MEH: compute3 nodes/c2 mostly idle so using for staticsky for couple hours. and kicking registration so not waiting to revert and un-stall itself.
=== Wednesday : 2014.06.25 ===
 * 08:45 (EDT) MEH:ipp030 down.. power cycled [wiki:ipp030.crash.20140625]
=== Thursday : 2014.06.26 ===
 * 07:55 (EDT) MEH: registration 170 exposures behind, kicked and moving again
{{{
crash for: register_imfile.pl --exp_id 760419 --tmp_class_id ota47 --tmp_exp_name o6834g0217o --uri neb://ipp030.0/gpc1/20140626/o6834g0217o/o6834g0217o.ota47.fits --logfile neb://ipp030.0/gpc1/20140626/o6834g0217o.760419/o6834g0217o.760419.reg.ota47.log --bytes 23063040 --md5sum 5d3046529d950808f4a0b6d8a1f15b66 --sunset 03:30:00 --sunrise 17:30:00 --summit_dateobs 2014-06-26T08:41:43.000000 --dbname gpc1 --verbose
job exit status: CRASH
job host: ipp030
job dtime: 8.739945
job exit date: Wed Jun 25 23:07:27 2014
hostname: ipp030

-- had to manually do 
regtool -updateprocessedimfile -exp_id 760419 -class_id XY47 -set_state pending_burntool -dbname gpc1

}}}
 * 09:10(EDT) MEH: still far behind in processing, compute3 nodes idle so throwing into processing for a few hours
 * 1!:25(EDT) MEH: stdsci pantasks struggling to stay loaded, needing regular restart and doing others as well
 * 12:00(EDT) MEH: once nightly finished, using idle compute3 nodes for small test MD.PV3 processing 
=== Friday : 2014.06.27 ===
 * 02:40(EDT) MEH: ipp030 having issues with becoming unresponsive/high load, turning off in processing
 * 18:10(PDT) MEH: Haydn looking into ipp030 crash around 0710(HST) this morning. -- setting neb-host down and removing from pantasks_hosts.input
  * some stdscience jobs need to probably be cleared and might as well restart stdsci
{{{
-- both have files on ipp030 so won't finish until back up
system failure for: diff_skycell.pl --threads @MAX_THREADS@ --diff_id 570810 --skycell_id skycell.0615.037 --diff_skyfile_id 33349104 --outroot neb://stsci17.1/gpc1/ThreePi.nt/2014/06/27/RINGS.V3/skycell.0615.037/RINGS.V3.skycell.0615.037.dif.570810 --redirect-output --run-state new --inverse --dbname gpc1 --verbose

system failure for: diff_skycell.pl --threads @MAX_THREADS@ --diff_id 570810 --skycell_id skycell.0615.037 --diff_skyfile_id 33349104 --outroot neb://stsci17.1/gpc1/ThreePi.nt/2014/06/27/RINGS.V3/skycell.0615.037/RINGS.V3.skycell.0615.037.dif.570810 --redirect-output --run-state new --inverse --dbname gpc1 --verbose
}}}
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===