== PS1 IPP Czar Logs for the week 2011.10.17 - 2011.10.23 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.10.17 ===
 * 11:00 - 11:15 update pantasks died. Restated by bills. distribution is backed up. Restarted pantasks
 * 12:45 Mark: set ipp029 down in nebulous for Cindy to install new ram. 
 * 13:40 Added back to nebulous in repair state initially. Will set to up in nebulous and add back in gradually for testing.
 * 14:15 One stdscience job started on ipp029 ran for ~15min before it crashed.
 * 20:45 stdscience seems to be under-running, restarting.
=== Tuesday : 2011.10.18 ===
 * 06:00 registration stalled. restarted pantasks and stalled. started running again after ~20 minutes.
 * 06:20 ippc13 down for 5hr. rebooted.
 * 06:25 warprun stalled
{{{
Reading FITS file /data/ipp027.0/nebulous/c7/38/1467947204.gpc1:ThreePi.nt:2011:10:18:o5852g0316o.408936:o5852g0316o.408936.ch.327264.XY44.ch.wt.fits failed.

perl ~ipp/src/ipp-20110622/tools/runchipimfile.pl --chip_id  327264  --class_id XY44 --redirect-output 
}}}
 * 06:30 registration now stuck with ipp033 mount to ipp028, running again after
{{{
/usr/local/sbin/force.umount ipp028
}}}
 * 06:35 stdscience pantasks down. restarted...
 * 06:50 registration stalled again... restarted. summitcopy stalled with 26 images left. restarted. looks like o5852g0500o/o5852g0500o46.fits got stalled with mount problem and failed with 
{{{
failure for: summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o5852g0500o/o5852g0500o46.fits --filename neb://ipp029.0/gpc1/20111018/o5852g0500o/o5852g0500o.ota46.fits --summit_id 404735 --exp_name o5852g0500o --inst gpc1 --telescope ps1 --class chip --class_id ota46 --bytes 51831360 --md5 7e8c954ded69e0537d465a1f457ebc26 --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous
}}}
 summitcopy still not loading and running manually fails to insert row now. Entry looks okay, no fault. Talking with Bill to trace out why/how to fix.
 * 09:50 Bill traced it to variable CLASS_ID not found and put in a temporary fix.
 * 10:00 will delay shutdown for nebulous repair until remaining data looks to make it through and MOPS gets postage stamps. 
 * 11:00 systems shutdown, apache off for nebulous/ippdb02 repair.
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===