== PS1 IPP Czar Logs for the week 2014.05.19 - 2014.05.25 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.05.19 ===

 * 07:30 EAM : setting warp off while nightly science finished diffs; also bumping up the pol to 600.  actually, thing are running slow, so I'm restarting stdscience. 
 * 12:20 EAM : for a test of relastro, I am turning off ipp04x for processing in stdscience
 * 14:20 HAF: heather stopped stack for ippc05 replacement drive (shouldn't take too long)  
 * 17:05 HAF: haydn repaired 42/36/c05 - only 42/c05 are up (he's still working on 36)
 * 17:10 HAF: postage stamp server ws down. restarted, not sure why it died.
 * 19:33 HAF: turned off Lap.* labels in stdsci and stack, as suggested by gene for nighly LAP processing. will turn on again in the AM

=== Tuesday : 2014.05.20 ===
 * 8:ish HAF/GENE added labels back into stack/stdsci, reverted stuff
 * 09:31 Bill: regenerated a couple of missing burntool tables
{{{
ipp@ippc18:/home/panstarrs/ipp>perl ~bills/ipp/tools/fixburntool --exp_id 144139 --class_id XY74
/home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 144139 --class_id XY74 --this_uri neb://ipp015.0/gpc1/20100302/o5257g0090o/o5257g0090o.ota74.fits --previous_uri neb://ipp015.0/gpc1/20100302/o5257g0089o/o5257g0089o.ota74.fits
Running [/home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 144139 --class_id XY74 --this_uri neb://ipp015.0/gpc1/20100302/o5257g0090o/o5257g0090o.ota74.fits --previous_uri neb://ipp015.0/gpc1/20100302/o5257g0089o/o5257g0089o.ota74.fits]...
      Deleted 1 chipProcessedImfiles
      Updated 0 chipProcessedImfiles
Success
ipp@ippc18:/home/panstarrs/ipp>perl ~bills/ipp/tools/fixburntool --exp_id 175176 --class_id XY15
/home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 175176 --class_id XY15 --this_uri neb://ipp040.0/gpc1/20100530/o5346g0436o/o5346g0436o.ota15.fits --previous_uri neb://ipp040.0/gpc1/20100530/o5346g0435o/o5346g0435o.ota15.fits
Running [/home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 175176 --class_id XY15 --this_uri neb://ipp040.0/gpc1/20100530/o5346g0436o/o5346g0436o.ota15.fits --previous_uri neb://ipp040.0/gpc1/20100530/o5346g0435o/o5346g0435o.ota15.fits]...
      Deleted 1 chipProcessedImfiles
      Updated 0 chipProcess
}}}
 * 09:54 Bill: restarted pstamp pantasks. It was running slowly because of a tired pcontrol.
 * 14:54 Bill: ippc30 is getting nfs overload from pstamp jobs. set one set of c2 nodes to off.
  * 15:00 Bill installed new version of pstamp_finish.pl which does not recompute the file size and md5sum and uses the values calculated in pstamp_job_run.pl 
 * 17:00 HAF is restarting the pantasks at the request of Bill.
 * 17:38 HAF: ok stuff is restarted and same as last night I pulled out the labels for LAP. Will re-add them tomorrow
 * 22:30 MEH: adding LAP.ThreePi.20130717.pole back into stack since uses nodes independent of stdsci -- will watch for couple hours to verify rate is similar

=== Wednesday : 2014.05.21 ===
mark is czar
 * 01:10 MEH: stdsci rate seems similar w/ stack running (as well as staticsky) on compute nodes. leaving in -- though it seems the default stack allocation of nodes is a bit to high on the c2 group w/ staticsky, someone has modified things and they are not initially compatible -- '''warning whenever restarting pantasks...'''
 * 08:00 MEH: leftover stuck dark -- o6798g0020d
 * 09:50 MEH: restart summitcopy+registration with ipp036,042 back in processing
  * recent restarts of pantasks_servers have not been rolling over log files -- 
 * 11:30 MEH: tasking some s3 nodes since unused currently -- taking all s3 nodes from stdsci for other tasks..
 * 16:00 MEH: restarting stdsci and stack -- turning MD06 back on, fixing auto-overload of c2 since being used by staticsky, forcing days old logs to actually rotate in order to track problems easier
{{{
 -- giving all of c2 to staticsky (3x) since out of plane 
}}}
 * 19:50 MEH: reminder LAP label out of stdsci for nightly processing once it starts
=== Thursday : 2014.05.22 ===

=== Friday : 2014.05.23 ===

=== Saturday : 2014.05.24 ===

=== Sunday : 2014.05.25 ===