PS1 IPP Czar Logs for the week 2014.05.19 - 2014.05.25

(Up to PS1 IPP Czar Logs)

Monday : 2014.05.19

• 07:30 EAM : setting warp off while nightly science finished diffs; also bumping up the pol to 600. actually, thing are running slow, so I'm restarting stdscience.
• 12:20 EAM : for a test of relastro, I am turning off ipp04x for processing in stdscience
• 14:20 HAF: heather stopped stack for ippc05 replacement drive (shouldn't take too long)
• 17:05 HAF: haydn repaired 42/36/c05 - only 42/c05 are up (he's still working on 36)
• 17:10 HAF: postage stamp server ws down. restarted, not sure why it died.
• 19:33 HAF: turned off Lap.* labels in stdsci and stack, as suggested by gene for nighly LAP processing. will turn on again in the AM

Tuesday : 2014.05.20

• 8:ish HAF/GENE added labels back into stack/stdsci, reverted stuff
• 09:31 Bill: regenerated a couple of missing burntool tables

  ipp@ippc18:/home/panstarrs/ipp>perl ~bills/ipp/tools/fixburntool --exp_id 144139 --class_id XY74
  /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 144139 --class_id XY74 --this_uri neb://ipp015.0/gpc1/20100302/o5257g0090o/o5257g0090o.ota74.fits --previous_uri neb://ipp015.0/gpc1/20100302/o5257g0089o/o5257g0089o.ota74.fits
  Running [/home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 144139 --class_id XY74 --this_uri neb://ipp015.0/gpc1/20100302/o5257g0090o/o5257g0090o.ota74.fits --previous_uri neb://ipp015.0/gpc1/20100302/o5257g0089o/o5257g0089o.ota74.fits]...
        Deleted 1 chipProcessedImfiles
        Updated 0 chipProcessedImfiles
  Success
  ipp@ippc18:/home/panstarrs/ipp>perl ~bills/ipp/tools/fixburntool --exp_id 175176 --class_id XY15
  /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 175176 --class_id XY15 --this_uri neb://ipp040.0/gpc1/20100530/o5346g0436o/o5346g0436o.ota15.fits --previous_uri neb://ipp040.0/gpc1/20100530/o5346g0435o/o5346g0435o.ota15.fits
  Running [/home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 175176 --class_id XY15 --this_uri neb://ipp040.0/gpc1/20100530/o5346g0436o/o5346g0436o.ota15.fits --previous_uri neb://ipp040.0/gpc1/20100530/o5346g0435o/o5346g0435o.ota15.fits]...
        Deleted 1 chipProcessedImfiles
        Updated 0 chipProcess
  

• 09:54 Bill: restarted pstamp pantasks. It was running slowly because of a tired pcontrol.
• 14:54 Bill: ippc30 is getting nfs overload from pstamp jobs. set one set of c2 nodes to off.
  • 15:00 Bill installed new version of pstamp_finish.pl which does not recompute the file size and md5sum and uses the values calculated in pstamp_job_run.pl
• 17:00 HAF is restarting the pantasks at the request of Bill.
• 17:38 HAF: ok stuff is restarted and same as last night I pulled out the labels for LAP. Will re-add them tomorrow
• 22:30 MEH: adding LAP.ThreePi.20130717.pole back into stack since uses nodes independent of stdsci -- will watch for couple hours to verify rate is similar

Wednesday : 2014.05.21

mark is czar

• 01:10 MEH: stdsci rate seems similar w/ stack running (as well as staticsky) on compute nodes. leaving in -- though it seems the default stack allocation of nodes is a bit to high on the c2 group w/ staticsky, someone has modified things and they are not initially compatible -- warning whenever restarting pantasks...
• 08:00 MEH: leftover stuck dark -- o6798g0020d
• 09:50 MEH: restart summitcopy+registration with ipp036,042 back in processing
  • recent restarts of pantasks_servers have not been rolling over log files --
• 11:30 MEH: tasking some s3 nodes since unused currently -- taking all s3 nodes from stdsci for other tasks..
• 16:00 MEH: restarting stdsci and stack -- turning MD06 back on, fixing auto-overload of c2 since being used by staticsky, forcing days old logs to actually rotate in order to track problems easier

   -- giving all of c2 to staticsky (3x) since out of plane 
  

• 19:50 MEH: reminder LAP label out of stdsci for nightly processing once it starts
• 23:20 MEH: fault 5 diffims to clear, poor weather so LAP label back in until data or sleep

Thursday : 2014.05.22

mark is czar

• 06:30 MEH: appears to be stuck files from last nights power glitch
• 08:46 Bill: in the process of restarting the postage stamp server. Waiting for a huge request to finish parsing.
• 09:45 MEH: summit files available again, LAP label out until nightly finished

Friday : 2014.05.23

Saturday : 2014.05.24

Sunday : 2014.05.25