PS1 IPP Czar Logs for the week 2013.03.04 - 2013.03.11

(Up to PS1 IPP Czar Logs)

Monday : 2013.03.04

mark is czar

  • 07:20 MEH: to push remaining 3PI nightly warps through
  • 08:00 MEH: nightly science data through so do regular restart of stdscience
  • 10:40 Bill: Richard came by and was concerned that MOPS's postage stamp requests are behind. Lowered priority of MOPS.2 label. Restarted pstamp pantasks because there didn't seem to be enough jobs in the queues. pcontrol was spinning I think.
  • 14:20 MEH: something making stsci03 unhappy starting ~1400 driving load >100..

Tuesday : 2013.03.05

mark is czar

  • 09:00 MEH: with nightly science finished, time for regular restart of stdscience
  • 09:42 Bill: queued 8723 staticsky runs
  • 09:45 Bill: replaced corrupted instance of o5498g0622o.ota26.fits on ipp064 with it's replicant from ippb01.1
  • 17:20 MEH: large LAP warp buildup again, until nightly science starts to get some stacks queued

Wednesday : 2013.03.06

  • 09:51 Bill: postage stamp server pantasks died around 9:30. It's been very busy. Restarted it.

Thursday : 2013.03.07

  • 12:45 MEH: stopping processing to setup+build new tag ipp-20130307
  • 15:20 MEH: tag build finished, now testing
    • running test sample MD07 exposures from last night
    • stopped stdscience, looks like it is redoing MD WSdiffs, query may need to be adjusted.. -- Chris found was from old warps that had missed being cleaned up, sent them now
    • will have to slowly go through and drop duplicates, messed up publishRun entries and need to fix first.
    • some duplicates also from MD refstack having duplicate skycells, missed when replacing poor skycells. tracing back and changing label for ones not used before.
  • 18:00 MEH: set duplicate diffims to state drop.
    • 23 MD diffims were made (MD05,06,07), registered like "2013-03-08 01%"
    • 34 3PI diffims were made registered like "2013-03-07 2% -- no these are normal nightly science that finished up ~12:20. so was just MD
  • 20:30 MEH: additional duplicate diffs showing up in publishing, set state drop and turned on for nightly science.
  • 20:45 MEH: re-activating LAP label, notice 212k camera stage new entries in distribution. removing label from distribution until determine if supposed to be.
  • 21:30 MEH: also looks like isp was having summitcopy trouble, not sure what changed

Friday : 2013.03.08

  • 05:30 Bill: unintended camera distribution was due to a change in disttool that changed the default to not require magic. Unfortunately, the LAP camera runs were still set up for distribution but they weren't being queued because the correspondng survey.add.dist command in stdscience/input was set to require destreaking. Disabled the targets and dropped the runs.
  • 07:04 Bill: several camera runs are repeatedly failing due to something with the background stuff turned
  • 07:15 Bill: fixed bug in ppStatsFromMetadata.config that was causing staticskytool -addskycalresult to fail
  • 07:41 Bill: warp has problems as well Turning revert off.
  • 08:28 Bill: warp failure rate for the currently queued LAP data is nearly 100% turning it off
  • 08:34 Bill: ran hosts.update in the update pantasks to hurry up the pending mpia requests

Saturday : 2013.03.09

  • 18:05 MEH: stdscience running about half loaded, needs regular restart and doing before nightly science

Sunday : 2013.03.10