PS1 IPP Czar Logs for the week 2013.11.04 - 2013.11.10

(Up to PS1 IPP Czar Logs)

Monday : 2013.11.04

  • 12:20 Set stdscience and pstamp to stop in preparation for daily restart.
  • 12:43 stdscience, pstamp and update restarted
  • 15:00 just noticed stdscience has been running with a poll limit of 30 instead of 300. Not sure why
    • MEH: clearing out final MD02.pv2 issues -- thanks for resetting (have actually also bumped to 500 since update processes continuing with MD10.pv2

Tuesday : 2013.11.05

  • 09:17 restarted stdscience
  • Cleared three chips which had fault = 26. I thought that I'd cleared all of those but there are 1414 of them with nonzero quality. Most are from 2011 lap and test runs.
    • MEH: have been fixing misc ones etc with MD, if possible, please don't bulk repair any related to MD.
  • 18:22 restarted stdscience (twice a day seems good) set poll back to 500 (which I think causes pcontrol to back up sooner but that's anecdotal)
    • yes if update processing, then before+after nightly is pretty much necessary. 500 is more to help when things get into warp only case and to keep the queue full but also can help dump more into the pool/book when it gets >100k Njobs and keeping that > available nodes...

Wednesday : 2013.11.06

  • 15:00 MEH: manually managing c3 stack and deepstack allocation -- current cut deep stacks are low enough ram use to run normal stacks but not enough for 2x deep stack and LAP could use the extra stack nodes

Thursday : 2013.11.07

  • 07:15 Bill: restarted stdscience, pstamp, update, publishing, distribution, cleanup, summitcopy, and registration pantasks
  • 15:30 MEH: w/o nightly data, not making much progress clearing out LAP stacks so realloc attempt -- -3x c3 from stdsci, +1x c3 stk

Friday : 2013.11.08

  • 09:37 Bill: turned warp revert off to take a look at an unusual recurring assertion failure in psphot
  • 10:01 Bill: warptool -updateskyfile -fault 0 -set_quality 3006 -warp_id 858484 -skycell_id skycell.2197.019. There is a bug in psphot that triggered an assertion failure. Running with code that detects the problem and avoids the assertion this skycell which has few pixels and those that exist look like pure noise gets "Unable to determine PSF" quality=3006. So I manually updated the skycell.

Saturday : 2013.11.09

  • 16:00 MEH: available disk space minimal, turning off LAP (removing label from stdsci) for now. restarting stdsci for nightly and putting c3 allocation back to stdsci now that LAP stacks mostly caught up.
    • finished md10.pv2 check and sending chips to cleanup -- freed up enough space on stsci10-19 to turn off from red for nightly processing..

Sunday : 2013.11.10