PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2013.03.25

  • 12:40 MEH: again stdscience needs regular restart, doing before turning on MD01.pv1.20130325
  • 13:50 MEH: distribution counters wacky.. just going to do a full restart to track MD to setup for distribution
  • 14:40 MEH: stack counters also wacky.. going to also do a full restart to more easily track reredo of MD stack photometry on newer refstacks
    • unclear where staticsky should be running and it is still commented out in stack, going to restart stack with staticsky running there since deepstack shutdown and may need to use it for full deepstacks soon
  • 16:10 MEH: compute,compute2 are 2x in stack and staticsky overloading those. don't feel like monkeying around with allocation again, taking staticsky out of stack and starting deepstack with 1x compute3 -- will decide where to balance the compute3 from stdscience or stack later

Tuesday : 2013.03.26

  • 09:45 Bill: The czartool page is working. It draws the upper plots and then browser goes into wait loop before showing the pantasks or other status. Tried restarting czarpoll on ippc11 and apache on ippdb03 with no help
  • 10:00 Bill: restarted pstamp and update pantasks It looks like we had a period of database timeouts over the weekend. set pstamp to stop to debug a dependency checking problem
  • 10:45 Bill: czartool problem was caused by ipp001 being out of space. Who knew?
  • 10:50 restarted cleanup
  • 10:55 MEH: stdscience needs its regular restart, adding next MD03.pv1.20130326 label
  • 14:37 Bill's special stacks are done. bills.20130322 removed from th e pantasks

Wednesday : 2013.03.27

  • 09:30 MEH: regular restart of stdscience
  • 10:00 MEH: while dealing with some broken chips..
  • 10:25 Bill: stare00 set to off in stdscience. Rita wants to open it up and look at the memory chips.
  • 11:37 CZW: stopping processing to allow Rita to check ipp064 and compute nodes.
  • 12:20 MEH: restarted mysql crashed on ipp026,035,049,054,058,061,065
  • 12:50 CZW: restarting processing.
  • 22:10 MEH: ippc01:/tmp/nebulous_server.log using all local diskspace. likely don't need to save, but moving to /export/ippc01/tmp and from czarlogs before
    rm /tmp/nebulous_server.log ; touch /tmp/nebulous_server.log ; /etc/init.d/apache2 graceful
    -- nebulous_server.log needs to be owned by apache, likely want to stop apache if archiving the log

Thursday : 2013.03.28

  • 11:30 MEH: sending all the re-re-reprocessing MD chips to cleanup, unclear plans for deepstacks so keeping the warps for now
  • 21:00 Bill: restarted summit copy and registration pantasks as they have been running for awhile

Friday : YYYY.MM.DD

  • 16:25 MEH: doing regular restart of stdscience before nightly data starts

Saturday : YYYY.MM.DD

Sunday : 2013.03.31

  • 01:20 MEH: with no/little nightly data, restarting deepstack (to run.. deepstack tests) using compute3. rebuilt ippconfig with a no-cut deep stack reduction config