[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.12.10 ===
 * 08:40 Bill dropped a chip that had no detections and popped an assertion failure chiptool -dropprocessedimfile -set_quality 99 -chip_id 725138 -class_id XY76
 * 08:42 set sts.rerun.20121129 back to active 565 pending warps. They are slow because our pantasks host assigment is designed for maximum use of cpu cores, but sts fields are memory hogs.
 * 09:50 MEH: ipp011 neb-host up to see if can crash it again
 * 12:00 MEH: rebooting ipp026 from weekend trouble, adding back into processing and neb-host up
 * 13:00 MEH: ipp011 neb-host repair, being a glock tarpit (holding +20 mins). things slowly clearing now.
 * 16:30 MEH: clearing some remaing stalled STS jobs on ipp066. doing a reboot of ipp066, restart of stack and stdscience and distribution
 * 18:35 MEH: looks like ipp029 down... nothing on console.. apparently didn't like having nothing to do..
  * of course, not rebooting..
  * ipp029 neb-host down, taking out of processing (should be already)
  * nightly science now proceeding  @19:20
=== Tuesday : 2012.11.11 ===
 * 09:13 Bill stopped distribution. There are some recurring faults that need to be debugged. May be related to ipp029 outage.
 * 09:15 bill set skycal to off in preparation for an adding a column to the skycalResult table
 * 09:35 Serge: Restarted mysql servers on all storage nodes but: ipp024, ipp025, ipp026, and ipp029 (down). Need to play the {{{~schastel/dev/FilesMonitoring/NodeRelated/maintenance/refresh_status_table_creation_mhpcc.sql}}} script on them.
 * 10:25 Serge: When ipp029 is back, install the following crontab (as schastel user): {{{~schastel/dev/FilesMonitoring/NodeRelated/deployment/scripts/crontab.ipp029}}}
 * 10:45 Bill: stopped processing for rebuild for schema change
 * 11:15 MEH: with processing stopped, did a double restart of rpc.statd on ipp020 (was using large amount of swap space)
 * 11:45 Bill: Set pantasks to run. Restarted stdscience pantasks without compute3 started deepstack with 2 x compute3
 * 20:50 MEH: apparently stack pantasks isn't running.. MD nightly waiting to run. can't see any crash messages, was it not restarted earlier today? not sure why roboczar didn't report it.
  * Bill notes likely silent crash from tag being rebuilt earlier, why roboczar didn't report (and haven't seen in previous full group shutdown warnings in a while)
  * MEH had checked wrong list -- roboczar wasn't watching stack or publishing -- adding to czartool/czarconfig.xml, now it watches ''stdscience distribution summitcopy registration pstamp stack publishing''
=== Wednesday : 2012.12.12 ===
 Bill is the czar today
 * 07:40 added a third set of compute3 hosts to deepstack
 * 08:10 queued 73000 skycal lap skycal runs for the region RA < 227 dec < 10
 * 16:51 set ipp029 to repair mode (should have done this much earlier)
 * 19:15 stdscience is sluggish. Setting to stop in preparation for restart
=== Thursday : 2012.12.13 ===
 Bill is czar today
 * 05:43 restarted stdscience with some tweaks to skycal.pro to start jobs more often if there is work to do
 * 10:55 noticed that ipp020 has become a job staller. Killed running jobs there and set host to off in pstamp and stdscience
 * 11:00 restarted ipp020 rpc.statd
 * 11:08 dropped the 620 skycalRuns in the kepler region that had excess matched detections. Set the 124 associated staticskyRuns to be re-run (set result.fault skycalRun.state = 'new' and revert. These will now run with the new psphotStack.
 * 12:27 added more hosts to deepstack pantasks. 2 x stare, 1 x compute2 1 x wave4. Will need to remove these before nightfall
 * 13:14 restarted rpc.statd on ipp019
 * 16:57 set the extra hosts added to deepstack at 12:27 to off
=== Friday : 2012.12.14 ===
 Mark is czar

 * 06:20 Bill: set stare. compute2, and wave4 hosts to on in deepstack
 * 08:00 MEH: ipp020 hung mounts, hanging on to ISP dark image registration and cleanup run from yesterday. restarting registration, will restart ipp020 in hour or so.
  * after a full nfs/rpc stop for a bit and then start, rpc restart, all mounts are back and nfs doesn't report export of cache tainted @09:20, no reboot as hanging jobs cleared.
  * all mounts check out okay while system mostly idle
 * 10:25 ipp020 mount trouble may be back -- and cleared, maybe after did mysql restart there
 * 11:13 bill cleaned up the staticsky mess that he made and queued new runs for the affected skycells
 * 11:15 MEH: mysql running amok on most of data nodes, python refresh_table.py running from past 3 days, suspect stalled and will need to start restarting mysql on all machines..
  * ipp028 mysql using 300% CPU and 11 refresh_table.py processes
  * ipp039 mysql using 300% CPU and 4 refresh_table.py processes
=== Saturday : 2012.12.15 ===

=== Sunday : 2012.12.16 ===