(Up to PS1 IPP Czar Logs)

Monday : 2012.12.10

• 08:40 Bill dropped a chip that had no detections and popped an assertion failure chiptool -dropprocessedimfile -set_quality 99 -chip_id 725138 -class_id XY76
• 08:42 set sts.rerun.20121129 back to active 565 pending warps. They are slow because our pantasks host assigment is designed for maximum use of cpu cores, but sts fields are memory hogs.
• 09:50 MEH: ipp011 neb-host up to see if can crash it again
• 12:00 MEH: rebooting ipp026 from weekend trouble, adding back into processing and neb-host up
• 13:00 MEH: ipp011 neb-host repair, being a glock tarpit (holding +20 mins). things slowly clearing now.
• 16:30 MEH: clearing some remaing stalled STS jobs on ipp066. doing a reboot of ipp066, restart of stack and stdscience and distribution
• 18:35 MEH: looks like ipp029 down... nothing on console.. apparently didn't like having nothing to do..
  • of course, not rebooting..
  • ipp029 neb-host down, taking out of processing (should be already)
  • nightly science now proceeding @19:20

Tuesday : 2012.11.11

• 09:13 Bill stopped distribution. There are some recurring faults that need to be debugged. May be related to ipp029 outage.
• 09:15 bill set skycal to off in preparation for an adding a column to the skycalResult table
• 09:35 Serge: Restarted mysql servers on all storage nodes but: ipp024, ipp025, ipp026, and ipp029 (down). Need to play the ~schastel/dev/FilesMonitoring/NodeRelated/maintenance/refresh_status_table_creation_mhpcc.sql script on them.
• 10:25 Serge: When ipp029 is back, install the following crontab (as schastel user): ~schastel/dev/FilesMonitoring/NodeRelated/deployment/scripts/crontab.ipp029
• 10:45 Bill: stopped processing for rebuild for schema change
• 11:15 MEH: with processing stopped, did a double restart of rpc.statd on ipp020 (was using large amount of swap space)
• 11:45 Bill: Set pantasks to run. Restarted stdscience pantasks without compute3 started deepstack with 2 x compute3
• 20:50 MEH: apparently stack pantasks isn't running.. MD nightly waiting to run. can't see any crash messages, was it not restarted earlier today? not sure why roboczar didn't report it.
  • Bill notes likely silent crash from tag being rebuilt earlier, why roboczar didn't report (and haven't seen in previous full group shutdown warnings in a while)
  • MEH had checked wrong list -- roboczar wasn't watching stack or publishing -- adding to czartool/czarconfig.xml, now it watches stdscience distribution summitcopy registration pstamp stack publishing

Wednesday : 2012.12.12

Bill is the czar today

• 07:40 added a third set of compute3 hosts to deepstack
• 08:10 queued 73000 skycal lap skycal runs for the region RA < 227 dec < 10
• 16:51 set ipp029 to repair mode (should have done this much earlier)
• 19:15 stdscience is sluggish. Setting to stop in preparation for restart

Thursday : 2012.12.13

Bill is czar today

• 05:43 restarted stdscience with some tweaks to skycal.pro to start jobs more often if there is work to do
• 10:55 noticed that ipp020 has become a job staller. Killed running jobs there and set host to off in pstamp and stdscience
• 11:00 restarted ipp020 rpc.statd
• 11:08 dropped the 620 skycalRuns in the kepler region that had excess matched detections. Set the 124 associated staticskyRuns to be re-run (set result.fault skycalRun.state = 'new' and revert. These will now run with the new psphotStack.
• 12:27 added more hosts to deepstack pantasks. 2 x stare, 1 x compute2 1 x wave4. Will need to remove these before nightfall
• 13:14 restarted rpc.statd on ipp019
• 16:57 set the extra hosts added to deepstack at 12:27 to off

Friday : 2012.12.14

• 06:20 Bill: set stare. compute2, and wave4 hosts to on in deepstack
• 08:00 MEH: ipp020 hung mounts, hanging on to ISP dark image registration and cleanup run from yesterday. restarting registration, will restart ipp020 in hour or so.
  • after a full nfs/rpc stop for a bit and then start, rpc restart, all mounts are back and nfs doesn't report export of cache tainted

Saturday : 2012.12.15

Sunday : 2012.12.16