(Up to PS1 IPP Czar Logs)

Monday : 2012.12.10

• 08:40 Bill dropped a chip that had no detections and popped an assertion failure chiptool -dropprocessedimfile -set_quality 99 -chip_id 725138 -class_id XY76
• 08:42 set sts.rerun.20121129 back to active 565 pending warps. They are slow because our pantasks host assigment is designed for maximum use of cpu cores, but sts fields are memory hogs.
• 09:50 MEH: ipp011 neb-host up to see if can crash it again
• 12:00 MEH: rebooting ipp026 from weekend trouble, adding back into processing and neb-host up
• 13:00 MEH: ipp011 neb-host repair, being a glock tarpit (holding +20 mins). things slowly clearing now.
• 16:30 MEH: clearing some remaing stalled STS jobs on ipp066. doing a reboot of ipp066, restart of stack and stdscience and distribution
• 18:35 MEH: looks like ipp029 down... nothing on console.. apparently didn't like having nothing to do..
  • of course, not rebooting..
  • ipp029 neb-host down, taking out of processing (should be already)
  • nightly science now proceeding @19:20

Tuesday : 2012.11.11

• 09:13 Bill stopped distribution. There are some recurring faults that need to be debugged. May be related to ipp029 outage.
• 09:15 bill set skycal to off in preparation for an adding a column to the skycalResult table
• 09:35 Serge: Restarted mysql servers on all storage nodes but: ipp024, ipp025, ipp026, and ipp029 (down). Need to play the ~schastel/dev/FilesMonitoring/NodeRelated/maintenance/refresh_status_table_creation_mhpcc.sql script on them.
• 10:25 Serge: When ipp029 is back, install the following crontab (as schastel user): ~schastel/dev/FilesMonitoring/NodeRelated/deployment/scripts/crontab.ipp029
• 10:45 Bill: stopped processing for rebuild for schema change
• 11:15 MEH: with processing stopped, did a double restart of rpc.statd on ipp020 (was using large amount of swap space)
• 11:45 Bill: Set pantasks to run. Restarted stdscience pantasks without compute3 started deepstack with 2 x compute3
• 20:50 MEH: apparently stack pantasks isn't running.. MD nightly waiting to run. can't see any crash messages, was it not restarted earlier today? not sure why roboczar didn't report it.
  • Bill notes likely silent crash from tag being rebuilt earlier, why roboczar didn't report (and haven't seen in previous full group shutdown warnings in a while)
  • MEH had checked wrong list -- roboczar wasn't watching stack or publishing -- adding to czartool/czarconfig.xml, now it watches stdscience distribution summitcopy registration pstamp stack publishing

Wednesday : 2012.12.12

Bill is the czar today

• 07:40 added a third set of compute3 hosts to deepstack
• 08:10 queued 73000 skycal lap skycal runs for the region RA < 227 dec < 10

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD