PS1 IPP Czar Logs for the week 2014-02-17 - 2014-02-23

(Up to PS1 IPP Czar Logs)

Monday : 2014-02-17

• 11:30 MEH: ipp056 dumping regular messages, setting to repair until more info -- Haydn reports disk on way to failing, swapped out and raid rebuilding

  Feb 17 11:18:38 ipp056 MR_MONITOR[15606]: <MRMON113> Controller ID: 0  Unexpected sense: PD = --:--:10 - Unrecovered read error, CDB =  0x28  0x00  0x5b  0xc7  0xe8  0x8f  0x00  0x00  0x04  0x00 , Sense =  0xf0  0x00  0x03  0x5b  0xc7  0xe8  0x91  0x0a  0x00  0x00  0x00  0x00  0x11  0x00  0x00  0x00  0x00  0x00 
  Feb 17 11:18:39 ipp056 MR_MONITOR[15606]: <MRMON057> Controller ID: 0  Consistency Check corrected medium error: (VD 0 Location 0x5bc7e834, PD --:--:10 Location 0x5bc7e834)
  Feb 17 11:18:39 ipp056 MR_MONITOR[15606]: <MRMON110> Controller ID: 0  Corrected medium error during recovery: PD --:--:10 Location 0x5bc7e834
  
  

• 12:20 MEH: also looks like the ippdb02 mysql backup to ippc63.1 encountered odd state and aborted at 09:31.. will need to fix so can start mysql again on ippdb02..
  • script gets week number and of course 08,09 are problems as bash interprets 0x as octal notation..
  • modified script and crontab, will restart backup process 13:31 and mysql normally restarted when finished..

Tuesday : 2014-02-18

• 08:00 MEH: ippdb02 backup to ippc63 still running, should finish later today
• 11:33 Bill: reverted staticsky runs with fault 2 (33 runs) and 3 (11 runs) since the information in the logs indicates that the failures were due to cluster problems.
• 14:00 MEH: odd pstamp/update faults w/ data_group M31.rp.2013.20110102, chip.revert.off for a bit
• 14:11 CZW: repointed the LAP queue file to the active one. This has been off since the timeout issue last week (still unresolved). Sudden thought while typing this out: the point where it ran into problems was skycell.2211/i, which is the 3664th entry in the old queue file. Since it needs to check the previous entries to ensure they haven't been done yet, this can probably stretch out to 10 minutes just in checking old entries. I've cut the currently active queue down to a three hour slice, to see if that keeps the number of things to check in check.
• 18:00 MEH: ipp064 volume degraded, already in neb-host repair for dvo work
• 20:50 MEH: ipp023 is unresponsive.. looks like crashed little after 20:20, neb-host down.. -- nothing on console, like ipp026, another weak node with history of crashing..
  • 22:00 looks like ipp023 down has pissed off ippc01 as well -- display says general protection fault occurred, will need to power cycle as well..
• 23:35 MEH: ippc01 power cycled and back up, mess of stalled jobs and mounts cleared. restarting pantasks slowly
• 00:20 MEH: there is a mix of undocumented manual setting/changes not tracked for normal loading using the ~ipp/ippconfig/pantasks_hosts.input.... have to untangle before stdsci+stk+staticsky can be started..
• 00:30 MEH: summitcopy 500 read timeouts ongoing..

Wednesday : 2014-02-19

• 09:30 MEH: manaully raising some node usage for stack
• 10:00 MEH: clearing out some LAP burn.tbl faults found missing on ippb02 ( since ipp023 is down). will be a mess of cam+warp+stack faults while ipp023 is down
• 10:57 Bill: reverted all faulted staticsky runs. Looks like many of the fault 4s were due to system crashes or memory problems. Rerunning them all to get a better handle on which are due to bugs in the code.
• 12:20 MEH: Haydn replaced ipp023 mobo, back up. neb-host repair for time being. all x.revert.on again -- '
• 15:50 MEH: ipp037,041,045 seem to be running 3x staticsky, but have only 8 core.. and are data nodes, manually reducing to 2x..
• 16:15 Bill: set staticsky to stop in preparation for rebuilding psphotStack with a couple of bug fixes
  • psphotStack rebuilt setting to run. Reverted the fault 4s caused by the bug. Some will work now. Others will get fault 3.
• 16:46 Bill: manually edited some staticskyRuns that have inputs with <1% good_frac.
  • sky_id 459346 deleted input stack_id 2480292
  • sky_id 459737 deleted input stack_id 2561398
  • sky_id 462496 deleted input stack_id 3000094
  • sky_id 460690 set to drop only has 1 input good_frac 0.000507
  • sky_id 461050 all inputs too sparse
• 18:40 MEH: sky looking clear and NEO night, so rebalancing system to push more stdsci (if other not-czars are turning things on/off, it really needs to get logged here)..
• 19:15 MEH: ipp040,043,050,052,053 taking excessive time in registration, neb-host repair and off in registration for now
• 20:10 MEH: turning off nodes on in summitcopy (wave1, ipp037,041,045,047) -- total 46 or so running
  • ipp019 has problem with network, cannot manually wget from conductor there

    request failed: 500 Can't connect to conductor.ifa.hawaii.edu:80 (connect: Network is unreachable) at /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/dsget line 150.
    

Thursday : 2014-02-20

Friday : 2014-02-21

Saturday : 2014-02-22

Sunday : 2014-02-23