PS1 IPP Czar Logs for the week 2013.09.09 - 2013.09.14

(Up to PS1 IPP Czar Logs)

Monday : 2013.09.09

mark is czar

• 08:40 MEH: finishing clearing broken LAP chips, then regular restart of stdsci
• 09:05 MEH: 4 pantasks_servers running on ippc15 by ipp, should be only 3.. restarting most pantasks.. and 5 rather than 3 running on ippc04 -- culprits were publishing 2x, stack 2x, registration 2x loaded.. kind of excessive over-starting..
  • wonder if the 2x registration has been source of odd registration hangup lately..
• 14:15 MEH: apparently ippc06 rebooted around @1345.. -- nothing on console.. someone bump it?
  • took down cleanup and detrend.. restarting..
• 14:30 MEH: ipp047 back online! apparently was never taken out of ippconfig/pantasks_hosts.input so taking out for now and putting into neb-host repair like siblings
• 15:00 MEH: ippc06 rebooting itself again.. -- taking out of processing, restarting cleanup and detrend pantasks
• 23:50 MEH: czarpoll fairly sluggish most of evening, not clear what from. going to restart

Tuesday : 2013.09.10

mark is czar

• 07:10 MEH: ipp033 down, 238 exposures behind.. -- nothing on console, back up and registration+processing continues
  • ipp033 is in repair so unclear why would stall registration -- was a check_burntool problem for neb://ipp033.0, so used regtool to clear and not have to wait, but..

    o6545g0361o  XY52 0 check_burntool neb://ipp033.0/gpc1/20130910/o6545g0361o/o6545g0361o.ota52.fits	#??? regtool -updateprocessedimfile -exp_id 654541 -class_id XY52 -set_state pending_burntool -dbname gpc1
    

• 07:55 MEH: czar stuff is very slow still, looks like ippdb01 is very busy still. dvo related? -- czarpage and status pretty much useless, updates maybe every 30-60 mins..
• 08:10 MEH: 3PI diffim fault 5, ongoing problem

  VectorFitPolynomial1DOrd (psMinimizePolyFit.c:610): unknown psLib error
       Could not solve linear equations.
  
  difftool -updatediffskyfile -set_quality 14006 -diff_id 476426 -skycell_id skycell.1049.062 -fault 0
  

• 08:15 MEH: registration finished
• 09:50 MEH: stdscience in dire need of regular restart..
• 10:50 MEH: distribution odd counts, restarting as well
• 11:50 MEH: apparently MD01,02 night stacks not made yet..
• 12:20 MEH: stack pantask odd counts, just shutdown for restart as well..
• 12:30 MEH: ippdb01 now <50% cpu usage with minimal things in processlist after cleaning out, start just distribution, can it advance runs?
  • seems to timeout? after ~700s but does advance what is found.. doesn't seem right
• 13:00 MEH: something wrong, even stacktool for MD night stacks take >5 min and seem to just timeout -- restart mysql on ippdb01
  • queries running normally again -- MD night stacks in progress, distribution running, LAP back on
  • state on restart -- running since 7/30, RAM 93%, 16G swap, often CPU@700% -- 30d regular restart useful when full 24/7 processing?

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD