PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2013.10.07

• 09:20 Bill restarted pstamp pantasks
• 17:30 CZW: ipp051 and ipp063 had to be power cycled, as they had memory issues due to galactic plane.
• 22:50 MEH: surprise, registration stuck.. 70 exposures behind -- restarting registration seems to have cleared
• 00:10 MEH: then 2 OSS chips stalling on ipp028 for ~3ks.. restarted stdsci, STS and LAP label out
  • stsci10-19 also down and not off.. something was odd..
• 01:20 MEH: OSS warps MIA.. restarting stdsci again found them. also five OSS had fault 5

  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1501.041 -diff_id 483695 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1413.083 -diff_id 483713 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1413.095 -diff_id 483713 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1322.020 -diff_id 483726 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1322.090 -diff_id 483726 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1414.007 -diff_id 483727 -fault 0
  

Tuesday : 2013.10.08

mark is czar

• 07:00 MEH: nightly mostly done, STS back in. 3PI fault 5 diffim

  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1324.041 -diff_id 483889 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1332.035 -diff_id 483964 -fault 0
  
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1233.080 -diff_id  483855 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1234.079 -diff_id  483859 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1235.098 -diff_id  483863 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1331.020 -diff_id  483978 -fault 0
  

• 09:20 MEH: ipp034 back up, put back into neb-host repair. ipp0016 put into repair over weekend because problematic, put back up now after rebooted sunday..
• 09:25 adding LAP label back in now --
• 09:40 MEH: continuing general preventative maintenance -- restarting long running pantasks (summit, cleanup, distribution, publishing, stack; did registration last night) -- been month+ since last log archive to clear up space on ippc18.0
  • ippc18 disk is overused, cannot rsync the logs ippc18.0 to ippc18.1 w/o over-driving load and killing processing.. suspect too many accounts are writing too many logs to home disk. have to set rsync to <800KPS
• 09:50 MEH: ipp049 had long running (>2hr) pswarp and ppImage, killed and reverted but looks like another pair ppImage stalling --
• 10:00 MEH: working in nodes that have been recently fixed
  • ipp031,032
  • ipp047
• 11:15 MEH: still no LAP stacks, putting 2x compute3 from stack into pstamp for MOPS (18.5k jobs)
  • possibly a lot of data for MOPS on ipp017 requested at once? driving load >200.. settling down now. need to check if also was from access by UMD for high throughput tests at that time
• 12:00 MEH: addressing low disk space on ippc0x machines due to /tmp/nebulous_server.log
• 12:35 MEH: MOPS stamps finished, restarting pstamp and putting the 2x compute3 back to stack
• 13:00 MEH: LAP has worked through the stalled updates and many stacks being loaded. switching STS reprocessing label back to its original 202 priority (this is above LAP)
• 13:50 MEH: ipp047 has be on/off being horrible overloaded in RAM and load.. probably also need powercycle -- finally got a login and killed off ppImage, Heather also stopping jobs and seems to be back -- have to watch the RAM use by mysql
• 14:00 MEH: looks like ippdb01 has died.. -- nothing on console, powercycle seems to be rebooting now (will take a terribly long time for RAM check)
• 16:00 MEH: MD10z stack fault 5 due to all input warps having bad PSF. just setting to quality 42, probably should be 13006 (PPSTACK_ERR_DATA)

  stacktool -dbname gpc1 -updatesumskyfile -stack_id 2809682 -fault 5 -set_quality 42
  stacktool -updaterun -dbname gpc1 -stack_id 2809682  -set_state full
  

• 16:10 MEH: ippmonitor hasn't been updating because czarpoll crashed when ippdb01 went down. restarted on ippc11
• 17:40 MEH: ipp032 looks like it went down.. -- ganglia doesn't indicate any overload so likely is down.. -- not rebooting on powercycle..
• 18:30 MEH: stdsci restart for nightly science, dropping STS label for the night and will let LAP run as is easier on system with updates
• 23:00 MEH: more diffim fault 5 from marginal data and psLib error

  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.056 -diff_id 484115  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.083 -diff_id 484115  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.053 -diff_id 484119  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.036 -diff_id 484126  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.072 -diff_id 484139  -fault 0
  

Wednesday : 2013.10.09

mark is czar

• 07:00 MEH: registration stalled again, regpeek showed was stuck with (couple time XY62 last night and then few times this morning to get the data registered)

  o6574g0216o  XY62 0 check_burntool neb://ipp041.0/gpc1/20131009/o6574g0216o/o6574g0216o.ota62.fits	#??? regtool -updateprocessedimfile -exp_id 664070 -class_id XY62 -set_state pending_burntool -dbname gpc1
  

  • 72 remaining exposure now clearing
  • MD01 warp also fault 3 and clearing, making stacks
• 09:10 MEH: nightly through except two exposures wedged in warp stage, will deal with when actually get into office...

  o6574g0239o
  o6574g0253o
  

  • LAP warps only to build up stack pool before switching back to STS -- LAP update order appears scrambled, maybe after new RA chunk added, many are short 10-20 warps to finish and make stacks and need chips so this may take hour or so to start getting enough through

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD