PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2018.06.18

• MEH: ipp105.0 red in nebulous, set neb-host repair until space freed up
• MEH: ipp113.0 pstamp server data location also filling up, set neb-host repair until more space freed up -- turned on cleanup for pstamp data on ipp113 through end of May and back off again in preparation for more large MOPS.2 requests, but just leave in neb-host repair

Tuesday : YYYY.MM.DD

Wednesday : 2018.06.20

• MEH: ippdb10 crashed/offline last night ~2000 HST -- nothing on console, no prompt -- power cycled and not rebooted, set power off
• MEH: ippc20 appears down in ganglia ~1600 HST, unclear what cab is in -- assume in mez but ippc20@… stalls trying to connect -- apparently Gene asked Haydn to try installing ubuntu, but wasn't clear notification this was done

Thursday : 2018.06.21

• MEH: ippdb10 crashed again, nothing on console -- Haydn fixed the power outlet so power cycle works now and back up
• MEH: long running diffs in ps2 -- memory ~28%

    0   ippx080    BUSY   5308.58 0.0.0.876c  0 diff_skycell.pl --threads @MAX_THREADS@ --diff_id 3345 --skycell_id skycell.0952.011 --diff_skyfile_id 184290 --outroot neb://ipp103.1/gpc2/OSS.nt/2018/06/21/RINGS.V3/skycell.0952.011/RINGS.V3.skycell.0952.011.dif.3345 --redirect-output --run-state new --inverse --reduction SWEETSPOT --dbname gpc2 --verbose 
    1   ippx073    BUSY   4580.04 0.0.0.90ed  0 diff_skycell.pl --threads @MAX_THREADS@ --diff_id 3350 --skycell_id skycell.0866.040 --diff_skyfile_id 184566 --outroot neb://ipp103.0/gpc2/OSS.nt/2018/06/21/RINGS.V3/skycell.0866.040/RINGS.V3.skycell.0866.040.dif.3350 --redirect-output --run-state new --inverse --reduction SWEETSPOT --dbname gpc2 --verbose 
    2   ippx046    BUSY   3780.44 0.0.0.980c  0 diff_skycell.pl --threads @MAX_THREADS@ --diff_id 3354 --skycell_id skycell.0952.011 --diff_skyfile_id 184804 --outroot neb://ipp103.1/gpc2/OSS.nt/2018/06/21/RINGS.V3/skycell.0952.011/RINGS.V3.skycell.0952.011.dif.3354 --redirect-output --run-state new --inverse --reduction SWEETSPOT --dbname gpc2 --verbose 
  
  
  difftool -dbname gpc2 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 3345 -skycell_id skycell.0952.011
  difftool -dbname gpc2 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 3350 -skycell_id skycell.0866.040
  difftool -dbname gpc2 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 3354 -skycell_id skycell.0952.011
  

Friday : 2018.06.22

• MEH: MOPS/Weryk processing setup -- adjusted crontab group for Weryk (and for other users allowed to use) on older data nodes and few compute nodes (had group stb-admin rather than crontab), fix /local/tmp permissions on most older compute nodes and the ippx nodes to be drwxrwxrwt as on the other nodes

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD