PS1 IPP Czar Logs for the week 2015.12.28 - 2016.01.03

(Up to PS1 IPP Czar Logs)

Monday : 2015.12.28

• 06:28 Bill: pstamp server was idle, so set all ps_ud% chip, warp, and diff data to be cleaned. Restarted pstamp server pantasks since it was using cpu time for now apparent reason.
• 16:45 EAM: restarting pantasks to be ready for the night.
  • everything is up and running, ready for the night.

Tuesday : 2015.12.29

• 07:00 MEH: QUB processing still running, setting up MOPS diffims using the QUB data -- probably finish by ~1100
• 09:26 MEH: clearing some OSS/3PI fault 5 WSdiffs

  difftool -dbname gpc1  -updatediffskyfile -fault 0 -set_quality 42 -diff_id 1297397  -skycell_id skycell.2193.048
  difftool -dbname gpc1  -updatediffskyfile -fault 0 -set_quality 42 -diff_id 1297561 -skycell_id skycell.0756.084
  difftool -dbname gpc1  -updatediffskyfile -fault 0 -set_quality 42 -diff_id 1297564 -skycell_id skycell.0840.017
  difftool -dbname gpc1  -updatediffskyfile -fault 0 -set_quality 42 -diff_id 1297601 -skycell_id skycell.0838.020
  

• 10:16 MEH: looking at some common fault nodes -- ippc54 looks like /export/ippc54.1 isn't mounted and tmp dir not available for processing there
• 17:22 MEH: restarting all pantasks to clear changes from last night and auto-remove ippc54 from processing until time to get ippc54.1 issue resolved
• 19:51 MEH: many faults w/ error: mkdir /data/ipp024.0/nebulous: No such file or directory -- set neb-host repair

Wednesday : 2015.12.30

• 12:25 MEH: test boosting pstamp pantasks to increase the rate QUB stamps are provided from last night
• 13:34 MEH: data nodes on path to filling up, need to start looking at products to clean up soon -- may also need to redo product targeting to use the older nodes with space as well
• 14:23 MEH: doing restart of nightly pantasks to return to default configurations now that cleanup is finished

Thursday : 2015.12.31

• 23:33 EAM: stopping & restarting the pantasks

Friday : 2016.01.01

• 15:06 MEH: restarting summitcopy+registration to use manual test changes in ipphosts.mhpcc.config for raw data to go to the 40T nodes w/ space to help balance space usage since monitoring processing again tonight
• 19:48 MEH: noticing ipp046 higher load.. seems to have only 1 cpu from repair on 12/23 to get back online -- since this is a datanode and needs to be online more than needs to be processing, it needed to be removed (or at least reduced) in loaded use in pantasks (ippconfig/pantasks_hosts.input)...

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD