PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.04.18

  • MEH:
    • moving stdscience pantasks back to its original ippc02 node so not sharing ippc01 s/ summitcopy
    • testing survey.pro changes for making the SSdiff have a long needed target label arg..
    • testing summitcopy time stamps in log
    • turning off stack nodes to compare processing rate during next MD07 night
    • running ~ipp pantasks log bzip

Tuesday : 2016.04.19

  • 15:30 MEH: boosting pstamp to get through a large backlog of QUB stamps before nightly processing

Wednesday : 2016-04-20

  • 14:45 CZW: I'm disabling the wave 3 nodes that are still in use in the pantasks_hosts.input file, and will be adding 3x x2 to stdscience, and 1x x3 to all the other pantasks that had 1x s2.
  • 15:30 CZW: ipp/pantasks restarted with new pantasks_hosts.input file. All servers now have equal or slightly higher number of active nodes than before.

Thursday : YYYY.MM.DD

  • 13:30 CZW: Stopped pantasks for config SVN repair.
  • 13:30 CZW: Rebooting wave 3 machines.
  • 13:50 CZW: ipp033 rebooted but does not see the home directory. I am not rebooting any more machines at this time.
  • 15:00 CZW: Rebooting has resumed. Beginning safety rsyncs on ipp033, ipp035, and ipp036.
  • 23:50 MEH: pstamp seems to be hosed due to some change today -- ~ipp/ippconfig/site.config was modified to an incompatible version
    #default access level for users that do not have specific entries in the tables
    PSTAMP_DEFAULT_ACCESS_LEVEL S32 1
    --> required!
    
    PSTAMP_PRESERVE_DAYS STR 14 -> 30
    
    --> should be commented out while ipp017 is down? not sure what problems that would cause
              CATDIR.017     STR    /data/ipp017.0/ipp/ippRefs/catdir.refcat.20140713.v0
    

Friday : 2016.04.22

  • 01:55 MEH: confirmed rate drop during longer exposure CFA/MD07 observations and not from stack processing, will turn stack back on in morning to run normally
  • 17:02 HAF: restart of pantasks - notes that there are jammed jobs (dsget / regimfile) on ipp104 that jammed at exactly the same time as a bunch of rsyncs started..

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD