PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.04.10

  • 01:40 MEH: summitcopy stalled on o7853g0221o because of fault -- doesn't seem to clear automatically.. 91 exposures behind in registgration
    • revertcopied since 202 isn't a commonfault(?)
      downloading starting  Sun Apr  9 23:47
      *** stderr ***
      stderr 54498
      request failed: 502 Proxy Error at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/dsget line 155.
      Unable to perform dsget: 202 at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/summit_copy.
      
  • 15:25 MEH: setting some datanodes repair->up to ease space crunch -- ipp118-120,122;

Tuesday : 2017-04-11

  • 14:00 CZW: Started the reinsert commands for the retired nodes as ipptest in a screen session on ipp118. This seems to have caused a high load average on ipp122, but it doesn't seem to be sluggish in any way. These jobs will run on ipp118, ipp119, ipp120, and ipp122. If someone notices a problem, they should stop cleanly with just a control-C in the screen session.

Wednesday : 2017-04-12

  • 17:50 CZW: It doesn't look like processing was restarted after the ITC work. I will restart the ippitc pantasks. Cleanup probably didn't run, either, so I'll trigger that if necessary.

Thursday : 2017-04-13

  • 18:45 CZW: Gene has set ippdb01 as nebulous master due to the disk failures on ippdb06 and restarted the apache servers. The ippitc user configuration has been updated, and I'm going to restart the pantasks servers.
    • I will also be leaving the reinsert jobs off for the weekend. I can determine if we have all the files we need from the log files, so reinserting the files isn't essential to

Friday : 2017.04.14

Saturday : 2017.04.15

  • 05:30 EAM: one download failed and was not cleared -- i reverted it and it is moving along again:
    pztool -revertcopied -exp_name o7858g0304o -fault 202 -dbname gpc1 -telescope ps1 -inst gpc1
    

Sunday : YYYY.MM.DD