PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Non-standard Processing

In an effort to improve communication, so we know what's going on better, HAF thinks we should list in the czar pages additional (non-standard) processing - so that we all know what's going on (no requirements for others to help run it). HAF will start:

HAF:

• ipp054 - ipp081: addstar processing for full force / diff. If you notice problems and need to reboot / stop this, please do the following:
  • contact HAF via email or phone (all should have my #)
  • if you feel addstar needs to be stopped and can't reach her:

           ssh ippdvo2@ippc19
           cd addstar.ipp054 (replace with machine that is problematic)
           pantasks_client
           stop
    

    • and then check status: wait for addstar run to finish, wait for minidvodb.premerge to finish (otherwise, risk corruption of mini)
• ~stsci03 is currently running ipptopsps for SAS39 FW. This is fairly resilient - no need to stop it if problem. you can access it via:~ no longer running

           ssh ippdvo2@ippc19
           screen -r fw
  

  • you can ctrl-c to stop it if necessary (but ipptopsps really doesn't care, it's fine.. it gracefully dies off if gpc1/nebulous die off, for example)

---

Monday : 2015.07.27

• HAF restarted summitcopy - shoved aside the summitExp faults (I think -- these are darks, and checkexp and others look "happy" now):

  delete from summitExp where fault = 110 and summit_id > 945000 and imfiles is null;;
  
  update summitExp set fault = 0 where fault = 110 and summit_id > 945000 ;
  

Tuesday : 2015.07.28

• 14:00 HAF : daily restart of pantasks

Wednesday : 2015.07.29

• 19:35 EAM : daily restart of pantasks

Thursday : 2015.07.30

• 04:40 Bill : reduced priority of ps_ud_MOPS label below nightly science labels since there seem to be a lot of requests pending
• 04:45 gpc1 progress is quite slow. gpc2 is using the cycles again.
• 8:26 HAF: registration is borked, download is slow.. I kicked registration

      16	8:25	regtool -updateprocessedimfile -exp_id 950693 -class_id XY54 -set_state pending_burntool -dbname gpc1
      17	8:26	regtool -updateprocessedimfile -exp_id 950693 -class_id XY15 -set_state pending_burntool -dbname gpc1
  

• 10:12 HAF: kicked registration - this is self preservation to make sure my stuff is working.. I am not the czar

   regtool -updateprocessedimfile -exp_id 950770 -class_id XY72 -set_state pending_burntool -dbname gpc1
  
  

• 14:45 CZW: started 10x check script on stare04 (to keep it out of the way) that is scanning lists of raw image prototypes and checking that we have two copies on the cluster. If not, it replicates copies to fix this. This is a pre-emptive scan to ensure that after shipping stsci nodes, we have two copies of the raw data.

Friday : 2015.07.31

• 00:00 MEH: only 4 data nodes for nightly processing and one is nearly full.. removing WS diffs untill morning as the use a lot of space
  • Ken has said gpc1 has priority over gpc2 and gpc1 >100 behind, removing gpc2 from summitcopy and stdsci until morning
  • there are no notes for ipp073 being in repair.. rsync isn't running on ipp074 it seems and why it was put into repair, as was ipp079 per email from Bill days ago.. unclear if rsyncs are still going to use these or not and why not put back up. past email noted ippb06 will be target for rsync but it really needs to be logged on the czar pages
  • ipp082 needs its BBU.. that also seems to have been ignored. it may be a better target for rsyncs

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD