PS1 IPP Czar Logs for the week 2019.01.07 - 2019.01.13

(Up to PS1 IPP Czar Logs)

Monday : 2019.01.07

• CCL: note from PS1_IPP_Czarlog_20181119, pstamp jobs stuck >7 days

  pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id 912774
  pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id 912781
  pstamptool -dbname ippRequestServer -dbserver ipp113 -updatereq  -set_state stop -req_id 912828
  

Tuesday : 2019.01.08

Wednesday : 2019.01.09

Thursday : 2019.01.10

• MEH: removed pstamp from auto-restart to prevent stamp requests still parsing from being corrupted, will restart pstamp once the stamp requests have finished

Friday : 2019.01.11

• MEH: Serge is testing out running some jobs on the IPP nodes (ipps, ippx, ippx35-63) when nightly processing has finished
  • nebulous db (ippdb09) not responding well, adjustments being made

Saturday : 2019.01.12

• MEH: ippc70 crashed last night -- power cycle not bringing back online, power off until can be looked at -- comment out in ippconfig/nebservers.csh (in ~ippitc, ~ippps2, ~ipp) and change NEB_HOST to ippc71.. restart pantasks in fresh login just to be sure
• MEH: nagios should have alerted us, Gavin found emails being held moderator since 12/10 -- large backlog of email now available
• MEH: ippc70 crash wedged registration --

  select * from rawImfile where exp_id>=1440841 and data_state not like "full";
  
  | exp_id  | class_id | data_state       |
  +---------+----------+------------------+
  | 1440841 | XY45     | check_burntool   | 
  | 1440842 | XY45     | pending_burntool | 
  ....
  
  regtool -updateprocessedimfile -exp_id 1440841 -class_id XY45 -set_state pending_burntool -dbname gpc1
  
  

Sunday : 2019.01.13