PS1 IPP Czar Logs for the week 2015.04.13 - 2015.04.19

(Up to PS1 IPP Czar Logs)

Monday : 2015.04.13

• 05:20 Bill: Richard reports that the results from two exposures have not been received by MOPS. The problem was one of the warps had a fault 3 which doesn't get reverted. It was a database connection problem. Seems like we should change the fault code from 3 (config error) to 2 (data error) so it would be reverted.
• 13:15 CZW: Restarting stdlanl pantasks and removing the auto-queue queue file entry from the input file to cull down the number of active lapRuns. The cleanup was timing out due to the large number of input exposures, and that launched 37 runs that should not be active yet.
• 22:00 EAM: restarted pv3diff & pv3diffleft

Tuesday : 2015.04.14

• 06:15 EAM : stsci10 is down, rebooting it now.
• 10:20 EAM : started pv3skycal under ~ippsky/pv3skycal (1 x storage nodes for now).
• 10:30 EAM : restarting pv3diffleft.

Wednesday : 2015.04.15

• 03:58 Bill: MOPS reports two exposures stalled. Each had a skycell with faulted fault 4. One succeeded after reverting. The other was cleared with warptool -updateskyfile -warp_id 1531927 -skycell_id skycell.0938.019 -fault 0 -set_quality 42
• 07:38 Bill: restarted pstamp pantasks
• 21:40 MEH: stdsci needs its regular restart

Thursday : 2015.04.16

• 03:45 MEH: cleared OSS diffim fault

  Error in subtraction:
   -> VectorFitPolynomial1DOrd (psMinimizePolyFit.c:633): unknown psLib error
       Could not solve linear equations.
  
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0764.041 -diff_id  1013770 -fault 0
  
  

• 09:55 MEH: doing regular restart of all pantasks, maybe will clear up problem summitcopy is having
• 10:25 EAM: queued 06h for skycal

Friday : 2015.04.17

Saturday : 2015.04.18

Sunday : 2015.04.19