PS1 IPP Czar Logs for the week 2012.01.02 - 2012.01.08

(Up to PS1 IPP Czar Logs)

Monday : 2011.01.02

Bill is czar today

• 09:00 Got about 350 exposures last night. Nightly science is finished 3Pi diffs and is working on the MD05 stacks. LAP is proceeding.
• 09:15 Restarted stdscience, pstamp, and update pantasks. The pcontrols were spinnning.
• 21:45 I've got a bad feeling about this.... ipp060 panic'd. Faults all over czartool. A really suprisingly high number
• 22:00 Stopped replication pantasks. Doing file stuff during periods like this seems like a bad plan.

Tuesday : 2012.01.03

• 14:30 Mark looking into and fixing some out of order warps cleaned in LAP causing it to stall. Many were overlapping so the out of order cleaned warps was holding up all final stacks. fixed with dirty one-liner to batch file for each LAP id until updated all necessary warps (Chris probably has better command)

  laptool -listrun -seq_id 8 -state run -dbname gpc1 -simple
  laptool -exposures -simple -dbname gpc1 -lap_id 2073
  laptool -exposures -simple -dbname gpc1 -lap_id 2073 | cut -d " " -f26,27 | grep cleaned | awk '{print "warptool -dbname gpc1 -setskyfiletoupdate -set_label LAP.ThreePi.20110809 -warp_id "$1}' > fixwarpupdate_lap2073.bat
  ...
  

• 15:00 looks like some chip updates having trouble too, looking into. it looks like there was a conflict in postage stamp update/clean with LAP need of chip for example chip_id=362588

  2234 RINGS.V3 skycell.2341 z.00000 run 2012-01-02T00:33:06.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 250043 362588 362592 F T T to_magic update 0 60 339462 full 0 1 321786 full 323487 update 9 75 -1 198849 cleaned 7 63
  
  select chip_id,state,label,data_state,fault from chipRun join chipProcessedImfile using (chip_id) where chip_id=362588;
  |  362588 | update | ps_ud_WEB.UP | cleaned    |     0 | 
  |  362588 | update | ps_ud_WEB.UP | full       |     0 | 
  

• 16:00 restarting update but leaving stopped until fixed LAP needed chips for update. reset the chip_ids goto_cleaned and setimfiletoupdate. updated and remaining warps updated. 800+ final stacks running. update pantasks set back to run.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD