PS1 IPP Czar Logs for the week 2012.01.02 - 2012.01.08

(Up to PS1 IPP Czar Logs)

Monday : 2011.01.02

Bill is czar today

• 09:00 Got about 350 exposures last night. Nightly science is finished 3Pi diffs and is working on the MD05 stacks. LAP is proceeding.
• 09:15 Restarted stdscience, pstamp, and update pantasks. The pcontrols were spinnning.
• 21:45 I've got a bad feeling about this.... ipp060 panic'd. Faults all over czartool. A really suprisingly high number
• 22:00 Stopped replication pantasks. Doing file stuff during periods like this seems like a bad plan.

Tuesday : 2012.01.03

• 14:30 Mark looking into and fixing some out-of-order warps cleaned in LAP causing it to stall. Many were overlapping so the out-of-order cleaned warps was holding up all final stacks. fixed with dirty one-liner to batch file for each LAP id until updated all necessary warps (Chris probably has better command)

  laptool -listrun -seq_id 8 -state run -dbname gpc1 -simple
  laptool -exposures -simple -dbname gpc1 -lap_id 2073
  laptool -exposures -simple -dbname gpc1 -lap_id 2073 | cut -d " " -f26,27 | grep cleaned | awk '{print "warptool -dbname gpc1 -setskyfiletoupdate -set_label LAP.ThreePi.20110809 -warp_id "$1}' > fixwarpupdate_lap2073.bat
  ...
  

• 15:00 looks like some chip updates having trouble too and looking into. looks like there was a conflict in postage stamp update/clean with LAP chips, for example chip_id=362588

  2234 RINGS.V3 skycell.2341 z.00000 run 2012-01-02T00:33:06.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 250043 362588 362592 F T T to_magic update 0 60 339462 full 0 1 321786 full 323487 update 9 75 -1 198849 cleaned 7 63
  
  select chip_id,state,label,data_state,fault from chipRun join chipProcessedImfile using (chip_id) where chip_id=362588;
  |  362588 | update | ps_ud_WEB.UP | cleaned    |     0 | 
  |  362588 | update | ps_ud_WEB.UP | full       |     0 | 
  

• 16:00 restarting update but leaving stopped until fixed LAP chips needing update. reset the chip_ids to goto_cleaned and the updated with -setimfiletoupdate. the chips updated okay as well as the remaining warps. 800+ final stacks running. update pantasks set back to run. overloading stack pantasks with wave4 for a couple hours, will remove before nightly science starts.

Wednesday : 2012.01.04

• 07:30 Mark: appears LAP is getting stuck again like before with at least warps being cleaned that it is expecting to be updated.
• 09:20 fixing during Ken's talk. adding extra wave4 to stack pantasks again.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD