== PS1 IPP Czar Logs for the week 2012.01.02 - 2012.01.08 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.01.02 ===
Bill is czar today
 * 09:00 Got about 350 exposures last night. Nightly science is finished 3Pi diffs and is working on the MD05 stacks. LAP is proceeding.
 * 09:15 Restarted stdscience, pstamp, and update pantasks. The pcontrols were spinnning.
 * 21:45 I've got a bad feeling about this.... ipp060 panic'd. Faults all over czartool. A really suprisingly high number
 * 22:00 Stopped replication pantasks. Doing file stuff during periods like this seems like a bad plan.
  
=== Tuesday : 2012.01.03 ===
 * 14:30 Mark looking into and fixing some out-of-order warps cleaned in LAP causing it to stall. Many were overlapping so the out-of-order cleaned warps was holding up all final stacks. fixed with dirty one-liner to batch file for each LAP id until updated all necessary warps (Chris probably has better command)
{{{
laptool -listrun -seq_id 8 -state run -dbname gpc1 -simple
laptool -exposures -simple -dbname gpc1 -lap_id 2073
laptool -exposures -simple -dbname gpc1 -lap_id 2073 | cut -d " " -f26,27 | grep cleaned | awk '{print "warptool -dbname gpc1 -setskyfiletoupdate -set_label LAP.ThreePi.20110809 -warp_id "$1}' > fixwarpupdate_lap2073.bat
...
}}}
 * 15:00 looks like some chip updates having trouble too and looking into. looks like there was a conflict in postage stamp update/clean with LAP chips, for example chip_id=362588
{{{
2234 RINGS.V3 skycell.2341 z.00000 run 2012-01-02T00:33:06.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 250043 362588 362592 F T T to_magic update 0 60 339462 full 0 1 321786 full 323487 update 9 75 -1 198849 cleaned 7 63

select chip_id,state,label,data_state,fault from chipRun join chipProcessedImfile using (chip_id) where chip_id=362588;
|  362588 | update | ps_ud_WEB.UP | cleaned    |     0 | 
|  362588 | update | ps_ud_WEB.UP | full       |     0 | 
}}}
 * 16:00 restarting update but leaving stopped until fixed LAP chips needing update. reset the chip_ids to goto_cleaned and the updated with -setimfiletoupdate. the chips updated okay as well as the remaining warps. 800+ final stacks running. update pantasks set back to run. overloading stack pantasks with wave4 for a couple hours, will remove before nightly science starts.
=== Wednesday : 2012.01.04 ===
 * 07:30 Mark: appears LAP is getting stuck again like before with at least warps being cleaned that it is expecting to be updated. 
 * 09:20 fixing during Ken's talk. adding extra wave4 to stack pantasks again (seems to increase the number of faults some however).
 * 09:50 again LAP running against pstamp update/cleans
{{{
|  362027 | update | ps_ud_WEB.UP | cleaned    |     0 | 
|  362027 | update | ps_ud_WEB.UP | full       |     0 | 
}}}
 * 10:30 trouble chips
{{{
| chip_id | state | label                | data_state | fault | quality | class_id |
|  322711 | update | LAP.ThreePi.20110809 | update     |     2 |       0 | XY26     | 
--> extn in chip, mv to .bad and cp'd good instance

|  322715 | update | LAP.ThreePi.20110809 | update     |     0 |      42 | XY26     | 
--> also extn problem in chip, mv'd .bad and cp'd good instance. however, couldn't get quality back to 0, just wanted to change fault 
chiptool -updateprocessedimfile -set_quality 0 -chip_id 322715 -class_id XY26 -dbname gpc1 -fault 0
--> manually set to full
}}}
 * 11:00 seem to have ended up with extra warps to update that aren't needed by LAP... waiting until LAP stalls again to cleanup 
=== Thursday : YYYY.MM.DD ===

 * 10:00 CZW: Launched SweetSpot WSdiffs manually.  This seems to be fairly easy to detect automatically, so this should probably be automated within the next week or two.  The command used was:
{{{
 difftool -definewarpstack -good_frac 0.2 -warp_label MSS.nightlyscience -stack_label MSS.nightlyscience -set_label MSS.nightlyscience -set_workdir neb://@HOST@.0/gpc1/MSS.nightlyscience/2012/01/05/ -available -set_reduction WARPSTACK -set_dist_group SweetSpot -dbname gpc1
}}}
 * 10:01 CZW: Kicked LAP.  Three runs were waiting on four chipRuns to be updated, and although the run was marked for update, none of the individual chipImfiles were.  I re-updated these chipRuns using my script (~watersc1/bin/run_update_for_chip_id.pl) and these now seem to be running (which should in turn get LAP processing to complete for these lapRuns).
 * 10:16 CZW: We had 116 warpRuns set to update that weren't needed.  I've sent these all to cleanup.
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===