(Up to PS1 IPP Czar Logs)

Monday : 2012-08-13

  • 08:00 (Bill) LAP has been running all night but czartool thinks that only chip processing has occurred. Restarted stdscience. Lots of timeouts for chiptool -pendingmfile jobs in pantasks.stdout.log
  • 08:17 stdscience stopped to investigate LAP wierdness
  • 08:20 There are no cam runs because LAP is not doing new chip processing, it is updating previously cleaned runs. There is no warp processing because the original files are still on disk because we aren't currently cleaning up warps.
  • 11:50 With advice from Chris, modified lap_science.pl to not queue updates for exposures that have warps in full state. Set the currently updating chipRuns (which are not needed) to be cleaned.
  • 13:26 set two faulted diff skycells out of their misery
    difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 266709 -skycell_id skycell.053
    difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 266714 -skycell_id skycell.046
  • (approx) 15:00 (Bill) Chip XY66 is causing memory explosion for several exposures. Stopped stdscience and killed off all outstanding ppImage proceses. Unfortunately forgot to set it back to run after that was done.

Tuesday : 2012-08-14

  • 09:00 restarted stdscience which has been left stopped all night.
  • 11:00 Bill All queued LAP stacks are done. Set lap label to inactive to avoid memory problems until nightly science is completed.
  • 13:30 All pantasks shut down for rebuild with changes to psphot.
  • 13:45 pantasks restarted (except for deepstack which currently has nothing to do)
  • 17:20 Bill found the root cause of the ppImage memory explosion and integrated a fix into the tag.

Wednesday : 2012-08-15

  • 08:45 (Serge) Fixed LAP:
    • gpc1/20100128/o5224g0297o/o5224g0297o.ota32.burn.tbl ; gpc1/20100626/o5373g0017o/o5373g0017o.ota56.burn.tbl ;
    • gpc1/20100624/o5371g0139o/o5371g0139o.ota65.fits ; gpc1/20100624/o5371g0148o/o5371g0148o.ota26.fits ; gpc1/20100624/o5371g0147o/o5371g0147o.ota64.fits ; gpc1/20100624/o5371g0124o/o5371g0124o.ota12.fits ;
  • 09:10 Bill updated lap_science.pl to not exit with status equal to the number of stacks queued. That behavior isn't needed and is distracting since it looks like an error.

Thursday : 2012-08-16

  • 14:44 (Serge): LAP
    • Recovered gpc1/20100624/o5371g0104o/o5371g0104o.ota65.fits;
    • Fixed gpc1/20100530/o5346g0044o/o5346g0044o.ota14.burn.tbl
  • 14:00 (Bill) restarted stdscience
  • 15:25 (Bill) stopped then restarted stdscience to pick up a bug fix in psphotEfficiency.c
  • 15:57 (Bill) reverted 4 lap stacks that faulted with fault 2

Friday : 2012-08-17

Happy Statehood Day

  • 06:30 (Bill) Fixed a couple of missing burntool tables. nightly science is nearly complete 6 exposures at various stages in pipeline

Saturday : 2012-08-18

  • 07:30 (Bill) Fixed a couple of missing burntool tables. nightly science is nearly complete 2 expouses left
  • 07:33 stdscience is sluggish: pcontrol eating cpu, jobs in books but not in controller list, restarting pantasks.

Sunday : 2012-08-19

  • 08:00 - 09:00 (Bill) No data last night. 192 chip faults. Turned out that 180 of them were three engineering exposures posing as 3PI. Dropped them. The rest were bad burntool tables or missing raw files. All but one was repaired. exp 180185 XY64 has been lost forever. Stopping all pantasks in preparation for restart.
  • 09:45 Stopped processing briefly to rebuild psphot. A change that Bill made last week which added a recipe value broke warp updates because the value did not exist in the mdc file. Fixed that.
  • 11:00 warp is falling quite a ways behind chip processing. Turning off chip processing for an hour or so to see if warp catches up
  • 12:32 going out. Turning chip back on
  • 13:48 chip.off pending runs: chip: 766 warp: 939
  • 15:12 chip.on pending: chip: 816 warp: 716