PS1 IPP Czar Logs for the week 2012.04.16 - 2012.04.22

(Up to PS1 IPP Czar Logs)

Monday : 2012.04.16

Bill is czar today

  • 07:45 Nightly science has finished chip processing. Everything looks fine.

Tuesday : 2012.04.17

  • 10:45 deep stack shut down. There is nothing for it to do right now. We have processed all of the lap staticsky runs at galactic latitude < 20 that find less than 40000 peaks
  • 21:50 Mark: with the predicted poor weather, queued up MD08.GR0 chip->warp to run normally in pantasks. ~475 exposures

Wednesday : 2012.04.18

  • 08:20 Mark: MD08.GR0-g finished, 3 lost OTA out of 475 exposures.
  • 10:25 Mark: MD08.GR0-r queued up for chip->warp processing since tonight looking better, ~500 exposures, and keep a daily processing group by filter for MD08.
  • 14:00 Mark: stdscience struggling (could even see decay in rate from last night). played around with stopping and resetting books, not real help. Chris listed books and look not be full of finished runs. so restarting.
  • 16:00 Mark: adding stacks back to deepstack and restarting. there is a note in the input file about not having stacking running with the LAP label, so commented the add.label LAP.ThreePi?.20110809. keeping staticsky module loaded since want to run MD08 staticsky.

Thursday : 2012.04.19

  • The ippRequest server database has some crashed tables. pstamp, distribution/rcserver, publishing, and apache on ippc17 have been shut down while we try to repair the tables.
  • repair table pstampJob fixed the problem
  • 13:20 Mark: MD08.GR0 continues with next filter, chips for previous filter fixed/ignored and cleaned. nightly stacks to start running after get rate baseline for chip->warp.
  • 16:30 Mark: compute3 had been removed from stdscience and stack (and put into deepstack for staticsky), putting back into stdscience+stack for now until deepstack fully active again.
  • 23:10 Mark: pileup of nightlyscience in distribution stage, setting for a bit to push out

Friday : 2012.04.20

  • 00:05 Mark: distribution cleared, putting destreak.revert.on again
  • 09:40 distribution backed up again, flipping destreak.revert.
  • 11:30 distribution of time sensitive SSdiffs has been finished, 3PI and STS still going
  • 12:30 GR0 nightly stack reprocessing has achieved the camel mode (wave mode?) see ganglia, sync'd start/finish.
  • 14:00 distribution finished, flipping destreak.revert back on

Saturday : 2012.04.21

  • 11:20 Mark: massive pileup in distribution, flipping
  • 11:25 ~100 still to register still? looking into. had to run
    regtool -updateprocessedimfile -exp_id 479086 -class_id XY11 -set_state pending_burntool -dbname gpc1
  • 16:00 distribution taking forever, going to restart it along with registration and stdscience for the semi-daily restart flush
  • 18:00 distribution of last nights data finished, flipping destreak.revert.on

Sunday : 2012.04.22