PS1 IPP Czar Logs for the week 2011.02.21 - 2011.02.27

(Up to PS1 IPP Czar Logs)

Monday : 2011.02.21

  • 11:20 EAM : the updated ppStack was not writing the header metadata for the unconvolved stack. I fixed this (trunk and ops tag) and re-launched the stacks for 2011.02.20 and 02.21 (labels like MD03.ns.redo).
  • 13:27 heather : added MD06.jtrp to stdsci processsing, set MD04.jtrp MD05.jtrp warps and chips to clean

Tuesday : 2011.02.22

  • 15:41 (heather) MD06.jtrp had been removed, dropped the bad exposures (with a type of ENGINEERING), readded the label back in. Left stdscience in the same state it was in when I got here (stopped).
  • 21:40 bill ran "perl ~bills/sts.raw/queueds 178223 190515 go" to queue a number of STS raw exposures for destreaking to the alternate path_base (I need to do some information transfer :) At the current time we can't use a survey task to queue the dist runs so I'll need to run ~bills/sts.raw/queeudist once in awhile.

Wednesday : 2011.02.23

Bill is czar today.

  • 10:22 (heather) I'm running my own stdsci (current trunk, on ippc14) for testing the new config changes.
  • 11:15 one chip got stuck in ppImage. Bill put it out of it's misery. See ticket 1459.
  • we also have 4 ThreePi? diff failures that Gene will look at later.
  • One magicRun got stuck. There was an entry in the magicToTree book that said it was in run state, but controller had no entry for it and there was nothing in magicTree or in the logs. magic.reset got it going.
pantasks: show.book magicToTree
magic_id                 126508
exp_id                   304795
camera                     GPC1
workdir         /data/ipp053.0/gpc1_destreak/ThreePi.nightlyscience
ra                     1.033026
decl                   0.138943
tess_id                RINGS.V0
priority                    400
dbname                     gpc1
pantaskState                RUN
npages: 1
  • set label STS.2010.raw to inactive. Since the pendingcomponent query for disttool is divided up by stage, this was interfering with nightly science distribution.
  • 13:01 stacktool -updaterun -set_state drop -set_note 'dropped due to ticket 1427' -stack_id 232072
  • 13:51 extended the retention time for ThreePi? diff images from 10 to 12 days.
  • 16:28 Fix bug in the prepare output function. Don't raise an error when delete fails for moved file. That is the whole point of the interface
  • 20:45 all of the postamp stamp requests are done. Set all of the ps_ud labels to goto_cleaned. ippdb00 is the only node red

Thursday : 2011.02.24

  • (10:20) restarted distribution with a new destreak.pro that converts from @HOST@ to the proper host.
  • (11:51) heather investigated some more md06.jtrp - looks like nebulous orphan/widows again
  • (11:51) heather added MD07.jtrp to stdscience, and magictest.3Pi.20110222 to ipp distribution
  • (11:51) heather is still running her own stdsci for magictesting (testing of new ippconfig recipes).
  • (12:04) queued more sts raw data for destreak and distribution.
  • (12:55) ipp005 became unresponsive after it and ippdb00 jumped to high load. Power cycled it at 1300
  • (14:00) fixed a couple of problems with magic_destreak_revert.pl with runs that aren't replace mode. (Then fixed it one more time)
  • (14:28) heather reverted the md%jtrps.

Friday : 2011.02.25

Serge is czar today (but no data)

  • 09:25: pstamp replication supervision is now installed in czartool (SVN 30760). For information, on the replication slave (ippc19), in mysql as root, I did:
CREATE USER 'ippMonitor'@'ipp004.ifa.hawaii.edu' IDENTIFIED BY 'ippMonitor';
GRANT REPLICATION CLIENT ON *.* TO 'ippMonitor'@'ipp004.ifa.hawaii.edu';
FLUSH PRIVILEGES;
  • (12:27) bills fixed the reference catalog at the pole and started up processing of the CNP data. Queued new runs for 2010-10-13 = 2010-11-01 which were either taken before the entry for the CNP survey was added, or were purged because they failed.
  • (14:34) replication is getting crazy. I stopped and restarted it.
  • (19:30) bill queued ThreePi?.136 for 2011-01-21 and 2011-01-22 exposures for reprocessing. 2011-01-06 deferred until I get confirmation from Ken that they should be included.

Saturday : 2011.02.26

  • (04:00) all but one chip ThreePi?.136 finished. Queued data from 2011-01-23 (over 200 exposures). Killed 3 CNP chips that seemed to be stuck in psphot. ! threepi.136 may be in this way as well
  • set CNP label to inactive. turned off chip.revert for now.
  • 8:15 moved warp from stdscience to distribution
  • 11:30 The experiment worked well warp is now keeping up.
  • 11:48 turned off publish.revert since there are > 40000 faults.
  • 20:10 lots of faults in the last couple of hours but it's Sunday I'm no longer looking. Good night

Sunday : 2011.02.27