| Version 43 (modified by , 15 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD
(Up to PS1 IPP Czar Logs)
Monday : 2011-07-25
- 09:07 Serge: A lot of red in publishing.
pubtool -revert -dbname gpc1 -client_id 5. Mops want to have ANW3, ANO1, and OSS asap. - 09:24 Serge:
chip.offso that ANO1 quad gets processed/reverted faster. OSS and ANW3 (but 2 exposures because of bad cam quality) all published. - 09:34 Serge:
warp.off. 8 exposures in ANO1 to go - 09:43 Serge:
pubtool -definerun -client_id 5 -label ThreePi.nightlyscience -dbname gpc1 - 09:47 Serge: All MOPS quads published. chip.on, warp.on
- 12:12 CZW: Restarted stack pantasks, as it did not appear to be properly queuing new jobs (and SAS had 900 stacks sitting idle).
- 15:52 Bills: added M31 label to stdscience and updated the input file. Did not turn on warp-stack diffs yet. We need to examine the exposures relative to the current template.
- 16:46 Mark: re-enabled MD10 for W-S diffs tonight.
Tuesday : 2011-07-26
- 09:14: Serge: Heather ran and fixed? the summitcopy problem
summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o5768g0297o/o5768g0297o43.fits --filename neb://ipp026.0/gpc1/20110726/o5768g0297o/o5768g0297o.ota43.fits --summit_id 364059 --exp_name o5768g0297o --inst gpc1 --telescope ps1 --class chip --class_id ota43 --bytes 49432320 --md5 424f69d8594ef727d7b78ce03d0a7a43 --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous
- 09:36: Bill queued sts diffs for July 3.
- 10:03: Bill/Serge killed some ppSub running for ages (ipp043/diff_id 147113267983.34 seconds, ippc28/diff_id 147235/204905.41 seconds, ipp038/diff_id 147554/74371.20 seconds)
- 12:05: Serge: chip.off so that mops have OSS asap
- 12:56: Serge: warp.off
- 13:12: Serge: chip.on, warp.on
- 13:20: Serge: ippdb03 is down
- 13:40: Serge: ipp026 is down
- 15:30: CZW: paused lap processing to track down and fix a bug that was preventing SAS2.12 gri runs from queuing stacks correctly. This is resolved now.
- 17:30: Serge: By Gene's request, system stop and shut down
- 17:55: Serge: all pantasks shutdown
- 18:00: Serge: all pantasks running
ipp@ipp004:/home/panstarrs/ipp>./check_system.sh start.server [...] ipp@ipp004:/home/panstarrs/ipp>./check_system.sh start pantasks server cleanup has been started (host: ippc07) pantasks server distribution has been started (host: ippc15) pantasks server pstamp has been started (host: ippc17) pantasks server publishing has been started (host: ippc08) pantasks server registration has been started (host: ipp052) pantasks server replication has been started (host: ippc19) pantasks server stack has been started (host: ippc05) pantasks server stdscience has been started (host: ippc16) pantasks server summitcopy has been started (host: ipp050) pantasks server update has been started (host: ippc13) ipp@ipp004:/home/panstarrs/ipp>./check_system.sh run pantasks server cleanup is running (host: ippc07) pantasks server distribution is running (host: ippc15) pantasks server pstamp is running (host: ippc17) pantasks server publishing is running (host: ippc08) pantasks server registration is running (host: ipp052) pantasks server replication is running (host: ippc19) pantasks server stack is running (host: ippc05) pantasks server stdscience is running (host: ippc16) pantasks server summitcopy is running (host: ipp050) pantasks server update is running (host: ippc13)
- 18:07: CZW: restarted replication pantasks with new input file that limits the number of hosts being used for shuffle. This does not seem to have had a significant impact on the shuffle speed.
- 18:20: Serge: fixed and restarted replication of ippdb01 onto ippdb03
- 18:20+epsilon Serge: ipp030 is down
Wednesday : 2011-07-27
- 13:36 Bill ipp030 is still down causing lots of faults. I turned magic, destreak and dist revert off to find out the extent of the errors
- 15:57 Bill turned the reverts back on since ipp030 has finished it's brain transplant.
Thursday : 2011-07-28
- 09:16 Serge: chip.off, warp.off
- 09:20 Serge: killed ppSub on ipp036 (diff 148435) and on ipp049 (148445) running for about 3 hours
- 09:21 Serge: pubtool -revert -label (OSS.nightlyscience, MD10.nightlyscience, MD08.nightlyscience)
- 09:22 Serge: warp.on, chip.on
- 09:55 Serge: trying to fix the 400+ failed published items
mysql> select pub_id, client_id, diffRun.state, diffRun.data_group, diffRun.dist_group from publishRun join publishDone using(pub_id) join diffRun on stage_id = diff_id where publishDone.fault > 0 and publishRun.state ='new' and diffRun.dist_group ='ThreePi'; +--------+-----------+-------+------------------+------------+ | pub_id | client_id | state | data_group | dist_group | +--------+-----------+-------+------------------+------------+ | 237134 | 5 | full | ThreePi.20110713 | ThreePi | -> publishRun.state set to drop (likely too old) | 242765 | 1 | full | ThreePi.20110727 | ThreePi | \ | 242871 | 1 | full | ThreePi.20110727 | ThreePi | | | 242873 | 1 | full | ThreePi.20110726 | ThreePi | | | 242882 | 1 | full | ThreePi.20110726 | ThreePi | | | 242928 | 1 | full | ThreePi.20110726 | ThreePi | | | 242929 | 1 | full | ThreePi.20110726 | ThreePi | | | 242930 | 1 | full | ThreePi.20110726 | ThreePi | | | 242945 | 1 | full | ThreePi.20110726 | ThreePi | |--> All fixed with pubtool -revert -pub_id <pub_id> | 242974 | 1 | full | ThreePi.20110726 | ThreePi | | | 242996 | 5 | full | ThreePi.20110728 | ThreePi | | | 243046 | 1 | full | ThreePi.20110728 | ThreePi | | | 243050 | 1 | full | ThreePi.20110728 | ThreePi | | | 243054 | 1 | full | ThreePi.20110728 | ThreePi | | | 243059 | 5 | full | ThreePi.20110728 | ThreePi | / +--------+-----------+-------+------------------+------------+ 15 rows in set (0.01 sec) - 10:45 Serge: Trying to fix failed warps (neb-mv missing log file + revert), e.g.:
neb-mv neb://ipp014.0/gpc1/ThreePi.nt/2011/07/27/o5769g0171o.368889/o5769g0171o.368889.wrp.228460.skycell.1578.028.log neb://trash/o5769g0171o.368889.wrp.228460.skycell.1578.028.log warptool -revertwarped -dbname gpc1 -warp_id 228460
- 11:10 Serge: Fixed failed cam (trace files had no nebulous instances but the problem was shown as if the fits was corrupted/inexistant in the log)
- 11:21 Bill fixed a zero sized raw file instance by copying a good copy on top of it: cp /data/ippb01.2/nebulous/3d/7d/1101912367.gpc1:20101008:o5477g0061o:o5477g0061o.ota75.fits /data/ipp025.0/nebulous/3d/7d/900677111.gpc1:20101008:o5477g0061o:o5477g0061o.ota75.fits
- 11:15 Bill fixed some corrupt mask files with runcamexp cam_id 240526 and 240529
- 11:50 Serge: Fixed all ThreePi.nightlyscience failed warps
- 12:13 Bill fixed a bug in magic_destreak_revert.pl which caused camera stage faults to got to failed_revert. cleared all destreak statefaults
- 12:25 Serge:
UPDATE publishRun JOIN publishDone using(pub_id) join diffRun on stage_id = diff_id SET publishRun.state='drop' WHERE publishRun.state='new' and diffRun.dist_group='ThreePi.offnight';to fix the 417 exposures which were queued to publishing by mistake - 13:40 Bill added module backgound.pro to stdscience to do the background restoration for the M31 images.
- 13:51 Bill queued STS diffs for July 4 - 9
Friday : 2011-07-29
Saturday : 2011-07-30
Sunday : 2011-07-31
Note:
See TracWiki
for help on using the wiki.
