| Version 27 (modified by , 16 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2010.12.06 - 2010.12.12
(Up to PS1 IPP Czar Logs)
Monday : 2010.12.06 : Roy
Not much data last night due to telescope azimuth amplifier errors. 123 science exposures in total, 99 through burntool.
bills notes: I forgot to mention that ipp007 has been relphot on /data/ipp007.0/gpc1/catdirs/ThreePi.V1.20101118 since last night. The working directory is ~ipp/dvo/relphot
11:00 bills: Mark Huber reported this morning that no data has been distributed since 11/30. rcDestination.dbhost was still set to ippdb02 so filsets were getting registered in the wrong database. At 08:08 Fixed this by faulting the affected filesets and letting revert / rerun recreate them. Later Armin reported that some were still missing from 11/30. Bill's date cut on the fix in HST while the time stamp is in UTC. The correct fix was to fault and then revert all filesets registered between '2010-11-30 13:33:58' and '2010-12-06'
11:30 bills: stackonly shut down so as not to interfere with detrend memory hogs.
11:38 heather: I started detrend as myself about ~ half an hour ago (with the current ipp trunk) to run another iteration on 585. I used whatever the default hosts were in detrend from months ago, and this caused load chaos. I stopped detrend and when the queue clears out I will chose different/less hosts.
13:13 roy: Turned destreak reverts on (63 faults). Czartool was lying about it already being on.
15:04 bills: turned destreak on and revert off. ran magictool -reverttree -label ThreePi.nightlyscience
15:28 eam; shutdown all pantasks, updated ipp user to ipp-20101206, restarted all pantasks
16:30 roy: all stdscience processing is complete
17:30 bills: Lots of unlogged activity the past few hours updating the production build to the new tag. Distribution shut off pending some distribution updates by bills.
17:49 bills: most all chip updates failing. update processing stopped.
18:57 bills: added label test_warpstack_forced to stdscience. As the label implies it is a test.
Tuesday : 2010.12.07
Heather is czar but this is all contributed by bills
05:27 restarted distribution pantasks. rcserver left turned off until bill tests changes to receive_file.pl
06:00 all distRuns for the stack-stack diffs made last night have some components that failed. From the file ownership and time stamps it appears that the script compressing the existing config dump files operated on these new ones creating files that are doubly compressed. They need two doses of gunzip. I.E. gunzip -c file.mdc | gunzip -c > outfile.mdc
06:30 stated up update pantasks running as bills using his ipp-20101029 build. Turned off MOPS.2 label to give full attention to the small number of requests that came in over the web interface.
07:30 many postage stamp requests completed, but now many are blocked because bills' build can't handle compressed config dump files.
08:30 Got fixes to ppImage. Rebuilt production build and restarted update as ipp user. Reverted chip faults from ps_ud_WEB and ps_ud_WEB.UP
09:59 1250 chips processed by update only 1 fault! 49 MD03.refstack skycells left to stack. Started them in stackonly pantasks on ipp042. hosts ipp040-44 are in use.
13:15 Needed to whack a few moles with regard to inconsistencies data_state and faults in the tables but we have had no non-revertable faults in chip and destreak update for several hours. Bill giddy with happiness.
Noticed that publishing was failing 100% of the time. Turned out to be a bug in publish_file.pl "if it isn't tested it doesn't work"
15:58 (heather) Heather is running detrend as heather on 038, 039, 046-053, c00-c10 (since 10am). After investigation minidvodb_create faults with bills, discovered that addstar has not been restarted since 11-23 (this is terrible, we rebuilt the ipp since then). Restarting addstar.
16:16 (bills) Queued 480 STS diffRuns label STS.20101202
16:46 (bills) stdscience diff.revert turned off
21:25 (bills) came back from reception to find that the update pantasks died around 17:42. Ken's team has submitted some postage stamp request files that resulted in nearly 7000 jobs. Nearly all of these will require update processing. Update processing is still going well. I need to do some work on destreak revert and cleanup to handle gone files.
Wednesday : 2010.12.08
9:57 (heather) no data from last night. I'm using the same hosts as I did yesterday to process ~30 images for the static mask (in a stdscience like pantasks running on c14).
