| Version 17 (modified by , 16 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2010.12.06 - 2010.12.12
(Up to PS1 IPP Czar Logs)
Monday : 2010.12.06 : Roy
Not much data last night due to telescope azimuth amplifier errors. 123 science exposures in total, 99 through burntool.
bills notes: I forgot to mention that ipp007 has been relphot on /data/ipp007.0/gpc1/catdirs/ThreePi.V1.20101118 since last night. The working directory is ~ipp/dvo/relphot
11:00 bills: Mark Huber reported this morning that no data has been distributed since 11/30. rcDestination.dbhost was still set to ippdb02 so filsets were getting registered in the wrong database. At 08:08 Fixed this by faulting the affected filesets and letting revert / rerun recreate them. Later Armin reported that some were still missing from 11/30. Bill's date cut on the fix in HST while the time stamp is in UTC. The correct fix was to fault and then revert all filesets registered between '2010-11-30 13:33:58' and '2010-12-06'
11:30 bills: stackonly shut down so as not to interfere with detrend memory hogs.
11:38 heather: I started detrend as myself about ~ half an hour ago (with the current ipp trunk) to run another iteration on 585. I used whatever the default hosts were in detrend from months ago, and this caused load chaos. I stopped detrend and when the queue clears out I will chose different/less hosts.
13:13 roy: Turned destreak reverts on (63 faults). Czartool was lying about it already being on.
15:04 bills: turned destreak on and revert off. ran magictool -reverttree -label ThreePi.nightlyscience
15:28 eam; shutdown all pantasks, updated ipp user to ipp-20101206, restarted all pantasks
16:30 roy: all stdscience processing is complete
17:30 bills: Lots of unlogged activity the past few hours updating the production build to the new tag. Distribution shut off pending some distribution updates by bills.
17:49 bills: most all chip updates failing. update processing stopped.
18:57 bills: added label test_warpstack_forced to stdscience. As the label implies it is a test.
Tuesday : 2010.12.07
05:27 restarted distribution pantasks. rcserver left turned off until bill tests changes to receive_file.pl
06:00 all distRuns for the stack-stack diffs made last night have some components that failed. From the file ownership and time stamps it appears that the script compressing the existing config dump files operated on these new ones creating files that are doubly compressed. They need two doses of gunzip. I.E. gunzip -c file.mdc | gunzip -c > outfile.mdc
