PS1 IPP Czar Logs for the week 2011.02.06 - 2011.02.10

(Up to PS1 IPP Czar Logs)

Monday : 2011.02.06

Roy is czar

• 09:00: only 70 science images, all through the pipeline
• 12:45 Serge: replication/shuffling seemed to be stuck. Following Chris advice, I shut it down and restarted it.
• 13:30: Roy: ran Bill's script to repair 'lost' instance of Exp: 193145, OTA: 15

repair_bad_instance -e 193145 -c XY15 -l

• 13:40 fixing bad burn.tbl as example for Roy

neb-stat --validate neb://ipp049.0/gpc1/20100629/o5376g0254o/o5376g0254o.ota54.burn.tbl
ipp_apply_burntool_single.pl --class_id XY54 --exp_id 188719 --dbname gpc1 --this_uri neb://ipp049.0/gpc1/20100629/o5376g0254o/o5376g0254o.ota54.fits --previous_uri neb://ipp049.0/gpc1/20100629/o5376g0253o/o5376g0253o.ota54.fits
neb-repair neb://ipp049.0/gpc1/20100629/o5376g0254o/o5376g0254o.ota54.burn.tbl

• 13:50: Roy: Trying one on my own:

neb-stat -validate neb://ipp051.0/gpc1/20100719/o5396g0026o/o5396g0026o.ota64.burn.tbl
ipp_apply_burntool_single.pl --class_id XY64 --exp_id 193152 --dbname gpc1 --this_uri neb://ipp051.0/gpc1/20100719/o5396g0026o/o5396g0026o.ota64.fits --previous_uri neb://ipp051.0/gpc1/20100719/o5396g0025o/o5396g0025o.ota64.fits
neb-repair neb://ipp051.0/gpc1/20100719/o5396g0026o/o5396g0026o.ota64.burn.tbl

• same for:

--class_id XY65 --exp_id 193166
--class_id XY51 --exp_id 193145

• and finally:

repair_bad_instance -e 193162 -c XY26 -r

Tuesday : 2011.02.07

Roy is czar

• 08:00: Roy: No data last night
• 08:00: Roy: LAP stuck again. One chip with a single corrupted copy and two with no copies at all, so:

repair_bad_instance -e 183688 -c XY26 -l
repair_bad_instance -e 183688 -c XY32 -l
repair_bad_instance -e 183664 -c XY65 -l

• 09:42 postage stamp server and data store shut down for database recovery.
• 10:00: Roy: stuck stacks, so:

stacktool -revertsumskyfile -fault 2 -label LAP.ThreePi.20110809 -dbname gpc1

• 11:18 Bill A couple of distribution fileset faults were lingering probably due to ippc17 failures. The failure was dsreg "fileset already exists" deleted them and the registrations completed.
• 11:40: Roy: more stuck stacks:

stacktool -revertsumskyfile -fault 2 -label LAP.ThreePi.20110809 -dbname gpc1

• 12:10 Earlier Bill queued all of the ps_ud% labels for cleanup. Cleanup was proceeding slowly. Restarted cleanup pantasks. Much better
• 14:55: Roy: Another lost chip:

repair_bad_instance -e 179899 -c XY76 -l

• 15:19 Bill queued some M31 data in g z and y filters for processing. Some of these exposures are old so there may be some problems.
• 15:33 As predicted some of the data has old burntool_state. Set those run's labels to M31.gzy.old.burntool
• 16:15: Roy:

repair_bad_instance -e 191531 -c XY76 -r

• 16:20: Roy: a 'bad' burn.tbl:

neb-stat -validate neb://ipp042.0/gpc1/20100614/o5361g0284o/o5361g0284o.ota23.burn.tbl
ipp_apply_burntool_single.pl --class_id XY23 --exp_id 181037 --dbname gpc1 --this_uri neb://ipp042.0/gpc1/20100614/o5361g0284o/o5361g0284o.ota23.fits --previous_uri neb://ipp042.0/gpc1/20100614/o5361g0283o/o5361g0283o.ota23.fits
neb-repair neb://ipp042.0/gpc1/20100614/o5361g0284o/o5361g0284o.ota23.burn.tbl

• 16:40: Roy: 3 more stuck stacks:

stacktool -revertsumskyfile -fault 2 -label LAP.ThreePi.20110809 -dbname gpc1

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD