PS1 IPP Czar Logs for the week 2011.09.19 - 2011.19.25

(Up to PS1 IPP Czar Logs)

Monday : 2011.09.19

• 08:00: Roy: everything is down from the summit.
• 08:30: Roy: ipp033 is back, so ran:

neb-host ipp033 up

• 09:15: Bill: Postage stamp server is caught up. Set all data with labels ps_ud% to be cleaned.
• 09:20: Bill: cleared all failed_revert faults.
• 10:10: Bill: set lap priority to 410 to allow it to finish it's destreak.
• 11:00: Bill: set lap priority back to 200
• 12:10 Chris/Mark: stdscience going slow. pcontrol large CPU use, stopping panstasks for restart.
• 12:29: Roy: shutdown stdscience, and restarted it
• 15:28: Bill fixed bug in ppMops that was causing some faults.
• 16:06: Bill fixed bug in his STS queueing script. Ran it to queue diffs for Sept 5 which were blocked by the bug
• 18:00: Mark removed labels from stdscience not being used tonight, MD03,04,05,06,07,08, PI, STD.nightlyscience, to test processing hiccups.
• 20:10 few frames of MD08 were observed, adding it back in.

Tuesday : 2011.09.20

• 08:35: Roy: registration stuck. regpeek revealed:

o5824g0388o  XY11 0 check_burntool neb://ipp008.0/gpc1/20110920/o5824g0388o/o5824g0388o.ota11.fits

Got exp_id using

SELECT DISTINCT exp_id FROM rawImfile WHERE exp_name = 'o5824g0388o';
+--------+
| exp_id |
+--------+
| 395112 | 
+--------+

Then ran

regtool -updateprocessedimfile -exp_id 395112 -class_id XY11 -set_state pending_burntool -dbname gpc1

• 08:40: Roy: Same as above, this time for exposure o5824g0398o
• 08:42: Roy: And again, exposure o5824g0399o
• 8:48 heather redid one of the failed burntool commands in the pantasks log.
• 9:00 CZW: restarted distribution server as it had the 100% CPU pantasks, and was processing slowly.
• 09:45 Bill: update server was down. Restarted it
• 10:19 Started magic_cleanup server in order to recover disk space on ipp053 ~bills/magic_cleanup.
• 14:47 CZW: Stopped stdscience to restart it.

Wednesday : 2011-09-21

• 1:20 CZW: Stopped distribution to restart it.
• 18:20 CZW: Preparation for stare night:
  • Stopped stdscience, distribution, cleanup, stack pantasks to prevent any processing
  • Disabled stare node processing using script (~ipp/stare_nodes.sh off)
  • Disabled future stare node processing by editing ~ipp/ippconfig/pantasks_hosts.input
  • Disabled nebulous allocation to all but the most full hosts using script: ~ipp/stare/pre_stare_night

Thursday : 2011-09-22

• 10:25 Stopped pantasks. Set nebulous state back to normal. Began restarting the pantasks.
• 10:36 Fixed warp with missing psf file runwarpskycell.pl --warp_id 260390 --skycell_id skycell.0783.030. Reverted the stack that wanted it.
• 10:50 Oops summit copy is only about 1/2 done with last night's data. Set nebulous back to pre_stare_night. Also rerunning the warp didn't take. Strange process, cleanup, update yields bad quality problem.

Friday : 2011-09-23

Saturday : 2011-09-24

Sunday : 2011-09-25