PS1 IPP Czar Logs for the week 2011.09.19 - 2011.19.25

(Up to PS1 IPP Czar Logs)

Monday : 2011.09.19

• 08:00: Roy: everything is down from the summit.
• 08:30: Roy: ipp033 is back, so ran:

neb-host ipp033 up

• 09:15: Bill: Postage stamp server is caught up. Set all data with labels ps_ud% to be cleaned.
• 09:20: Bill: cleared all failed_revert faults.
• 10:10: Bill: set lap priority to 410 to allow it to finish it's destreak.
• 11:00: Bill: set lap priority back to 200
• 12:10 Chris/Mark: stdscience going slow. pcontrol large CPU use, stopping panstasks for restart.
• 12:29: Roy: shutdown stdscience, and restarted it
• 15:28: Bill fixed bug in ppMops that was causing some faults.
• 16:06: Bill fixed bug in his STS queueing script. Ran it to queue diffs for Sept 5 which were blocked by the bug
• 18:00: Mark removed labels from stdscience not being used tonight, MD03,04,05,06,07,08, PI, STD.nightlyscience, to test processing hiccups.
• 20:10 few frames of MD08 were observed, adding it back in.

Tuesday : 2011.09.20

• 08:35: Roy: registration stuck. regpeek revealed:

o5824g0388o  XY11 0 check_burntool neb://ipp008.0/gpc1/20110920/o5824g0388o/o5824g0388o.ota11.fits

Got exp_id using

SELECT DISTINCT exp_id FROM rawImfile WHERE exp_name = 'o5824g0388o';
+--------+
| exp_id |
+--------+
| 395112 | 
+--------+

Then ran

regtool -updateprocessedimfile -exp_id 395112 -class_id XY11 -set_state pending_burntool -dbname gpc1

• 08:40: Roy: Same as above, this time for exposure o5824g0398o
• 08:42: Roy: And again, exposure o5824g0399o
• 8:48 heather redid one of the failed burntool commands in the pantasks log.
• 9:00 CZW: restarted distribution server as it had the 100% CPU pantasks, and was processing slowly.
• 09:45 Bill: update server was down. Restarted it
• 10:19 Started magic_cleanup server in order to recover disk space on ipp053 ~bills/magic_cleanup.
• 14:47 CZW: Stopped stdscience to restart it.

Wednesday : 2011-09-21

• 1:20 CZW: Stopped distribution to restart it.
• 18:20 CZW: Preparation for stare night:
  • Stopped stdscience, distribution, cleanup, stack pantasks to prevent any processing
  • Disabled stare node processing using script (~ipp/stare_nodes.sh off)
  • Disabled future stare node processing by editing ~ipp/ippconfig/pantasks_hosts.input
  • Disabled nebulous allocation to all but the most full hosts using script: ~ipp/stare/pre_stare_night

Thursday : 2011-09-22

• 10:25 Stopped pantasks. Set nebulous state back to normal. Began restarting the pantasks.
• 10:36 Fixed warp with missing psf file runwarpskycell.pl --warp_id 260390 --skycell_id skycell.0783.030. Reverted the stack that wanted it.
• 10:50 Oops summit copy is only about 1/2 done with last night's data. Set nebulous back to pre_stare_night. Also rerunning the warp didn't take. Strange process, cleanup, update yields bad quality problem.

• 18:15 CZW: Updated currently pending_burntool entries to avoid a large amount of useless burntool processing:

   select 'regtool -updateprocessedimfile -exp_id ',exp_id,'-class_id',class_id,'-set_state full -burntool_state -14' from rawImfile where dateobs > '2011-09-22T07:34:03.000000' AND dateobs < '2011-09-22T16:00:22.000000' AND data_state = 'pending_burntool';
  

• 19:00 heather restarted summitcopy (so it had 30 hosts instead of 60). Earlier in the day someone had set it to 160 (!) connections to camera's datastore, which caused denial of service. More than 30 connections causes performance issues for camera while it's taking data. Don't do that.
• 22:30 CZW: ran post_stare_night script to re-enable all the hosts in nebulous. Restarted stdscience and distribution pantasks. Manually ran cleanup for 2011-09-22 and 2011-09-23 to ensure that we've cleaned everything that we should.

Friday : 2011-09-23

• 09:15 Serge: restarted publishing
• 09:20 Serge: no exposure has been registered (they seem to have been copied though). Shutdown and restart registration

Saturday : 2011-09-24

Sunday : 2011-09-25