PS1 IPP Czar Logs for the week 2012.03.26 - 2012.04.01

(Up to PS1 IPP Czar Logs)

Monday : 2012.03.26

Tuesday : 2012.03.27

  • 23:00 CZW: Restarting stdscience because we have a backlog of exposures at the chip stage, and pantasks only has 35 chipImfile jobs running.

Wednesday : 2012.03.28

Bill is czar today

  • 13:00 stopped stdscience and deepstack to prepare for ipp041 going offline
  • 13:46 Shut down ipp041 for mother board swap.
  • 16:00 ipp041 back up with original mother board. There is a problem with the power connectors on the new one
  • 17:00 restart stdscience

Thursday : 2012.03.29

Bill is czar today

  • 07:00 distribution is proceeding slowly. pcontrols on ippc15 spinning. Restarted distribution and publishing pantasks.
  • 07:10 Load and network usage are periodically going low. Not sure what's happening. Noticed many glockfile processes hanging around. Several pspshotStack processes are swapping on the compute3 notes. summitcopy and registration are not finished yet
  • 07:15 set deepstack pantasks to stop 36 processes running.
  • 11:45 both staticsky and stdscience are working on the center of the galaxy. This means a lot of sources and thus a lot of memory used. ipp009 is backed up with 6 pswarp processes trying to write big cmf files (in an inefficient manner) and are swapping against them and czarpoll.
  • Afternoon: restarted stdscience, pstamp, and update.
  • Deepstack pantasks is shut off. The pending runs are all near galactic center and grow too big. I will start it up later after I queue new runs in a safer area.

Friday : 2012.03.30

Mark is czar

  • 08:00 looks like nightly science is downloaded and finished processing
  • 12:30 deepstack has been restarted and is processing staticsky runs for projection cells with abs(galactic latitude) > 10
  • 14:37 Bill has started a pantasks running out of ~bills/destreak which is restoring uncensored camera stage files.
  • 18:04 CZW I've launched a side pantasks to run the SAS reduction. This is using wave2-4, compute, and compute 3 nodes, each with a single instance. The pantasks server is running on ippc25, and the directory with the ptolemy.rc file is /data/ippc18.0/home/watersc1/pantasks_debugging/sideband_science.

Saturday : 2012.03.31

Sunday : 2012.04.01