PS1 IPP Czar Logs for the week 2013.07.08 - 2013.07.14

(Up to PS1 IPP Czar Logs)

Monday : 2013.07.08

• 08:04 Bill stopped distribution. It is falling all over itself because the new stsci machines do not have /local/ipp set up correctly. (It was set to be a link to /export/ipp053.0 which is inaccessible except from ipp053)
• 08:23 Set up /local/ipp on stsci10 - 19 (rm /local/ipp ; mkdir -p /export/$hostname.0/ipp/tmp; ln -s /export/$hostname.0/ipp /local ; mkdir /export/$hostname.0/ipp/gpc1 ; rsync -av /data/ipp053.0/ipp/gpc1/tess /local/ipp/gpc1
• 08:38 Bill: more distribution woes. The wave 2 nodes have absolutely no space in their /export partitions. Thus their /local/ipp/tmp has no space, so distribution tasks cannot run on those nodes. Restarted distribution with wave2 and wave3 nodes turned off and some compute nodes added in.
• 10:09 Bill: set M31.test and STS.rerun data to be cleaned. We now have 8000 chipRuns in state goto_cleaned
• 11:00 EAM : set all relevant machines with > 98% full to 'repair'. started dvo purge-temp to remove unneeded temp files from ipptopsps ops.
• 11:01 EAM : ipp054 crashed : ipp054.20130708.crashlog
  • console server for ipp054 (and ipp055 at least) does not allow power control -- contacted Gavin (who contacted Rita) to fix).
• 23:30 MEH: taking 1x compute3 and compute2_himem from stdscience to run MD09.deeptest staticsky

Tuesday : 2013.07.09

• earlier Bill: set all chip, warp, and diff runs with ps_ud% labels to goto_cleaned
• 10:45 Bill: shut down ippc17 so Rita can check the fans. Since this is the data store machine we need to gracefully shut down some services
  • shut down pstamp and update pantasks
  • shut down apache with /etc/init.d/apache2 stop
  • shut down mysqld with mysqladmin shutdown (this takes awhile)
  • shut down host with shutdown -h now
    • I actually didn't know about the -h but I add it here for future reference
• 11:04 Bill: forgot about some other services dependent on ippc17 because of the data store. distribution cleanup (which is now hung) rcserver task in distribution (which was idle) and ipptopsps which writes to the data store.
• 11:35 Bill: restarted cleanup pantasks with dist.cleanup.off
• 12:24 Bill: pstamp restarted. Added a couple of sets of compute3 nodes to work on the mops backlog.
• 12:33 restarted update and set dist.cleanup.on (in cleanup) and rcserver.on (in distribution) Everything is back online.
• 13:10 MEH: returning 1x compute3 + compute2_himem from MD09.deeptest staticsky run back to stdsci
• 21:45 Bill: set cleanup to stop. Will restart with standard host config.

Wednesday : 2013.07.10

• 06:45 Bill: added some hosts to the cleanup pantasks. wave2, compute2, compute3

Thursday : 2013.07.11

Friday : 2013.07.12

Saturday : 2013.07.13

Sunday : 2013.07.14