PS1 IPP Czar Logs for the week 2013.07.08 - 2013.07.14

(Up to PS1 IPP Czar Logs)

Monday : 2013.07.08

• 08:04 Bill stopped distribution. It is falling all over itself because the new stsci machines do not have /local/ipp set up correctly. (It was set to be a link to /export/ipp053.0 which is inaccessible except from ipp053)
• 08:23 Set up /local/ipp on stsci10 - 19 (rm /local/ipp ; mkdir -p /export/$hostname.0/ipp/tmp; ln -s /export/$hostname.0/ipp /local ; mkdir /export/$hostname.0/ipp/gpc1 ; rsync -av /data/ipp053.0/ipp/gpc1/tess /local/ipp/gpc1
• 08:38 Bill: more distribution woes. The wave 2 nodes have absolutely no space in their /export partitions. Thus their /local/ipp/tmp has no space, so distribution tasks cannot run on those nodes. Restarted distribution with wave2 and wave3 nodes turned off and some compute nodes added in.
• 10:09 Bill: set M31.test and STS.rerun data to be cleaned. We now have 8000 chipRuns in state goto_cleaned
• 11:00 EAM : set all relevant machines with > 98% full to 'repair'. started dvo purge-temp to remove unneeded temp files from ipptopsps ops.
• 11:01 EAM : ipp054 crashed : ipp054.20130708.crashlog
  • console server for ipp054 (and ipp055 at least) does not allow power control -- contacted Gavin (who contacted Rita) to fix).

Tuesday : 2013.07.09

Wednesday : 2013.07.10

Thursday : 2013.07.11

Friday : 2013.07.12

Saturday : 2013.07.13

Sunday : 2013.07.14