PS1 IPP Czar Logs for the week 2013.02.11 - 2013.02.17

(Up to PS1 IPP Czar Logs)

Monday : 2013.02.11

• 11:20 Bill queued two M31 chip runs as a test.
• 11:30 Bill: fixed burntool table for o5257g0058o.ota22.fits
• 14:15 Bill: restarted stack after fixing typo in the input file
• 14:20 CZW: stopping processing to make a database change to allow testing of the new background code.
• 15:56 CZW: processing up again.

Tuesday : 2013-02-12

• 14:45 CZW: Added czw.bkg_test.ipp label to stack pantasks to check differences in stacking between the current working tag and the current trunk.

Wednesday : YYYY.MM.DD

Thursday : 2012.02.14

• 9:30 Bill: The cluster seems to have become somwhat wedged in the past half our. Setting stdscience to stop for a bit.
• 09:38 programs that are stuck are all trying to write files to /data/ipp063. Set neb-host ipp063 repair and restarted rpc.statd twice
• 09:53 several programs are also stuck writing to ipp007 set it to repair as well
• 10:12 rebooting ipp063. Restarting stdscience
• 10:18 neb-host ipp063 up ipp007 still in repair
• 10:27 neb-host ipp007 up
• 10:54 and now ipp041 is being a problem child. setting to repair
• 11:12 ipp007 hanging replicate requests. Setting to repair until nightly science finishes. This unclogged the queue to let the nightly chips finish. Setting chip off.
• 11:36 rfixed a slew of broken instances

neb://ipp025.0/gpc1/20110115/o5576g0197o/o5576g0197o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0208o/o5576g0208o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0196o/o5576g0196o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0198o/o5576g0198o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0213o/o5576g0213o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0215o/o5576g0215o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0216o/o5576g0216o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0217o/o5576g0217o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0218o/o5576g0218o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0250o/o5576g0250o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0267o/o5576g0267o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0192o/o5576g0192o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0194o/o5576g0194o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0195o/o5576g0195o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0212o/o5576g0212o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0203o/o5576g0203o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0220o/o5576g0220o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0251o/o5576g0251o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0252o/o5576g0252o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0268o/o5576g0268o.ota76.fits
neb://ipp025.0/gpc1/20110115/o5576g0269o/o5576g0269o.ota76.fits
neb://ipp025.0/gpc1/20110116/o5577g0181o/o5577g0181o.ota76.fits
neb://ipp025.0/gpc1/20110116/o5577g0198o/o5577g0198o.ota76.fits

• 13:04 Heather turned chip back on

Friday : 2013.02.15

mark is czar

• 08:00: MEH: nightly science looks finished, stdscience needs its regular restart and will do before SSdiffs get started
• 08:20 restarting gmond on ipp018,020,057,063 so ganglia memory report is correct
• 10:00 czar page hanging because replication problem on ? -- ippdb02 is out of disk space, db00 <70G -- all processing stopped except pstamp/update, deactivate LAP label to finish SSdiffims from last night in stdscience
• 12:30 Serge cleaned up some space a couple months, but disk upgrades very much needed. Processing restarted.
• 14:00 MEH: restart mysql on ipp010,012,026,057,058 found down when repairing LAP chip

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD