(Up to PS1 IPP Czar Logs)

Monday : 2012.09.03

• 11:40 EAM : some NFS hung mounts on ipp016 (from stsci). I am stopping processing to clear out hung jobs
• 12:40 EAM : I tried to clear the NFS mounts, with no luck. I rebooted ipp016 and restarted processing

Tuesday : YYYY.MM.DD

• 11:44 CZW: Restarted stdscience because the rate plots where showing <50 exp/hour.

Wednesday : YYYY.MM.DD

• 09:02 (Serge): Stopped replication on ippc63.
• 09:10 (Serge): Started rsync to ippc62 (/export/ippc62.0/backup_nebulous/20120905 ; screen session: backup_nebulous on ippc62 rsync -e rsh -avz ippc63:/export/ippc63.0/mysql .)
• 13:05 (Serge): chip.revert.off for LAP surgery
  • Fixed:
    • gpc1/20100607/o5354g0061o/o5354g0061o.ota35.burn.tbl
    • gpc1/20100228/o5255g0601o/o5255g0601o.ota62.burn.tbl
    • gpc1/20100228/o5255g0603o/o5255g0603o.ota24.burn.tbl
    • gpc1/20100607/o5354g0213o/o5354g0213o.ota35.burn.tbl
    • gpc1/20100607/o5354g0252o/o5354g0252o.ota06.burn.tbl
    • gpc1/20100607/o5354g0240o/o5354g0240o.ota34.burn.tbl
    • gpc1/20100607/o5354g0242o/o5354g0242o.ota13.burn.tbl
    • gpc1/20100607/o5354g0260o/o5354g0260o.ota13.burn.tbl
    • gpc1/20100607/o5354g0261o/o5354g0261o.ota06.burn.tbl
  • Recovered:
    • gpc1/20100607/o5354g0236o/o5354g0236o.ota02.fits
    • gpc1/20100607/o5354g0261o/o5354g0261o.ota65.fits
    • gpc1/20100616/o5363g0091o/o5363g0091o.ota37.fits
    • gpc1/20100616/o5363g0111o/o5363g0111o.ota26.fits
    • gpc1/20100616/o5363g0110o/o5363g0110o.ota44.fits
    • gpc1/20100617/o5364g0138o/o5364g0138o.ota52.fits
    • gpc1/20100619/o5366g0205o/o5366g0205o.ota52.fits
    • gpc1/20100619/o5366g0223o/o5366g0223o.ota51.fits
    • gpc1/20100619/o5366g0202o/o5366g0202o.ota44.fits
  • Lost
    • gpc1/20100617/o5364g0160o/o5364g0160o.ota65.fits (XY65 182678 581641)
• 13:55 (Serge): chip.revert.on
• 14:50 (Serge): Shutting everything down (E-mail from Gavin about power outage at mhpcc)
• 18:10 (haf/czw): haf restarted processing (all but addstars and deepstack... deepstack was never shut off?). czw restarted nebdiskd, which kicked out ipp013 and ipp046 (these will be off tonight). haf is doing stuff with addstars, so ignore if they are off.
• 19:00 (haf) restarted czarpoll/robovzar

Thursday : YYYY.MM.DD

• 07:00 (Serge): Started slave on ippdb02.
• 07:05 (Serge): Registration was stuck but now seems to run again.

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD