PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.01.25

• 16:00 EAM: ipp007 rebooted to catch a disk, disk array rebuilding (temporarily at repair in nebulous).
• 16:01 EAM: stopping pantasks for restart
  • 16:05 EAM: restart is done

Tuesday : 2016.01.26

• 02:00 MEH: looks like ipp017 is unresponsive for past 5 min, nothing on console -- set neb-host down (from repair) to prevent possible stalling for detrends, no jobs are running on it
  • try power cycle after 10 min -- back up after F1 for keyboard error -- leaving neb-host down for night
• 02:36 MEH: ipp017 down again ~5 min ago.. setting power off for tonight
• 16:00 EAM : stopping the pantasks for a restart. Haydn is working on replacement mobos for ipp017, ippx052, ipx070, all currently off.
• 20:06 MEH: summit fault on o7414g0102o -- will finish clearing after SNIFS time finished.. dell switches had to be cycles and made the exposure appear missing and fault set -- manually removed fault once things back up and seems okay

  update summitExp set fault=0 where fault=110 and exp_name="o7414g0102o";
  

Wednesday : 2016.01.27

• 14:39 MEH: boosting pstamp for QUB and then regular restart of nightly pantasks

Thursday : 2016.01.28

• 07:12 MEH: summitcopy and registration been having some >1200s stalled jobs occasionally lately -- manually killed to finish nightly downloads
• 07:25 MEH: ippc09 disk full -- clear all the other ippc0x machines used for nebulous as well
• 11:09 MEH: looks like OSS may have out-of-order diffs -- ps1_29_2690, ps1_29_2523 missing visit 1 due to FWHM>12 pix at cam stage. ps1_29_2690 has visit 2-3,3-4 but ps1_29_2523 only has visit 2-3. will wait for MOPS to ask if wanted.
• 16:08 MEH: regular restart of nightly pantasks

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD