PS1 IPP Czar Logs for the week 2013.08.12 - 2013.08.18

(Up to PS1 IPP Czar Logs)

Monday : 2013.08.12

• 08:47 Bill: update pantasks has stuck jobs (88,000 seconds) probably due to the down node. Set pantasks to stop in preparation for clearing out the jobs
• 09:10 Bill restarted update and pstamp pantasks
• -9:20 Bill droped skycell that was repeatedly faulting: difftool -updatediffskyfile -set_quality 42 -fault 0 -diff_id 463628 -skycell_id skycell.1226.089
• 14:00 CZW: added do.more.summary macro to the stack pantasks, which allows the stack.summary tasks to run continually. I've limited the number of simultaneous run tasks to 10 to try and keep the load from getting too high (this task needs to read 100 stack skycells, so it's somewhat memory intensive).
• 20:00 Bill: earlier queued 105 M31 exposures to be processed. Now have second thoughts. Compromise: removed M31.rp.2013 label from stdscience. Left M31.rp.2013.bgsub in. This will do background subtracted chip - warp processing, but will defer background uncorrected chip, chip_bg, and warp_bg processing until the label is added back in.

Tuesday : YYYY.MM.DD

• 01:55 MEH: stsci19 has been down for ~3hrs.. stalling all processing. setting neb-host down and see if can triage things until morning
  • not looking good. may be more than just stsci19, often seeing timeout when checking mounts to ipp052.0-ipp066.0 -- leaving until morning
• 07:55 Bill: turning revert tasks off in stdscience faulted tasks just fault again
• 08:00 Bill: summit copy has 132 incomplete downloads 66 copied but not registered
  • killed a couple of stuck registration processes (> 20000 sec)
  • ran pztool -clearcommonfaults
  • making some progress now

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD