PS1 IPP Czar Logs for the week 2013.08.12 - 2013.08.18

(Up to PS1 IPP Czar Logs)

Monday : 2013.08.12

• 08:47 Bill: update pantasks has stuck jobs (88,000 seconds) probably due to the down node. Set pantasks to stop in preparation for clearing out the jobs
• 09:10 Bill restarted update and pstamp pantasks
• -9:20 Bill droped skycell that was repeatedly faulting: difftool -updatediffskyfile -set_quality 42 -fault 0 -diff_id 463628 -skycell_id skycell.1226.089
• 14:00 CZW: added do.more.summary macro to the stack pantasks, which allows the stack.summary tasks to run continually. I've limited the number of simultaneous run tasks to 10 to try and keep the load from getting too high (this task needs to read 100 stack skycells, so it's somewhat memory intensive).
• 20:00 Bill: earlier queued 105 M31 exposures to be processed. Now have second thoughts. Compromise: removed M31.rp.2013 label from stdscience. Left M31.rp.2013.bgsub in. This will do background subtracted chip - warp processing, but will defer background uncorrected chip, chip_bg, and warp_bg processing until the label is added back in.

Tuesday : YYYY.MM.DD

• 01:55 MEH: stsci19 has been down for ~3hrs.. stalling all processing. setting neb-host down and see if can triage things until morning
  • not looking good. may be more than just stsci19, often seeing timeout when checking mounts to ipp052.0-ipp066.0 -- leaving until morning
• 07:55 Bill: turning revert tasks off in stdscience faulted tasks just fault again
• 08:00 Bill: summit copy has 132 incomplete downloads 66 copied but not registered
  • killed a couple of stuck registration processes (> 20000 sec)
  • ran pztool -clearcommonfaults
  • making some progress now
  • regtool -updateprocessedimfile -class_id XY53 -exp_id 642199 -set_state pending_burntool -burntool_state 0
• 08:25 ipp058 has 14 "sync" processes all hung which is causing several scripts to not exit.
• 08:35 restarted registration pantasks. hosts ipp058 and ipp065 set to off because they hang in sync
• 08:39 registration is running many more processes after restart. burntool is making progress again. XY14 and XY27 are behind
• 08:49 Bill restarted pstamp pantasks
• 09:32 Bill ran pztool -clearcommonfaults to revert faulted imfile from o6517g0198o ota24 (files were copied perviously fault was "can't lock file"). May require further intervention.
• 09:38 EAM : a number of machines (ipp014, ipp015, ipp016, ipp019, ipp020, probably others) have ipp052 mounts making things slow. I rebooted ipp013, but it will be a pain to reboot all of these. Haydn is checking into ipp052, so we'll leave it for now.
• 13:51 Bill: added label M31.rp.2013 to stdscience

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD