PS1_IPP_CzarLog_20120423 – Pan-STARRS IPP

Context Navigation

Version 16 (modified by Mark Huber, 14 years ago) ( diff )
--

Mark czar tagging in for Roy

08:08 Bill: two chip_imfile processes have been running since around midnight making no progress. Killed the processes and reverted the faults. They quickly finished. These are the case of the 2 three pi exposures that haven't finished.
09:20 Mark: covering for Roy, stare03 out of processing for disk install
11:15 stare03 disks installed, back into processing.
12:00 with ganglia DOA, manually keeping eye on systems while MD08.GR0-z chip-warp runs with
```
~ipp/who_uses_the_cluster/sshToNodes.py ipp 'uptime'
```
17:10 going to restart stdscience once MD08.GR0-z finished, pcontrol running 100% cpu but still peaked @100 chip exposures/hr. will queue up MD08.GR0-y afterwards to run along nightly science and finish up.
20:30 stdscience dragging even after restart, try restart again. looked to be a pileup of jobs wanting ipp039.
22:30 with the new nodes, able to run MD08.GR0-y processing alongside nightly science and finish ~500 exposures chip->warp before morning.

Mark is czar

07:00 nightly science still downloading, full night of observations but seems slower than normal. been seeing chip_imfile/chip stage taking >1ks on various machines, also seems longer than normal -- look like some crowded fields.
09:30 nightly science 99.9% done
10:45 destreak_restore has been holding up the remaining 3PI nightly science and time sensitive MD SSdiffs from distribution.. setting destreak.revert.off until nightly science out.