PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.02.15

• 13:00 EAM : I am going to launch relphot using most ippx nodes and the ipps nodes, and probably some of the ipp080 - ipp104 nodes. I'm restarting stdscience pantasks now to ensure nightly jobs are completed before launching relphot.
• 13:30 EAM : I have started relphot for the full PV3 master database. this will likely take 24-48 hours.

Tuesday : YYYY.MM.DD

• 10:30 CZW: ipp/replication pantasks running ippb03.0 OTA replication.

Wednesday : YYYY.MM.DD

Thursday : 2016.02.18

• 18:00 EAM : dvo operations are using most of the memory for the ipp0XX nodes. I'm stopping the pantaskses and I will remove the storage nodes from the processing set for standard processing.
• 21:20 EAM : summitcopy was stuck for a while and I had some trouble tracking down the reason. apparently, two exposures had a failure in summitcopy advance. using the command pztool -toadvance -dbname gpc1 -limit 10 showed the problem exposures. in summitcopy pantasks, the command show.books and then show.book pzPendingAdvance revealed the two exposures still on the books as if they were running -- they must have died in a way which pantasks dropped. stopping summitcopy and calling reset.advance cleared that book and made things right again.

Friday : 2016.02.19

• 09:40 MEH: clearing unrecoverable fault 4 (warp curve of growth) and fault 5 (diff)

  warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.0534.058 -warp_id 1688791 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.1550.056 -diff_id 1341246  -fault 0
  

• 11:30 MEH: will do regular restart of nightly pantasks once WSdiffs finish -- including registration which seems to have crashed on its own?
  • registration was in an odd state because someone started the outdated publishing pantasks which uses the same machine and port.. both pantasks had to be manually killed, now restart can normally be done..
• 15:20 CZW: Launching md5sum calculations across cluster so we can confirm with STSCI that they have all the updated files correct. This is going to mostly involve the ipp0[6-9]X computers with a few other nodes. I'm going to let it run in waves, so not all of these computers are busy at the same time. It's single thread per-host, so the load should only increase by about 1.

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD