PS1 IPP Czar Logs for the week 2016.07.11 - 2016.07.17

(Up to PS1 IPP Czar Logs)

Monday : 2016.07.11

Tuesday : 2016.07.12

Wednesday : 2016.07.13

  • 09:00 EAM: exposure o7582g0220o had a bad chip and was dropped, but still showed up in the to-be-downloaded list. i've deleted the bad chip:
    summitcopy pantasks stop
    mysql> delete from summitImfile WHERE summitImfile.exp_name like 'o7582g0220o' and summit_id = 1113804 and class_id = 'ota51';
    mysql> update summitExp set imfiles = 59 WHERE exp_name = 'o7582g0220o';
    pztool -dbname gpc1 -revertcopied -exp_name o7582g0220o -inst gpc1 -telescope ps1 -fault 110
    pztool -updatepzexp -exp_name o7582g0220o -inst gpc1 -telescope ps1 -set_state run -summit_id 1113804 -dbname gpc1
    summitcopy pantasks run
    
  • 09:05 EAM: clear a bad fault:
    warptool -updateskyfile -set_quality 42 -fault 0 -warp_id 1748945 -skycell_id skycell.2639.084 -dbname gpc1
    
  • 10:55 MEH: QUB.LOWDEC.update.20160712 chip+warp updates running for week on ippsXX nodes, pantasks on ippmops:stdscience:ipps00 if needs to be stopped
  • 15:00 EAM: stopping & restarting pantasks

Thursday : 2016.07.14

  • 11:43 MEH: QUB.LOWDEC.update.20160712 chip+warp updates also running on c2 group nodes (~half loaded, daytime only)
  • 19:45 EAM: restarting pantasks

Friday : 2016.07.15

Saturday : 2016.07.16

  • 08:30 MEH: ipp063 looks to be having trouble, large wait cpu state since ~0600 -- neb-host repair helps some, may need to have NFS restarted or just rebooted (looks like many disk/medium errors in messages and ps-ipp-ops emails so rebooting is probably not a good idea until raid can be looked at)
    • nightly cleared, seems fine now with just updates running when in repair

Sunday : 2016.07.17