PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.06.23

• 20:30(EDT) MEH: restarting pantasks, been running a while and need some stamps quickly

Tuesday : 2014.06.24

• 10:00(EDT) MEH: compute3 nodes/c2 mostly idle so using for staticsky for couple hours. and kicking registration so not waiting to revert and un-stall itself.

Wednesday : 2014.06.25

• 08:45 (EDT) MEH:ipp030 down.. power cycled ipp030.crash.20140625

Thursday : 2014.06.26

• 07:55 (EDT) MEH: registration 170 exposures behind, kicked and moving again

  crash for: register_imfile.pl --exp_id 760419 --tmp_class_id ota47 --tmp_exp_name o6834g0217o --uri neb://ipp030.0/gpc1/20140626/o6834g0217o/o6834g0217o.ota47.fits --logfile neb://ipp030.0/gpc1/20140626/o6834g0217o.760419/o6834g0217o.760419.reg.ota47.log --bytes 23063040 --md5sum 5d3046529d950808f4a0b6d8a1f15b66 --sunset 03:30:00 --sunrise 17:30:00 --summit_dateobs 2014-06-26T08:41:43.000000 --dbname gpc1 --verbose
  job exit status: CRASH
  job host: ipp030
  job dtime: 8.739945
  job exit date: Wed Jun 25 23:07:27 2014
  hostname: ipp030
  
  -- had to manually do 
  regtool -updateprocessedimfile -exp_id 760419 -class_id XY47 -set_state pending_burntool -dbname gpc1
  
  

• 09:10(EDT) MEH: still far behind in processing, compute3 nodes idle so throwing into processing for a few hours
• 1!:25(EDT) MEH: stdsci pantasks struggling to stay loaded, needing regular restart and doing others as well
• 12:00(EDT) MEH: once nightly finished, using idle compute3 nodes for small test MD.PV3 processing

Friday : 2014.06.27

• 02:40(EDT) MEH: ipp030 having issues with becoming unresponsive/high load, turning off in processing
• 18:10(PDT) MEH: Haydn looking into ipp030 crash around 0710(HST) this morning. -- setting neb-host down and removing from pantasks_hosts.input
  • some stdscience jobs need to probably be cleared and might as well restart stdsci

    -- both have files on ipp030 so won't finish until back up
    system failure for: diff_skycell.pl --threads @MAX_THREADS@ --diff_id 570810 --skycell_id skycell.0615.037 --diff_skyfile_id 33349104 --outroot neb://stsci17.1/gpc1/ThreePi.nt/2014/06/27/RINGS.V3/skycell.0615.037/RINGS.V3.skycell.0615.037.dif.570810 --redirect-output --run-state new --inverse --dbname gpc1 --verbose
    
    system failure for: diff_skycell.pl --threads @MAX_THREADS@ --diff_id 570810 --skycell_id skycell.0615.037 --diff_skyfile_id 33349104 --outroot neb://stsci17.1/gpc1/ThreePi.nt/2014/06/27/RINGS.V3/skycell.0615.037/RINGS.V3.skycell.0615.037.dif.570810 --redirect-output --run-state new --inverse --dbname gpc1 --verbose
    

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD