PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : YYYY.MM.DD

Wednesday : 2018.10.03

• MEH: o8394h0565o XY00 also fault 4 in chip like XY36, manually set quality 42 to clear

  chiptool -dbname gpc2 -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 15331 -class_id XY00
  

• MEH: skyflats also have obs_mode OSS

Thursday : 2018.10.04

• 15:45 EAM : running MD03 exposures (g,r,z) for Euclid data release (chip -> warp) using ~ipptest/stdscience (ippc26 - ippc127, ex ippc70-ippc75). Processing is setup to stop at 6pm and restart at 6am.
• MEH: Haydn will be testing ethernet and fan setting on ippb18,20,21 -- neb-host down for a while now since unstable so may be off/on until hear back from him
• MEH: Ken notes changes made to PS2 code that may change the position angle calculation -- need to setup a larger grid search angle in camera/psastro -- made change directly to ops code

  /data/ippc18.0/home/ippps2/psconfig/default.lin64/share/ippconfig/gpc2/psastro.config
  PSASTRO.GRID.MIN.ANGLE F32 -10.0 
  PSASTRO.GRID.MAX.ANGLE F32 +10.0 
  

  • warp.off, fake.off, camera.off -- camera.on for first few exposures and look at log for grid soln -- looks mostly unchanged to previous day -- return to normal setting, camera.on,fake.on,warp.on
• MEH: ippx001-012, x025-x032 power cycled a little before 2000 HST, ippx042-x044+others lost network -- ippx005 slow coming back again? turn off in stdscience
  • looks like happened again ~20:40 HST -- turning off 3x x0,x1 and adding 4x s4+5+6+7 datanodes
• MEH: another chip fault 4 case like XY36 -- manually set quality 42

  chiptool -dbname gpc2 -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 15860 -class_id XY46
  

• MEH: download rate from summit is a bit slower than normal ~40 exp/hr instead of 50 exp/hr -- Craig notes servers doing other key work all in one queue until can get back to Maui and finish fixing the systems -- should be okay for now, last chunk downloading at end of night (~70 exp behind) and at end of night things will also speed up as well
• MEH: exposure o8396h0008o stuck in fake stage, suspect from xnodes rebooting putting in odd state -- missing XY41 so manually ran

  fake_imfile.pl --exp_id 29846 --fake_id 14569 --class_id XY41 --chiproot=neb://any/gpc2/OSS.nt/2018/10/05//o8396h0008o.29846/o8396h0008o.29846.ch.15848 --camroot=neb://any/gpc2/OSS.nt/2018/10/05//o8396h0008o.29846/o8396h0008o.29846.cm.15400 --camera GPC2 --outroot neb://any/gpc2/OSS.nt/2018/10/05//o8396h0008o.29846/o8396h0008o.29846.fk.14569 --reduction SWEETSPOT --dbname gpc2 --verbose
  

Friday : 2018.10.05

• MEH: looks like either lost power or network to H02 cabinet -- will need to reallocate ps2:stdscience(04->c23),summitcopy(00->c24),registration(02->c25) and ps1:pstamp(c33->c26) pantasks, set neb-host down ipp032,058-066 as long as they are powered off and not just network offline, update ~ippps2/ippconfig/site.config and comment out PS1.REF.20170919 for ipp058-066
  • visited the ITC -- nothing on fire, really hot, or smoking/smell burnt -- systems appear booted but w/o network so could not log in to check for how long -- manually power cycled stare00,02,04 to clear any ps2 pantasks if still trying to run

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD