PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2018.07.02

• MEH: ipps05 went offline ~18:45 HST, nothing on console -- tried power cycle twice and not booting, power off until can be looked at

Tuesday : 2018.07.03

• MEH: camera issue last night -- first test dark came out as o8302g0106o, another came out as c8302g0003d -- c8302g0003d registered ok, o8302g0106o stalled in registration

Wednesday : 2018.07.04

• MEH: for some reason PS2 WWdiffs were not being automatically queued... no errors anywhere and pantasks seemed to be running normally since restart last afternoon... manually restarting seemed to clear whatever the problem was...
• MEH: ippx057 showing io cpu wait states slowing processing on ps2, remove from processing until can look at
• MEH: another set of excessively long running WWdiffs killed and set quality 42 -- dtime for ppSub and ppImage show this isn't uncommon and often hides during the night processing and why rate is possibly slower than should be -- additional/temporary masking suggested at IPP meeting should help improve both cases

Thursday : YYYY.MM.DD

Friday : 2018.07.06

• MEH: HVAC down at IfA today @noon through the weekend -- no cooling in Mez, power down non-critical nodes

  ipp001 -- mysql shutdown
  ipp022 -- mysql shutdown
  ipp003 -- mysql shutdown, inform QUB
  ippb04 -- neb-host down, 
  ippb05 -- neb-host down,
  

• MEH: IfA computer room B224 also losing AC.. have to power down ippc20,c21,c22 -- c20 with ubuntu should have done shutdown, manually powered off after halt
• MEH: summit fault on non-critical dark (o8306g0009d) fault 250 -- bad exposure, NULL imfiles on conductor

  line 2: field : does not conform to (?-xism:^\d+$): o8306g0009d01.fits|||chip|ota01   | at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/lib/DataStore/FileSet.pm line 226
  

  • no clear way to clear, entry in summitExp but not in pzDownloadExp to set a state to drop and clear fault reporting -- manually set exp_type in summitExp to not get confused by other faults on ippMonitor page but will trigger summit fault alert email
  • manually dropping from summitExp as done in czarlog past only proceeded to be re-added -- manually inserted entry into psDownloadExp with state drop to clear fault reporting

Saturday : 2018.07.07

• MEH: missing PS1 Euclid twilight and dark exposures in morning -- cannot connect to conductor
  • Curt confirms network to summit -- The HO summit network went down serious 4:45am due to a Hawaiian Telcom outage. HT was notified around 5am.
  • Curt added ps-obs list to IfA HO list

Sunday : 2018.07.08

• MEH: all PS2 exposures quality error 4007 from failed astrometry --
  • allowing larger GRID ANGLE range have many succeeding around -4 to -6 -- not all, still testing
  • also needs a large GRID offset range ~1800, set to 3000 in SWEETSPOT_TEST manually added ~ippps2/psconfig/default.lin65/share/ippconfig for ippps2 (ie not compiled...)
  • sample set for ps1_28_5512 object looks ok through WWdiff -- full reprocessing of night with SWEETSPOT_TEST hack running label gpc2:mehtest180708 -- but helps to queue w/ hacked reduction to use.. try again and now 150 WWdiffs for 300 exposures
• MEH: IfA A/C back on, power up Mez computers -- ipp001,003 ok, ippb04 (missing drive), b05 (POST multiple cycle finally booted) Gavin addressed, ipp022 Gavin will power on in morning