PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : YYYY.MM.DD

Wednesday : 2018.01.31

• MEH: chip repeat fault 2 failure -- blocking automatic alt-diffs v2-v3 at end of night.. -- manually set quality 42 and fault 0, if happens again should be looked into

   	o8149g0350o 	XY27 	1371055 	2050838 	new 	OSS.nightlyscience 	OSS.20180131 	SweetSpot 	2 
  gpc1/OSS.nt/2018/01/31/o8149g0350o.1371055/o8149g0350o.1371055.ch.2050838.XY27.log
  
      Failed to calculate the min/max of the input vector.
          Auxiliary mask not enabled.
            Cell 0: 1 1
        PHOTCODE is GPC1.i.XY27
        --- psphot Model Background ---
          Unable to measure image background: too few data points (0) (0 failures)
          Unable to measure image background: too few data points (0) (1 failures)
          Unable to measure image background: too few data points (0) (2 failures)
      Failed to estimate background using ROBUST_MEDIAN for (434x423, (row0,col0) = (0,0)
      Failed to estimate background using ROBUST_MEDIAN for (434x800, (row0,col0) = (0,23)
      Failed to estimate background using ROBUST_MEDIAN for (434x800, (row0,col0) = (0,423)
          Unable to measure image background: too few data points (0) (100 failures)
                  Unable to measure image background: too few data points (0) (200 failures)
  Unable to measure image background: too few data points (0) (200 failures)
      Failed to estimate background using ROBUST_MEDIAN for (800x800, (row0,col0) = (2434,3223)
                  Unable to measure image background: too few data points (0) (300 failures)
  Unable to measure image background: too few data points (0) (300 failures)
      Failed to estimate background for 169 of 169 subimages
  Error in the psphot readout analysis
   -> psphotModelBackgroundReadout (psphotModelBackground.c:275): invalid data
       failed to build background image
   -> psphotModelBackgroundReadoutFileIndex (psphotModelBackground.c:394): invalid data
       Unable to generate background model
   -> psphotModelBackground (psphotModelBackground.c:413): invalid data
       failed to model background for PSPHOT.INPUT entry 0
      In ppImageMaskStats: 0 483835616
      In ppImageMaskStats: 23590328 0.675939 0.324061 0.000000 0.000000
  
  ....
  
  Number of leaks to display: 500
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/ppStatsFromMetadata /data/ipp118.1/nebulous/46/80/10454753075.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.stats - CHIP_IMFILE]...
      All data points have the same value: 20631.320312.
      All data points have the same value: 20631.320312.
  -bg 20631.320312 -bg_mean_stdev 0.000000 -bg_stdev 70.661133 -dtime_detrend 13.332060 -dtime_photom 1.159112 -dtime_total 118.431503 -n_stars 0.000000 -n_extended 0.000000 -n_cr 0.000000 -quality 0 -ver_pslib '40242' -ver_psmodules '40242' -ver_psphot '40242' -ver_psastro '40242' -ver_ppstats '40242' -ver_ppimage '40242' -maskfrac_npix 23590328 -maskfrac_static 0.675938 -maskfrac_dynamic 0.324062 -maskfrac_magic 0.000000 -maskfrac_advisory 0.000000 
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/funpack -S /data/ipp118.1/nebulous/ff/52/10454753063.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.ch.fits > /dev/null]...
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/funpack -S /data/ipp118.1/nebulous/94/2c/10454753064.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.ch.mk.fits > /dev/null]...
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/funpack -S /data/ipp118.1/nebulous/d5/cb/10454753065.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.ch.wt.fits > /dev/null]...
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/funpack -S /data/ipp118.1/nebulous/c7/30/10454753067.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.b1.fits > /dev/null]...
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/funpack -S /data/ipp118.1/nebulous/e7/fa/10454753068.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.b2.fits > /dev/null]...
  Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/funpack -S /data/ipp118.1/nebulous/af/8c/10454753066.gpc1:OSS.nt:2018:01:31:o8149g0350o.1371055:o8149g0350o.1371055.ch.2050838.XY27.mdl.fits > /dev/null]...
  Couldn't find expected output file: neb://ipp118.1/gpc1/OSS.nt/2018/01/31//o8149g0350o.1371055/o8149g0350o.1371055.ch.2050838.XY27.cmf at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 830
  	main::my_die('Couldn\'t find expected output file: neb://ipp118.1/gpc1/OSS....', 1371055, 2050838, 'XY27', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 799
  	main::check_output('neb://ipp118.1/gpc1/OSS.nt/2018/01/31//o8149g0350o.1371055/o8...', 1) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 554
  
  

• MEH: regular cleanup of active distribution datastores of old products
• MEH: regular bzip of pantask logs and mirror of ippitc account
• MEH: had to manually reset state of 2 diffs for QUB stamps, caught in cleanup but not cleaned
• MEH: neb-host adjustments --
  • ipp032 repair -- getting full, MOPS may need for gpc2 stamps until can update disk space
  • ipp105.0 up -- has space again (not red)
  • ipp115.0+1 -- red for while, put into repair to preserve space

Thursday : 2018.02.01

• MEH: moving ippqub:stdscience_ws (and other pantasks as needed) to run primarily on ippc65 since ippitc nightly pantasks running on ippc66,c67,c69 (not clear why c69 is used rather than c68...) in order to free up ippc106 for full compute node processing

Friday : YYYY.MM.DD

Saturday : 2018.02.03

• MEH: adding crontab setups for ippitc --
  • ippitc@ippc65 crontab to check pstamp every 3 hrs (avoiding 1530 pantasks restart) and restart pstamp if >100k in ~ippitc/chk_pstamp100k_cron_c65
  • ippitc@ippc64 crontab to quick/temporarily bzip rotated log files and rsync a mirror copy to /data/ipp032.0/ippitc/home_mirror_daily using ippitc_bziplog_mirror.csh (added to crontab where daily pantasks restart is)
  • reminder ippitc@ippc67 also runs crontab for fault 5 diff, repeat fault 4 warp to quality 42 end of night in ~ippitc/fault_cron_c67

Sunday : YYYY.MM.DD