Summary for SAS41

Pixel Processing

Scripts from ~ipptest/SAS40 copied into in ~ipptest/SAS41 -- see README

Need to match processing user w/ tag used for LAP.PV3

• ch/cam/wrp -- ipplanl/psconfig/ipp-pv3-20140717.lin64, added pantasks ~ipplanl/stdsas
• stk -- ipplanl/psconfig/ipp-pv3-20140717.lin64, added pantasks ~ipplanl/stdsas
• ss/sc -- ippsky/psconfig/ipp-trunk-20150624.lin64, added ~ippsky/ss_sc_ffsas
• ff -- ippsky/psconfig/ipp-20150329.lin64, added ~ippsky/ss_sc_ffsas
• diff -- ipptest/psconfig/ipp-20150312.lin64, added ~ipptest/diffsas

Manually add label 20151208 to ippMonitor and restart czarPoll ~ipp/src/ippMonitor/czartool/czartool/CzarDb.pm

Chip/cam/wrp:

• script for SAS40 didn't use reduction LAP_SCIENCE, added for SAS41

  ./00_chips.cmd SAS.20151208 commit
  

Stack:

./01_stacks.cmd SAS.20151208 commit

• have fault 5 case --

  4974985 	skycell.1404.093 	SAS.20151208 	SAS.20151208 	SAS 	i.00000 	RINGS.V3 	new 	5 
        Input seeing FWHMs:
        Input 0: 3.988422
        Input 1: 3.858631
        Input 2: 1.000000
        Input 3: 0.989007
        PSF FWHM distribution: limit: 10.000000 (0.994503 0.005497 0.500000) (3.923527 0.064895 0.500000) 0.000000
      PSF 0 is completely bad --- not including in envelope calculation.
      PSF 1 is completely bad --- not including in envelope calculation.
      PSF 2 is completely bad --- not including in envelope calculation.
      PSF 3 is completely bad --- not including in envelope calculation.
      Data error code: 32cf
  

  • in SAS39/40 PSF was acceptable and stack was made -- manually set quality 42 --

    stacktool -dbname gpc1 -updatesumskyfile  -set_quality 42 -stack_id 4974985 -fault 0
    stacktool -dbname gpc1 -updaterun  -set_state full -stack_id 4974985 
    

• summary stacks made after all issues cleared -- stack.summary.on

Staticsky+skycal:

• have to run before full-force for listing to use

  ./02_ssky.cmd SAS.20151208 commit
  ./03_skycal.cmd SAS.20151208 commit
  

Full Force:

./05_fforce.cmd SAS.20151208 commit

• repeating fault 4 --

  neb://any/SAS2/SAS.20151208/RINGS.V3/skycell.1316.075/RINGS.V3.skycell.1316.075.wrp.1651301.ff.1038546.log
  
  Error in the psphot image loop
  
   -> psVectorStats (psStats.c:1922): unknown psLib error
       Failed to calculate vector minimum and maximum
   -> pmSourceRoughClass (pmSource.c:799): unknown psLib error
       failed to measure SN / moments stats
   -> psphotRoughClassRegion (psphotRoughClass.c:238): Programming error
       programming error calling pmSourceRoughClass
   -> psphotImageLoop (psphotImageLoop.c:136): Programming error
       failure in psphotReadout for chip 0, cell 0, readout 0
  

  • manually set quality 42 --

    fftool -dbname gpc1 -updateresult -ff_id 1038546 -warp_id 1651301 -fault 0 -quality 42 
    

Diff:

./06_diff.cmd SAS.20151208 commit

• ipp-20150312.lin64 broken for difftool queuing.. using ipp-pv3-20140717
• check DV5 format and inv.cmf being made -- yes

STScI will want pixels -- should be able to use same method for fetching the missing files (once missing LAP files finished)

• ~file sizes
  • warp ~1.66 TB (FITS including mdl only ~3GB)
  • stack ~384GB (unconv FITS), skycal cmf ~25GB
  • diff ~12 GB (cmf, subkernel), ~16 GB (cmf, subkernel, mdc, mdl.fits)
  • stack summary ~7.4GB
• setup access in 00_default_vhost.conf for ippc20-ippc29 under /data/ipp060.0/SAS41/ split by warp, diff, stack, sumstk and index listing of files
  • STScI downloading SAS data from ippc26-ippc29, using 100 wget for total 400 connections to those 4 machines -- they are seeing ~200Mbps per machine -- ~2.1TB in ~8 hours. locally see ~10-20MB/s on network i/o on those machines

Release tables:

• setup initial entries

  releasetool -dbname gpc1 -definerelease -set_surveyID 30 -set_release_name SAS.41 -set_release_state pending  -set_priority 0 -set_dataRelease 41
  
  select * from ippRelease where release_name like 'SAS.41';
  | rel_id | surveyID | release_name       | release_state | dataRelease | priority | dvodb                 | ubercal_file        | accessLevelMin |
  +--------+----------+--------------------+---------------+-------------+----------+-----------------------+---------------------+----------------+
  |    163 |       30 | SAS.41             | pending       |          41 |        0 | NULL                  | NULL                |              0 | 
  
  
  releasetool -dbname gpc1 -definerelexp -set_state processed -release_name SAS.41 -label SAS.20151208
  
  releasetool -dbname gpc1 -definerelstack -release_name SAS.41 -label SAS.20151208 -set_state processed -set_stack_type deep -simple
  

• need to wait for calibration of SAS before finalize

Internal diffs:

CZW: I'm defining a set of diffs that use SAS.20151208 as the source for both stacks and warps. The diff processing label will be SAS.20151208.internal, with a data_group of SAS.20151208.20160119. I'm using a copied and modified 06_diff.cmd in the ipptest/SAS41 directory to launch this, and will be reusing the ipptest/diffsas pantasks on ippc63.