= Summary for SAS41 = 
[[PageOutline]]


== Pixel Processing ==

Scripts from ~ipptest/SAS40 copied into in ~ipptest/SAS41 -- see README

Need to match processing user w/ tag used for LAP.PV3 
 * ch/cam/wrp -- ipplanl/psconfig/ipp-pv3-20140717.lin64, added pantasks ~ipplanl/stdsas
 * stk -- ipplanl/psconfig/ipp-pv3-20140717.lin64, added pantasks ~ipplanl/stdsas
 * ss/sc -- ippsky/psconfig/ipp-trunk-20150624.lin64, added ~ippsky/ss_sc_ffsas
 * ff -- ippsky/psconfig/ipp-20150329.lin64, added ~ippsky/ss_sc_ffsas
 * diff -- ipptest/psconfig/ipp-20150312.lin64, added ~ipptest/diffsas

Manually add label 20151208 to ippMonitor and restart czarPoll
~ipp/src/ippMonitor/czartool/czartool/CzarDb.pm


Chip/cam/wrp:
 * script for SAS40 didn't use reduction LAP_SCIENCE, added for SAS41
{{{
./00_chips.cmd SAS.20151208 commit
}}}


Stack:
{{{
./01_stacks.cmd SAS.20151208 commit
}}}
 * have fault 5 case --
{{{
4974985 	skycell.1404.093 	SAS.20151208 	SAS.20151208 	SAS 	i.00000 	RINGS.V3 	new 	5 
      Input seeing FWHMs:
      Input 0: 3.988422
      Input 1: 3.858631
      Input 2: 1.000000
      Input 3: 0.989007
      PSF FWHM distribution: limit: 10.000000 (0.994503 0.005497 0.500000) (3.923527 0.064895 0.500000) 0.000000
    PSF 0 is completely bad --- not including in envelope calculation.
    PSF 1 is completely bad --- not including in envelope calculation.
    PSF 2 is completely bad --- not including in envelope calculation.
    PSF 3 is completely bad --- not including in envelope calculation.
    Data error code: 32cf
}}}
  * in SAS39/40 PSF was acceptable and stack was made -- manually set quality 42 -- 
{{{
stacktool -dbname gpc1 -updatesumskyfile  -set_quality 42 -stack_id 4974985 -fault 0
stacktool -dbname gpc1 -updaterun  -set_state full -stack_id 4974985 
}}}
 * summary stacks made after all issues cleared --  stack.summary.on


Staticsky+skycal:
 * have to run before full-force for listing to use 
{{{
./02_ssky.cmd SAS.20151208 commit
./03_skycal.cmd SAS.20151208 commit
}}}


Full Force:
{{{
./05_fforce.cmd SAS.20151208 commit
}}}
 * repeating fault 4 --  
{{{
neb://any/SAS2/SAS.20151208/RINGS.V3/skycell.1316.075/RINGS.V3.skycell.1316.075.wrp.1651301.ff.1038546.log

Error in the psphot image loop

 -> psVectorStats (psStats.c:1922): unknown psLib error
     Failed to calculate vector minimum and maximum
 -> pmSourceRoughClass (pmSource.c:799): unknown psLib error
     failed to measure SN / moments stats
 -> psphotRoughClassRegion (psphotRoughClass.c:238): Programming error
     programming error calling pmSourceRoughClass
 -> psphotImageLoop (psphotImageLoop.c:136): Programming error
     failure in psphotReadout for chip 0, cell 0, readout 0
}}}
   * manually set quality 42 --
{{{
fftool -dbname gpc1 -updateresult -ff_id 1038546 -warp_id 1651301 -fault 0 -quality 42 
}}}


Diff:
{{{
./06_diff.cmd SAS.20151208 commit
}}}
 * ipp-20150312.lin64 broken for difftool queuing.. using ipp-pv3-20140717
 * check DV5 format and inv.cmf being made -- yes


STScI will want pixels -- should be able to use same method for fetching the missing files (once missing LAP files finished)
 * ~file sizes
  * warp ~1.66 TB (FITS including mdl only ~3GB)
  * stack ~384GB (unconv FITS), skycal cmf ~25GB
  * diff ~12 GB (cmf, subkernel), ~16 GB (cmf, subkernel, mdc, mdl.fits)
  * stack summary ~7.4GB
 * setup access in 00_default_vhost.conf for ippc20-ippc29 under /data/ipp060.0/SAS41/ split by warp, diff, stack, sumstk and index listing of files
  * STScI downloading SAS data from ippc26-ippc29, using 100 wget for total 400 connections to those 4 machines -- they are seeing ~200Mbps per machine -- ~2.1TB in ~8 hours. locally see ~10-20MB/s on network i/o on those machines



Release tables:
 * setup initial entries
{{{
releasetool -dbname gpc1 -definerelease -set_surveyID 30 -set_release_name SAS.41 -set_release_state pending  -set_priority 0 -set_dataRelease 41

select * from ippRelease where release_name like 'SAS.41';
| rel_id | surveyID | release_name       | release_state | dataRelease | priority | dvodb                 | ubercal_file        | accessLevelMin |
+--------+----------+--------------------+---------------+-------------+----------+-----------------------+---------------------+----------------+
|    163 |       30 | SAS.41             | pending       |          41 |        0 | NULL                  | NULL                |              0 | 


releasetool -dbname gpc1 -definerelexp -set_state processed -release_name SAS.41 -label SAS.20151208

releasetool -dbname gpc1 -definerelstack -release_name SAS.41 -label SAS.20151208 -set_state processed -set_stack_type deep -simple
}}}
 * need to wait for calibration of SAS before finalize

Internal diffs:

CZW: I'm defining a set of diffs that use SAS.20151208 as the source for both stacks and warps.  The diff processing label will be SAS.20151208.internal, with a data_group of SAS.20151208.20160119.  I'm using a copied and modified 06_diff.cmd in the ipptest/SAS41 directory to launch this, and will be reusing the ipptest/diffsas pantasks on ippc63.