IPP for PS2

(Up to the Main Page)

This page lists information related to the IPP processing of the PS2 data as of 2018-04-09.

Processing

Processing is running as the ippps2 user using a checkout of the trunk code. It is set up to match the ippitc user as closely as possible. There are pantasks directories defined for

pantasks	host	port	cluster loading
detrend	stare04	2001:2009	stare*3 ippx085-88*6
registration	stare02	2001:2009	hosts.registration
stdscience	stare04	2010:2019	stare*3 ippx077-88*6
summitcopy	stare00	2010:2019	hosts.summitcopy

These hosts were chosen to not interfere with GPC1 nightly processing. The registration and summitcopy hosts were only being run during the day when the configuration was written, so the duplication was not a problem. These should be changed before GPC2 switches over to full night observing.

The stdscience pantasks has the nighlyscience labels for OSS, ESS, MSS, ThreePi, SNIa, QUB, QUB.ic, CFA, FSS, K2, and Bright loaded. Only OSS.nightlyscience is configured to publish results to MOPS. The standard stdscience modules are loaded (chip-warp, diff, nighly_science, publish, survey).

The processing database is tgpc2, running on ippdb08.

Configuration changes

Although processing is running on the trunk, I (CZW) have modified some of the code to work-around issues. These will likely be committed to the trunk once I'm convinced there is no impact on non-GPC2 data, and when changes made for expediency have proper solutions identified.

• ippTasks/nightly_science.pro hard codes a set of camera map entries. This is changed to reflect the new database. There is also a syntax fix to correct new stricter interpretation.
• ppImage/src/ppImageMaskStats.c did not cleanly handle the case in which an entire OTA is masked. Some GPC2 detectors had this issue. The code change sets this to a quality value of 5007.
• ippScripts/scripts/camera_exp.pl has a list of cameras to skip construction of background images. This is not a clean solution, but I added GPC2 to this list.
• ippScripts/scripts/detrend_norm_calc.pl assumes a normalization of 0.0 or NAN is always wrong, but this is not the case when a detector has no good pixels. This is not a clean solution, but I set the normalization to 1.0 in this case, and do not throw an error.
• Ohana/src/skycalc/Makefile add sunmoon to the list of commands to build.
• ippconfig/recipes/nightly_science.config add simple (i) filter names instead of the correct long form (i.00002).
• ippconfig/gpc2/psastro.config be extremely permissive and allow the max FWHM to be 100 pixels. Use 2017 refcat.
• ippconfig/gpc2/ppImage.config add PPIMAGE_BKGCONT recipe with BACKGROUND.CONTINUITY disabled.

Known issues

• The filter is not being interpreted from the header correctly. The configuration lists the FILTERID keyword being used for both FPA.FILTERID and FPA.FILTER concepts. The header lists FILTERID='i.00002 ' AND FILTER='i ', but the tgpc2 database lists the filter as 'i'. The header keywords and configuration both match what is being done with GPC1, so this mis-interpretation is not understood.
• The nightly_science desperate diff routine depends on the existence of a morning dark to signal that all observations have been taken. These are not currently being taken (although Angie has emailed and may set this up soon), which prevented one diff from being constructed automatically. I launched the diff with the command

difftool -definewarpwarp -dbname tgpc2 -set_workdir neb://@HOST@.0/tgpc2/OSS.nt/2018/03/30 -set_label czw.up -set_data_group OSS.20180330 -set_reduction SWEETSPOT -set_dist_group SweetSpot -exp_id 5729 -template_exp_id 5737 -input_label czw.up -template_label czw.up -good_frac 0.1 -backwards -trace db 5 -rerun.

• There is no cleanup pantasks, so nightly science exposures go to label="goto_cleaned" and state="goto_cleaned". For the one night of science data, I moved them back using STAGEtool -dbname tgpc2 -updaterun -label goto_cleaned -set_label czw.up -set_state full.
• The current MASK is a DARKMASK+FLATMASK that does not exclude static bad pixels/columns that are not caught by those processes.
• The i-band GPC1 stacks with label="LAP.PV3.20140730.final" are being ingested into the tgpc2 database to allow for warp (GPC2) - stack (GPC1) difference image processing. The stackRun.filter column will need to be modified, as difftool blocks diffs that do not have the same filter. The ingest process uses the stack_id value from the gpc1 database, so the tgpc2.stackRun and tgpc2.stackSumSkyfile tables will have a much higher auto-increment value. The stackInputSkyfile entries are lost, as they reference warpSkyfile entries that do not exist.