IPP for PS2

(Up to the Main Page)

This page lists information related to the IPP processing of the PS2 data as of 2018-04-09.

Processing

Processing is running as the ippps2 user using a checkout of the trunk code. It is set up to match the ippitc user as closely as possible. There are pantasks directories defined for

pantasks	host	port	cluster loading
detrend	stare04	2001:2009	stare*3 ippx085-88*6
registration	stare02	2001:2009	hosts.registration
stdscience	stare04	2010:2019	stare*3 ippx065-88*6
summitcopy	stare00	2010:2019	hosts.summitcopy

These hosts were chosen to not interfere with GPC1 nightly processing. The registration and summitcopy hosts were only being run during the day when the configuration was written, so the duplication was not a problem. These should be changed before GPC2 switches over to full night observing.

The stdscience pantasks has the nighlyscience labels for OSS, ESS, MSS, ThreePi, SNIa, QUB, QUB.ic, CFA, FSS, K2, and Bright loaded. Only OSS.nightlyscience is configured to publish results to MOPS. The standard stdscience modules are loaded (chip-warp, diff, nighly_science, publish, survey).

The processing database is tgpc2, running on ippdb08.

Configuration changes

Although processing is running on the trunk, I (CZW) have modified some of the code to work-around issues. These will likely be committed to the trunk once I'm convinced there is no impact on non-GPC2 data, and when changes made for expediency have proper solutions identified.

• ippTasks/nightly_science.pro hard codes a set of camera map entries. This is changed to reflect the new database. There is also a syntax fix to correct new stricter interpretation.
• ppImage/src/ppImageMaskStats.c did not cleanly handle the case in which an entire OTA is masked. Some GPC2 detectors had this issue. The code change sets this to a quality value of 5007.
• ippScripts/scripts/camera_exp.pl has a list of cameras to skip construction of background images. This is not a clean solution, but I added GPC2 to this list.
• ippScripts/scripts/detrend_norm_calc.pl assumes a normalization of 0.0 or NAN is always wrong, but this is not the case when a detector has no good pixels. This is not a clean solution, but I set the normalization to 1.0 in this case, and do not throw an error.
• Ohana/src/skycalc/Makefile add sunmoon to the list of commands to build.
• ppSub/src/ppSubCamera.c modifies the camera name in the config object to allow warp-stack diffs to be constructed using GPC1 LAP stacks.
• ippconfig/recipes/nightly_science.config add simple (i) filter names instead of the correct long form (i.00002).
• ippconfig/gpc2/psastro.config be extremely permissive and allow the max FWHM to be 100 pixels. Use 2017 refcat.
• ippconfig/gpc2/ppImage.config add PPIMAGE_BKGCONT recipe with BACKGROUND.CONTINUITY disabled.

Known issues

• The filter is not being interpreted from the header correctly. The configuration lists the FILTERID keyword being used for both FPA.FILTERID and FPA.FILTER concepts. The header lists FILTERID='i.00002 ' AND FILTER='i ', but the tgpc2 database lists the filter as 'i'. The header keywords and configuration both match what is being done with GPC1, so this mis-interpretation is not understood.
  • A closer look shows that GPC1 uses FILTERID for both concepts. ppStats uses the FPA.FILTER concept to populate the filter, so this is the source of the problem. I'm hesitant to fix this right away, as it will require changing all the entries in the database already. It might be best to wait until things are more stable and we're in a true commissioning phase.
• The nightly_science desperate diff routine depends on the existence of a morning dark to signal that all observations have been taken. These are not currently being taken, which prevented one diff from being constructed automatically. I launched the diff with the command

difftool -definewarpwarp -dbname tgpc2 -set_workdir neb://@HOST@.0/tgpc2/OSS.nt/2018/03/30 -set_label czw.up -set_data_group OSS.20180330 -set_reduction SWEETSPOT -set_dist_group SweetSpot -exp_id 5729 -template_exp_id 5737 -input_label czw.up -template_label czw.up -good_frac 0.1 -backwards -trace db 5 -rerun.

• These darks are now being taken, but as of 2018-04-17, the obs_mode does not match the expected value. If this continues, I will remove this restriction from the configuration file (a comment of 'darks_morn%' is probably sufficiently restrictive).

• There is no cleanup pantasks, so nightly science exposures go to label="goto_cleaned" and state="goto_cleaned". For the one night of science data, I moved them back using STAGEtool -dbname tgpc2 -updaterun -label goto_cleaned -set_label czw.up -set_state full.
• The current MASK is a DARKMASK+FLATMASK that does not exclude static bad pixels/columns that are not caught by those processes.
• Occasionally the OBJECT field is not cleared before engineering data is taken. This has not resulted in non-science exposures being processed, but it is a possibility.
• The i-band GPC1 stacks with label="LAP.PV3.20140730.final" are being ingested into the tgpc2 database to allow for warp (GPC2) - stack (GPC1) difference image processing. The stackRun.filter column will need to be modified, as difftool blocks diffs that do not have the same filter. The ingest process uses the stack_id value from the gpc1 database, so the tgpc2.stackRun and tgpc2.stackSumSkyfile tables will have a much higher auto-increment value. The stackInputSkyfile entries are lost, as they reference warpSkyfile entries that do not exist.
  • Warp (GPC2) - stack (GPC1) difference images do not currently work, because there is a database JOIN from the stackInputSkyfile table to the warpRun table, I assume to prevent a stack from being subtracted from an input warp. However, as the stack entries are fakes from the GPC1 database, this JOIN fails, and no stacks are found when the diff is attempted. This is not a queuing issue, as the diffInputSkyfile entries were constructed with appropriate warp1/stack2 values. It is only when processing the diff does the issue come up. I think the solution is to create a sacrificial warp that will be the "input" to all ingested stacks.
    • I have added stackInputSkyfile entries for all i-band stacks that point at warp_id 60. For all of them. This lie should allow WS diffs to complete, as the camera will match.
    • Due to issues with ppSub and the way files are read, I have had to make a very hacky edit to ppSubCamera.c to allow these diffs to be generated. After loading the warp images (from GPC2), ppSub throws an error when it attempts to read the stack (from GPC1) because the camera in the pmConfig object does not match the camera determined from the stack header. This appears to be a fundamental limitation of the pmFPAfileDefine codebase, and I could not find an easy solution. Instead, upon failing to read the stack, I have the code (near ppSubCamera.c:215) substitute the string GPC1 for GPC2 in the config->cameraName string. This allows a second attempt to read the stack (and all subsequent stack products) to succeed, as the header matches the config expectation. Once all stack products have been loaded, the config->cameraName string is set back to the original GPC2 value, and the difference processing continues as normal. This generates images that appear reasonable, but this edit is not a good long-term solution. The substitution is hardcoded, and ignores the fundamental issue that skycell products should be treated equally by the code, regardless of camera.