Index: branches/czw_branch/20170908/ippToPsps/config/latextables/README =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/README (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/README (revision 40483) @@ -0,0 +1,9 @@ +Instructions to parse/check stuff between ipptopsps and the paper + +copy main.tex from sharelatex here + +run 'perl parseipptopspsallflags.pl', this parses tables.IN.vot as well as main.tex into different txt files + +(ipptopsps.DetectionFlags.txt and latex.DetectionFlags.txt, for example) + +do diffs on the 8 different tables: are they as we expect? (note: some have funny whitespace characters i've been unable to diagnose). Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags.txt (revision 40483) @@ -0,0 +1,33 @@ +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||PSFMODEL||0x00000001||1||Source fitted with a psf model (linear or non-linear).|| +||EXTMODEL||0x00000002||2||Source fitted with an extended-source model.|| +||FITTED||0x00000004||4||Source fitted with non-linear model (PSF or EXT; good or bad).|| +||FAIL||0x00000008||8||Fit (non-linear) failed (non-converge; off-edge; run to zero).|| +||POOR||0x00000010||16||Fit succeeds, but low-S/N, high-Chisq, or large (for PSF -- drop?).|| +||PAIR||0x00000020||32||Source fitted with a double PSF.|| +||PSFSTAR||0x00000040||64||Source used to define PSF model.|| +||SATSTAR||0x00000080||128||Source model peak is above saturation.|| +||BLEND||0x00000100||256||Source is a blend with other sources.|| +||EXTERNAL||0x00000200||512||Source based on supplied input position.|| +||BADPSF||0x00000400||1024||Failed to get good estimate of object's PSF.|| +||DEFECT||0x00000800||2048||Source is thought to be a defect.|| +||SATURATED||0x00001000||4096||Source is thought to be saturated pixels (bleed trail).|| +||CR_LIMIT||0x00002000||8192||Source has crNsigma above limit.|| +||EXT_LIMIT||0x00004000||16384||Source has extNsigma above limit.|| +||MOMENTS_FAILURE||0x00008000||32768||Could not measure the moments.|| +||SKY_FAILURE||0x00010000||65536||Could not measure the local sky.|| +||SKYVAR_FAILURE||0x00020000||131072||Could not measure the local sky variance.|| +||BELOW_MOMENTS_SN||0x00040000||262144||Moments not measured due to low S/N.|| +||UNDEF_1||0x00080000||524288||Unused bit value.|| +||BIG_RADIUS||0x00100000||1048576||Poor moments for small radius, try large radius.|| +||AP_MAGS||0x00200000||2097152||Source has an aperture magnitude.|| +||BLEND_FIT||0x00400000||4194304||Source was fitted as a blend.|| +||EXTENDED_FIT||0x00800000||8388608||Full extended fit was used.|| +||EXTENDED_STATS||0x01000000||16777216||Extended aperture stats calculated.|| +||LINEAR_FIT||0x02000000||33554432||Source fitted with the linear fit.|| +||NONLINEAR_FIT||0x04000000||67108864||Source fitted with the non-linear fit.|| +||RADIAL_FLUX||0x08000000||134217728||Radial flux measurements calculated.|| +||SIZE_SKIPPED||0x10000000||268435456||Size could not be determined.|| +||PEAK_ON_SPIKE||0x20000000||536870912||Peak lands on diffraction spike.|| +||PEAK_ON_GHOST||0x40000000||1073741824||Peak lands on ghost or glint.|| +||PEAK_OFF_CHIP||0x80000000||2147483648||Peak lands off edge of chip.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags2.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags2.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags2.txt (revision 40483) @@ -0,0 +1,24 @@ +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||DIFF_WITH_SINGLE||0x00000001||1||Difference source matched to a single positive detection.|| +||DIFF_WITH_DOUBLE||0x00000002||2||Difference source matched to positive detections in both images.|| +||MATCHED||0x00000004||4||Source generated based on another image (forced photometry at source location).|| +||ON_SPIKE||0x00000008||8||More than 25% of (PSF-weighted) pixels land on diffraction spike.|| +||ON_STARCORE||0x00000010||16||More than 25% of (PSF-weighted) pixels land on starcore.|| +||ON_BURNTOOL||0x00000020||32||More than 25% of (PSF-weighted) pixels land on burntool.|| +||ON_CONVPOOR||0x00000040||64||More than 25% of (PSF-weighted) pixels land on convpoor.|| +||PASS1_SRC||0x00000080||128||Source detected in first pass analysis|| +||HAS_BRIGHTER_NEIGHBOR||0x00000100||256||Peak is not the brightest in its footprint|| +||BRIGHT_NEIGHBOR_1||0x00000200||512||Flux_negative / (r^2 flux_positive) > 1.|| +||BRIGHT_NEIGHBOR_10||0x00000400||1024||Flux_negative / (r^2 flux_positive) > 10.|| +||DIFF_SELF_MATCH||0x00000800||2048||Positive detection match is probably this source.|| +||SATSTAR_PROFILE||0x00001000||4096||Saturated source is modeled with a radial profile.|| +||ECONTOUR_FEW_PTS||0x00002000||8192||Too few points to measure the elliptical contour.|| +||RADBIN_NAN_CENTER||0x00004000||16384||Radial bins failed with too many NaN center bin.|| +||PETRO_NAN_CENTER||0x00008000||32768||Petrosian (1976) radial bins failed with too many NaN center bin.|| +||PETRO_NO_PROFILE||0x00010000||65536||Petrosian (1976) not built becaues radial bins missing.|| +||PETRO_INSIG_RATIO||0x00020000||131072||Insignificant measurement of Petrosian (1976) ratio.|| +||PETRO_RATIO_ZEROBIN||0x00040000||262144||Petrosian (1976) ratio in the 0th bin (likely bad).|| +||EXT_FITS_RUN||0x00080000||524288||Attempted to run extended fits on this source.|| +||EXT_FITS_FAIL||0x00100000||1048576||At least one of the model fits failed.|| +||EXT_FITS_RETRY||0x00200000||2097152||One of the model fits was retried with new window.|| +||EXT_FITS_NONE||0x00400000||4194304||All of the model fits failed.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags3.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags3.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.DetectionFlags3.txt (revision 40483) @@ -0,0 +1,30 @@ +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||NOCAL||0x00000001||1||Detection ignored for this analysis (photcode; time range) -- internal only.|| +||POOR_PHOTOM||0x00000002||2||Detection is photometry outlier.|| +||SKIP_PHOTOM||0x00000004||4||Detection was ignored for photometry measurement.|| +||AREA||0x00000008||8||Detection near image edge.|| +||POOR_ASTROM||0x00000010||16||Detection is astrometry outlier.|| +||SKIP_ASTROM||0x00000020||32||Detection was ignored for astrometry measurement.|| +||USED_OBJ||0x00000040||64||Detection was used during update objects|| +||USED_CHIP||0x00000080||128||Detection was used during update chips to measure astrometry with IRLS fit.|| +||BLEND_MEAS||0x00000100||256||Detection is within radius of multiple objects.|| +||BLEND_OBJ||0x00000200||512||Multiple detections within radius of object.|| +||WARP_USED||0x00000400||1024||Measurement used to find mean warp photometry.|| +||UNMASKED_ASTRO||0x00000800||2048||Detection was unmasked in update chips to determine astrometry parameter errors.|| +||BLEND_MEAS_X||0x00001000||4096||Detection is within radius of multiple objects across catalogs.|| +||ARTIFACT||0x00002000||8192||Detection is thought to be non-astronomical.|| +||SYNTH_MAG||0x00004000||16384||Magnitude is synthetic.|| +||PHOTOM_UBERCAL||0x00008000||32768||Externally-supplied zero point from ubercal analysis.|| +||STACK_PRIMARY||0x00010000||65536||This stack measurement is in the primary skycell.|| +||STACK_PHOT_SRC||0x00020000||131072||This measurement supplied the stack photometry.|| +||ICRF_QSO||0x00040000||262144||This measurement is an ICRF reference position.|| +||IMAGE_EPOCH||0x00080000||524288||This measurement is registered to the image epoch (not tied to the reference catalog epoch).|| +||PHOTOM_PSF||0x00100000||1048576||This measurement is used for the mean PSF magnitude.|| +||PHOTOM_APER||0x00200000||2097152||This measurement is used for the mean aperture magnitude.|| +||PHOTOM_KRON||0x00400000||4194304||This measurement is used for the mean Kron (1980) magnitude.|| +||MASKED_PSF||0x01000000||16777216||This measurement is masked based on IRLS weights for the mean PSF magnitude.|| +||MASKED_APER||0x02000000||33554432||This measurement is masked based on IRLS weights for the mean aperture magnitude.|| +||MASKED_KRON||0x04000000||67108864||This measurement is masked based on IRLS weights for the mean Kron (1980) magnitude.|| +||OBJECT_HAS_2MASS||0x10000000||268435456||This measurement comes from an object with 2MASS data.|| +||OBJECT_HAS_GAIA||0x20000000||536870912||This measurement comes from an object with Gaia data.|| +||OBJECT_HAS_TYCHO||0x40000000||1073741824||This measurement comes from an object with Tycho data.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ForcedGalaxyShapeFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ForcedGalaxyShapeFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ForcedGalaxyShapeFlags.txt (revision 40483) @@ -0,0 +1,5 @@ +||NO_ERROR||0x00000000||0||No error condition raised.|| +||FAIL_FIT||0x00000001||1||Fit failed to converge or was degenerate|| +||TOO_FEW||0x00000002||2||Not enough points to fit the model|| +||OUT_OF_RANGE||0x00000004||4||Fit minimum too far outside data range|| +||BAD_ERROR||0x00000008||8||Invalid size error (nan or inf)|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ImageFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ImageFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ImageFlags.txt (revision 40483) @@ -0,0 +1,12 @@ +||NEW||0x00000000||0||No relphot / relastro attempted.|| +||PHOTOM_NOCAL||0x00000001||1||Used within relphot to mean 'don't apply fit'.|| +||PHOTOM_POOR||0x00000002||2||Relphot says image is bad (dMcal > limit).|| +||PHOTOM_SKIP||0x00000004||4||External information image has bad photometry.|| +||PHOTOM_FEW||0x00000008||8||Currently too few measurements for good value for photometry.|| +||ASTROM_NOCAL||0x00000010||16||User-set value used within relastro: ignore.|| +||ASTROM_POOR||0x00000020||32||Relastro says image is bad (dR,dD > limit).|| +||ASTROM_FAIL||0x00000040||64||Relastro fit diverged, fit not applied.|| +||ASTROM_SKIP||0x00000080||128||External information image has bad astrometry.|| +||ASTROM_FEW||0x00000100||256||Currently too few measurements for good value for astrometry.|| +||PHOTOM_UBERCAL||0x00000200||512||Externally-supplied photometry zero point from ubercal analysis.|| +||ASTROM_GMM||0x00000400||1024||Image was fitted to positions corrected by the galaxy motion model.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectFilterFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectFilterFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectFilterFlags.txt (revision 40483) @@ -0,0 +1,22 @@ +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||SECF_STAR_FEW||0x00000001||1||Used within relphot: skip star.|| +||SECF_STAR_POOR||0x00000002||2||Used within relphot: skip star.|| +||SECF_USE_SYNTH||0x00000004||4||Synthetic photometry used in average measurement.|| +||SECF_USE_UBERCAL||0x00000008||8||Ubercal photometry used in average measurement.|| +||SECF_HAS_PS1||0x00000010||16||PS1 photometry used in average measurement.|| +||SECF_HAS_PS1_STACK||0x00000020||32||PS1 stack photometry exists.|| +||SECF_HAS_TYCHO||0x00000040||64||Tycho photometry used for synthetic magnitudes.|| +||SECF_FIX_SYNTH||0x00000080||128||Synthetic magnitudes repaired with zeropoint map.|| +||SECF_RANK_0||0x00000100||256||Average magnitude calculated in 0th pass.|| +||SECF_RANK_1||0x00000200||512||Average magnitude calculated in 1st pass.|| +||SECF_RANK_2||0x00000400||1024||Average magnitude calculated in 2nd pass.|| +||SECF_RANK_3||0x00000800||2048||Average magnitude calculated in 3rd pass.|| +||SECF_RANK_4||0x00001000||4096||Average magnitude calculated in 4th pass.|| +||SECF_STACK_PRIMARY||0x00004000||16384||PS1 stack photometry comes from primary skycell.|| +||SECF_STACK_BESTDET||0x00008000||32768||PS1 stack best measurement is a detection (not forced).|| +||SECF_STACK_PRIMDET||0x00010000||65536||PS1 stack primary measurement is a detection (not forced).|| +||SECF_HAS_SDSS||0x00100000||1048576||This photcode has SDSS photometry.|| +||SECF_HAS_HSC||0x00200000||2097152||This photcode has HSC photometry.|| +||SECF_HAS_CFH||0x00400000||4194304||This photcode has CFH photometry (mostly Megacam).|| +||SECF_HAS_DES||0x00800000||8388608||This photcode has DES photometry.|| +||SECF_OBJ_EXT||0x01000000||16777216||Extended in this band.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectInfoFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectInfoFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectInfoFlags.txt (revision 40483) @@ -0,0 +1,32 @@ +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||FEW||0x00000001||1||Used within relphot; skip star.|| +||POOR||0x00000002||2||Used within relphot; skip star.|| +||ICRF_QSO||0x00000004||4||Object IDed with known ICRF quasar (may have ICRF position measurement)|| +||HERN_QSO_P60||0x00000008||8||Identified as likely QSO (Hernitschek et al 2015), P_QSO >= 0.60|| +||HERN_QSO_P05||0x00000010||16||Identified as possible QSO (Hernitschek et al 2015), P_QSO >= 0.05|| +||HERN_RRL_P60||0x00000020||32||Identified as likely RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.60|| +||HERN_RRL_P05||0x00000040||64||Identified as possible RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.05|| +||HERN_VARIABLE||0x00000080||128||Identified as a variable based on ChiSq? (Hernitschek et al 2015)|| +||TRANSIENT||0x00000100||256||Identified as a non-periodic (stationary) transient|| +||HAS_SOLSYS_DET||0x00000200||512||At least one detection identified with a known solar-system object (asteroid or other).|| +||MOST_SOLSYS_DET||0x00000400||1024||Most detections identified with a known solar-system object (asteroid or other).|| +||LARGE_PM||0x00000800||2048||Star with large proper motion|| +||RAW_AVE||0x00001000||4096||Simple weighted average position was used (no IRLS fitting)|| +||FIT_AVE||0x00002000||8192||Average position was fitted|| +||FIT_PM||0x00004000||16384||Proper motion model was fitted|| +||FIT_PAR||0x00008000||32768||Parallax model was fitted|| +||USE_AVE||0x00010000||65536||Average position used (not PM or PAR)|| +||USE_PM||0x00020000||131072||Proper motion used (not AVE or PAR)|| +||USE_PAR||0x00040000||262144||Parallax used (not AVE or PM)|| +||NO_MEAN_ASTROM||0x00080000||524288||Mean astrometry could not be measured|| +||STACK_FOR_MEAN||0x00100000||1048576||Stack position used for mean astrometry|| +||MEAN_FOR_STACK||0x00200000||2097152||Mean astrometry used for stack position|| +||BAD_PM||0x00400000||4194304||Failure to measure proper-motion model|| +||EXT||0x00800000||8388608||Extended in our data (eg; PS)|| +||EXT_ALT||0x01000000||16777216||Extended in external data (eg; 2MASS)|| +||GOOD||0x02000000||33554432||Good-quality measurement in our data (eg; PS)|| +||GOOD_ALT||0x04000000||67108864||Good-quality measurement in external data (eg; 2MASS)|| +||GOOD_STACK||0x08000000||134217728||Good-quality object in the stack (> 1 good stack measurement)|| +||BEST_STACK||0x10000000||268435456||The primary stack measurements are the best measurements|| +||SUSPECT_STACK||0x20000000||536870912||Suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement)|| +||BAD_STACK||0x40000000||1073741824||Poor-quality stack object (no more than 1 good or suspect measurement).|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectQualityFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectQualityFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/ipptopsps.ObjectQualityFlags.txt (revision 40483) @@ -0,0 +1,9 @@ +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||QF_OBJ_EXT||0x00000001||1||Extended in our data (eg; PS).|| +||QF_OBJ_EXT_ALT||0x00000002||2||Extended in external data (eg; 2MASS).|| +||QF_OBJ_GOOD||0x00000004||4||Good-quality measurement in our data (eg; PS).|| +||QF_OBJ_GOOD_ALT||0x00000008||8||Good-quality measurement in external data (eg; 2MASS).|| +||QF_OBJ_GOOD_STACK||0x00000010||16||Good-quality object in the stack (> 1 good stack measurement).|| +||QF_OBJ_BEST_STACK||0x00000020||32||The primary stack measurements are the best measurements.|| +||QF_OBJ_SUSPECT_STACK||0x00000040||64||Suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement).|| +||QF_OBJ_BAD_STACK||0x00000080||128||Poor-quality stack object (no more than 1 good or suspect measurement).|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags.txt (revision 40483) @@ -0,0 +1,34 @@ +||Name||Hexadecimal||Value||Description|| +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||PSFMODEL||0x00000001||1||Source fitted with a psf model (linear or non-linear).|| +||EXTMODEL||0x00000002||2||Source fitted with an extended-source model.|| +||FITTED||0x00000004||4||Source fitted with non-linear model (PSF or EXT; good or bad).|| +||FAIL||0x00000008||8||Fit (non-linear) failed (non-converge; off-edge; run to zero).|| +||POOR||0x00000010||16||Fit succeeds; but low-S/N; high-Chisq; or large (for PSF -- drop?).|| +||PAIR||0x00000020||32||Source fitted with a double PSF.|| +||PSFSTAR||0x00000040||64||Source used to define PSF model.|| +||SATSTAR||0x00000080||128||Source model peak is above saturation.|| +||BLEND||0x00000100||256||Source is a blend with other sources.|| +||EXTERNAL||0x00000200||512||Source based on supplied input position.|| +||BADPSF||0x00000400||1024||Failed to get good estimate of object's PSF.|| +||DEFECT||0x00000800||2048||Source is thought to be a defect.|| +||SATURATED||0x00001000||4096||Source is thought to be saturated pixels (bleed trail).|| +||CR_LIMIT||0x00002000||8192||Source has crNsigma above limit.|| +||EXT_LIMIT||0x00004000||16384||Source has extNsigma above limit.|| +||MOMENTS_FAILURE||0x00008000||32768||Could not measure the moments.|| +||SKY_FAILURE||0x00010000||65536||Could not measure the local sky.|| +||SKYVAR_FAILURE||0x00020000||131072||Could not measure the local sky variance.|| +||BELOW_MOMENTS_SN||0x00040000||262144||Moments not measured due to low S/N.|| +||UNDEF_1||0x00080000||524288||Unused bit value.|| +||BIG_RADIUS||0x00100000||1048576||Poor moments for small radius; try large radius.|| +||AP_MAGS||0x00200000||2097152||Source has an aperture magnitude.|| +||BLEND_FIT||0x00400000||4194304||Source was fitted as a blend.|| +||EXTENDED_FIT||0x00800000||8388608||Full extended fit was used.|| +||EXTENDED_STATS||0x01000000||16777216||Extended aperture stats calculated.|| +||LINEAR_FIT||0x02000000||33554432||Source fitted with the linear fit.|| +||NONLINEAR_FIT||0x04000000||67108864||Source fitted with the non-linear fit.|| +||RADIAL_FLUX||0x08000000||134217728||Radial flux measurements calculated.|| +||SIZE_SKIPPED||0x10000000||268435456||Size could not be determined.|| +||PEAK_ON_SPIKE||0x20000000||536870912||Peak lands on diffraction spike.|| +||PEAK_ON_GHOST||0x40000000||1073741824||Peak lands on ghost or glint.|| +||PEAK_OFF_CHIP||0x80000000||2147483648||Peak lands off edge of chip.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags2.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags2.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags2.txt (revision 40483) @@ -0,0 +1,25 @@ +||Name||Hexadecimal||Value||Description|| +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||DIFF_WITH_SINGLE||0x00000001||1||Difference source matched to a single positive detection.|| +||DIFF_WITH_DOUBLE||0x00000002||2||Difference source matched to positive detections in both images.|| +||MATCHED||0x00000004||4||Source generated based on another image (forced photometry at source location).|| +||ON_SPIKE||0x00000008||8||More than 25% of (PSF-weighted) pixels land on diffraction spike.|| +||ON_STARCORE||0x00000010||16||More than 25% of (PSF-weighted) pixels land on starcore.|| +||ON_BURNTOOL||0x00000020||32||More than 25% of (PSF-weighted) pixels land on burntool.|| +||ON_CONVPOOR||0x00000040||64||More than 25% of (PSF-weighted) pixels land on convpoor.|| +||PASS1_SRC||0x00000080||128||Source detected in first pass analysis|| +||HAS_BRIGHTER_NEIGHBOR||0x00000100||256||Peak is not the brightest in its footprint|| +||BRIGHT_NEIGHBOR_1||0x00000200||512||Flux_negative / (r^2 flux_positive) > 1.|| +||BRIGHT_NEIGHBOR_10||0x00000400||1024||Flux_negative / (r^2 flux_positive) > 10.|| +||DIFF_SELF_MATCH||0x00000800||2048||Positive detection match is probably this source.|| +||SATSTAR_PROFILE||0x00001000||4096||Saturated source is modeled with a radial profile.|| +||ECONTOUR_FEW_PTS||0x00002000||8192||Too few points to measure the elliptical contour.|| +||RADBIN_NAN_CENTER||0x00004000||16384||Radial bins failed with too many NaN center bin.|| +||PETRO_NAN_CENTER||0x00008000||32768||Petrosian (1976) radial bins failed with too many NaN center bin.|| +||PETRO_NO_PROFILE||0x00010000||65536||Petrosian (1976) not built becaues radial bins missing.|| +||PETRO_INSIG_RATIO||0x00020000||131072||Insignificant measurement of Petrosian (1976) ratio.|| +||PETRO_RATIO_ZEROBIN||0x00040000||262144||Petrosian (1976) ratio in the 0th bin (likely bad).|| +||EXT_FITS_RUN||0x00080000||524288||Attempted to run extended fits on this source.|| +||EXT_FITS_FAIL||0x00100000||1048576||At least one of the model fits failed.|| +||EXT_FITS_RETRY||0x00200000||2097152||One of the model fits was retried with new window.|| +||EXT_FITS_NONE||0x00400000||4194304||All of the model fits failed.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags3.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags3.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.DetectionFlags3.txt (revision 40483) @@ -0,0 +1,32 @@ +||Name||Hexadecimal||Value||Description|| +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||NOCAL||0x00000001||1||Detection ignored for this analysis (photcode; time range) -- internal only.|| +||POOR_PHOTOM||0x00000002||2||Detection is photometry outlier.|| +||SKIP_PHOTOM||0x00000004||4||Detection was ignored for photometry measurement.|| +||AREA||0x00000008||8||Detection near image edge.|| +||POOR_ASTROM||0x00000010||16||Detection is astrometry outlier.|| +||SKIP_ASTROM||0x00000020||32||Detection was ignored for astrometry measurement.|| +||USED_OBJ||0x00000040||64||Detection was used during update objects|| +||USED_CHIP||0x00000080||128||Detection was used during update chips to measure astrometry with IRLS fit.|| +||BLEND_MEAS||0x00000100||256||Detection is within radius of multiple objects.|| +||BLEND_OBJ||0x00000200||512||Multiple detections within radius of object.|| +||WARP_USED||0x00000400||1024||Measurement used to find mean warp photometry.|| +||UNMASKED_ASTRO||0x00000800||2048||Detection was unmasked in update chips to determine astrometry parameter errors.|| +||BLEND_MEAS_X||0x00001000||4096||Detection is within radius of multiple objects across catalogs.|| +||ARTIFACT||0x00002000||8192||Detection is thought to be non-astronomical.|| +||SYNTH_MAG||0x00004000||16384||Magnitude is synthetic.|| +||PHOTOM_UBERCAL||0x00008000||32768||Externally-supplied zero point from ubercal analysis.|| +||STACK_PRIMARY||0x00010000||65536||This stack measurement is in the primary skycell.|| +||STACK_PHOT_SRC||0x00020000||131072||This measurement supplied the stack photometry.|| +||ICRF_QSO||0x00040000||262144||This measurement is an ICRF reference position.|| +||IMAGE_EPOCH||0x00080000||524288||This measurement is registered to the image epoch|| +||||||||(not tied to the reference catalog epoch).|| +||PHOTOM_PSF||0x00100000||1048576||This measurement is used for the mean PSF magnitude.|| +||PHOTOM_APER||0x00200000||2097152||This measurement is used for the mean aperture magnitude.|| +||PHOTOM_KRON||0x00400000||4194304||This measurement is used for the mean Kron (1980) magnitude.|| +||MASKED_PSF||0x01000000||16777216||This measurement is masked based on IRLS weights for the mean PSF magnitude.|| +||MASKED_APER||0x02000000||33554432||This measurement is masked based on IRLS weights for the mean aperture magnitude.|| +||MASKED_KRON||0x04000000||67108864||This measurement is masked based on IRLS weights for the mean Kron (1980) magnitude.|| +||OBJECT_HAS_2MASS||0x10000000||268435456||This measurement comes from an object with 2MASS data.|| +||OBJECT_HAS_GAIA||0x20000000||536870912||This measurement comes from an object with Gaia data.|| +||OBJECT_HAS_TYCHO||0x40000000||1073741824||This measurement comes from an object with Tycho data.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ImageFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ImageFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ImageFlags.txt (revision 40483) @@ -0,0 +1,13 @@ +||Name||Hexadecimal||Value||Description|| +||NEW||0x00000000||0||No relphot / relastro attempted.|| +||PHOTOM_NOCAL||0x00000001||1||Used within relphot to mean 'don't apply fit'.|| +||PHOTOM_POOR||0x00000002||2||Relphot says image is bad (dMcal > limit).|| +||PHOTOM_SKIP||0x00000004||4||External information image has bad photometry.|| +||PHOTOM_FEW||0x00000008||8||Currently too few measurements for good value for photometry.|| +||ASTROM_NOCAL||0x00000010||16||User-set value used within relastro: ignore.|| +||ASTROM_POOR||0x00000020||32||Relastro says image is bad (dR,dD > limit).|| +||ASTROM_FAIL||0x00000040||64||Relastro fit diverged, fit not applied.|| +||ASTROM_SKIP||0x00000080||128||External information image has bad astrometry.|| +||ASTROM_FEW||0x00000100||256||Currently too few measurements for good value for astrometry.|| +||PHOTOM_UBERCAL||0x00000200||512||Externally-supplied photometry zero point from ubercal analysis.|| +||ASTROM_GMM||0x00000400||1024||Image was fitted to positions corrected by the galaxy motion model.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectFilterFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectFilterFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectFilterFlags.txt (revision 40483) @@ -0,0 +1,23 @@ +||Name||Hexadecimal||Value||Description|| +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||SECF_STAR_FEW||0x00000001||1||Used within relphot: skip star.|| +||SECF_STAR_POOR||0x00000002||2||Used within relphot: skip star.|| +||SECF_USE_SYNTH||0x00000004||4||Synthetic photometry used in average measurement.|| +||SECF_USE_UBERCAL||0x00000008||8||Ubercal photometry used in average measurement.|| +||SECF_HAS_PS1||0x00000010||16||PS1 photometry used in average measurement.|| +||SECF_HAS_PS1_STACK||0x00000020||32||PS1 stack photometry exists.|| +||SECF_HAS_TYCHO||0x00000040||64||Tycho photometry used for synthetic magnitudes.|| +||SECF_FIX_SYNTH||0x00000080||128||Synthetic magnitudes repaired with zeropoint map.|| +||SECF_RANK_0||0x00000100||256||Average magnitude calculated in 0th pass.|| +||SECF_RANK_1||0x00000200||512||Average magnitude calculated in 1st pass.|| +||SECF_RANK_2||0x00000400||1024||Average magnitude calculated in 2nd pass.|| +||SECF_RANK_3||0x00000800||2048||Average magnitude calculated in 3rd pass.|| +||SECF_RANK_4||0x00001000||4096||Average magnitude calculated in 4th pass.|| +||SECF_STACK_PRIMARY||0x00004000||16384||PS1 stack photometry comes from primary skycell.|| +||SECF_STACK_BESTDET||0x00008000||32768||PS1 stack best measurement is a detection (not forced).|| +||SECF_STACK_PRIMDET||0x00010000||65536||PS1 stack primary measurement is a detection (not forced).|| +||SECF_HAS_SDSS||0x00100000||1048576||This photcode has SDSS photometry.|| +||SECF_HAS_HSC||0x00200000||2097152||This photcode has HSC photometry.|| +||SECF_HAS_CFH||0x00400000||4194304||This photcode has CFH photometry (mostly megacam).|| +||SECF_HAS_DES||0x00800000||8388608||This photcode has DES photometry.|| +||SECF_OBJ_EXT||0x01000000||16777216||Extended in this band.|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectInfoFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectInfoFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectInfoFlags.txt (revision 40483) @@ -0,0 +1,36 @@ +||Name||Hexadecimal||Value||Description|| +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||FEW||0x00000001||1||Used within relphot; skip star.|| +||POOR||0x00000002||2||Used within relphot; skip star.|| +||ICRF_QSO||0x00000004||4||Object IDed with known ICRF quasar|| +||||||||(may have ICRF position measurement)|| +||HERN_QSO_P60||0x00000008||8||Identified as likely QSO (Hernitschek et al 2015);|| +||||||||P_QSO >= 0.60|| +||HERN_QSO_P05||0x00000010||16||Identified as possible QSO (Hernitschek et al 2015), P_QSO >= 0.05|| +||HERN_RRL_P60||0x00000020||32||Identified as likely RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.60|| +||HERN_RRL_P05||0x00000040||64||Identified as possible RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.05|| +||HERN_VARIABLE||0x00000080||128||Identified as a variable based on ChiSq? (Hernitschek et al 2015)|| +||TRANSIENT||0x00000100||256||Identified as a non-periodic (stationary) transient|| +||HAS_SOLSYS_DET||0x00000200||512||At least one detection identified with a known solar-system object (asteroid or other).|| +||MOST_SOLSYS_DET||0x00000400||1024||Most detections identified with a known solar-system object (asteroid or other).|| +||LARGE_PM||0x00000800||2048||Star with large proper motion|| +||RAW_AVE||0x00001000||4096||Simple weighted average position was used (no IRLS fitting)|| +||FIT_AVE||0x00002000||8192||Average position was fitted|| +||FIT_PM||0x00004000||16384||Proper motion model was fitted|| +||FIT_PAR||0x00008000||32768||Parallax model was fitted|| +||USE_AVE||0x00010000||65536||Average position used (not PM or PAR)|| +||USE_PM||0x00020000||131072||Proper motion used (not AVE or PAR)|| +||USE_PAR||0x00040000||262144||Parallax used (not AVE or PM)|| +||NO_MEAN_ASTROM||0x00080000||524288||Mean astrometry could not be measured|| +||STACK_FOR_MEAN||0x00100000||1048576||Stack position used for mean astrometry|| +||MEAN_FOR_STACK||0x00200000||2097152||Mean astrometry used for stack position|| +||BAD_PM||0x00400000||4194304||Failure to measure proper-motion model|| +||EXT||0x00800000||8388608||Extended in our data (eg; PS)|| +||EXT_ALT||0x01000000||16777216||Extended in external data (eg; 2MASS)|| +||GOOD||0x02000000||33554432||Good-quality measurement in our data (eg; PS)|| +||GOOD_ALT||0x04000000||67108864||Good-quality measurement in external data (eg; 2MASS)|| +||GOOD_STACK||0x08000000||134217728||Good-quality object in the stack (> 1 good stack measurement)|| +||BEST_STACK||0x10000000||268435456||The primary stack measurements are the best measurements|| +||SUSPECT_STACK||0x20000000||536870912||Suspect object in the stack|| +||||||||(no more than 1 good measurement, 2 or more suspect or good stack measurement)|| +||BAD_STACK||0x40000000||1073741824||Poor-quality stack object (no more than 1 good or suspect measurement)|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectQualtiyFlags.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectQualtiyFlags.txt (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/latex.ObjectQualtiyFlags.txt (revision 40483) @@ -0,0 +1,10 @@ +||Name||Hexadecimal||Value||Description|| +||DEFAULT||0x00000000||0||Initial value; resets all bits.|| +||QF_OBJ_EXT||0x00000001||1||Extended in our data (eg; PS).|| +||QF_OBJ_EXT_ALT||0x00000002||2||Extended in external data (eg; 2MASS).|| +||QF_OBJ_GOOD||0x00000004||4||Good-quality measurement in our data (eg; PS).|| +||QF_OBJ_GOOD_ALT||0x00000008||8||Good-quality measurement in external data (eg; 2MASS).|| +||QF_OBJ_GOOD_STACK||0x00000010||16||Good-quality object in the stack (> 1 good stack measurement).|| +||QF_OBJ_BEST_STACK||0x00000020||32||The primary stack measurements are the best measurements.|| +||QF_OBJ_SUSPECT_STACK||0x00000040||64||Suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement).|| +||QF_OBJ_BAD_STACK||0x00000080||128||Poor-quality stack object (no more than 1 good or suspect measurement).|| Index: branches/czw_branch/20170908/ippToPsps/config/latextables/parse_tableIN_sub.pl =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/parse_tableIN_sub.pl (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/parse_tableIN_sub.pl (revision 40483) @@ -0,0 +1,45 @@ +#$table = 'ObjectInfoFlags'; +$table='ObjectQualityFlags'; +$table=$ARGV[1]; +#$table = 'ImageFlags'; +# +#
+#
+#
+#
+ +#
+ +$go = 0; +while (<>) { + + + + +if ($_ =~ /\/|/g; + $_ =~s /\s+\/|/g; + $_ =~s /\/|/g; + $_ =~s /\<\/TR\>/|/g; + $_ =~s /\s+\<\/TD\>/|/g; + $_ =~s /\<\/TD\>/|/g; + $_ =~s /^\s+//; + + print $_; + } + } +} + +} Index: branches/czw_branch/20170908/ippToPsps/config/latextables/parseipptopspsallflags.pl =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/parseipptopspsallflags.pl (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/parseipptopspsallflags.pl (revision 40483) @@ -0,0 +1,19 @@ +#`rm ipptopsps.*lags.txt`; +`perl parse_tableIN_sub.pl ../tables.IN.vot ObjectInfoFlags > ipptopsps.ObjectInfoFlags.txt`; + `perl parse_tableIN_sub.pl ../tables.IN.vot ObjectQualityFlags > ipptopsps.ObjectQualityFlags.txt`; + `perl parse_tableIN_sub.pl ../tables.IN.vot ObjectFilterFlags > ipptopsps.ObjectFilterFlags.txt`; + `perl parse_tableIN_sub.pl ../tables.IN.vot DetectionFlags > ipptopsps.DetectionFlags.txt`; + + `perl parse_tableIN_sub.pl ../tables.IN.vot DetectionFlags2 > ipptopsps.DetectionFlags2.txt`; + `perl parse_tableIN_sub.pl ../tables.IN.vot DetectionFlags3 > ipptopsps.DetectionFlags3.txt`; + `perl parse_tableIN_sub.pl ../tables.IN.vot ImageFlags > ipptopsps.ImageFlags.txt`; + `perl parse_tableIN_sub.pl ../tables.IN.vot ForcedGalaxyShapeFlags > ipptopsps.ForcedGalaxyShapeFlags.txt`; + +`perl parselatex.pl main.tex ObjectInfoFlags > latex.ObjectInfoFlags.txt`; +`perl parselatex.pl main.tex ObjectQualityFlags > latex.ObjectQualtiyFlags.txt`; +`perl parselatex.pl main.tex ObjectFilterFlags > latex.ObjectFilterFlags.txt`; +`perl parselatex.pl main.tex DetectionFlags > latex.DetectionFlags.txt`; +`perl parselatex.pl main.tex DetectionFlags2 > latex.DetectionFlags2.txt`; +`perl parselatex.pl main.tex DetectionFlags3 > latex.DetectionFlags3.txt`; +`perl parselatex.pl main.tex ImageFlags > latex.ImageFlags.txt`; +`perl parselatex.pl main.tex ForceGalaxyShapeFlags > latex.ForcedGalaxyShapeFlags.txt`; Index: branches/czw_branch/20170908/ippToPsps/config/latextables/parselatex.pl =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/latextables/parselatex.pl (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/config/latextables/parselatex.pl (revision 40483) @@ -0,0 +1,28 @@ +$table = $ARGV[1]; +$go = 0; +while (<>) { + if ($_ =~ /caption/) { + if ($_ =~ /$table\}/) { + $go = 1; + } else { + $go = 0; + } + + } + if ($go==1) { + if ($_ =~ /\&/) { + #this is a good line (?) +# print $_; + chomp $_; + $_ =~ s/\\//g; + $_ =~ s/\s+&/&/g; + $_ =~ s/&\s+/&/g; + $_ =~ s/\{em //g; + $_ =~ s/\}//g; + $_ =~ s/\$//g; + my @split = split ('&', $_); + $split[3] =~ s/\s+$//; + print "||$split[0]||$split[1]||$split[2]||$split[3]||\n"; + } + } +} Index: branches/czw_branch/20170908/ippToPsps/config/tables.DO.vot =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/tables.DO.vot (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/config/tables.DO.vot (revision 40483) @@ -7,8 +7,8 @@ - IAU name for this object. + IAU-approved name for this object. - - Alternate Pan-STARRS name for this object. + + Alternate Sexigesimal name for this object. Index: branches/czw_branch/20170908/ippToPsps/config/tables.IN.vot =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/tables.IN.vot (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/config/tables.IN.vot (revision 40483) @@ -576,12 +576,12 @@ - - - + + + - + - + @@ -614,5 +614,5 @@ - + @@ -669,5 +669,5 @@ - + @@ -715,5 +715,5 @@ - + @@ -770,33 +770,33 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -822,28 +822,29 @@ - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + @@ -869,12 +870,12 @@ - - - - - - - - + + + + + + + + @@ -900,8 +901,8 @@ - - - - + + + + Index: branches/czw_branch/20170908/ippToPsps/config/tables.OB.vot =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/tables.OB.vot (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/config/tables.OB.vot (revision 40483) @@ -9,8 +9,8 @@ - IAU name for this object. - - - Alternate Pan-STARRS name for this object. + IAU-approved name for this object. + + + Alternate Sexigesimal name for this object. Index: branches/czw_branch/20170908/ippToPsps/config/tables.ST.vot =================================================================== --- branches/czw_branch/20170908/ippToPsps/config/tables.ST.vot (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/config/tables.ST.vot (revision 40483) @@ -93,5 +93,8 @@ - Locally derived photometric zero point for this stack. + Locally derived photometric zero point for this stack (PSF-like measurements only). + + + Locally derived photometric zero point for this stack (Aperture-like measurements only). @@ -231,4 +234,7 @@ Information flag bitmask indicating details of the g filter stack photometry. Values listed in DetectionFlags3. + + Information flag bitmask indicating details of the g filter stack photometry. Values listed in ObjectFilterFlags. + Number of input frames/exposures contributing to the g filter stack detection. @@ -287,4 +293,7 @@ Information flag bitmask indicating details of the r filter stack photometry. Values listed in DetectionFlags3. + + Information flag bitmask indicating details of the r filter stack photometry. Values listed in ObjectFilterFlags. + Number of input frames/exposures contributing to the r filter stack detection. @@ -343,4 +352,7 @@ Information flag bitmask indicating details of the i filter stack photometry. Values listed in DetectionFlags3. + + Information flag bitmask indicating details of the i filter stack photometry. Values listed in ObjectFilterFlags. + Number of input frames/exposures contributing to the i filter stack detection. @@ -399,4 +411,7 @@ Information flag bitmask indicating details of the z filter stack photometry. Values listed in DetectionFlags3. + + Information flag bitmask indicating details of the z filter stack photometry. Values listed in ObjectFilterFlags. + Number of input frames/exposures contributing to the z filter stack detection. @@ -454,4 +469,7 @@ Information flag bitmask indicating details of the y filter stack photometry. Values listed in DetectionFlags3. + + + Information flag bitmask indicating details of the y filter stack photometry. Values listed in ObjectFilterFlags. @@ -463,5 +481,5 @@
NEW 0x000000000No relphot / relastro attempted.
PHOTOM_NOCAL 0x000000011Used within relphot to mean 'don't apply fit'. | user-set value used withing relphot: ignore (EAM!)
PHOTOM_POOR 0x000000022Relphot says image is bad (dMcal > limit). (EAM!)
PHOTOM_SKIP 0x000000044External information image has bad photometry (EAM).
PHOTOM_NOCAL 0x000000011Used within relphot to mean 'don't apply fit'.
PHOTOM_POOR 0x000000022Relphot says image is bad (dMcal > limit).
PHOTOM_SKIP 0x000000044External information image has bad photometry.
PHOTOM_FEW 0x000000088Currently too few measurements for good value for photometry.
ASTROM_NOCAL 0x0000001016User-set value used within relastro: ignore (EAM!).
ASTROM_NOCAL 0x0000001016User-set value used within relastro: ignore.
ASTROM_POOR 0x0000002032Relastro says image is bad (dR,dD > limit).
ASTROM_FAIL 0x0000004064Relastro fit diverged, fit not applied.
ASTROM_SKIP 0x00000080128External information image has bad astrometry. (EAM)
ASTROM_SKIP 0x00000080128External information image has bad astrometry.
ASTROM_FEW 0x00000100256Currently too few measurements for good value for astrometry.
PHOTOM_UBERCAL0x00000200512Externally-supplied photometry zero point from ubercal analysis.
EXTMODEL 0x000000022 Source fitted with an extended-source model.
FITTED 0x000000044 Source fitted with non-linear model (PSF or EXT; good or bad).
FAIL 0x000000088 Fit (non-linear) failed (non-converge, off-edge, run to zero).
FAIL 0x000000088 Fit (non-linear) failed (non-converge; off-edge; run to zero).
POOR 0x0000001016 Fit succeeds, but low-S/N, high-Chisq, or large (for PSF -- drop?).
PAIR 0x0000002032 Source fitted with a double PSF.
DIFF_WITH_SINGLE 0x000000011 Difference source matched to a single positive detection.
DIFF_WITH_DOUBLE 0x000000022 Difference source matched to positive detections in both images.
MATCHED 0x000000044 source generated based on another image (forced photometry at source location).
MATCHED 0x000000044 Source generated based on another image (forced photometry at source location).
ON_SPIKE 0x000000088 More than 25% of (PSF-weighted) pixels land on diffraction spike.
ON_STARCORE 0x0000001016 More than 25% of (PSF-weighted) pixels land on starcore.
DEFAULT 0x000000000 Initial value; resets all bits.
NOCAL 0x000000011 Detection ignored for this analysis (photcode, time range) -- internal only.
NOCAL 0x000000011 Detection ignored for this analysis (photcode; time range) -- internal only.
POOR_PHOTOM 0x000000022 Detection is photometry outlier.
SKIP_PHOTOM 0x000000044 Detection was ignored for photometry measurement.
FEW 0x000000011 Used within relphot; skip star.
POOR 0x000000022 Used within relphot; skip star.
ICRF_QSO 0x000000044 object IDed with known ICRF quasar (may have ICRF position measurement)
HERN_QSO_P60 0x000000088 identified as likely QSO (Hernitschek et al 2015), P_QSO >= 0.60
HERN_QSO_P05 0x0000001016 identified as possible QSO (Hernitschek et al 2015), P_QSO >= 0.05
HERN_RRL_P60 0x0000002032 identified as likely RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.60
HERN_RRL_P05 0x0000004064 identified as possible RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.05
HERN_VARIABLE 0x00000080128 identified as a variable based on ChiSq (Hernitschek et al 2015)
TRANSIENT 0x00000100256 identified as a non-periodic (stationary) transient
HAS_SOLSYS_DET 0x00000200512 at least one detection identified with a known solar-system object (asteroid or other).
MOST_SOLSYS_DET 0x000004001024 most detections identified with a known solar-system object (asteroid or other).
LARGE_PM 0x000008002048 star with large proper motion
RAW_AVE 0x000010004096 simple weighted average position was used (no IRLS fitting)
FIT_AVE 0x000020008192 average position was fitted
FIT_PM 0x0000400016384 proper motion model was fitted
FIT_PAR 0x0000800032768 parallax model was fitted
USE_AVE 0x0001000065536 average position used (not PM or PAR)
USE_PM 0x00020000131072 proper motion used (not AVE or PAR)
USE_PAR 0x00040000262144 parallax used (not AVE or PM)
NO_MEAN_ASTROM 0x00080000524288 mean astrometry could not be measured
STACK_FOR_MEAN 0x001000001048576 stack position used for mean astrometry
MEAN_FOR_STACK 0x002000002097152 mean astrometry used for stack position
BAD_PM 0x004000004194304 failure to measure proper-motion model
EXT 0x008000008388608 extended in our data (eg, PS)
EXT_ALT 0x0100000016777216 extended in external data (eg, 2MASS)
GOOD 0x0200000033554432 good-quality measurement in our data (eg,PS)
GOOD_ALT 0x0400000067108864 good-quality measurement in external data (eg, 2MASS)
GOOD_STACK 0x08000000134217728 good-quality object in the stack (> 1 good stack measurement)
BEST_STACK 0x10000000268435456 the primary stack measurements are the best measurements
SUSPECT_STACK 0x20000000536870912 suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement)
BAD_STACK 0x400000001073741824 poor-quality stack object (no more than 1 good or suspect measurement)
ICRF_QSO 0x000000044 Object IDed with known ICRF quasar (may have ICRF position measurement)
HERN_QSO_P60 0x000000088 Identified as likely QSO (Hernitschek et al 2015), P_QSO >= 0.60
HERN_QSO_P05 0x0000001016 Identified as possible QSO (Hernitschek et al 2015), P_QSO >= 0.05
HERN_RRL_P60 0x0000002032 Identified as likely RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.60
HERN_RRL_P05 0x0000004064 Identified as possible RR Lyra (Hernitschek et al 2015), P_RRLyra >= 0.05
HERN_VARIABLE 0x00000080128 Identified as a variable based on ChiSq? (Hernitschek et al 2015)
TRANSIENT 0x00000100256 Identified as a non-periodic (stationary) transient
HAS_SOLSYS_DET 0x00000200512 At least one detection identified with a known solar-system object (asteroid or other).
MOST_SOLSYS_DET 0x000004001024 Most detections identified with a known solar-system object (asteroid or other).
LARGE_PM 0x000008002048 Star with large proper motion
RAW_AVE 0x000010004096 Simple weighted average position was used (no IRLS fitting)
FIT_AVE 0x000020008192 Average position was fitted
FIT_PM 0x0000400016384 Proper motion model was fitted
FIT_PAR 0x0000800032768 Parallax model was fitted
USE_AVE 0x0001000065536 Average position used (not PM or PAR)
USE_PM 0x00020000131072 Proper motion used (not AVE or PAR)
USE_PAR 0x00040000262144 Parallax used (not AVE or PM)
NO_MEAN_ASTROM 0x00080000524288 Mean astrometry could not be measured
STACK_FOR_MEAN 0x001000001048576 Stack position used for mean astrometry
MEAN_FOR_STACK 0x002000002097152 Mean astrometry used for stack position
BAD_PM 0x004000004194304 Failure to measure proper-motion model
EXT 0x008000008388608 Extended in our data (eg; PS)
EXT_ALT 0x0100000016777216 Extended in external data (eg; 2MASS)
GOOD 0x0200000033554432 Good-quality measurement in our data (eg; PS)
GOOD_ALT 0x0400000067108864 Good-quality measurement in external data (eg; 2MASS)
GOOD_STACK 0x08000000134217728 Good-quality object in the stack (> 1 good stack measurement)
BEST_STACK 0x10000000268435456 The primary stack measurements are the best measurements
SUSPECT_STACK 0x20000000536870912 Suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement)
BAD_STACK 0x400000001073741824 Poor-quality stack object (no more than 1 good or suspect measurement).
DEFAULT 0x000000000 Initial value; resets all bits.
SECF_STAR_FEW 0x000000011 Used within relphot: skip star.
SECF_STAR_POOR 0x000000022 Used within relphot: skip star.
SECF_USE_SYNTH 0x000000044 Synthetic photometry used in average measurement.
SECF_USE_UBERCAL 0x000000088 Ubercal photometry used in average measurement.
SECF_HAS_PS1 0x0000001016 PS1 photometry used in average measurement.
SECF_HAS_PS1_STACK0x0000002032 PS1 stack photometry exists.
SECF_HAS_TYCHO 0x0000004064 Tycho photometry used for synthetic magnitudes.
SECF_FIX_SYNTH 0x00000080128 Synthetic magnitudes repaired with zeropoint map.
SECF_RANK_0 0x00000100256 Average magnitude uses only rank 0 detections.
SECF_RANK_1 0x00000200512 Average magnitude uses only rank 1 detections.
SECF_RANK_2 0x000004001024 Average magnitude uses only rank 2 detections.
SECF_RANK_3 0x000008002048 Average magnitude uses only rank 3 detections.
SECF_RANK_4 0x000010004096 Average magnitude uses only rank 4 detections.
SECF_STACK_PRIMARY0x0000400016384 PS1 stack photometry comes from primary skycell.
SECF_STACK_BESTDET0x0000800032768 PS1 stack best measurement is a detection (not forced).
SECF_STACK_PRIMDET0x0001000065536 PS1 stack primary measurement is a detection (not forced).
SECF_HAS_SDSS 0x001000001048576 This photcode has SDSS photometry (EAM??).
SECF_HAS_HSC 0x002000002097152 This photcode has HSC photometry (EAM??).
SECF_HAS_CFH 0x004000004194304 This photcode has CFH photometry (mostly Megacam) (EAM??).
SECF_HAS_DES 0x008000008388608 This photcode has DES photometry (EAM??).
SECF_OBJ_EXT 0x0100000016777216 Extended in this band.
DEFAULT 0x000000000 Initial value; resets all bits.
SECF_STAR_FEW 0x000000011 Used within relphot: skip star.
SECF_STAR_POOR 0x000000022 Used within relphot: skip star.
SECF_USE_SYNTH 0x000000044 Synthetic photometry used in average measurement.
SECF_USE_UBERCAL 0x000000088 Ubercal photometry used in average measurement.
SECF_HAS_PS1 0x0000001016 PS1 photometry used in average measurement.
SECF_HAS_PS1_STACK 0x0000002032 PS1 stack photometry exists.
SECF_HAS_TYCHO 0x0000004064 Tycho photometry used for synthetic magnitudes.
SECF_FIX_SYNTH 0x00000080128 Synthetic magnitudes repaired with zeropoint map.
SECF_RANK_0 0x00000100256 Average magnitude calculated in 0th pass.
SECF_RANK_1 0x00000200512 Average magnitude calculated in 1st pass.
SECF_RANK_2 0x000004001024 Average magnitude calculated in 2nd pass.
SECF_RANK_3 0x000008002048 Average magnitude calculated in 3rd pass.
SECF_RANK_4 0x000010004096 Average magnitude calculated in 4th pass.
SECF_STACK_PRIMARY 0x0000400016384 PS1 stack photometry includes a primary skycell.
SECF_STACK_BESTDET 0x0000800032768 PS1 stack best measurement is a detection (not forced).
SECF_STACK_PRIMDET 0x0001000065536 PS1 stack primary measurement is a detection (not forced).
SECF_STACK_PRIMARY_MULTIPLE0x0001000065536 PS1 stack object has multiple primary measurements.
SECF_HAS_SDSS 0x001000001048576 This photcode has SDSS photometry.
SECF_HAS_HSC 0x002000002097152 This photcode has HSC photometry.
SECF_HAS_CFH 0x004000004194304 This photcode has CFH photometry (mostly Megacam).
SECF_HAS_DES 0x008000008388608 This photcode has DES photometry.
SECF_OBJ_EXT 0x0100000016777216 Extended in this band.
DEFAULT 0x000000000 Initial value; resets all bits.
QF_OBJ_EXT 0x000000011 Extended in our data (eg, PS).
QF_OBJ_EXT_ALT 0x000000022 Extended in external data (eg, 2MASS).
QF_OBJ_GOOD 0x000000044 Good-quality measurement in our data (eg,PS).
QF_OBJ_GOOD_ALT 0x000000088 Good-quality measurement in external data (eg, 2MASS).
QF_OBJ_GOOD_STACK 0x0000001016 good-quality object in the stack (> 1 good stack measurement)
QF_OBJ_BEST_STACK 0x0000002032 the primary stack measurements are the best measurements.
QF_OBJ_SUSPECT_STACK0x0000004064 suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement).
QF_OBJ_BAD_STACK 0x00000080128 poor-quality stack object (no more than 1 good or suspect measurement).
QF_OBJ_EXT 0x000000011 Extended in our data (eg; PS).
QF_OBJ_EXT_ALT 0x000000022 Extended in external data (eg; 2MASS).
QF_OBJ_GOOD 0x000000044 Good-quality measurement in our data (eg; PS).
QF_OBJ_GOOD_ALT 0x000000088 Good-quality measurement in external data (eg; 2MASS).
QF_OBJ_GOOD_STACK 0x0000001016 Good-quality object in the stack (> 1 good stack measurement).
QF_OBJ_BEST_STACK 0x0000002032 The primary stack measurements are the best measurements.
QF_OBJ_SUSPECT_STACK0x0000004064 Suspect object in the stack (no more than 1 good measurement, 2 or more suspect or good stack measurement).
QF_OBJ_BAD_STACK 0x00000080128 Poor-quality stack object (no more than 1 good or suspect measurement).
NO_ERROR 0x000000000 No error condition raised.
FAIL_FIT 0x000000011 fit failed to converge or was degenerate
TOO_FEW 0x000000022 not enough points to fit the model
OUT_OF_RANGE 0x000000044 fit minimum too far outside data range
BAD_ERROR 0x000000088 invalid size error (nan or inf)
FAIL_FIT 0x000000011 Fit failed to converge or was degenerate
TOO_FEW 0x000000022 Not enough points to fit the model
OUT_OF_RANGE 0x000000044 Fit minimum too far outside data range
BAD_ERROR 0x000000088 Invalid size error (nan or inf)
- + Contains the PSF, Kron (1980), and aperture fluxes for all filters in a single row, along with point-source object shape parameters. See StackObjectThin table for discussion of primary, secondary, and best detections. References: Kron, R. G. 1980, ApJS, 43, 305. @@ -587,5 +605,8 @@ - Photometric zeropoint for the g filter stack. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + Photometric zeropoint for the g filter stack (PSF-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + + + Photometric zeropoint for the g filter stack (APERTURE-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. @@ -694,5 +715,8 @@ - Photometric zeropoint for the r filter stack. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + Photometric zeropoint for the r filter stack (PSF-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + + + Photometric zeropoint for the r filter stack (APERTURE-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. @@ -801,5 +825,8 @@ - Photometric zeropoint for the i filter stack. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + Photometric zeropoint for the i filter stack (PSF-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + + + Photometric zeropoint for the i filter stack (APERTURE-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. @@ -908,5 +935,8 @@ - Photometric zeropoint for the z filter stack. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + Photometric zeropoint for the z filter stack (PSF-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + + + Photometric zeropoint for the z filter stack (APERTURE-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. @@ -1015,5 +1045,8 @@ - Photometric zeropoint for the y filter stack. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + Photometric zeropoint for the y filter stack (PSF-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + + + Photometric zeropoint for the y filter stack (APERTURE-like magnitudes only). Necessary for converting listed fluxes and magnitudes back to measured ADU counts. @@ -4790,5 +4823,5 @@ - Photometric zeropoint. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. + Photometric zeropoint of frame/exposure. Necessary for converting listed fluxes and magnitudes back to measured ADU counts. Index: branches/czw_branch/20170908/ippToPsps/jython/batch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/batch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/batch.py (revision 40483) @@ -68,5 +68,5 @@ raise else: - logger.errorPair("missing fits item?", str(id)) + logger.errorPair("missing fits item for batch ID" + str(id), "continue") # define the dvo table names Index: branches/czw_branch/20170908/ippToPsps/jython/detectionbatch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/detectionbatch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/detectionbatch.py (revision 40483) @@ -60,6 +60,10 @@ meta = self.gpc1Db.getCameraStageMeta(self.id) if not meta: - self.logger.errorPair("Could not get", "camera metadata") - raise + self.logger.infoPair("Could not get camera metadata", "trying with no warp id") + meta = self.gpc1Db.getCameraStageMetaNoWarpID(self.id) + if not meta: + self.logger.errorPair("Could not get camera metadata", "now what?") + raise + # set a flag indicating that we have the new columns added in version 5 of the @@ -80,5 +84,6 @@ self.chipID = meta[6]; self.warpID = meta[7]; - + + if not self.analysisVer: self.analysisVer = -999 @@ -815,24 +820,24 @@ sqlLine.group("a.ra", "b.ra") sqlLine.group("a.dec", "b.dec_") - sqlLine.group("a.zp", "b.zp") + sqlLine.group("a.zp", "b.zpPSF") sqlLine.group("a.telluricExt", "b.telluricExt") sqlLine.group("a.airmass", "b.airmass") sqlLine.group("a.infoFlag3", "b.flags") sqlLine.group("a.expTime", "b.expTime") - sqlLine.group("a.psfFlux", "a.psfFlux * b.zpFactor") - sqlLine.group("a.psfFluxErr", "a.psfFluxErr * b.zpFactor") - sqlLine.group("a.apFlux", "a.apFlux * b.zpFactor") - sqlLine.group("a.apFluxErr", "a.apFluxErr * b.zpFactor") - sqlLine.group("a.kronFlux", "a.kronFlux * b.zpFactor") - sqlLine.group("a.kronFluxErr", "a.kronFluxErr * b.zpFactor") - sqlLine.group("a.sky", "a.sky * b.zpFactor") - sqlLine.group("a.skyErr", "a.skyErr * b.zpFactor") + sqlLine.group("a.psfFlux", "a.psfFlux * b.zpFactorPSF") + sqlLine.group("a.psfFluxErr", "a.psfFluxErr * b.zpFactorPSF") + sqlLine.group("a.apFlux", "a.apFlux * b.zpFactorAPER") + sqlLine.group("a.apFluxErr", "a.apFluxErr * b.zpFactorAPER") + sqlLine.group("a.kronFlux", "a.kronFlux * b.zpFactorAPER") + sqlLine.group("a.kronFluxErr", "a.kronFluxErr * b.zpFactorAPER") + sqlLine.group("a.sky", "a.sky * b.zpFactorAPER") + sqlLine.group("a.skyErr", "a.skyErr * b.zpFactorAPER") sql = sqlLine.makeEquals("WHERE a.ippDetectID = b.ippDetectID AND b.imageID = " + str(imageID)) - ## a.psfFlux = a.psfFlux * 3630.78 * POW(10.0, -0.4*b.zp), \ + ## a.psfFlux = a.psfFlux * 3630.78 * POW(10.0, -0.4*b.zpPSF), \ # instrumental flux vs Janskies: - # b.ZP is defined to include the airmass term and zero point offset from relative calibration or ubercal + # b.zpPSF is defined to include the airmass term and zero point offset from relative calibration or ubercal # (see dvopsps/src/insert_detections_dvopsps_catalog.c:311 # mag_AB = -2.5*log(flux_DN/sec) + ZP Index: branches/czw_branch/20170908/ippToPsps/jython/diffbatch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/diffbatch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/diffbatch.py (revision 40483) @@ -457,5 +457,5 @@ sqlLine.group("a.raErr", "b.raErr") # NOTE: This is supplied above as well, is this needed?? sqlLine.group("a.decErr", "b.decErr") # NOTE: This is supplied above as well, is this needed?? - sqlLine.group("a.zp", "b.zp") + sqlLine.group("a.zp", "b.zpPSF") sqlLine.group("a.telluricExt", "b.telluricExt") sqlLine.group("a.airmass", "b.airmass") @@ -633,17 +633,17 @@ a.ra = b.ra, \ a.dec = b.dec_, \ - a.zp = b.zp, \ + a.zp = b.zpPSF, \ a.telluricExt = b.telluricExt, \ a.airmass = b.airmass, \ a.expTime = b.expTime, \ a.DinfoFlag3 = b.flags \ - a.DpsfFlux = a.DpsfFlux * b.zpFactor, \ - a.DpsfFluxErr = a.DpsfFluxErr * b.zpFactor, \ - a.apFlux = a.apFlux * b.zpFactor, \ - a.apFluxErr = a.apFluxErr * b.zpFactor, \ - a.kronFlux = a.kronFlux * b.zpFactor, \ - a.kronFluxErr = a.kronFluxErr * b.zpFactor \ - a.sky = a.sky * b.zpFactor, \ - a.skyErr = a.skyErr * b.zpFactor \ + a.DpsfFlux = a.DpsfFlux * b.zpFactorPSF, \ + a.DpsfFluxErr = a.DpsfFluxErr * b.zpFactorPSF, \ + a.apFlux = a.apFlux * b.zpFactorAPER, \ + a.apFluxErr = a.apFluxErr * b.zpFactorAPER, \ + a.kronFlux = a.kronFlux * b.zpFactorAPER, \ + a.kronFluxErr = a.kronFluxErr * b.zpFactorAPER \ + a.sky = a.sky * b.zpFactorAPER, \ + a.skyErr = a.skyErr * b.zpFactorAPER \ WHERE a.ippDetectID = b.ippDetectID \ AND b.imageID = " + str(imageID) Index: branches/czw_branch/20170908/ippToPsps/jython/diffobjectbatch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/diffobjectbatch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/diffobjectbatch.py (revision 40483) @@ -265,6 +265,6 @@ sqlLine.group("ra", "RA_MEAN") sqlLine.group("dec_", "DEC_MEAN") - sqlLine.group("diffObjName", "IAU_NAME") - sqlLine.group("diffObjPSOName", "PSO_NAME") + sqlLine.group("diffObjName", "PSO_NAME") + sqlLine.group("diffObjNameHMS", "PSX_NAME") sqlLine.group("nDetections", "'0'") sql = sqlLine.makeRaw(") SELECT ", " FROM " + cptTableName) Index: branches/czw_branch/20170908/ippToPsps/jython/dvo.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/dvo.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/dvo.py (revision 40483) @@ -101,6 +101,8 @@ if not self.loadImages(): + self.logger.debugPair("FAILED to load images", "exiting") os._exit(1) + print "DONE load images (resetAllTables)" ''' @@ -145,7 +147,28 @@ self.logger.infoPair("DVO Images.dat file", "NOT up-to-date") + # check on the format: + try: + imagesHeaderPHU = self.readFitsHeader("PHU", path) + except Exception,e: + # print the error, wait a few secs then try again + print str(e) + self.logger.infoPair("Failed to find PHU for Images.dat", path) + raise + + if "FORMAT" in imagesHeaderPHU: + fileFormat = imagesHeaderPHU["FORMAT"] + else: + self.logger.infoPair("Missing FORMAT for Images.dat", path) + raise + + self.logger.infoPair("got FORMAT for Images.dat", fileFormat) + + if not (fileFormat == "PS1_V6"): + self.logger.infoPair("Invalid FORMAT for Images.dat (PS1_V6)", fileFormat) + return False + self.importFits( path, - "SOURCE_ID IMAGE_ID EXTERN_ID FLAGS MCAL SECZ PHOTCODE XPIX_SYS_ERR YPIX_SYS_ERR N_FIT_ASTROM MAG_SYS_ERR N_FIT_PHOTOM", + "SOURCE_ID IMAGE_ID EXTERN_ID FLAGS MCAL_PSF MCAL_APER SECZ PHOTCODE XPIX_SYS_ERR YPIX_SYS_ERR N_FIT_ASTROM MAG_SYS_ERR N_FIT_PHOTOM", self.scratchDb.dvoImagesTable) self.logger.infoPair("Adding primary key to", self.scratchDb.dvoImagesTable) @@ -559,6 +582,7 @@ self.logger.debugPair("Reading IPP table", table.name) table = stilts.tpipe(table, cmd='explodeall') - #adds an index to all the tables - # table = stilts.tpipe(table, cmd='addcol table_index $0') + + # adds an index to all the tables + # table = stilts.tpipe(table, cmd='addcol table_index $0') # IPP FITS files are littered with infinity values. Remove them @@ -566,4 +590,15 @@ table = stilts.tpipe(table, cmd='replaceval -Infinity null *') table = stilts.tpipe(table, cmd='replaceval Infinity null *') + + # verify that all columns in 'columns' exist in the tabl + ## myColumns = table.columns() + ## for wcol in columns: + ## foundCol = False + ## for tcol in myColumns: + ## if (wcol == tcol.name): + ## print "found wcol " + wcol + ## foundCol = True; + ## break + ## if not foundCol: attempts = 0 @@ -575,6 +610,7 @@ table.cmd_keepcols(columns).write(self.scratchDb.url + '#' + tableName) break - except: - # wait a few secs then try again + except Exception,e: + # print the error, wait a few secs then try again + print str(e) time.sleep(2) attempts = attempts + 1 @@ -663,4 +699,30 @@ ''' + Find and read a header extension. + ''' + def readFitsHeader(self, extname, filename): + + cmd = "fhead -n %s %s" % (extname, filename) + p = Popen(cmd, shell=True, stdout=PIPE) + output = p.communicate()[0] + + header = {} + + # split the output header bytes into key/value pairs for each 80-byte line + for i in range(0, len(output), 80): + # print "line %d : %s" % (i, output[i:i+80]) + record = output[i:i+80] + + match = re.match('^(HIERARCH )*([a-zA-Z0-9-_\.]+)\s*=\s+\'*([a-zA-Z0-9-+_\.:\s@#\(\)\,]+)\'*\\/*', record) + + if match: + param = match.group(2) + value = match.group(3).strip() + if value == "NaN": value = "NULL" + header[param] = value + + return header + + ''' ingest skyregion into MySQL database using the native DVO program dvopsps Populates dvoDetections and dvoDone with FITS tables from DVO for the defined sky area, this @@ -688,5 +750,9 @@ # make sure we have an up-to-date Images table - self.loadImages() # this is in nativeIngestDetections (remove?) + if not self.loadImages(): # this is in nativeIngestDetections (remove?) + self.logger.debugPair("FAILED to load images", "exiting") + os._exit(1) + + print "DONE load images (nativeIngestDetections)" # the box dimensions are the area used to select the items of Index: branches/czw_branch/20170908/ippToPsps/jython/dvodetections.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/dvodetections.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/dvodetections.py (revision 40483) @@ -24,4 +24,5 @@ super(DvoDetections, self).__init__(logger, config, skychunk, ippToPspsDb, scratchDb, gpc1Db) + print "----- DvoDetections is not used, is it?? -----" os._exit(1) Index: branches/czw_branch/20170908/ippToPsps/jython/dvodiffdetections.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/dvodiffdetections.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/dvodiffdetections.py (revision 40483) @@ -14,4 +14,5 @@ ''' A class for ingesting DVO detections into MySQL +XXX EAM 20151029 : I do not believe this class is used at all. I am poisoning it with an exit ''' class DvoDetections(Dvo): @@ -25,4 +26,7 @@ super(DvoDetections, self).__init__(logger, config, skychunk, ippToPspsDb, scratchDb, gpc1Db) # super(DvoDetections, self).__init__(logger, config, skychunk, ippToPspsDb, scratchDb) + + print "----- DvoDetections is not used, is it?? -----" + os._exit(1) # declare DVO file types of interest Index: branches/czw_branch/20170908/ippToPsps/jython/dvoforcedobjects.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/dvoforcedobjects.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/dvoforcedobjects.py (revision 40483) @@ -48,4 +48,7 @@ cmd += " -cpt " + region + if self.config.test: + cmd += " -test-mode" + if self.skychunk.parallel: if self.scratchDb.dbHost == "localhost": Index: branches/czw_branch/20170908/ippToPsps/jython/forcedwarpbatch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/forcedwarpbatch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/forcedwarpbatch.py (revision 40483) @@ -424,6 +424,17 @@ # self.selectDvoObjIDs() self.logger.infoPair("inserting dvo info: find imageID", "from externID") - imageID = self.scratchDb.getImageIDFromExternIDandPhotcodeRange(self.warpSkyFileID[num],12000,12500) - self.logger.infoPair("I don't like this", "it is hardwired to gpc1") + + # get the imageID: this code has cases for simtest vs gpc1 cameras + # if self.config.test: + self.logger.infoPair("current camera is", self.config.camera) + if self.config.camera == "gpc1": + self.logger.infoPair("I don't like this", "it is hardwired to gpc1") + imageID = self.scratchDb.getImageIDFromExternIDandPhotcodeRange(self.warpSkyFileID[num],12000,12500) + if self.config.camera == "simtest": + imageID = self.scratchDb.getImageIDFromExternIDandPhotcodeRange(self.warpSkyFileID[num],15100,15500) + if imageID == 0: + self.logger.infoPair ("failed to get imageID for", str(num)) + raise + self.logger.infoPair("updating", pspsTableName) @@ -433,17 +444,17 @@ sqlLine.group("a.ippObjID", "b.ippObjID") sqlLine.group("a.dvoRegionID","b.catID") - sqlLine.group("a.zp", "b.zp") + sqlLine.group("a.zp", "b.zpPSF") sqlLine.group("a.telluricExt","b.telluricExt") sqlLine.group("a.airmass", "b.airmass") sqlLine.group("a.FinfoFlag3", "b.flags") - sqlLine.group("a.FpsfFlux", "a.FpsfFlux * b.zpFactor") - sqlLine.group("a.FpsfFluxErr", "a.FpsfFluxErr * b.zpFactor") - sqlLine.group("a.FapFlux", "a.FapFlux * b.zpFactor") - sqlLine.group("a.FapFluxErr", "a.FapFluxErr * b.zpFactor") - sqlLine.group("a.FkronFlux", "a.FkronFlux * b.zpFactor") - sqlLine.group("a.FkronFluxErr", "a.FkronFluxErr * b.zpFactor") - sqlLine.group("a.Fsky", "a.Fsky * b.zpFactor") - sqlLine.group("a.FskyErr", "a.FskyErr * b.zpFactor") + sqlLine.group("a.FpsfFlux", "a.FpsfFlux * b.zpFactorPSF") + sqlLine.group("a.FpsfFluxErr", "a.FpsfFluxErr * b.zpFactorPSF") + sqlLine.group("a.FapFlux", "a.FapFlux * b.zpFactorAPER") + sqlLine.group("a.FapFluxErr", "a.FapFluxErr * b.zpFactorAPER") + sqlLine.group("a.FkronFlux", "a.FkronFlux * b.zpFactorAPER") + sqlLine.group("a.FkronFluxErr", "a.FkronFluxErr * b.zpFactorAPER") + sqlLine.group("a.Fsky", "a.Fsky * b.zpFactorAPER") + sqlLine.group("a.FskyErr", "a.FskyErr * b.zpFactorAPER") sql = sqlLine.makeEquals("WHERE a.ippDetectID = b.ippDetectID AND b.imageID = " +str( imageID)) @@ -457,7 +468,15 @@ self.logger.infoPair("row count of ForcedWarpMeasurement after cull of null objID ", rowCountAfter) if rowCountAfter == 0: - self.skipBatch = True - self.logger.infoPair ("skip this batch", "it is bad") - return False + if self.haveLensPSF[num] ==0: + self.logger.infoPair("skipping this table", "no lenspsf and no dvo, run this:") + self.logger.infoPair("update addRun join addProcessedExp using (add_id) set fault = 43 where add_id = ",str(self.number[num])) + self.skipBatch = True + return False + else: + self.skipBatch = True + self.logger.infoPair ("skip this batch", "it is bad") + return False + + self.logger.infoPair("Adding 'row' columns to", pspsTableName) @@ -557,15 +576,15 @@ sqlLine = sqlUtility("UPDATE " + pspsTableName + " as a, " + self.scratchDb.dvoDetectionTable + " as b SET ") - sqlLine.group("a.flxR5", "a.flxR5 * b.zpFactor") - sqlLine.group("a.flxR5Err", "a.flxR5Err * b.zpFactor") - sqlLine.group("a.flxR5Std", "a.flxR5Std * b.zpFactor") - - sqlLine.group("a.flxR6", "a.flxR6 * b.zpFactor") - sqlLine.group("a.flxR6Err", "a.flxR6Err * b.zpFactor") - sqlLine.group("a.flxR6Std", "a.flxR6Std * b.zpFactor") - - sqlLine.group("a.flxR7", "a.flxR7 * b.zpFactor") - sqlLine.group("a.flxR7Err", "a.flxR7Err * b.zpFactor") - sqlLine.group("a.flxR7Std", "a.flxR7Std * b.zpFactor") + sqlLine.group("a.flxR5", "a.flxR5 * b.zpFactorAPER") + sqlLine.group("a.flxR5Err", "a.flxR5Err * b.zpFactorAPER") + sqlLine.group("a.flxR5Std", "a.flxR5Std * b.zpFactorAPER") + + sqlLine.group("a.flxR6", "a.flxR6 * b.zpFactorAPER") + sqlLine.group("a.flxR6Err", "a.flxR6Err * b.zpFactorAPER") + sqlLine.group("a.flxR6Std", "a.flxR6Std * b.zpFactorAPER") + + sqlLine.group("a.flxR7", "a.flxR7 * b.zpFactorAPER") + sqlLine.group("a.flxR7Err", "a.flxR7Err * b.zpFactorAPER") + sqlLine.group("a.flxR7Std", "a.flxR7Std * b.zpFactorAPER") sql = sqlLine.makeEquals("WHERE a.ippDetectID = b.ippDetectID AND b.imageID = " +str( imageID)) @@ -657,4 +676,5 @@ # each of the "populate*" methods below add their tables to the list: self.tablesToExport=[] + tables = [] # list of tables to calculate min/max objid on... for num in self.number: @@ -671,11 +691,12 @@ self.logger.infoPair("Populating", myTable) if not self.populateForcedWarpMeasurement(num): - self.logger.infoPair ("skippping ForcedWarpMeasurement", str(num)) + self.logger.infoPair ("failed on ForcedWarpMeasurement", str(num)) #continue raise self.tablesToExport.append(myTable) - - self.logger.infoPair("setting min/max objid from", myTable) - self.setMinMaxObjID([myTable]) + tables.append(myTable) + #self.logger.infoPair("setting min/max objid from", myTable) + + #self.setMinMaxObjID([myTable]) # populateForcedWarpMeasurement also populates ForcedWarpMasked (from ForcedWarpMeasurement) @@ -697,7 +718,9 @@ self.populateForcedWarpToImage(num) self.tablesToExport.append(myTable) + self.logger.infoPair("setting min/max objid from", "all the fw tables") + self.setMinMaxObjID(tables) + self.logger.infoPair("finishing","populatePspsTables") + - self.logger.infoPair("finishing","populatePspsTables") - return True @@ -762,4 +785,5 @@ self.haveLensPSF[num] = 0 self.haveLensing[num] = 0 + for table in tables: # check for AP_NPIX columns: missing for SAS39 Index: branches/czw_branch/20170908/ippToPsps/jython/gpc1db.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/gpc1db.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/gpc1db.py (revision 40483) @@ -511,6 +511,8 @@ files = glob.glob(pathBase + ".cmf") + if len(files) < 1: + self.logger.errorPair("No files found for forced warp path", pathBase) + return None - if len(files) < 1: return None fits = Fits(self.logger, self.config, files[0]) self.logger.infoPair("cmf file",files[0]) @@ -760,4 +762,6 @@ meta = [] + + sql = "SELECT exp_id, exp_name, camRun.dist_group, camRun.software_ver, rawExp.bg, rawExp.bg_stdev, chip_id, warp_id \ FROM camRun \ @@ -773,6 +777,5 @@ rs.first() except: - self.logger.errorPair("Can't query for", "camera meta data") - + self.logger.errorPair("Can't query for", "camera meta data") try: meta.append(rs.getInt(1)) @@ -786,6 +789,42 @@ except: self.logger.errorPair("getCameraStageMeta()", "empty meta data") - return meta + + def getCameraStageMetaNoWarpID(self, camID): + + self.logger.debug("Querying GPC1 for camera meta data") + + meta = [] + + + sql = "SELECT exp_id, exp_name, camRun.dist_group, camRun.software_ver, rawExp.bg, rawExp.bg_stdev, chip_id, -999 \ + FROM camRun \ + JOIN chipRun USING(chip_id) \ + JOIN rawExp USING(exp_id) \ + LEFT JOIN fakeRun USING(cam_id) \ + LEFT JOIN warpRun USING(fake_id) \ + WHERE camRun.cam_id = %d" % camID + + try: + rs = self.executeQuery(sql) + rs.first() + except: + self.logger.errorPair("Can't query for", "camera meta data") + try: + meta.append(rs.getInt(1)) + meta.append(rs.getString(2)) + meta.append(rs.getString(3)) + meta.append(rs.getString(4)) + meta.append(rs.getString(5)) + meta.append(rs.getString(6)) + meta.append(rs.getInt(7)) + meta.append(rs.getInt(8)) + except: + self.logger.errorPair("getCameraStageMeta()", "empty meta data") + + return meta + + + ''' @@ -818,5 +857,4 @@ files = [] - #there are a couple of states for smfversion @@ -829,5 +867,4 @@ # use (file).smf if camRun.state = 'full' and if there is only a .smf # fault out if camRun.state != 'full' -- that is a race condition we don't want. - if (smfversion == "not_reproc") or (smfversion == "use_new"): @@ -904,5 +941,7 @@ self.logger.infoPair("smf files:", files) - if len(files) < 1: return None + if len(files) < 1: + self.logger.errorPair("No files found for cam path", path) + return None return Fits(self.logger, self.config, files[0]) # TODO just returning first file - check @@ -974,5 +1013,7 @@ # print "stack cmf files:", files - if len(files) < 1: return None + if len(files) < 1: + self.logger.errorPair("No files found for stack path", pathBase) + return None # if we get here, then the cmf is readable, now check the stack_id Index: branches/czw_branch/20170908/ippToPsps/jython/ippjytest =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/ippjytest (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/ippjytest (revision 40483) @@ -14,5 +14,5 @@ set JARDIR = $datadir/jar set JYDIR = $datadir/jython -# set JYDIR = . +set JYDIR = . setenv IPPTOPSPS_DATA $datadir/ipptopsps Index: branches/czw_branch/20170908/ippToPsps/jython/ipptopspsdb.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/ipptopspsdb.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/ipptopspsdb.py (revision 40483) @@ -27,5 +27,5 @@ self.skychunk = skychunk - self.MAX_FAILS = 5 + self.MAX_FAILS = 15 ''' Index: branches/czw_branch/20170908/ippToPsps/jython/objectbatch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/objectbatch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/objectbatch.py (revision 40483) @@ -312,6 +312,6 @@ sqlLine.group("raMean", "RA_MEAN") sqlLine.group("decMean", "DEC_MEAN") - sqlLine.group("objName", "IAU_NAME") - sqlLine.group("objPSOName", "PSO_NAME") + sqlLine.group("objName", "PSO_NAME") + sqlLine.group("objNameHMS", "PSX_NAME") sqlLine.group("epochMean", "EPOCH_MEAN") sqlLine.group("raMeanErr", "RA_ERR") Index: branches/czw_branch/20170908/ippToPsps/jython/queue.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/queue.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/queue.py (revision 40483) @@ -134,5 +134,5 @@ #if (dec > 85): raise # this may be a problem for poles # HAF XXX - if (dec > 90): raise + ####if (dec > 90): raise # print "dec, dD: ", dec, dD ,self.skychunk.maxDec @@ -178,5 +178,5 @@ dec = dec + 2*dD #if (dec > 85): raise - if (dec > 90): raise + ###if (dec > 90): raise #dR = 0.5*self.skychunk.boxSize / math.cos(math.radians(dec)) dR = 0.5*self.skychunk.boxSize Index: branches/czw_branch/20170908/ippToPsps/jython/scratchdb.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/scratchdb.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/scratchdb.py (revision 40483) @@ -271,5 +271,7 @@ zeroPoint = -999 - sql = "SELECT MCAL, SECZ, C_LAM, K FROM %s AS a JOIN %s as b WHERE EXTERN_ID = %s AND a.PHOTCODE = b.CODE" % (self.dvoImagesTable, self.dvoPhotcodesTable, (externID)) + # MCAL_PSF, MCAL_APER, SECZ are from dvoImagesTabl + # C_LAM, K are from dvoPhotcodeTable + sql = "SELECT MCAL_PSF, SECZ, C_LAM, K FROM %s AS a JOIN %s as b WHERE EXTERN_ID = %s AND a.PHOTCODE = b.CODE" % (self.dvoImagesTable, self.dvoPhotcodesTable, (externID)) rs = self.executeQuery(sql) if not rs: @@ -288,4 +290,34 @@ zeroPoint = 0.001*Clam + Klam*(airmass - 1.0) - Mcal return zeroPoint + + ''' + Gets zero point for image (PSF and APER) + ''' + def getStackZeroPoint(self, externID): + + zeroPointPSF = -999 + zeroPointAPER = -999 + + # MCAL_PSF, MCAL_APER, SECZ are from dvoImagesTabl + # C_LAM, K are from dvoPhotcodeTable + sql = "SELECT MCAL_PSF, MCAL_APER, SECZ, C_LAM, K FROM %s AS a JOIN %s as b WHERE EXTERN_ID = %s AND a.PHOTCODE = b.CODE" % (self.dvoImagesTable, self.dvoPhotcodesTable, (externID)) + rs = self.executeQuery(sql) + if not rs: + print "missing result set for imageID query" + os._exit(2) + + if not rs.first(): + self.logger.infoPair("no zero point; image is missing, externID: ", externID) + return zeroPoint + + McalPSF = rs.getFloat(1) + McalAPER = rs.getFloat(2) + airmass = rs.getFloat(3) + Clam = rs.getFloat(4) + Klam = rs.getFloat(5) + + zeroPointPSF = 0.001*Clam + Klam*(airmass - 1.0) - McalPSF + zeroPointAPER = 0.001*Clam + Klam*(airmass - 1.0) - McalAPER + return (zeroPointPSF, zeroPointAPER) ''' @@ -576,5 +608,6 @@ EXTERN_ID INT, \ FLAGS INT, \ - MCAL FLOAT, \ + MCAL_PSF FLOAT, \ + MCAL_APER FLOAT, \ SECZ FLOAT, \ PHOTCODE SMALLINT, \ @@ -607,6 +640,8 @@ raErr FLOAT, \ decErr FLOAT, \ - zp FLOAT, \ - zpFactor FLOAT, \ + zpPSF FLOAT, \ + zpFactorPSF FLOAT, \ + zpAPER FLOAT, \ + zpFactorAPER FLOAT, \ telluricExt FLOAT, \ airmass FLOAT, \ @@ -620,4 +655,5 @@ flags INT, \ objflags INT, \ + filtflags INT, \ PRIMARY KEY (imageID, ippDetectID), \ KEY (objID, detectID), \ Index: branches/czw_branch/20170908/ippToPsps/jython/stackbatch.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/stackbatch.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/stackbatch.py (revision 40483) @@ -307,12 +307,13 @@ # zp correction should comes from DVO (unless image is not in DVO) - zpImage = self.scratchDb.getImageZeroPoint(stackID) - - if (zpImage == "NULL"): - zpImage = zp + (zpImagePSF, zpImageAPER) = self.scratchDb.getStackZeroPoint(stackID) + + # zp is the value in the header, zpImage includes the McalPSF value from DVO + if (zpImagePSF == "NULL"): zpImagePSF = zp + if (zpImageAPER == "NULL"): zpImageAPER = zpImagePSF detectionThreshold = "NULL" - if (magref != "NULL") and (zpImage != "NULL") and (expTime != "NULL"): - detectionThreshold = magref + zpImage + 2.5 * math.log10(expTime) + if (magref != "NULL") and (zpImagePSF != "NULL") and (expTime != "NULL"): + detectionThreshold = magref + zpImagePSF + 2.5 * math.log10(expTime) # insert stack metadata into table @@ -340,5 +341,6 @@ # sqlLine.group("psfFwhm_max", fwhm_maj_uq) sqlLine.group("astroScat", astroscat) - sqlLine.group("photoZero", zpImage) + sqlLine.group("photoZero", zpImagePSF) + sqlLine.group("photoZeroAperture", zpImageAPER) sqlLine.group("photoScat", zpErr) sqlLine.group("nAstroRef", header['NASTRO']) @@ -429,5 +431,6 @@ sqlLine.group("a."+filter+"Epoch", str(stackEpoch)) - sqlLine.group("a."+filter+"PSFMag", "b.Mpsf") + # the Mag values below are set in dvopsps/insert_detections_dvopsps_catalog.c using either zpPSF (PSFMag) or zpAPER (KronMag, ApMag) + sqlLine.group("a."+filter+"PSFMag", "b.Mpsf") sqlLine.group("a."+filter+"PSFMagErr", "b.dMpsf") sqlLine.group("a."+filter+"KronMag", "b.Mkron") @@ -440,4 +443,5 @@ sqlLine.group("a."+filter+"infoFlag3", "b.flags") + sqlLine.group("a."+filter+"infoFlag4", "b.filtflags") sql = sqlLine.makeEquals("WHERE a.objID = b.objID AND a." + filter + "ippDetectID = b.ippDetectID AND b.imageID = " + str(imageID)) @@ -557,14 +561,15 @@ sqlLine = sqlUtility("UPDATE " + tablename + " AS a, " + self.scratchDb.dvoDetectionTable + " as b SET") sqlLine.group("a."+filter+"expTime", "b.expTime") - sqlLine.group("a."+filter+"zp", "b.zp") - sqlLine.group("a."+filter+"PSFFlux", "a."+filter+"PSFFlux * b.zpFactor") - sqlLine.group("a."+filter+"PSFFluxErr", "a."+filter+"PSFFluxErr * b.zpFactor") - sqlLine.group("a."+filter+"ApFlux", "a."+filter+"ApFlux * b.zpFactor") - sqlLine.group("a."+filter+"ApFluxErr", "a."+filter+"ApFluxErr * b.zpFactor") - sqlLine.group("a."+filter+"KronFlux", "a."+filter+"KronFlux * b.zpFactor") - sqlLine.group("a."+filter+"KronFluxErr", "a."+filter+"KronFluxErr * b.zpFactor") - - sqlLine.group("a."+filter+"sky", "a."+filter+"sky * b.zpFactor") - sqlLine.group("a."+filter+"skyErr", "a."+filter+"skyErr * b.zpFactor") + sqlLine.group("a."+filter+"zp", "b.zpPSF") + sqlLine.group("a."+filter+"zpAPER", "b.zpAPER") + sqlLine.group("a."+filter+"PSFFlux", "a."+filter+"PSFFlux * b.zpFactorPSF") + sqlLine.group("a."+filter+"PSFFluxErr", "a."+filter+"PSFFluxErr * b.zpFactorPSF") + sqlLine.group("a."+filter+"ApFlux", "a."+filter+"ApFlux * b.zpFactorAPER") + sqlLine.group("a."+filter+"ApFluxErr", "a."+filter+"ApFluxErr * b.zpFactorAPER") + sqlLine.group("a."+filter+"KronFlux", "a."+filter+"KronFlux * b.zpFactorAPER") + sqlLine.group("a."+filter+"KronFluxErr", "a."+filter+"KronFluxErr * b.zpFactorAPER") + + sqlLine.group("a."+filter+"sky", "a."+filter+"sky * b.zpFactorAPER") + sqlLine.group("a."+filter+"skyErr", "a."+filter+"skyErr * b.zpFactorAPER") # where should these go? @@ -700,4 +705,7 @@ imageID = self.imageIDs[filter] self.logger.infoPair("selecting imageID", imageID) + + # NOTE: for model magnitudes, we use zpPSF not zpAPER since they are PSF-matched + # (maybe that is the wrong solution?) # model calibrated magnitude = instrumental magnitude + 2.5*log10(exptime) + ZP (added below from dvoDetectionTable) @@ -765,5 +773,5 @@ sqlLine = sqlUtility("UPDATE " + tablename + " AS a, " + self.scratchDb.dvoDetectionTable + " as b SET") - sqlLine.group("a." + filter + model + "Mag", "a." + filter + model + "Mag + b.zp") + sqlLine.group("a." + filter + model + "Mag", "a." + filter + model + "Mag + b.zpPSF") sql = sqlLine.makeEquals("WHERE a.objID = b.objID AND a." + filter + "ippDetectID = b.ippDetectID AND b.imageID = " + str(imageID)) @@ -884,5 +892,4 @@ # PETRO_MAG is instrumental + 2.5log(exptime) + FPA.ZP # we want to apply the DVO zero point, so remove FPA.ZP: - ## XXX is this statement true: is FPA.ZP + 2.5log(exptime) applied to PETRO_MAG? hdrZP = str(self.zpImage[filter]) @@ -897,5 +904,5 @@ sqlLine.group("a."+filter+"haveData", "'1'") - sqlLine.group("a." + filter + "petMag", "b.PETRO_MAG - " + hdrZP) + sqlLine.group("a." + filter + "petMag", "b.PETRO_MAG - " + hdrZP) # note that this is **subtracting** the hdrZP (zpAPER is added in below) sqlLine.group("a." + filter + "petMagErr", "1.0 / b.PETRO_MAG_ERR") # still inverted? (yes) @@ -913,5 +920,5 @@ # modify petMag to apply the zero point sqlLine = sqlUtility("UPDATE " + tablename + " AS a, " + self.scratchDb.dvoDetectionTable + " as b SET") - sqlLine.group("a." + filter + "petMag", "a." + filter + "petMag + b.zp") + sqlLine.group("a." + filter + "petMag", "a." + filter + "petMag + b.zpAPER") sql = sqlLine.makeEquals("WHERE a.objID = b.objID AND a." + filter + "ippDetectID = b.ippDetectID AND b.imageID = " + str(imageID)) @@ -1019,20 +1026,22 @@ # care of this, but does not? - sqlLine = sqlUtility("UPDATE " + tablename + " SET") - for radius in [1, 2, 3, 4, 5, 6, 7, 8, 9]: - if radius < minRadius: continue - if radius > maxRadius: continue - pspsRadius = radius + 2 - sqlLine.group(filter + prefix + "flxR" + str(pspsRadius), "-999") - sqlLine.group(filter + prefix + "flxR" + str(pspsRadius) + "Err", "-999") - sqlLine.group(filter + prefix + "flxR" + str(pspsRadius) + "Std", "-999") - sqlLine.group(filter + prefix + "flxR" + str(pspsRadius) + "Fill", "-999") - - sql = sqlLine.makeEquals("") - - try: self.scratchDb.execute(sql) - except: - self.logger.errorPair('failed sql',sql) - raise + # chopping this out to see if it still works + + #sqlLine = sqlUtility("UPDATE " + tablename + " SET") + #for radius in [1, 2, 3, 4, 5, 6, 7, 8, 9]: + # if radius < minRadius: continue + # if radius > maxRadius: continue + # pspsRadius = radius + 2 + # sqlLine.group(filter + prefix + "flxR" + str(pspsRadius), "-999") + # sqlLine.group(filter + prefix + "flxR" + str(pspsRadius) + "Err", "-999") + # sqlLine.group(filter + prefix + "flxR" + str(pspsRadius) + "Std", "-999") + # sqlLine.group(filter + prefix + "flxR" + str(pspsRadius) + "Fill", "-999")# + + #sql = sqlLine.makeEquals("") + + # try: self.scratchDb.execute(sql) + # except: + # self.logger.errorPair('failed sql',sql) + # raise # set the flux values from the cmf file @@ -1110,7 +1119,7 @@ pspsNum = str(int(number) + 2) field = filter + prefix + "flxR" + pspsNum - sqlLine.group(field, field + " * b.zpFactor") - sqlLine.group(field + "Err", field + "err" + " * b.zpFactor") - sqlLine.group(field + "Std", field + "Std" + " * b.zpFactor") + sqlLine.group(field, field + " * b.zpFactorAPER") + sqlLine.group(field + "Err", field + "err" + " * b.zpFactorAPER") + sqlLine.group(field + "Std", field + "Std" + " * b.zpFactorAPER") sql = sqlLine.makeEquals("WHERE a.objID = b.objID AND a." + filter + "ippDetectID = b.ippDetectID AND b.imageID = " + str(imageID)) @@ -1120,4 +1129,5 @@ self.logger.infoPair("failed sql",sql) raise + ''' Populates the StackToImage table Index: branches/czw_branch/20170908/ippToPsps/jython/testCode.py =================================================================== --- branches/czw_branch/20170908/ippToPsps/jython/testCode.py (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/jython/testCode.py (revision 40483) @@ -46,4 +46,10 @@ def importIppTables(self): + if len(sys.argv) != 2: + print "USAGE: ippjython testCode.py (filename)" + os._exit(2) + + self.filename = sys.argv[1] + try: tables = stilts.treads(self.filename) @@ -54,11 +60,33 @@ for table in tables: - print "import smf table " + table.name + print "import table " + table.name # need to generate an index on the IPP_IDET column # table = stilts.tpipe(table, cmd='addcol table_index $0') - table = stilts.tpipe(table, cmd='explodeall') + try: + table = stilts.tpipe(table, cmd='explodeall') + except Exception,e: + # print the error, wait a few secs then try again + print str(e) + return False - print "read smf table " + table.name + try: + myColumns = table.columns() + except Exception,e: + # print the error, wait a few secs then try again + print str(e) + return False + + print "got myColumns" + + try: + for column in myColumns: + print "got column " + column.name + except Exception,e: + # print the error, wait a few secs then try again + print str(e) + return False + + print "read table " + table.name # drop any previous tables before import @@ -69,13 +97,13 @@ # IPP FITS files are littered with infinities, so remove these - print "Removing Infinity values from all columns" - table = stilts.tpipe(table, cmd='replaceval -Infinity null *') - table = stilts.tpipe(table, cmd='replaceval Infinity null *') - - try: - table.write(self.url + '#' + table.name) - except: - print "Problem writing table '" + table.name + "' to the database" - os._exit(4) + ## print "Removing Infinity values from all columns" + ## table = stilts.tpipe(table, cmd='replaceval -Infinity null *') + ## table = stilts.tpipe(table, cmd='replaceval Infinity null *') + ## + ## try: + ## table.write(self.url + '#' + table.name) + ## except: + ## print "Problem writing table '" + table.name + "' to the database" + ## os._exit(4) return True @@ -103,4 +131,5 @@ print "alt word1" + self.importIppTables() # self.connectMysql(argv) Index: branches/czw_branch/20170908/ippToPsps/test/fulltest.sh =================================================================== --- branches/czw_branch/20170908/ippToPsps/test/fulltest.sh (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/test/fulltest.sh (revision 40483) @@ -16,4 +16,27 @@ end setenv IPPTOPSPS_DATA $datadir/ipptopsps + +set DRversion = "DR2" +set myScript = "none" +if ($DRversion == "DR1") then + set myScript = mkgpc1data.dvo + set CATDIR_CAM = catdir.cam + set CATDIR_STK = catdir.stk + set CATDIR_WRP = catdir.wrp + set CATDIR_DIF = catdir.dif +endif +if ($DRversion == "DR2") then + set myScript = mkgpc1data.dr2.dvo + set CATDIR_CAM = catdir.mrg + set CATDIR_STK = catdir.mrg + set CATDIR_WRP = catdir.mrg + set CATDIR_DIF = catdir.dif +endif +if ($DRversion == "none") then + echo "invalid DRversion $DRversion" + exit 2 +endif + +set coords = `$myScript -dump-region` # before running this test, you must have a mysql user: @@ -217,6 +240,6 @@ if ($mkgpc1) then echo "" - echo " ---- mkgpc1data.dvo : creating the dvo & gpc1 database entries ----" - mkgpc1data.dvo + echo " ---- $myScript : creating the dvo & gpc1 database entries ----" + $myScript if ($status) then echo "failed to generate the data & dvo, quitting" @@ -283,4 +306,6 @@ echo "0 " >> initbatch.dat # queue_FW echo "0 " >> initbatch.dat # queue_FO + echo "0 " >> initbatch.dat # queue_FG + echo "0 " >> initbatch.dat # queue_GO echo "1 " >> initbatch.dat # active echo "0 " >> initbatch.dat # parallel @@ -313,9 +338,9 @@ echo "0 " >> cambatch.dat # datastore_publish echo "catdir.cam " >> cambatch.dat # dvo_label - echo "$OUTDIR/catdir.cam " >> cambatch.dat # dvo_location - echo "9 " >> cambatch.dat # min_ra - echo "11 " >> cambatch.dat # max_ra - echo "19 " >> cambatch.dat # min_dec - echo "21 " >> cambatch.dat # max_dec + echo "$OUTDIR/$CATDIR_CAM " >> cambatch.dat # dvo_location + echo "$coords[1] " >> cambatch.dat # min_ra + echo "$coords[2] " >> cambatch.dat # max_ra + echo "$coords[3] " >> cambatch.dat # min_dec + echo "$coords[4] " >> cambatch.dat # max_dec echo "2 " >> cambatch.dat # box_size echo "$OUTDIR " >> cambatch.dat # base_path @@ -334,4 +359,6 @@ echo "0 " >> cambatch.dat # queue_FW echo "0 " >> cambatch.dat # queue_FO + echo "0 " >> cambatch.dat # queue_FG + echo "0 " >> cambatch.dat # queue_GO echo "1 " >> cambatch.dat # active echo "0 " >> cambatch.dat # parallel @@ -366,9 +393,9 @@ echo "0 " >> objectbatch.dat # datastore_publish echo "catdir.cam " >> objectbatch.dat # dvo_label - echo "$OUTDIR/catdir.cam " >> objectbatch.dat # dvo_location - echo "9 " >> objectbatch.dat # min_ra - echo "11 " >> objectbatch.dat # max_ra - echo "19 " >> objectbatch.dat # min_dec - echo "21 " >> objectbatch.dat # max_dec + echo "$OUTDIR/$CATDIR_CAM " >> objectbatch.dat # dvo_location + echo "$coords[1] " >> objectbatch.dat # min_ra + echo "$coords[2] " >> objectbatch.dat # max_ra + echo "$coords[3] " >> objectbatch.dat # min_dec + echo "$coords[4] " >> objectbatch.dat # max_dec echo "2 " >> objectbatch.dat # box_size echo "$OUTDIR " >> objectbatch.dat # base_path @@ -387,4 +414,6 @@ echo "0 " >> objectbatch.dat # queue_FW echo "0 " >> objectbatch.dat # queue_FO + echo "0 " >> objectbatch.dat # queue_FG + echo "0 " >> objectbatch.dat # queue_GO echo "1 " >> objectbatch.dat # active echo "0 " >> objectbatch.dat # parallel @@ -419,9 +448,9 @@ echo "0 " >> stackbatch.dat # datastore_publish echo "catdir.stk " >> stackbatch.dat # dvo_label - echo "$OUTDIR/catdir.stk " >> stackbatch.dat # dvo_location - echo "9 " >> stackbatch.dat # min_ra - echo "11 " >> stackbatch.dat # max_ra - echo "19 " >> stackbatch.dat # min_dec - echo "21 " >> stackbatch.dat # max_dec + echo "$OUTDIR/$CATDIR_STK " >> stackbatch.dat # dvo_location + echo "$coords[1] " >> stackbatch.dat # min_ra + echo "$coords[2] " >> stackbatch.dat # max_ra + echo "$coords[3] " >> stackbatch.dat # min_dec + echo "$coords[4] " >> stackbatch.dat # max_dec echo "2 " >> stackbatch.dat # box_size echo "$OUTDIR " >> stackbatch.dat # base_path @@ -440,4 +469,6 @@ echo "0 " >> stackbatch.dat # queue_FW echo "0 " >> stackbatch.dat # queue_FO + echo "0 " >> stackbatch.dat # queue_FG + echo "0 " >> stackbatch.dat # queue_GO echo "1 " >> stackbatch.dat # active echo "0 " >> stackbatch.dat # parallel @@ -472,9 +503,9 @@ echo "0 " >> forcebatch.dat # datastore_publish echo "catdir.wrp " >> forcebatch.dat # dvo_label - echo "$OUTDIR/catdir.wrp " >> forcebatch.dat # dvo_location - echo "9 " >> forcebatch.dat # min_ra - echo "11 " >> forcebatch.dat # max_ra - echo "19 " >> forcebatch.dat # min_dec - echo "21 " >> forcebatch.dat # max_dec + echo "$OUTDIR/$CATDIR_WRP " >> forcebatch.dat # dvo_location + echo "$coords[1] " >> forcebatch.dat # min_ra + echo "$coords[2] " >> forcebatch.dat # max_ra + echo "$coords[3] " >> forcebatch.dat # min_dec + echo "$coords[4] " >> forcebatch.dat # max_dec echo "2 " >> forcebatch.dat # box_size echo "$OUTDIR " >> forcebatch.dat # base_path @@ -493,4 +524,6 @@ echo "1 " >> forcebatch.dat # queue_FW echo "0 " >> forcebatch.dat # queue_FO + echo "0 " >> forcebatch.dat # queue_FG + echo "0 " >> forcebatch.dat # queue_GO echo "1 " >> forcebatch.dat # active echo "0 " >> forcebatch.dat # parallel @@ -525,9 +558,9 @@ echo "0 " >> forcedobjectbatch.dat # datastore_publish echo "catdir.wrp " >> forcedobjectbatch.dat # dvo_label - echo "$OUTDIR/catdir.wrp " >> forcedobjectbatch.dat # dvo_location - echo "9 " >> forcedobjectbatch.dat # min_ra - echo "11 " >> forcedobjectbatch.dat # max_ra - echo "19 " >> forcedobjectbatch.dat # min_dec - echo "21 " >> forcedobjectbatch.dat # max_dec + echo "$OUTDIR/$CATDIR_WRP " >> forcedobjectbatch.dat # dvo_location + echo "$coords[1] " >> forcedobjectbatch.dat # min_ra + echo "$coords[2] " >> forcedobjectbatch.dat # max_ra + echo "$coords[3] " >> forcedobjectbatch.dat # min_dec + echo "$coords[4] " >> forcedobjectbatch.dat # max_dec echo "2 " >> forcedobjectbatch.dat # box_size echo "$OUTDIR " >> forcedobjectbatch.dat # base_path @@ -546,4 +579,6 @@ echo "0 " >> forcedobjectbatch.dat # queue_FW echo "1 " >> forcedobjectbatch.dat # queue_FO + echo "0 " >> forcedobjectbatch.dat # queue_FG + echo "0 " >> forcedobjectbatch.dat # queue_GO echo "1 " >> forcedobjectbatch.dat # active echo "0 " >> forcedobjectbatch.dat # parallel @@ -579,9 +614,9 @@ echo "0 " >> diffbatch.dat # datastore_publish echo "catdir.dif " >> diffbatch.dat # dvo_label - echo "$OUTDIR/catdir.dif " >> diffbatch.dat # dvo_location - echo "9 " >> diffbatch.dat # min_ra - echo "11 " >> diffbatch.dat # max_ra - echo "19 " >> diffbatch.dat # min_dec - echo "21 " >> diffbatch.dat # max_dec + echo "$OUTDIR/$CATDIR_DIF " >> diffbatch.dat # dvo_location + echo "$coords[1] " >> diffbatch.dat # min_ra + echo "$coords[2] " >> diffbatch.dat # max_ra + echo "$coords[3] " >> diffbatch.dat # min_dec + echo "$coords[4] " >> diffbatch.dat # max_dec echo "2 " >> diffbatch.dat # box_size echo "$OUTDIR " >> diffbatch.dat # base_path @@ -600,4 +635,6 @@ echo "0 " >> diffbatch.dat # queue_FW echo "0 " >> diffbatch.dat # queue_FO + echo "0 " >> diffbatch.dat # queue_FG + echo "0 " >> diffbatch.dat # queue_GO echo "1 " >> diffbatch.dat # active echo "0 " >> diffbatch.dat # parallel @@ -631,9 +668,9 @@ echo "0 " >> diffobjbatch.dat # datastore_publish echo "catdir.dif " >> diffobjbatch.dat # dvo_label - echo "$OUTDIR/catdir.dif " >> diffobjbatch.dat # dvo_location - echo "9 " >> diffobjbatch.dat # min_ra - echo "11 " >> diffobjbatch.dat # max_ra - echo "19 " >> diffobjbatch.dat # min_dec - echo "21 " >> diffobjbatch.dat # max_dec + echo "$OUTDIR/$CATDIR_DIF " >> diffobjbatch.dat # dvo_location + echo "$coords[1] " >> diffobjbatch.dat # min_ra + echo "$coords[2] " >> diffobjbatch.dat # max_ra + echo "$coords[3] " >> diffobjbatch.dat # min_dec + echo "$coords[4] " >> diffobjbatch.dat # max_dec echo "2 " >> diffobjbatch.dat # box_size echo "$OUTDIR " >> diffobjbatch.dat # base_path @@ -652,4 +689,6 @@ echo "0 " >> diffobjbatch.dat # queue_FW echo "0 " >> diffobjbatch.dat # queue_FO + echo "0 " >> diffobjbatch.dat # queue_FG + echo "0 " >> diffobjbatch.dat # queue_GO echo "1 " >> diffobjbatch.dat # active echo "0 " >> diffobjbatch.dat # parallel Index: branches/czw_branch/20170908/ippToPsps/test/gdb.sh =================================================================== --- branches/czw_branch/20170908/ippToPsps/test/gdb.sh (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/test/gdb.sh (revision 40483) @@ -0,0 +1,21 @@ + +define t1 + set $i = 0 + while ($i < 50) + printf "%d %d %f\n", $i, secfilt[$i].stackBestOff, secfilt[$i].MpsfStk + set $i = $i + 1 + end +end + +define t2 + set $i = 0 + while ($i < Ncstack) + if (cstack[$i]) + printf "%d %s\n", $i, cstack[$i] + else + printf "%d %s\n", $i, "NULL" + end + set $i = $i + 1 + end +end + Index: branches/czw_branch/20170908/ippToPsps/test/mkgpc1.sh =================================================================== --- branches/czw_branch/20170908/ippToPsps/test/mkgpc1.sh (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/test/mkgpc1.sh (revision 40483) @@ -15,4 +15,14 @@ endif +if ("$1" == "user.sudo") then + if ($#argv != 4) goto usage; + set dbhost = $2 + set dbuser = $3 + set dbpass = $4 + + sudo mysql -h $dbhost -u root -p -e "grant all on *.* to $dbuser@"$dbhost" identified by '$dbpass'" + exit 0; +endif + if ("$1" == "create") then if ($#argv < 4) goto usage; @@ -28,4 +38,10 @@ # create a database with completely minimal tables needed to test mysql -h $dbhost -u $dbuser $dbpass -e "CREATE DATABASE $dbname" + if ($status) then + echo "ERROR: failed to create mysql database $dbname ($0)" + echo " NOTE: before running this test, you must have a mysql user:" + echo " mkgpc1.sh user localhost dvo dvo (requires root password)" + exit 1 + endif mysql -h $dbhost -u $dbuser $dbpass $dbname < gpc1schema.sql exit 0; @@ -45,9 +61,10 @@ (mysql -B -h $dbhost -u $dbuser $dbpass -e "show databases" | grep ^$dbname\$) >& /dev/null if ($status) then - echo "database $dbname does not yet exist" + echo "INFO: database $dbname does not yet exist ($0)" exit 0 endif - # mysql -h $dbhost -u $dbuser $dbpass $dbname -e "show tables" + echo mysql -B -h $dbhost -u $dbuser $dbpass $dbname -e "show tables" + mysql -h $dbhost -u $dbuser $dbpass $dbname -e "show tables" mysql -h $dbhost -u $dbuser $dbpass $dbname -e "describe ippToPspsFake" >& /dev/null if ($status) then @@ -91,4 +108,5 @@ endif + echo mysql -B -h $dbhost -u $dbuser $dbpass $dbname -e "show tables" set ntable = `mysql -B -h $dbhost -u $dbuser $dbpass $dbname -e "show tables" | wc -l` echo ntable: $ntable @@ -112,6 +130,13 @@ echo " delete an existing test db (enter pass 3x)" echo "" + echo " mkgpc1.sh forcedelete (dbhost) (dbname) (dbuser) [dbpass]" + echo " delete an existing test db (enter pass 3x), even if the number of tables is wrong" + echo "" echo " mkgpc1.sh user (dbhost) (dbuser) (password)" echo " create a new user and password for the test db" echo "" + echo " mkgpc1.sh user.sudo (dbhost) (dbuser) (password)" + echo " create a new user and password for the test db, using sudo mysql" + echo " (this option is needed if your mysql installation uses the auth_socket plugin" + echo "" exit 2 Index: branches/czw_branch/20170908/ippToPsps/test/mkgpc1data.dr2.dvo =================================================================== --- branches/czw_branch/20170908/ippToPsps/test/mkgpc1data.dr2.dvo (revision 40483) +++ branches/czw_branch/20170908/ippToPsps/test/mkgpc1data.dr2.dvo (revision 40483) @@ -0,0 +1,823 @@ +#!/usr/bin/env dvo +# -*-sh-*- +input tap.dvo + +if (not($?VERBOSE)) set VERBOSE = 0 +if (not($?PARALLEL)) set PARALLEL = 0 + +$SMALLTEST = 1 + +## this script must be run in this directory, output data goes to 'testdata' +$OUTDIR = testdata + +# note that ipptopsps and gpc1db need the path ex extension +# I'm setting the base filename without the .smf and add it in mkcatdir.* + +# some global value for ease of access: +$RA = 9.93 +$DEC = -0.03 +$dRA = 1.0 +$dDEC = 1.0 + +$SKYCELL = skycell.1133.081 +$TESS_ID = RINGS.V3 +$BAD_PSFQF_FRAC = 0.0 +$ADDNOISE = "-no-noise" +$ADDNOISE = "" + +$CATFORMAT = PS1_V5 +$CMFFORMAT = PS1_V5 +$DIFFORMAT = PS1_DV5 + +# a note on double quotes and TESS_ID: +# since the variables TESS_ID and SKYCELL are used below to construct the list, their values should NOT be wrapped in double quotes +# or those double quotes are embedded in the strings in the list +# when TESS_ID and SKYCELL appear in mysql commands as values they need to be protected with double quotes so mysql does not +# interpret them as db table fields + +### XXX some of the ipptopsps functions try to find an image based on EXTERN_ID alone; this should be in combination with SOURCE_ID + +# imagedata describes the full fake exposure set. the fields are +list imagedata +# exp_id stk_id filebase M_off date time warpfile filter difffile + 1 101 $OUTDIR/test.cam 0.000 2010/01/01 01:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 2 101 $OUTDIR/test.cam -0.025 2010/01/01 02:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 3 101 $OUTDIR/test.cam 0.025 2010/01/01 03:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 4 101 $OUTDIR/test.cam 0.010 2010/01/01 04:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 5 102 $OUTDIR/test.cam 0.000 2010/01/01 05:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 6 102 $OUTDIR/test.cam -0.025 2010/01/01 06:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 7 102 $OUTDIR/test.cam 0.025 2010/01/01 07:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 8 102 $OUTDIR/test.cam 0.010 2010/01/01 08:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 9 103 $OUTDIR/test.cam 0.000 2010/01/01 09:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 10 103 $OUTDIR/test.cam -0.025 2010/01/01 10:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 11 103 $OUTDIR/test.cam 0.025 2010/01/01 11:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 12 103 $OUTDIR/test.cam 0.010 2010/01/01 12:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 13 104 $OUTDIR/test.cam -0.025 2010/01/02 01:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 14 104 $OUTDIR/test.cam 0.025 2010/01/02 02:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 15 104 $OUTDIR/test.cam 0.010 2010/01/02 03:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 16 104 $OUTDIR/test.cam 0.000 2010/01/02 04:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 17 105 $OUTDIR/test.cam -0.025 2010/01/02 05:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 18 105 $OUTDIR/test.cam 0.025 2010/01/02 06:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 19 105 $OUTDIR/test.cam 0.010 2010/01/02 07:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 20 105 $OUTDIR/test.cam 0.000 2010/01/02 08:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 21 106 $OUTDIR/test.cam -0.025 2010/01/02 09:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 22 106 $OUTDIR/test.cam 0.025 2010/01/02 10:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 23 106 $OUTDIR/test.cam 0.010 2010/01/02 11:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 24 106 $OUTDIR/test.cam -0.020 2010/01/02 12:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff +end + +list stackdata +# stk_id sky_id filename M_off filter forcedgalaxy + 101 1 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.010 g $OUTDIR/$TESS_ID.$SKYCELL.fgal + 102 2 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.020 g $OUTDIR/$TESS_ID.$SKYCELL.fgal + 103 3 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.030 g $OUTDIR/$TESS_ID.$SKYCELL.fgal + 104 1 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.025 r $OUTDIR/$TESS_ID.$SKYCELL.fgal + 105 2 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.035 r $OUTDIR/$TESS_ID.$SKYCELL.fgal + 106 3 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.045 r $OUTDIR/$TESS_ID.$SKYCELL.fgal +end + +macro mkfull + init.db + + mkcatdir.cam $CMFFORMAT $CATFORMAT + mkcatdir.stk $CMFFORMAT $CATFORMAT + mkcatdir.wrp $CMFFORMAT\_Lensing $CATFORMAT + mkcatdir.dif $DIFFORMAT PS1_V6 + insert.stack.set + + exec rm -rf testdata/catdir.mrg + + # if CATFORMAT is not specified, dvomerge will default to the .ptolemyrc version + # CATFORMAT, SKY_DEPTH here must match the addstar calls + # (or else I should copy the first catdir instead of using dvomerge) + # (or else dvomerge should be smart enough to inherit these from the input if the output does not exist) + exec dvomerge -matched-tables testdata/catdir.cam into testdata/catdir.mrg -D CATFORMAT $CATFORMAT -D SKY_DEPTH 4 + exec dvomerge -matched-tables testdata/catdir.stk into testdata/catdir.mrg + exec dvomerge -matched-tables testdata/catdir.wrp into testdata/catdir.mrg + + # basic relphot calibration, imitating the original PV3.3 calibration + echo relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 -use-mcal-psf-for-stack-aper + exec relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 -use-mcal-psf-for-stack-aper + + # test parallel-regions version, updating only stacks and warps, and applying the McalPSF vs McalAPER + exec cp RegionHost.dat testdata/catdir.mrg + + # parallel version is now setting the format in my local build + echo relphot -parallel-regions -region-hosts RegionHost.dat -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv g,r -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 -imfreeze -mosfreeze -only-stacks-and-warps -update-catformat PS1_V6 -nloop 0 + exec relphot -parallel-regions -region-hosts RegionHost.dat -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv g,r -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 -imfreeze -mosfreeze -only-stacks-and-warps -update-catformat PS1_V6 -nloop 0 + + # echo relphot -images -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv g,r -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 -imfreeze -mosfreeze -only-stacks-and-warps -update-catformat PS1_V6 -nloop 0 + # exec relphot -images -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv g,r -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 -imfreeze -mosfreeze -only-stacks-and-warps -update-catformat PS1_V6 -nloop 0 +end + +macro init.db + + $dbhost = localhost + $dbname = gpc1test + $dbuser = dvo + $dbpass = dvo + + $gscfile = `gconfig GSCFILE` + if ("$gscfile" == "not found") + echo "problem with dvo / ptolemy configuration. check ~/.ptolemyrc" + break + end + + exec mkgpc1.sh delete $dbhost $dbname $dbuser $dbpass + exec mkgpc1.sh create $dbhost $dbname $dbuser $dbpass + + dbconnect $dbhost $dbuser $dbname -p $dbpass + + # system-wide info + dbinsert ippToPspsFake (my_id) values (2) + dbinsert skycell (radeg, decdeg, tess_id, skycell_id) values ($RA, $DEC, "$TESS_ID", "$SKYCELL") +end + +# create a populated catdir with a set of cmf files, save the filenames in list for use by insert.exp +macro mkcatdir.cam + if ($0 != 3) + echo "mkcatdir.cam (cmftype) (dvotype)" + break + end + + # tapPLAN 4 + + mkdir $OUTDIR + + local i catdir ID stkID rawfile cmffile offset + $catdir = $OUTDIR/catdir.cam + exec rm -rf $catdir + + $TIMEFORMAT = mjd + $TIMEREF = 2001/01/01,00:00:00 + + for i 0 $imagedata:n + list word -split $imagedata:$i + $ID = $word:0 + $stkID = $word:1 + sprintf rawfile "%s.%02d.txt" $word:2 $ID + sprintf cmffile "%s.%02d.smf" $word:2 $ID + $offset = $word:3 + sprintf myDATE "%s" $word:4 + sprintf myTIME "%s" $word:5 + $filter = $word:7 + + if ($SMALLTEST && ($stkID != 101) && ($stkID != 104)) continue + + echo ctimes -abs $myDATE,$myTIME -var mjd + ctimes -abs $myDATE,$myTIME -var mjd + + # XXX for a simple test of ippToPsps, i need to generate smf files + # with some correspondence to gpc1 exposure smfs + + # this means: + # 1) a PHU with some basic header data + # 2) add extensions with EXTNAME of XYnn + + # NOTE: mkcmf does not populate a PHU, so we have to fake it + + # create an empty header and populate with the desired keywords + mcreate dummy 0 0 + keyword dummy MJD-OBS -wf $mjd + keyword dummy FILTERID -w $filter.00000 + keyword dummy EXPTIME -wf 1.0 + + keyword dummy ZPT_ERR -wf 0.01 + keyword dummy EXPREQ -wf 1.0 + keyword dummy AIRMASS -wf 1.0 + keyword dummy RA -wf $RA + keyword dummy DEC -wf $DEC + keyword dummy CTYPE1 -w "RA---DIS" + keyword dummy CTYPE2 -w "DEC--DIS" + keyword dummy CRVAL1 -wf $RA + keyword dummy CRVAL2 -wf $DEC + keyword dummy CRPIX1 -wf 0.0 + keyword dummy CRPIX2 -wf 0.0 + keyword dummy CDELT1 -wf {1.0/3600} + keyword dummy CDELT2 -wf {1.0/3600} + keyword dummy PC001001 -wf 1.0 + keyword dummy PC001002 -wf 0.0 + keyword dummy PC002001 -wf 0.0 + keyword dummy PC002002 -wf 1.0 + keyword dummy NPLYTERM -wd 0 + keyword dummy PCA1X3Y0 -wf 0.0 + keyword dummy PCA1X2Y1 -wf 0.0 + keyword dummy PCA1X1Y2 -wf 0.0 + keyword dummy PCA1X0Y3 -wf 0.0 + keyword dummy PCA1X2Y0 -wf 0.0 + keyword dummy PCA1X1Y1 -wf 0.0 + keyword dummy PCA1X0Y2 -wf 0.0 + keyword dummy PCA2X3Y0 -wf 0.0 + keyword dummy PCA2X2Y1 -wf 0.0 + keyword dummy PCA2X1Y2 -wf 0.0 + keyword dummy PCA2X0Y3 -wf 0.0 + keyword dummy PCA2X2Y0 -wf 0.0 + keyword dummy PCA2X1Y1 -wf 0.0 + keyword dummy PCA2X0Y2 -wf 0.0 + wd dummy $cmffile + + mkinput $offset $rawfile + echo mkcmf -photcode SIMTEST.$filter.Chip $ADDNOISE -append $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 0 -coords + exec mkcmf -photcode SIMTEST.$filter.Chip $ADDNOISE -append $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 0 -coords + + if ($i == 3) + echo "NOTE: not adding image $i to DVO : this simulates images with poor astrometry" + continue + end + + if (0 && ($i == 5)) + echo "NOTE: adding image $i 2x to DVO, but without detections : this simulates a failure in dvo contruction" + + echo addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -image -dup-images + exec addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -image -dup-images + end + + if (0 && ($i == 7)) + echo "NOTE: adding detections from image $i 2x to DVO, but with wrong photcode and duplicate image ID : this simulates a failure in dvo construction" + echo addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -dup-images -image-id-override 7 -photcode 2MASS_J + exec addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -dup-images -image-id-override 7 -photcode 2MASS_J + end + + echo addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass + exec addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass + end + + if ($SMALLTEST) + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update + + echo relastro -v -D CATDIR $catdir -D CAMERA simtest -images -update-chips -update-all-cameras -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -apply-offsets + exec relastro -v -D CATDIR $catdir -D CAMERA simtest -images -update-chips -update-all-cameras -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -apply-offsets + else + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + end + + if ($PARALLEL) + $hostname = `hostname` + exec rm -f $catdir/HostTable.dat + output $catdir/HostTable.dat + echo "# ID Hostname Catdir" + echo " 1 $hostname $catdir.p1" + echo " 2 $hostname $catdir.p2" + echo " 3 $hostname $catdir.p3" + output stdout + exec dvodist -out $catdir + end + + # for i 0 $offset:n + # tapOK {abs(Mcal[$i] - Mcal[0] - $offset:$i) < 0.001} "Mcal $i" + # end + + # tapDONE +end + +# create a populated catdir with a set of cmf files, save the filenames in list for use by insert.exp +macro mkcatdir.stk + if ($0 != 3) + echo "mkcatdir.stk (cmftype) (dvotype)" + break + end + + # tapPLAN 4 + + mkdir $OUTDIR + + local i catdir stkID skyID rawfile cmffile offset + $catdir = $OUTDIR/catdir.stk + exec rm -rf $catdir + + for i 0 $stackdata:n + list word -split $stackdata:$i + $stkID = $word:0 + $skyID = $word:1 + sprintf rawfile "%s.%02d.txt" $word:2 $stkID + sprintf cmffile "%s.%02d.cmf" $word:2 $stkID + $offset = $word:3 + $filter = $word:4 + + ## this is not generating photcodes recognized as stack by relphot + if ($SMALLTEST && ($stkID != 101) && ($stkID != 104)) continue + + mkinput $offset $rawfile + echo mkcmf -stack -photcode SIMTEST.$filter.SkyChip $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords -aper-offset 0.15 + exec mkcmf -stack -photcode SIMTEST.$filter.SkyChip $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords -aper-offset 0.15 + + # add some required header keywords: + # exec echo "HIERARCH FPA.ZP = {25.+$offset} / Magnitude zero point" > tmp.hdr + # exec fits_insert $cmffile tmp.hdr + + echo addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -D ZERO_POINT_OPTION CHIP_HEADER + exec addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -D ZERO_POINT_OPTION CHIP_HEADER + end + + if ($SMALLTEST) + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -averages -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -averages -update + else + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + end + + if ($PARALLEL) + $hostname = `hostname` + exec rm -f $catdir/HostTable.dat + output $catdir/HostTable.dat + echo "# ID Hostname Catdir" + echo " 1 $hostname $catdir.p1" + echo " 2 $hostname $catdir.p2" + echo " 3 $hostname $catdir.p3" + output stdout + exec dvodist -out $catdir + end + + # for i 0 $offset:n + # tapOK {abs(Mcal[$i] - Mcal[0] - $offset:$i) < 0.001} "Mcal $i" + # end + + # tapDONE +end + +# create a populated catdir with a set of cmf files, save the filenames in list for use by insert.exp +macro mkcatdir.wrp + if ($0 != 3) + echo "mkcatdir.wrp (cmftype) (dvotype)" + break + end + + # tapPLAN 4 + + mkdir $OUTDIR + + local i catdir ID stkID rawfile cmffile offset + $catdir = $OUTDIR/catdir.wrp + exec rm -rf $catdir + + $TIMEFORMAT = mjd + $TIMEREF = 2001/01/01,00:00:00 + + for i 0 $imagedata:n + list word -split $imagedata:$i + $ID = $word:0 + $stkID = $word:1 + sprintf rawfile "%s.%02d.txt" $word:2 $ID; # use the cam-stage rawfile + sprintf cmffile "%s.%02d.cmf" $word:6 $ID + $offset = $word:3 + sprintf myDATE "%s" $word:4 + sprintf myTIME "%s" $word:5 + $filter = $word:7 + + if ($SMALLTEST && ($stkID != 101) && ($stkID != 104)) continue + + echo ctimes -abs $myDATE,$myTIME -var mjd + ctimes -abs $myDATE,$myTIME -var mjd + + ## this is not generating photcodes recognized as stack by relphot + + mkinput $offset $rawfile + echo mkcmf -forcedwarp -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 2 -tess_id $TESS_ID -skycell $SKYCELL -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + exec mkcmf -forcedwarp -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 2 -tess_id $TESS_ID -skycell $SKYCELL -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + + echo addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -xrad + exec addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -xrad + end + + # add the forcedgalaxy files for the above warps: + for i 0 $stackdata:n + list word -split $stackdata:$i + $stkID = $word:0 + sprintf rawfile "%s.%02d.txt" $word:2 $stkID + sprintf cmffile "%s.%02d.smf" $word:5 $stkID + $offset = $word:3 + $filter = $word:4 + + ## this is not generating photcodes recognized as stack by relphot + if ($SMALLTEST && ($stkID != 101) && ($stkID != 104)) continue + + echo mkcmf -forcedgalaxy -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords + exec mkcmf -forcedgalaxy -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords + + echo loadgalphot -D CATDIR $catdir -D CAMERA simtest -photcode SIMTEST.$filter.ForcedWarp -image-id $stkID $cmffile + exec loadgalphot -D CATDIR $catdir -D CAMERA simtest -photcode SIMTEST.$filter.ForcedWarp -image-id $stkID $cmffile + end + + if ($SMALLTEST) + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -averages -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -averages -update + else + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + end + + echo dvolens -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update-objects -update + exec dvolens -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update-objects -update + + if ($PARALLEL) + $hostname = `hostname` + exec rm -f $catdir/HostTable.dat + output $catdir/HostTable.dat + echo "# ID Hostname Catdir" + echo " 1 $hostname $catdir.p1" + echo " 2 $hostname $catdir.p2" + echo " 3 $hostname $catdir.p3" + output stdout + exec dvodist -out $catdir + end + + # for i 0 $offset:n + # tapOK {abs(Mcal[$i] - Mcal[0] - $offset:$i) < 0.001} "Mcal $i" + # end + + # tapDONE +end + +macro insert.exp + if ($0 != 4) + echo "USAGE: insert.exp (ID) (filename) (filter)" + break + end + + local myID filename filter + $myID = $1 + $filename = $2 + $filter = $3 + + # a single exposure + echo $myDATE + echo $myTIME + + dbinsert rawExp (exp_id, exp_name, exp_time, filter, bg, bg_stdev, ra, decl, dateobs) values ($myID, "test.$myID", 10.0, "$filter.00000", 100.0, 2.0, {$RA*3.1415/180.0}, {$DEC*3.1415/180.0}, '$myDATE $myTIME') + dbinsert chipRun (exp_id, chip_id) values ($myID, $myID) + dbinsert camRun (chip_id, cam_id, dist_group, software_ver) values ($myID, $myID, "testgroup", "38000M") + dbinsert camProcessedExp (cam_id, path_base, fault) values ($myID, "$filename", 0) + dbinsert fakeRun (fake_id, cam_id) values ($myID, $myID) + dbinsert warpRun (warp_id, fake_id, tess_id, state) values ($myID, $myID, "$TESS_ID", "full") + dbinsert warpSkyCellMap (warp_id, skycell_id, class_id) values ($myID, "$SKYCELL", "XY00") + dbinsert warpImfile (warp_id, skycell_id, warp_skyfile_id) values ($myID, "$SKYCELL", $myID) + + dbinsert addRun (add_id, stage, state, stage_id, minidvodb_name) values ($myID, "cam", "full", $myID, "catdir.cam") + dbinsert addProcessedExp (add_id, fault) values ($myID, 0) +end + +macro insert.wrp + if ($0 != 4) + echo "USAGE: insert.wrp (warpID) (stackID) (filename)" + break + end + + local wrpID stkID filename + $wrpID = $1 + $stkID = $2 + $filename = $3 + + # a single exposure + echo $myDATE + echo $myTIME + + dbinsert addRun (add_id, stage, state, stage_id, stage_extra1, minidvodb_name) values ($wrpID, "fullforce", "full", $stkID, $wrpID, "catdir.wrp") + + # we have a single fullForceRun for each stack, so we can match fullForceRun ids to stackRun ids + dbinsert fullForceInput (ff_id, warp_id) values ($stkID, $wrpID) + dbinsert fullForceResult (ff_id, warp_id, software_ver, path_base) values ($stkID, $wrpID, "38000M", "$filename") +end + +# this inserts a single stack, adding the exposures associated with this stack +macro insert.stack + if ($0 != 5) + echo "USAGE: insert.stack (stkID) (skyID) (stkfile) (filter)" + break + end + + local i stkID skyID expID expfile wrpfile stkfile filter + + $stkID = $1 + $skyID = $2 + $stkfile = $3 + $filter = $4 + + $skycalID = $stkID + 20 + + dbinsert stackRun (stack_id, skycell_id, filter, software_ver) values ($stkID, "$SKYCELL", "$filter.00000", "37500") + + # we have a single fullForceRun for each stack, so we can match fullForceRun ids to stackRun ids + dbinsert fullForceRun (ff_id, skycal_id) values ($stkID, $skycalID) + + $TIMEFORMAT = mjd + $TIMEREF = 2001/01/01,00:00:00 + + $mjdsum = 0.0 + $Nstk = 0 + + for i 0 $imagedata:n + echo $imagedata:$i + list tmp -split $imagedata:$i + if ($stkID != $tmp:1) continue + + $expID = $tmp:0 + sprintf myDATE "%s" $tmp:4 + sprintf myTIME "%s" $tmp:5 + +# sprintf rawfile "%s.%02d.txt" $tmp:2 $ID + sprintf expfile "%s.%02d" $tmp:2 $expID + sprintf wrpfile "%s.%02d" $tmp:6 $expID + + $myFilter = $tmp:7 + insert.exp $expID $expfile $myFilter + if ($myFilter != $filter) + echo "*** WARNING: filter mis-match between exp and stk: $myFilter vs $filter" + break + end + + insert.wrp $expID $stkID $wrpfile + + dbinsert stackInputSkyfile (stack_id, warp_id) values ($stkID, $expID) + + echo ctimes -abs $myDATE,$myTIME -var mjd + ctimes -abs $myDATE,$myTIME -var mjd + $mjdsum += $mjd + $Nstk ++ + end + $mjd = $mjdsum / $Nstk + + dbinsert stackSumSkyfile (stack_id, mjd_obs) values ($stkID, $mjd) + dbinsert skycalRun (sky_id, skycal_id, stack_id) values ($skyID, $skycalID, $stkID) + dbinsert skycalResult (skycal_id, path_base) values ($skycalID, "$stkfile") + + dbinsert addRun (add_id, stage, state, stage_id, minidvodb_name) values ($stkID, "skycal", "full", $skycalID, "catdir.stk") +end + +macro mkcatdir.dif + if ($0 != 3) + echo "mkcatdir.cam (cmftype) (dvotype)" + break + end + + # tapPLAN 4 + + mkdir $OUTDIR + + local i catdir ID stkID rawfile cmffile offset + $catdir = $OUTDIR/catdir.dif + exec rm -rf $catdir + + $TIMEFORMAT = mjd + $TIMEREF = 2001/01/01,00:00:00 + + for i 0 $imagedata:n + list word -split $imagedata:$i + $ID = $word:0 + $stkID = $word:1 + sprintf rawfile "%s.%02d.txt" $word:2 $stkID + sprintf cmffile "%s.%02d.smf" $word:8 $stkID + sprintf invfile "%s.%02d.inv.smf" $word:8 $stkID + $offset = $word:3 + sprintf myDATE "%s" $word:4 + sprintf myTIME "%s" $word:5 + $filter = $word:7 + + if ($SMALLTEST && ($stkID != 101) && ($stkID != 104)) continue + + echo ctimes -abs $myDATE,$myTIME -var mjd + ctimes -abs $myDATE,$myTIME -var mjd + + mkinput $offset $rawfile + $diff_skyfile_id = $ID + 25 + echo mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + exec mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + + echo mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $invfile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + exec mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $invfile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + + insert.dif $ID $ID $stkID $word:8 + + if ($i == 3) + echo "NOTE: not adding image $i to DVO : this simulates images with poor astrometry" + continue + end + + exec rm -rf addstar.txt + output addstar.txt + echo $cmffile + echo $invfile + output stdout + echo addstar -diff-inv -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest -list addstar.txt -D CATFORMAT $2 -quick-airmass + exec addstar -diff-inv -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest -list addstar.txt -D CATFORMAT $2 -quick-airmass + end + if ($SMALLTEST) + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update -boundary-tree tess.3pi.fits -is-diff-db -statmode WT_MEAN + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update -boundary-tree tess.3pi.fits -is-diff-db -statmode WT_MEAN + else + echo relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -images g,r,i,z,y -update + end + + if ($PARALLEL) + $hostname = `hostname` + exec rm -f $catdir/HostTable.dat + output $catdir/HostTable.dat + echo "# ID Hostname Catdir" + echo " 1 $hostname $catdir.p1" + echo " 2 $hostname $catdir.p2" + echo " 3 $hostname $catdir.p3" + output stdout + exec dvodist -out $catdir + end + + # for i 0 $offset:n + # tapOK {abs(Mcal[$i] - Mcal[0] - $offset:$i) < 0.001} "Mcal $i" + # end + + # tapDONE +end + + +macro insert.stack.set + + local i stkID skyID cmffile filter + + for i 0 $stackdata:n + list word -split $stackdata:$i + $stkID = $word:0 + $skyID = $word:1 + sprintf cmffile "%s.%02d" $word:2 $stkID + $filter = $word:4 + + if ($SMALLTEST && ($stkID != 101) && ($stkID != 104)) continue + + insert.stack $stkID $skyID $cmffile $filter + end + + # I am setting these databases in a very fake way, with same in and out names! + + # catdirs for the exposures + dbinsert minidvodbRun (minidvodb_name, minidvodb_id, state) values ("catdir.cam", 1, "merged") + dbinsert minidvodbProcessed (minidvodb_id, fault) values (1, 0) + + dbinsert mergedvodbRun (minidvodb_id, merge_id, state, mergedvodb) values (1, 1, "full", "catdir.cam") + dbinsert mergedvodbProcessed (merge_id, fault) values (1, 0) + + # catdirs for the stack + dbinsert minidvodbRun (minidvodb_name, minidvodb_id, state) values ("catdir.stk", 2, "merged") + dbinsert minidvodbProcessed (minidvodb_id, fault) values (2, 0) + + dbinsert mergedvodbRun (minidvodb_id, merge_id, state, mergedvodb) values (2, 2, "full", "catdir.stk") + dbinsert mergedvodbProcessed (merge_id, fault) values (2, 0) + + # catdirs for the forced warps + dbinsert minidvodbRun (minidvodb_name, minidvodb_id, state) values ("catdir.wrp", 3, "merged") + dbinsert minidvodbProcessed (minidvodb_id, fault) values (3, 0) + + dbinsert mergedvodbRun (minidvodb_id, merge_id, state, mergedvodb) values (3, 3, "full", "catdir.wrp") + dbinsert mergedvodbProcessed (merge_id, fault) values (3, 0) + + # catdirs for the diffs + dbinsert minidvodbRun (minidvodb_name, minidvodb_id, state) values ("catdir.dif", 4, "merged") + dbinsert minidvodbProcessed (minidvodb_id, fault) values (4, 0) + + dbinsert mergedvodbRun (minidvodb_id, merge_id, state, mergedvodb) values (4, 4, "full", "catdir.dif") + dbinsert mergedvodbProcessed (merge_id, fault) values (4, 0) + +end + +macro insert.dif + if ($0 != 5) + echo "USAGE: insert.dif (diffID) (warpID) (stackID) (filename)" + break + end + + local difID wrpID stkID filename + $difID = $1 + $wrpID = $2 + $stkID = $3 + $filename = $4 + + echo $myDATE + echo $myTIME + + $addID = $difID + 100 + $diffSkyfileID = $difID + 25; + dbinsert addRun (add_id, stage, state, stage_id, stage_extra1, minidvodb_name) values ($addID, "diff", "full", $difID, $diffSkyfileID, "catdir.dif") + dbinsert addProcessedExp (add_id, fault) values ($addID, 0) + + dbinsert diffRun (diff_id, tess_id, state, diff_mode, software_ver) values ($difID, "$TESS_ID", "full", 2, "38000M") + dbinsert diffInputSkyfile (diff_id, warp1, stack2, diff_skyfile_id, tess_id, skycell_id) values ($difID, $wrpID, $stkID, $diffSkyfileID, "$TESS_ID", "$SKYCELL" ) + dbinsert diffSkyfile (diff_id, skycell_id, path_base) values ($difID, "$SKYCELL", "$filename") +end + +# make a simple input file for mkcmf +macro mkinput + if ($0 != 3) + echo "mkinput (offset) (output)" + break + end + + local output + $output = $2 + + exec rm -f $output + + local i j ra dec + output $output + for i 10 1024 100 + for j 10 1024 100 + $ra = $RA + $i*0.25/3600 + $dec = $DEC + $j*0.25/3600 + fprintf " %10.6f %10.6f %7.3f %6.1f %6.1f %7.3f" $ra $dec {10.0 + $1 + 2.5*($i + $j)/1000.0} $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + end + end + # XXX add 2 duplicate detections + $i = 110; $j = 210 + # fprintf " %10.6f %10.6f %7.3f %6.1f %6.1f %7.3f" $ra $dec {10.0 + $1 + 2.5*($i + $j)/1000.0} $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + # fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + $i = 210; $j = 110 + # fprintf " %10.6f %10.6f %7.3f %6.1f %6.1f %7.3f" $ra $dec {10.0 + $1 + 2.5*($i + $j)/1000.0} $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + # fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + output stdout +end + +macro check.mags + if ($0 != 3) + echo "USAGE: check.mags (catdir) (ver)" + break + end + + # NOTE: relphot refuses to set the warp averages if there is no boundary tree: otherwise it cannot choose the correct subset of measurements + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv -averages -boundary-tree tess.3pi.fits + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits + + catdir $1 + skyregion {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} + avextract ra dec g:psf:chp g:kron:chp g:psf:stk g:kron:stk g:psf:wrp g:kron:wrp + + set dg:psf:chp:stk = g:psf:chp - g:psf:stk + set dg:psf:chp:wrp = g:psf:chp - g:psf:wrp + + set dg:kron:chp:stk = g:kron:chp - g:kron:stk + set dg:kron:chp:wrp = g:kron:chp - g:kron:wrp + + set dg:psf:kron:chp = g:psf:chp - g:kron:chp + set dg:psf:kron:stk = g:psf:stk - g:kron:stk + set dg:psf:kron:wrp = g:psf:wrp - g:kron:wrp + + lim -n 0$2 g:psf:chp -0.2 0.2; clear; box; plot g:psf:chp dg:psf:chp:stk -c blue -pt 7; plot g:psf:chp dg:psf:chp:wrp -c red -pt 2 + label -x "g:psf:chp" -y "g:psf:chp - g:psf:stk(blue),wrp(red)" + resize 600 400 + + lim -n 1$2 g:kron:chp -0.2 0.2; clear; box; plot g:kron:chp dg:kron:chp:stk -c blue -pt 7; plot g:kron:chp dg:kron:chp:wrp -c red -pt 2 + label -x "g:kron:chp" -y "g:kron:chp - g:kron:stk(blue),wrp(red)" + resize 600 400 + + lim -n 2$2 g:psf:chp -0.2 0.2; clear; box; plot g:psf:chp dg:psf:kron:chp -c green -pt 7; plot g:psf:chp dg:psf:kron:stk -c blue -pt 2; plot g:psf:chp dg:psf:kron:wrp -c red -pt 2 + label -x "g:psf:chp" -y "g:psf:chp - g:kron:chp(green),stk(blue),wrp(red)" + resize 600 400 + +# for i 0 10 +# echo ra[$i] dec[$i] g:psf:chp[$i] g:kron:chp[$i] g:psf:stk[$i] g:psf:wrp[$i] +# end +end + +macro check.mags.meas + if ($0 != 3) + echo "USAGE: check.mags.meas (catdir) (ver)" + break + end + + # NOTE: relphot refuses to set the warp averages if there is no boundary tree: otherwise it cannot choose the correct subset of measurements + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv -averages -boundary-tree tess.3pi.fits + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits + + catdir $1 + skyregion {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} + + mextract ra dec mag:rel:psf mag:rel:kron mag:ave:psf mag:ave:kron photcode + set dM:psf = mag:ave:psf - mag:rel:psf + lim -n 0$2 mag:ave:psf dM:psf; clear; box; plot mag:ave:psf dM:psf + label -x "mag:ave:psf" -y "mag:ave:psf - mag:rel:psf" + resize 600 400 + + set dM:kron = mag:ave:kron - mag:rel:kron + lim -n 1$2 mag:ave:kron dM:kron; clear; box; plot mag:ave:kron dM:kron + label -x "mag:ave:kron" -y "mag:ave:kron - mag:rel:kron" + resize 600 400 + + set dM:psf:kron = mag:rel:psf - mag:rel:kron + lim -n 2$2 mag:rel:kron dM:psf:kron; clear; box; plot mag:rel:kron dM:psf:kron + label -x "mag:rel:kron" -y "mag:rel:psf - mag:rel:kron" + resize 600 400 +end + +if ($SCRIPT) + if ($argv:n > 0) + if ("$argv:0" == "-parallel") + $PARALLEL = 1 + end + if ("$argv:0" == "-dump-region") + echo {$RA - $dRA} {$RA + $dRA} {$DEC - $dDEC} {$DEC + $dDEC} + exit 0 + end + end + mkfull + if (not($STATUS)) exit 1 + exit 0 +end + Index: branches/czw_branch/20170908/ippToPsps/test/mkgpc1data.dvo =================================================================== --- branches/czw_branch/20170908/ippToPsps/test/mkgpc1data.dvo (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/test/mkgpc1data.dvo (revision 40483) @@ -17,45 +17,52 @@ $RA = 10.0 $DEC = 20.0 -$SKYCELL = "skycell.1133.081" -$TESS_ID = "RINGS.V3" -$BAD_PSFQF_FRAC = 0.2 +$SKYCELL = skycell.1133.081 +$TESS_ID = RINGS.V3 +$BAD_PSFQF_FRAC = 0.0 +$ADDNOISE = "-no-noise" + +# a note on double quotes and TESS_ID: +# since the variables TESS_ID and SKYCELL are used below to construct the list, their values should NOT be wrapped in double quotes +# or those double quotes are embedded in the strings in the list +# when TESS_ID and SKYCELL appear in mysql commands as values they need to be protected with double quotes so mysql does not +# interpret them as db table fields # imagedata describes the full fake exposure set. the fields are list imagedata -# exp_id stk_id filebase M_off date time warpfile filter difffile - 1 1 $OUTDIR/test.01 0.000 2010/01/01 01:00:00 $OUTDIR/warp.01 g $OUTDIR/diff.01 - 2 1 $OUTDIR/test.02 -0.025 2010/01/01 02:00:00 $OUTDIR/warp.02 g $OUTDIR/diff.02 - 3 1 $OUTDIR/test.03 0.025 2010/01/01 03:00:00 $OUTDIR/warp.03 g $OUTDIR/diff.03 - 4 1 $OUTDIR/test.04 0.010 2010/01/01 04:00:00 $OUTDIR/warp.04 g $OUTDIR/diff.04 - 5 2 $OUTDIR/test.05 0.000 2010/01/01 05:00:00 $OUTDIR/warp.05 g $OUTDIR/diff.05 - 6 2 $OUTDIR/test.06 -0.025 2010/01/01 06:00:00 $OUTDIR/warp.06 g $OUTDIR/diff.06 - 7 2 $OUTDIR/test.07 0.025 2010/01/01 07:00:00 $OUTDIR/warp.07 g $OUTDIR/diff.07 - 8 2 $OUTDIR/test.08 0.010 2010/01/01 08:00:00 $OUTDIR/warp.08 g $OUTDIR/diff.08 - 9 3 $OUTDIR/test.09 0.000 2010/01/01 09:00:00 $OUTDIR/warp.09 g $OUTDIR/diff.09 - 10 3 $OUTDIR/test.10 -0.025 2010/01/01 10:00:00 $OUTDIR/warp.10 g $OUTDIR/diff.10 - 11 3 $OUTDIR/test.11 0.025 2010/01/01 11:00:00 $OUTDIR/warp.11 g $OUTDIR/diff.11 - 12 3 $OUTDIR/test.12 0.010 2010/01/01 12:00:00 $OUTDIR/warp.12 g $OUTDIR/diff.12 - 13 4 $OUTDIR/test.13 -0.025 2010/01/02 01:00:00 $OUTDIR/warp.13 r $OUTDIR/diff.13 - 14 4 $OUTDIR/test.14 0.025 2010/01/02 02:00:00 $OUTDIR/warp.14 r $OUTDIR/diff.14 - 15 4 $OUTDIR/test.15 0.010 2010/01/02 03:00:00 $OUTDIR/warp.15 r $OUTDIR/diff.15 - 16 4 $OUTDIR/test.16 0.000 2010/01/02 04:00:00 $OUTDIR/warp.16 r $OUTDIR/diff.16 - 17 5 $OUTDIR/test.17 -0.025 2010/01/02 05:00:00 $OUTDIR/warp.17 r $OUTDIR/diff.17 - 18 5 $OUTDIR/test.18 0.025 2010/01/02 06:00:00 $OUTDIR/warp.18 r $OUTDIR/diff.18 - 19 5 $OUTDIR/test.19 0.010 2010/01/02 07:00:00 $OUTDIR/warp.19 r $OUTDIR/diff.19 - 20 5 $OUTDIR/test.20 0.000 2010/01/02 08:00:00 $OUTDIR/warp.20 r $OUTDIR/diff.20 - 21 6 $OUTDIR/test.21 -0.025 2010/01/02 09:00:00 $OUTDIR/warp.21 r $OUTDIR/diff.21 - 22 6 $OUTDIR/test.22 0.025 2010/01/02 10:00:00 $OUTDIR/warp.22 r $OUTDIR/diff.22 - 23 6 $OUTDIR/test.23 0.010 2010/01/02 11:00:00 $OUTDIR/warp.23 r $OUTDIR/diff.23 - 24 6 $OUTDIR/test.24 -0.020 2010/01/02 12:00:00 $OUTDIR/warp.24 r $OUTDIR/diff.24 +# exp_id stk_id filebase M_off date time warpfile filter difffile + 1 1 $OUTDIR/test.cam 0.000 2010/01/01 01:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 2 1 $OUTDIR/test.cam -0.025 2010/01/01 02:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 3 1 $OUTDIR/test.cam 0.025 2010/01/01 03:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 4 1 $OUTDIR/test.cam 0.010 2010/01/01 04:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 5 2 $OUTDIR/test.cam 0.000 2010/01/01 05:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 6 2 $OUTDIR/test.cam -0.025 2010/01/01 06:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 7 2 $OUTDIR/test.cam 0.025 2010/01/01 07:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 8 2 $OUTDIR/test.cam 0.010 2010/01/01 08:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 9 3 $OUTDIR/test.cam 0.000 2010/01/01 09:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 10 3 $OUTDIR/test.cam -0.025 2010/01/01 10:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 11 3 $OUTDIR/test.cam 0.025 2010/01/01 11:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 12 3 $OUTDIR/test.cam 0.010 2010/01/01 12:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp g $OUTDIR/$TESS_ID.$SKYCELL.diff + 13 4 $OUTDIR/test.cam -0.025 2010/01/02 01:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 14 4 $OUTDIR/test.cam 0.025 2010/01/02 02:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 15 4 $OUTDIR/test.cam 0.010 2010/01/02 03:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 16 4 $OUTDIR/test.cam 0.000 2010/01/02 04:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 17 5 $OUTDIR/test.cam -0.025 2010/01/02 05:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 18 5 $OUTDIR/test.cam 0.025 2010/01/02 06:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 19 5 $OUTDIR/test.cam 0.010 2010/01/02 07:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 20 5 $OUTDIR/test.cam 0.000 2010/01/02 08:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 21 6 $OUTDIR/test.cam -0.025 2010/01/02 09:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 22 6 $OUTDIR/test.cam 0.025 2010/01/02 10:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 23 6 $OUTDIR/test.cam 0.010 2010/01/02 11:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff + 24 6 $OUTDIR/test.cam -0.020 2010/01/02 12:00:00 $OUTDIR/$TESS_ID.$SKYCELL.wrp r $OUTDIR/$TESS_ID.$SKYCELL.diff end list stackdata -# stk_id sky_id filename M_off filter forcedgalaxy - 1 1 $OUTDIR/stack.01 0.010 g $OUTDIR/fgal.01 - 2 2 $OUTDIR/stack.02 0.020 g $OUTDIR/fgal.02 - 3 3 $OUTDIR/stack.03 0.030 g $OUTDIR/fgal.03 - 4 1 $OUTDIR/stack.04 0.025 r $OUTDIR/fgal.04 - 5 2 $OUTDIR/stack.05 0.035 r $OUTDIR/fgal.05 - 6 3 $OUTDIR/stack.06 0.045 r $OUTDIR/fgal.06 +# stk_id sky_id filename M_off filter forcedgalaxy + 1 1 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.010 g $OUTDIR/$TESS_ID.$SKYCELL.fgal + 2 2 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.020 g $OUTDIR/$TESS_ID.$SKYCELL.fgal + 3 3 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.030 g $OUTDIR/$TESS_ID.$SKYCELL.fgal + 4 1 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.025 r $OUTDIR/$TESS_ID.$SKYCELL.fgal + 5 2 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.035 r $OUTDIR/$TESS_ID.$SKYCELL.fgal + 6 3 $OUTDIR/$TESS_ID.$SKYCELL.stk 0.045 r $OUTDIR/$TESS_ID.$SKYCELL.fgal end @@ -67,4 +74,12 @@ mkcatdir.dif PS1_DV5 PS1_V5 insert.stack.set + + exec rm -rf testdata/catdir.mrg + exec dvomerge -matched-tables testdata/catdir.cam into testdata/catdir.mrg + exec dvomerge -matched-tables testdata/catdir.stk into testdata/catdir.mrg + exec dvomerge -matched-tables testdata/catdir.wrp into testdata/catdir.mrg + + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region 8 12 18 22 -update -vv -averages -boundary-tree tess.3pi.fits + exec relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region 8 12 18 22 -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits -cloud-limit 0.1 end @@ -75,7 +90,11 @@ $dbuser = dvo $dbpass = dvo -# $dbuser = czw -# $dbpass = czw - + + $gscfile = `gconfig GSCFILE` + if ("$gscfile" == "not found") + echo "problem with dvo / ptolemy configuration. check ~/.ptolemyrc" + break + end + exec mkgpc1.sh delete $dbhost $dbname $dbuser $dbpass exec mkgpc1.sh create $dbhost $dbname $dbuser $dbpass @@ -85,5 +104,5 @@ # system-wide info dbinsert ippToPspsFake (my_id) values (2) - dbinsert skycell (radeg, decdeg, tess_id, skycell_id) values ($RA, $DEC, $TESS_ID, $SKYCELL) + dbinsert skycell (radeg, decdeg, tess_id, skycell_id) values ($RA, $DEC, "$TESS_ID", "$SKYCELL") end @@ -110,6 +129,6 @@ $ID = $word:0 $stkID = $word:1 - $rawfile = $word:2\.txt - $cmffile = $word:2\.smf + sprintf rawfile "%s.%02d.txt" $word:2 $ID + sprintf cmffile "%s.%02d.smf" $word:2 $ID $offset = $word:3 sprintf myDATE "%s" $word:4 @@ -172,6 +191,6 @@ mkinput $offset $rawfile - echo mkcmf -photcode SIMTEST.$filter.Chip -no-noise -append $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 0 - exec mkcmf -photcode SIMTEST.$filter.Chip -no-noise -append $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 0 + echo mkcmf -photcode SIMTEST.$filter.Chip $ADDNOISE -append $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 0 -coords + exec mkcmf -photcode SIMTEST.$filter.Chip $ADDNOISE -append $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 0 -coords if ($i == 3) @@ -198,12 +217,12 @@ if ($SMALLTEST) - echo relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update - - echo relastro -v -D CATDIR $catdir -images -update-chips -region 8 12 18 22 -update -apply-offsets - exec relastro -v -D CATDIR $catdir -images -update-chips -region 8 12 18 22 -update -apply-offsets + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update + + echo relastro -v -D CATDIR $catdir -D CAMERA simtest -images -update-chips -update-all-cameras -region 8 12 18 22 -update -apply-offsets + exec relastro -v -D CATDIR $catdir -D CAMERA simtest -images -update-chips -update-all-cameras -region 8 12 18 22 -update -apply-offsets else - echo relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update end @@ -246,6 +265,6 @@ $stkID = $word:0 $skyID = $word:1 - $rawfile = $word:2\.txt - $cmffile = $word:2\.cmf + sprintf rawfile "%s.%02d.txt" $word:2 $stkID + sprintf cmffile "%s.%02d.cmf" $word:2 $stkID $offset = $word:3 $filter = $word:4 @@ -255,6 +274,6 @@ mkinput $offset $rawfile - echo mkcmf -stack -photcode SIMTEST.$filter.SkyChip $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL - exec mkcmf -stack -photcode SIMTEST.$filter.SkyChip $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL + echo mkcmf -stack -photcode SIMTEST.$filter.SkyChip $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords + exec mkcmf -stack -photcode SIMTEST.$filter.SkyChip $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords # add some required header keywords: @@ -267,9 +286,9 @@ if ($SMALLTEST) - echo relphot -D CATDIR $catdir -region 8 12 18 22 -averages -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -averages -update + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -averages -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -averages -update else - echo relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update end @@ -315,6 +334,6 @@ $ID = $word:0 $stkID = $word:1 - $rawfile = $word:2\.txt; # use the cam-stage rawfile - $cmffile = $word:6\.cmf + sprintf rawfile "%s.%02d.txt" $word:2 $ID; # use the cam-stage rawfile + sprintf cmffile "%s.%02d.smf" $word:6 $ID $offset = $word:3 sprintf myDATE "%s" $word:4 @@ -330,6 +349,6 @@ mkinput $offset $rawfile - echo mkcmf -forcedwarp -photcode SIMTEST.$filter.ForcedWarp $rawfile $cmffile -extroot SkyChip -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 2 -tess_id $TESS_ID -skycell $SKYCELL -bad-psfqf-frac $BAD_PSFQF_FRAC - exec mkcmf -forcedwarp -photcode SIMTEST.$filter.ForcedWarp $rawfile $cmffile -extroot SkyChip -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 2 -tess_id $TESS_ID -skycell $SKYCELL -bad-psfqf-frac $BAD_PSFQF_FRAC + echo mkcmf -forcedwarp -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 2 -tess_id $TESS_ID -skycell $SKYCELL -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + exec mkcmf -forcedwarp -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $ID -sourceID 2 -tess_id $TESS_ID -skycell $SKYCELL -bad-psfqf-frac $BAD_PSFQF_FRAC -coords echo addstar -D SKY_DEPTH 4 -D CATDIR $catdir -D CAMERA simtest $cmffile -D CATFORMAT $2 -quick-airmass -xrad @@ -341,6 +360,6 @@ list word -split $stackdata:$i $stkID = $word:0 - $rawfile = $word:2\.txt - $cmffile = $word:5\.cmf + sprintf rawfile "%s.%02d.txt" $word:2 $stkID + sprintf cmffile "%s.%02d.smf" $word:5 $stkID $offset = $word:3 $filter = $word:4 @@ -349,21 +368,21 @@ if ($SMALLTEST && ($stkID != 1) && ($stkID != 4)) continue - echo mkcmf -forcedgalaxy -photcode SIMTEST.$filter.ForcedWarp $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL - exec mkcmf -forcedgalaxy -photcode SIMTEST.$filter.ForcedWarp $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL - - echo loadgalphot -D CATDIR $catdir -photcode SIMTEST.$filter.ForcedWarp -image-id $stkID $cmffile - exec loadgalphot -D CATDIR $catdir -photcode SIMTEST.$filter.ForcedWarp -image-id $stkID $cmffile + echo mkcmf -forcedgalaxy -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords + exec mkcmf -forcedgalaxy -photcode SIMTEST.$filter.ForcedWarp $ADDNOISE $rawfile $cmffile -extroot SkyChip -date 2008/1/1 -time $i\:00:00 -mjd 54466.0 -radec $RA $DEC -type $1 -imageID $stkID -sourceID 1 -tess_id $TESS_ID -skycell $SKYCELL -coords + + echo loadgalphot -D CATDIR $catdir -D CAMERA simtest -photcode SIMTEST.$filter.ForcedWarp -image-id $stkID $cmffile + exec loadgalphot -D CATDIR $catdir -D CAMERA simtest -photcode SIMTEST.$filter.ForcedWarp -image-id $stkID $cmffile end if ($SMALLTEST) - echo relphot -D CATDIR $catdir -region 8 12 18 22 -averages -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -averages -update + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -averages -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -averages -update else - echo relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update - end - - echo dvolens -D CATDIR $catdir -region 8 12 18 22 -update-objects -update - exec dvolens -D CATDIR $catdir -region 8 12 18 22 -update-objects -update + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update + end + + echo dvolens -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -update-objects -update + exec dvolens -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -update-objects -update if ($PARALLEL) @@ -406,7 +425,7 @@ dbinsert camProcessedExp (cam_id, path_base, fault) values ($myID, "$filename", 0) dbinsert fakeRun (fake_id, cam_id) values ($myID, $myID) - dbinsert warpRun (warp_id, fake_id, tess_id, state) values ($myID, $myID, $TESS_ID, "full") - dbinsert warpSkyCellMap (warp_id, skycell_id, class_id) values ($myID, $SKYCELL, "XY00") - dbinsert warpImfile (warp_id, skycell_id, warp_skyfile_id) values ($myID, $SKYCELL, $myID) + dbinsert warpRun (warp_id, fake_id, tess_id, state) values ($myID, $myID, "$TESS_ID", "full") + dbinsert warpSkyCellMap (warp_id, skycell_id, class_id) values ($myID, "$SKYCELL", "XY00") + dbinsert warpImfile (warp_id, skycell_id, warp_skyfile_id) values ($myID, "$SKYCELL", $myID) dbinsert addRun (add_id, stage, state, stage_id, minidvodb_name) values ($myID, "cam", "full", $myID, "catdir.cam") @@ -452,5 +471,5 @@ $skycalID = $stkID + 20 - dbinsert stackRun (stack_id, skycell_id, filter, software_ver) values ($stkID, $SKYCELL, "$filter.00000", "37500") + dbinsert stackRun (stack_id, skycell_id, filter, software_ver) values ($stkID, "$SKYCELL", "$filter.00000", "37500") # we have a single fullForceRun for each stack, so we can match fullForceRun ids to stackRun ids @@ -520,7 +539,7 @@ $ID = $word:0 $stkID = $word:1 - $rawfile = $word:2\.txt - $cmffile = $word:8\.cmf - $invfile = $word:8\.inv.cmf + sprintf rawfile "%s.%02d.txt" $word:2 $stkID + sprintf cmffile "%s.%02d.smf" $word:8 $stkID + sprintf invfile "%s.%02d.inv.smf" $word:8 $stkID $offset = $word:3 sprintf myDATE "%s" $word:4 @@ -535,9 +554,9 @@ mkinput $offset $rawfile $diff_skyfile_id = $ID + 25 - echo mkcmf -diff -photcode SIMTEST.$filter.SkyChip -no-noise -extroot SkyChip $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC - exec mkcmf -diff -photcode SIMTEST.$filter.SkyChip -no-noise -extroot SkyChip $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC - - echo mkcmf -diff -photcode SIMTEST.$filter.SkyChip -no-noise -extroot SkyChip $rawfile $invfile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC - exec mkcmf -diff -photcode SIMTEST.$filter.SkyChip -no-noise -extroot SkyChip $rawfile $invfile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC + echo mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + exec mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $cmffile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + + echo mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $invfile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords + exec mkcmf -diff -photcode SIMTEST.$filter.SkyChip $ADDNOISE -extroot SkyChip $rawfile $invfile -date $myDATE -time $myTIME -mjd $mjd -radec $RA $DEC -type $1 -imageID $diff_skyfile_id -sourceID 36 -bad-psfqf-frac $BAD_PSFQF_FRAC -coords insert.dif $ID $ID $stkID $word:8 @@ -557,9 +576,9 @@ end if ($SMALLTEST) - echo relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update -boundary-tree tess.3pi.fits -is-diff-db -statmode WT_MEAN - exec relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update -boundary-tree tess.3pi.fits -is-diff-db -statmode WT_MEAN + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update -boundary-tree tess.3pi.fits -is-diff-db -statmode WT_MEAN + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -nloop 0 -D STAR_TOOFEW 0 -update -boundary-tree tess.3pi.fits -is-diff-db -statmode WT_MEAN else - echo relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update - exec relphot -D CATDIR $catdir -region 8 12 18 22 -images g,r,i,z,y -update + echo relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update + exec relphot -D CATDIR $catdir -D CAMERA simtest -region 8 12 18 22 -images g,r,i,z,y -update end @@ -592,5 +611,5 @@ $stkID = $word:0 $skyID = $word:1 - $cmffile = $word:2 + sprintf cmffile "%s.%02d.cmf" $word:2 $stkID $filter = $word:4 @@ -652,7 +671,7 @@ dbinsert addProcessedExp (add_id, fault) values ($addID, 0) - dbinsert diffRun (diff_id, tess_id, state, diff_mode, software_ver) values ($difID, $TESS_ID, "full", 2, "38000M") - dbinsert diffInputSkyfile (diff_id, warp1, stack2, diff_skyfile_id, tess_id, skycell_id) values ($difID, $wrpID, $stkID, $diffSkyfileID, $TESS_ID, $SKYCELL ) - dbinsert diffSkyfile (diff_id, skycell_id, path_base) values ($difID, $SKYCELL, "$filename") + dbinsert diffRun (diff_id, tess_id, state, diff_mode, software_ver) values ($difID, "$TESS_ID", "full", 2, "38000M") + dbinsert diffInputSkyfile (diff_id, warp1, stack2, diff_skyfile_id, tess_id, skycell_id) values ($difID, $wrpID, $stkID, $diffSkyfileID, "$TESS_ID", "$SKYCELL" ) + dbinsert diffSkyfile (diff_id, skycell_id, path_base) values ($difID, "$SKYCELL", "$filename") end @@ -669,17 +688,57 @@ exec rm -f $output - local i j + local i j ra dec output $output for i 10 1024 100 for j 10 1024 100 - fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + $ra = 10.0 + $i*0.25/3600 + $dec = 20.0 + $j*0.25/3600 + fprintf " %10.6f %10.6f %7.3f %6.1f %6.1f %7.3f" $ra $dec {10.0 + $1 + 2.5*($i + $j)/1000.0} $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} end end # XXX add 2 duplicate detections $i = 110; $j = 210 - fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + # fprintf " %10.6f %10.6f %7.3f %6.1f %6.1f %7.3f" $ra $dec {10.0 + $1 + 2.5*($i + $j)/1000.0} $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + # fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} $i = 210; $j = 110 - fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + # fprintf " %10.6f %10.6f %7.3f %6.1f %6.1f %7.3f" $ra $dec {10.0 + $1 + 2.5*($i + $j)/1000.0} $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} + # fprintf " %6.1f %6.1f %7.3f" $i $j {-15.0 + $1 + 2.5*($i + $j)/1000.0} output stdout +end + +macro check.mags + + # NOTE: relphot refuses to set the warp averages if there is no boundary tree: otherwise it cannot choose the correct subset of measurements + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region 8 12 18 22 -update -vv -averages -boundary-tree tess.3pi.fits + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region 8 12 18 22 -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits + + catdir testdata/catdir.mrg + skyregion 8 12 18 22 + avextract ra dec g:psf:chp g:kron:chp g:psf:stk g:psf:wrp + + set dg:psf:chp:stk = g:psf:chp - g:psf:stk + set dg:psf:chp:wrp = g:psf:chp - g:psf:wrp + lim g:psf:chp dg:psf:chp:stk; clear; box; plot g:psf:chp dg:psf:chp:stk -c blue -pt 7; plot g:psf:chp dg:psf:chp:wrp -c red -pt 2 + +# for i 0 10 +# echo ra[$i] dec[$i] g:psf:chp[$i] g:kron:chp[$i] g:psf:stk[$i] g:psf:wrp[$i] +# end +end + +macro check.mags.meas + + # NOTE: relphot refuses to set the warp averages if there is no boundary tree: otherwise it cannot choose the correct subset of measurements + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region 8 12 18 22 -update -vv -averages -boundary-tree tess.3pi.fits + # relphot -D CATDIR testdata/catdir.mrg -D CAMERA simtest -region 8 12 18 22 -update -vv -images g,r -reset -reset-zpts -D STAR_TOOFEW 0 -boundary-tree tess.3pi.fits + + catdir testdata/catdir.mrg + skyregion 8 12 18 22 + + mextract ra dec mag:rel:psf mag:rel:kron mag:ave:psf mag:ave:kron photcode + set dM:psf = mag:ave:psf - mag:rel:psf + lim -n 0 mag:ave:psf dM:psf; clear; box; plot mag:ave:psf dM:psf + + set dM:kron = mag:ave:kron - mag:rel:kron + lim -n 1 mag:ave:kron dM:kron; clear; box; plot mag:ave:kron dM:kron end Index: branches/czw_branch/20170908/ippToPsps/test/notes.txt =================================================================== --- branches/czw_branch/20170908/ippToPsps/test/notes.txt (revision 40128) +++ branches/czw_branch/20170908/ippToPsps/test/notes.txt (revision 40483) @@ -1,2 +1,8 @@ + +2017.11.28 EAM: + + updating the test suite (using my desktop kukui -- ubuntu 14.04). + + * need to install xmllint to run ipptopsps In order to have a reasonably complete test, I need to have a bunch of things that mimic the IPP infrastructure