== IPP for PS1 == [[PageOutline]] (Up to the [wiki:WikiStart Main Page]) This page lists information related to the IPP processing of the PS1 data. === Data Distribution === * [wiki:GPC1_DataDistribution] * [wiki:PS1_DataRequestManagement PS1 Data Distribution Requests & Processing Management] * [wiki:NightlyStacks Nightly stack processing information] === Outstanding Issues === * [wiki:NebulousRelatedErrors Errors related to Nebulous (NFS? mysql? other issues?)] === IPP Operations === * [wiki:Processing] Processing Czar information * [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs] * [wiki:IPP_NodeUse] * [wiki:PS1_Operations] Tag creation * [wiki:NightlyScienceCode] Nightly science automation code * [wiki:NightlyScienceV2] New nightly science code documentation * [wiki:Nebulous_Notes Nebulous related notes] * [wiki:ChipStage_Timing] Statistics on the processing time required for chip stage * [wiki:PS1_IASC_Chips] Creating a dataset for the IASC Outreach Program * [wiki:LAP_Science Large Area Processing organization] * [wiki:PV3_failed_data Log of stack and warp data that has failed in PV3] === MHPCC Cluster === * [wiki:WeekendDiskChanges '''Weekend Disk Changes'''] : information about contacting the MHPCC weekend support if we lose drives on the weekend * [wiki:Production_Cluster Production Cluster] * [wiki:Production_Cluster_Status Production Cluster Status] * [wiki:PS1_Operations PS1 Operations] * [wiki:Cluster_Storage_Notes Cluster Storage Notes] * [wiki:GPC_Summit_Copy GPC Summit Copy] : EAM notes for running the summit copy system (Feb 2008) * [http://ps1wiki.ifa.hawaii.edu/DataStore Data Store] * [wiki:SummitCopyAndRegistration Summit Copy Status] * [wiki:DatabaseBackups IPP mysql database backups] * [wiki:ThroughputDemo_20091104 Throughput Demo 2009.11.04] * [wiki:Nebulous_Notes Nebulous related notes] * [wiki:Nebulous_Issues Nebulous Issues that need addressing in the future] * [wiki:New_Nebulous_Code 2018 Update to Nebulous to enable location awareness.] * [wiki:Shuffle_notes Issues relating to shuffling/reinserting data.] === Configuration Information === * [wiki:PS1_DVO_Seed_Database] : Creating the DVO Database used to seed the PS1 Science DVO Database * [wiki:PS1_Tessellations]: Generation of the tessellations used for PS1 science === Science Image Analysis === * [wiki:DVO_Reorganization_20150827] New locations for DVO Databases after the STScI machines were shipped * [wiki:PS1_3pi_reference_catalogs] Summary of the 20120524 and 20140331 reference catalogs * [wiki:PS1_3pi_DVO_20120606 3pi DVO Database 2012.06.06] Information about the 3pi DVO database snapshot 2012.06.06 * [wiki:GPC1_Defects GPC1 Defects] Examples of interesting GPC1 defects * [wiki:PS1_3pi_DVO_20110421 3pi DVO Database 2011.04.21] Information about the 3pi DVO database snapshot 2011.04.21 * [wiki:Background_Oversubtraction Background Model and Oversubtraction] * [wiki:Background_Model_Stacks Background Model and Stacks] * [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Performance_Tests_July2010 Performance Tests by DRAVG (July 2010)] * [wiki:ppSubModifications_201009 ppSub Modifications September 2010] : Discussion of ppSub modifications * [wiki:MD04_Stack_Photometry_20100819 MD04 Stack Photometry 2010.08.19] : Exploration of stack photometry issues seen in MD04 * [wiki:PS1_MD04_RefStack PS1 MD04 RefStack] : Creation of the MD04 reference stacks] * [wiki:Stackphot_20100514 Stackphot Initial Release] * [wiki:PS1_SAS_AstromPhotom_20100310 SAS Astrometry & Photometry Test 2010.03.10] * [wiki:PS1_rates Completeness / False Positive rates] : Completeness and false positive rates for the MD09 reprocessing of 11/19 * [wiki:PS1_depth] : first-pass look as PS1 depth * [wiki:PS1_color_diagrams_July2009] : Color-color and color-magnitude diagrams from the July 2009 data release * [wiki:PS1_Science_Processing_Status_20090705] : Science Processing Status for April-June Medium Deep fields (MD06, MD07, MD08) * [wiki:PS1_GPC1_GlintMasks] : Glint Masking examples 2009.07.03 * [wiki:PS1_GPC1_Glints_20090629] : Glint Sources 2009.06.29 * [wiki:PS1_GPC1_Magic_Status_20090628] : Status of Magic analysis issues 2009.06.28 * [wiki:PS1_GPC1_Magic_And_Short_Exposures] : Analysis of magic troubles for short exposures. * [wiki:PS1_GPC1_Rotator_Measurements] : EAM Reports on analysis of PS1 rotation sequence images * [wiki:GPC_Tests_2008.06 GPC Tests 2008.06] : EAM notes on running the science analysis on GPC1 images * [wiki:GPC_PR_Images GPC PR Images] : EAM notes on processing GPC1 PR images (Apr 2008) * [wiki:GPC_Notes_2008.02 GPC Notes 2008.02] : J.Koppenhoefer notes on running IPP on GPC1 data (Feb 2008) * [wiki:GPC_Notes GPC Notes] : EAM notes on running IPP on GPC1 data (Aug 2007) * [wiki:GPC_Notes_2007.12 GPC Notes 2007.12] : EAM notes on running IPP on GPC1 data (Dec 2007) * [wiki:PS1_GPC1_GhostModel_V0] : Ghost Model Analysis (Run 2 data) * [wiki:MaskFeatures.pattern.20090722] : Examples of features that trigger Magic * [wiki:Image_Artifacts] Examples of known image artifacts * [wiki:MoonGlintTest] Examples of moon glints. * [wiki:ppSub_Issues] Concerns about image subtraction -- EAM 2010.01.04 * [wiki:ppSub_vs_Hotpants] Comparison between ppSub and hotpants PSF-matching image subtraction codes * [wiki:EdgeRolloff] Check of the edge of cell rolloff issue * [wiki:Quality_Flags] Counts and logfile examples of non-zero chip and warp quality flags. * [wiki:DetectorLinearity] Detector linearity work. * [wiki:psphot_detection_comparison] Comparison of detections with different signal-to-noise using psphot * [wiki:GPC_BrightStarAstrometry_Issue] Analysis of astrometry offset first found in STS data. * [wiki:Background_Continuity] Discussion of attempts to improve background smoothness by ensuring cell-to-cell continuity. * [wiki:Stack_Rejection_Discussion] Discussion of stacking and input rejection. * [wiki:MOPS_diff_false_positives] Discussion of MOPS false positives/difference images. * [wiki:GPC1_1D_convolution] Discussion of Gaussian 1-D convolution code change. * [wiki:Stack_Sky_Levels] Discussion of Stack image sky/background levels. * [wiki:Stack_Algorithm] Discussion of current algorithm used in ppStack * [wiki:Coverage_Maps] Discussion of the stack summary/coverage map processing. * [wiki:FullForcePhotometry] Discussion of the forced photometry to be performed on individual exposures using static sky object parameters * [wiki:GalaxyParameters] Discussion of the galaxy parameter measurements performed on extended sources === Detrend Creation Information === * [wiki:GPC1_Detrend_Documentation] : General documentation on GPC1 detrends/creation/detselect. * [wiki:GPC1_Camera_Artifacts_Bathtub_Rings] : Bathtub Rings and Related Issues * [wiki:GPC1_detrend_extended_notes] : information about the various detRuns, peculiarities, and manual changes * [wiki:GPC1_FlatField_Correction_20091003 GPC1 Flat-field correction 2009.10] : Description of the flat-field correction analysis process * [wiki:GPC1_Mask_20101215 GPC1 Mask 20101215] : Manual updates to the GPC1 bad pixel mask (2010-12-15) * [wiki:GPC1_Mask_20100122 GPC1 Mask 20100122] : Manual updates to the GPC1 bad pixel mask since the camera mods 2010.01.14 * [wiki:GPC1_Mask_2009.06.22 GPC1 Mask 20090622] : Manual updates to the GPC1 bad pixel mask * [wiki:GPC1_Mask_2009.05 GPC1 Mask 2009.05] : Analysis of the GPC1 bad pixel mask -- Run 3 (reconfigured camera) * [wiki:GPC1_Dark_Linefits_20090506 GPC1 Dark Linefits] : examination of the bias correction based on pre and post-scan fitting * [wiki:GPC1_Mask_2008.06 GPC1 Mask 2008.06] : Analysis of the GPC1 bad pixel mask * [wiki:GPC_FlatField_Correction_2008.11 GPC FlatField Correction 2008.11] : GPC1 flat-field correction analysis -- some success! * [wiki:GPC_FlatField_Correction_2008.10_pt2 GPC FlatField Correction 2008.10 pt2] : GPC1 flat-field correction analysis -- further notes * [wiki:GPC_FlatField_Correction_2008.10 GPC FlatField Correction 2008.10] : GPC1 flat-field correction analysis * [wiki:GPC_Dark_Trends GPC Dark Trends] : J.Koppenhoefer notes and figures on dark current as a function of temperature and exposure time (Feb/Mar 2008) * [wiki:GPC_Mask_Results GPC Mask Results] : S.Rodney notes and figures showing GPC1 bad pixel mask results (Fall 2007) * [wiki:GPC_DarkMask GPC DarkMask] : S.Rodney notes on building/characterizing bad pixel masks from GPC1 darks (Fall 2007) * [wiki:GPC_Detrend_Registration GPC Detrend Registration] : script for registering pre-made GPC1 detrends * [wiki:GPC1_FPA_Mask_20100525 GPC1 Mask analysis 20100525] : Notes on the mask fraction for the full FPA. * [wiki:GPC1_Mask_Stats_Week_One] : Check of the mask statistics from the first week of processing with maskstats enabled. * [wiki:Background_Dark_Model] : Analysis of the dark model contribution to background signal, and results of making a new dark (2012-03-08). * [wiki:GPC1_Reference_Catalog_Stats] : Analysis of the reliability of the reference catalog based on multiple SAS2 processings. * [wiki:GPC1_FFT_Bias_Study] : Study of bias images with FFTs. * [wiki:GPC1_Bias_Pattern_Study] : Study of common mode structure along bias cells * [wiki:GPC1_M31_OTA_Offset_Study] : Examination of the offsets between OTAs visible in the M31 background preserved images. * [wiki:GPC1_Video_Dark_Models] : Study of the influence of video cells on the dark current. === Interesting & Complex Bugs === * [wiki:PS1_timing_problems] : History of problems with the PS1 / GPC1 / IPP reported times. * [wiki:stack_background_example] : Example of the stack background offset problem * [wiki:PS1_2MASS_astrometry] : Problems with 2MASS astrometry and the synthetic catalog (July 2009) * [wiki:update_versus_magic Update Processsing versus magic] * [wiki:duplicate_addstars] * [wiki:MOPS_trailed_psfs] : Issues with trail fitting claiming point source objects are trailed. === Postage Stamp Server === * [wiki:PostageStampServer PostageStampServer] === Photometric Calibration === * [wiki:PS1_Photometric_System PS1 Photometric System] : Detailed analysis and results for PS1 bandpasses and photometry by JT * [wiki:Calibration_Concept_Discussion] : Summary of some of the photometric calibration issues for 3pi * [wiki:GPC_Zero_Points GPC Zero Points] : EAM notes on using Stripe 82 chip output processed by Johannes to get zero points * [wiki:GPC_Zero_Points_July09] : Investigation of GPC1 zero points from the July 2009 data release * [wiki:synth_catalog_July09] : Color magnitude diagrams for the synthetic catalog as compared to SDSS * [wiki:zpt_variation_july09] : Variation in zero point relative to SDSS * [wiki:FaintBias] : Attempt to empirically remove the faint-end bias === Astrometric Calibration === * [wiki:GalacticPlane_and_wBand_20140526] : Astrometry improvements to allow psastro to better handle the Galactic Plane, especially in w-band === Collimation & Alignment === * [wiki:GPC1_C&A_Results_2008.07 GPC1 C&A Results 2008.07] === Committees === * The Data Reduction, Analysis and Verification (DRAV) Group is the formal link between the PS1SC and the IPP team. DRAV coordinates IPP testing outside the IPP team at IfA and has ~monthly telecons bringing together everyone working on or with IPP. Email contact is dravg_at_ps1sc.org, group membership and telecon minutes are posted on the [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG PS1 wiki's DRAVG page] * [wiki:DRAVG_test_suite_images_photometry] List of IPP test suite images candidates === Miscellaneous === * [wiki:Magic.20081221.example magic.20081221.example] -- discussion of the impact of IPP and Magic on the GPC1 masking. * [wiki:Magic.20081112.example magic.20081112.example] -- discussion of the state of magic as of Nov 12, and example masks * [wiki:Pordig.20081029.sample pordig.20081029.sample] -- discussion of the single-image analysis and data examples. * [wiki:ipp.psps.ids ipp.psps.ids] -- discussion of how data is identified as it moves though IPP to PSPS * [wiki:PS1_Graveyard] -- missing raw chips * [wiki:External_Supercomputing_Notes] -- CZW notes on IPP processing on an external supercomputer cluster. * [wiki:asinh_compression] -- How to deal with converting asinh compressed images to something that can be used. * [wiki:ITC_shuffle] -- CZW notes on the shuffle of data to the ITC cluster. * [wiki:PS1_Graveyard] -- missing raw chips