== Artifacts and Magic Streaks : Feb 2010 Verification Data == 

Over the past few months, one of the major development goals has been to reduce the impact of artifacts (optical and instrumental) on the data quality and the area masked by Magic.

Perhaps the most significant instrumental artifact has been the row-by-row variations in the bias level (sometimes described as 'streakies' or correlated read noise).  This effect varies from cell-to-cell and chip-to-chip.  For some cells, there is an overall bias offset which is constant for a given readout of a given row, but randomly change from row to row (and from exposure to exposure for a given row).  For some cells, this bias offset drifts up or down as the row is read, with a somewhat random amount for each row.  For most of the past year, in order to reduce the triggers for Magic, the IPP has been forced to subtract a row-by-row 2nd-order polynomial fit to the sky level in order to remove this signal (the 'PATTERN.ROW correction').  This correction procedure significantly affects the flux of objects which are a few 10s of pixels or larger.  

Since this impact was unacceptable for many science goals, we have been trying to mitigate the problem without damaging the astronomical sources.  The main effort has been on the camera readout process: modification of the readout timings has helped to reduce the level of the 'streakies'.  In addition, the camera group has worked to make a cleaner measurement of the overscan so that the correction can be applied from region unaffected by the astronomical signal.  This work has resulted in much cleaner images, but some cells are still susceptible to the effect.  We are currently continuing to apply the 'PATTERN.ROW' correction to a small subset of the cells in the camera (169 cells, or about 4.5% of the area), but the rest can be processed without this correction.

We have also attempted to address other sources of magic false positives, in both the IPP where possible and in the magic software itself.  Over the past two weeks, we have run three iterations of a single test suite to validate magic.  The test suite included data representing all of the survey modes.  Below are plotted histograms of the number of magic streaks found for each of the three test runs, also broken down by the survey type and the filter.  Note that the intermediate analysis used only a subset of roughly 300 exposures, while the first and last iteration used the complete set of roughly 750 exposures.  Also note that in these plots, the horizontal axis uses bins of '''log10(streaks)'''. 

Notice that the streak numbers depend greatly on the survey type.  For the most part, the lower the sky background and the higher the density of stars, the more sensitive will Magic be to the instrumental artifacts and persistence structures.  In addition, the area near the Geo belt (-3.5 Dec as seen from Hawaii) will have a higher density of real satellites.  For the full set of data in this test suite, the final configuration yielded a median of 49 streaks per exposure and a maximum of 279.  

'''After the last test set, we missed a streak.  The Magic team adjusted the detection threshold just slightly (to 2.35 sigma from 2.4 sigma) to enable that detection.  As a result, the expected streak rates are ~15% larger than shown here.'''

magicverify.%.3.% = first test
magictest.%.20100205 = intermediate test
magictest.%.20100210% = most recent (best test)
|| magicverify.%.3.% = first test|| magictest.%.20100205 = intermediate test||magictest.%.20100210% = most recent (best test)||
|| all streaks  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-2.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-1.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-3.jpg,100%)]] ||
||  all r band ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-4.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-5.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-6.jpg,100%)]] ||
||  all z band ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-7.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-8.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-9.jpg,100%)]] ||
||  all i band ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-10.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-11.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-12.jpg,100%)]] ||
||  all g band ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-13.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-14.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-15.jpg,100%)]] ||
||  all y band ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-17.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-16.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-18.jpg,100%)]] ||
||  all w band ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-20.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-19.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-21.jpg,100%)]] ||

||  3pi all ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-23.jpg,100%)]] ||only has z band ||[[Image(source:/trunk/doc/magicverification/mhistogram-24.jpg,100%)]] ||
||  3pi z ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-25.jpg,100%)]]||[[Image(source:/trunk/doc/magicverification/mhistogram-22.jpg,100%)]]  ||[[Image(source:/trunk/doc/magicverification/mhistogram-26.jpg,100%)]] ||
||  3pi y ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-27.jpg,100%)]] || no y band test ||[[Image(source:/trunk/doc/magicverification/mhistogram-28.jpg,100%)]] ||
||  dec 07 all ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-30.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-29.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-31.jpg,100%)]] |||
||  dec 07 g ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-33.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-32.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-34.jpg,100%)]] |||
||  dec 07 r ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-36.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-35.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-37.jpg,100%)]] |||
||  dec 07 i ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-38.jpg,100%)]] ||no i band image ||[[Image(source:/trunk/doc/magicverification/mhistogram-39.jpg,100%)]] |||
||  dec 35 all ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-40.jpg,100%)]] ||only r band ||[[Image(source:/trunk/doc/magicverification/mhistogram-41.jpg,100%)]] |||
||  dec 35 g ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-42.jpg,100%)]] ||no g band ||[[Image(source:/trunk/doc/magicverification/mhistogram-43.jpg,100%)]] |||
||  dec 35 r ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-45.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-44.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-46.jpg,100%)]] |||
||  dec 35 i ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-47.jpg,100%)]] ||no i band ||[[Image(source:/trunk/doc/magicverification/mhistogram-48.jpg,100%)]] |||
||  lgl all ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-49.jpg,100%)]] ||just one band ||[[Image(source:/trunk/doc/magicverification/mhistogram-50.jpg,100%)]] |||
||  lgl g ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-51.jpg,100%)]] ||no g band ||[[Image(source:/trunk/doc/magicverification/mhistogram-52.jpg,100%)]] |||
|| lgl r ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-54.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-53.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-55.jpg,100%)]] |||
||  lgl i ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-56.jpg,100%)]] ||no i band ||[[Image(source:/trunk/doc/magicverification/mhistogram-57.jpg,100%)]] |||
|| shortmd all ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-59.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-58.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-60.jpg,100%)]] |||
|| shortmd g ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-62.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-61.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-63.jpg,100%)]] |||
|| shortmd r ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-65.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-64.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-66.jpg,100%)]] |||
|| shortmd i ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-68.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-67.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-69.jpg,100%)]] |||
|| shortmd z ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-71.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-70.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-72.jpg,100%)]] |||
|| shortmd y ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-74.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-73.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-75.jpg,100%)]] |||
|| md i ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-76.jpg,100%)]] || not tested ||[[Image(source:/trunk/doc/magicverification/mhistogram-77.jpg,100%)]] |||
|| md20 all  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-79.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-78.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-80.jpg,100%)]] |||
|| md20 g  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-82.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-81.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-83.jpg,100%)]] |||
|| md20 r  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-84.jpg,100%)]] ||no r ||[[Image(source:/trunk/doc/magicverification/mhistogram-85.jpg,100%)]] |||

|| md20 i  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-87.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-86.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-88.jpg,100%)]] |||
|| md20 z  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-90.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-89.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-91.jpg,100%)]] |||
|| md20 y  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-93.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-92.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-94.jpg,100%)]] |||
|| m31 r  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-95.jpg,100%)]] ||no r ||[[Image(source:/trunk/doc/magicverification/mhistogram-96.jpg,100%)]] |||
|| ss all  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-97.jpg,100%)]] || r only ||[[Image(source:/trunk/doc/magicverification/mhistogram-98.jpg,100%)]] |||
|| ss r  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-100.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-99.jpg,100%)]] ||[[Image(source:/trunk/doc/magicverification/mhistogram-101.jpg,100%)]] |||
|| ss w  ||
||[[Image(source:/trunk/doc/magicverification/mhistogram-102.jpg,100%)]] || no w band ||[[Image(source:/trunk/doc/magicverification/mhistogram-103.jpg,100%)]] |||