| | 8 | I spent the week understanding detailed issues with the ubercal'ed dvo database / reference database. I completed an initial run of the full sky relphot analysis on Monday. However, I discovered surprising NAN values in the database, which I tracked down to the ubercal table. It turns out that ubercal sets the zero point for rejected images to NAN (I thought they were excluded from the table). I updated the code to tread NAN zero-pointed images as not part of ubercal, and re-ran the process. I also ran relastro on the full sky, but discovered that too many images had unreliable solutions. Looking through the analysis, I realized this was due to the selection stars used to perform the calibration: in order to avoid blowing up the memory footprint, I limit the density of stars to some specific value. After basic S/N cuts, I was selecting that subset randomly. This allowed too may objects with only a limited number of overlap detections to be used, which meant some images had very few stars. I made a few changes: 1) I bumped up the density (from 1000 to 2500 stars per square degree), 2) I modified the selection to preference objects with many measurements, 3) I relaxed the S/N cuts a bit to give a bit better sampling for all images. This helped to reduce greatly the number of bad astrometry failures, but pushed the memory up alot. I needed to reduce the fraction of the sky per chunk to fit in the footprint of ipp064 (48GB). The analysis now runs on 13 chunks: 12 RA slices from -60 : 60 dec and the polar cap. Although the I/O time is quite reasonable, with the parallelized analysis, the processing time is now substantial : something like 40 hours is needed to do the analysis of all chips in the database. I am looking at threading that part of the code for the future. |
