ISP SAS Processing

Heather is attempting to process the SAS area of sky in skyprobe.

Issues

  • ippMonitor / isp is broken - it looks at an old database? Heather doesn't know how to fix this
  • there are multiple dark detrends for isp and heather doesn't remember why. The one for 2011 (det_id 14) is missing, so heather set that to drop at the moment to continue processing
    • assume they were not replicated and was on a machine that is dead or out of nebulous...
  • need to fix summitcopy/registration to ingest the comment from isp data, data after a certain date does not have a comment.

First steps

  • compiled ipp trunk as of 9-7-2012 as heather, running ispstdscience as heather on ipp007 (neither of these matter, just for reference)
  • we want warps and stacks - likely the gpc1 skycells are not idea. Heather (with help of gene) created new skycells for isp, called ISP.V0. They are essentially identical to gpc1, except the pixel size is much larger (5.57), and each projection cell is not subdivided (gpc1 subdivides by 10x10)
    • these are located at /local/ipp/isp/ISP.V0 on all relevant machines
    • the magical command: skycells -mode RINGS -scale 5.57 -nx 1 -ny 1 -fix-ns -D CATDIR ISP.V0 -overlap 30 30 -skyparity
  • queued (repeatedly) sas.v0 stuff. this overlaps with MD09, so did the following to queue:
    • chiptool -dbname isp -definebyquery -set_label sas.v0 -dateobs_begin 2010-01-01T00:00:00 -comment '%ps1%' -dateobs_end 2012-10-02T00:00:00 -set_workdir neb://@HOST@.0/isp/sas.v0 -ra_min 328 -ra_max 339 -decl_min -6 -decl_max 6 -set_end_stage warp -set_tess_id ISP.V0 -simple
  • here's the status of processing:
     mysql> select count(*), state from chipRun where label = 'SAS.v0' group by state;
    +----------+-------+
    | count(*) | state |
    +----------+-------+
    |     3300 | full  | 
    |       21 | new   | 
    +----------+-------+
    2 rows in set (0.02 sec)
    
    mysql> select count(*), state from camRun where label = 'SAS.v0' group by state;
    +----------+-------+
    | count(*) | state |
    +----------+-------+
    |     3300 | full  | 
    +----------+-------+
    1 row in set (0.01 sec)
    
    mysql> select count(*), state from warpRun where label = 'SAS.v0' group by state;
    +----------+-------+
    | count(*) | state |
    +----------+-------+
    |      143 | full  | 
    |     2335 | new   | 
    +----------+-------+
    2 rows in set (0.01 sec)
    
    

hmmm... warp is still running.... This seems unexpected... What do the statistics look like?

first look at some warps.

mysql> select uri from warpRun join warpSkyfile using (warp_id) where label = 'SAS.v0' and state = 'full' and warp_id = 29;
+---------------------------------------------------------------------------------------+
| uri                                                                                   |
+---------------------------------------------------------------------------------------+
| neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1404.fits | 
| neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1405.fits | 
| NULL                                                                                  | 
| neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1493.fits | 
| neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1494.fits | 
| NULL                                                                                  | 
| NULL                                                                                  | 
| NULL                                                                                  | 
+---------------------------------------------------------------------------------------+
8 rows in set (0.00 sec)

database says this one is processed, and it tried 8 skycells (?). I loaded up the 4 good ones in ds9:

 ds9 -mosaicwcs `neb-locate --path neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1404.fits` `neb-locate --path neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1405.fits` `neb-locate --path neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1493.fits` `neb-locate --path neb://any/isp/sas.v0/o5345i0213o01.89259/o5345i0213o01.89259.wrp.29.skycell.1494.fits`

Hey, this doesn't look so bad, except for it being out of focus. D'oh...

according to the database, teh fwhm is 17... yuuuuck!

here are some statistics on the camRun:

mysql> select count(*), min(fwhm_major), max(fwhm_major), min(sigma_ra), max(sigma_ra)  from camRun join camProcessedExp using (cam_id) where label = 'SAS.v0' limit 1;
+----------+------------------+-----------------+------------------+-----------------+
| count(*) | min(fwhm_major)  | max(fwhm_major) | min(sigma_ra)    | max(sigma_ra)   |
+----------+------------------+-----------------+------------------+-----------------+
|     3300 | 0.89642798900604 | 28.676900863647 | 0.44685900211334 | 37.357189178467 | 
+----------+------------------+-----------------+------------------+-----------------+

and here's what I think are potentially good: I require sigma_ra < 1 (is that reasonable?), n_astrom > 0 (otherwise, astrometry failed), and fwhm_major < 5 (no ideas if this is reasonable or not...) Note that we do have odd problems including dome occultation, so we do expect some fraction to suck.

mysql> select count(*), min(fwhm_major), max(fwhm_major)  from camRun join camProcessedExp using (cam_id) where label = 'SAS.v0' and sigma_ra < 1 and n_astrom > 0 and fwhm_major < 5;
+----------+-----------------+-----------------+
| count(*) | min(fwhm_major) | max(fwhm_major) |
+----------+-----------------+-----------------+
|     1356 |  1.417240023613 | 4.9119601249695 | 
+----------+-----------------+-----------------+
1 row in set (0.01 sec)

Here's the breakdown by filter:

mysql> select filter, count(*), min(fwhm_major), max(fwhm_major)  from camRun join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where camRun.label = 'SAS.v0' and sigma_ra < 1 and n_astrom > 0 and fwhm_major < 5 group by filter;
+--------+----------+-----------------+-----------------+
| filter | count(*) | min(fwhm_major) | max(fwhm_major) |
+--------+----------+-----------------+-----------------+
| g      |      284 |         3.01373 |         3.65707 | 
| i      |      719 |         1.41724 |         4.91196 | 
| r      |      311 |          1.6642 |         4.58159 | 
| z      |       42 |         1.48253 |         1.67822 | 
+--------+----------+-----------------+-----------------+
4 rows in set (0.03 sec)

I'm a little surprised at the lack of y, but y is horrible to process with skyprobe, so I'm not going to worry about it for now... Clearly, there are a lot of images in gri, and a few in z (?). This should be enough to stack. I can sort by skycells:

mysql> select filter, skycell_id, count(*), min(fwhm_major), max(fwhm_major)  from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where camRun.label = 'SAS.v0' and sigma_ra < 1 and n_astrom > 0 and fwhm_major < 5 and warpSkyfile.quality = 0 group by filter, skycell_id;
+--------+--------------+----------+-----------------+-----------------+
| filter | skycell_id   | count(*) | min(fwhm_major) | max(fwhm_major) |
+--------+--------------+----------+-----------------+-----------------+
| g      | skycell.1314 |        1 |         3.59727 |         3.59727 | 
| g      | skycell.1315 |        2 |          3.6018 |          3.6311 | 
| g      | skycell.1316 |        2 |          3.6018 |          3.6311 | 
| g      | skycell.1404 |        2 |         3.59727 |         3.60911 | 
| g      | skycell.1405 |        4 |         3.59727 |          3.6311 | 
| g      | skycell.1406 |        3 |          3.6018 |          3.6382 | 
| g      | skycell.1407 |        1 |          3.6382 |          3.6382 | 
| g      | skycell.1494 |        1 |         3.60911 |         3.60911 | 
| g      | skycell.1495 |        1 |          3.6382 |          3.6382 | 
| g      | skycell.1496 |        1 |          3.6382 |          3.6382 | 
| i      | skycell.1314 |        2 |         1.63937 |         1.66888 | 
| i      | skycell.1315 |        1 |         1.66888 |         1.66888 | 
| i      | skycell.1404 |        4 |         1.63937 |         2.42179 | 
| i      | skycell.1405 |        4 |         1.63937 |         1.67396 | 
| i      | skycell.1406 |        3 |         1.64807 |         1.67396 | 
| i      | skycell.1407 |        1 |         1.64807 |         1.64807 | 
| i      | skycell.1493 |        4 |         1.76622 |         2.42179 | 
| i      | skycell.1494 |        4 |         1.67151 |         2.42179 | 
| i      | skycell.1495 |        3 |         1.64807 |         1.67396 | 
| i      | skycell.1582 |        1 |         1.81341 |         1.81341 | 
| r      | skycell.1314 |        2 |         1.80198 |         1.94848 | 
| r      | skycell.1315 |        1 |         1.94848 |         1.94848 | 
| r      | skycell.1404 |        4 |         1.80198 |         3.14089 | 
| r      | skycell.1405 |        4 |         1.80198 |         3.14089 | 
| r      | skycell.1406 |        3 |         1.91914 |         1.98396 | 
| r      | skycell.1407 |        1 |         1.91914 |         1.91914 | 
| r      | skycell.1493 |        4 |         1.85289 |         3.14089 | 
| r      | skycell.1494 |        4 |         1.85289 |         3.14089 | 
| r      | skycell.1495 |        3 |         1.91914 |         1.98396 | 
| r      | skycell.1496 |        2 |         1.91914 |         1.98396 | 
| z      | skycell.1493 |        1 |          1.6754 |          1.6754 | 
+--------+--------------+----------+-----------------+-----------------+
31 rows in set (0.00 sec)

of course, it helps if we've processed more than a handful of stuff.... ugh... need to wait now...

Monday

A quick check of the db says that it's processed 920, 1558 left still to go. I wonder why it is so slow for warps? It's still processing according to pantasks_client, with only a minimal number of faults (6 out of 145424). Why are there so many processed skycells?!

This is seriously weird. For example, for warp_id = 955, there are 504 skycells, all but 2 have crappy quality. What gives?

Sometimes it has a reasonable < 10 number of skycells, but often it has hundreds... I'm still investigating, but it seems that the camRuns that have hundreds of skycells are what I would call reasonable (ie, good, sigma_ra, good nastrom, good astrometry, good focus..)

ahh.. debugged it.. turns out to be a problem in dvoImageOverlaps - specific to non-mosaic type of images. I need to set the polyorder to 1 for non mosaics (this is set to 1 for mosaics), and then I get reasonable numbers of skycells. I've dropped and renamed the old warp stuff, and restarted at sas.v1. It's processing now, with about 8 hosts, and in the past hour has processed something like 100 or so warpRuns (out of many more). This looks like it's working, will see how it works tomorrow (for stacking?)

Tuesday

Still marching along - I only have a handful of hosts so I increased the number. about 20 hours after I put in the fix (note, also checked into svn), 1059 have completed with another 1400 remaining.. Hopefully it will finish by tomorrow and I can try to stack.

Stacks!

09-19-2012

First attempts at stacking... the warps had finished last week, but then I was sidetracked by psps stuff. I'm returning to it now. All but 3 skycells stacked.. I am not going to worry about that for now. Here is the list of good warps for stacking:

mysql> select warpRun.label,skycell_id, filter, quality, count(*) from rawExp join chipRun using (exp_id) join camRun using (chip_id) join fakeRun using (cam_id) join warpRun using (fake_id) join warpSkyfile using (warp_id) where warpRun.label = 'SAS.v1' and warpRun.state = 'full' and quality = 0 group by skycell_id, filter, warpRun.label;
+--------+--------------+--------+---------+----------+
| label  | skycell_id   | filter | quality | count(*) |
+--------+--------------+--------+---------+----------+
| sas.v1 | skycell.1224 | g      |       0 |       34 | 
| sas.v1 | skycell.1224 | i      |       0 |       55 | 
| sas.v1 | skycell.1224 | r      |       0 |       32 | 
| sas.v1 | skycell.1224 | z      |       0 |        5 | 
| sas.v1 | skycell.1225 | g      |       0 |      115 | 
| sas.v1 | skycell.1225 | i      |       0 |      226 | 
| sas.v1 | skycell.1225 | r      |       0 |      116 | 
| sas.v1 | skycell.1225 | z      |       0 |       20 | 
| sas.v1 | skycell.1226 | g      |       0 |       95 | 
| sas.v1 | skycell.1226 | i      |       0 |      203 | 
| sas.v1 | skycell.1226 | r      |       0 |      105 | 
| sas.v1 | skycell.1226 | z      |       0 |       23 | 
| sas.v1 | skycell.1227 | g      |       0 |       32 | 
| sas.v1 | skycell.1227 | i      |       0 |       73 | 
| sas.v1 | skycell.1227 | r      |       0 |       33 | 
| sas.v1 | skycell.1227 | z      |       0 |        9 | 
| sas.v1 | skycell.1314 | g      |       0 |      106 | 
| sas.v1 | skycell.1314 | i      |       0 |      223 | 
| sas.v1 | skycell.1314 | r      |       0 |      100 | 
| sas.v1 | skycell.1314 | z      |       0 |       35 | 
| sas.v1 | skycell.1315 | g      |       0 |      253 | 
| sas.v1 | skycell.1315 | i      |       0 |      522 | 
| sas.v1 | skycell.1315 | r      |       0 |      251 | 
| sas.v1 | skycell.1315 | z      |       0 |       67 | 
| sas.v1 | skycell.1316 | g      |       0 |      214 | 
| sas.v1 | skycell.1316 | i      |       0 |      477 | 
| sas.v1 | skycell.1316 | r      |       0 |      205 | 
| sas.v1 | skycell.1316 | y      |       0 |        4 | 
| sas.v1 | skycell.1316 | z      |       0 |       67 | 
| sas.v1 | skycell.1317 | g      |       0 |       68 | 
| sas.v1 | skycell.1317 | i      |       0 |      148 | 
| sas.v1 | skycell.1317 | r      |       0 |       70 | 
| sas.v1 | skycell.1317 | z      |       0 |       16 | 
| sas.v1 | skycell.1404 | g      |       0 |      125 | 
| sas.v1 | skycell.1404 | i      |       0 |      234 | 
| sas.v1 | skycell.1404 | r      |       0 |      101 | 
| sas.v1 | skycell.1404 | z      |       0 |       27 | 
| sas.v1 | skycell.1405 | g      |       0 |      253 | 
| sas.v1 | skycell.1405 | i      |       0 |      519 | 
| sas.v1 | skycell.1405 | r      |       0 |      229 | 
| sas.v1 | skycell.1405 | y      |       0 |        3 | 
| sas.v1 | skycell.1405 | z      |       0 |       80 | 
| sas.v1 | skycell.1406 | g      |       0 |      214 | 
| sas.v1 | skycell.1406 | i      |       0 |      471 | 
| sas.v1 | skycell.1406 | r      |       0 |      201 | 
| sas.v1 | skycell.1406 | y      |       0 |       10 | 
| sas.v1 | skycell.1406 | z      |       0 |       67 | 
| sas.v1 | skycell.1407 | g      |       0 |       72 | 
| sas.v1 | skycell.1407 | i      |       0 |      156 | 
| sas.v1 | skycell.1407 | r      |       0 |       69 | 
| sas.v1 | skycell.1407 | y      |       0 |        4 | 
| sas.v1 | skycell.1407 | z      |       0 |       17 | 
| sas.v1 | skycell.1493 | g      |       0 |       44 | 
| sas.v1 | skycell.1493 | i      |       0 |       72 | 
| sas.v1 | skycell.1493 | r      |       0 |       37 | 
| sas.v1 | skycell.1493 | z      |       0 |       11 | 
| sas.v1 | skycell.1494 | g      |       0 |      110 | 
| sas.v1 | skycell.1494 | i      |       0 |      188 | 
| sas.v1 | skycell.1494 | r      |       0 |       85 | 
| sas.v1 | skycell.1494 | y      |       0 |        2 | 
| sas.v1 | skycell.1494 | z      |       0 |       26 | 
| sas.v1 | skycell.1495 | g      |       0 |      102 | 
| sas.v1 | skycell.1495 | i      |       0 |      199 | 
| sas.v1 | skycell.1495 | r      |       0 |       94 | 
| sas.v1 | skycell.1495 | y      |       0 |        7 | 
| sas.v1 | skycell.1495 | z      |       0 |       19 | 
| sas.v1 | skycell.1496 | g      |       0 |       44 | 
| sas.v1 | skycell.1496 | i      |       0 |       70 | 
| sas.v1 | skycell.1496 | r      |       0 |       41 | 
| sas.v1 | skycell.1496 | y      |       0 |        2 | 
| sas.v1 | skycell.1496 | z      |       0 |        1 | 
| sas.v1 | skycell.1582 | g      |       0 |        1 | 
| sas.v1 | skycell.1582 | i      |       0 |        1 | 
| sas.v1 | skycell.1583 | i      |       0 |        3 | 
| sas.v1 | skycell.1583 | r      |       0 |        2 | 
| sas.v1 | skycell.1584 | g      |       0 |        3 | 
| sas.v1 | skycell.1584 | i      |       0 |        9 | 
| sas.v1 | skycell.1584 | r      |       0 |        2 | 
| sas.v1 | skycell.1584 | y      |       0 |        1 | 
| sas.v1 | skycell.1585 | i      |       0 |        1 | 
+--------+--------------+--------+---------+----------+
80 rows in set (0.07 sec)

and here's the breakdown of quality:

mysql> select quality, count(*) from warpSkyfile join warpRun using (warp_id) where label = 'SAS.v1' group by quality;
+---------+----------+
| quality | count(*) |
+---------+----------+
|       0 |     8039 | 
|     257 |      629 | 
|    3006 |      651 | 
|    8007 |     1409 | 
+---------+----------+
4 rows in set (0.13 sec)

right this moment I'm not going to look into why so many things have such crummy quality.

This is my stacktool command:

stacktool -definebyquery -set_workdir neb://@HOST@.0/isp/sas.v0 -set_label SAS.v0 -select_skycell_id skycell.1496 -select_label sas.v1 -dbname isp -min_num 2

I just went from the last skycell up and kept queueing until they stopped faulting. That happened at skycell.1496, which is taking a while because I am running it on ipp007, and the load is already high from LAP.

ooooh... 2 interesting reasons why this didn't work:

  • I forgot about the oof guys and queued them up
  • default recipe for convolve does crazy stuff (maybe because of the oofs, maybe for other reasons)

Here is a picture of conv and unconv stack (zoomed in, about 1/4 of the skycell). It's actually not too bad, considering...

One of the stacks (8) doesn't have any of the bad focus ones. I pulled up the unconvolved and one of it's warps. Some problems:

  • not visible, but the edge of the skycell you can tell that the astrometry isn't perfect. This effect is not apparent on stack_id = 6.
  • the convolved fits file for stack_id = 8 looks just as bad as for the one with bad focus. Something weird is happening, and it is not related to focus.

Here is a comparison of stack vs warp (zoomed in, this is about 1/4 of the skycell)

Here is the proper query to get what we want (fwhm < 5):

select warpRun.label,skycell_id, filter, warpSkyfile.quality, count(*) from rawExp join chipRun using (exp_id) join camRun using (chip_id) join 
camProcessedExp using (cam_id) join fakeRun using (cam_id) join warpRun using (fake_id) join warpSkyfile using (warp_id) where warpRun.label = 'SAS.v1' 
and warpRun.state = 'full' and warpSkyfile.quality = 0 and fwhm_major < 5 group by skycell_id, filter, warpRun.label;
+--------+--------------+--------+---------+----------+
| label  | skycell_id   | filter | quality | count(*) |
+--------+--------------+--------+---------+----------+
| sas.v1 | skycell.1224 | g      |       0 |       34 | 
| sas.v1 | skycell.1224 | i      |       0 |       52 | 
| sas.v1 | skycell.1224 | r      |       0 |       32 | 
| sas.v1 | skycell.1224 | z      |       0 |        5 | 
| sas.v1 | skycell.1225 | g      |       0 |      115 | 
| sas.v1 | skycell.1225 | i      |       0 |      212 | 
| sas.v1 | skycell.1225 | r      |       0 |      115 | 
| sas.v1 | skycell.1225 | z      |       0 |       16 | 
| sas.v1 | skycell.1226 | g      |       0 |       95 | 
| sas.v1 | skycell.1226 | i      |       0 |      182 | 
| sas.v1 | skycell.1226 | r      |       0 |      104 | 
| sas.v1 | skycell.1226 | z      |       0 |       15 | 
| sas.v1 | skycell.1227 | g      |       0 |       32 | 
| sas.v1 | skycell.1227 | i      |       0 |       60 | 
| sas.v1 | skycell.1227 | r      |       0 |       33 | 
| sas.v1 | skycell.1227 | z      |       0 |        3 | 
| sas.v1 | skycell.1314 | g      |       0 |      106 | 
| sas.v1 | skycell.1314 | i      |       0 |      208 | 
| sas.v1 | skycell.1314 | r      |       0 |      100 | 
| sas.v1 | skycell.1314 | z      |       0 |       26 | 
| sas.v1 | skycell.1315 | g      |       0 |      253 | 
| sas.v1 | skycell.1315 | i      |       0 |      496 | 
| sas.v1 | skycell.1315 | r      |       0 |      250 | 
| sas.v1 | skycell.1315 | z      |       0 |       58 | 
| sas.v1 | skycell.1316 | g      |       0 |      214 | 
| sas.v1 | skycell.1316 | i      |       0 |      446 | 
| sas.v1 | skycell.1316 | r      |       0 |      204 | 
| sas.v1 | skycell.1316 | y      |       0 |        1 | 
| sas.v1 | skycell.1316 | z      |       0 |       60 | 
| sas.v1 | skycell.1317 | g      |       0 |       68 | 
| sas.v1 | skycell.1317 | i      |       0 |      134 | 
| sas.v1 | skycell.1317 | r      |       0 |       70 | 
| sas.v1 | skycell.1317 | z      |       0 |       13 | 
| sas.v1 | skycell.1404 | g      |       0 |      125 | 
| sas.v1 | skycell.1404 | i      |       0 |      217 | 
| sas.v1 | skycell.1404 | r      |       0 |      100 | 
| sas.v1 | skycell.1404 | z      |       0 |       21 | 
| sas.v1 | skycell.1405 | g      |       0 |      253 | 
| sas.v1 | skycell.1405 | i      |       0 |      492 | 
| sas.v1 | skycell.1405 | r      |       0 |      228 | 
| sas.v1 | skycell.1405 | y      |       0 |        1 | 
| sas.v1 | skycell.1405 | z      |       0 |       69 | 
| sas.v1 | skycell.1406 | g      |       0 |      214 | 
| sas.v1 | skycell.1406 | i      |       0 |      450 | 
| sas.v1 | skycell.1406 | r      |       0 |      201 | 
| sas.v1 | skycell.1406 | y      |       0 |        1 | 
| sas.v1 | skycell.1406 | z      |       0 |       66 | 
| sas.v1 | skycell.1407 | g      |       0 |       72 | 
| sas.v1 | skycell.1407 | i      |       0 |      148 | 
| sas.v1 | skycell.1407 | r      |       0 |       69 | 
| sas.v1 | skycell.1407 | z      |       0 |       16 | 
| sas.v1 | skycell.1493 | g      |       0 |       44 | 
| sas.v1 | skycell.1493 | i      |       0 |       65 | 
| sas.v1 | skycell.1493 | r      |       0 |       37 | 
| sas.v1 | skycell.1493 | z      |       0 |        7 | 
| sas.v1 | skycell.1494 | g      |       0 |      110 | 
| sas.v1 | skycell.1494 | i      |       0 |      173 | 
| sas.v1 | skycell.1494 | r      |       0 |       84 | 
| sas.v1 | skycell.1494 | z      |       0 |       18 | 
| sas.v1 | skycell.1495 | g      |       0 |      102 | 
| sas.v1 | skycell.1495 | i      |       0 |      186 | 
| sas.v1 | skycell.1495 | r      |       0 |       94 | 
| sas.v1 | skycell.1495 | z      |       0 |       18 | 
| sas.v1 | skycell.1496 | g      |       0 |       44 | 
| sas.v1 | skycell.1496 | i      |       0 |       63 | 
| sas.v1 | skycell.1496 | r      |       0 |       41 | 
| sas.v1 | skycell.1496 | z      |       0 |        1 | 
| sas.v1 | skycell.1582 | g      |       0 |        1 | 
| sas.v1 | skycell.1582 | i      |       0 |        1 | 
| sas.v1 | skycell.1583 | i      |       0 |        3 | 
| sas.v1 | skycell.1583 | r      |       0 |        2 | 
| sas.v1 | skycell.1584 | g      |       0 |        3 | 
| sas.v1 | skycell.1584 | i      |       0 |        9 | 
| sas.v1 | skycell.1584 | r      |       0 |        2 | 
| sas.v1 | skycell.1585 | i      |       0 |        1 | 
+--------+--------------+--------+---------+----------+
75 rows in set (0.05 sec)

That has a max fwhm of 5.... I shuffled the old stacks to SAS.v0.nocut, so I will requeue with the fwhm cut to see how that turns out...

stacktool -definebyquery -set_workdir neb://@HOST@.0/isp/sas.v0 -set_label SAS.v0 -select_skycell_id skycell.1496 -select_label sas.v1 -dbname isp -min_num 2 -select_fwhm_major_max 5

2013-03-28

Ok, return to SAS from long ago. Here is what I have learned:

  • stacks were not complete and the majority had faulted (I assume due to LAP processing hogging stuff from before). I restarted the stacks, they finished, but they have problems:
    • overrejection of images (why?)
    • there are nan edges on everything (why?)
    • we need Gene to help with the config I think
  • where are the zeropoints?
    • after a lot of investigations, the zeropoints are there, but the default doesn't grab from the chip_header - this has since been fixed by gene in addstar_scripts. I think if I remake the dvo it will work.
  • make a sas dvodb, since I am blocked on stacks at the moment. first of all, where are we:
mysql> select count(*), state, quality, fault from camRun join camProcessedExp using (cam_id) where label = 'SAS.v0' group by state, quality, fault;
+----------+-------+---------+-------+
| count(*) | state | quality | fault |
+----------+-------+---------+-------+
|     2478 | full  |       0 |     0 | 
|      822 | full  |    4007 |     0 | 
+----------+-------+---------+-------+
2 rows in set (0.03 sec)

and here are the 'good' images:

mysql> select count(*) from camRun join camProcessedExp using (cam_id) where label = 'SAS.v0' and quality = 0 and fault = 0 and fwhm_major < 5 and sigma_ra < 1 and  n_astrom >0;
+----------+
| count(*) |
+----------+
|     1356 | 
+----------+
1 row in set (0.01 sec)

These cuts are as above, but listed here for posterity:

  • fwhm_major < 5 (or else it is likely out of focus
  • sigma_ra < 1 or else the astrometry is likely bad
  • n_astrom > 0 or else the astrometry IS bad

There are no tools for addtool to query that. I think I have to do addtool -definebyquery 1300 times to make a dvo. I can do that

Attachments