LAP Czaring

• Return to processing documentation
• Return to LAP documentation

Overall czaring sequence

• Check the current runs: laptool -listrun -seq_id 8 -state run
  • If everything was recently registered, or has a defined stack association id, it's probably ok.
  • If a run is old, or looks stuck, look at the exposures: laptool -exposures -lap_id LLL
    • Is something stuck because of a fault? Revert it.
    • Is something failing after a revert? It probably has a corrupt file, check the log and fix it using the runXXXfile.pl command from the tools directory.
    • Is something weird? If you can't figure out why exposure X is holding things up, kick it out so we can move on: laptool -updateexp -lap_id LLL -exp_id XXX -set_data_state drop

Problems that may arise due to old data

• The major issue is the loss of raw imfiles due to now-resolved problems in the shuffle code. If a chip imfile continually fails, and neb-repair notes that the raw file is "consistent" (this is mentioned in the chip stage log file), the file is probably corrupt. This can be verified by running neb-stat --validate, and checking that there are no good instances of this file anymore. Since the remaining files are valid, we do not want to drop the entire exposure from the lapRun. To remove only this file from processing, and to mark it as bad to prevent future processing, run the following commands:

   chiptool -updateprocessedimfile -set_state full -chip_id CHIP_ID -class_id CLASS_ID
   chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id CHIP_ID -class_id CLASS_ID
   regtool -updateprocessedimfile -set_state corrupt -class_id CLASS_ID -exp_id EXP_ID
  

• revised Nov. 2011 -

  chiptool -dropprocessedimfile -set_quality 42 -chip_id 337393 -class_id XY53
  regtool -updateprocessedimfile -set_ignored -exp_id 195736 -class_id XY53
  

Be aware that ippb00 is currently having disk issues, and is going up and down. I dropped a few OTAs from further processing that had a missing MHPCC instance and an instance on ippb00 that was also being reported as missing, but since this is also the case when the host is down, I did not flag the raw imfile as corrupt.

Defining a new region to work on

In the ipp/trunk/tools directory is a script LAP_regiontool.pl. This takes the boundaries of the region to be defined, and produces a list of laptool -definerun commands that can be used as a queuelist file to ensure new runs are automatically launched as old ones complete (defined in pantasks using lap.define.queue /path/to/queuelist.file). The RA is assumed to be in hours unless the --degrees option is passed. The commands to generate the current list of regions are listed below:

 alias LAP_RT 'LAP_regiontool.pl --seq_id 11 --label LAP.ThreePi.20120706 --dist_group LAP.ThreePi --tess_id RINGS.V3 --radius 5 --dbname gpc1'
 LAP_RT --ra_min 21 --ra_max 22 --dec_min -30 --dec_max 10 > region_A.queue
 LAP_RT --ra_min 23 --ra_max 24 --dec_min -10 --dec_max 10 > region_B.queue
 LAP_RT --ra_min 20 --ra_max 21 --dec_min -30 --dec_max -20 > region_C.queue
 LAP_RT --ra_min 22 --ra_max 23 --dec_min -10 --dec_max 10 > region_D.queue
 LAP_RT --ra_min 23 --ra_max 24 --dec_min 30 --dec_max 50 > region_E.queue
 LAP_RT --ra_min 21 --ra_max 23 --dec_min 30 --dec_max 40 > region_F.queue
 LAP_RT --ra_min 21 --ra_max 22 --dec_min 10 --dec_max 30 > region_G.queue
 LAP_RT --ra_min 18 --ra_max 22 --dec_min 70 --dec_max 90 > region_H.queue

 LAP_RT --ra_min 22 --ra_max 23 --dec_min 10 --dec_max 30 > region_I.queue
 LAP_RT --ra_min 23 --ra_max 24 --dec_min 10 --dec_max 30 > region_J.queue
 LAP_RT --ra_min 22 --ra_max 23 --dec_min -30 --dec_max -10 > region_K.queue
 LAP_RT --ra_min 23 --ra_max 24 --dec_min -30 --dec_max -10 > region_L.queue
 LAP_RT --ra_min 20 --ra_max 21 --dec_min -20 --dec_max 30 > region_M.queue

 LAP_RT --ra_min 21 --ra_max 22 --dec_min 40 --dec_max 70  > region_N.queue
 LAP_RT --ra_min 22 --ra_max 23 --dec_min 40 --dec_max 90  > region_O.queue
 LAP_RT --ra_min 23 --ra_max 24 --dec_min 50 --dec_max 90  > region_P.queue

 LAP_RT --ra_min 20 --ra_max 21 --dec_min 30 --dec_max 70  | tac > region_Q.queue
 LAP_RT --ra_min 19 --ra_max 20 --dec_min -30 --dec_max 70 | tac > region_R.queue
 LAP_RT --ra_min 18 --ra_max 19 --dec_min -30 --dec_max 70 | tac > region_S.queue

 LAP_RT --ra_min 0 --ra_max 1 --dec_min -30 --dec_max 90 > 00hours.queue
 LAP_RT --ra_min 1 --ra_max 2 --dec_min -30 --dec_max 90 | tac > 01hours.queue

Caveat: this script blocks the region into bands in declination, to allow regions of five filter coverage to be generated quickly. Because of this, it cycles through the filters in a different way for each block, as it selects the last filter of the previous block as the first filter of the current block.

Increasing the number of simultaneous active lapRuns

Given a queue list file, the number of active lapRuns can be increased by finding the command line for the next unprocessed projection cell/filter, and issuing that command. However, only one command should be issued at any time. Due to the way the exposure sharing works, having two lapRuns with the state of 'new' in the same filter can cause multiple chipRuns to be queued for the same exposure. To prevent this from happening, wait until there are no more lapRuns with state 'new' before adding a second lapRun.

For safety, it's generally best to use the trickle_add.pl script in the ~ipp/lap directory to launch multiple new runs. This script accepts a set of laptool -definerun commands in a file (or from stdin, such as with head -10 queue | trickle_add.pl), and launches them one at a time, ensuring that the previous command is in state run before issuing the next command. This helps the exposure linking so that all exposures are shared fully.

Decreasing the number of simultaneous active lapRuns

If there are too many currently active runs, you must wait for the processing to finish. A replacement run is queued during the cleanup phase, using the queue list file defined in pantasks using 'lap.define.queue'. If this queue file is not defined (or points at a non-existing file, or an empty file, etc.), no replacement run will be queued, and the cleanup phase will clean and deactivate the finished run. To make this easier, an empty queue file exists ('/home/panstarrs/ipp/lap/off.queue') that can be used to cull down the number of active runs.

Determining when a block is finished

Projection cells are identified by an index number that increases by one with increasing RA, starting at 0 degrees. Once an entire band is finished at that declination, the declination is incremented by 4 degrees and another band of projection cells is defined, again incrementing by one. The consequence of this is that projection cells in the same part of the sky have large jumps in the index number as the declination changes. As the five color blocks are also divided by declination, these jumps can be used to determine when a block has been fully processed. Looking at a section of the output of laptool -listrun -seq_id 8 -simple:

1252 8 RINGS.V3 skycell.1585 y.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T11:36:11.000000 0 9223372036854775807 4534
1253 8 RINGS.V3 skycell.1586 y.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T11:40:29.000000 0 9223372036854775807 4535
1254 8 RINGS.V3 skycell.1999 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T11:45:21.000000 0 9223372036854775807 4537
1255 8 RINGS.V3 skycell.2000 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T11:48:44.000000 0 9223372036854775807 4538
1256 8 RINGS.V3 skycell.2001 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T12:11:46.000000 0 9223372036854775807 4540
1257 8 RINGS.V3 skycell.2002 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T13:02:06.000000 0 9223372036854775807 4541
1258 8 RINGS.V3 skycell.2003 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T13:10:41.000000 0 9223372036854775807 4542
1259 8 RINGS.V3 skycell.2004 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T14:07:01.000000 0 9223372036854775807 4545
1260 8 RINGS.V3 skycell.2005 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T14:16:41.000000 0 9223372036854775807 4546
1261 8 RINGS.V3 skycell.2076 g.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-12T14:27:39.000000 0 9223372036854775807 4550
...
1297 8 RINGS.V3 skycell.2004 r.00000 done LAP.ThreePi.20110809 LAP.ThreePi 2011-10-14T17:00:22.000000 0 9223372036854775807 4599
1298 8 RINGS.V3 skycell.2005 r.00000 run LAP.ThreePi.20110809 LAP.ThreePi 2011-10-14T23:02:57.000000 0 9223372036854775807 4600

LAP run 1298 is the lowest numbered run that is still processing. Therefore the block that contains skycell.2005 is not complete in five colors. Continuing backwards, the jump from skycell.1586 to skycell.1999 is large, so we seem to have full five-color completion for any sequence 8 data prior to 1254. There is another jump between lap_id 1260 and 1261, but that is just a regular step in declination.

Once we know that we've completed processing of exposures in lapRuns < 1254, we can get a list of exp_ids from the database:

select DISTINCT exp_id from lapRun JOIN lapExp USING(lap_id) where seq_id = 8 AND lap_id < 1254;
...
11972 rows in set (0.07 sec)

Examples

The key to the lapExp lines below:

entry	note
lap_id
tess_id
projection_cell
filter
lapRun_state
registered	Date the lapRun was created. Ages greater than 24 hours suggest a problem.
fault	lapRun.fault Currently not set because we don't catch any fault states.
label
dist_group
exp_id	exp_id for this exposure
chip_id	chip_id for the processing of this exposure
pair_id	chip_id of the exposure that will be paired with this one for differencing
private	Boolean for "Will this exposure possibly span multiple projection_cells in the diff?"
pairwise	Boolean: "Will this exposure be pairwise diffed?"
active	Boolean: "Is this exposure currently in use?"
data_state	lapExp.data_state
chipRun_state	state of chipRun
chip_bad_quality	count of components with non-zero quality. Used to determine if an exposure should be dropped
chip_component_count	count of all components
cam_id	cam_id associated with this exposure
camRun_state	state of camRun
cam_bad_quality	count of components with non-zero quality
cam_component_count	count of all components
fake_id
fakeRun_state
warp_id
warpRun_state
warp_bad_quality
warp_component_count
magicked	warpRun.magicked. Used to check if the warps have been destreaked for use in stacks.
diff_id
diffRun_state
diff_bad_quality
diff_component_count

After calling laptool -exposures -lap_id 1230 -simple:

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97873 307033 307034 F T T full full 0 60 283612 full 0 1 267013 full 270040 full 12 71 219206 174035 cleaned 2 59

This is fine. It was generated from an update call, and so that's why the diffRun.state is cleaned, but the other states are full.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97902 307036 307035 F T T to_magic update 0 60 283614 full 0 1 267017 full 270044 update 10 72 -1 174036 cleaned 4 60

This is also ok. It's another exposure being updated, but is waiting on the chip stage to finish updating. The lapExp.data_state value of "to_magic" denotes that LAP is waiting for this to destreak the warp stage.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 98666 319194 319195 F T T full full 0 60 296302 full 0 1 279553 full 281037 full 7 68 225586 178478 full 4 61

This exposure was first processed as part of this lapRun, and has completely finished regular processing. There should be no problems with this exposure.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 101218 307039 9223372036854775807 T F T to_diff full 0 60 283617 full 0 1 267018 full 270045 full 10 71 0 9223372036854775807  0 0

This exposure does not have a pair, and so the private bit is set to TRUE, and the pairwise bit is set to FALSE. Since we have not made any stacks yet, this exposure is waiting for that. Since it is likely to span multiple projection cells, it may not get a diff as part of this lapRun. In this case, it will have lapExp.data_state set to 'drop' when the lapRun reaches the diff check.

1241 RINGS.V3 skycell.1586 r.00000 run 2011-10-11T15:26:03.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97951 319256 319258 F T T to_diff full 0 60 296359 full 0 1 279618 full 281149 full 6 67 0 9223372036854775807  0 0

Regular pairwise diff. It's waiting on other data so the stacks can be made, and once that's done, will do the diff.

1241 RINGS.V3 skycell.1586 r.00000 run 2011-10-11T15:26:03.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97976 319461 319462 F T T new new 0 0 9223372036854775807  0 0 9223372036854775807  9223372036854775807  0 0 9223372036854775807 9223372036854775807  0 0

Regular new exposure. No processing has been done yet, but there shouldn't be any problems with it.

1241 RINGS.V3 skycell.1586 r.00000 run 2011-10-11T15:26:03.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 98678 319235 319233 F T T exists full 0 60 296343 full 0 1 279655 full 281187 full 4 68 0 9223372036854775807  0 0

Similarly, this is an exposure that was already largely processed as part of another lapRun.

1209 RINGS.V3 skycell.1408 g.00000 run 2011-10-10T13:55:40.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 384906 314005 314002 F T T drop cleaned 0 60 291112 full 1 1 9223372036854775807  9223372036854372036854775807  0 0

This exposure was dropped from further processing because it has a bad quality value at the camera stage (probably a problem with the astrometry).

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 209356 307042 307045 F T T to_magic error_cleaned 0 60 283621 full 0 1 267023 full 270050 update 13 75 -1 174042 cleaned 3 62

The chip processing for this exposure wasn't fully cleaned, and due to a peculiarity with the update code, it wasn't fully updated. This can be fixed by calling chiptool:

chiptool -updateprocessedimfile -set_state cleaned -chip_id 307042 -class_id XY46
chiptool -updaterun -set_state cleaned -chip_id 307042
chiptool -setimfiletoupdate -chip_id 307042 -set_label LAP.ThreePi.20110809

1222 RINGS.V3 skycell.1494 i.00000 run 2011-10-10T17:49:40.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 208760 306559 306557 F T T to_magic update 0 60 282803 full 0 1 266222 full 269249 update 7 71 -1 173576 cleaned 5 64

This exposure looks fine (it's waiting for update to finish), but as update faults aren't shown in czartool, there may be problems. chiptool sorts the jobs by chip_id, so update jobs (which have smaller, older chip_id values) always run before new jobs. If you see a lapExp line like this, but are processing new jobs, then there's probably a hidden fault that needs to be tracked down and fixed.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 209356 307042 307045 F T T to_magic cleaned 0 60 283621 full 0 1 267023 full 270050 cleaned 13 75 -1 174042 cleaned 3 62

A line similar to this showed up today (2011-11-01). It seems that this exposure did not get updated when the lapRun was queued, and continued to not get updated when the three other lapRuns that contained it were queued. I don't have a good explanation for this at the moment, but this was easily resolved by running the update commands for the chip, warp, and destreak runs (I grabbed that bit of code out of lap_science.pl, and put it in /home/panstarrs/watersc1/bin/run_update_for_chip_id.pl. This might be useful if this problem crops up again, as I currently don't see any obvious reason why the update wouldn't happen. The script takes the chip_id to be updated, and runs the commands as needed).