LAP Czaring

• Return to processing documentation
• Return to LAP documentation

Overall czaring sequence

• Check the current runs: laptool -listrun -seq_id 8 -state run
  • If everything was recently registered, or has a defined stack association id, it's probably ok.
  • If a run is old, or looks stuck, look at the exposures: laptool -exposures -lap_id LLL
    • Is something stuck because of a fault? Revert it.
    • Is something failing after a revert? It probably has a corrupt file, check the log and fix it.
    • Is something weird? If you can't figure out why exposure X is holding things up, kick it out so we can move on: laptool -updateexp -lap_id LLL -exp_id XXX -set_data_state drop

Problems that may arise due to old data

• The major issue is the loss of raw imfiles due to now-resolved problems in the shuffle code. If a chip imfile continually fails, and neb-repair notes that the raw file is "consistent" (this is mentioned in the chip stage log file), the file is probably corrupt. This can be verified by running neb-stat --validate, and checking that there are no good instances of this file anymore. To remove this file from processing, and to mark it as bad to prevent future processing, do the following commands:

   chiptool -updateprocessedimfile -set_state full -chip_id CHIP_ID -class_id CLASS_ID
   chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id CHIP_ID -class_id CLASS_ID
   regtool -updateprocessedimfile -set_state corrupt -class_id CLASS_ID -exp_id EXP_ID
  

Examples

The key to the lapExp lines below:

entry	note
lap_id
tess_id
projection_cell
filter
lapRun_state
registered	Date the lapRun was created. Ages greater than 24 hours suggest a problem.
fault	lapRun.fault Currently not set because we don't catch any fault states.
label
dist_group
exp_id	exp_id for this exposure
chip_id	chip_id for the processing of this exposure
pair_id	chip_id of the exposure that will be paired with this one for differencing
private	Boolean for "Will this exposure possibly span multiple projection_cells in the diff?"
pairwise	Boolean: "Will this exposure be pairwise diffed?"
active	Boolean: "Is this exposure currently in use?"
data_state	lapExp.data_state
chipRun_state	state of chipRun
chip_bad_quality	count of components with non-zero quality. Used to determine if an exposure should be dropped
chip_component_count	count of all components
cam_id	cam_id associated with this exposure
camRun_state	state of camRun
cam_bad_quality	count of components with non-zero quality
cam_component_count	count of all components
fake_id
fakeRun_state
warp_id
warpRun_state
warp_bad_quality
warp_component_count
magicked	warpRun.magicked. Used to check if the warps have been destreaked for use in stacks.
diff_id
diffRun_state
diff_bad_quality
diff_component_count

After calling laptool -exposures -lap_id 1230 -simple:

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97873 307033 307034 F T T full full 0 60 283612 full 0 1 267013 full 270040 full 12 71 219206 174035 cleaned 2 59

This is fine. It was generated from an update call, and so that's why the diffRun.state is cleaned, but the other states are full.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97902 307036 307035 F T T to_magic update 0 60 283614 full 0 1 267017 full 270044 update 10 72 -1 174036 cleaned 4 60

This is also ok. It's another exposure being updated, but is waiting on the chip stage to finish updating. The lapExp.data_state value of "to_magic" denotes that LAP is waiting for this to destreak the warp stage.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 98666 319194 319195 F T T full full 0 60 296302 full 0 1 279553 full 281037 full 7 68 225586 178478 full 4 61

This exposure was first processed as part of this lapRun, and has completely finished regular processing. There should be no problems with this exposure.

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 101218 307039 9223372036854775807 T F T to_diff full 0 60 283617 full 0 1 267018 full 270045 full 10 71 0 9223372036854775807  0 0

This exposure does not have a pair, and so the private bit is set to TRUE, and the pairwise bit is set to FALSE. Since we have not made any stacks yet, this exposure is waiting for that. Since it is likely to span multiple projection cells, it may not get a diff as part of this lapRun. In this case, it will have lapExp.data_state set to 'drop' when the lapRun reaches the diff check.

1241 RINGS.V3 skycell.1586 r.00000 run 2011-10-11T15:26:03.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97951 319256 319258 F T T to_diff full 0 60 296359 full 0 1 279618 full 281149 full 6 67 0 9223372036854775807  0 0

Regular pairwise diff. It's waiting on other data so the stacks can be made, and once that's done, will do the diff.

1241 RINGS.V3 skycell.1586 r.00000 run 2011-10-11T15:26:03.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 97976 319461 319462 F T T new new 0 0 9223372036854775807  0 0 9223372036854775807  9223372036854775807  0 0 9223372036854775807 9223372036854775807  0 0

Regular new exposure. No processing has been done yet, but there shouldn't be any problems with it.

1241 RINGS.V3 skycell.1586 r.00000 run 2011-10-11T15:26:03.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 98678 319235 319233 F T T exists full 0 60 296343 full 0 1 279655 full 281187 full 4 68 0 9223372036854775807  0 0

Similarly, this is an exposure that was already largely processed as part of another lapRun.

1209 RINGS.V3 skycell.1408 g.00000 run 2011-10-10T13:55:40.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 384906 314005 314002 F T T drop cleaned 0 60 291112 full 1 1 9223372036854775807  9223372036854372036854775807  0 0

This exposure was dropped from further processing because it has a bad quality value at the camera stage (probably a problem with the astrometry).

1230 RINGS.V3 skycell.1405 r.00000 run 2011-10-11T01:58:46.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 209356 307042 307045 F T T to_magic error_cleaned 0 60 283621 full 0 1 267023 full 270050 update 13 75 -1 174042 cleaned 3 62

The chip processing for this exposure wasn't fully cleaned, and due to a peculiarity with the update code, it wasn't fully updated. This can be fixed by calling chiptool:

chiptool -updateprocessedimfile -set_state cleaned -chip_id 307042 -class_id XY46
chiptool -updaterun -set_state cleaned -chip_id 307042
chiptool -setimfiletoupdate -chip_id 307042 -set_label LAP.ThreePi.20110809

1222 RINGS.V3 skycell.1494 i.00000 run 2011-10-10T17:49:40.000000 0 LAP.ThreePi.20110809 LAP.ThreePi 208760 306559 306557 F T T to_magic update 0 60 282803 full 0 1 266222 full 269249 update 7 71 -1 173576 cleaned 5 64

This exposure looks fine (it's waiting for update to finish), but as update faults aren't shown in czartool, there may be problems. chiptool sorts the jobs by chip_id, so update jobs (which have smaller, older chip_id values) always run before new jobs. If you see a lapExp line like this, but are processing new jobs, then there's probably a hidden fault that needs to be tracked down and fixed.