Changes between Version 189 and Version 190 of Processing

Timestamp:

Sep 9, 2013, 1:41:02 PM (13 years ago)

Author:

watersc1

Comment:

--

Processing

@@ -323 +323 @@
 LAP should run largely without any intervention other than regular reverting and repairing corrupt files.  [wiki:LAP_Czar A detailed look at how to check the status of LAP processing is given here.] 
 
+== CZW PV2 issues ==
+
+I have written a few scripts that I've been using to quietly fix PV2 processing issues.  The details are given below.  Note that none of these scripts call any commands that update/reprocess/copy data.  After identifying a suggested course of action, the commands that should be issued are printed to the screen.  This prevents accidents that could arise from undefined values in the scripts, requiring a person to copy and execute these commands.
+
+=== Stacking faults ===
+
+There are remaining unresolved errors in ppStack that cause some stacks to fail with fault 4 or 5.  LAP does not block on these faults, as they are considered unrepairable.  These can probably be resolved after the ppStack errors are fully diagnosed and repaired.  However, until that happens, I've been simply setting these stackRuns to 'drop':
+{{{
+ stacktool -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 4
+ stacktool -updaterun -set_state drop -state new -label LAP.ThreePi.20130717 -fault 5
+}}}
+
+Another error that is less clear is the occurrence of fault 2 issues with stacks.  These are rarely due to actual NFS errors, but seem to be caused by errors in the warp update process.  The symptom is that the warpSkyfile is set to data_state = 'full', but without the pswarp command actually completing successfully.  These cause the stacks to continually revert and fail due to the missing warpSkyfile.  I've written a script {{{/home/panstarrs/watersc1/PV2.LAP.20130717/fixes/unstick_fault2.pl}}} to help fix these faults.  The script scans the logfiles for the faulted stacks, uses the missing file mentioned in that file to determine what needs to be done to fix the missing file.  It then prints the {{{runwarpskycell.pl}}} that needs to be run to fix this missing file.  In the case that the chip images have been cleaned for this exposure (based on the message printed in the logfile), the appropriate chiptool command is also returned to update it.  As the stack log doesn't change, if a chip update is required, it will continue to print that message, even after the chipRun has been updated.
+
+=== Missing OTA/BT TABLE files ===
+
+Due to the failure of ipp047, I've been using another script {{{/home/panstarrs/watersc1/bin/chip_auto_repair_helper.pl}}} to generate commands that can be used to identify hidden copies of missing single instance raw files.  After identifying which chipImfile has faulted, it checks the existence of the fits image, and if that is found and looks reasonable (exists, not zero-byte), it checks the associated burntool table.  If one of these files is missing a reasonable copy, the {{{deneb-locate.py}}} script is called to look for a hidden copy.  If one is found, the appropriate copy command is printed.  A large amount of diagnostic/md5sum information is also returned for verification.
 = Queuing/Dequeuing data =