Changes between Version 7 and Version 8 of CleanupUpdate

Timestamp:

Nov 15, 2012, 12:57:23 PM (14 years ago)

Author:

bills

Comment:

--

CleanupUpdate

@@ -43 +43 @@
 }}}
 
-compress log file. Should we rename it or keep it in place?
+When cleanup is successfully completed gzip the log file. Sample of one 91% reduction in size 517698 compressed to 46548. Perhaps
+make the script a bit less verbose in success cases.
 
 === Chip stage ===
-find smf for associated camRun. If found and is not corrupted, delete cmfs otherwise leave them on disk.
+    - find smf for associated camRun. If found and is not corrupted, delete cmfs otherwise leave them on disk.
+    - delete b1 fits ?
+    - delete trace files ? (they are empty)
+    - compress log files
 
+Once implemented we will start one big pass through the cleaned up data to implement these space reductions. This will consist of
+ * Modify query for pendingcleanupimfile to (optionally) list all components not just those in full state
+ * set cleaned runs back to state goto_cleaned with a different label so we can distinguish between regular and second pass cleanup
+=== Warp stage ===
+    - modify query to optionally list all components not just those in full state
+    - delete sources (they are regenerated by update anyways)
+    - delete skycell template files (they are regnerated by update)
+    - delete psf file (it is regenerated on by update)
+    - compress log files
 
-{{{
-    if (goto_purged)
-        delete psf
-        should we delete the background model? 
-            It is only useful if the corresponding warp is in state full.
-    if (goto_scrubbed) 
-        delete all of the files we delete in goto_purged as well
-
-}}}
-
-CZW: The background model should be retained unless we are confident we never wish to use this run (or it's warp/stack descendants) again.  The code I'm working on now allows for the generation of stack-level background models for runs with cleaned chips and warps, so the above statement that it's only useful if the warp is full is no longer accurate.
-=== Warp stage ===
-
-{{{
-    if (goto_purged)
-        delete cmf
-        delete psf
-        delete skycell.template
-
-    if (goto_scrubbed)
-        delete the files deleted in goto_purged as well as the normal cleanup files
-
-}}}
+=== Camera Stage ===
+We found about 15000 censored camera runs that were not restored. I missed some labels apparently. These are running now. This will
+remove 2 copies of the smf for each run.
 
 === Changes to other stages ===