Changes between Initial Version and Version 1 of CleanupUpdate

Timestamp:

Nov 14, 2012, 2:31:44 PM (14 years ago)

Author:

bills

Comment:

--

CleanupUpdate

@@ +1 @@
+This page contains some notes on the IPP cleanup and update mechanism focusing on some changes proposed in November 2012.
+
+While the ipp production cluster has a vast amount of storage, there is not nearly enough to save all the processed data
+products. A process has been established to delete large data files retaining enough information that they can be
+regenerated at a later date.
+
+=== Cleanup states ===
+In the cleanup process runs are transitioned to one of 3 dates
+    * cleanup         Delete ephemeral files only. Leave files behind so that images can be updated.
+    * purged          More (but perhaps not all) output files are deleted. For example logs survive. Since recipe information is lost images cannot be regenerated from this state.
+    * scrubbed        Don't recall what this is for. It seems to delete the same files as regular cleanup. But only if the config dump file is missing. '''XXX: Should this be more like purged or cleaned?
+'''
+
+=== Update operation ===
+Recipes and other configuration is read from config dump file and those parameters are used to
+regenerate the images. The goal is to be as close to the original processing as possible, but since
+the code is in general different it might be just "close"
+
+
+{{{
+if (mdc file is not available)
+    # this should only apply to runs prior to replication of config dump files
+    if (mdc file is temporarily not available)
+        images are regenerated using current recipes but dump file is not saved
+    else
+        images are regenerated using current recipes but new config dump file is saved
+
+
+}}}
+
+== Proposed Change to chip updates ==
+Currently if cmf or psf is missing we rerun photometry.  Change to only do this if the psf file is missing. Do not recreate cmf file.
+
+
+=== Cleanup Changes ===
+
+== In general ==
+redirect output to a nebulous file whose path is based on the workdir. For example
+
+{{{
+
+    neb://ipp040.0/gpc1/ThreePi.DM.20100401/o5238g0323o.133649/o5238g0323o.133649.ch.79656.cleanup.log
+
+}}}
+
+compress log file. Should we rename it or keep it in place?
+
+== chip stage ==
+find smf for associated camRun. If found and is not corrupted, delete cmfs otherwise leave them on disk.
+
+
+{{{
+    if (goto_purged)
+        delete psf
+        should we delete the background model? 
+            It is only useful if the corresponding warp is in state full.
+    if (goto_scrubbed) 
+        delete all of the files we delete in goto_purged as well
+
+}}}
+
+== warp stage ==
+
+{{{
+    if (goto_purged)
+        delete cmf
+        delete psf
+        delete skycell.template
+
+    if (goto_scrubbed)
+        delete the files deleted in goto_purged as well as the normal cleanup files
+
+}}}
+
+Changes to other stages: TBD