This page contains some notes on the IPP cleanup and update mechanism focusing on some changes proposed in November 2012.

While the ipp production cluster has a vast amount of storage, there is not nearly enough to save all the processed data
products. A process has been established to delete large data files retaining enough information that they can be
regenerated at a later date.

=== Cleanup states ===
In the cleanup process runs are transitioned to one of 3 dates
    * cleanup         Delete ephemeral files only. Leave files behind so that images can be updated.
    * purged          More (but perhaps not all) output files are deleted. For example logs survive. Since recipe information is lost images cannot be regenerated from this state.
    * scrubbed        Don't recall what this is for. It seems to delete the same files as regular cleanup. But only if the config dump file is missing. '''XXX: Should this be more like purged or cleaned?
'''

=== Update operation ===
Recipes and other configuration is read from config dump file and those parameters are used to
regenerate the images. The goal is to be as close to the original processing as possible, but since
the code is in general different it might be just "close"


{{{
if (mdc file is not available)
    # this should only apply to runs prior to replication of config dump files
    if (mdc file is temporarily not available)
        images are regenerated using current recipes but dump file is not saved
    else
        images are regenerated using current recipes but new config dump file is saved


}}}

== Proposed Change to chip updates ==
Currently if cmf or psf is missing we rerun photometry.  Change to only do this if the psf file is missing. Do not recreate cmf file.


=== Cleanup Changes ===

== In general ==
redirect output to a nebulous file whose path is based on the workdir. For example

{{{

    neb://ipp040.0/gpc1/ThreePi.DM.20100401/o5238g0323o.133649/o5238g0323o.133649.ch.79656.cleanup.log

}}}

compress log file. Should we rename it or keep it in place?

== chip stage ==
find smf for associated camRun. If found and is not corrupted, delete cmfs otherwise leave them on disk.


{{{
    if (goto_purged)
        delete psf
        should we delete the background model? 
            It is only useful if the corresponding warp is in state full.
    if (goto_scrubbed) 
        delete all of the files we delete in goto_purged as well

}}}

== warp stage ==

{{{
    if (goto_purged)
        delete cmf
        delete psf
        delete skycell.template

    if (goto_scrubbed)
        delete the files deleted in goto_purged as well as the normal cleanup files

}}}

Changes to other stages: TBD