This page contains some notes on the IPP cleanup and update mechanism focusing on some changes proposed in November 2012.

While the ipp production cluster has a vast amount of storage, there is not nearly enough to save all the processed data products. A process has been established to delete large data files retaining enough information that they can be regenerated at a later date.

Cleanup states

In the cleanup process runs are transitioned to one of 3 dates

• cleanup Delete ephemeral files only. Leave files behind so that images can be updated.
• purged More (but perhaps not all) output files are deleted. For example logs survive. Since recipe information is lost images cannot be regenerated from this state.
• scrubbed Don't recall what this is for. It seems to delete the same files as regular cleanup. But only if the config dump file is missing. XXX: Should this be more like purged or cleaned?

Update operation

Recipes and other configuration is read from config dump file and those parameters are used to regenerate the images. The goal is to be as close to the original processing as possible, but since the code is in general different it might be just "close"

if (mdc file is not available)
    # this should only apply to runs prior to replication of config dump files
    if (mdc file is temporarily not available)
        images are regenerated using current recipes but dump file is not saved
    else
        images are regenerated using current recipes but new config dump file is saved

Proposed Change to chip updates

Currently if cmf or psf is missing we rerun photometry. Change to only do this if the psf file is missing. Do not recreate cmf file.

Cleanup Changes

General

redirect output to a nebulous file whose path is based on the workdir. For example

    neb://ipp040.0/gpc1/ThreePi.DM.20100401/o5238g0323o.133649/o5238g0323o.133649.ch.79656.cleanup.log

compress log file. Should we rename it or keep it in place?

Chip stage

find smf for associated camRun. If found and is not corrupted, delete cmfs otherwise leave them on disk.

    if (goto_purged)
        delete psf
        should we delete the background model? 
            It is only useful if the corresponding warp is in state full.
    if (goto_scrubbed) 
        delete all of the files we delete in goto_purged as well

Warp stage

    if (goto_purged)
        delete cmf
        delete psf
        delete skycell.template

    if (goto_scrubbed)
        delete the files deleted in goto_purged as well as the normal cleanup files

Changes to other stages: TBD