This page contains some notes on the IPP cleanup and update mechanism focusing on some changes proposed in November 2012.

While the IPP production cluster has a vast amount of storage, there is not nearly enough to save all the processed data products. A process has been established to delete large data files retaining enough information that they can be regenerated at a later date.

Cleanup states

In the cleanup process runs are transitioned to one of 3 states

  • cleanup Delete ephemeral files only. Leave files behind so that images can be updated.
  • purged More (but perhaps not all) output files are deleted. For example logs survive. Since recipe information is lost images cannot be regenerated from this state.
  • scrubbed Like cleaned but cannot be updated because mdc file is missing. (Actually they can be since update detects missing config file and uses current recipes.

CZW: The idea behind the "scrubbed" state was to clearly note runs that could not be updated, as they had no config file. This distinguishes them from the "cleaned" runs. This currently retains catalogs, making it similar to an intermediate of "cleaned" and "purged" (it cannot be regenerated, but it is not completely eliminated).

Update operation

Recipes and other configuration is read from config dump file and those parameters are used to regenerate the images. The goal is to be as close to the original processing as possible, but since the code is in general different it might be just "close"

if (mdc file is not available)
    # this should only apply to runs prior to replication of config dump files
    if (mdc file is temporarily not available)
        images are regenerated using current recipes but dump file is not saved
        images are regenerated using current recipes but new config dump file is saved

Proposed Change to chip updates

Currently if cmf or psf is missing we rerun photometry. Change to only do this if the psf file is missing. Do not recreate cmf file.

Cleanup Changes


redirect output to a nebulous file whose path is based on the workdir. For example


When cleanup is successfully completed gzip the log file. Sample of one 91% reduction in size 517698 compressed to 46548. Perhaps make the script a bit less verbose in success cases.

Chip stage

  • find smf for associated camRun. If found and is not corrupted, delete cmfs otherwise leave them on disk.
  • delete b1 fits ?
  • delete trace files ? (they are empty)
  • gzip log files

Once implemented we will start one big pass through the cleaned up data to implement these space reductions. This will consist of

  • Modify query for pendingcleanupimfile to (optionally) list all components not just those in full state
  • set cleaned runs back to state goto_cleaned with a different label so we can distinguish between regular and second pass cleanup

Warp stage

  • modify query to optionally list all components not just those in full state
  • delete sources (they are regenerated by update anyways)
  • delete skycell template files (they are regnerated by update)
  • delete psf file (it is regenerated on by update)
  • gzip log files

Note: deleting the sources file is a big win. We could have been doing it all along as warp update always regenerates them

Camera Stage

We found about 15000 censored camera runs that were not restored. I missed some labels apparently. These are running now. This will remove 2 copies of the smf for each run.

Changes to other stages