Manifest Missing Files

Before transferring the STSCI nodes, we did two kinds of shuffle operations.

• P1 shuffle to clear IPP files off the STSCI nodes. This was a slow process due to the number of database interactions necessary to ensure no file database key and disk location was misassociated.
• P2 shuffle to check that all 3-pi PV3 files that we were shipping were stored on the STSCI nodes.

The note that not all files included in the manifests were found is concerning, but not overly so. The manifest files were created during the P2 shuffle process. A command ran that identified all files for a given stage that needed to be transferred, did that transfer, and wrote the disk location into the manifest for each file. Based on the command lists, there should be

Stage	Number of manifest files	Note
warp	374519
stack	998886
diff	391241	I believe the shuffle was started before culling all duplicates from the diff stage.

When the P2 shuffle command detected that something was wrong with the shuffle, it deleted the manifest it had created to prevent confusion. However, the fact that there are files missing indicates this failure detection was not perfect. I believe the issue has something to do with files being cached incorrectly, as the majority of the missing files are small warp-stage mdc files (~120k each).

Size of problem

From Bernie Shiao's email of 2015-11-04, there are 17359607 files missing out of the 182590119 files listed in the manifests. I (CZW) am currently running a scan of his list of missing files, and checking for instances on the IPP cluster. For files that we have an instance of (such as the majority-by-count 12147688 missing mdc files), this scan is creating a mimic directory tree that can be rsynced to download all those files into the manifest-correct location.

Unfortunately, we recently ran a series of cleanup operations on the PV3 warp label, in order to make the IPP disks and IPP gpc1 database consistent. This is needed to allow updates to run on those warps, as otherwise, the system thinks that the warp images are available on a temporarily inaccessible disk. This cleanup deleted the IPP copies of the warp images and cmfs that are listed in the missing file list. The scan is identifying the warp_id and skycell_id combinations that need to be updated to regenerate the missing files. Of the 1303837 files scanned, only 635633 update operations are required, indicating that on average, about two files are missing from each warp_id/skycell_id file set. Assuming this is approximately half of the data (unclear), and using the 70-skycells-per-warp average, this translates to about 20k warp update runs, which is less than a month of processing time. In terms of disk space/network transfer, this is about 51 TB. A "worst case scenario" in which each of the 2029138 fits images identified as missing all need unique updates gives closer to 30k warp updates, and 85TB of disk space.

The procedure for fixing this issue is likely to be something like the following:

1. The STSCI scan of manifest files completes. If this is already done, then the current missing list is complete.
2. Run the IPP side scan to determine which files are already on disk. This is running, and is expected to complete over the weekend.
3. From that list, run the update process either in one large chunk (if time is critical), or as a series of batches to spread the IPP disk usage and network transfers out (if IPP disk usage is more critical than usual).
4. Iterate steps 1-3 until 3-pi PV3 file convergence is reached.

Extra files

In addition to the 3-pi PV3 files listed in the manifest, the STSCI disks have a large number of "extra" files. The P1 shuffle mentioned above was running slowly, and with a hard shipping deadline, it was decided to stop that process and switch to an rsync strategy. With this strategy, a list of file exceptions was created to prevent PV3 data from being copied back, and then rsync was run with those exceptions to transfer a copy of all the files to staging disks elsewhere in the IPP cluster. After shipping, these copies were reinserted into our nebulous database at their new locations, allowing us to shift the database operations. This was largely successful, but left the original STSCI copy in place. These files can be deleted, but I would recommend waiting to do so until some point in the future.

Although these files should have an IPP copy, some files were matched by the exceptions that were not intended. Mark Huber has files that were unavailable after the disk shipment, and it may be more efficient to transfer those files back from STSCI instead of regenerating them here.

In addition, it would be best to double check that any raw OTA data that is still on the STSCI disks have sufficient IPP copies before deleting those. We did run a verification of this prior to shipment, but it's best to be careful with the raw data. If the STSCI scan is already indexing files that are not listed in the manifest files, those can be checked fairly quickly.

November 10 update

The initial scan indicates that there are 2593402 missing files that need update, in 699669 warp/skycell pairs. This indicates on average all four major components (image, mask, variance, catalog) were missing in a particular warp/skycell pair. This means that fewer updates are necessary than the previous estimate, with 10k warp update runs (using the 70 skycell-per-warp average). The disk usage for this will be more managable, around 28 TB.