| | 8 | We are finally pushing data out to the consortium again. We distributed MD09 data, and should be able to distribute M31 images by Monday. The burntool processing has been running slowly, but Heather & I made some modifications that allows the burntool to be run without so much hand-holding -- I had failed to increment the burntool version field, so Heather needed to watch carefully for processing errors. Heather & Chris have also been working on issues related to cluster data management. After Heather found a raw image file which had been corrupted (either in copy or in situ on disk), she has been running a script to check the md5sum for the raw images. In the cases so far, the corrupted images had a second uncorrupted image elsewhere in the cluster (due to the Nebulous duplications). Most of the examples we've seen appear to have been caused by a period where two summit copy processes were running at the same time, so hopefully there are not any unrecoverably damaged files. Bill S updated the schema and summit copy process to allow us to track compressed md5sum and file size (previously we were only tracking this for the uncompressed values reported by the summit). Chris has been making good progress on running the cleanup analysis. This is getting critical since many of our nodes are at 95% full, and our total usage is roughly 70% of capacity. There are less than 100k raw exposures, which should account for about 270TB, but we are using 560TB of our storage. The remainder is taken by the many versions of data analysis tests and distributions, so cleaning those old analysis runs will get us back most of our storage space. |
