| | 7 | |
| | 8 | I worked on two side projects this past week related to data storage. First, I added compression I/O support to the IPP metadata I/O functions. This will allow us to apply gzip all of the MDC configuration files which are saved at each stage. Uncompressed, those files now take up some 30TB of space on our cluster; compressed they should be about 2TB, so this is an interesting savings. |
| | 9 | |
| | 10 | The other project was motivated by the realisation that our data replication has some equivalence to RAID 1, but it could be a RAID 5. We currently duplicate all raw exposures across the cluster to avoid loss of data if we lose a single storage node. Instead, we could identify groups of N files that are distributed across N unique hosts, and save a parity image of the N images on another unique host. This parity image would be the same size as the largest of the N input images. Like RAID 5, we could reproduce any single one of the N input images with the parity image and the remaining N-1 images. Thus we would be protected against the loss of a single node, just as we are currently protected. This data protection strategy would only require (1 + 1/N) times the input data, instead of 2 times the input data, as in the current strategy. There are some downsides to this strategy: recovery is somewhat slower (or blocks the rest of processing more) since it requires all N-1 remaining files to be read; the management is a bit more complex, and automation of the management within Nebulous would take some coding work. However, for some of the less in-demand data, we could reduce our storage footprint using manual scripts without too much effort. |
| | 11 | |
| | 12 | I created a simple program to generate these parity images and to repair an image. I intend for us to move the older detrend images, the in-focus focus images, and the stare data to this backup strategy soon, which would save us some 60 - 80TB. |
| | 13 | |
| | 14 | I also started work on the photometric and astrometric reference catalog: I made a subset database of the ThreePi dvo database with the high-quality sources, and started to run some relphot and relastro tests. I also started work on the STS-desired forced photometry code in the difference image analysis. |
