Index: /trunk/doc/design/distrib.v0.txt =================================================================== --- /trunk/doc/design/distrib.v0.txt (revision 21522) +++ /trunk/doc/design/distrib.v0.txt (revision 21522) @@ -0,0 +1,160 @@ + +Releasing GPC1 images + +This note discusses the process of preparing GPC1 images for release and distributing the +data to the PS1 Science Consortium. + +There are a number of somewhat unique features of the PS1 project that must be considered + + 1. The requirement that satellite streaks be masked in the images + 2. The massive size of the data + 3. The fact that the data is stored inside secure facility at MHPCC + +The note is rather free form. Lines marked with '*' indicate tasks that need to get +done to complete this project. + +Magic Masking +------------- + +All images to be released must be run through the magic destreaking process. In this process +difference images are examined for potential satellite streaks and pixels along the line +of the streaks and are removed from the images. For the raw images the excised pixels are +saved so that the complete original images may be recovered if necessary. + +Additionally those pixels that were not included in the science and magic analyses +must be masked because they may contain undeteced satellite streaks. +This masking is not performed as part of the magic process so that they may be included in +subsequent analysis inside the IPP system. This masking is applied to images as part of the +release process. + + +De-streaking +-------------------- + +A pantasks task is needed that runs periodicaly to queue destreaking runs with the command +magicdstool -definebyquery (this mode does not exist yet and thus needs to be coded) + +* Create the pantasks 'destreak' task +* Create the magicdstool -definebyquery mode + +A pantasks task performs the destreaking operations replacing the images in the nebulous database +with the destreaked versions. The original images are deleted and a flag is set in the +images' row in the corresponding database table that indicates that magic has been performed. + +Once the destreaking runs are complete the images are ready for release processing. + +Releasing images +---------------- + +The release processing consists of the following steps + + 1. identify those data that are to be prepared for release + 2. the 'excise the masked pixel' step for science images + 3. build DataStore file sets containing the images and any other data products (.cmf files, psf etc) + +This process is very similar to the processing done by the Postage Stamp server and so it makes sense +to build upon that request processing system. + +* We need to be able to specify criteria for what science images are to be released. + +For example, do we only release raw and processed detrend images? +Do we release chip processed or warp images from certain surveys? (The Pan planets team has +shown an interest in analyzing the chip level images). + +* We need a task that applies the criteria and creates 'get_image' type postage stamp request. +pstamptool -definebyquery ? +(* this mode does not currently exist. Postage stamp jobs are specified +by a request file which is externally prepared and then referenced by the db. The request file is +parsed into a set of zero or more jobs. It probably makes sense to have the capability to skip the +request file step and create the jobs directly.) + +* We need to define how we organize the data into Data Store products and file sets. + +We have a whole lot of data. The DataStore only has 3 levels of hierarchy + root + product + filesetA1 + file + file + filesetA2 + file + file + productB + filesetB2 + etc. + +We probably want to have filesets correspond to exposures. To keep the products from getting too +long, we may want to divide up the products by date. + +Perhaps something like: +product 20081106_4776 +fileset o4776g0144o_raw +file o4776g0144o.ota01.fits + + +Since these data will quickly get large, (we don't have storage space budgeted for a complete second +copy of the data): + +* We need to be able to purge release file sets based on some criteria. + +* For images that are being released in order to create a backup copy, it would make sense to get +notification from MPG that the images have been successfully transferred. Need a mechanism for that. + +* The data store needs to be able to recognize and report to users when requested file sets +are no longer available because they have been purged. + + +Presentation and data delivery +------------------------------ +The primary interface to the data is through the Data Store. The data store is implementated +as cgi scripts exectuted by an apache web server. + +Since we do not have space in Manoa to store the images here we need to leave them at MHPCC. +Since the network there is restricted we will need to provide a reverse proxy server in +Manoa which will forward the requests to the apache server(s) on Maui. + +* We need to restrict access to the data store by user. Perhaps by product? + +* We need to log bandwith used by these users had have the ability to control throttle usage to +insure that the primary users are serviced. + +XXXX The following isn't very thought out. There are two issues I'm trying to solve at once. + 1. Maximizing bandwidth for data transfer. + 2. Simplifying the client interface. + +The standard data store use pattern is for users to query the product list for new file sets +and then copy the files with HTTP GET requests. This method is used by the IPP for the summit +copy process. This method has the drawback that the client must handle any exceptional situations +and manage errors and retries when necessary. + +* It might be more efficient to supply an rsync server interface to the data repository. This program +already implements error correction features, detection of differnces between the source and +destination, etc. + +We would need to provide a reverse proxy server in manoa to handle forward the rsync +procotol requests to maui. + +On the other hand for efficiency we probably want to distribute the data store's files among +a set of nodes. (This would make the rsync solution problematic.) +This way we could use a set of apache servers and redirect the requests to the various nodes. +Perhaps we would need a set of reverse proxy servers in manoa as well. + + +"Mirror of data at MPG" +----------------------- + +The destreaked raw images will be copied to the MPG cluster in Garching to serve as a backup and +a source for analysis there. + +It will be very confusing to have different database ids for the images located there. +We need a way to copy the database tables for the images so that they can be preserved. + +It has been suggested that what might be done is to copy the database and +set all of the processed image entries to 'cleaned'. Then they could be regenerated using 'update'. + +* We would need to provide the metadata config dump of the configuration that was used to process +the images. + +* Will it be a problem that the reprocessing is done with the streak and masked pixels removed? + +* Perhaps we could provide the database dump and the config files as part of the raw image transfer Index: /trunk/doc/design/remote.copy.txt =================================================================== --- /trunk/doc/design/remote.copy.txt (revision 21522) +++ /trunk/doc/design/remote.copy.txt (revision 21522) @@ -0,0 +1,123 @@ + ++ The data will be copied in batches which represent a single entry in + a single stage. This means, eg, a single chipRun, a single rawExp, or + a single warpRun, etc. + ++ We need hand-shaking between the source and target sites. + ++ Data Store: + + product: distribution (restricted access) + fileset: chipRun chip_id + files.... + ++ remote copy system includes these stages: + + 1) rc_getstages + + query the source datastore for stages to fetch + + this may use the last-seen entry to query for only the new + entries + + insert the new stages into rcDownloadStage + + 2) rc_copystage + + for a single rcDownloadStage entry, download the complete fileset + + * we can do this with the wget command to ensure the full set is + downloaded, and allow for retries + + * the result files should be placed in a local temporary + location. + + insert the downloaded data into the local database: + + * check the database version in the manifest + - if it is not equal to the current version, raise an error + + * create nebulous entries for each of the files + - for each file in the manifest, create the single nebulous + entry and copy the file to the corresponding instance. + - allow the remote site to replicate entries at this time + + * execute the SQL db update code + + * update the rcDownloadStage entry + + * post an entry on our datastore to acknowledge success or + failure + + * on success insert the stage info into our upload tables. + +---------- + +rc_manifest: + +STAGE: +ID: +DB_VERSION : version number of source database +DB_DATA : filename.sql +NUMBER OF FILES : NNN +# neb name ==> bundle name +neb://path/name/o4541g0012o.ota11.fits ==> o4541g0012o.ota11.fits +neb://path/name/o4541g0012o.ota12.fits ==> o4541g0012o.ota12.fits +neb://path/name/o4541g0012o.ota13.fits ==> o4541g0012o.ota13.fits + +----------- + +database tables: + +rcDownloadStage: + id + ds_entry + stage (needed here?) + fileset uri + state + fault + +rcDistributeStage + id + stage + fileset uri + destination + state + +------------ + +once the rcDistributeStage is built, we need a task to monitor for +successful download by the destination system. Here are the tasks +needed: + +1) build a book of remote destinations (static book loaded from + config?) + +2) for each remote destination, examine the remote datastore + (PRODUCT name needed for download results) + + * we need to track the last seen entry + ** we need this information for several systems: summit copy, + remote copy, distribute + + for each new entry, update rcDistributeStage with new state + (success / failure) + +------------ + +build the rcDistributeStage table entries: + +new entries to be distributed should be added after the data is (a) +identified to be distributed, and (b) a distribution bundle is +generated. + +* Does this replicate PStamp server tables / interfaces? + +Steps to do (run for each of the distributed stages) + +1) identify the entries in the stage to be distributed: + * entry meets the needed condition (exists, is 'full', is magicked + -- this last may be optional depending on the data source). + + * entry is not in the rcDistributeStage + +2) do work