Changes between Version 180 and Version 181 of ippToPsps
- Timestamp:
- Mar 7, 2012, 11:54:44 AM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPsps
v180 v181 99 99 === Setting up a loading machine === 100 100 101 Although this can be changed in the config file, {{{ippToPsps}}} uses a 'scratch' MySQL database on {{{localhost}}} while processing. So, to to any loading, a local MySQL server should be up-and-running, with a database created as the {{{ipp}}} user for use by {{{ippToPsps}}}. The default name for this is {{{ipptopsps_scratch}}}.101 Although this can be changed in the config, {{{ippToPsps}}} uses a 'scratch' MySQL database on {{{localhost}}} while processing. So, to to any loading, a local MySQL server should be up-and-running, with a database created as the {{{ipp}}} user for use by {{{ippToPsps}}}. The default name for this is {{{ipptopsps_scratch}}}. 102 102 103 103 Because {{{ippToPsps}}} uses this local database as a scratch database, there are large number of inserts and deletions, meaning large log files. For this reason, it is important to configure MySQL to store its data on a local partition, and not {{{/var}}} as is the default. For example, on {{{ipp005}}}: … … 162 162 {{{ 163 163 cd trunk/ippToPsps/jython 164 ./run.sh load .py someConfig.xml164 ./run.sh loader.py someConfig 165 165 }}} 166 166 … … 172 172 173 173 {{{ 174 ./run.sh load.py someConfig.xmlinit174 ./run.sh load.py init 175 175 }}} 176 176 … … 179 179 === Queuing === 180 180 181 Before any loading clients can do anything, you must queue something up. To do this run: 182 183 {{{ 184 ./run.sh queue.py someConfig 185 }}} 186 181 187 When run, {{{load.py}}} first queues up all the exposures or stacks that it needs to process, then works through them sequentially, doing detections first then stacks. The master list comes from the gpc1 {{{addRun}}} table, which lists all items currently merged into the DVO database we are using. From this list, {{{ippToPsps}}} subtracts items that have already been successfully loaded to the datastore already, and queues up the remainder. 182 188 183 189 Because it is possible, and usual, to run multiple versions of {{{load.py}}} in order to speed up loading time, the methods to queue and begin processing a new batch form a critical section. This simply means that the 'batch' table in the {{{ippToPsps}}} database is locked by a client looking for a new item to process, then released afterwards. 184 190 185 An important factor in the queuing of items is the current ''epoch'', which is set in the config file.191 An important factor in the queuing of items is the current ''epoch'', which is set in the config. 186 192 187 193 === Epochs === 188 194 189 Each config file requires an entry in the {{{options/epoch}}} field. An ''epoch'' in the context of {{{ippToPsps}}} is the date that we count as ''the beginning''. If we loaded IPP data to PSPS once and only once, this would not be necessary: we would queue up all available exposures and stacks and simply load them. However, the early stages of the project have required multiple re-loads of data while the IPP perfects the science, necessitating that the same exposures and stacks are loaded more than once. Because {{{load.py}}} queues available items by taking into account those that have already been published to PSPS, we have to give it an epoch date from which to accurately determine those exposure that have been loaded or not. Generally speaking, the epoch is reset every time we delete the main PSPS database.195 Each config requires an entry in the {{{epoch}}} column. An ''epoch'' in the context of {{{ippToPsps}}} is the date that we count as ''the beginning''. If we loaded IPP data to PSPS once and only once, this would not be necessary: we would queue up all available exposures and stacks and simply load them. However, the early stages of the project have required multiple re-loads of data while the IPP perfects the science, necessitating that the same exposures and stacks are loaded more than once. Because {{{queue.py}}} queues available items by taking into account those that have already been published to PSPS, we have to give it an epoch date from which to accurately determine those exposure that have been loaded or not. Generally speaking, the epoch is reset every time we delete the main PSPS database. 190 196 191 197 === Test mode === … … 200 206 == Running {{{dvoToMySQL}}} == 201 207 202 Like all other {{{ippToPsps}}} components, {{{dvoToMySQL}}} takes one argument which is a path to a config file. For example:208 Like all other {{{ippToPsps}}} components, {{{dvoToMySQL}}} takes one argument which is a name of a config in the database. For example: 203 209 204 210 {{{ … … 217 223 * when a batch has been successfully merged into the PSPS database, then final copy on local disk may be deleted 218 224 219 The logic for this is that errors may occur during the merge phase and it is useful to have local copies of offending batches for debugging purposes. This standard behavior can be changed by setting the appropriate values in the ''deletion'' section of the config file.225 The logic for this is that errors may occur during the merge phase and it is useful to have local copies of offending batches for debugging purposes. This standard behavior can be changed by setting the appropriate values in the ''deletion'' section of the config. 220 226 221 227 To actually perform the clean, run the {{{cleanup.py}}} program like this:
