Here are the instructions for Bill/Gene/anyone else to build the dvo, fix ipptopsp and run ipptopsps: There are far more instructions on [wiki:ippToPsps], these are just the quick version with specifics for SA10: == IPPTOPSPS INSTRUCTIONS == There are a number of columns to be fixed, which I haven't had a chance to get to yet (and likely won't). Here is a table of remaining changes to be checked/verified for ipptopsps. These live here: {{{ ~ipp/src/ippTopsps/jython/ stackbatch.py objectbatch.py detectionbatch.py }}} * these need to be checked in stack: * !StackDetection * psfFlux/psfFluxErr (formerly instFlux/instFluxErr) - was unclear if this was just a name change or do we grab something else instead? * peakFlux -> covert to adu/s * sky/skyErr -> convert to adu/s/pixel * sgSep -> needs a formula * psfLikelihood -> fhe formula was modded, (see wiki) * apFlux/apFluxErr -> these were once mags, need to be converted to flux, also needs exptime correction * !StackApFlx * isophotMag/Err -> were mags, need to be converted to flux * petFlux/Err -> were mags, need to be converted to flux * !StackModelFit * devFlux/Err -> were mags, need to be converted to flux * expFlux/Err -> were mags, need to be converted to flux * serFlux/Err -> were mags, need to be converted to flux * !stackDetectionCalib * need ra/dec/raErr/decErr to be populated * need zp/zpErr to be populated * need expTime to be populated * need airMass to be populated * these need to be checked in detection: * Detection * psfFlux/psfFluxErr (formerly instFlux/instFluxErr) - was unclear if this was just a name change or do we grab something else instead? * peakFlux -> covert to adu/s * sky/skyErr -> convert to adu/s/pixel * sgSep -> needs a formula * psfLikelihood -> fhe formula was modded, (see wiki) * apFlux/apFluxErr -> these were once mags, need to be converted to flux * !DetectionCalib * need ra/dec/raErr/decErr to be populated * need zp/zpErr to be populated * need expTime to be populated * need airMass to be populated * Object: * [filter]stackDetectID needs to be populated - I think the only other change was a flux to a mag and I did that already I advise to make some tests of these to make sure that the batches contain what you want and such. There is already a config for this, SAS.2120905.sa10.test1 (yes, typo in the date). The config is located in the ipptopsps database, (ippdb01, same user/pass as gpc1), and you can see it here: {{{ mysql> select * from config where name = 'SAS.2120905.sa10.test1'; +---------------------+------------------------+-------------------+----------------+-------------------+-------------- +------------------------------------------+--------+--------+---------+---------+----------+--------------------------- +--------------+--------------+------------------+----------------+---------------------+--------+-------------+---------- +----------+----------+--------+ | timestamp | name | datastore_product | datastore_type | datastore_publish | dvo_label | dvo_location | min_ra | max_ra | min_dec | max_dec | box_size | base_path | data_release | delete_local | delete_datastore | delete_dxlayer | epoch | survey | psps_survey | queue_P2 | queue_ST | queue_OB | active | +---------------------+------------------------+-------------------+----------------+-------------------+-------------- +------------------------------------------+--------+--------+---------+---------+----------+--------------------------- +--------------+--------------+------------------+----------------+---------------------+--------+-------------+---------- +----------+----------+--------+ | 2012-10-26 11:51:12 | SAS.2120905.sa10.test1 | PSPS_test | IPP_PSPS | 1 | SAS.20120905 | /data/ipp005.0/gpc1/catdirs/SAS/20120901 | 327 | 341 | -7 | 7 | 2 | /data/ipp005.0/ipptopsps/ | 10 | 1 | 1 | 1 | 2012-10-26 11:53:00 | 3PI | SA10 | 0 | 0 | 1 | 1 | +---------------------+------------------------+-------------------+----------------+-------------------+-------------- +------------------------------------------+--------+--------+---------+---------+----------+--------------------------- +--------------+--------------+------------------+----------------+---------------------+--------+-------------+---------- +----------+----------+--------+ 1 row in set (0.00 sec) }}} == testing ipptopsps changes == you can turn off/on queue_P2/queue_ST/queue_OB as needed, and to test do the following as ipp in ~/src/ippToPsps/jython on ipp005 or ipp006 {{{ psconfig ipptrunk-20120913 ./run.sh queue.py SAS.2120905.SA10.test1 ./run.sh loader.py SAS.2120905.SA10.test1 }}} The first command changes the psconfig to grab the proper libraries for dvograbber The second command makes a list of what needs to be made into batches (so if you turn off/on ST/P2/OB it will only pick up the ones that are on) The third command starts loading The batches get placed on http://datastore.ipp.ifa.hawaii.edu/PSPS_test/, you can pick them up from there and play with them. === do psconfig ipptrunk-20120913 prior to any of the ./run.py commands - this picks up the correct libraries for dvo. === == recompiling for the new SAS == there will be a dvo schema change - there are 2 ways to work with this: * Step 1: * update the tag and recompile the tag * check out a copy of the truck and access it by something like ipptrunk-2012date * Step 2: once you are linked to the proper dvo library, recompile dvograbber (in ~/src/ippTopsps/src) I honestly don't remember how I did this, wheter from ~src/ippToPsps or from the directory that contains dvograbber, and what if any things I did to make sure the make files picked up the correct libraries. I did stuff multiple times... as ipp on ipp005 you can test it like this: ../src/dvograbber ../config/settings.xml ipptopsps_scratch3 /data/ipp005.0/gpc1/catdirs/SAS/20120901 to see if it works (change to new SAS directory) * Step 3: once you verify dvograbber works it should be ok... I hope... == queueing the new SAS == === MAKE sure you have used the proper psconfig for this otherwise it will not work (dvograbber will fail!!) === * do psconfig ipptrunk-20120913 for testing * do psconfig ipptrunk-(whatever) for the actual loading for PSPS (see above section) === Step 1: make new config === {{{ ./run.sh queue.py edit }}} answer questions, basically give it the same answers as for SAS.2120905.sa10.test1 with a different name/ dvo_label / dvo_location, and put it on datastore_product = PSPS_JHU, and tell it to queue everything (P2/ST/OB). for epoch give it whatever the current time is - I don't understand why but that's the rules from Roy. if you make errors correct it by going into mysql and editing that entry in config table === Step 2: make an init batch (only do this once) === {{{ ./run.sh loader.py (configname) init }}} this should just work - i've not had problems in the past. === Step 3: start loading. === you can do this step multiple times to load on multiple machines. There are certain machines that can do loading, but I recommend doing it on ipp005 and ipp006 (up to 3 on ipp005, and I think 2-3 times on ipp006) {{{ ./run.sh loader.py (configname) }}} == trouble shooting == === reverting errors === First is how do you tell? There is funny logic here. Preferably you will know how many OB/ST/P2 batches there are. Do queries in addRun to find out how many cam/stacks there are (cam should be 620, stacks are a little variable due to those few that don't staticsky) objects = number of cpt files in SAS dvodb. Do a query in the ipptopsps database to get counts of stuff (example from SAS.v9): {{{ mysql> select batch_type, count(*) from batch where processed = 1 and loaded_to_datastore =1 and dvo_db = 'SAS.20120905' and datastore_product = 'PSPS_JHU' group by batch_type; +------------+----------+ | batch_type | count(*) | +------------+----------+ | IN | 1 | | OB | 368 | | P2 | 620 | | ST | 2967 | +------------+----------+ 4 rows in set (0.12 sec) }}} those counts should add up to what you expect (it does for sas_v9). Note that you can't just count up the ones that have not processed, because once a batch fails it is forgotten about (we don't recycle batch numbers, we just make a new batch with the same contents): {{{ mysql> select batch_type, count(*) from batch where dvo_db = 'SAS.20120905' and datastore_product = 'PSPS_JHU' group by batch_type, processed;+------------+----------+ | batch_type | count(*) | +------------+----------+ | IN | 1 | | OB | 123 | | OB | 368 | | P2 | 620 | | ST | 2967 | +------------+----------+ 5 rows in set (0.13 sec) }}} Those 123 OBs failed during some craziness the last time (always due to nfs/system/neb/mysql hiccups) and were requeued... Always only trust the final count of what's been processed and loaded_to_datastore... To try again, requeue and reload: {{{ ./run.sh queue.py SAS.(name) ./run.sh loader.py SAS.(name) }}} it might take a couple of iterations, depending on the nature of the error. check database to verify it has all. === logs === These live in /data/ipp005.0/ipptopsps/logs === databases === * the main ipptopsps database has information about the config and the batches. * each loader has a database as well on teh machines they are running on: mysql -h localhost -u ipp ipptopsps_scratch (or ipptopsps_scratch2 or ipptopsps_scratch3) useful to diagnose stuff when working on ipptopsps == IPPDVO recompile of ipp trunk instructions == ippdvo runs off of the trunk. To differentiate I call them by the date that I cecked out the trunk... So once the changes are in and it is ready to do, make adirectory called src/ipptrunk-201210day check out and compile the trunk, and edit .tcshrc to use that psconfig instead. I keep the old ones around just in case... == ADDSTAR INSTRUCTIONS == This is to be run on addstarlap (ipp005) since that is where the other SASs live I was pondering about just reusing the old cam stage from previously, but I don't necessarily trust that since I think Gene changed some stuff in the dvo, also, it's substantially easier to explain without queueing older minidvodbs, and also, building 620 cam stages isn't going to take too long. I generally build the cam stage first and then the staticsky stage. For simplicity you can do both simultaneously (I don't think it matters): Bill edits: We are using the skycal stage not staticsky. I have updated the following to reflect that to queue the minidvodb stuff: {{{ survey.add.addstar SAS.cam SAS_v8 SAS.20121029 SAS.20121029 cam survey.add.addstar SAS.skycal SAS_v10 SAS.20121029 SAS.20121029 skycal add.minidvodb SAS.20121029 SAS.20121029.MINI SAS.20121029 1 GPC1 add.label SAS.20121029 }}} once you are satisfied that skycal addstars are working properly and doing the right stuff, you can turn on creation/merge of the merge dvodb: {{{ add.mergedvodb SAS.20121029 survey.add.mergedvodb SAS.20121029 SAS.20121029 SAS.20121029 }}} Note that once the skycals have completed, there will still be that last minidvodb that is in 'active', since it kicks to be merged after a day. If all the skycals have been addstarred, you can set the state (with mysql) of the current active minidvodb for this to the state 'to_be_merged' which will cause it to go get merged (wait for it...) Once the merge dvodb is built, it needs to be 'spiffed': These are the commands to use. (Edited to use the corrected tree file and to use the v10 catdir name {{{ relphot g,r,i,z,y -nloop 0 -update -boundary-tree ~ipp/RINGS.V3.tree.v1.fits -D CAMERA gpc1 -D CATDIR /data/ipp005.0/gpc1/catdirs/SAS/20121029 relastro -update-objects -reset -update -D CAMERA gpc1 -D CATDIR /data/ipp005.0/gpc1/catdirs/SAS/20121029 }}} Please verify with Gene that these are the commands.