Here are the instructions for Bill/Gene/anyone else to build the dvo, fix ipptopsp and run ipptopsps: There are far more instructions on ippToPsps, these are just the quick version with specifics for SA10:
IPPTOPSPS INSTRUCTIONS
There are a number of columns to be fixed, which I haven't had a chance to get to yet (and likely won't). Here is a table of remaining changes
to be checked/verified for ipptopsps. These live here:
~ipp/src/ippTopsps/jython/ stackbatch.py objectbatch.py detectionbatch.py
- these need to be checked in stack:
- StackDetection
- psfFlux/psfFluxErr (formerly instFlux/instFluxErr) - was unclear if this was just a name change or do we grab something else instead?
- peakFlux -> covert to adu/s
- sky/skyErr -> convert to adu/s/pixel
- sgSep -> needs a formula
- psfLikelihood -> fhe formula was modded, (see wiki)
- apFlux/apFluxErr -> these were once mags, need to be converted to flux, also needs exptime correction
- StackApFlx
- isophotMag/Err -> were mags, need to be converted to flux
- petFlux/Err -> were mags, need to be converted to flux
- StackModelFit
- devFlux/Err -> were mags, need to be converted to flux
- expFlux/Err -> were mags, need to be converted to flux
- serFlux/Err -> were mags, need to be converted to flux
- !stackDetectionCalib
- need ra/dec/raErr/decErr to be populated
- need zp/zpErr to be populated
- need expTime to be populated
- need airMass to be populated
- StackDetection
- these need to be checked in detection:
- Detection
- psfFlux/psfFluxErr (formerly instFlux/instFluxErr) - was unclear if this was just a name change or do we grab something else instead?
- peakFlux -> covert to adu/s
- sky/skyErr -> convert to adu/s/pixel
- sgSep -> needs a formula
- psfLikelihood -> fhe formula was modded, (see wiki)
- apFlux/apFluxErr -> these were once mags, need to be converted to flux
- DetectionCalib
- need ra/dec/raErr/decErr to be populated
- need zp/zpErr to be populated
- need expTime to be populated
- need airMass to be populated
- Object:
- [filter]stackDetectID needs to be populated - I think the only other change was a flux to a mag and I did that already
- Detection
I advise to make some tests of these to make sure that the batches contain what you want and such. There is already a config for this, SAS.2120905.sa10.test1 (yes, typo in the date). The config is located in the ipptopsps database, (ippdb01, same user/pass as gpc1), and you can see it here:
mysql> select * from config where name = 'SAS.2120905.sa10.test1'; +---------------------+------------------------+-------------------+----------------+-------------------+-------------- +------------------------------------------+--------+--------+---------+---------+----------+--------------------------- +--------------+--------------+------------------+----------------+---------------------+--------+-------------+---------- +----------+----------+--------+ | timestamp | name | datastore_product | datastore_type | datastore_publish | dvo_label | dvo_location | min_ra | max_ra | min_dec | max_dec | box_size | base_path | data_release | delete_local | delete_datastore | delete_dxlayer | epoch | survey | psps_survey | queue_P2 | queue_ST | queue_OB | active | +---------------------+------------------------+-------------------+----------------+-------------------+-------------- +------------------------------------------+--------+--------+---------+---------+----------+--------------------------- +--------------+--------------+------------------+----------------+---------------------+--------+-------------+---------- +----------+----------+--------+ | 2012-10-26 11:51:12 | SAS.2120905.sa10.test1 | PSPS_test | IPP_PSPS | 1 | SAS.20120905 | /data/ipp005.0/gpc1/catdirs/SAS/20120901 | 327 | 341 | -7 | 7 | 2 | /data/ipp005.0/ipptopsps/ | 10 | 1 | 1 | 1 | 2012-10-26 11:53:00 | 3PI | SA10 | 0 | 0 | 1 | 1 | +---------------------+------------------------+-------------------+----------------+-------------------+-------------- +------------------------------------------+--------+--------+---------+---------+----------+--------------------------- +--------------+--------------+------------------+----------------+---------------------+--------+-------------+---------- +----------+----------+--------+ 1 row in set (0.00 sec)
testing ipptopsps changes
you can turn off/on queue_P2/queue_ST/queue_OB as needed, and to test do the following as ipp in ~/src/ippToPsps/jython on ipp005 or ipp006
psconfig ipptrunk-20120913 ./run.sh queue.py SAS.2120905.SA10.test1 ./run.sh loader.py SAS.2120905.SA10.test1
The first command changes the psconfig to grab the proper libraries for dvograbber
The second command makes a list of what needs to be made into batches (so if you turn off/on ST/P2/OB it will only pick up the ones that are on)
The third command starts loading
The batches get placed on http://datastore.ipp.ifa.hawaii.edu/PSPS_test/, you can pick them up from there and play with them.
do psconfig ipptrunk-20120913 prior to any of the ./run.py commands - this picks up the correct libraries for dvo.
recompiling for the new SAS
there will be a dvo schema change - there are 2 ways to work with this:
- Step 1:
- update the tag and recompile the tag
- check out a copy of the truck and access it by something like ipptrunk-2012date
- Step 2: once you are linked to the proper dvo library, recompile dvograbber (in ~/src/ippTopsps/src) I honestly don't remember how I did this, wheter from ~src/ippToPsps or from the directory that contains dvograbber, and what if any things I did to make sure the make files picked up the correct libraries. I did stuff multiple times... as ipp on ipp005 you can test it like this: ../src/dvograbber ../config/settings.xml ipptopsps_scratch3 /data/ipp005.0/gpc1/catdirs/SAS/20120901 to see if it works (change to new SAS directory)
- Step 3: once you verify dvograbber works it should be ok... I hope...
queueing the new SAS
MAKE sure you have used the proper psconfig for this otherwise it will not work (dvograbber will fail!!)
- do psconfig ipptrunk-20120913 for testing
- do psconfig ipptrunk-(whatever) for the actual loading for PSPS (see above section)
Step 1: make new config
./run.sh queue.py edit
answer questions, basically give it the same answers as for SAS.2120905.sa10.test1 with a different name/ dvo_label / dvo_location, and put it on datastore_product = PSPS_JHU, and tell it to queue everything (P2/ST/OB). for epoch give it whatever the current time is - I don't understand why but that's the rules from Roy.
if you make errors correct it by going into mysql and editing that entry in config table
Step 2: make an init batch (only do this once)
./run.sh loader.py (configname) init
this should just work - i've not had problems in the past.
Step 3: start loading.
you can do this step multiple times to load on multiple machines. There are certain machines that can do loading, but I recommend doing it on ipp005 and ipp006 (up to 3 on ipp005, and I think 2-3 times on ipp006)
./run.sh loader.py (configname)
trouble shooting
reverting errors
First is how do you tell? There is funny logic here. Preferably you will know how many OB/ST/P2 batches there are. Do queries in addRun to find out how many cam/stacks there are (cam should be 620, stacks are a little variable due to those few that don't staticsky) objects = number of cpt files in SAS dvodb. Do a query in the ipptopsps database to get counts of stuff (example from SAS.v9):
mysql> select batch_type, count(*) from batch where processed = 1 and loaded_to_datastore =1 and dvo_db = 'SAS.20120905' and datastore_product = 'PSPS_JHU' group by batch_type; +------------+----------+ | batch_type | count(*) | +------------+----------+ | IN | 1 | | OB | 368 | | P2 | 620 | | ST | 2967 | +------------+----------+ 4 rows in set (0.12 sec)
those counts should add up to what you expect (it does for sas_v9). Note that you can't just count up the ones that have not processed, because once a batch fails it is forgotten about (we don't recycle batch numbers, we just make a new batch with the same contents):
mysql> select batch_type, count(*) from batch where dvo_db = 'SAS.20120905' and datastore_product = 'PSPS_JHU' group by batch_type, processed;+------------+----------+ | batch_type | count(*) | +------------+----------+ | IN | 1 | | OB | 123 | | OB | 368 | | P2 | 620 | | ST | 2967 | +------------+----------+ 5 rows in set (0.13 sec)
Those 123 OBs failed during some craziness the last time (always due to nfs/system/neb/mysql hiccups) and were requeued... Always only trust the final count of what's been processed and loaded_to_datastore...
To try again, requeue and reload:
./run.sh queue.py SAS.(name) ./run.sh loader.py SAS.(name)
it might take a couple of iterations, depending on the nature of the error. check database to verify it has all.
logs
These live in /data/ipp005.0/ipptopsps/logs
databases
- the main ipptopsps database has information about the config and the batches.
- each loader has a database as well on teh machines they are running on: mysql -h localhost -u ipp ipptopsps_scratch (or ipptopsps_scratch2 or ipptopsps_scratch3) useful to diagnose stuff when working on ipptopsps
IPPDVO recompile of ipp trunk instructions
ippdvo runs off of the trunk. To differentiate I call them by the date that I cecked out the trunk... So once the changes are in and it is ready to do, make adirectory called src/ipptrunk-201210day check out and compile the trunk, and edit .tcshrc to use that psconfig instead. I keep the old ones around just in case...
ADDSTAR INSTRUCTIONS
This is to be run on addstarlap (ipp005) since that is where the other SASs live
I was pondering about just reusing the old cam stage from previously, but I don't necessarily trust that since I think Gene changed some stuff in the dvo, also, it's substantially easier to explain without queueing older minidvodbs, and also, building 620 cam stages isn't going to take too long.
I generally build the cam stage first and then the staticsky stage. For simplicity you can do both simultaneously (I don't think it matters):
Bill edits: We are using the skycal stage not staticsky. I have updated the following to reflect that
to queue the minidvodb stuff:
survey.add.addstar SAS.cam SAS_v8 SAS.20121029 SAS.20121029 cam survey.add.addstar SAS.skycal SAS_v10 SAS.20121029 SAS.20121029 skycal add.minidvodb SAS.20121029 SAS.20121029.MINI SAS.20121029 1 GPC1 add.label SAS.20121029
once you are satisfied that skycal addstars are working properly and doing the right stuff, you can turn on creation/merge of the merge dvodb:
add.mergedvodb SAS.20121029 survey.add.mergedvodb SAS.20121029 SAS.20121029 SAS.20121029
Note that once the skycals have completed, there will still be that last minidvodb that is in 'active', since it kicks to be merged after a day. If all the skycals have been addstarred, you can set the state (with mysql) of the current active minidvodb for this to the state 'to_be_merged' which will cause it to go get merged (wait for it...)
Once the merge dvodb is built, it needs to be 'spiffed':
These are the commands to use. (Edited to use the corrected tree file and to use the v10 catdir name
relphot g,r,i,z,y -nloop 0 -update -boundary-tree ~ipp/RINGS.V3.tree.v1.fits -D CAMERA gpc1 -D CATDIR /data/ipp005.0/gpc1/catdirs/SAS/20121029 relastro -update-objects -reset -update -D CAMERA gpc1 -D CATDIR /data/ipp005.0/gpc1/catdirs/SAS/20121029
Please verify with Gene that these are the commands.
