Notes on ingest of PV3 ipptopsps go here
Current version of ipptopsps
using ippdvo2 account:
ippdvo2@ippc19:/home/panstarrs/ippdvo2>psconfig /home/panstarrs/ippdvo2/psconfig : ipptrunk.20160518 : lin64
Current status of ipptopsps
- all of these are located in datastores like 'PSPS_PV3_ST_SLICE_X'
- IN - needs to be generated
- OB - the boundaries need to be regenerated once Gene investigates
- ST - edges are avoided; found 1 hole (non-boundary) so far with dvo duplicates, and 2 batches with invalid cmf info.
- P2 - Wait for Gene
- FW - Wait for Gene
- FG/FO - Need to verify ffg minis before handoff to Gene, then wait for Gene.
- DF/DO - addstar in progress. large number of faults due to missing files on space telescope machines
Fastest way to set up mysql on 'new' machines
as root on the machine in question
cp /etc/mysql/my.cnf /etc/mysql/my.cnf.orig cp ~heather/mysqlconfig/my.cnf /etc/mysql/my.cnf /usr/bin/mysql_install_db /etc/init.d/mysql restart
wait a bit, it can take a bit of time to start, and the status it displays when doing mysql restart is generally useless. check if it is running with mysql -h localhost -u root
now, as ippdvo2 on that machine (not sure why, but it doesn't take as root), the following will add the necessary accounts, initialize and install the scratch database.
mysql -h localhost -u root CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo'; GRANT ALL on *.* to 'dvo'@'%'; CREATE USER 'dvo'@'localhost' IDENTIFIED BY 'dvo'; GRANT ALL on *.* to 'dvo'@'localhost'; create database ipptopsps_scratch; flush privileges; exit ippjython setupScratchDb.py
current ipptopsps machines
- ipp006 is the database machine, name of database is ipptopsps2
- ippx001 - ippx048 are currently set up for ipptopsps (correct schema, mysql, etc)
current checker scripts
I am running ipptopsps on ippdvo2@ippc19 as a series of screens.
ippdvo2@ippc19:/home/panstarrs/ippdvo2>screen -list There are screens on: 16937.ippx038 (Detached) 18061.ippx040 (Detached) 21114.ipp0061 (Detached) 14854.ippx025 (Detached) 8346.ippx002 (Detached) 17709.ippx039 (Detached) 12913.ippx012 (Detached) 12513.ippx011 (Detached) 30664.ippx024 (Detached) 6716.ippx001 (Detached) 15710.ippx027 (Detached) 28930.ippx015 (Detached) 29697.ippx022 (Detached) 9390.ippx032 (Detached) 11759.ippx035 (Detached) 15441.ippx026 (Detached) 10147.ippx034 (Detached) 14845.ippx048 (Detached) 9809.ippx033 (Detached) 31005.ippx029 (Detached) 14303.ippx020 (Detached) 29540.ippx021 (Detached) 31320.ippx030 (Detached) 27989.ippx008 (Detached) 29405.ippx005 (Detached) 13691.ippx018 (Detached) 29204.ippx016 (Detached) 16163.ippx028 (Detached) 14377.ippx046 (Detached) 20417.ippx031 (Detached) 30106.ippx023 (Detached) 11621.ippx009 (Detached) 10348.ippx003 (Detached) 13826.ippx044 (Detached) 13109.ippx042 (Detached) 11023.ippx004 (Detached) 12503.ippx036 (Detached) 12892.ippx041 (Detached) 25079.ippx006 (Detached) 13980.ippx045 (Detached) 28605.ippx014 (Detached) 14692.ippx047 (Detached) 27623.ippx007 (Detached) 13412.ippx017 (Detached) 13945.ippx019 (Detached) 13492.ippx043 (Detached) 12115.ippx010 (Detached) 16621.ippx037 (Detached) 28308.ippx013 (Detached) 49 Sockets in /var/run/screen/S-ippdvo2.
you can check the status of loaders by the following:
cd ~heather/checkloaders perl checkloaders.pl loaderlist.txt
It will output the status of loaders:
loader.py running on ippx018 loader.py running on ippx019 problem with ippx020 problem with ippx021 loader.py running on ippx022 loader.py running on ippx023 problem with ippx024 loader.py running on ippx025 problem with ippx026 problem with ippx027 loader.py running on ippx028 problem with ippx029 loader.py running on ippx030 problem with ippx031 problem with ippx032 loader.py running on ippx033 loader.py running on ippx034 loader.py running on ippx035 loader.py running on ippx036 loader.py running on ippx037 loader.py running on ippx038
In general, only need to worry about the ones with 'problem'- attach to them (screen -r ippx031 for example), and see what's going on. mysql/io/neb/gpc1 weirdness will cause the loaders to die if they encounter any problems. Generally ok to hit up and resume, unless it indicates it is out of work:
2016-06-02 13:38:05.658 | INFO | -------------------------------------------------------------------------- 2016-06-02 13:38:05.661 | INFO | No outstanding boxes nothing to do here 2016-06-02 13:38:05.664 | INFO | Boxes with unprocessed data 0 2016-06-02 13:38:05.664 | INFO | numAttempts 0 2016-06-02 13:38:05.664 | INFO | done with processing boxes, numAttempts: 1 USING parallel 2016-06-02 13:38:05.677 | INFO | No outstanding boxes nothing to do here 2016-06-02 13:38:05.677 | INFO | Boxes with unprocessed data 0 2016-06-02 13:38:05.678 | INFO | numAttempts 1 2016-06-02 13:38:05.680 | INFO | done with processing boxes, numAttempts: 2 USING parallel 2016-06-02 13:38:05.690 | INFO | loader.py exited completed
If it is out of work, move on to the next skychunk to load, being extra careful NOT to have the same skychunk loaded by 2 loaders at the same time (causes race conditions)
