Changes between Initial Version and Version 1 of IppToPspsIngestOf3PiPv3

Timestamp:

Jun 2, 2016, 1:39:21 PM (10 years ago)

Author:

heather

Comment:

--

IppToPspsIngestOf3PiPv3

@@ +1 @@
+Notes on ingest of PV3 ipptopsps go here
+
+== Current version of ipptopsps == 
+
+using ippdvo2 account: 
+
+{{{
+ippdvo2@ippc19:/home/panstarrs/ippdvo2>psconfig
+/home/panstarrs/ippdvo2/psconfig : ipptrunk.20160518 : lin64
+}}}
+
+== Current status of ipptopsps ==
+
+ * all of these are located in datastores like 'PSPS_PV3_ST_SLICE_X' 
+ * IN - needs to be generated
+ * OB - the boundaries need to be regenerated once Gene investigates
+ * ST - edges are avoided; found 1 hole (non-boundary) so far with dvo duplicates, and 2 batches with invalid cmf info.
+ * P2 - Wait for Gene
+ * FW - Wait for Gene
+ * FG/FO - Need to verify ffg minis before handoff to Gene, then wait for Gene.
+ * DF/DO - addstar in progress. large number of faults due to missing files on space telescope machines
+
+== Fastest way to set up mysql on 'new' machines ==
+
+as root on the machine in question
+{{{
+cp /etc/mysql/my.cnf /etc/mysql/my.cnf.orig
+cp ~heather/mysqlconfig/my.cnf /etc/mysql/my.cnf 
+/usr/bin/mysql_install_db
+/etc/init.d/mysql restart
+}}}
+
+wait a bit, it can take a bit of time to start, and the status it displays when doing mysql restart is generally useless. check if it is running with mysql -h localhost -u root
+
+now, as ippdvo2 on that machine (not sure why, but it doesn't take as root), the following will add the necessary accounts, initialize and install the scratch database.
+
+{{{
+mysql -h localhost -u root
+CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo';
+GRANT ALL on *.* to 'dvo'@'%'; 
+CREATE USER 'dvo'@'localhost' IDENTIFIED BY 'dvo';
+GRANT ALL on *.* to 'dvo'@'localhost';    
+create database ipptopsps_scratch;
+flush privileges;
+exit
+ippjython setupScratchDb.py
+}}}
+
+== current ipptopsps machines ==
+
+ * ipp006 is the database machine, name of database is ipptopsps2
+ * ippx001 - ippx048 are currently set up for ipptopsps (correct schema, mysql, etc)
+
+== current checker scripts ==
+
+I am running ipptopsps on ippdvo2@ippc19 as a series of screens.  
+
+{{{
+ippdvo2@ippc19:/home/panstarrs/ippdvo2>screen -list
+There are screens on:
+        16937.ippx038   (Detached)
+        18061.ippx040   (Detached)
+        21114.ipp0061   (Detached)
+        14854.ippx025   (Detached)
+        8346.ippx002    (Detached)
+        17709.ippx039   (Detached)
+        12913.ippx012   (Detached)
+        12513.ippx011   (Detached)
+        30664.ippx024   (Detached)
+        6716.ippx001    (Detached)
+        15710.ippx027   (Detached)
+        28930.ippx015   (Detached)
+        29697.ippx022   (Detached)
+        9390.ippx032    (Detached)
+        11759.ippx035   (Detached)
+        15441.ippx026   (Detached)
+        10147.ippx034   (Detached)
+        14845.ippx048   (Detached)
+        9809.ippx033    (Detached)
+        31005.ippx029   (Detached)
+        14303.ippx020   (Detached)
+        29540.ippx021   (Detached)
+        31320.ippx030   (Detached)
+        27989.ippx008   (Detached)
+        29405.ippx005   (Detached)
+        13691.ippx018   (Detached)
+        29204.ippx016   (Detached)
+        16163.ippx028   (Detached)
+        14377.ippx046   (Detached)
+        20417.ippx031   (Detached)
+        30106.ippx023   (Detached)
+        11621.ippx009   (Detached)
+        10348.ippx003   (Detached)
+        13826.ippx044   (Detached)
+        13109.ippx042   (Detached)
+        11023.ippx004   (Detached)
+        12503.ippx036   (Detached)
+        12892.ippx041   (Detached)
+        25079.ippx006   (Detached)
+        13980.ippx045   (Detached)
+        28605.ippx014   (Detached)
+        14692.ippx047   (Detached)
+        27623.ippx007   (Detached)
+        13412.ippx017   (Detached)
+        13945.ippx019   (Detached)
+        13492.ippx043   (Detached)
+        12115.ippx010   (Detached)
+        16621.ippx037   (Detached)
+        28308.ippx013   (Detached)
+49 Sockets in /var/run/screen/S-ippdvo2.
+}}}
+
+you can check the status of loaders by the following:
+
+{{{
+cd ~heather/checkloaders
+perl checkloaders.pl loaderlist.txt
+}}}
+
+It will output the status of loaders:
+
+{{{
+loader.py running on ippx018
+loader.py running on ippx019
+problem with ippx020
+problem with ippx021
+loader.py running on ippx022
+loader.py running on ippx023
+problem with ippx024
+loader.py running on ippx025
+problem with ippx026
+problem with ippx027
+loader.py running on ippx028
+problem with ippx029
+loader.py running on ippx030
+problem with ippx031
+problem with ippx032
+loader.py running on ippx033
+loader.py running on ippx034
+loader.py running on ippx035
+loader.py running on ippx036
+loader.py running on ippx037
+loader.py running on ippx038
+}}}
+
+In general, only need to worry about the ones with 'problem'- attach to them (screen -r ippx031 for example), and see what's going on.   mysql/io/neb/gpc1 weirdness will cause the loaders to die if they encounter any problems.  Generally ok to hit up and resume, unless it indicates it is out of work:
+
+{{{
+2016-06-02 13:38:05.658 |    INFO | --------------------------------------------------------------------------
+2016-06-02 13:38:05.661 |    INFO | No outstanding boxes                    nothing to do here
+2016-06-02 13:38:05.664 |    INFO | Boxes with unprocessed data             0
+2016-06-02 13:38:05.664 |    INFO | numAttempts                             0
+2016-06-02 13:38:05.664 |    INFO | done with processing boxes, numAttempts: 1
+USING parallel
+2016-06-02 13:38:05.677 |    INFO | No outstanding boxes                    nothing to do here
+2016-06-02 13:38:05.677 |    INFO | Boxes with unprocessed data             0
+2016-06-02 13:38:05.678 |    INFO | numAttempts                             1
+2016-06-02 13:38:05.680 |    INFO | done with processing boxes, numAttempts: 2
+USING parallel
+2016-06-02 13:38:05.690 |    INFO | loader.py exited                        completed
+}}}
+
+If it is out of work, move on to the next skychunk to load, being extra careful NOT to have the same skychunk loaded by 2 loaders at the same time (causes race conditions)