heather's notes of stuff for ingest of OB 3pi PV2

step 1

• compile trunk - called ipptrunk.20141009 as ippdvo
• make datastores for OB:

  dsprodtool --add PSPS_PV2_OB_SLICE00 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE02 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE04 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE06 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE08 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE10 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE12 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE14 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE16 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE18 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE20 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE22 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  

• this was chosen to make roughly 10k per slice, to make it manageable.

step 2 queue

there is a script in queueLoaders in ~heather - I need to add to tools.

basically, loader name is PSPS_PV2_OB_SLICE00

step 3 set up stare nodes because they are cranky ?

• various iterations of start/restart of mysql , manually deleting locked tables and reconfiguring the scratch databases

step 4 add stsci loaders

magic incantations to configure mysql

    mysql -u root
    CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo';
    GRANT ALL on *.* to 'dvo'@'%';    
grant all privileges on *.* to dvo@localhost identified by 'dvo';
flush privileges;
    show grants; 
exit
mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch"
ippjython setupScratchDb.py

Current configuration of loaders

The current configuration of loading— ippdvo@ippc18, several screens (screen -r stare00 etc)

This is the configuration of loaders

stare00 — slice 00 stare01 — slice 06 stare03 — slice 02 stare04 — slice 04 ipp006 —slice 08 stsci00 — slice 10 stsci01 — slice 12 stsci02 — slice 14 stsci04 — slice 16 stsci05 —slice 18 stsci06 —slice 20 stsci07 —slice 22

Checking/Restarting logs

Saturday, 7:30am

all loaders are dead. restarted them

Saturday, 12:00pm

3 loaders needed to be restarted (stsci/ipp006)

Saturday, 8:00pm

• first query is how many are pending for each slice
• second is total number published
• third is how many total are remaining

 select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     6500 | PSPS_PV2_OB_SLICE00 | 
|     7444 | PSPS_PV2_OB_SLICE02 | 
|    10242 | PSPS_PV2_OB_SLICE04 | 
|    10468 | PSPS_PV2_OB_SLICE06 | 
|     8023 | PSPS_PV2_OB_SLICE08 | 
|     6486 | PSPS_PV2_OB_SLICE10 | 
|     7043 | PSPS_PV2_OB_SLICE12 | 
|     9248 | PSPS_PV2_OB_SLICE14 | 
|    11355 | PSPS_PV2_OB_SLICE16 | 
|    12498 | PSPS_PV2_OB_SLICE18 | 
|    12606 | PSPS_PV2_OB_SLICE20 | 
|     9308 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.21 sec)

+----------+
| count(*) |
+----------+
|    11662 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   111221 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

Loaders which needed restart:

• stare00 - restart
• stare03 -restart
• ipp006 -restart
• stsci00 , 01, 03, 06, 07 -restart

Sunday 9:15am

Loaders whcih need restart: stare01 stare03 ipp006 stsci00 stsci01 stsci03 stsci04 stsci05

(stare00/04/stsci6/7 did not)

select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     5817 | PSPS_PV2_OB_SLICE00 | 
|     7359 | PSPS_PV2_OB_SLICE02 | 
|     9422 | PSPS_PV2_OB_SLICE04 | 
|    10425 | PSPS_PV2_OB_SLICE06 | 
|     7978 | PSPS_PV2_OB_SLICE08 | 
|     6381 | PSPS_PV2_OB_SLICE10 | 
|     6999 | PSPS_PV2_OB_SLICE12 | 
|     9094 | PSPS_PV2_OB_SLICE14 | 
|    11213 | PSPS_PV2_OB_SLICE16 | 
|    12397 | PSPS_PV2_OB_SLICE18 | 
|    12021 | PSPS_PV2_OB_SLICE20 | 
|     8814 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.20 sec)

+----------+
| count(*) |
+----------+
|    14630 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   107920 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.14 sec)

sunday 7:00pm

I made a script to email me every hour, it checks the status of the 12 loaders. Nothing knocked over. Here is the status of loading, 6000 loaded since the am:

mysql>  select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     5329 | PSPS_PV2_OB_SLICE00 | 
|     6768 | PSPS_PV2_OB_SLICE02 | 
|     9142 | PSPS_PV2_OB_SLICE04 | 
|     9717 | PSPS_PV2_OB_SLICE06 | 
|     7210 | PSPS_PV2_OB_SLICE08 | 
|     5734 | PSPS_PV2_OB_SLICE10 | 
|     6509 | PSPS_PV2_OB_SLICE12 | 
|     8459 | PSPS_PV2_OB_SLICE14 | 
|    10572 | PSPS_PV2_OB_SLICE16 | 
|    12055 | PSPS_PV2_OB_SLICE18 | 
|    11508 | PSPS_PV2_OB_SLICE20 | 
|     8313 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    20544 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   101316 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

mysql> 

stare03 stare04 were stuck - I restarted them after killing the processlist in mysql