heather's notes of stuff for ingest of OB 3pi PV2

step 1

• compile trunk - called ipptrunk.20141009 as ippdvo
• make datastores for OB:

  dsprodtool --add PSPS_PV2_OB_SLICE00 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE02 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE04 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE06 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE08 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE10 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE12 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE14 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE16 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE18 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE20 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  dsprodtool --add PSPS_PV2_OB_SLICE22 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
  

• this was chosen to make roughly 10k per slice, to make it manageable.

step 2 queue

there is a script in queueLoaders in ~heather - I need to add to tools.

basically, loader name is PSPS_PV2_OB_SLICE00

step 3 set up stare nodes because they are cranky ?

• various iterations of start/restart of mysql , manually deleting locked tables and reconfiguring the scratch databases

step 4 add stsci loaders

magic incantations to configure mysql

    mysql -u root
    CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo';
    GRANT ALL on *.* to 'dvo'@'%';    
grant all privileges on *.* to dvo@localhost identified by 'dvo';
flush privileges;
    show grants; 
exit
mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch"
ippjython setupScratchDb.py

Current configuration of loaders

The current configuration of loading— ippdvo@ippc18, several screens (screen -r stare00 etc)

This is the configuration of loaders

stare00 — slice 00 stare01 — slice 06 stare03 — slice 02 stare04 — slice 04 ipp006 —slice 08 stsci00 — slice 10 stsci01 — slice 12 stsci02 — slice 14 stsci04 — slice 16 stsci05 —slice 18 stsci06 —slice 20 stsci07 —slice 22

Checking/Restarting logs

Saturday, 7:30am

all loaders are dead. restarted them

Saturday, 12:00pm

3 loaders needed to be restarted (stsci/ipp006)

Saturday, 8:00pm

• first query is how many are pending for each slice
• second is total number published
• third is how many total are remaining

 select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     6500 | PSPS_PV2_OB_SLICE00 | 
|     7444 | PSPS_PV2_OB_SLICE02 | 
|    10242 | PSPS_PV2_OB_SLICE04 | 
|    10468 | PSPS_PV2_OB_SLICE06 | 
|     8023 | PSPS_PV2_OB_SLICE08 | 
|     6486 | PSPS_PV2_OB_SLICE10 | 
|     7043 | PSPS_PV2_OB_SLICE12 | 
|     9248 | PSPS_PV2_OB_SLICE14 | 
|    11355 | PSPS_PV2_OB_SLICE16 | 
|    12498 | PSPS_PV2_OB_SLICE18 | 
|    12606 | PSPS_PV2_OB_SLICE20 | 
|     9308 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.21 sec)

+----------+
| count(*) |
+----------+
|    11662 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   111221 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

Loaders which needed restart:

• stare00 - restart
• stare03 -restart
• ipp006 -restart
• stsci00 , 01, 03, 06, 07 -restart

Sunday 9:15am

Loaders whcih need restart: stare01 stare03 ipp006 stsci00 stsci01 stsci03 stsci04 stsci05

(stare00/04/stsci6/7 did not)

select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     5817 | PSPS_PV2_OB_SLICE00 | 
|     7359 | PSPS_PV2_OB_SLICE02 | 
|     9422 | PSPS_PV2_OB_SLICE04 | 
|    10425 | PSPS_PV2_OB_SLICE06 | 
|     7978 | PSPS_PV2_OB_SLICE08 | 
|     6381 | PSPS_PV2_OB_SLICE10 | 
|     6999 | PSPS_PV2_OB_SLICE12 | 
|     9094 | PSPS_PV2_OB_SLICE14 | 
|    11213 | PSPS_PV2_OB_SLICE16 | 
|    12397 | PSPS_PV2_OB_SLICE18 | 
|    12021 | PSPS_PV2_OB_SLICE20 | 
|     8814 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.20 sec)

+----------+
| count(*) |
+----------+
|    14630 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   107920 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.14 sec)

sunday 7:00pm

I made a script to email me every hour, it checks the status of the 12 loaders. Nothing knocked over. Here is the status of loading, 6000 loaded since the am:

mysql>  select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     5329 | PSPS_PV2_OB_SLICE00 | 
|     6768 | PSPS_PV2_OB_SLICE02 | 
|     9142 | PSPS_PV2_OB_SLICE04 | 
|     9717 | PSPS_PV2_OB_SLICE06 | 
|     7210 | PSPS_PV2_OB_SLICE08 | 
|     5734 | PSPS_PV2_OB_SLICE10 | 
|     6509 | PSPS_PV2_OB_SLICE12 | 
|     8459 | PSPS_PV2_OB_SLICE14 | 
|    10572 | PSPS_PV2_OB_SLICE16 | 
|    12055 | PSPS_PV2_OB_SLICE18 | 
|    11508 | PSPS_PV2_OB_SLICE20 | 
|     8313 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    20544 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   101316 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

mysql> 

stare03 stare04 were stuck - I restarted them after killing the processlist in mysql

Mon 8:30 am

mysql>  select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     4716 | PSPS_PV2_OB_SLICE00 | 
|     6752 | PSPS_PV2_OB_SLICE02 | 
|     9141 | PSPS_PV2_OB_SLICE04 | 
|     8800 | PSPS_PV2_OB_SLICE06 | 
|     6170 | PSPS_PV2_OB_SLICE08 | 
|     4832 | PSPS_PV2_OB_SLICE10 | 
|     6298 | PSPS_PV2_OB_SLICE12 | 
|     7490 | PSPS_PV2_OB_SLICE14 | 
|     9629 | PSPS_PV2_OB_SLICE16 | 
|    11493 | PSPS_PV2_OB_SLICE18 | 
|    10847 | PSPS_PV2_OB_SLICE20 | 
|     7609 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    27399 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|    93777 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

stsci01 loader fell over stare03 stare04 were funny needed restart

monday 9pm

several loaders fell over, they were fine until 7pm, they started failing a few at a time, this is the current status as of 9:30:

loader.py running on stare00
problem with stare01
loader.py running on stare03
loader.py running on stare04
loader.py running on ipp006
loader.py running on stsci00
problem with stsci01
problem with stsci03
problem with stsci04
loader.py running on stsci05
problem with stsci06
problem with stsci07

select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     4518 | PSPS_PV2_OB_SLICE00 | 
|     6751 | PSPS_PV2_OB_SLICE02 | 
|     9140 | PSPS_PV2_OB_SLICE04 | 
|     8021 | PSPS_PV2_OB_SLICE06 | 
|     5433 | PSPS_PV2_OB_SLICE08 | 
|     4266 | PSPS_PV2_OB_SLICE10 | 
|     5802 | PSPS_PV2_OB_SLICE12 | 
|     6752 | PSPS_PV2_OB_SLICE14 | 
|     9076 | PSPS_PV2_OB_SLICE16 | 
|    11103 | PSPS_PV2_OB_SLICE18 | 
|    10323 | PSPS_PV2_OB_SLICE20 | 
|     7104 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    32304 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|    88289 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.12 sec)

• stare00 is stuck
• 01 needed restart
• 03 is stuck
• 04 is stuck
• ipp006 is stuck (unstucked it)
• stsci00 is stuck (not any more?)
• 01 needed restart
• 03/04/05/06/07 needed restart

• stares were a bit more involved:

{{ mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps2_scratch" mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch" ippjython setupScratchDb.py }}

• stsci07 needed a second restart