== heather's notes of stuff for ingest of OB 3pi PV2 ==

=== step 1 ===

 * compile trunk - called ipptrunk.20141009 as ippdvo 
 * make datastores for OB:
{{{
dsprodtool --add PSPS_PV2_OB_SLICE00 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE02 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE04 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE06 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE08 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE10 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE12 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE14 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE16 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE18 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE20 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE22 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
}}}
 * this was chosen to make roughly 10k per slice, to make it manageable.


=== step 2 queue ===

there is a script in queueLoaders in ~heather - I need to add to tools.

basically, loader name is PSPS_PV2_OB_SLICE00

=== step 3 set up stare nodes because they are cranky ? ===

* various iterations of start/restart of mysql , manually deleting locked tables and reconfiguring the scratch databases

=== step 4 add stsci loaders ===

magic incantations to configure mysql

{{{
    mysql -u root
    CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo';
    GRANT ALL on *.* to 'dvo'@'%';    
grant all privileges on *.* to dvo@localhost identified by 'dvo';
flush privileges;
    show grants; 
exit
mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch"
ippjython setupScratchDb.py
}}}


== Current configuration of loaders ==

The current configuration of loading— ippdvo@ippc18, several screens (screen -r stare00 etc)

This is the configuration of loaders

stare00 — slice 00
stare01 — slice 06
stare03 — slice 02
stare04 — slice 04
ipp006  —slice 08
stsci00 — slice 10
stsci01 — slice 12
stsci02 — slice 14
stsci04 — slice 16
stsci05 —slice 18
stsci06 —slice 20
stsci07 —slice 22

== Checking/Restarting logs ==

=== Saturday, 7:30am ===  
 all loaders are dead. restarted them

=== Saturday, 12:00pm === 
 3 loaders needed to be restarted (stsci/ipp006)

=== Saturday, 8:00pm === 
 * first query is how many are pending for each slice
 * second is total number published
 * third is how many total are remaining
 
{{{
 select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     6500 | PSPS_PV2_OB_SLICE00 | 
|     7444 | PSPS_PV2_OB_SLICE02 | 
|    10242 | PSPS_PV2_OB_SLICE04 | 
|    10468 | PSPS_PV2_OB_SLICE06 | 
|     8023 | PSPS_PV2_OB_SLICE08 | 
|     6486 | PSPS_PV2_OB_SLICE10 | 
|     7043 | PSPS_PV2_OB_SLICE12 | 
|     9248 | PSPS_PV2_OB_SLICE14 | 
|    11355 | PSPS_PV2_OB_SLICE16 | 
|    12498 | PSPS_PV2_OB_SLICE18 | 
|    12606 | PSPS_PV2_OB_SLICE20 | 
|     9308 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.21 sec)

+----------+
| count(*) |
+----------+
|    11662 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   111221 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)
}}}
Loaders which needed restart:

 * stare00 - restart
 * stare03 -restart
 * ipp006 -restart
 * stsci00 , 01, 03, 06, 07 -restart

=== Sunday 9:15am ===

Loaders whcih need restart:
stare01
stare03
ipp006
stsci00
stsci01
stsci03
stsci04
stsci05

(stare00/04/stsci6/7 did not)
{{{
select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     5817 | PSPS_PV2_OB_SLICE00 | 
|     7359 | PSPS_PV2_OB_SLICE02 | 
|     9422 | PSPS_PV2_OB_SLICE04 | 
|    10425 | PSPS_PV2_OB_SLICE06 | 
|     7978 | PSPS_PV2_OB_SLICE08 | 
|     6381 | PSPS_PV2_OB_SLICE10 | 
|     6999 | PSPS_PV2_OB_SLICE12 | 
|     9094 | PSPS_PV2_OB_SLICE14 | 
|    11213 | PSPS_PV2_OB_SLICE16 | 
|    12397 | PSPS_PV2_OB_SLICE18 | 
|    12021 | PSPS_PV2_OB_SLICE20 | 
|     8814 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.20 sec)

+----------+
| count(*) |
+----------+
|    14630 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   107920 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.14 sec)

}}}

=== sunday 7:00pm ===

I made a script to email me every hour, it checks the status of the 12 loaders.  Nothing knocked over. Here is the status of loading, 6000 loaded since the am:

{{{
mysql>  select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     5329 | PSPS_PV2_OB_SLICE00 | 
|     6768 | PSPS_PV2_OB_SLICE02 | 
|     9142 | PSPS_PV2_OB_SLICE04 | 
|     9717 | PSPS_PV2_OB_SLICE06 | 
|     7210 | PSPS_PV2_OB_SLICE08 | 
|     5734 | PSPS_PV2_OB_SLICE10 | 
|     6509 | PSPS_PV2_OB_SLICE12 | 
|     8459 | PSPS_PV2_OB_SLICE14 | 
|    10572 | PSPS_PV2_OB_SLICE16 | 
|    12055 | PSPS_PV2_OB_SLICE18 | 
|    11508 | PSPS_PV2_OB_SLICE20 | 
|     8313 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    20544 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   101316 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

mysql> 

}}}

stare03 stare04 were stuck - I restarted them after killing the processlist in mysql


=== Mon 8:30 am ===
{{{
mysql>  select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     4716 | PSPS_PV2_OB_SLICE00 | 
|     6752 | PSPS_PV2_OB_SLICE02 | 
|     9141 | PSPS_PV2_OB_SLICE04 | 
|     8800 | PSPS_PV2_OB_SLICE06 | 
|     6170 | PSPS_PV2_OB_SLICE08 | 
|     4832 | PSPS_PV2_OB_SLICE10 | 
|     6298 | PSPS_PV2_OB_SLICE12 | 
|     7490 | PSPS_PV2_OB_SLICE14 | 
|     9629 | PSPS_PV2_OB_SLICE16 | 
|    11493 | PSPS_PV2_OB_SLICE18 | 
|    10847 | PSPS_PV2_OB_SLICE20 | 
|     7609 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    27399 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|    93777 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)

}}}

stsci01 loader fell over
stare03 stare04 were funny needed restart

=== monday 9pm ===

several loaders fell over, they were fine until 7pm, they started failing a few at a time, this is the current status as of 9:30:

{{{
loader.py running on stare00
problem with stare01
loader.py running on stare03
loader.py running on stare04
loader.py running on ipp006
loader.py running on stsci00
problem with stsci01
problem with stsci03
problem with stsci04
loader.py running on stsci05
problem with stsci06
problem with stsci07
}}}

{{{
select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     4518 | PSPS_PV2_OB_SLICE00 | 
|     6751 | PSPS_PV2_OB_SLICE02 | 
|     9140 | PSPS_PV2_OB_SLICE04 | 
|     8021 | PSPS_PV2_OB_SLICE06 | 
|     5433 | PSPS_PV2_OB_SLICE08 | 
|     4266 | PSPS_PV2_OB_SLICE10 | 
|     5802 | PSPS_PV2_OB_SLICE12 | 
|     6752 | PSPS_PV2_OB_SLICE14 | 
|     9076 | PSPS_PV2_OB_SLICE16 | 
|    11103 | PSPS_PV2_OB_SLICE18 | 
|    10323 | PSPS_PV2_OB_SLICE20 | 
|     7104 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.19 sec)

+----------+
| count(*) |
+----------+
|    32304 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|    88289 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.12 sec)


}}}

 * stare00 is stuck
 * 01 needed restart
 * 03 is stuck
 * 04 is stuck
 * ipp006 is stuck (unstucked it)
 * stsci00 is stuck (not any more?)
 * 01 needed restart
 * 03/04/05/06/07 needed restart

 * stares 00 03 04were a bit more involved:  

{{{
mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps2_scratch"
mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch"
ippjython setupScratchDb.py
}}}

 * stsci07 needed a second restart

=== Tue 8:30 am ===

someone cranked the cluster up to 11 last night..  All my stuff stopped.

{{{
mysql>  select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     4516 | PSPS_PV2_OB_SLICE00 | 
|     6749 | PSPS_PV2_OB_SLICE02 | 
|     9139 | PSPS_PV2_OB_SLICE04 | 
|     7953 | PSPS_PV2_OB_SLICE06 | 
|     5334 | PSPS_PV2_OB_SLICE08 | 
|     3726 | PSPS_PV2_OB_SLICE10 | 
|     5426 | PSPS_PV2_OB_SLICE12 | 
|     6105 | PSPS_PV2_OB_SLICE14 | 
|     8944 | PSPS_PV2_OB_SLICE16 | 
|    11090 | PSPS_PV2_OB_SLICE18 | 
|    10216 | PSPS_PV2_OB_SLICE20 | 
|     7087 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.18 sec)

+----------+
| count(*) |
+----------+
|    34068 | 
+----------+
1 row in set (0.02 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|    86285 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)


}}}

 * stare00 - stalled --- > moved to stsci08
 * stare01 -stopped restarted
 * stare03 stalled

 * stare04 stalled --->  moved to stsci09
 * ipp006 restarted
 *stsci 00 /01 /03 ok
 * stsci 04 restarted
 * stsci 05 / 06 restarted
 * stsci 07 restarted