== heather's notes of stuff for ingest of OB 3pi PV2 ==

=== step 1 ===

 * compile trunk - called ipptrunk.20141009 as ippdvo 
 * make datastores for OB:
{{{
dsprodtool --add PSPS_PV2_OB_SLICE00 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE02 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE04 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE06 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE08 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE10 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE12 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE14 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE16 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE18 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE20 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
dsprodtool --add PSPS_PV2_OB_SLICE22 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
}}}
 * this was chosen to make roughly 10k per slice, to make it manageable.


=== step 2 queue ===

there is a script in queueLoaders in ~heather - I need to add to tools.

basically, loader name is PSPS_PV2_OB_SLICE00

=== step 3 set up stare nodes because they are cranky ? ===

* various iterations of start/restart of mysql , manually deleting locked tables and reconfiguring the scratch databases

=== step 4 add stsci loaders ===

magic incantations to configure mysql

{{{
    mysql -u root
    CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo';
    GRANT ALL on *.* to 'dvo'@'%';    
grant all privileges on *.* to dvo@localhost identified by 'dvo';
flush privileges;
    show grants; 
exit
mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch"
ippjython setupScratchDb.py
}}}


== Current configuration of loaders ==

The current configuration of loading— ippdvo@ippc18, several screens (screen -r stare00 etc)

This is the configuration of loaders

stare00 — slice 00
stare01 — slice 06
stare03 — slice 02
stare04 — slice 04
ipp006  —slice 08
stsci00 — slice 10
stsci01 — slice 12
stsci02 — slice 14
stsci04 — slice 16
stsci05 —slice 18
stsci06 —slice 20
stsci07 —slice 22

== Checking/Restarting logs ==

=== Saturday, 7:30am ===  
 all loaders are dead. restarted them

=== Saturday, 12:00pm === 
 3 loaders needed to be restarted (stsci/ipp006)

=== Saturday, 8:00pm === 
 * first query is how many are pending for each slice
 * second is total number published
 * third is how many total are remaining
 
{{{
 select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%';
+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|     6500 | PSPS_PV2_OB_SLICE00 | 
|     7444 | PSPS_PV2_OB_SLICE02 | 
|    10242 | PSPS_PV2_OB_SLICE04 | 
|    10468 | PSPS_PV2_OB_SLICE06 | 
|     8023 | PSPS_PV2_OB_SLICE08 | 
|     6486 | PSPS_PV2_OB_SLICE10 | 
|     7043 | PSPS_PV2_OB_SLICE12 | 
|     9248 | PSPS_PV2_OB_SLICE14 | 
|    11355 | PSPS_PV2_OB_SLICE16 | 
|    12498 | PSPS_PV2_OB_SLICE18 | 
|    12606 | PSPS_PV2_OB_SLICE20 | 
|     9308 | PSPS_PV2_OB_SLICE22 | 
+----------+---------------------+
12 rows in set (0.21 sec)

+----------+
| count(*) |
+----------+
|    11662 | 
+----------+
1 row in set (0.01 sec)

+----------+---------------------+
| count(*) | datastore_product   |
+----------+---------------------+
|   111221 | PSPS_PV2_OB_SLICE00 | 
+----------+---------------------+
1 row in set (0.13 sec)
}}}
Loaders which needed restart:

 * stare00 - restart
 * stare03 -restart
 * ipp006 -restart
 * stsci00 , 01, 03, 06, 07 -restart