| Version 24 (modified by , 11 years ago) ( diff ) |
|---|
heather's notes of stuff for ingest of OB 3pi PV2
step 1
- compile trunk - called ipptrunk.20141009 as ippdvo
- make datastores for OB:
dsprodtool --add PSPS_PV2_OB_SLICE00 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE02 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE04 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE06 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE08 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE10 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE12 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE14 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE16 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE18 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE20 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices' dsprodtool --add PSPS_PV2_OB_SLICE22 --type IPP-PSPS --description 'PSPS 3Pi PV2 OB slices'
- this was chosen to make roughly 10k per slice, to make it manageable.
step 2 queue
there is a script in queueLoaders in ~heather - I need to add to tools.
basically, loader name is PSPS_PV2_OB_SLICE00
step 3 set up stare nodes because they are cranky ?
- various iterations of start/restart of mysql , manually deleting locked tables and reconfiguring the scratch databases
step 4 add stsci loaders
magic incantations to configure mysql
mysql -u root
CREATE USER 'dvo'@'%' IDENTIFIED BY 'dvo';
GRANT ALL on *.* to 'dvo'@'%';
grant all privileges on *.* to dvo@localhost identified by 'dvo';
flush privileges;
show grants;
exit
mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch"
ippjython setupScratchDb.py
Current configuration of loaders
The current configuration of loading— ippdvo@ippc18, several screens (screen -r stare00 etc)
This is the configuration of loaders
stare00 — slice 00 stare01 — slice 06 stare03 — slice 02 stare04 — slice 04 ipp006 —slice 08 stsci00 — slice 10 stsci01 — slice 12 stsci02 — slice 14 stsci04 — slice 16 stsci05 —slice 18 stsci06 —slice 20 stsci07 —slice 22
Checking/Restarting logs
Saturday, 7:30am
all loaders are dead. restarted them
Saturday, 12:00pm
3 loaders needed to be restarted (stsci/ipp006)
Saturday, 8:00pm
- first query is how many are pending for each slice
- second is total number published
- third is how many total are remaining
select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 6500 | PSPS_PV2_OB_SLICE00 | | 7444 | PSPS_PV2_OB_SLICE02 | | 10242 | PSPS_PV2_OB_SLICE04 | | 10468 | PSPS_PV2_OB_SLICE06 | | 8023 | PSPS_PV2_OB_SLICE08 | | 6486 | PSPS_PV2_OB_SLICE10 | | 7043 | PSPS_PV2_OB_SLICE12 | | 9248 | PSPS_PV2_OB_SLICE14 | | 11355 | PSPS_PV2_OB_SLICE16 | | 12498 | PSPS_PV2_OB_SLICE18 | | 12606 | PSPS_PV2_OB_SLICE20 | | 9308 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.21 sec) +----------+ | count(*) | +----------+ | 11662 | +----------+ 1 row in set (0.01 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 111221 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.13 sec)
Loaders which needed restart:
- stare00 - restart
- stare03 -restart
- ipp006 -restart
- stsci00 , 01, 03, 06, 07 -restart
Sunday 9:15am
Loaders whcih need restart: stare01 stare03 ipp006 stsci00 stsci01 stsci03 stsci04 stsci05
(stare00/04/stsci6/7 did not)
select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 5817 | PSPS_PV2_OB_SLICE00 | | 7359 | PSPS_PV2_OB_SLICE02 | | 9422 | PSPS_PV2_OB_SLICE04 | | 10425 | PSPS_PV2_OB_SLICE06 | | 7978 | PSPS_PV2_OB_SLICE08 | | 6381 | PSPS_PV2_OB_SLICE10 | | 6999 | PSPS_PV2_OB_SLICE12 | | 9094 | PSPS_PV2_OB_SLICE14 | | 11213 | PSPS_PV2_OB_SLICE16 | | 12397 | PSPS_PV2_OB_SLICE18 | | 12021 | PSPS_PV2_OB_SLICE20 | | 8814 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.20 sec) +----------+ | count(*) | +----------+ | 14630 | +----------+ 1 row in set (0.01 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 107920 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.14 sec)
sunday 7:00pm
I made a script to email me every hour, it checks the status of the 12 loaders. Nothing knocked over. Here is the status of loading, 6000 loaded since the am:
mysql> select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 5329 | PSPS_PV2_OB_SLICE00 | | 6768 | PSPS_PV2_OB_SLICE02 | | 9142 | PSPS_PV2_OB_SLICE04 | | 9717 | PSPS_PV2_OB_SLICE06 | | 7210 | PSPS_PV2_OB_SLICE08 | | 5734 | PSPS_PV2_OB_SLICE10 | | 6509 | PSPS_PV2_OB_SLICE12 | | 8459 | PSPS_PV2_OB_SLICE14 | | 10572 | PSPS_PV2_OB_SLICE16 | | 12055 | PSPS_PV2_OB_SLICE18 | | 11508 | PSPS_PV2_OB_SLICE20 | | 8313 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.19 sec) +----------+ | count(*) | +----------+ | 20544 | +----------+ 1 row in set (0.02 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 101316 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.13 sec) mysql>
stare03 stare04 were stuck - I restarted them after killing the processlist in mysql
Mon 8:30 am
mysql> select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 4716 | PSPS_PV2_OB_SLICE00 | | 6752 | PSPS_PV2_OB_SLICE02 | | 9141 | PSPS_PV2_OB_SLICE04 | | 8800 | PSPS_PV2_OB_SLICE06 | | 6170 | PSPS_PV2_OB_SLICE08 | | 4832 | PSPS_PV2_OB_SLICE10 | | 6298 | PSPS_PV2_OB_SLICE12 | | 7490 | PSPS_PV2_OB_SLICE14 | | 9629 | PSPS_PV2_OB_SLICE16 | | 11493 | PSPS_PV2_OB_SLICE18 | | 10847 | PSPS_PV2_OB_SLICE20 | | 7609 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.19 sec) +----------+ | count(*) | +----------+ | 27399 | +----------+ 1 row in set (0.02 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 93777 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.13 sec)
stsci01 loader fell over stare03 stare04 were funny needed restart
monday 9pm
several loaders fell over, they were fine until 7pm, they started failing a few at a time, this is the current status as of 9:30:
loader.py running on stare00 problem with stare01 loader.py running on stare03 loader.py running on stare04 loader.py running on ipp006 loader.py running on stsci00 problem with stsci01 problem with stsci03 problem with stsci04 loader.py running on stsci05 problem with stsci06 problem with stsci07
select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 4518 | PSPS_PV2_OB_SLICE00 | | 6751 | PSPS_PV2_OB_SLICE02 | | 9140 | PSPS_PV2_OB_SLICE04 | | 8021 | PSPS_PV2_OB_SLICE06 | | 5433 | PSPS_PV2_OB_SLICE08 | | 4266 | PSPS_PV2_OB_SLICE10 | | 5802 | PSPS_PV2_OB_SLICE12 | | 6752 | PSPS_PV2_OB_SLICE14 | | 9076 | PSPS_PV2_OB_SLICE16 | | 11103 | PSPS_PV2_OB_SLICE18 | | 10323 | PSPS_PV2_OB_SLICE20 | | 7104 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.19 sec) +----------+ | count(*) | +----------+ | 32304 | +----------+ 1 row in set (0.02 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 88289 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.12 sec)
- stare00 is stuck
- 01 needed restart
- 03 is stuck
- 04 is stuck
- ipp006 is stuck (unstucked it)
- stsci00 is stuck (not any more?)
- 01 needed restart
- 03/04/05/06/07 needed restart
- stares 00 03 04were a bit more involved:
mysql -h localhost -u dvo -pdvo -e "drop database ipptopsps2_scratch" mysql -h localhost -u dvo -pdvo -e "create database ipptopsps2_scratch" ippjython setupScratchDb.py
- stsci07 needed a second restart
Tue 8:30 am
someone cranked the cluster up to 11 last night.. All my stuff stopped.
mysql> select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 4516 | PSPS_PV2_OB_SLICE00 | | 6749 | PSPS_PV2_OB_SLICE02 | | 9139 | PSPS_PV2_OB_SLICE04 | | 7953 | PSPS_PV2_OB_SLICE06 | | 5334 | PSPS_PV2_OB_SLICE08 | | 3726 | PSPS_PV2_OB_SLICE10 | | 5426 | PSPS_PV2_OB_SLICE12 | | 6105 | PSPS_PV2_OB_SLICE14 | | 8944 | PSPS_PV2_OB_SLICE16 | | 11090 | PSPS_PV2_OB_SLICE18 | | 10216 | PSPS_PV2_OB_SLICE20 | | 7087 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.18 sec) +----------+ | count(*) | +----------+ | 34068 | +----------+ 1 row in set (0.02 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 86285 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.13 sec)
- stare00 - stalled --- > moved to stsci08
- stare01 -stopped restarted
- stare03 stalled
- stare04 stalled ---> moved to stsci09
- ipp006 restarted *stsci 00 /01 /03 ok
- stsci 04 restarted
- stsci 05 / 06 restarted
- stsci 07 restarted
Tue 10:45am
- stsci10 needs restart
mysql> mysql> select count(*), datastore_product from pending join box on box_ijoin skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 4407 | PSPS_PV2_OB_SLICE00 | | 6621 | PSPS_PV2_OB_SLICE02 | | 9086 | PSPS_PV2_OB_SLICE04 | | 7859 | PSPS_PV2_OB_SLICE06 | | 5157 | PSPS_PV2_OB_SLICE08 | | 3597 | PSPS_PV2_OB_SLICE10 | | 5311 | PSPS_PV2_OB_SLICE12 | | 5973 | PSPS_PV2_OB_SLICE14 | | 8830 | PSPS_PV2_OB_SLICE16 | | 11001 | PSPS_PV2_OB_SLICE18 | | 10128 | PSPS_PV2_OB_SLICE20 | | 7006 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.18 sec) +----------+ | count(*) | +----------+ | 35241 | +----------+ 1 row in set (0.02 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 84976 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.13 sec)
Tuesday 7:00pm
I've been checking my email (the hourly reports) - there's been no need to restart things since the am. The status of loading at 7pm:
mysql> mysql> select count(*), datastore_product from pending join box on box_ijoin skychunk on name = skychunk where name like 'PSPS%' group by datastore_product ;select count(*) from batch where dvo_db like 'LAP.PV2%' and loaded_to_datastore =1; select count(*), datastore_product from pending join box on box_id = id join skychunk on name = skychunk where name like 'PSPS%'; +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 3941 | PSPS_PV2_OB_SLICE00 | | 6088 | PSPS_PV2_OB_SLICE02 | | 8448 | PSPS_PV2_OB_SLICE04 | | 7323 | PSPS_PV2_OB_SLICE06 | | 4570 | PSPS_PV2_OB_SLICE08 | | 3138 | PSPS_PV2_OB_SLICE10 | | 4984 | PSPS_PV2_OB_SLICE12 | | 5379 | PSPS_PV2_OB_SLICE14 | | 8387 | PSPS_PV2_OB_SLICE16 | | 10801 | PSPS_PV2_OB_SLICE18 | | 9729 | PSPS_PV2_OB_SLICE20 | | 6618 | PSPS_PV2_OB_SLICE22 | +----------+---------------------+ 12 rows in set (0.18 sec) +----------+ | count(*) | +----------+ | 40287 | +----------+ 1 row in set (0.02 sec) +----------+---------------------+ | count(*) | datastore_product | +----------+---------------------+ | 79406 | PSPS_PV2_OB_SLICE00 | +----------+---------------------+ 1 row in set (0.13 sec)
wed 8am
ipp006 loading fell over about 7am, stsci10 at about 8 Both restarted.
Status of loaders loader.py running on stsci08 loader.py running on stare01 loader.py running on stsci11 loader.py running on stsci10 problem with ipp006 loader.py running on stsci00 loader.py running on stsci01 loader.py running on stsci03 loader.py running on stsci09 loader.py running on stsci05 loader.py running on stsci06 loader.py running on stsci07 loader numbers count(*) datastore_product 3274 PSPS_PV2_OB_SLICE00 5192 PSPS_PV2_OB_SLICE02 7483 PSPS_PV2_OB_SLICE04 6500 PSPS_PV2_OB_SLICE06 3704 PSPS_PV2_OB_SLICE08 2446 PSPS_PV2_OB_SLICE10 4362 PSPS_PV2_OB_SLICE12 4437 PSPS_PV2_OB_SLICE14 7626 PSPS_PV2_OB_SLICE16 10362 PSPS_PV2_OB_SLICE18 9081 PSPS_PV2_OB_SLICE20 6031 PSPS_PV2_OB_SLICE22 total done count(*) 48296 pending count(*) datastore_product 70497 PSPS_PV2_OB_SLICE00
here is the coverage of batches (green = good, white = needs to be done, black = failed, need to retry) - please click to get big version
wed 7:30pm
- restarted ipp006, stsci00, stsci06, stsci07, stsci11.
Status of loaders
loader.py running on stsci08 loader.py running on stare01 loader.py running on stsci11 loader.py running on stsci10 problem with ipp006 problem with stsci00 loader.py running on stsci01 loader.py running on stsci03 loader.py running on stsci09 loader.py running on stsci05 loader.py running on stsci06 problem with stsci07 loader.py running on stsci13 loader.py running on stsci14 loader.py running on stsci15 loader numbers count(*) datastore_product 2563 PSPS_PV2_OB_SLICE00 4278 PSPS_PV2_OB_SLICE02 6638 PSPS_PV2_OB_SLICE04 5726 PSPS_PV2_OB_SLICE06 2928 PSPS_PV2_OB_SLICE08 1789 PSPS_PV2_OB_SLICE10 3792 PSPS_PV2_OB_SLICE12 3637 PSPS_PV2_OB_SLICE14 6725 PSPS_PV2_OB_SLICE16 9777 PSPS_PV2_OB_SLICE18 8444 PSPS_PV2_OB_SLICE20 5388 PSPS_PV2_OB_SLICE22 total done count(*) 56315
thurs 8am
stsci09 and stsci06 fell over, they have been restarted
Status of loaders loader.py running on stsci08 loader.py running on stare01 loader.py running on stsci11 loader.py running on stsci10 loader.py running on ipp006 loader.py running on stsci00 loader.py running on stsci01 loader.py running on stsci03 problem with stsci09 loader.py running on stsci05 problem with stsci06 loader.py running on stsci07 loader.py running on stsci13 loader.py running on stsci14 loader.py running on stsci15 loader numbers count(*) datastore_product 1908 PSPS_PV2_OB_SLICE00 3430 PSPS_PV2_OB_SLICE02 5814 PSPS_PV2_OB_SLICE04 4989 PSPS_PV2_OB_SLICE06 2295 PSPS_PV2_OB_SLICE08 1250 PSPS_PV2_OB_SLICE10 3225 PSPS_PV2_OB_SLICE12 2771 PSPS_PV2_OB_SLICE14 5824 PSPS_PV2_OB_SLICE16 8828 PSPS_PV2_OB_SLICE18 7701 PSPS_PV2_OB_SLICE20 4890 PSPS_PV2_OB_SLICE22 total done count(*) 64283 pending count(*) datastore_product 52925 PSPS_PV2_OB_SLICE00
Attachments (3)
- obplot.png (167.2 KB ) - added by 11 years ago.
- obplot.20150124.png (156.5 KB ) - added by 11 years ago.
- obplot.20150123.png (173.8 KB ) - added by 11 years ago.
Download all attachments as: .zip
Note:
See TracWiki
for help on using the wiki.
