This wiki is to track the merge progress of stuff

step 0

remove/rename all prior incantations of LAP.20120706 merge because we had problems. This has been done - basically all old stuff has name of .drop added to it

step 1

Kepler. Ken wants to merge kepler and has declared Kepler to consist of ra: 282-300 and dec:40-50 - this is not all of kepler but the majority of it. Kepler exists in slice18 (both skycal and cam). To find whic ones are specifically kepler:

mysql> select minidvodb_name, ra*180/3.1415927 ,decl*180/3.1415927, count(*) from addRun join camRun on stage_id = cam_id join chipRun using (chip_id) join rawExp using (exp_id) where addRun.minidvodb_group = 'LAP.slice18.cam' and stage = 'cam' and ra*180/3.1415927 > 282 and ra*180/3.1415927 < 300 and decl*180/3.1415927 > 40 and decl*180/3.1415927 < 50 group by minidvodb_name;
+----------------------+------------------+--------------------+----------+
| minidvodb_name       | ra*180/3.1415927 | decl*180/3.1415927 | count(*) |
+----------------------+------------------+--------------------+----------+
| LAP.slice18.cam.1802 | 283.532343323818 |   43.8456519204415 |       77 | 
| LAP.slice18.cam.1805 | 299.307475472552 |   40.0563955983218 |       83 | 
| LAP.slice18.cam.1808 | 284.438418767653 |   41.7542987033297 |       32 | 
| LAP.slice18.cam.1810 | 284.438590654989 |   41.7544705906657 |       77 | 
| LAP.slice18.cam.1812 | 283.909578730559 |   48.0151803255718 |      157 | 
| LAP.slice18.cam.1815 | 296.846106116811 |   40.0083244400205 |      143 | 
| LAP.slice18.cam.1818 | 286.823928512439 |   48.9638838287344 |      106 | 
| LAP.slice18.cam.1821 | 292.127741447833 |   44.1639299709348 |      134 | 
| LAP.slice18.cam.1823 | 297.915188050953 |   41.2683732044577 |      140 | 
| LAP.slice18.cam.1825 | 282.540667986655 |   44.6111808192068 |      110 | 
| LAP.slice18.cam.1827 | 299.947698503374 |   46.6876944296439 |       12 | 
| LAP.slice18.cam.1849 | 282.167099509749 |   44.9916820853321 |        1 | 
+----------------------+------------------+--------------------+----------+
12 rows in set (1.42 sec)

Therefore, for the first tests, need to only merge 12 cam minidvodbs (vs 84). Oh! Wait - what about edges (+/- 3 degrees?) I need to redo the query. However, first things first, these are the first 11 minidvos to ingest so I should queue and start that. They have been queued (yay). Actually, there aren't that many extra past those so all of slice18cam was queued

Step 2

manually build the mergedvodb. This is because merging needs some help, and gene wants me to hack something together and this isn't as nice as it will eventually be. Eventually, you will be able to set the kind of merge to do (parallel or not), and it will be tracked in the database. For now, there are a few manual steps to be done, followed by forcing the scripts to do parallel. Ugly.

• Manual step 1: rsync as before
• Manual step 2: make and edit the hosttable.dat so that it knows where the parallel stuff will live. For this one, it is something like /data/ippxxx.0/gpc1/catdirs/parallel/LAP.ThreePi.20120706.2069 where 2069 is the merge_id of the first one to be merged in. This is to prevent clashes between recycled dvodb names, should that happen in the future.
• Manual step 3: create the directories for hosttable.dat - I had to add gpc1, then gpc1/catdirs/ then gpc1/catdirs/parallel to most for this time but if we keep the same structure that won't be needed in the future.
• Manual step 4: do dvodist -out /path/to/where/rsynced/above . for me it was dvodist -out /data/ipp005.0/gpc1/catdirs/LAP.ThreePi.20120706
• Manual step 5: assuming previous 4 steps were successful (check!), then do mergetool -addmerged -merge_id 2069 -mergedvodb LAP.20120706 -dtime_verify 42 -dtime_merge 42 -dtime_script 28880 -fault 0 -dbname gpc1 -epoch 2013-02-05 note the times all say 42 except for the script time which is roughly 8 hours (that's a guess on my part...). 42 is the answer to the universe, also used when someone manually does stuff.

• oooookay!!! I think we are almost ready. First step is to rebuild dvo to pick up the new stuff (dvodist -out for example)
• Rebuild step 1 : rebuild current ipp (here)
• rebuild step 2: add hacks to force to be parallel
• rebuild step 3: restart and test. sacrifice various animals if necessary.
• rebuild step 4: no animal sacrifices necessary yet...

• to check: there are 152 or so faulted addstuffs..why?? note I don't think it has to do with what I've been doing ... oh it is...I fail at compiling.. I think it's okay though... need to check...

restart

The mergedvodb that I created failed because of file permissions. I have fixed the directory permission problem but I need to start over:

• step 1 rsync (done)-> /data/ipp005.0/gpc1/catdirs/LAP.ThreePi.20120706.20130208 == this is a horrible name but at least it's descriptive
• step 2 (done) fix hosttable.dat
• step 3 (done) dvodist -out
• step 4 (done) fix the mergedvodbRun tables - want to redo the merging.
  • check that addstarlap and addstar do not have anything in show.mergedvodb or survey.show.mergedvodb (I deleted them)
  • use these commands to move stuff out of the way in the db

    mysql> update mergedvodbRun set mergedvodb_path = '/data/ipp005.0/gpc1/catdirs/LAP.ThreePi.20120706.drop.20130207' where mergedvodb = 'LAP.20120706';
    Query OK, 74 rows affected (0.04 sec)
    Rows matched: 74  Changed: 74  Warnings: 0
    
    mysql> update mergedvodbRun set mergedvodb = 'LAP.20120706.drop.20130207' where mergedvodb = 'LAP.20120706';
    Query OK, 74 rows affected (0.01 sec)
    Rows matched: 74  Changed: 74  Warnings: 0
    

  • move stuff out of the way on the ipp005.0 side;

    ippdvo@ipp005:/data/ipp005.0/gpc1/catdirs>mv LAP.ThreePi.20120706.drop2 LAP.ThreePi.20120706.drop.20130207
    

  • pantasks_client stuff:

    pantasks: mergedvodb.merge.off
    

  • queue:

    survey.add.mergedvodb LAP.20120706.cam LAP.20120706 LAP.slice18.cam
    

  • remove from queue:

    survey.del.mergedvodb LAP.20120706.cam
    

  • and repeat for .skycal
• step 5 (done)- fake the mergeRun stuff for the first column

  mergetool -addmerged -merge_id 2143 -mergedvodb LAP.20120706 -dtime_verify 42 -dtime_merge 42 -dtime_script 28880 -fault 0 -dbname gpc1 -epoch 2013-02-09
  

• step 6 -restart - at this point it's just mergedvodb.merge.on I think.

addstar -resort

• first attempt at addstar -resort -- all sorts of problems, there was a bug in the resort code. Gene fixed this and noted that he's pretty sure dvomerge already sorts. However, we did the resorting anyways.

• second attempt: svn up of all of ippcode on ippdvo's ipptrunk.20130205 directory, stop and shutdown of addstars, rebuild of ipp, restart of addstars, and restart of addstar -resort. It failed, due to:

  can't lock file /data/ipp016.0/gpc1/catdirs/parallel/LAP.ThreePi.20120706.2143//n1500/1566.00.cpt
  ERROR: failure to open/create catalog file /data/ipp016.0/gpc1/catdirs/parallel/LAP.ThreePi.20120706.2143//n1500/1566.00.cpt
  

• third attempt: retry addstar_client resort on ipp016:{{{addstar_client -resort -hostID 16 -D CATDIR /data/ipp005.0/gpc1/catdirs/LAP.ThreePi.20120706 -hostdir /data/ipp016.0/gpc1/catdirs/parallel/LAP.ThreePi.20120706.2143/ -region 0.000000 360.000000 -90.000000 90.000000

}}}, same failure as before.

• fourth attempt:
  • gene removed/sorted out the lock file problem. redo of addstar_client -resort -hostID 16 -D CATDIR /data/ipp005.0/gpc1/catdirs/LAP.ThreePi.20120706 -hostdir /data/ipp016.0/gpc1/catdirs/parallel/LAP.ThreePi.20120706.2143/ -region 0.000000 360.000000 -90.000000 90.000000 on ipp016 and it reports success.

rest of dvodb calibration

• setphot - edit run.setphot to turn on realrun and parallel. running : src/ipptrunk.20130205/tools/run.setphot /data/ipp005.0/gpc1/catdirs/LAP.ThreePi.20120706 >& setphot.20130211.txt
• relphot
• relastro