Introduction

We have had problems with addstar + merging in the past, and we need to fix them. There are several failure modes which have caused problems in the past, or which I imagine could cause problems:

• the addstar machine (ipp004 and/or ipp005) fills up
• there are nfs related issues (this may cause problems with dvomerge?)
• if the merge gets corrupted, we do not stop and this causes further problems
• we do not back up the minidvodbs often enough
• we do not back up the merge
• we merge the minidvodbs again, causing database chaos

The way addstar works now

Queueing

We have several scripts and survey tasks to queue up the camera stage and the staticsky stage for running addstar. New addRuns are queued and grouped by a minidvodb_group. This minidvodb_group is used to prevent multiple camRuns for the same exp_id from being queued. The first one in is the first one used.

Ingesting addstar runs

Initially, addRuns are queued with a minidvodb_group, but no minidvodb_name. When addstar is running, it looks for the current (active state) minidvodb for a given minidvodb_group, and ingests that add_id into that minidvodb. Once completed, the addRun is updated to reflect where the cam/staticsky was ingested.

Minidvodb Tasks

• Creation
  • When certain criteria are met, a new minidvodb is created. Currently a new minidvodb is created when the current active minidvodb has more than 500 add_ids or is older than 1 day. When a new minidvodb is created, the old active minidvodb is retired (set to a state of "to_be_merged".

• Verification and Merging
  • When the tasks detect a new minidvodb in a state of "to_be_merged", the following happens:
    • relphot -- run relphot on the minidvodb -- fail if it doesn't relphot EAM : this step is not needed
    • addstar -resort -- resort the minidvodb -- fail if it doesn't resort
    • verify that it is resorted -- fail if it isn't
    • dvomerge the minidvodb into the big dvodb -- if there is a failure here, it unfortunately blindly goes onto the next minidvodb to merge, which causes corruption.

MinidvodbCopy

There are a set of tasks and db tables to make backup copies of the minidvodbs. More details can be seen here (​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/MinidvodbCopy). MinidvodbCopy takes a minidvodb_group, host, and destination path to copy minidvodbs somewhere. I haven't run into problems with it yet. It has checks for diskspace - it doesn't transfer unless there is enough room.

Backups of the big dvodb

These are done by hand, and very sporadically. We often merge hundreds of minidvodbs before making a backup. We often lose track of where the backups go (or they get modified and thus can't be used). This is a horribly inefficient way to do this, and most recoveries take forever because we have to merge back together all the minidvodbs (this takes a month).

What I would like to change

I would like to split up the minidvodb tasks and tables to something like this:

The minidvodb tasks/tables/scripts

• minidvodbRun
• minidvodbProcessed
• minidvodbCopy

These would be mostly unchanged, with the exception that they are NOT merged into the big database. Once a minidvodb is relphot'd/resorted/verified, it is now ready for merging and copying a backup to ippmanoa. EAM : if we can make the ATRC machines available via rsync to the consortium, then maybe we should put them there (though for now, ippmanoa is fine

Mergedvodb tasks/tables/scripts

• mergedvodbRun
• mergedvodbProcessed
• mergedvodbCopy

These would be for tracking/queueing/backing up the big dvodbs. Some ideas:

• no queuing up a merge if there is a previously failed merge for a given big dvodb (this makes sense- allows us to fix it).
• mergedvodbCopy - my idea is after each successful merge, do an rsync backup of this. I don't know how long this takes, but we can investigate this
• we can easily create one-off smaller dvodbs if necessary this way, and track it and have it in the database (handy for roy and probably others)

I would also like to run some cron job that detects full disk space on ipp004/ipp005, stops the addstars if necessary, and emails me (or the group) to notify us.

How this would address the addstar problems

• the addstar machine (ipp004 and/or ipp005) fills up -- solved with a cron job
• there are nfs related issues (this may cause problems with dvomerge?) -- we will have a backup of the old dvodb to recover from
• if the merge gets corrupted, we do not stop and this causes further problems -- we will now stop (and we have a backup).
• we do not back up the minidvodbs often enough -- we will back up minidvodbs on ippb03
• we do not back up the merge -- we will back this up now
• we merge the minidvodbs again, causing database chaos -- this would be done as a new mergedvodbRun (instead of hacking at the minidvodb tables)

• what if we have some crazy corruption (like the wrong data in the db?) -- I don't have any good way for this, except to keep a spare older *good* dvodb to rebuild from.