Ticket #119: metadata.txt

The idea of the psMetadata structure is to enable two competing needs
of the metadata system.  The purpose of the psHash in psMetadata
is to allow quick access to the string-based keys.  The purpose of the
psList in psMetadata is to maintain a specific order of key/value
pairs.  Partly, this is conceived to allows the psMetadata structure
to carry the data from a FITS header in such a way that the act of
reading in and writing out a single FITS header does not change the
header.  

Some background explanation of FITS headers may help clarify this (and
may help with the issues in bug #120).  A FITS header is an ASCII
segment which preceeds each data segment in a FITS file.  The FITS
header components have a required sequence for some keywords, and a
set of rules governing the format.  An example of a few lines are:

SIMPLE  =                    T / file does conform to FITS standard
BLANK   =               -32768 / Value used for NULL pixels
DATE-OBS= '2004-06-16'         / Observation start date  (yyyy-mm-dd)
COMMENT  ---------------- DATA BLOCK ---------------------------------
COMMENT  --------------- PARAMETERS FOR SUPRIME_CAM ------------------
EXPTIME  = 1.05                / exposure time for image
COMMENT  ---------------- FOO ----------------------------------------

the keys in FITS headers are always <= 8 upper-case characters, and
are usually followed by the = sign (the exception are the comment type
of keys, which have somewhat different rules in general).  The key is
followed (after the =) by the value, followed in turn by an optional
comment, separated by ' / '.  The data values may be integers, floats,
strings, and booleans (I note that we don't have boolean types in the
psMetadataType enum, do we?).  Keep in mind that the CFITSIO tools
provide the mechanisms to read in the FITS header blocks correctly
from a file and to extract the key/value/comments with correct
interpretation of the formats.  

If we require the ability to reconstruct the original order of the
metadata items in a FITS header (and we do require this), then I don't
think we can drop the psList. 

 However, in looking at the needs, it seems like we may not have
defined the interaction between the hash and the ITEM_SET list quite
right.  For the hash, I claim we want only one entry per hash key.
Unique items are no problem, the bucket.data just points to the
corresponding psMetadataItem entry (and both have the same value for
'key').  For the non-unique entries, we can't have multiple hash
entries with the same name.  Also, we don't care about getting to a
specific non-unique entry fast by the hash.  Instead, there should be
a single hash entry for each non-unique key which points at a
psMetadataItem container with a type of 'ITEM_SET'.  the list in this
set would then point at the collection of all non-unique entries.  On
the other side, the psList entries should each point at the individual
psMetadataItem entries, whether unique or not.  I have drawn a picture
to illustrate this.  (attached).  

If I want add a psMetadataItem, I should do the following things:

look for the key in the psHash.  

  if it exists, does it point to an entry with type = ITEM_SET?  
 
    If YES, I can add it to the end of that psList (ie, being a little
    sloppy, metadata->hash->bucket->data->items) as well as to the end
    of the psMetadata list (metadata->list).

    If NO, this is an error (attempt to repeat a UNIQUE entry).

  if such an entry does not exist, add it to metadata->hash and
  metadata->list.  

  (note that this still allows me to add a psMetadataItem of type
  ITEM_SET which should have a NULL data value.  Also note that if it
  should be an error to attempt to add a psMetadataItem of type
  ITEM_SET which already exists).

To delete a unique entry by index, I can get to it from
metadata->list, which gets me a unique psMetadataItem, delete the
entry from the list, and then also delete the entry with that name
from the metadata->hash.  

To delete a unique entry by name is also trivial: I get to it from the
metadata->hash, then I need to find the entry in metadata->list with
the same ID (not fast, requires looping over almost all entries).  To
delete a n

To delete a non-unique entry, the SDRS says you delete all non-unique
entries (not sure I agree with this!).  To do this by index, I would
get to the item from metadata->list.  Then I would get the entry point
from metadata->hash with the same name and delete everything in that
container.  

To delete a non-unique entry by name, you need to get to the
containing entry, then loop over the metadata->list and delete all
entries by their ID, then delete the container in metadata->hash.