IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 38340


Ignore:
Timestamp:
May 30, 2015, 7:59:48 PM (11 years ago)
Author:
eugene
Message:

more work on compression : cfitsio assumes gzip, not zlib (I have made uncompress general )

Location:
branches/eam_branches/ohana.20150429/src
Files:
17 edited

Legend:

Unmodified
Added
Removed

  • branches/eam_branches/ohana.20150429/src/libfits/extern/gzip.c

    r38339 r38340  
    22# include <gfitsio.h>
    33# include <zlib.h>
     4# define EXTRA_VERBOSE 1
    45
    56/** the two functions in this file re-implement the uncompress function of zlib. 
    … …  
    1011
    1112/* zlib.h -- interface of the 'zlib' general purpose compression library
    12   version 1.2.3, July 18th, 2005
    13 
    14   Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler
    15 
    16   This software is provided 'as-is', without any express or implied
    17   warranty.  In no event will the authors be held liable for any damages
    18   arising from the use of this software.
    19 
    20   Permission is granted to anyone to use this software for any purpose,
    21   including commercial applications, and to alter it and redistribute it
    22   freely, subject to the following restrictions:
    23 
    24   1. The origin of this software must not be misrepresented; you must not
    25      claim that you wrote the original software. If you use this software
    26      in a product, an acknowledgment in the product documentation would be
    27      appreciated but is not required.
    28   2. Altered source versions must be plainly marked as such, and must not be
    29      misrepresented as being the original software.
    30   3. This notice may not be removed or altered from any source distribution.
    31 
    32   Jean-loup Gailly        Mark Adler
    33   jloup@gzip.org          madler@alumni.caltech.edu
    34 
    35 
    36   The data format used by the zlib library is described by RFCs (Request for
    37   Comments) 1950 to 1952 in the files http://www.ietf.org/rfc/rfc1950.txt
    38   (zlib format), rfc1951.txt (deflate format) and rfc1952.txt (gzip format).
     13   version 1.2.3, July 18th, 2005
     14
     15   Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler
     16
     17   This software is provided 'as-is', without any express or implied
     18   warranty.  In no event will the authors be held liable for any damages
     19   arising from the use of this software.
     20
     21   Permission is granted to anyone to use this software for any purpose,
     22   including commercial applications, and to alter it and redistribute it
     23   freely, subject to the following restrictions:
     24
     25   1. The origin of this software must not be misrepresented; you must not
     26   claim that you wrote the original software. If you use this software
     27   in a product, an acknowledgment in the product documentation would be
     28   appreciated but is not required.
     29   2. Altered source versions must be plainly marked as such, and must not be
     30   misrepresented as being the original software.
     31   3. This notice may not be removed or altered from any source distribution.
     32
     33   Jean-loup Gailly        Mark Adler
     34   jloup@gzip.org          madler@alumni.caltech.edu
     35
     36
     37   The data format used by the zlib library is described by RFCs (Request for
     38   Comments) 1950 to 1952 in the files http://www.ietf.org/rfc/rfc1950.txt
     39   (zlib format), rfc1951.txt (deflate format) and rfc1952.txt (gzip format).
    3940*/
    4041
    4142/*
    42      The 'zlib' compression library provides in-memory compression and
     43  The 'zlib' compression library provides in-memory compression and
    4344  decompression functions, including integrity checks of the uncompressed
    4445  data.  This version of the library supports only one compression method
    … …  
    4647  stream interface.
    4748
    48      Compression can be done in a single step if the buffers are large
     49  Compression can be done in a single step if the buffers are large
    4950  enough (for example if an input file is mmap'ed), or can be done by
    5051  repeated calls of the compression function.  In the latter case, the
    … …  
    5253  (providing more output space) before each call.
    5354
    54      The compressed data format used by default by the in-memory functions is
     55  The compressed data format used by default by the in-memory functions is
    5556  the zlib format, which is a zlib wrapper documented in RFC 1950, wrapped
    5657  around a deflate stream, which is itself documented in RFC 1951.
    5758
    58      The library also supports reading and writing files in gzip (.gz) format
     59  The library also supports reading and writing files in gzip (.gz) format
    5960  with an interface similar to that of stdio using the functions that start
    6061  with "gz".  The gzip format is different from the zlib format.  gzip is a
    6162  gzip wrapper, documented in RFC 1952, wrapped around a deflate stream.
    6263
    63      This library can optionally read and write gzip streams in memory as well.
    64 
    65      The zlib format was designed to be compact and fast for use in memory
     64  This library can optionally read and write gzip streams in memory as well.
     65
     66  The zlib format was designed to be compact and fast for use in memory
    6667  and on communications channels.  The gzip format was designed for single-
    6768  file compression on file systems, has a larger header than zlib to maintain
    6869  directory information, and uses a different, slower check method than zlib.
    6970
    70      The library does not install any signal handler. The decoder checks
     71  The library does not install any signal handler. The decoder checks
    7172  the consistency of the compressed data, so the library should never
    7273  crash even in case of corrupted input.
    … …  
    116117}
    117118
    118 // XXX ??? is cfitsio writing data which is incompatible with zlib ???
     119# define ESCAPE(RET) { fprintf (stderr, "gzip error in %s @ %s:%d\n", __func__, __FILE__, __LINE__); return (RET); }
     120
     121// NOTE: CFITSIO uses gzip format, not just zlib
    119122int gfits_uncompress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen)
    120123{
    121     z_stream stream;
    122     int i, err;
    123 
    124     stream.next_in = (Bytef*)source;
    125     stream.avail_in = (uInt)sourceLen;
    126     /* Check for source > 64K on 16-bit machine: */
    127     if ((uLong)stream.avail_in != sourceLen) return Z_BUF_ERROR;
    128 
    129     stream.next_out = dest;
    130     stream.avail_out = (uInt)*destLen;
    131     if ((uLong)stream.avail_out != *destLen) return Z_BUF_ERROR;
    132 
    133     stream.zalloc = Z_NULL;
    134     stream.zfree  = Z_NULL;
    135     stream.opaque = Z_NULL;
    136 
    137     if (0) {
    138       fprintf (stderr, "inp unc: ");
    139       for (i = 0; i < sourceLen; i++) {
    140         fprintf (stderr, "0x%02hhx ", source[i]);
    141       }
    142       fprintf (stderr, "\n");
    143     }
    144 
    145     // err = inflateInit2(&stream, -15);
    146     err = inflateInit(&stream);
    147     if (0) fprintf (stderr, "inp buffers unc 0: %d => %d => %d\n", stream.avail_in, stream.avail_out, (int) stream.total_out);
    148 
    149     if (err != Z_OK) return err;
    150 
    151     err = inflate(&stream, Z_FINISH);
    152     if (0) fprintf (stderr, "out buffers unc 1: %d => %d => %d\n", stream.avail_in, stream.avail_out, (int) stream.total_out);
    153     if (err != Z_STREAM_END) {
    154         inflateEnd(&stream);
    155         if (err == Z_NEED_DICT || (err == Z_BUF_ERROR && stream.avail_in == 0)) return Z_DATA_ERROR;
    156         return err;
    157     }
    158     if (0) fprintf (stderr, "out buffers unc 2: %d => %d => %d\n", stream.avail_in, stream.avail_out, (int) stream.total_out);
    159 
    160     if (0) {
    161       fprintf (stderr, "out unc: ");
    162       for (i = 0; i < stream.total_out; i++) {
    163         fprintf (stderr, "0x%02hhx ", dest[i]);
    164       }
    165       fprintf (stderr, "\n");
    166     }
    167 
    168     assert (stream.total_out <= *destLen);
    169     *destLen = stream.total_out;
    170 
    171     err = inflateEnd(&stream);
    172     return err;
     124  z_stream stream;
     125  int i, err;
     126 
     127  stream.next_in = (Bytef*)source;
     128  stream.avail_in = (uInt)sourceLen;
     129
     130  /* Check for source > 64K on 16-bit machine: */
     131  if ((uLong)stream.avail_in != sourceLen) ESCAPE (Z_BUF_ERROR);
     132 
     133  stream.next_out = dest;
     134  stream.avail_out = (uInt)*destLen;
     135  if ((uLong)stream.avail_out != *destLen) ESCAPE (Z_BUF_ERROR);
     136 
     137  stream.zalloc = Z_NULL;
     138  stream.zfree  = Z_NULL;
     139  stream.opaque = Z_NULL;
     140 
     141  if (EXTRA_VERBOSE) {
     142    fprintf (stderr, "inp unc: ");
     143    for (i = 0; i < sourceLen; i++) {
     144      fprintf (stderr, "0x%02hhx ", source[i]);
     145    }
     146    fprintf (stderr, "\n");
     147  }
     148 
     149  // a value of 32 tells inflate to use the window size used in the compression AND to look for both zlib and gzip headers
     150  err = inflateInit2(&stream, 32);
     151  if (err != Z_OK) ESCAPE (err);
     152 
     153  err = inflate(&stream, Z_FINISH);
     154  if (err != Z_STREAM_END) {
     155    inflateEnd(&stream);
     156    if (err == Z_NEED_DICT || (err == Z_BUF_ERROR && stream.avail_in == 0)) ESCAPE (Z_DATA_ERROR);
     157    ESCAPE(err);
     158  }
     159 
     160  if (EXTRA_VERBOSE) {
     161    fprintf (stderr, "out unc: ");
     162    for (i = 0; i < stream.total_out; i++) {
     163      fprintf (stderr, "0x%02hhx ", dest[i]);
     164    }
     165    fprintf (stderr, "\n");
     166  }
     167
     168  assert (stream.total_out <= *destLen);
     169  *destLen = stream.total_out;
     170
     171  err = inflateEnd(&stream);
     172  return err;
    173173}
    174174
    … …  
    182182
    183183  /* Check for source > 64K on 16-bit machine: */
    184   if ((uLong)stream.avail_in != sourceLen) return Z_BUF_ERROR;
     184  if ((uLong)stream.avail_in != sourceLen) ESCAPE (Z_BUF_ERROR);
    185185
    186186  stream.next_out = dest;
    187187  stream.avail_out = (uInt)*destLen; // allocated space
    188   if ((uLong)stream.avail_out != *destLen) return Z_BUF_ERROR;
     188  if ((uLong)stream.avail_out != *destLen) ESCAPE (Z_BUF_ERROR);
    189189
    190190  stream.zalloc = Z_NULL;
    … …  
    192192  stream.opaque = Z_NULL;
    193193
    194   if (0) {
     194  if (EXTRA_VERBOSE) {
    195195    fprintf (stderr, "inp cmp: ");
    196196    for (i = 0; i < sourceLen; i++) {
    … …  
    202202  // the '1' is the compression level: make this a user argument
    203203  // NOTE: cfitsio uses a fixed value of 1
    204   err = deflateInit(&stream, 1);
     204  // I am using deflateInit2 because cfitsio expects gzip, not just zlib
     205  err = deflateInit2(&stream, 1, Z_DEFLATED, 31, 8, Z_DEFAULT_STRATEGY);
    205206  if (0) fprintf (stderr, "inp buffers cmp 0: %d => %d => %d\n", stream.avail_in, stream.avail_out, (int) stream.total_out);
    206207
    207   if (err != Z_OK) return err;
     208  if (err != Z_OK) ESCAPE(err);
    208209
    209210  // XXX this is written to do the compression in a single pass.  it could be re-done to have
    210211  // the compression occur in a series of steps
    211212  err = deflate(&stream, Z_FINISH);
    212   if (0) fprintf (stderr, "out buffers cmp 1: %d => %d => %d\n", stream.avail_in, stream.avail_out, (int) stream.total_out);
    213 
    214213  if (err != Z_STREAM_END) {
    215214    err = deflateEnd(&stream);
    216     return Z_BUF_ERROR;
    217   }
    218   if (0) fprintf (stderr, "out buffers cmp 2: %d => %d => %d\n", stream.avail_in, stream.avail_out, (int) stream.total_out);
    219 
    220   if (0) {
     215    ESCAPE (Z_BUF_ERROR);
     216  }
     217
     218  if (EXTRA_VERBOSE) {
    221219    fprintf (stderr, "out cmp: ");
    222220    for (i = 0; i < stream.total_out; i++) {

  • branches/eam_branches/ohana.20150429/src/libfits/include/gfitsio.h

    r38335 r38340  
    205205
    206206int     gfits_byteswap_zdata           PROTO((char *zdata, int Nzdata, int bitpix));
    207 int     gfits_extension_is_compressed  PROTO((Header *header));
     207int     gfits_extension_is_compressed_image  PROTO((Header *header));
     208int     gfits_extension_is_compressed_table  PROTO((Header *header));
    208209int     gfits_uncompressed_data_pixsize PROTO((char *cmptype, int out_bitpix, char **optname, char **optvalue, int Noptions));
    209210int     gfits_vartable_heap_pixsize    PROTO((char format));

  • branches/eam_branches/ohana.20150429/src/libfits/matrix/F_compress_M.c

    r38329 r38340  
    139139  int tile_pixsize = gfits_uncompressed_data_pixsize (zcmptype, header[0].bitpix, NULL, NULL, 0);
    140140
     141  int Nzdata_alloc = raw_pixsize*max_tile_size + 100;
    141142  ALLOCATE (raw,   char, raw_pixsize*max_tile_size);
    142   ALLOCATE (zdata, char, raw_pixsize*max_tile_size);
     143  ALLOCATE (zdata, char, Nzdata_alloc);
    143144
    144145  // init the otile[] counters
    … …  
    155156    if (!gfits_collect_data (matrix, raw, Nraw, raw_pixsize, otile, oblank, ztile, zblank, zscale, zzero)) ESCAPE(A);
    156157
     158    if (!strcasecmp(zcmptype, "GZIP_1")) {
     159      if (!gfits_byteswap_zdata (raw, Nraw, raw_pixsize)) ESCAPE(A);
     160    }
     161
    157162    // XXX the tile must not be > 2GB
    158163   
    159164    // optname, optvalue = NULL, Noptions = 0
    160165
    161     int Nzdata = raw_pixsize*max_tile_size; // available space, replaced with actual output size on compression
     166    int Nzdata = Nzdata_alloc; // available space, replaced with actual output size on compression
    162167    if (!gfits_compress_data (zdata, &Nzdata, zcmptype, NULL, NULL, 0, raw, Nraw, tile_pixsize)) ESCAPE(A);
    163168
    … …  
    298303        SETUP_INSIZE (int, 4);
    299304        break;
     305      case 8:
     306        SETUP_INSIZE (double, 8);
     307        break;
    300308      default:
    301309        abort();

  • branches/eam_branches/ohana.20150429/src/libfits/matrix/F_compress_data.c

    r38334 r38340  
    2727int gfits_compress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
    2828
     29# define ESCAPE(RET) { fprintf (stderr, "gzip error in %s @ %s:%d\n", __func__, __FILE__, __LINE__); return (RET); }
     30
    2931int gfits_compress_data (char *zdata, int *Nzdata, char *cmptype,
    3032                         char **optname, char **optvalue, int Nopt,
    … …  
    5355    status = gfits_compress ((Bytef *) zdata, &destLen, (Bytef *) rawdata, Nbytes);
    5456    *Nzdata = destLen;
    55     if (status != Z_OK) {
    56       fprintf (stderr, "error in compress (GZIP)\n");
    57       return (FALSE);
    58     }
     57    if (status != Z_OK) ESCAPE(FALSE);
    5958    return (TRUE);
    6059  }

  • branches/eam_branches/ohana.20150429/src/libfits/matrix/F_compress_utils.c

    r38339 r38340  
    131131}
    132132
    133 int gfits_extension_is_compressed (Header *header) {
     133int gfits_extension_is_compressed_image (Header *header) {
    134134
    135135    int has_extension, has_extname, has_zimage, zimage;
    … …  
    146146    }
    147147
     148    return (FALSE);
     149}
     150
     151int gfits_extension_is_compressed_table (Header *header) {
     152
     153    int has_ztable, ztable;
     154    has_ztable  = gfits_scan_alt (header, "ZTABLE", "%t", 1, &ztable);
     155    if (has_ztable && ztable) return (TRUE);
    148156    return (FALSE);
    149157}

  • branches/eam_branches/ohana.20150429/src/libfits/matrix/F_uncompress_M.c

    r38339 r38340  
    249249  // size of a pixel in the output from the decompression routine
    250250  odata_pixsize = gfits_uncompressed_data_pixsize (cmptype, header[0].bitpix, optname, optvalue, Noptions);
    251   ALLOCATE (out, char, odata_pixsize*max_tile_size);
     251
     252  int Nout_alloc = odata_pixsize*max_tile_size;
     253  ALLOCATE (out, char, Nout_alloc);
    252254
    253255  off_t row;
    … …  
    258260
    259261    // expected output size for this tile
    260     Nout = gfits_tile_size (matrix, otile, ztile);
     262    // Nout = gfits_tile_size (matrix, otile, ztile);
     263    Nout = Nout_alloc;
    261264
    262265    zdata = gfits_varlength_column_pointer (ftable, &zdef, row, &Nzdata);
    … …  
    273276    // XXX the tile must not be > 2GB
    274277    if (!gfits_uncompress_data ((char *)zdata, Nzdata, cmptype, optname, optvalue, Noptions, out, &Nout, odata_pixsize)) ESCAPE;
     278
     279    if (!strcasecmp(cmptype, "GZIP_1")) {
     280      if (!gfits_byteswap_zdata (out, Nout, odata_pixsize)) ESCAPE;
     281    }
    275282
    276283    // copy the uncompressed pixels into their correct locations           
    … …  
    387394        SETUP_INSIZE (int, 4);
    388395        break;
     396      case 8:
     397        SETUP_INSIZE (double, 8);
     398        break;
    389399      default:
    390400        abort();

  • branches/eam_branches/ohana.20150429/src/libfits/matrix/F_uncompress_data.c

    r38334 r38340  
    2626int gfits_uncompress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen);
    2727
     28# define ESCAPE(RET) { fprintf (stderr, "gzip error in %s @ %s:%d\n", __func__, __FILE__, __LINE__); return (RET); }
     29
    2830int gfits_uncompress_data (char *zdata, int Nzdata, char *cmptype, char **optname, char **optvalue, int Nopt, char *outdata, int *Nout, int out_pixsize) {
    2931
    … …  
    4951    // XXX shouldn't we validate the result : we think we know the size
    5052    status = gfits_uncompress ((Bytef *) outdata, &tNout, (Bytef *) zdata, Nzdata);
    51     if (status != Z_OK) {
    52       fprintf (stderr, "error in uncompress (GZIP)\n");
    53       return (FALSE);
    54     }
     53    if (status != Z_OK) ESCAPE(FALSE);
    5554    *Nout = tNout / out_pixsize;
    5655

  • branches/eam_branches/ohana.20150429/src/libfits/table/F_uncompress_T.c

    r38335 r38340  
    9898
    9999  // allocate the intermediate storage buffers
    100   ALLOCATE (raw,   char, max_width*ztilelen);
     100  int Nraw_alloc = max_width*ztilelen + 100;
     101  ALLOCATE (raw,   char, Nraw_alloc);
    101102  ALLOCATE (zdata, char, max_width*ztilelen);
    102103
    … …  
    120121   
    121122      int Nrows = (row == Ntile - 1) ? ztilelast : ztilelen;
    122       int Nraw = Nrows*fields[i].Nvalues; // number of pixels
    123 
     123      int Nraw = Nraw_alloc; // expected number of pixel: Nrows*fields[i].Nvalues
    124124      if (!gfits_uncompress_data (zdata, Nzdata, fields[i].zctype, NULL, NULL, 0, raw, &Nraw, fields[i].pixsize)) ESCAPE(A);
    125125

  • branches/eam_branches/ohana.20150429/src/libfits/test/compress.sh

    r38339 r38340  
     1
     2macro testall
     3
     4  foreach bitpix 8 16 32 -32 -64
     5    echo ===== bitpix = $bitpix =====
     6    test1 $bitpix
     7  end
     8end
    19
    210macro test1
    3  mcreate z 500 500
     11 if ($0 != 2)
     12   echo "USAGE: test_image (bitpix)"
     13   break
     14 end
     15
     16 mcreate z 8 8
    417 set x = xramp(z)
    5  wd x test.raw.fits -bitpix -32 -bzero 0 -bscale 1
    6  wd x test.cmp.fits -bitpix -32 -bzero 0 -bscale 1 -compress
     18 wd x test.raw.fits -bitpix $1 -bzero 0 -bscale 1
     19 wd x test.cmp.fits -bitpix $1 -bzero 0 -bscale 1 -compress
    720 rd b test.cmp.fits -x 0
    821 set d = x - b
    922 stat d
    10  exec fpack -g -S test.raw.fits > test.fpk.fits
    11  exec funpack -S test.cmp.fits > test.fun.fits
     23
     24 if ($bitpix == -64)
     25   echo "*** bitpix $bitpix fails fpack / funpack ***"
     26 else
     27   exec fpack -g -S test.raw.fits > test.fpk.fits
     28   echo "load fpack version"
     29   rd c test.fpk.fits -x 0
     30   set d = x - c
     31   stat d
     32 end
     33
     34 if ($bitpix == -64)
     35   echo "*** bitpix $bitpix fails fpack / funpack ***"
     36 else
     37   exec funpack -S test.cmp.fits > test.fun.fits
     38   rd e test.fun.fits
     39   set d = x - e
     40   stat d
     41 end
    1242end
    1343
    … …  
    2555macro test3
    2656
     57 $NPT = 10
     58 create x 0 $NPT
     59 set y = x^2
     60 create ID 1 {$NPT + 1} -int
     61
     62 write -fits test test.raw.tbl x y ID
     63 write -fits test test.cmp.tbl x y ID -compress-mode NONE
     64
     65 foreach f x y ID
     66   set $f\_raw = $f
     67   delete $f
     68 end
     69
     70 echo "===== raw ====="
     71 data test.raw.tbl
     72 read -fits test x y ID
     73
     74 foreach f x y ID
     75   set dv = $f - $f\_raw
     76   vstat dv
     77 end
     78 delete -q x y ID
     79
     80 echo "===== cmp ====="
     81 data test.cmp.tbl
     82 read -fits test x y ID
     83
     84 foreach f x y ID
     85   set dv = $f - $f\_raw
     86   vstat dv
     87 end
     88end
     89
     90macro test4
     91
    2792 create x 0 100
    2893 set y = x^2
     94 create ID 1 101 -int
    2995
    30  create ID 1 101 -int
    3196 write -fits test test.raw.tbl x y ID
     97 write -fits test test.cmp.tbl x y ID -compress
    3298
    33  write -fits test test.cmp.tbl x y ID -compress
     99 foreach f x y ID
     100   set $f\_raw = $f
     101   delete $f
     102 end
     103
     104 echo "===== raw ====="
     105 data test.raw.tbl
     106 read -fits test x y ID
     107
     108 foreach f x y ID
     109   set dv = $f - $f\_raw
     110   vstat dv
     111 end
     112 delete -q x y ID
     113
     114 echo "===== cmp ====="
     115 data test.cmp.tbl
     116 read -fits test x y ID
     117
     118 foreach f x y ID
     119   set dv = $f - $f\_raw
     120   vstat dv
     121 end
    34122end

  • branches/eam_branches/ohana.20150429/src/libfits/test/imagecomp.c

    r38335 r38340  
    33# include "tap_ohana.h"
    44
    5 int test_compress (char *zcmptype);
     5int test_compress (int bitpix, char *zcmptype);
    66
    77int main (int argc, char **argv) {
    … …  
    1111  diag ("libfits imagecomp.c tests");
    1212
    13   test_compress ("NONE");
    14   test_compress ("GZIP_1");
     13  test_compress (  8, "NONE");
     14  test_compress ( 16, "NONE");
     15  test_compress ( 32, "NONE");
     16  test_compress (-32, "NONE");
     17  test_compress (-64, "NONE");
     18
     19  test_compress (  8, "GZIP_1");
     20  test_compress ( 16, "GZIP_1");
     21  test_compress ( 32, "GZIP_1");
     22  test_compress (-32, "GZIP_1");
     23  test_compress (-64, "GZIP_1");
    1524
    1625  exit (0);
    … …  
    2029# define NY 100
    2130
    22 int test_compress (char *zcmptype) { // make a table, compress, uncompress, compare : use compression zcmptype
     31int test_compress (int bitpix, char *zcmptype) { // make a table, compress, uncompress, compare : use compression zcmptype
    2332
    2433  Header rawheader;
    … …  
    3140  ftable.header = &theader;
    3241
    33   diag ("--- starting test_compress with zcmptype %s ---", zcmptype);
     42  diag ("--- starting test_compress with bitpix %d, zcmptype %s ---", bitpix, zcmptype);
    3443  ok (gfits_init_header (&rawheader), "init'ed the raw header");
    3544  ok (gfits_init_matrix (&rawmatrix), "init'ed the raw matrix");
    … …  
    3847
    3948  /* assign the necessary internal values */
    40   rawheader.bitpix   = -32;
     49  rawheader.bitpix   = bitpix;
    4150  rawheader.Naxes = 2;
    4251  rawheader.Naxis[0] = NX;
    … …  
    4655  ok (gfits_create_header (&rawheader),          "created header");
    4756  ok (gfits_create_matrix (&rawheader, &rawmatrix), "created matrix");
    48 
    49   float *rawdata = (float *) rawmatrix.buffer;
     57 
     58  char   *rawdata_char   = (char   *) rawmatrix.buffer;
     59  short  *rawdata_short  = (short  *) rawmatrix.buffer;
     60  int    *rawdata_int    = (int    *) rawmatrix.buffer;
     61  float  *rawdata_float  = (float  *) rawmatrix.buffer;
     62  double *rawdata_double = (double *) rawmatrix.buffer;
     63 
    5064  int ix, iy;
    5165  for (ix = 0; ix < NX; ix++) {
    5266    for (iy = 0; iy < NY; iy++) {
    53       rawdata[ix + NX*iy] = ix * iy*iy;
     67      switch (bitpix) {
     68        case   8: rawdata_char  [ix + NX*iy] = ix + iy*iy; break;
     69        case  16: rawdata_short [ix + NX*iy] = ix + iy*iy; break;
     70        case  32: rawdata_int   [ix + NX*iy] = ix + iy*iy; break;
     71        case -32: rawdata_float [ix + NX*iy] = ix + iy*iy; break;
     72        case -64: rawdata_double[ix + NX*iy] = ix + iy*iy; break;
     73        default: myAbort ("bad bitpix value");
     74      }
    5475    }
    5576  }     
    … …  
    6485  int Nbad = 0;
    6586
    66   float *outdata = (float *) outmatrix.buffer;
     87  char   *outdata_char   = (char   *) outmatrix.buffer;
     88  short  *outdata_short  = (short  *) outmatrix.buffer;
     89  int    *outdata_int    = (int    *) outmatrix.buffer;
     90  float  *outdata_float  = (float  *) outmatrix.buffer;
     91  double *outdata_double = (double *) outmatrix.buffer;
     92
    6793  for (ix = 0; ix < NX; ix++) {
    6894    for (iy = 0; iy < NY; iy++) {
    69       if (rawdata[ix + NX*iy] != outdata[ix + NX*iy]) Nbad ++;
     95      switch (bitpix) {
     96        case   8: if (rawdata_char  [ix + NX*iy] != outdata_char  [ix + NX*iy]) Nbad ++; break;
     97        case  16: if (rawdata_short [ix + NX*iy] != outdata_short [ix + NX*iy]) Nbad ++; break;
     98        case  32: if (rawdata_int   [ix + NX*iy] != outdata_int   [ix + NX*iy]) Nbad ++; break;
     99        case -32: if (rawdata_float [ix + NX*iy] != outdata_float [ix + NX*iy]) Nbad ++; break;
     100        case -64: if (rawdata_double[ix + NX*iy] != outdata_double[ix + NX*iy]) Nbad ++; break;
     101        default: myAbort ("bad bitpix value");
     102      }
    70103    }
    71104  }     

  • branches/eam_branches/ohana.20150429/src/libfits/test/tablecomp.c

    r38335 r38340  
    8282  off_t Nrow;
    8383
    84   short  *SEQ_t = gfits_get_bintable_column_data (outheader, outtable, "SEQ", type, &Nrow, &Ncol); ok (!strcmp (type, "short"),  "read short  table column");
    85   int     *ID_t = gfits_get_bintable_column_data (outheader, outtable,  "ID", type, &Nrow, &Ncol); ok (!strcmp (type, "int"),    "read int    table column");
    86   float    *X_t = gfits_get_bintable_column_data (outheader, outtable,   "X", type, &Nrow, &Ncol); ok (!strcmp (type, "float"),  "read float  table column");
    87   double   *Y_t = gfits_get_bintable_column_data (outheader, outtable,   "Y", type, &Nrow, &Ncol); ok (!strcmp (type, "double"), "read double table column");
    88    
     84  short  *SEQ_t = gfits_get_bintable_column_data (outheader, outtable, "SEQ", type, &Nrow, &Ncol); ok (!strcmp (type, "short"),  "read short  table column"); ok (Nrow == Nval, "right number of rows"); ok (Ncol == 1, "right number of cols");
     85  int     *ID_t = gfits_get_bintable_column_data (outheader, outtable,  "ID", type, &Nrow, &Ncol); ok (!strcmp (type, "int"),    "read int    table column"); ok (Nrow == Nval, "right number of rows"); ok (Ncol == 1, "right number of cols");
     86  float    *X_t = gfits_get_bintable_column_data (outheader, outtable,   "X", type, &Nrow, &Ncol); ok (!strcmp (type, "float"),  "read float  table column"); ok (Nrow == Nval, "right number of rows"); ok (Ncol == 1, "right number of cols");
     87  double   *Y_t = gfits_get_bintable_column_data (outheader, outtable,   "Y", type, &Nrow, &Ncol); ok (!strcmp (type, "double"), "read double table column"); ok (Nrow == Nval, "right number of rows"); ok (Ncol == 1, "right number of cols");
     88
    8989  // count mismatched values
    9090  int Nseq = 0;

  • branches/eam_branches/ohana.20150429/src/opihi/cmd.data/rd.c

    r38338 r38340  
    8585      return (FALSE);
    8686    }
    87     if (gfits_extension_is_compressed (&buf[0].header)) {
     87    if (gfits_extension_is_compressed_image (&buf[0].header)) {
    8888        IsCompressed = TRUE;
    8989    }
    … …  
    118118      Nword = 1;
    119119      IsCompressed = FALSE;
    120       if (gfits_extension_is_compressed (&buf[0].header)) {
     120      if (gfits_extension_is_compressed_image (&buf[0].header)) {
    121121        if (!strcmp (CCDKeyword, "EXTNAME")) Nword = 1;
    122122        IsCompressed = TRUE;
    … …  
    194194
    195195    if (!gfits_fread_ftable_data (f, &ftable, FALSE)) { fprintf (stderr, "problem reading compressed image table data\n"); fclose (f); return FALSE; }
    196     fprintf (stderr, "rd 1: ");
    197     for (i = 0; i < 32; i++) {
    198       fprintf (stderr, "0x%0hhx ", ftable.buffer[i]);
    199     }
    200     fprintf (stderr, "\n");
    201  
    202196    if (!gfits_byteswap_varlength_column (&ftable, 1)) { fprintf (stderr, "problem doing byteswap on compressed image metadata column\n"); fclose (f); gfits_free_table (&ftable); return FALSE; }
    203     fprintf (stderr, "rd 2: ");
    204     for (i = 0; i < 32; i++) {
    205       fprintf (stderr, "0x%0hhx ", ftable.buffer[i]);
    206     }
    207     fprintf (stderr, "\n");
    208 
    209197    if (!gfits_uncompress_image (&buf[0].header, &buf[0].matrix, &ftable)) { fprintf (stderr, "problem uncompressing the image data\n"); fclose (f); gfits_free_table (&ftable); return FALSE; }
    210198

  • branches/eam_branches/ohana.20150429/src/opihi/cmd.data/read_vectors.c

    r38062 r38340  
    412412  table.header = &header;
    413413
     414  int IsCompressed = FALSE;
     415
    414416  /* load appropriate extension (if extname is a number, use count) */
    415417  if (Nextend > -1) {
    … …  
    430432    if (!gfits_load_header (f, &header)) ESCAPE ("error reading header for extension %d\n", Nextend);
    431433
    432     if (getSizes) {
     434    IsCompressed = gfits_extension_is_compressed_table (&header);
     435
     436    if (getSizes && !IsCompressed) {
    433437      read_table_sizes (&header);
    434438      if (CCDKeyword != NULL) free (CCDKeyword);
    435439      gfits_free_header (&header);
    436440      return TRUE;
     441    }
     442    if (IsCompressed) {
     443      if (getSizes) {
     444        gprint (GP_ERR, "%s[%s] is compressed: -sizes is invalid\n", filename, extname);
     445        gfits_free_header (&header);
     446        return FALSE;
     447      }
     448      if ((start > 0) || (Nrows > -1)) {
     449        gprint (GP_ERR, "%s[%s] is compressed: must read entire table\n", filename, extname);
     450        gfits_free_header (&header);
     451        return FALSE;
     452      }
    437453    }
    438454
    … …  
    483499      }
    484500
    485       if (getSizes) {
     501      IsCompressed = gfits_extension_is_compressed_table (&header);
     502
     503      if (getSizes && !IsCompressed) {
    486504        read_table_sizes (&header);
    487505        if (CCDKeyword != NULL) free (CCDKeyword);
    … …  
    489507        return TRUE;
    490508      }
     509      if (IsCompressed) {
     510        if (getSizes) {
     511          gprint (GP_ERR, "%s[%s] is compressed: -sizes is invalid\n", filename, extname);
     512          gfits_free_header (&header);
     513          return FALSE;
     514        }
     515        if ((start > 0) || (Nrows > -1)) {
     516          gprint (GP_ERR, "%s[%s] is compressed: must read entire table\n", filename, extname);
     517          gfits_free_header (&header);
     518          return FALSE;
     519        }
     520      }
    491521
    492522      if (Nrows == -1) {
    … …  
    503533
    504534      if (!gfits_fread_ftable_range (f, padIfShort, &table, start, Nrows)) ESCAPE ("error reading table for extension %d\n", Nextend);
    505 
    506       // if (!gfits_fread_ftable_data (f, &table, padIfShort)) ESCAPE ("error reading table for extension\n");
    507 
    508535      break;
    509536    }
    … …  
    518545  Binary = !strcmp (type, "BINTABLE");
    519546  Ny = header.Naxis[1];
     547
     548  Header *outheader = &header;
     549  FTable *outtable  = &table;
     550
     551  Header rawheader;
     552  FTable rawtable;
     553  if (IsCompressed) {
     554    rawtable.header = &rawheader;
     555    if (!gfits_uncompress_table (&table, &rawtable)) ESCAPE ("failed to uncompress table");
     556    outheader = &rawheader;
     557    outtable  = &rawtable;
     558    gfits_free_header (&header);
     559    gfits_free_table (&table);
     560    Ny = rawheader.Naxis[1];
     561  }
    520562
    521563  /* find columns which match requested vectors */
    … …  
    527569    Nval = 0;
    528570    if (Binary) {
    529       if (!gfits_get_bintable_column_type (&header, argv[i], type, &Nval)) ESCAPE ("requested field not found");
    530       if (!gfits_get_bintable_column (&header, &table, argv[i], &data)) ESCAPE ("error reading data from specified field");
     571      if (!gfits_get_bintable_column_type (outheader, argv[i], type, &Nval)) ESCAPE ("requested field not found");
     572      if (!gfits_get_bintable_column (outheader, outtable, argv[i], &data)) ESCAPE ("error reading data from specified field");
    531573    } else {
    532       if (!gfits_get_table_column_type (&header, argv[i], type)) ESCAPE ("requested field not found");
    533       if (!gfits_get_table_column (&header, &table, argv[i], &data)) ESCAPE ("error reading data from specified field");
     574      if (!gfits_get_table_column_type (outheader, argv[i], type)) ESCAPE ("requested field not found");
     575      if (!gfits_get_table_column (outheader, outtable, argv[i], &data)) ESCAPE ("error reading data from specified field");
    534576    }
    535577    if (Nval == 0) ESCAPE ("no data for field in table");
    … …  
    594636  }
    595637  if (CCDKeyword != NULL) free (CCDKeyword);
    596   gfits_free_table (&table);
    597   gfits_free_header (&header);
     638  gfits_free_table (outtable);
     639  gfits_free_header (outheader);
    598640  return (TRUE);
    599641}

  • branches/eam_branches/ohana.20150429/src/opihi/cmd.data/wd.c

    r38338 r38340  
    205205    ftable.header = &theader;
    206206
    207     int i;
    208207    gfits_compress_image (&temp_header, &temp_matrix, &ftable, NULL, CompressMode);
    209     fprintf (stderr, "wd 1: ");
    210     for (i = 0; i < 32; i++) {
    211       fprintf (stderr, "0x%0hhx ", ftable.buffer[i]);
    212     }
    213     fprintf (stderr, "\n");
    214 
    215208    gfits_byteswap_varlength_column (&ftable, 1);
    216     fprintf (stderr, "wd 2: ");
    217     for (i = 0; i < 32; i++) {
    218       fprintf (stderr, "0x%0hhx ", ftable.buffer[i]);
    219     }
    220     fprintf (stderr, "\n");
    221    
    222209    gfits_fwrite_Theader (f, &theader);
    223210    gfits_fwrite_table  (f, &ftable);

  • branches/eam_branches/ohana.20150429/src/opihi/cmd.data/write_vectors.c

    r38330 r38340  
    6363  }
    6464
    65   int compress = FALSE;
    66   if ((N = get_argument (argc, argv, "-compress"))) {
    67     remove_argument (N, &argc, argv);
    68     compress = TRUE;
     65  char *compress = NULL;
     66  if ((N = get_argument (argc, argv, "-compress-mode"))) {
     67    remove_argument (N, &argc, argv);
     68    compress = strcreate(argv[N]);
     69    remove_argument (N, &argc, argv);
     70  } else if ((N = get_argument (argc, argv, "-compress"))) {
     71    remove_argument (N, &argc, argv);
     72    compress = strcreate("GZIP_1");
    6973    if (!FITS) {
    7074      fprintf (stderr, "NOTE: write_vectors -compress has no effect on non-FITS\n");

  • branches/eam_branches/ohana.20150429/src/opihi/include/dvomath.h

    r38330 r38340  
    167167
    168168/* vector IO functions */
    169 int           WriteVectorTableFITS  PROTO((char *filename, char *extname, Vector **vec, int Nvec, int append, int compress, char *format));
     169int           WriteVectorTableFITS  PROTO((char *filename, char *extname, Vector **vec, int Nvec, int append, char *compress, char *format));
    170170Vector      **ReadVectorTableFITS   PROTO((char *filename, char *extname, int *Nvec));
    171171

  • branches/eam_branches/ohana.20150429/src/opihi/lib.shell/VectorIO.c

    r38330 r38340  
    7272 
    7373// write a set of vectors to a FITS file (vectors names become fits column names)
    74 int WriteVectorTableFITS (char *filename, char *extname, Vector **vec, int Nvec, int append, int compress, char *format) {
     74int WriteVectorTableFITS (char *filename, char *extname, Vector **vec, int Nvec, int append, char *compress, char *format) {
    7575 
    7676  Header rawheader;
    … …  
    106106  if (compress) {
    107107    cmptable.header = &cmpheader;
    108     if (!gfits_compress_table (&rawtable, &cmptable, 10, "GZIP_1")) goto escape;
     108    if (!gfits_compress_table (&rawtable, &cmptable, 10, compress)) goto escape;
    109109    outtable = &cmptable;
    110110    outheader = &cmpheader;
Note: See TracChangeset for help on using the changeset viewer.