Index: /branches/rel-1-0/PS-IPP-Config/.cvsignore =================================================================== --- /branches/rel-1-0/PS-IPP-Config/.cvsignore (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/.cvsignore (revision 11645) @@ -0,0 +1,3 @@ +Build +_build +blib Index: /branches/rel-1-0/PS-IPP-Config/Build.PL =================================================================== --- /branches/rel-1-0/PS-IPP-Config/Build.PL (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/Build.PL (revision 11645) @@ -0,0 +1,21 @@ +use Module::Build; +# See perldoc Module::Build for details of how this works + +Module::Build->new( + module_name => 'PS::IPP::Config', + dist_version => '0.01', + author => 'Paul Price and Joshua Hoblitt ', + license => 'gpl', + create_makefile_pl => 'passthrough', + requires => { + 'Carp' => 0, + 'Class::Accessor::Fast' => '0.19', + 'PS::IPP::Metadata::Config' => '0.07', + 'Statistics::Descriptive' => '2.6', + 'URI' => 0, + }, + recommends => { + 'Test::Distribution' => '1.22', + }, + +)->create_build_script; Index: /branches/rel-1-0/PS-IPP-Config/LICENSE =================================================================== --- /branches/rel-1-0/PS-IPP-Config/LICENSE (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/LICENSE (revision 11645) @@ -0,0 +1,260 @@ +The General Public License (GPL) +Version 2, June 1991 + +Copyright (C) 1989, 1991 Free Software Foundation, Inc. 675 Mass Ave, +Cambridge, MA 02139, USA. Everyone is permitted to copy and distribute +verbatim copies of this license document, but changing it is not allowed. + +Preamble + +The licenses for most software are designed to take away your freedom to share +and change it. By contrast, the GNU General Public License is intended to +guarantee your freedom to share and change free software--to make sure the +software is free for all its users. This General Public License applies to most of +the Free Software Foundation's software and to any other program whose +authors commit to using it. (Some other Free Software Foundation software is +covered by the GNU Library General Public License instead.) You can apply it to +your programs, too. + +When we speak of free software, we are referring to freedom, not price. Our +General Public Licenses are designed to make sure that you have the freedom +to distribute copies of free software (and charge for this service if you wish), that +you receive source code or can get it if you want it, that you can change the +software or use pieces of it in new free programs; and that you know you can do +these things. + +To protect your rights, we need to make restrictions that forbid anyone to deny +you these rights or to ask you to surrender the rights. These restrictions +translate to certain responsibilities for you if you distribute copies of the +software, or if you modify it. + +For example, if you distribute copies of such a program, whether gratis or for a +fee, you must give the recipients all the rights that you have. You must make +sure that they, too, receive or can get the source code. And you must show +them these terms so they know their rights. + +We protect your rights with two steps: (1) copyright the software, and (2) offer +you this license which gives you legal permission to copy, distribute and/or +modify the software. + +Also, for each author's protection and ours, we want to make certain that +everyone understands that there is no warranty for this free software. If the +software is modified by someone else and passed on, we want its recipients to +know that what they have is not the original, so that any problems introduced by +others will not reflect on the original authors' reputations. + +Finally, any free program is threatened constantly by software patents. We wish +to avoid the danger that redistributors of a free program will individually obtain +patent licenses, in effect making the program proprietary. To prevent this, we +have made it clear that any patent must be licensed for everyone's free use or +not licensed at all. + +The precise terms and conditions for copying, distribution and modification +follow. + +GNU GENERAL PUBLIC LICENSE +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND +MODIFICATION + +0. This License applies to any program or other work which contains a notice +placed by the copyright holder saying it may be distributed under the terms of +this General Public License. The "Program", below, refers to any such program +or work, and a "work based on the Program" means either the Program or any +derivative work under copyright law: that is to say, a work containing the +Program or a portion of it, either verbatim or with modifications and/or translated +into another language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not covered by +this License; they are outside its scope. The act of running the Program is not +restricted, and the output from the Program is covered only if its contents +constitute a work based on the Program (independent of having been made by +running the Program). Whether that is true depends on what the Program does. + +1. You may copy and distribute verbatim copies of the Program's source code as +you receive it, in any medium, provided that you conspicuously and appropriately +publish on each copy an appropriate copyright notice and disclaimer of warranty; +keep intact all the notices that refer to this License and to the absence of any +warranty; and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and you may at +your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Program or any portion of it, thus +forming a work based on the Program, and copy and distribute such +modifications or work under the terms of Section 1 above, provided that you also +meet all of these conditions: + +a) You must cause the modified files to carry prominent notices stating that you +changed the files and the date of any change. + +b) You must cause any work that you distribute or publish, that in whole or in +part contains or is derived from the Program or any part thereof, to be licensed +as a whole at no charge to all third parties under the terms of this License. + +c) If the modified program normally reads commands interactively when run, you +must cause it, when started running for such interactive use in the most ordinary +way, to print or display an announcement including an appropriate copyright +notice and a notice that there is no warranty (or else, saying that you provide a +warranty) and that users may redistribute the program under these conditions, +and telling the user how to view a copy of this License. (Exception: if the +Program itself is interactive but does not normally print such an announcement, +your work based on the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If identifiable +sections of that work are not derived from the Program, and can be reasonably +considered independent and separate works in themselves, then this License, +and its terms, do not apply to those sections when you distribute them as +separate works. But when you distribute the same sections as part of a whole +which is a work based on the Program, the distribution of the whole must be on +the terms of this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to +work written entirely by you; rather, the intent is to exercise the right to control +the distribution of derivative or collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program with the +Program (or with a work based on the Program) on a volume of a storage or +distribution medium does not bring the other work under the scope of this +License. + +3. You may copy and distribute the Program (or a work based on it, under +Section 2) in object code or executable form under the terms of Sections 1 and 2 +above provided that you also do one of the following: + +a) Accompany it with the complete corresponding machine-readable source +code, which must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange; or, + +b) Accompany it with a written offer, valid for at least three years, to give any +third party, for a charge no more than your cost of physically performing source +distribution, a complete machine-readable copy of the corresponding source +code, to be distributed under the terms of Sections 1 and 2 above on a medium +customarily used for software interchange; or, + +c) Accompany it with the information you received as to the offer to distribute +corresponding source code. (This alternative is allowed only for noncommercial +distribution and only if you received the program in object code or executable +form with such an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for making +modifications to it. For an executable work, complete source code means all the +source code for all modules it contains, plus any associated interface definition +files, plus the scripts used to control compilation and installation of the +executable. However, as a special exception, the source code distributed need +not include anything that is normally distributed (in either source or binary form) +with the major components (compiler, kernel, and so on) of the operating system +on which the executable runs, unless that component itself accompanies the +executable. + +If distribution of executable or object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the source +code from the same place counts as distribution of the source code, even though +third parties are not compelled to copy the source along with the object code. + +4. You may not copy, modify, sublicense, or distribute the Program except as +expressly provided under this License. Any attempt otherwise to copy, modify, +sublicense or distribute the Program is void, and will automatically terminate +your rights under this License. However, parties who have received copies, or +rights, from you under this License will not have their licenses terminated so long +as such parties remain in full compliance. + +5. You are not required to accept this License, since you have not signed it. +However, nothing else grants you permission to modify or distribute the Program +or its derivative works. These actions are prohibited by law if you do not accept +this License. Therefore, by modifying or distributing the Program (or any work +based on the Program), you indicate your acceptance of this License to do so, +and all its terms and conditions for copying, distributing or modifying the +Program or works based on it. + +6. Each time you redistribute the Program (or any work based on the Program), +the recipient automatically receives a license from the original licensor to copy, +distribute or modify the Program subject to these terms and conditions. You +may not impose any further restrictions on the recipients' exercise of the rights +granted herein. You are not responsible for enforcing compliance by third parties +to this License. + +7. If, as a consequence of a court judgment or allegation of patent infringement +or for any other reason (not limited to patent issues), conditions are imposed on +you (whether by court order, agreement or otherwise) that contradict the +conditions of this License, they do not excuse you from the conditions of this +License. If you cannot distribute so as to satisfy simultaneously your obligations +under this License and any other pertinent obligations, then as a consequence +you may not distribute the Program at all. For example, if a patent license would +not permit royalty-free redistribution of the Program by all those who receive +copies directly or indirectly through you, then the only way you could satisfy +both it and this License would be to refrain entirely from distribution of the +Program. + +If any portion of this section is held invalid or unenforceable under any particular +circumstance, the balance of the section is intended to apply and the section as +a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other +property right claims or to contest validity of any such claims; this section has +the sole purpose of protecting the integrity of the free software distribution +system, which is implemented by public license practices. Many people have +made generous contributions to the wide range of software distributed through +that system in reliance on consistent application of that system; it is up to the +author/donor to decide if he or she is willing to distribute software through any +other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a +consequence of the rest of this License. + +8. If the distribution and/or use of the Program is restricted in certain countries +either by patents or by copyrighted interfaces, the original copyright holder who +places the Program under this License may add an explicit geographical +distribution limitation excluding those countries, so that distribution is permitted +only in or among countries not thus excluded. In such case, this License +incorporates the limitation as if written in the body of this License. + +9. The Free Software Foundation may publish revised and/or new versions of the +General Public License from time to time. Such new versions will be similar in +spirit to the present version, but may differ in detail to address new problems or +concerns. + +Each version is given a distinguishing version number. If the Program specifies a +version number of this License which applies to it and "any later version", you +have the option of following the terms and conditions either of that version or of +any later version published by the Free Software Foundation. If the Program does +not specify a version number of this License, you may choose any version ever +published by the Free Software Foundation. + +10. If you wish to incorporate parts of the Program into other free programs +whose distribution conditions are different, write to the author to ask for +permission. For software which is copyrighted by the Free Software Foundation, +write to the Free Software Foundation; we sometimes make exceptions for this. +Our decision will be guided by the two goals of preserving the free status of all +derivatives of our free software and of promoting the sharing and reuse of +software generally. + +NO WARRANTY + +11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS +NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE +COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM +"AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR +IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE +ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, +YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED +TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY +WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS +PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM +(INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY +OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS +BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS Index: /branches/rel-1-0/PS-IPP-Config/MANIFEST =================================================================== --- /branches/rel-1-0/PS-IPP-Config/MANIFEST (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/MANIFEST (revision 11645) @@ -0,0 +1,8 @@ +Build.PL +LICENSE +Makefile.PL +lib/PS/IPP/Config.pm +lib/PS/IPP/Metadata/List.pm +lib/PS/IPP/Metadata/Stats.pm +t/00_distribution.t +MANIFEST Index: /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Config.pm =================================================================== --- /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Config.pm (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Config.pm (revision 11645) @@ -0,0 +1,455 @@ +# Copyright (c) 2006 Paul Price, Joshua Hoblitt +# +# $Id: Config.pm,v 1.27 2007-02-01 02:22:31 eugene Exp $ + +package PS::IPP::Config; + +use strict; +use warnings FATAL => qw( all ); + +our $VERSION = '0.01'; + +use Carp qw( carp croak ); +use File::Spec; +use PS::IPP::Metadata::Config 0.07; +use Getopt::Long qw( GetOptions :config gnu_getopt pass_through ); # Set pass_through so we don't kill @ARGV +use URI; + +use base qw( Class::Accessor::Fast Exporter ); +__PACKAGE__->mk_accessors( qw( path ) ); + +our @EXPORT_OK = qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR +); + +our $PS_EXIT_SUCCESS = 0; +our $PS_EXIT_UNKNOWN_ERROR = 1; +our $PS_EXIT_SYS_ERROR = 2; +our $PS_EXIT_CONFIG_ERROR = 3; +our $PS_EXIT_PROG_ERROR = 4; +our $PS_EXIT_DATA_ERROR = 5; +our $PS_EXIT_TIMEOUT_ERROR = 6; + +our $parser = PS::IPP::Metadata::Config->new; # Metadata parser +$parser->overwrite(0); # Turn off overwrite + +#$::RD_TRACE = 1; +#$::RD_HINT = 1; +#use Data::Dumper; + +# Given a parsed metadata, parse it into an array of hashes +sub new { + my $class = shift; # Class name + my $camera = shift; # Camera name + + unless (defined $class) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + # Get location of ipprc file + my $name; # Name of ipprc file + GetOptions( 'site=s' => \$name ); + $name = $ENV{PS_SITE} if not defined $name; + $name = $ENV{HOME} . '/.ipprc' if not defined $name; + + open my $file, $name or croak "Unable to open ipprc file $name: $!"; + my @contents = <$file>; # Contents of the ipprc file + close $file; + + my $mdc = $parser->parse( join '', @contents); # The parsed metadata config + + my $path = _interpolate_env(_mdLookupStr($mdc, 'PATH')); # The path + croak "PATH is not set in $name\n" unless defined $path; + + # Interpolate environment variables in the work directory + + my $self = { + path => $path, # Path + camera => undef, # Camera configuration + _ipprc => $mdc # The parsed configuration + }; + bless $self, $class; + + $self->define_camera($camera) if defined $camera; + + return $self; +} + +# Define a camera to use +sub define_camera +{ + my $self = shift; # Configuration object + my $name = shift; # Camera name + + unless (defined $self and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + my $camera_list = _mdLookupMD($self->{_ipprc}, 'CAMERAS'); # List of cameras + my $filename = _mdLookupStr($camera_list, $name); # Filename of camera configuration + croak "Unable to find configuration file for camera $name\n" + if not defined $filename; + + my @path = split /:/, $self->{path}; # List of paths to try + my $found = 0; # Found it? + my $realfile; + my $tries = 0; + while (not $found) { + ## try a second time + foreach my $path (@path) { + $realfile = File::Spec->rel2abs( $filename, $path ); + if ($tries) { + print "re-trying ($tries): filename: $filename, path: $path -> "; + print "test: $realfile\n"; + } + if (-f $realfile) { + $found = 1; + last; + } + } + if (not $found) { + $tries ++; + if ($tries > 4) { + croak "Unable to find camera configuration file $filename\n" if not $found; + } + ## try again after a moment? + system ("df"); + select(undef, undef, undef, 0.25); + } + } + + # Read the file + open my $file, $realfile or croak "Unable to open camera configuration file $realfile: $!"; + my @contents = <$file>; + close $file; + $self->{camera} = $parser->parse( join '', @contents); # The parsed metadata config + + # XXX why isn't just $self being returned here? -JH + return defined $self->{camera}; +} + + +# select a datapath from config.site +sub datapath +{ + my $self = shift; # Configuration object + my $name = shift; # datapath name + + unless (defined $self and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + my $path_list = _mdLookupMD($self->{_ipprc}, 'DATAPATH'); # List of paths + my $pathname = _mdLookupStr($path_list, $name); # Path of interest + croak "Unable to find camera configuration file $pathname\n" if not defined $pathname; + + return $pathname; +} + +sub convert_filename_absolute +{ + my $self = shift; # Configuration object + my $name = shift; # raw name + + unless (defined $self and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + my $uri = URI->new($name) or die "Unable to parse URI: $name\n"; # URI + my $scheme = $uri->scheme(); # Scheme, e.g., file, path + if (lc($scheme) eq 'file') { + my $host = $uri->host(); + my $path; + if (defined $host) { + # The user has file://some/directory/ but *probably* wants + # /some/directory/ i.e., the URI is technically wrong + # (should be one or three leading slashes, not two). We + # choose to fix it and give the user what we think they + # want, rather than what they asked for. + $path = File::Spec->catfile( undef, $host, $uri->path() ); + } else { + $path = $uri->path(); + } + + return File::Spec->canonpath( $path ); + } + + if (lc($scheme) eq 'path') { + my $authority = $uri->authority(); + my $path = $self->datapath( $authority ); + my @segments = $uri->path_segments(); + return File::Spec->catfile( $path, @segments ); + } + + return $name; +} + +sub convert_filename_relative +{ + my $self = shift; # Configuration object + my $name = shift; # raw name + + unless (defined $self and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + $name = File::Spec->canonpath( $name); # Clean up + my ($vol, $dir, $file) = File::Spec->splitpath( $name ); + my @dirs = File::Spec->splitdir( $dir ); + pop @dirs; # Get rid of filename + + my $path_list = _mdLookupMD($self->{_ipprc}, 'DATAPATH'); # List of paths + my $best_path; + my $best_name; + my $best_score; + CFR_PATH_CHECK: foreach my $path_item (@$path_list) { + my $path_name = $path_item->{name}; # Name of the path + my $path = File::Spec->canonpath( $path_item->{value} ); # The path + my @path_dirs = File::Spec->splitdir( $path ); + + # Check if the path is suitable + next if scalar @path_dirs > scalar @dirs; + for (my $i = 0; $i < scalar @path_dirs; $i++) { + next CFR_PATH_CHECK if $path_dirs[$i] ne $dirs[$i]; + } + # It is suitable; see if it is 'best' (minimizes the number of directories) + my $score = scalar @path_dirs; + if (not defined $best_score or $score > $best_score) { + $best_path = $path; + $best_score = $score; + $best_name = $path_name; + } + } + my $uri = URI->new(); # URI to return + if (defined $best_score) { + my $rel = File::Spec->abs2rel( $name, $best_path ); + my @segments = File::Spec->splitdir( $rel ); + $uri->scheme( 'path' ); + $uri->authority( $best_name ); + $uri->path_segments( @segments ); + } else { + $uri->scheme( 'file' ); + $uri->path( $name ); + } + + return $uri->as_string(); +} + +# Return a rejection limit from the camera configuration +sub rejection +{ + my $self = shift; # Configuration object + my $name = shift; # Name of limit + my $type = lc(shift); # Type of frame + my $filter = shift; # Filter name, for extra qualification; optional + + unless (defined $self and defined $name and defined $type) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + my $camera = $self->{camera}; # Camera configuration + unless (defined $camera) { + carp "Camera has not yet been defined.\n"; + return undef; + } + + my $rejection = _mdLookupMD($camera, 'REJECTION'); # Rejection list + unless (defined $rejection) { + carp "Can't find REJECTION list in camera configuration.\n"; + return undef; + } + + TYPES: foreach my $item (@$rejection) { + if (lc($item->{name}) eq $type) { + croak "$name within REJECTION is not of type METADATA" unless $item->{class} eq "metadata"; + my $limits = $item->{value}; # List of rejection limits + + # Check the FILTER first + foreach my $limit (@$limits) { + if ($limit->{name} eq 'FILTER') { + if ($limit->{value} eq '*' or + (defined $filter and + $limit->{value} eq $filter)) { + last; + } + next TYPES; # Doesn't match --- look at the next one + } + } + + foreach my $limit (@$limits) { + return $limit->{value} if $limit->{name} eq $name; + } + carp "Unable to find limit $name in REJECTION list for $type.\n"; + return undef; + } + } + + if (not defined $filter) { + carp "Unable to find type $type with no FILTER in REJECTION list.\n"; + } else { + carp "Unable to find type $type with FILTER $filter in REJECTION list.\n"; + } + return undef; +} + +# Return a filename from the camera configuration FILERULES +sub filename +{ + my $self = shift; # Configuration object + my $name = shift; # Name of the file + my $output = shift; # For replacing {OUTPUT} + my $component = shift; # For replacing {CHIP.NAME} and {CELL.NAME} + + unless (defined $self and defined $name and defined $output) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + my $camera = $self->{camera}; # Camera configuration + unless (defined $camera) { + carp "Camera has not yet been defined.\n"; + return undef; + } + + my $filerules = _mdLookupMD($camera, 'FILERULES'); # File rules + unless (defined $filerules) { + carp "Can't find FILERULES list in camera configuration.\n"; + return undef; + } + + my $file = _mdLookupMDwithRedirect($filerules, $name); # The file of interest + unless (defined $file) { + carp "Can't find $name within FILERULES in camera configuration.\n"; + return undef; + } + + my $filename = _mdLookupStr($file, "FILENAME.RULE"); # File name + unless (defined $filename) { + carp "Can't find FILENAME.RULE for $name within FILERULES in camera configuration.\n"; + return undef; + } + + $filename =~ s/\{OUTPUT\}/$output/; + if ($filename =~ /\{CHIP\.NAME\}/) { + unless (defined $component) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + $filename =~ s/\{CHIP\.NAME\}/$component/; + $filename =~ s/\{CELL\.NAME\}/$component/; + } + + return $filename; +} + + +# Interpolate environment variables in a directory (or colon-delimited list of directories) +sub _interpolate_env +{ + my $dir = shift; # Directory of interest + + return undef unless defined $dir; + + while ($dir =~ /\$\{?(\w*)[\}\/:]?/) { + my $name = $1; # Environment variable name + my $value = $ENV{$name}; # Environment variable value +# print "Replacing $name with $value.\n"; + $dir =~ s/\$\{?$name\}?/$value\//; + } + return $dir; +} + +# Lookup the metadata, checking the type is STR +sub _mdLookupStr +{ + my $mdc = shift; # Metadata config to look up + my $name = shift; # Name of item to look up + + unless (defined $mdc and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + # Iterate through the array of hashes + foreach my $item (@$mdc) { + if ($item->{name} eq $name) { + carp "$name within metadata is not of type STR.\n" unless $item->{type} eq "STR"; + return $item->{value}; + } + } + + carp "Unable to find $name within metadata.\n"; + return undef; +} + +# Lookup the metadata, checking the type is MD +sub _mdLookupMD +{ + my $mdc = shift; # Metadata config to look up + my $name = shift; # Name of item to look ip + + unless (defined $mdc and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + # Iterate through the array of hashes + foreach my $item (@$mdc) { + if ($item->{name} eq $name) { + carp "$name within metadata is not of type METADATA.\n" unless $item->{class} eq "metadata"; + return $item->{value}; + } + } + + carp "Unable to find $name within metadata.\n"; + return undef; +} + + +# Lookup the metadata, checking the type is MD; may be redirection +sub _mdLookupMDwithRedirect +{ + my $mdc = shift; # Metadata config to look up + my $name = shift; # Name of item to look ip + + unless (defined $mdc and defined $name) { + carp "Programming error"; + exit($PS_EXIT_PROG_ERROR); + } + + # Iterate through the array of hashes + foreach my $item (@$mdc) { + if ($item->{name} eq $name) { + if ($item->{class} eq "metadata") { + return $item->{value}; + } + if ($item->{class} eq "scalar" and $item->{type} eq "STR") { + # Redirect + return _mdLookupMDwithRedirect($mdc, $item->{value}); + } + carp "$name within metadata is not of type METADATA or STR.\n"; + return undef; + } + } + + carp "Unable to find $name within metadata.\n"; + return undef; +} + + +1; + +__END__; + Index: /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Metadata/List.pm =================================================================== --- /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Metadata/List.pm (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Metadata/List.pm (revision 11645) @@ -0,0 +1,46 @@ +# Copyright (c) 2006 Paul Price, Joshua Hoblitt +# +# $Id: List.pm,v 1.1 2006-08-12 04:26:12 price Exp $ + +package PS::IPP::Metadata::List; + +use strict; +use warnings FATAL => qw( all ); + +our $VERSION = '0.01'; + +use Carp qw( carp ); + +# Allow exporting of function +use base qw( Exporter ); +our @EXPORT_OK = qw( parse_md_list ); + +#$::RD_TRACE = 1; +#$::RD_HINT = 1; +#use Data::Dumper; + +# Given a parsed metadata, parse it into an array of hashes +sub parse_md_list { + my $md = shift; # Parsed metadata, from PS::IPP::Metadata::Config + + my @array; # The array of hashes, to be returned + foreach my $mdItem (@$md) { + if ($mdItem->{class} ne "metadata") { + carp "MD element ", $mdItem->{name}, " isn't of type METADATA --- ignored.\n"; + next; + } + my %hash; # Hash element + my $mdComponents = $mdItem->{value}; # Components of the metadata + foreach my $data (@$mdComponents) { + $hash{$data->{name}} = $data->{value}; + } + push @array, \%hash; + } + + return \@array; +} + +1; + +__END__; + Index: /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Metadata/Stats.pm =================================================================== --- /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Metadata/Stats.pm (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/lib/PS/IPP/Metadata/Stats.pm (revision 11645) @@ -0,0 +1,252 @@ +# Copyright (c) 2006 Paul Price, Joshua Hoblitt +# +# $Id: Stats.pm,v 1.12 2006-10-12 23:42:56 price Exp $ + +package PS::IPP::Metadata::Stats; + +use strict; +use warnings FATAL => qw( all ); + +our $VERSION = '0.02'; + +use Carp qw( carp croak ); +use Statistics::Descriptive; + +use base qw( Class::Accessor::Fast ); +__PACKAGE__->mk_accessors( qw( bg_mean bg_stdev bg_mean_stdev constants variables bg_data bg_stdev_data ) ); + +#$::RD_TRACE = 1; +#$::RD_HINT = 1; +use Data::Dumper; + +sub new { + my $class = shift; # Class name + my $constants = shift; # Array of values that should be constant through the FPA + my $variables = shift; # Array of values that may be variable through the FPA + + my $self = { bg_mean => undef, # Mean of the mean backgrounds + bg_stdev => undef, # Standard deviation of the mean backgrounds + bg_mean_stdev => undef, # Mean of the standard deviation of the backgrounds + constants => $constants, # Array of values that should be constant through the FPA + variables => $variables, # Array of values that may be variable through the FPA + bg_data => [], # Array of background values + bg_stdev_data => [],# Array of background standard deviations + data => {} # The data + }; + + # Populate object + foreach my $constant (@$constants) { + $self->{data}->{$constant} = { type => 'constant', # Type of value (constant/variable) + value => undef # The actual variable + }; + } + foreach my $variable (@$variables) { + $self->{data}->{$variable} = { type => 'variable', # Type of value (constant/variable) + data => [], # Array of data values + mean => undef, # Mean of data values + stdev => undef # Standard deviation of data values + }; + } + + # Bless and return object + bless $self, $class; + return $self; +} + +# Given a parsed metadata from ppStats, assemble summary statistics +sub parse { + my $self = shift; # Where we'll put the information + my $md = shift; # Parsed metadata, from PS::IPP::Metadata::Config + + # count of parse errors + my $errors = 0; + +# $self->_parse_megacam($md); + $self->_parse_braindead($md); + + # Check that we found everything for the constants + my $constants = $self->constants(); # Array of constants + if (defined $constants) { + foreach my $constant (@$constants) { + if (not defined $self->{data}->{$constant}->{value}) { + carp "Unable to find value for ", $constant, "\n"; + $errors++; + } + } + } + + # Get mean, stdev for the variables + my $variables = $self->variables(); # Array of variables + if (defined $variables) { + foreach my $variable (@$variables) { + my $info = $self->{data}->{$variable}; # The information about this particular variable + my $array = $info->{data}; # The array of values collected + if (scalar @$array == 0) { + carp "Unable to find any values for ", $variable, "\n"; + $errors++; + next; + } + + # Get statistics on the value + my $stats = Statistics::Descriptive::Sparse->new(); # Statistics calculator + $stats->add_data(@$array); + $info->{mean} = $stats->mean(); + $info->{stdev} = $stats->standard_deviation(); + } + } + + # Get mean, stdev, mean stdev for the background + if (scalar @{$self->{bg_data}} > 0) { + my $meanStats = Statistics::Descriptive::Sparse->new(); # Statistics for mean + $meanStats->add_data(@{$self->{bg_data}}); + $self->bg_mean($meanStats->mean()); + # XXX the stddev of a set of 1, appears to be a statistics undefined + # value, however that doesn't do us much go so I'm changing undef to 0 + $self->bg_mean_stdev($meanStats->standard_deviation() || 0); + } else { + $self->bg_mean(undef); + $self->bg_mean_stdev(undef); + } + if (scalar @{$self->{bg_stdev_data}} > 0) { + my $stdevStats = Statistics::Descriptive::Sparse->new(); # Statistics for standard deviation + my @variances; + foreach my $number (@{$self->{bg_stdev_data}}) { + push @variances, $number**2; + } + $stdevStats->add_data(@variances); + $self->bg_stdev( sqrt( $stdevStats->mean() ) ); + } else { + $self->bg_stdev(undef); + } + + if ($errors) { + carp "a total of $errors parse errors occured"; + return; + } + + return $self; +} + +sub _parse_megacam +{ + my ($self, $md) = @_; + + # Descend the FPA hierarchy + foreach my $fpaItem (@$md) { + if ($fpaItem->{class} eq "metadata") { + my $chipName = $fpaItem->{name}; # Name of chip + my $chipData = $fpaItem->{value}; # Chip-level data + foreach my $chipItem (@$chipData) { + if ($chipItem->{class} eq "metadata") { + my $cellName = $chipItem->{name}; # Name of cell + my $cellData = $chipItem->{value}; # Cell-level data + my $bgName; # Name of the value we measured + my $bgStdevName; # Name of the stdev we measured + foreach my $cellItem (@$cellData) { + if ($cellItem->{name} =~ /^(SAMPLE|ROBUST|FITTED|CLIPPED)_/) { + # It's a statistic of some sort + if ($cellItem->{name} =~ /_STDEV$/) { + if (defined $bgStdevName) { + carp "Ignoring duplicate background stdev (", $cellItem->{name}, "). ", + "Original is ", $bgStdevName, "\n"; + next; + } + $bgStdevName = $cellItem->{name}; + push @{$self->{bg_stdev_data}}, $cellItem->{value}; + } else { + if (defined $bgName) { + carp "Ignoring duplicate background value (", $cellItem->{name}, "). ", + "Original is ", $bgName, "\n"; + next; + } + $bgName = $cellItem->{name}; + push @{$self->{bg_data}}, $cellItem->{value}; + } + } else { + $self->_check_values($cellItem); + } + } + } else { + $self->_check_values($chipItem); + } + } + } else { + $self->_check_values($fpaItem); + } + } + + return $self; +} + +sub _parse_braindead +{ + my ($self, $md) = @_; + + # descend through the fpa + foreach my $entry (@$md) { + # recurse on nested metadata + if ($entry->{class} eq 'metadata') { + $self->_parse_braindead($entry->{value}); + } + + if ($entry->{name} =~ /^(SAMPLE|ROBUST|FITTED|CLIPPED)_/) { + # It's a statistic of some sort + if ($entry->{name} =~ /_STDEV$/) { + push @{$self->{bg_stdev_data}}, $entry->{value}; + } else { + push @{$self->{bg_data}}, $entry->{value}; + } + } else { + $self->_check_values($entry); + } + } + + return $self; +} + +# Private function to check a metadata item for things we're interested in +sub _check_values { + my $self = shift; + my $mdItem = shift; # Metadata item to check + + my $name = $mdItem->{name}; # Name of the item + my $value = $mdItem->{value}; # Value of the item + + my $data = $self->{data}; # The data + return if not defined $data->{$name}; # Not interested + my $type = $data->{$name}->{type}; # Type of the item: constant or variable + + if ($type eq 'constant') { + if (defined $data->{$name}->{value} && ($data->{$name}->{value} ne $value)) { + carp "Warning: different value for ", $name, " found: ", $data->{$name}->{value}, " (old) vs ", + $value, " (new). Ignoring new value.\n"; + return; + } + $data->{$name}->{value} = $value; + return; + } + + if ($type eq 'variable') { + my $array = $data->{$name}->{data}; # Array containing all the values + push @$array, $value; + return; + } + + croak "Unrecognised type (", $type, ") for value of ", $name, "\n"; + return; +} + +# Return the data structure for a particular value, or the entire hash +sub data { + my $self = shift; + my $name = shift; # Name of data to return + + return $self->{data} if (not defined $name); + return $self->{data}->{$name}; +} + + +1; + +__END__; + Index: /branches/rel-1-0/PS-IPP-Config/t/00_distribution.t =================================================================== --- /branches/rel-1-0/PS-IPP-Config/t/00_distribution.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Config/t/00_distribution.t (revision 11645) @@ -0,0 +1,25 @@ +#!/usr/bin/env perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 00_distribution.t,v 1.1 2006-09-06 01:59:09 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ./t ); + +use Test::More; + +# example taken from Test::Distribution Pod + +BEGIN { + eval { + require Test::Distribution; + }; + if($@) { + plan skip_all => 'Test::Distribution not installed'; + } else { + import Test::Distribution; # not => qw( podcover ); + } +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/.cvsignore =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/.cvsignore (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/.cvsignore (revision 11645) @@ -0,0 +1,6 @@ +Build +META.yml +Makefile +Makefile.PL +_build +blib Index: /branches/rel-1-0/PS-IPP-Metadata-Config/Build.PL =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/Build.PL (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/Build.PL (revision 11645) @@ -0,0 +1,43 @@ +use Module::Build; +# See perldoc Module::Build for details of how this works + +my $class = Module::Build->subclass(code => <<'EOF'); +use File::Copy; +use File::stat; + +my $grammar = "config_grammar.txt"; +my $parser = "lib/PS/IPP/Metadata/Parser.pm"; + +sub ACTION_code { + my $self = shift; + + if (!-e $parser || (stat($grammar)->mtime > stat($parser)->mtime)) { + system("perl -MParse::RecDescent - $grammar PS::IPP::Metadata::Parser") == 0 + or die "Parse::RecDecent code gen failed: $?"; + move("Parser.pm", $parser) or die "move failed: $!"; + } + + $self->SUPER::ACTION_code(@_); +} +EOF + +$class->new( + module_name => 'PS::IPP::Metadata::Config', + dist_version_from => 'lib/PS/IPP/Metadata/Config.pm', + author => 'Joshua Hoblitt ', + license => 'gpl', + create_makefile_pl => 'passthrough', + requires => { + 'Carp' => 0, + 'Class::Accessor::Fast' => '0.19', + 'Params::Validate' => '0.72', + 'DateTime::Format::ISO8601' => '0.0402', + 'Parse::RecDescent' => '1.94', + }, + recommends => { + 'Test::Distribution' => '1.22', + }, + script_files => [qw( + scripts/mdc-dump + )], +)->create_build_script; Index: /branches/rel-1-0/PS-IPP-Metadata-Config/Changes =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/Changes (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/Changes (revision 11645) @@ -0,0 +1,54 @@ +Revision history for Perl module PS::IPP::Metadata::Config + +0.11 Thu Feb 1 15:42:28 HST 2007 + - add t/00_distribution.t + +0.10 Fri Nov 24 17:11:13 HST 2006 + - fix handling of NULL strings + - merge the 'time' production into the 'scalar' production for a ~20% speed boost + - remove support for "epoch format" times + - additional metadata structure tests + - merge metadata_name & metadata_end into metdata + - merge empty metdata case into main metadata production + - error check vector + - change type & vtype into regex alternations instead of rule alternations + +0.09 Thu Oct 26 16:16:28 HST 2006 + - fix a bug in Build.PL where a rebuilt grammar was not installed + - fix time tests that were dependent on MULTI times being broken + - fix MULTI time + - no longer require the 'Z' and the end of ISO8601 times + - implement NULL time support + +0.08 Wed Oct 11 16:09:26 HST 2006 + - add support for empty METADATAs + +0.07 Tue Oct 10 17:09:28 HST 2006 + - fix MULTI parsing of TYPEs + - minor test update + +0.06 Tue Oct 3 15:25:44 HST 2006 + - fix Build.PL script to not throw and error when $parser does not exist + - fix the trailing white space left in string values when followed by a + comment + +0.05 Mon Sep 25 12:06:22 HST 2006 + - fix a bug in the manifest + - fix Build.PL so the parser is only regenerated when the grammar has been + changed + +0.04 Mon Sep 25 11:50:20 HST 2006 + - add fractional second support to the ISO8601 format + +0.03 Fri Sep 22 14:27:14 HST 2006 + - add float nan/inf support + +0.02 Thu Aug 24 17:08:14 HST 2006 + - split grammar out of PS::IPP::Metadata::Config into grammar_config.txt + - precompile the parser as PS::IPP::Metadata::Pasrser + - change PS::IPP::Metadata::Config to use PS::IPP::Metadata::Pasrser + +0.01 Tue Feb 22 15:46:35 2005 + - original version; created by ExtUtils::ModuleMaker 0.32 + + Index: /branches/rel-1-0/PS-IPP-Metadata-Config/LICENSE =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/LICENSE (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/LICENSE (revision 11645) @@ -0,0 +1,260 @@ +The General Public License (GPL) +Version 2, June 1991 + +Copyright (C) 1989, 1991 Free Software Foundation, Inc. 675 Mass Ave, +Cambridge, MA 02139, USA. Everyone is permitted to copy and distribute +verbatim copies of this license document, but changing it is not allowed. + +Preamble + +The licenses for most software are designed to take away your freedom to share +and change it. By contrast, the GNU General Public License is intended to +guarantee your freedom to share and change free software--to make sure the +software is free for all its users. This General Public License applies to most of +the Free Software Foundation's software and to any other program whose +authors commit to using it. (Some other Free Software Foundation software is +covered by the GNU Library General Public License instead.) You can apply it to +your programs, too. + +When we speak of free software, we are referring to freedom, not price. Our +General Public Licenses are designed to make sure that you have the freedom +to distribute copies of free software (and charge for this service if you wish), that +you receive source code or can get it if you want it, that you can change the +software or use pieces of it in new free programs; and that you know you can do +these things. + +To protect your rights, we need to make restrictions that forbid anyone to deny +you these rights or to ask you to surrender the rights. These restrictions +translate to certain responsibilities for you if you distribute copies of the +software, or if you modify it. + +For example, if you distribute copies of such a program, whether gratis or for a +fee, you must give the recipients all the rights that you have. You must make +sure that they, too, receive or can get the source code. And you must show +them these terms so they know their rights. + +We protect your rights with two steps: (1) copyright the software, and (2) offer +you this license which gives you legal permission to copy, distribute and/or +modify the software. + +Also, for each author's protection and ours, we want to make certain that +everyone understands that there is no warranty for this free software. If the +software is modified by someone else and passed on, we want its recipients to +know that what they have is not the original, so that any problems introduced by +others will not reflect on the original authors' reputations. + +Finally, any free program is threatened constantly by software patents. We wish +to avoid the danger that redistributors of a free program will individually obtain +patent licenses, in effect making the program proprietary. To prevent this, we +have made it clear that any patent must be licensed for everyone's free use or +not licensed at all. + +The precise terms and conditions for copying, distribution and modification +follow. + +GNU GENERAL PUBLIC LICENSE +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND +MODIFICATION + +0. This License applies to any program or other work which contains a notice +placed by the copyright holder saying it may be distributed under the terms of +this General Public License. The "Program", below, refers to any such program +or work, and a "work based on the Program" means either the Program or any +derivative work under copyright law: that is to say, a work containing the +Program or a portion of it, either verbatim or with modifications and/or translated +into another language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not covered by +this License; they are outside its scope. The act of running the Program is not +restricted, and the output from the Program is covered only if its contents +constitute a work based on the Program (independent of having been made by +running the Program). Whether that is true depends on what the Program does. + +1. You may copy and distribute verbatim copies of the Program's source code as +you receive it, in any medium, provided that you conspicuously and appropriately +publish on each copy an appropriate copyright notice and disclaimer of warranty; +keep intact all the notices that refer to this License and to the absence of any +warranty; and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and you may at +your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Program or any portion of it, thus +forming a work based on the Program, and copy and distribute such +modifications or work under the terms of Section 1 above, provided that you also +meet all of these conditions: + +a) You must cause the modified files to carry prominent notices stating that you +changed the files and the date of any change. + +b) You must cause any work that you distribute or publish, that in whole or in +part contains or is derived from the Program or any part thereof, to be licensed +as a whole at no charge to all third parties under the terms of this License. + +c) If the modified program normally reads commands interactively when run, you +must cause it, when started running for such interactive use in the most ordinary +way, to print or display an announcement including an appropriate copyright +notice and a notice that there is no warranty (or else, saying that you provide a +warranty) and that users may redistribute the program under these conditions, +and telling the user how to view a copy of this License. (Exception: if the +Program itself is interactive but does not normally print such an announcement, +your work based on the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If identifiable +sections of that work are not derived from the Program, and can be reasonably +considered independent and separate works in themselves, then this License, +and its terms, do not apply to those sections when you distribute them as +separate works. But when you distribute the same sections as part of a whole +which is a work based on the Program, the distribution of the whole must be on +the terms of this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to +work written entirely by you; rather, the intent is to exercise the right to control +the distribution of derivative or collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program with the +Program (or with a work based on the Program) on a volume of a storage or +distribution medium does not bring the other work under the scope of this +License. + +3. You may copy and distribute the Program (or a work based on it, under +Section 2) in object code or executable form under the terms of Sections 1 and 2 +above provided that you also do one of the following: + +a) Accompany it with the complete corresponding machine-readable source +code, which must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange; or, + +b) Accompany it with a written offer, valid for at least three years, to give any +third party, for a charge no more than your cost of physically performing source +distribution, a complete machine-readable copy of the corresponding source +code, to be distributed under the terms of Sections 1 and 2 above on a medium +customarily used for software interchange; or, + +c) Accompany it with the information you received as to the offer to distribute +corresponding source code. (This alternative is allowed only for noncommercial +distribution and only if you received the program in object code or executable +form with such an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for making +modifications to it. For an executable work, complete source code means all the +source code for all modules it contains, plus any associated interface definition +files, plus the scripts used to control compilation and installation of the +executable. However, as a special exception, the source code distributed need +not include anything that is normally distributed (in either source or binary form) +with the major components (compiler, kernel, and so on) of the operating system +on which the executable runs, unless that component itself accompanies the +executable. + +If distribution of executable or object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the source +code from the same place counts as distribution of the source code, even though +third parties are not compelled to copy the source along with the object code. + +4. You may not copy, modify, sublicense, or distribute the Program except as +expressly provided under this License. Any attempt otherwise to copy, modify, +sublicense or distribute the Program is void, and will automatically terminate +your rights under this License. However, parties who have received copies, or +rights, from you under this License will not have their licenses terminated so long +as such parties remain in full compliance. + +5. You are not required to accept this License, since you have not signed it. +However, nothing else grants you permission to modify or distribute the Program +or its derivative works. These actions are prohibited by law if you do not accept +this License. Therefore, by modifying or distributing the Program (or any work +based on the Program), you indicate your acceptance of this License to do so, +and all its terms and conditions for copying, distributing or modifying the +Program or works based on it. + +6. Each time you redistribute the Program (or any work based on the Program), +the recipient automatically receives a license from the original licensor to copy, +distribute or modify the Program subject to these terms and conditions. You +may not impose any further restrictions on the recipients' exercise of the rights +granted herein. You are not responsible for enforcing compliance by third parties +to this License. + +7. If, as a consequence of a court judgment or allegation of patent infringement +or for any other reason (not limited to patent issues), conditions are imposed on +you (whether by court order, agreement or otherwise) that contradict the +conditions of this License, they do not excuse you from the conditions of this +License. If you cannot distribute so as to satisfy simultaneously your obligations +under this License and any other pertinent obligations, then as a consequence +you may not distribute the Program at all. For example, if a patent license would +not permit royalty-free redistribution of the Program by all those who receive +copies directly or indirectly through you, then the only way you could satisfy +both it and this License would be to refrain entirely from distribution of the +Program. + +If any portion of this section is held invalid or unenforceable under any particular +circumstance, the balance of the section is intended to apply and the section as +a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other +property right claims or to contest validity of any such claims; this section has +the sole purpose of protecting the integrity of the free software distribution +system, which is implemented by public license practices. Many people have +made generous contributions to the wide range of software distributed through +that system in reliance on consistent application of that system; it is up to the +author/donor to decide if he or she is willing to distribute software through any +other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a +consequence of the rest of this License. + +8. If the distribution and/or use of the Program is restricted in certain countries +either by patents or by copyrighted interfaces, the original copyright holder who +places the Program under this License may add an explicit geographical +distribution limitation excluding those countries, so that distribution is permitted +only in or among countries not thus excluded. In such case, this License +incorporates the limitation as if written in the body of this License. + +9. The Free Software Foundation may publish revised and/or new versions of the +General Public License from time to time. Such new versions will be similar in +spirit to the present version, but may differ in detail to address new problems or +concerns. + +Each version is given a distinguishing version number. If the Program specifies a +version number of this License which applies to it and "any later version", you +have the option of following the terms and conditions either of that version or of +any later version published by the Free Software Foundation. If the Program does +not specify a version number of this License, you may choose any version ever +published by the Free Software Foundation. + +10. If you wish to incorporate parts of the Program into other free programs +whose distribution conditions are different, write to the author to ask for +permission. For software which is copyrighted by the Free Software Foundation, +write to the Free Software Foundation; we sometimes make exceptions for this. +Our decision will be guided by the two goals of preserving the free status of all +derivatives of our free software and of promoting the sharing and reuse of +software generally. + +NO WARRANTY + +11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS +NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE +COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM +"AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR +IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE +ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, +YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED +TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY +WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS +PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM +(INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY +OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS +BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS Index: /branches/rel-1-0/PS-IPP-Metadata-Config/MANIFEST =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/MANIFEST (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/MANIFEST (revision 11645) @@ -0,0 +1,23 @@ +Build.PL +Changes +LICENSE +MANIFEST +META.yml +Makefile.PL +README +Todo +config_grammar.txt +docs/sdrs_grammar.txt +lib/PS/IPP/Metadata/Config.pm +lib/PS/IPP/Metadata/Config.pod +lib/PS/IPP/Metadata/Parser.pm +scripts/mdc-dump +t/00_distribution.t +t/01_load.t +t/02_examples.t +t/03_metadata.t +t/04_type.t +t/05_time.t +t/06_multi.t +t/07_floats.t +t/08_strings.t Index: /branches/rel-1-0/PS-IPP-Metadata-Config/README =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/README (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/README (revision 11645) @@ -0,0 +1,16 @@ +pod2text PS::IPP::Metadata::Config.pm > README + +If this is still here it means the programmer was too lazy to create the readme file. + +You can create it now by using the command shown above from this directory. + +At the very least you should be able to use this set of instructions +to install the module... + +perl Build.PL +./Build +./Build test +./Build install + + +If you are on a windows box you should use 'nmake' rather than 'make'. Index: /branches/rel-1-0/PS-IPP-Metadata-Config/Todo =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/Todo (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/Todo (revision 11645) @@ -0,0 +1,5 @@ +TODO list for Perl module PS::IPP::Metadata::Config + +- Nothing yet + + Index: /branches/rel-1-0/PS-IPP-Metadata-Config/config_grammar.txt =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/config_grammar.txt (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/config_grammar.txt (revision 11645) @@ -0,0 +1,354 @@ +# Copyright (c) 2005 Joshua Hoblitt +# +# $Id: config_grammar.txt,v 1.12 2006-11-25 02:29:05 jhoblitt Exp $ + +{ + use strict; + + use Carp qw( carp ); + use DateTime::Format::ISO8601; + + our @scope_stack; +} + +startrule: grammar eofile + { $item{grammar} } + +grammar: + statement(s) + { + $thisparser->{local} = pop @scope_stack; + [ grep $_->{class} !~ /comment_line/, @{$item{'statement(s)'}} ]; + } + +statement: + scalar + | comment_line + | metadata + | typedef + | typedef_declare + | multi_declare + | vector + | + +scalar: + name type comment(?) "\n" + { + $thisparser->{local}{name}{ $item{name} }++ + unless defined $thisparser->{local}{name}{ $item{name} }; + + $return = { + class => 'scalar', + name => $item{name}, + type => $item{type}, + value => $item[4], + }; + + $return->{comment} = $item{'comment(?)'}[0] + if $item{'comment(?)'}[0]; + + if ( defined $thisparser->{local}{name}{ $item{name} } ) { + $return->{multi}++ + if $thisparser->{local}{name}{ $item{name} } =~ /MULTI/; + } else { + $thisparser->{local}{name}{ $item{name} }++; + } + } + +vector: + vname vtype vvalue[ $item{vtype} ] comment(?) "\n" + { + $thisparser->{local}{name}{ $item{vname} }++ + unless defined $thisparser->{local}{name}{ $item{vname} }; + + $return = { + class => 'vector', + name => $item{vname}, + type => $item{vtype}, + value => $item{vvalue}, + comment => $item{comment} + }; + + $return->{comment} = $item{'comment(?)'}[0] + if $item{'comment(?)'}[0]; + } + | + +multi_declare: + name /MULTI/i comment(?) "\n" + { + if ( ! defined $thisparser->{local}{name}{ $item{name} } ) { + $thisparser->{local}{name}{ $item{name} } = "MULTI"; + $return = { class => 'comment_line' }; + } else { + $return = undef; + } + } + | + +# can't comit after the metadat_name here as it seems to break parsing empty +# metadatas +metadata: + name /METADATA/i comment(?) "\n" + + { push @scope_stack, $thisparser->{local}; $thisparser->{local} = {}; 1; } + grammar(?) + /END/i comment(?) "\n" + + { + if (defined(@{$item{'grammar(?)'}})) { + $return = { + class => 'metadata', + name => $item{name}, + value => @{$item{'grammar(?)'}}, + }; + } else { + $return = { + class => 'metadata', + name => $item{name}, + value => [], + }; + } + + $return->{comment} = $item[4][0] + if defined $item[4][0]; + + if ( defined $thisparser->{local}{name}{ $item{name} } ) { + $return->{multi}++ + if $thisparser->{local}{name}{ $item{name} } =~ /MULTI/; + } else { + $thisparser->{local}{name}{ $item{name} }++; + } + } + | + +comment_line: + comment(?) "\n" + {{ class => 'comment_line' }} + +typedef_declare: + 'TYPE' name name(s) comment(?) "\n" + { + if (defined $thisparser->{local}{typedef}{ $item{name} }) { + carp "refinition of TYPE $item{name}"; + $return = undef; + } else { + + my $type = { + class => 'typedef_declare', + name => $item{name}, + fields => $item{'name(s)'}, + length => scalar @{ $item{'name(s)'} }, + }; + + $thisparser->{local}{typedef}{ $item{name} } = $type; + + # don't return anything in the parse tree + $return = { class => 'comment_line' }; + } + } + | + +typedef: + name typedef_type word(s) comment(?) "\n" + { + my $type = $thisparser->{local}{typedef}{ $item{typedef_type} }; + + unless ( scalar @{ $item{'word(s)'} } == $type->{length} ) { + my $expect = $type->{length}; + my $got = scalar @{ $item{'word(s)'} }; + carp "\"$item{name} $item{typedef_type}\"" + . " does not have enough fields, epected: $expect, got: $got"; + $return = undef; + } else { + my @md; + my $i = 0; + foreach my $name ( @{ $type->{fields} } ) { + push @md, { + name => $name, + class => 'scalar', + type => 'STR', + value => $item{'word(s)'}[$i], + }; + + $i++; + } + + $return = { + class => 'metadata', + name => $item{name}, + value => \@md, + }; + + if ( defined $thisparser->{local}{name}{ $item{name} } ) { + $return->{multi}++ + if $thisparser->{local}{name}{ $item{name} } =~ /MULTI/; + } else { + $thisparser->{local}{name}{ $item{name} }++; + } + } + + } + | + + +typedef_type: name + { + if ( ! defined $thisparser->{local}{typedef}{ $item{name} } ) { + $return = undef; + } else { + $return = $item{name}; + } + } + +comment: + '#' to_end_of_line + { $item{to_end_of_line} } + +vname: + '@' name + { $item{name} } + +name: + /[a-z][.\w-]*/i + { + if ( $item[1] =~ /^(METADATA|END|TYPE)$/i ) { + $return = undef; + } else { + $return = $item[1]; + } + } + +# 'STR' seems to be the most common type, trying to match it first is a simple +# optimization. +type: + vtype + | /(STR + | STRING + | UTC + | UT1 + | TAI + | TT)/x + +vtype: + /(S8 + | S16 + | S32 + | S64 + | U8 + | U16 + | U32 + | U64 + | F32 + | F64 + | C32 + | C64 + | BOOL)/x + +S8: int +S16: int +S32: int +S64: int +U8 : int +U16: int +U32: int +U64: int +F32: float +F64: float +C32: float +C64: float +BOOL: bool +STR: string +STRING: string + +vvalue: + _vvalue[%arg](s) + { [ map { @$_ } @{$item[1]} ] } + +# backtracking optimization +_vvalue: + vector_sep(?) + { [ $item[1] ] } + +vector_sep: + /,|\s+/ + +int: + # $RE{num}{int} from Regexp::Common::number + /(?:(?:[+-]?)(?:[0-9]+))/ + + # based on $RE{num}{real} from Regexp::Common::number +float: + /(?:(?i)(?:[+-]?)(?:(?=[0-9]|[.])(?:[0-9]*)(?:(?:[.])(?:[0-9]{0,}))?)(?:(?:[Ee])(?:(?:[+-]?)(?:[0-9]+))|))/ + | inf + | nan + +inf: + /[+-]?inf(?:inity)?/i + { + if ($item[1] =~ /^-/) { + $return = "-inf"; + } else { + $return = "+inf"; + } + } +nan: + /[+-]?nan(?:\(\S*?\))?/i + { $return = "nan" } + +bool: + /[tf]/i + { $item[1] =~ /t/i ? 1 : 0 } + +string: + /(?:\S[^#\n]*)?[^#\n ]/ + # trim tailing the trailing white that's left when the string is + # followed by a comment (#) + { + $item[1] =~ s/\s*$//; + $return = $item[1]; + } + | NULL + +word: + /(?:[^#\s\n]+)/ + +to_end_of_line: + /[^\n]*/ + { + my $comment = $item[1]; + # remove leading whitespace + $comment =~ s/^\s+//; + # remove trailing whitespace + $comment =~ s/\s+$//; + + $return = $comment; + } + +UTC: + iso8601 + | NULL + +UT1: + iso8601 + | NULL + +TAI: + iso8601 + | NULL + +TT: + iso8601 + | NULL + +iso8601: + # based on code from DateTime::Format::ISO8601 + / \d{4} -?? \d\d -?? \d\d T?? \d\d :?? \d\d :?? \d\d (?:[\.,] (\d+))? Z?/x + { DateTime::Format::ISO8601->parse_datetime( $item[1] ) } + +NULL: + /NULL/ + { $return = "NULL" } + +eofile: + /^\z/ Index: /branches/rel-1-0/PS-IPP-Metadata-Config/docs/sdrs_grammar.txt =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/docs/sdrs_grammar.txt (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/docs/sdrs_grammar.txt (revision 11645) @@ -0,0 +1,144 @@ +startrule: statement(s) /\z/ + +statement: scalar | vector | multi_declare | comment + +comment: '#' end_of_line + +scalar: name type value comment | name type value + +vector: vname vtype vvalue comment | vname vtype vvalue + +multi_declare: name '*' end_of_line + +name: /[a-z][.\w]*/i + +vname: /\@[a-z][.\w]*/i + +type: + vtype | 'STR' | 'STRING' + +vtype: + 'S8' | + 'S16' | + 'S32' | + 'S64' | + 'U8' | + 'U16' | + 'U32' | + 'U64' | + 'F32' | + 'F64' | + 'C32' | + 'C64' | + 'BOOL' + +value: vector_value | string + +vvalue: vector_value vector_sep vvalue | vector_value vector_sep(?) + +vector_sep: ',' + +vector_value: float | int | bool + +int: integer_constant + +float: floating_constant + +bool: /[tf]\s+?/i + +string: + /\S[^#\n]*/ + +end_of_line: /[^\n]*/ + +### based on syntax found in "C A Reference Manual" + +# integer constants + +integer_constant: + decimal_constant integer_suffix(?) | + octal_constant integer_suffix(?) | + hexadecimal_constant integer_suffix(?) + +decimal_constant: + digit(2..) | + nonzero_digit + +octal_constant: + '0' octal_digit(s) | + '0' + +hexadecimal_constant: + hex_prefix hex_digit_sequence + +digit: + /[0-9]/ + +nonzero_digit: + /[1-9]/ + +octal_digit: + /[0-7]/ + +hex_digit: + /0_9a-f/i + +integer_suffix: + long_suffix unsigned_suffix(?) | + long_long_suffix unsigned_suffix(?) | + unsigned_suffix long_suffix(?) | + unsigned_suffix long_long_suffix(?) + +long_suffix: + /l/i + +long_long_suffix: + /ll/i + +unsigned_suffix: + /u/i + +# floating-point constants + +floating_constant: + decimal_floating_constant | + hexadecimal_floating_constant + +decimal_floating_constant: + digit_sequence exponent floating_suffix(?) | + dotted_digits exponent(?) floating_suffix(?) + +digit_sequence: + digit(s) + +dotted_digits: + digit_sequence '.' digit_sequence | + '.' digit_sequence | + digit_sequence '.' + +exponent: + /e/i sign_part digit_sequence + +sign_part: + /[+-]/ + +floating_suffix: + /[fl]/i + +hexadecimal_floating_constant: + hex_prefix dotted_hex_digits binary_exponent floating_suffix(?) | + hex_prefix hex_digit_sequence binary_exponent floating_suffix(?) + +hex_prefix: + /0x/i + +dotted_hex_digits: + hex_digit_sequence '.' hex_digit_sequence | + '.' hex_digit_sequence | + hex_digit_sequence '.' + +hex_digit_sequence: + hex_digit(s) + +binary_exponent: + /p/i sign_part(?) digit_sequence Index: /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/.cvsignore =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/.cvsignore (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/.cvsignore (revision 11645) @@ -0,0 +1,1 @@ +Parser.pm Index: /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/Config.pm =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/Config.pm (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/Config.pm (revision 11645) @@ -0,0 +1,137 @@ +# Copyright (c) 2005 Joshua Hoblitt +# +# $Id: Config.pm,v 1.30 2007-02-02 01:42:56 jhoblitt Exp $ + +package PS::IPP::Metadata::Config; + +use strict; +use warnings FATAL => qw( all ); + +our $VERSION = '0.11'; + +use Carp qw( carp ); +use PS::IPP::Metadata::Parser; + +use base qw( Class::Accessor::Fast ); +__PACKAGE__->mk_accessors( qw( overwrite ) ); + +#$::RD_TRACE = 1; +#$::RD_HINT = 1; +#use Data::Dumper; + +sub new { + my $class = shift; + + my $self = { _parser => PS::IPP::Metadata::Parser->new }; + + bless $self, $class; + + $self->overwrite( undef ); + + return $self; +} + +sub parse { + my ( $self, $metadata ) = @_; + + return undef if not defined $metadata; + return undef if $metadata =~ /^\s*$/; + + # remove any local data from a prevous run + # this is slightly faster then using Storable::dclone to clone a new parser + delete $self->{_parser}{local}; + + my $tree = $self->{_parser}->startrule( $metadata ); + + return undef unless defined $tree; + + # look for duplicate names, there should be none after processing the + # multi-symbols + $self->_merge_duplicates( $tree ) or return undef; + + # translate NULL values into perl undefs (Parse::RecDescent can't return + # undefs, args) + $self->_fix_nulls( $tree ) or return undef; + + #print Dumper($tree); + + return $tree; +} + +sub _merge_duplicates { + my ( $self, $tree ) = @_; + + # encountered names + my %names; + + # Iteratate through the parse tree looking for duplicate declarations and + # resolving them by discarding elements according to the value of + # 'overwrite'. + for (my $i = 0; $i < @{$tree}; $i++) { + my $elem = $tree->[$i]; + + # stop if the prevous pass removed the last element and called redo + last unless defined $elem; + + # recurse through nested metadata + if ( $elem->{class} eq "metadata" ) { + $self->_merge_duplicates( $elem->{value} ); + } + + # ignore elements with the "multi" flag + if ( defined $elem->{multi} ) { + delete $elem->{multi}; + next; + } + + if ( defined $names{ $elem->{name} } ) { + if ( $self->{overwrite} ) { + # remove the previous occurance + carp "duplicate variable name: ", $elem->{name} + , ", removed previous occurance\n"; + splice @{$tree}, $names{ $elem->{name} }, 1; + } else { + # remove the current occurance + carp "duplicate variable name: ", $elem->{name} + , ", removed\n"; + splice @{$tree}, $i, 1; + } + + # the list just got shorter by one element so we don't want to + # increment the cursor + redo; + } + + # add element name and location to record of previously seen names + $names{ $elem->{name} } = $i; + } + + return 1; +} + +sub _fix_nulls { + my ( $self, $tree ) = @_; + + # Iteratate through the parse tree looking for values of NULL and + # translating them into Perl undefs + for (my $i = 0; $i < @{$tree}; $i++) { + my $elem = $tree->[$i]; + + # recurse through nested metadata + if ( $elem->{class} eq "metadata" ) { + $self->_fix_nulls( $elem->{value} ); + } + + # force stringification of $elem->{value} -- specifically because you + # can't compare a DateTime object to a string value + if ("$elem->{value}" eq "NULL") { + $elem->{value} = undef; + } + } + + return 1; +} + +1; + +__END__ Index: /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/Config.pod =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/Config.pod (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/lib/PS/IPP/Metadata/Config.pod (revision 11645) @@ -0,0 +1,137 @@ +=pod + +=head1 NAME + +PS::IPP::Metadata::Config - Parses the psMetadata config language + +=head1 SYNOPSIS + + use PS::IPP::Metadata::Config; + + my $mdparser = PS::IPP::Metadata::Config->new; + $mdparser->overwrite( 1 ); + + my $config = $mdparser->parse( $example ); + +=head1 DESCRIPTION + +Parses the psMetadata configuration language into an AoH parse tree. + +=head1 USAGE + +=head2 Import Parameters + +This module accepts no arguments to it's C method and exports no +I. + +=head2 Methods + +=head3 Constructors + +=over 4 + +=item * new + +Accepts no parameters and returns a L object. + +=back + +=head3 Object Methods + +=over 4 + +=item * overwrite + +Controls the behavior of the parser when encountering keys that have been +defined more then once. A true value instructs the parser to discard all but +the latest declaration and a false value will preserve the first key parsed. +Accepts 0, 1, or undef. + +Defaults to undef. + +=item * parse + +Accepts a string. Returns an AoH structure on success or undef on failure. + +The format of the returned structure is as follows: + + [ + { + 'comment' => 'This is a comment', + 'value' => '123456789', + 'name' => 'Bob', + 'type' => 'U64', + 'class' => 'scalar' + }, + { + 'comment' => 'These are prime numbers', + 'value' => [ + '2', + '3' + ], + 'name' => '@primes', + 'type' => 'U8', + 'class' => 'vector' + }, + ... + ] + +=back + +=head1 DEVELOPER NOTES + +Currently, "multiple symbols" are translated into "STR" vectors as the +appropriate behavior isn't clear in the psLib SDRS. + +The parser does not have complete support for all of C99's floating point +constants or any support for complex numbers. + +None of the "extended" config language syntax, as defined in the SDRS, is +supported. + +=head1 REFERENCES + +=over 4 + +=item PSDC-430-007 + +Pan-STARRS PS-1 IPP Library Supplementary Design Requirements Specification +(SDRS): Section 5.3.4, "Configuration Files". + +=back + +=head1 CREDITS + +Just me, myself, and I. + +=head1 SUPPORT + +Please contact the author directly via e-mail. + +=head1 AUTHOR + +Joshua Hoblitt + +=head1 COPYRIGHT + +Copyright (C) 2005 Joshua Hoblitt. All rights reserved. + +This program is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free Software +Foundation; either version 2 of the License, or (at your option) any later +version. + +This program is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A +PARTICULAR PURPOSE. See the GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 Temple +Place - Suite 330, Boston, MA 02111-1307, USA. + +The full text of the license can be found in the LICENSE file included with +this module, or in the L Pod as supplied with Perl 5.8.1 and later. + +=head1 SEE ALSO + +=cut Index: /branches/rel-1-0/PS-IPP-Metadata-Config/scripts/mdc-dump =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/scripts/mdc-dump (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/scripts/mdc-dump (revision 11645) @@ -0,0 +1,146 @@ +#!/usr/bin/env perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: mdc-dump,v 1.2 2006-01-21 02:38:17 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use vars qw( $VERSION ); +$VERSION = '0.02'; + +use Data::Dumper qw( Dumper ); +use PS::IPP::Metadata::Config; + +use Getopt::Long qw( GetOptions :config auto_help auto_version ); +use Pod::Usage qw( pod2usage ); + +my ( $file, $overwrite, $verbose ); +GetOptions( + 'file=s' => \$file, + 'overwrite' => \$overwrite, + 'verbose' => \$verbose, +) || pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; + +# open STDIN if no file was specified +$file = '-' unless $file; + +if (defined $verbose) { + $::RD_TRACE = 1; + $::RD_HINT = 1; +} +my $md_parser = PS::IPP::Metadata::Config->new; + +if ( $overwrite ) { + $md_parser->overwrite( 1 ); +} else { + $md_parser->overwrite( 0 ); +} + +open( my $md, $file ) or die "can't open file: $!"; + +my $data = $md_parser->parse( do { local $/; <$md> } ); + +close( $md ); + +if ( $data ) { + print Dumper( $data ); +} else { + warn "Invalid Syntax, Parse failed.\n"; + exit 1; +} + +__END__ + +=pod + +=head1 NAME + +mddump.pl - validate psMetadataConfig files + +=head1 SYNOPSIS + + mddump.pl [--overwrite] [--verbose] --file + + or + + cat | mddump.pl [--overwrite] [--verbose] + + or + + mddump.pl [--help | -h | -? | --version] + +=head1 DESCRIPTION + +Parses psMetadataConfig formatted text either from a file or STDIN and prints a +Perl data structure that represents the metadata. + +=head1 OPTIONS + +=over 4 + +=item * --file | -f + +Accepts the name of a file to use as input. If this option is omitted the +STDIN will be read from. + +=item * --overwrite | -o + +A flag that turns L's overwrite behavior on. +Overwrite is always off unless this flag is specified. + +=item * --verbose | -v + +A flag that turns L's C and C tracing +flags. verbose is always off unless this flag is specified. + +=item * --help | -h | -? + +Prints usage information. + +=item * --version + +Prints the version. + +=back + +=head1 CREDITS + +Just me, myself, and I. + +=head1 SUPPORT + +Please contact the author directly via e-mail. + +=head1 AUTHOR + +Joshua Hoblitt + +=head1 COPYRIGHT + +Copyright (C) 2004 Joshua Hoblitt. All rights reserved. + +This program is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free Software +Foundation; either version 2 of the License, or (at your option) any later +version. + +This program is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A +PARTICULAR PURPOSE. See the GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 Temple +Place - Suite 330, Boston, MA 02111-1307, USA. + +The full text of the license can be found in the L Pod as supplied +with Perl 5.8.1 and later. + +=head1 SEE ALSO + +L + +=cut Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/00_distribution.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/00_distribution.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/00_distribution.t (revision 11645) @@ -0,0 +1,25 @@ +#!/usr/bin/env perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 00_distribution.t,v 1.1 2007-01-30 01:30:48 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ./t ); + +use Test::More; + +# example taken from Test::Distribution Pod + +BEGIN { + eval { + require Test::Distribution; + }; + if($@) { + plan skip_all => 'Test::Distribution not installed'; + } else { + import Test::Distribution not => qw( podcover ); + } +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/01_load.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/01_load.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/01_load.t (revision 11645) @@ -0,0 +1,14 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 01_load.t,v 1.1.1.1 2005-03-01 03:38:45 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +use Test::More tests => 1; + +BEGIN { use_ok( 'PS::IPP::Metadata::Config' ); } Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/02_examples.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/02_examples.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/02_examples.t (revision 11645) @@ -0,0 +1,385 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 02_examples.t,v 1.5 2006-07-12 02:49:53 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 9; +use PS::IPP::Metadata::Config; + +my $config_parser = PS::IPP::Metadata::Config->new; + +{ + my $config = $config_parser->parse(undef); + is($config, undef, "parsing undef returns undef"); +} + +{ + my $config = $config_parser->parse(''); + is($config, undef, "parsing an empty string returns undef"); +} + +{ + my $config = $config_parser->parse("\n\n\n\n\n"); + is($config, undef, "parsing an empty string returns undef"); +} + +{ +my $example = q{ +Double F64 1.23456789 # This is a comment +Float F32 0.98765#This is a comment too +String STR This is the string that forms the value #comment + + # This is a comment line and is to be ignored +boolean BOOL T # The value of `boolean' is `true' + +@primes U8 2,3 5 7,11,13 17 # These are prime numbers + +comment MULTI # The rest of this line is ignored, but `comment' is set to be non-unique +comment STR This +comment STR is +comment STR a +comment STR non-unique +comment STR key +Float F64 1.23456 # This generates a warning, and, if `overwrite' is `false', is ignored +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "SDRS example parsed"); + + my $tree = [ + { + name => 'Double', + class => 'scalar', + type => 'F64', + value => '1.23456789', + comment => 'This is a comment', + }, + { + name => 'Float', + class => 'scalar', + type => 'F32', + value => '0.98765', + comment => 'This is a comment too', + }, + { + name => 'String', + class => 'scalar', + type => 'STR', + value => 'This is the string that forms the value', + comment => 'comment', + }, + { + name => 'boolean', + class => 'scalar', + type => 'BOOL', + value => 1, + comment => q{The value of `boolean' is `true'}, + }, + { + name => 'primes', + class => 'vector', + type => 'U8', + value => [qw( 2 3 5 7 11 13 17 )], + comment => 'These are prime numbers', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'This', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'is', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'a', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'non-unique', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'key', + }, + ]; + + is_deeply( $config, $tree, "SDRS example structure, no overwrite" ); +} + +{ +my $example = q{ +Double F64 1.23456789 # This is a comment +Float F32 0.98765#This is a comment too +String STR This is the string that forms the value #comment + + # This is a comment line and is to be ignored +boolean BOOL T # The value of `boolean' is `true' + +@primes U8 2,3 5 7,11,13 17 # These are prime numbers + +comment MULTI # The rest of this line is ignored, but `comment' is set to be non-unique +comment STR This +comment STR is +comment STR a +comment STR non-unique +comment STR key +Float F64 1.23456 # This generates a warning, and, if `overwrite' is `false', is ignored +}; + $config_parser->overwrite( 1 ); + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "SDRS example parsed"); + + my $tree = [ + { + name => 'Double', + class => 'scalar', + type => 'F64', + value => '1.23456789', + comment => 'This is a comment', + }, + { + name => 'String', + class => 'scalar', + type => 'STR', + value => 'This is the string that forms the value', + comment => 'comment', + }, + { + name => 'boolean', + class => 'scalar', + type => 'BOOL', + value => 1, + comment => q{The value of `boolean' is `true'}, + }, + { + name => 'primes', + class => 'vector', + type => 'U8', + value => [qw( 2 3 5 7 11 13 17 )], + comment => 'These are prime numbers', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'This', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'is', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'a', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'non-unique', + }, + { + name => 'comment', + class => 'scalar', + type => 'STR', + value => 'key', + }, + { + name => 'Float', + class => 'scalar', + type => 'F64', + value => '1.23456', + comment => q{This generates a warning, and, if `overwrite' is `false', is ignored}, + }, + ]; + + is_deeply( $config, $tree, "SDRS example structure, with overwrite" ); +} + +{ +my $example = q{ +# these are examples of camera definition variables: +# skyprobe +NCELL S32 1 +NCHIP S32 1 +# FILENAME EXTNAME REGION CHIP BIASSEC +CELL.00 STR %f00.%x PHU [0,0:0,0] CHIP.00 [0,0:0,0] + +# megacam-raw +NCELL S32 72 +NCHIP S32 36 +# FILENAME EXTNAME DATASEC CHIP BIASSEC +CELL.00 STR %f.%x AMP00 [0,0:0,0] CHIP.00 BIASSEC +CELL.01 STR %f.%x AMP01 [0,0:0,0] CHIP.00 [2100,2110:0,4096] +CELL.02 STR %f.%x AMP02 [0,0:0,0] CHIP.01 [0,0:0,0] +CELL.03 STR %f.%x AMP03 [0,0:0,0] CHIP.01 [0,0:0,0] + +# megacam-splice +NCELL S32 72 +NCHIP S32 36 +# FILENAME EXTNAME REGION CHIP BIASSEC TRIMSEC +CELL.00 STR %f.%x CCD00 ASEC-00 CHIP.00 BSEC-00 DSEC-00 +CELL.01 STR %f.%x CCD00 ASEC-01 CHIP.00 BSEC-01 DSEC-01 +CELL.02 STR %f.%x CCD01 ASEC-00 CHIP.01 BSEC-00 DSEC-00 +CELL.03 STR %f.%x CCD01 ASEC-01 CHIP.01 BSEC-01 DSEC-01 + +# cfh12k-split +NCELL S32 12 +NCHIP S32 12 +# FILENAME EXTNAME REGION CHIP BIASSEC +CELL.00 STR %f/%f00.%x PHU [0,0:0,0] CHIP.00 [0,0:0,0] +CELL.01 STR %f/%f01.%x PHU [0,0:0,0] CHIP.01 [0,0:0,0] +CELL.02 STR %f/%f02.%x PHU [0,0:0,0] CHIP.02 [0,0:0,0] + +# cfh12k-mef +NCELL S32 12 +NCHIP S32 12 +# FILENAME EXTNAME REGION CHIP BIASSEC +CELL.00 STR %f.%x CHIP00 [0,0:0,0] CHIP.00 [0,0:0,0] +CELL.01 STR %f.%x CHIP01 [0,0:0,0] CHIP.01 [0,0:0,0] +CELL.02 STR %f.%x CHIP02 [0,0:0,0] CHIP.02 [0,0:0,0] + +#- REGION can be defined by a header keyword in IRAF format or by a explicit IRAF format +#- what is default for NAXIS1,2 for IRAF format? +#- Nreadout is always NAXIS3? + +# recipe file: +# this makes the assumption that, for a given camera, all chips & +# cells have the same recipe. this is probably a good start, but may +# not cut it in general. eg, it is already clear that for + +# recipe file must be a function of time and camera. +# + +# BIAS: +BIAS.IMAGE STR NONE +BIAS.IMAGE STR FILE:bias.fits +BIAS.IMAGE STR DB:BEST +BIAS.IMAGE STR DB:CLOSE + +BIAS.OVERSCAN STR HEADER:BIASSEC +BIAS.OVERSCAN STR RECIPE:[0,0:0,0] +BIAS.OVERSCAN STR NONE + +BIAS.OVERSCAN.STATS STR MEDIAN +BIAS.OVERSCAN.STATS STR MEAN + +BIAS.OVERSCAN.FIT STR SPLINE +BIAS.OVERSCAN.FIT.NPTS S32 5 + +BIAS.OVERSCAN.FIT STR POLYNOMIAL +BIAS.OVERSCAN.FIT.ORDER S32 3 +BIAS.OVERSCAN.FIT.NBIN S32 5 +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "GENE example"); +} + +{ +my $example = q{ +# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. + +# How to identify this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 72 +END + +# How to read this data +PHU STR FPA # The FITS file represents an entire FPA +EXTENSIONS STR CELLS # The extensions represent cells +EXTENSION_KEY STR EXTNAME # You get the extensions by looking at the EXTNAME header +IDENTIFIER STR OBSID # We identify the observation by the observation Id in the header + +# What's in the FITS file? +CONTENTS METADATA + TYPE CELL CHIP CELLTYPE + # Extension name, chip + amp00 CELL ccd00 science + amp01 CELL ccd00 science + amp02 CELL ccd01 science + amp03 CELL ccd01 science + guide CELL guide guide # A guide CCD thrown in, just for fun +END + +# Specify the cell data +CELLS METADATA + science METADATA # A science CCD + TYPE LOCATION SOURCE VALUE + BIASSEC LOCATION VALUE [1:10,1:4096];[1035:1084,1:4096] + TRIMSEC LOCATION VALUE [11:1034,1:4096] + # BIASSEC LOCATION HEADER BIASSEC + # TRIMSEC LOCATION HEADER TRIMSEC + END + guide METADATA # A guide CCD + TYPE LOCATION SOURCE VALUE + BIASSEC LOCATION VALUE [1:10,1:1024];[1035:1084,1:1024] + TRIMSEC LOCATION VALUE [11:1034,1:1024] + # BIASSEC LOCATION HEADER BIASSEC + # TRIMSEC LOCATION HEADER TRIMSEC + END + +END + + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + AIRMASS STR AIRMASS + EXPTIME STR EXPOSURE + DARKTIME STR EXPOSURE # No specific darktime header; use exposure time + FILTER STR FILTER + DATE-OBS STR DATE-OBS + TIME-OBS STR TIME-OBS + POSANGLE STR POSANG + RA STR OBJ-RA + DEC STR OBJ-DEC +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + RADECSYS STR ICRS +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS + GAIN dbEntry Camera gain chipId,cellId CHIP,CELL + READNOISE dbEntry Camera readNoise chipId,cellId CHIP,CELL + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). + +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "Paul/megacam example"); +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/03_metadata.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/03_metadata.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/03_metadata.t (revision 11645) @@ -0,0 +1,439 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 03_metadata.t,v 1.6 2006-11-25 01:15:47 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 13; +use PS::IPP::Metadata::Config; + +my $config_parser = PS::IPP::Metadata::Config->new; + +{ +my $example = q{ +CELL METADATA + EXTNAME STR CCD00 + BIASSEC STR BSEC-00 + CHIP STR CHIP.00 + NCELL S32 24 +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "SDRS example parsed"); + + my $tree = [ + { + name => 'CELL', + class => 'metadata', + value => [ + { + name => 'EXTNAME', + class => 'scalar', + type => 'STR', + value => 'CCD00' + }, + { + name => 'BIASSEC', + class => 'scalar', + type => 'STR', + value => 'BSEC-00' + }, + { + name => 'CHIP', + class => 'scalar', + type => 'STR', + value => 'CHIP.00' + }, + { + name => 'NCELL', + class => 'scalar', + type => 'S32', + value => 24, + }, + ], + } + ]; + + is_deeply( $config, $tree, "SDRS example structure"); +} + +{ +my $example = q{ +CELL METADATA # foo + EXTNAME STR CCD00 # bar + BIASSEC STR BSEC-00 # baz + CHIP STR CHIP.00 # zab + NCELL S32 24 # rab +END # oof +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "SDRS example + comments parsed"); + + my $tree = [ + { + name => 'CELL', + class => 'metadata', + value => [ + { + name => 'EXTNAME', + class => 'scalar', + type => 'STR', + value => 'CCD00', + comment => 'bar', + }, + { + name => 'BIASSEC', + class => 'scalar', + type => 'STR', + value => 'BSEC-00', + comment => 'baz', + }, + { + name => 'CHIP', + class => 'scalar', + type => 'STR', + value => 'CHIP.00', + comment => 'zab', + }, + { + name => 'NCELL', + class => 'scalar', + type => 'S32', + value => 24, + comment => 'rab', + }, + ], + comment => 'foo', + } + ]; + + is_deeply( $config, $tree, "SDRS example + comments structure"); +} + +{ +my $example = q{ +CELL METADATA + FOO METADATA + BAR STR BAZ + PING STR PONG + END + + EXTNAME STR CCD00 + BIASSEC STR BSEC-00 + CHIP STR CHIP.00 + NCELL S32 24 +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "nested metadata parsed"); + + my $tree = [ + { + name => 'CELL', + class => 'metadata', + value => [ + { + name => 'FOO', + class => 'metadata', + value => [ + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + { + name => 'EXTNAME', + class => 'scalar', + type => 'STR', + value => 'CCD00', + }, + { + name => 'BIASSEC', + class => 'scalar', + type => 'STR', + value => 'BSEC-00', + }, + { + name => 'CHIP', + class => 'scalar', + type => 'STR', + value => 'CHIP.00', + }, + { + name => 'NCELL', + class => 'scalar', + type => 'S32', + value => 24, + }, + ], + } + ]; + + is_deeply( $config, $tree, "nested metadata structure"); +} + +{ +my $example = q{ +FOO1 METADATA + FOO2 METADATA + FOO3 METADATA + FOO4 METADATA + FOO5 METADATA + FOO6 METADATA + BAR STR BAZ + PING STR PONG + END + BAR STR BAZ + PING STR PONG + END + BAR STR BAZ + PING STR PONG + END + BAR STR BAZ + PING STR PONG + END + BAR STR BAZ + PING STR PONG + END + BAR STR BAZ + PING STR PONG +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "deeply nested metadata"); + + my $tree = [ + { + name => 'FOO1', + class => 'metadata', + value => [ + { + name => 'FOO2', + class => 'metadata', + value => [ + { + name => 'FOO3', + class => 'metadata', + value => [ + { + name => 'FOO4', + class => 'metadata', + value => [ + { + name => 'FOO5', + class => 'metadata', + value => [ + { + name => 'FOO6', + class => 'metadata', + value => [ + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + { + name => 'BAR', + class => 'scalar', + type => 'STR', + value => 'BAZ', + }, + { + name => 'PING', + class => 'scalar', + type => 'STR', + value => 'PONG', + }, + ], + }, + ]; + + is_deeply( $config, $tree, "nested metadata structure"); +} + +{ +my $example = q{ +FOO1 METADATA + BAR STR BAZ + PING STR PONG + FOO2 METADATA + BAR STR BAZ + PING STR PONG + FOO3 METADATA + BAR STR BAZ + PING STR PONG + FOO4 METADATA + BAR STR BAZ + PING STR PONG + FOO5 METADATA + BAR STR BAZ + PING STR PONG + FOO6 METADATA + BAR STR BAZ + PING STR PONG + END + END + END + END + END +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "deeply nested metadata"); +} + +{ +my $example = q{ +FOO1 METADATA + BAR STR BAZ + FOO2 METADATA + BAR STR BAZ + FOO3 METADATA + BAR STR BAZ + FOO4 METADATA + BAR STR BAZ + FOO5 METADATA + BAR STR BAZ + FOO6 METADATA + BAR STR BAZ + PING STR PONG + END + PING STR PONG + END + PING STR PONG + END + PING STR PONG + END + PING STR PONG + END + PING STR PONG +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "deeply nested metadata"); +} + +{ +my $example = q{ +FOO1 METADATA + FOO2 METADATA + BAR STR BAZ + PING STR PONG + END + FOO3 METADATA + BAR STR BAZ + PING STR PONG + END +END +}; + my $config = $config_parser->parse( $example ); + + ok( defined( $config ), "two metadata at the same depth"); +} + +{ +my $example = q{ +foo METADATA +END +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + + ok( defined( $config ), "empty METADATA"); + + my $tree = [ + { + name => 'foo', + class => 'metadata', + value => [], + }, + ]; + + is_deeply( $config, $tree, "empty METADATA structure"); +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/04_type.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/04_type.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/04_type.t (revision 11645) @@ -0,0 +1,123 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 04_type.t,v 1.1 2005-03-23 01:53:55 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 9; +use PS::IPP::Metadata::Config; + +my $config_parser = PS::IPP::Metadata::Config->new; + +{ +my $example = q{ +TYPE CELL EXTNAME BIASSEC CHIP +CELL.00 CELL CCD00 BSEC-00 CHIP.00 +CELL.01 CELL CCD01 BSEC-01 CHIP.00 +}; + my $config = $config_parser->parse( $example ); + ok( defined( $config ), "basic TYPE"); +} + +{ +my $example = q{ +TYPE CELL EXTNAME BIASSEC CHIP # comment +CELL.00 CELL CCD00 BSEC-00 CHIP.00 # foo +CELL.01 CELL CCD01 BSEC-01 CHIP.00 # +}; + my $config = $config_parser->parse( $example ); + ok( defined( $config ), "TYPE with comments"); +} + +{ +my $example = q{ +TYPE CELL EXTNAME BIASSEC CHIP +CELL.00 CELL CCD00 BSEC-00 CHIP.00 +CELL.01 CELL CCD01 BSEC-01 CHIP.00 +FOO METADATA + TYPE CELL EXTNAME BIASSEC + CELL.00 CELL CCD00 BSEC-00 + CELL.01 CELL CCD01 BSEC-01 + FOO METADATA + TYPE CELL EXTNAME + CELL.00 CELL CCD00 + CELL.01 CELL CCD01 + END +END +}; + my $config = $config_parser->parse( $example ); + ok( defined( $config ), "TYPE not visible in lower scopes"); +} + +{ +my $example = q{ +TYPE CELL EXTNAME BIASSEC CHIP +FOO METADATA + CELL.00 CELL CCD00 BSEC-00 CHIP.00 +END +}; + my $config = $config_parser->parse( $example ); + ok( !defined( $config ), "TYPE not in scope"); +} + +{ +my $example = q{ +FOO METADATA + TYPE CELL EXTNAME BIASSEC CHIP +END +CELL.00 CELL CCD00 BSEC-00 CHIP.00 +}; + my $config = $config_parser->parse( $example ); + ok( !defined( $config ), "TYPE not in scope"); +} + +{ +my $example = q{ +FOO METADATA + TYPE CELL EXTNAME BIASSEC CHIP +END +BAR METADATA + CELL.00 CELL CCD00 BSEC-00 CHIP.00 +END +}; + my $config = $config_parser->parse( $example ); + ok( !defined( $config ), "TYPE not in scope"); +} + + +{ +my $example = q{ +TYPE CELL EXTNAME BIASSEC CHIP +CELL.00 CELL CCD00 BSEC-00 +CELL.01 CELL CCD01 BSEC-01 CHIP.00 +}; + my $config = $config_parser->parse( $example ); + ok( !defined( $config ), "TYPE with missing parameters"); +} + +{ +my $example = q{ +TYPE CELL EXTNAME BIASSEC CHIP +TYPE CELL EXTNAME BIASSEC CHIP +CELL.00 CELL CCD00 BSEC-00 CHIP.00 +CELL.01 CELL CCD01 BSEC-01 CHIP.00 +}; + my $config = $config_parser->parse( $example ); + ok( !defined( $config ), "TYPE redefinition"); +} + +{ +my $example = q{ +CELL.00 CELL CCD00 BSEC-00 CHIP.00 +CELL.01 CELL CCD01 BSEC-01 CHIP.00 +}; + my $config = $config_parser->parse( $example ); + ok( !defined( $config ), "missing TYPE declaration"); +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/05_time.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/05_time.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/05_time.t (revision 11645) @@ -0,0 +1,283 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 05_time.t,v 1.5 2006-11-22 23:54:50 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 15; +use PS::IPP::Metadata::Config; + +my $config_parser = PS::IPP::Metadata::Config->new; + +{ +my $example =<parse( $example ); + ok( defined( $config ), "NULL times parsed"); + + my $tree = [ + { + name => 'mytime', + class => 'scalar', + type => 'UTC', + value => undef, + }, + { + name => 'mytime', + class => 'scalar', + type => 'UT1', + value => undef, + }, + { + name => 'mytime', + class => 'scalar', + type => 'TAI', + value => undef, + }, + { + name => 'mytime', + class => 'scalar', + type => 'TT', + value => undef, + }, + ]; + + is_deeply( $config, $tree, "NULL times structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "NULL times with comments parsed"); + + my $tree = [ + { + name => 'mytime', + class => 'scalar', + type => 'UTC', + value => undef, + comment => 'comment', + }, + { + name => 'mytime', + class => 'scalar', + type => 'UT1', + value => undef, + comment => 'comment', + }, + { + name => 'mytime', + class => 'scalar', + type => 'TAI', + value => undef, + comment => 'comment', + }, + { + name => 'mytime', + class => 'scalar', + type => 'TT', + value => undef, + comment => 'comment', + }, + ]; + + is_deeply( $config, $tree, "NULL times with comments structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "basic ISO8601 parsed"); + + my $tree = [ + { + name => 'recently', + class => 'scalar', + type => 'UTC', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + { + name => 'recently', + class => 'scalar', + type => 'UT1', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + { + name => 'recently', + class => 'scalar', + type => 'TAI', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + { + name => 'recently', + class => 'scalar', + type => 'TT', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + ]; + + is_deeply( $config, $tree, "basic ISO8601 structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "basic ISO8601 w/Z parsed"); + + my $tree = [ + { + name => 'recently', + class => 'scalar', + type => 'UTC', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + { + name => 'recently', + class => 'scalar', + type => 'UT1', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + { + name => 'recently', + class => 'scalar', + type => 'TAI', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + { + name => 'recently', + class => 'scalar', + type => 'TT', + value => DateTime::Format::ISO8601->parse_datetime("2005-03-18T16:05:00Z"), + }, + ]; + + is_deeply( $config, $tree, "basic ISO8601 w/Z structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "basic IS8601 with fractional seconds parsed"); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "ISO8601 with comments parsed"); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "ISO8601 with comments and fractional seconds parsed"); +} + +#diag("epoch time format maybe deprecated"); +# +#{ +#my $example =<parse( $example ); +# ok( defined( $config ), "basic epoch time parsed"); +#} +# +#{ +#my $example =<parse( $example ); +# ok( defined( $config ), "epoch time with comments parsed"); +#} + +{ +my $example =<parse( $example ); + ok( !defined( $config ), "bad format"); +} + +{ +my $example =<parse( $example ); + ok( !defined( $config ), "bad format"); +} + +{ +my $example =<parse( $example ); + ok( !defined( $config ), "bad format"); +} + +{ +my $example =<parse( $example ); + ok( !defined( $config ), "bad format"); +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/06_multi.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/06_multi.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/06_multi.t (revision 11645) @@ -0,0 +1,346 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 06_multi.t,v 1.8 2006-10-11 02:49:05 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 15; +use PS::IPP::Metadata::Config; + +{ +my $example = q{ +foo MULTI +foo S8 -1 +foo STR bar baz +foo BOOL T +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + ok( defined( $config ), "basic MULTI"); +} + +{ +my $example = q{ +foo MULTI # foo +foo S8 -1 # bar +foo STR bar baz # baz +foo BOOL T # +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + ok( defined( $config ), "MULTI with comments"); +} + +{ +my $example = q{ +foo MULTI +foo S8 -1 +foo STR bar baz +foo BOOL T +bar METADATA + foo MULTI + foo S8 -1 + foo STR bar baz + foo BOOL T +END +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + ok( defined( $config ), "MULTI not visible in lower scopes"); +} + +{ +my $example = q{ +foo MULTI +foo METADATA + bar BOOL T +END +foo METADATA + bar BOOL T +END +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + + my $tree = [ + { + name => 'foo', + class => 'metadata', + value => [{ + name => 'bar', + class => 'scalar', + type => 'BOOL', + value => 1, + }], + }, + { + name => 'foo', + class => 'metadata', + value => [{ + name => 'bar', + class => 'scalar', + type => 'BOOL', + value => 1, + }], + }, + ]; + + is_deeply( $config, $tree, "MULTI METADATA structure, declared" ); +} + +{ +my $example = q{ +foo METADATA + bar BOOL T +END +foo METADATA + bar BOOL T +END +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + + my $tree = [ + { + name => 'foo', + class => 'metadata', + value => [{ + name => 'bar', + class => 'scalar', + type => 'BOOL', + value => 1, + }], + }, + ]; + + is_deeply( $config, $tree, "MULTI METADATA structure, not declared" ); +} + +{ +my $example = q{ +foo MULTI +TYPE bar a b c +foo bar x y z +foo bar x y z +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + ok( defined( $config ), "MULTI TYPE"); + + my $tree = [ + { + name => 'foo', + class => 'metadata', + value => [ + { + name => 'a', + class => 'scalar', + type => 'STR', + value => 'x', + }, + { + name => 'b', + class => 'scalar', + type => 'STR', + value => 'y', + }, + { + name => 'c', + class => 'scalar', + type => 'STR', + value => 'z', + }, + ], + }, + { + name => 'foo', + class => 'metadata', + value => [ + { + name => 'a', + class => 'scalar', + type => 'STR', + value => 'x', + }, + { + name => 'b', + class => 'scalar', + type => 'STR', + value => 'y', + }, + { + name => 'c', + class => 'scalar', + type => 'STR', + value => 'z', + }, + ], + }, + ]; + + is_deeply( $config, $tree, "MULTI TYPE structure, not declared" ); +} + +{ +my $example = q{ +TYPE bar a b c +foo MULTI +foo bar x y z +foo bar x y z +}; + + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + ok( defined( $config ), "MULTI TYPE"); + + my $tree = [ + { + name => 'foo', + class => 'metadata', + value => [ + { + name => 'a', + class => 'scalar', + type => 'STR', + value => 'x', + }, + { + name => 'b', + class => 'scalar', + type => 'STR', + value => 'y', + }, + { + name => 'c', + class => 'scalar', + type => 'STR', + value => 'z', + }, + ], + }, + { + name => 'foo', + class => 'metadata', + value => [ + { + name => 'a', + class => 'scalar', + type => 'STR', + value => 'x', + }, + { + name => 'b', + class => 'scalar', + type => 'STR', + value => 'y', + }, + { + name => 'c', + class => 'scalar', + type => 'STR', + value => 'z', + }, + ], + }, + ]; + + is_deeply( $config, $tree, "MULTI TYPE structure, not declared" ); +} + +{ +my $example = q{ +TYPE bar a b c +foo bar x y z +foo bar x y z +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + + my $tree = [ + { + name => 'foo', + class => 'metadata', + value => [ + { + name => 'a', + class => 'scalar', + type => 'STR', + value => 'x', + }, + { + name => 'b', + class => 'scalar', + type => 'STR', + value => 'y', + }, + { + name => 'c', + class => 'scalar', + type => 'STR', + value => 'z', + }, + ], + }, + ]; + + is_deeply( $config, $tree, "MULTI TYPE structure, not declared" ); +} + +{ +my $example = q{ +foo MULTI +bar METADATA + foo S8 -1 + foo STR bar baz +END +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + # ok because of overwrite + ok( defined( $config ), "MULTI not in scope"); +} + +{ +my $example = q{ +bar METADATA + foo MULTI +END +foo S8 -1 +foo STR bar baz +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + # ok because of overwrite + ok( defined( $config ), "MULTI not in scope"); +} + +{ +my $example = q{ +bar METADATA + foo MULTI +END +baz METADATA + foo S8 -1 +END +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + # ok because of overwrite + ok( defined( $config ), "MULTI not in scope"); +} + +{ +my $example = q{ +foo MULTI +foo MULTI +foo S8 -1 +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + ok( !defined( $config ), "MULTI redeclaration"); +} + +{ +my $example = q{ +foo S8 -1 +foo STR bar baz +}; + my $config = PS::IPP::Metadata::Config->new->parse( $example ); + # ok because of overwrite + ok( defined( $config ), "missing MULTI declaration"); +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/07_floats.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/07_floats.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/07_floats.t (revision 11645) @@ -0,0 +1,209 @@ +#!/usr/bin/perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 07_floats.t,v 1.1 2006-09-23 00:26:50 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 12; +use PS::IPP::Metadata::Config; + +my $config_parser = PS::IPP::Metadata::Config->new; + +{ +my $example =<parse( $example ); + ok( defined( $config ), "F32 nans parsed"); + + my $tree = [ + { + name => 'mynan', + class => 'scalar', + type => 'F32', + value => 'nan', + }, + { + name => 'mynan', + class => 'scalar', + type => 'F32', + value => 'nan', + }, + { + name => 'mynan', + class => 'scalar', + type => 'F32', + value => 'nan', + }, + { + name => 'mynan', + class => 'scalar', + type => 'F32', + value => 'nan', + }, + ]; + + is_deeply( $config, $tree, "F32 nans structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "F64 nans parsed"); + + my $tree = [ + { + name => 'mynan', + class => 'scalar', + type => 'F64', + value => 'nan', + }, + { + name => 'mynan', + class => 'scalar', + type => 'F64', + value => 'nan', + }, + { + name => 'mynan', + class => 'scalar', + type => 'F64', + value => 'nan', + }, + { + name => 'mynan', + class => 'scalar', + type => 'F64', + value => 'nan', + }, + ]; + + is_deeply( $config, $tree, "F64 nans structure" ); +} + +my $inftree = [ + # inf + { + name => 'myinf', + class => 'scalar', + type => 'F32', + value => '+inf', + }, + { + name => 'myinf', + class => 'scalar', + type => 'F32', + value => '+inf', + }, + { + name => 'myinf', + class => 'scalar', + type => 'F32', + value => '-inf', + }, + { + name => 'myinf', + class => 'scalar', + type => 'F32', + value => '+inf', + }, + { + name => 'myinf', + class => 'scalar', + type => 'F32', + value => '+inf', + }, + { + name => 'myinf', + class => 'scalar', + type => 'F32', + value => '-inf', + }, +]; + +{ +my $example =<parse( $example ); + ok( defined( $config ), "F32 infs parsed"); + + is_deeply( $config, $inftree, "F32 infs structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "F32 InFs parsed"); + + is_deeply( $config, $inftree, "F32 InFs structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "F32 iNfs parsed"); + + is_deeply( $config, $inftree, "F32 iNfs structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "F32 INFs parsed"); + + is_deeply( $config, $inftree, "F32 INFs structure" ); +} Index: /branches/rel-1-0/PS-IPP-Metadata-Config/t/08_strings.t =================================================================== --- /branches/rel-1-0/PS-IPP-Metadata-Config/t/08_strings.t (revision 11645) +++ /branches/rel-1-0/PS-IPP-Metadata-Config/t/08_strings.t (revision 11645) @@ -0,0 +1,75 @@ +#!/usr/bin/perl + +# Copyright (C) 2006 Joshua Hoblitt +# +# $Id: 08_strings.t,v 1.2 2006-11-04 01:25:32 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ); + +#$::RD_TRACE = 1; + +use Test::More tests => 4; +use PS::IPP::Metadata::Config; + +my $config_parser = PS::IPP::Metadata::Config->new; + +{ +my $example =<parse( $example ); + ok( defined( $config ), "strs parsed"); + + my $tree = [ + { + name => 'mystr', + class => 'scalar', + type => 'STR', + value => 'somevalue', + }, + { + name => 'mystr', + class => 'scalar', + type => 'STR', + value => 'somevaluebeforeacomment', + comment => 'some comment', + }, + ]; + + is_deeply( $config, $tree, "str structure" ); +} + +{ +my $example =<parse( $example ); + ok( defined( $config ), "NULL strs parsed"); + + my $tree = [ + { + name => 'mystr', + class => 'scalar', + type => 'STR', + value => undef, + }, + { + name => 'mystr', + class => 'scalar', + type => 'STR', + value => undef, + comment => 'some comment', + }, + ]; + + is_deeply( $config, $tree, "NULL str structure" ); +} Index: /branches/rel-1-0/dbconfig/.cvsignore =================================================================== --- /branches/rel-1-0/dbconfig/.cvsignore (revision 11645) +++ /branches/rel-1-0/dbconfig/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +ippdb.m4 +ippdb.mdc Index: /branches/rel-1-0/dbconfig/Makefile =================================================================== --- /branches/rel-1-0/dbconfig/Makefile (revision 11645) +++ /branches/rel-1-0/dbconfig/Makefile (revision 11645) @@ -0,0 +1,26 @@ +SHELL=/bin/sh +GLUEFORGE=`which glueforge` + +all: ippdb.mdc + +ippdb.mdc : ipp.m4 *.md + m4 ipp.m4 > ippdb.mdc + +## please leave the output target as ippdb.src and +## move to ippdb by hand for cvs import +install: ippdb.mdc + rm -rf ../ippdb.src + $(GLUEFORGE) -i ippdb.mdc --output ../ippdb.src + chmod +x ../ippdb.src/autogen.sh + +src: ippdb.mdc + $(GLUEFORGE) -i ippdb.mdc --output ../ippdb + chmod +x ../ippdb/autogen.sh + +foo: src + $(MAKE) -C ../ippdb install + +build : ippdb ippdb.mdc + +clean: + @rm -f *~ Index: /branches/rel-1-0/dbconfig/config.md =================================================================== --- /branches/rel-1-0/dbconfig/config.md (revision 11645) +++ /branches/rel-1-0/dbconfig/config.md (revision 11645) @@ -0,0 +1,5 @@ +glueforge METADATA + pkg_name STR ippdb + pkg_namespace STR ippdb + pkg_version STR 1.0.0 +END Index: /branches/rel-1-0/dbconfig/dimm.md =================================================================== --- /branches/rel-1-0/dbconfig/dimm.md (revision 11645) +++ /branches/rel-1-0/dbconfig/dimm.md (revision 11645) @@ -0,0 +1,21 @@ +# Table 13: DIMM Measurements Table +# Column Name Datatype Description +# Time date/time The time the DIMM observation was taken. +# sigmax float Raw dispersion in x. +# sigmay float Raw dispersion in y. +# FWHM float Dervied seeing full width at half maximum. +# RA float The coordinates of the measured star. +# DEC float The coordinates of the measured star. +# Exposure time float The exposure time of the DIMM observation. +# Telescope ID string source of the DIMM data + +dimm METADATA +# time DATETIME 2006-01-11T00:00:00 + sigmax F32 0.0 + sigmay F32 0.0 + fwhm F32 0.0 + ra F64 0.0 + decl F64 0.0 + expttime F32 0.0 + telescope_id STR 255 +END Index: /branches/rel-1-0/dbconfig/dome.md =================================================================== --- /branches/rel-1-0/dbconfig/dome.md (revision 11645) +++ /branches/rel-1-0/dbconfig/dome.md (revision 11645) @@ -0,0 +1,16 @@ +# Table 15: Dome Status Table +# Column Name Datatype Description +# Time date/time The time for which the dome status is valid. +# Azimuth float The azimuth of the dome. +# Open status boolean Whether the dome is open or not. +# Lights status boolean Whether lights are on in the dome or not. +# Track status boolean Whether dome is tracking telescope or not. + +dome METADATA +# time DATETIME 2006-01-11T00:00:00 + az F32 0.0 + open BOOL t + light BOOL t + # XXX is it possible for the dome slit to not track the telescope? ;) + track BOOL t +END Index: /branches/rel-1-0/dbconfig/ipp.m4 =================================================================== --- /branches/rel-1-0/dbconfig/ipp.m4 (revision 11645) +++ /branches/rel-1-0/dbconfig/ipp.m4 (revision 11645) @@ -0,0 +1,11 @@ +include(config.md) +dnl include(weather.md) +dnl include(skyp_transparency.md) +dnl include(skyp_absorption.md) +dnl include(skyp_emission.md) +dnl include(dimm.md) +dnl include(skyp_ir.md) +dnl include(dome.md) +dnl include(telescope.md) +include(tasks.md) +include(p4.md) Index: /branches/rel-1-0/dbconfig/p4.md =================================================================== --- /branches/rel-1-0/dbconfig/p4.md (revision 11645) +++ /branches/rel-1-0/dbconfig/p4.md (revision 11645) @@ -0,0 +1,82 @@ +# $Id: p4.md,v 1.5 2007-01-19 00:11:41 jhoblitt Exp $ + +# define a new p4run for a single skycell +p4Run METADATA + p4_id S32 0 # Primary Key AUTO_INCREMENT + survey_mode STR 64 # Key + state STR 64 # Key + workdir STR 255 + filter STR 64 + skycell_id STR 64 + tess_id STR 64 + ra F64 0.0 + decl F64 0.0 + input_ss STR 64 + output_ss STR 64 +END + +# the list of imfiles in our skycell keyed against phase 3 +p4InputImfile METADATA + p4_id S32 0 # Primary Key + exp_tag STR 64 # Primary Key + p3_version S32 0 # Primary Key +# class is not yet consistently carried through pXtools +# class STR 64 # Primary Key + class_id STR 64 # Primary Key +END + +p4WarpedImfile METADATA + p4_id S32 0 # Primary Key + exp_tag STR 64 # Primary Key + p3_version S32 0 # Primary Key +# class is not yet consistently carried through pXtools +# class STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 +# bg F64 0.0 +# bg_stdev F64 0.0 +# bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 +END + +p4StackedImfile METADATA + p4_id S32 0 # Primary Key +# class is not yet consistently carried through pXtools +# class STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 +# bg F64 0.0 +# bg_stdev F64 0.0 +# bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 +END + +# one diff image per input to the current stack +# XXX how will diff against the static sky be handled? +p4DiffImfile METADATA + p4_id S32 0 # Primary Key + exp_tag STR 64 # Primary Key + p3_version S32 0 # Primary Key +# class is not yet consistently carried through pXtools +# class STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 +# bg F64 0.0 +# bg_stdev F64 0.0 +# bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 +END + +# output from magic +p4MagicMaskImfile METADATA + p4_id S32 0 # Primary Key + exp_tag STR 64 # Primary Key + p3_version S32 0 # Primary Key +# class is not yet consistently carried through pXtools +# class STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 +END Index: /branches/rel-1-0/dbconfig/skyp_absorption.md =================================================================== --- /branches/rel-1-0/dbconfig/skyp_absorption.md (revision 11645) +++ /branches/rel-1-0/dbconfig/skyp_absorption.md (revision 11645) @@ -0,0 +1,25 @@ +# Table 11: Skyprobe Line Absorption Table (sample entries) +# Column Name Datatype Description +# Time date/time The time the LRProbe observation was taken. +# Disperser ID string ID of the dispersing element +# Atm Component 1 float The strength of the 1st atmospheric component. +# Atm Component 2 float The strength of the 2nd atmospheric component. +# Atm Component 3 float The strength of the 3rd atmospheric component. +# Disperser ID string ID of the dispersing element +# Number of stars int Number of stars used to measure the absorptions. +# Astrometry coords The astrometry used on the LRProbe image. +# Exposure time float The exposure time of the LRProbe image. +# Sky brightness float The measured sky (surface) brightness, in physical units. + +skyp_absorption METADATA +# time DATETIME 2006-01-10T00:00:00 + disperser_id STR 255 + atmcomp1 F32 0.0 + atmcomp2 F32 0.0 + atmcomp3 F32 0.0 + nstars S32 0 + ra F64 0.0 + decl F64 0.0 + exptime F32 0.0 + sky_bright F64 0.0 +END Index: /branches/rel-1-0/dbconfig/skyp_emission.md =================================================================== --- /branches/rel-1-0/dbconfig/skyp_emission.md (revision 11645) +++ /branches/rel-1-0/dbconfig/skyp_emission.md (revision 11645) @@ -0,0 +1,20 @@ +# Table 12: Skyprobe Line Emission Table (sample entries) +# Column Name Datatype Description +# Time date/time The time the LRProbe observation was taken. +# Disperser ID string ID of the dispersing element +# Atm Component 1 float The strength of the 1st atmospheric component. +# Atm Component 2 float The strength of the 2nd atmospheric component. +# Atm Component 3 float The strength of the 3rd atmospheric component. +# Continuum float The strength of the continuum emission. +# Disperser ID string ID of the dispersing element +# Exposure time float The exposure time of the LRProbe image. + +skyp_emission METADATA +# time DATETIME 2006-01-11T00:00:00 + disperser_id STR 255 + atmcomp1 F32 0.0 + atmcomp2 F32 0.0 + atmcomp3 F32 0.0 + continuum F32 0.0 + exptime F32 0.0 +END Index: /branches/rel-1-0/dbconfig/skyp_ir.md =================================================================== --- /branches/rel-1-0/dbconfig/skyp_ir.md (revision 11645) +++ /branches/rel-1-0/dbconfig/skyp_ir.md (revision 11645) @@ -0,0 +1,18 @@ +# Table 14: Near IR Wide-field Camera Results Table +# Column Name Datatype Description +# Time date/time The time the NIR observation was taken. +# Sky brightness float The sky (surface) brightness in the NIR observation. +# Sky variance float The variance in the sky (surface) brightness. +# Astrometry coords The astrometry used on the NIR image. +# FOV X float field width +# FOV Y float field height + +skyp_ir METADATA +# time DATETIME 2006-01-11T00:00:00 + sky_bright F64 0.0 + sky_var F64 0.0 + ra F64 0.0 + decl F64 0.0 + fov_x F32 0.0 + fov_y F32 0.0 +END Index: /branches/rel-1-0/dbconfig/skyp_transparency.md =================================================================== --- /branches/rel-1-0/dbconfig/skyp_transparency.md (revision 11645) +++ /branches/rel-1-0/dbconfig/skyp_transparency.md (revision 11645) @@ -0,0 +1,21 @@ +# Table 10: SkyProbe Transparency Table (sample entries) +# Column Name Datatype Description +# Time date/time The time the SkyProbe image was taken. +# Filter string Filter used for SkyProbe image. +# Transparency float The derived transparency. +# Number of stars int The number of stars used to measure the transparency. +# Astrometry coords The astrometry used on the SkyProbe image. +# Exposure time float The exposure time of the SkyProbe image. +# Sky brightness float The measured sky (surface) brightness, counts / second + +skyp_transparency METADATA +# time DATETIME 2006-01-10T00:00:00 # Primary Key + filter STR 255 + trans F64 0.0 + nstars S32 0 +# astrom + ra F64 0.0 + decl F64 0.0 + exptime F32 0.0 + sky_bright F64 0.0 +END Index: /branches/rel-1-0/dbconfig/tasks.md =================================================================== --- /branches/rel-1-0/dbconfig/tasks.md (revision 11645) +++ /branches/rel-1-0/dbconfig/tasks.md (revision 11645) @@ -0,0 +1,403 @@ +# $Id: tasks.md,v 1.115 2007-01-29 17:42:51 eugene Exp $ + +# this table records all exposure ID ever seen from the summit +# exp_id == fileset + +# note that dec is a MySQL reserved word +# note that use is a MySQL reserved word +# note that exp is a MySQL reserved word + +# for use with this stored procedure to generate exp_tags under Mysql 5+ +# +#CREATE FUNCTION genTag (exp_id varchar(64)) RETURNS VARCHAR(64) +#BEGIN +# UPDATE expTagCounter SET counter = LAST_INSERT_ID(counter + 1); +# RETURN CONCAT(exp_id, '.', LAST_INSERT_ID()); +#END +# +expTagCounter METADATA + counter U64 0 +END + +# list of source exposures -- updated as exposures are seen +# summitExp.imfiles is updated as filesets are queried default value should be +# -1 as NULL or 0 might be a valid value (empty fileset) +summitExp METADATA + exp_id STR 64 # Primary Key + camera STR 64 # Primary Key + telescope STR 64 # Primary Key + dateobs UTC NULL + exp_type STR 64 + uri STR 255 + imfiles S32 0 +END + +# class == type of file +# class_id == type set id +# list of source images -- updated as exposures/filesets are queried +summitImfile METADATA + exp_id STR 64 # Primary Key + camera STR 64 # Primary Key + telescope STR 64 # Primary Key + file_id STR 64 # Key + bytes S32 0 + md5sum STR 32 + class STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 +END + +# list of exposures that have had their imfiles/files registered (but not +# downloaded) +pzPendingExp METADATA + exp_id STR 64 # Primary Key + camera STR 64 # Primary Key + telescope STR 64 # Primary Key +END + +pzPendingImfile METADATA + exp_id STR 64 # Primary Key + camera STR 64 # Primary Key + telescope STR 64 # Primary Key + class STR 64 # Primary Key + class_id STR 64 # Primary Key + exp_tag STR 64 # Unique Key +END + +# list of exposures that have had all of their imfiles downloaded +pzDoneExp METADATA + exp_id STR 64 # Primary Key + camera STR 64 # Primary Key + telescope STR 64 # Primary Key +END + +pzDoneImfile METADATA + exp_id STR 64 # Primary Key + camera STR 64 # Primary Key + telescope STR 64 # Primary Key + class STR 64 # Primary Key + class_id STR 64 # Primary Key + exp_tag STR 64 # Unique Key + uri STR 255 +END + +newExp METADATA + exp_tag STR 64 # Primary Key + exp_id STR 64 # Key + camera STR 64 # Key + telescope STR 64 # Key + imfiles S32 0 + workdir STR 255 # destination for output files +END + +# class needs to be carried here so it can go into rawImfile and be normalized +# from there +newImfile METADATA + exp_tag STR 64 # Primary Key + class STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 +END + +# paired with rawImfile +rawExp METADATA + exp_tag STR 64 # Primary Key + exp_id STR 64 # Key + camera STR 64 + telescope STR 64 + dateobs UTC 0001-01-01T00:00:00Z + exp_type STR 64 + imfiles S32 0 + workdir STR 255 # destination for output files + filter STR 64 + airmass F32 0.0 + ra F64 0.0 + decl F64 0.0 + exp_time F32 0.0 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + alt F64 0.0 + az F64 0.0 + ccd_temp F32 0.0 + posang F64 0.0 + object STR 64 + fault S16 0 # NOT NULL +END + +rawImfile METADATA + exp_tag STR 64 # Primary Key + class STR 64 + class_id STR 64 # Primary Key + uri STR 255 + exp_type STR 64 + filter STR 64 + airmass F32 0.0 + ra F64 0.0 + decl F64 0.0 + exp_time F32 0.0 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + alt F64 0.0 + az F64 0.0 + ccd_temp F32 0.0 + posang F64 0.0 + object STR 64 + dateobs UTC 0001-01-01T00:00:00Z + fault S16 0 # NOT NULL +END + +# +# XXX - temporarily (???) stot carrying around class in imfiles after rawImfile +# + +p1PendingExp METADATA + exp_tag STR 64 # Primary Key + recipe STR 64 + p1_version S32 0 +END + +# do we need nclass? +# class is missing + +p2PendingExp METADATA + exp_tag STR 64 # Primary Key + recipe STR 64 + p1_version S32 0 + p2_version S32 0 + label STR 64 # key +END + +# uris in & out ? + +p2PendingImfile METADATA + exp_tag STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 + recipe STR 64 + p1_version S32 0 + p2_version S32 0 +END + +# nclass & ndone? +# and/or logging imfile done better? + +p2ProcessedExp METADATA + exp_tag STR 64 # Primary Key + p1_version S32 0 + p2_version S32 0 + label STR 64 # key +END + +p2Mask METADATA + label STR 64 # Primary Key +END + +# uris in & out ? +# state? + +p2ProcessedImfile METADATA + exp_tag STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + p1_version S32 0 + p2_version S32 0 + fault S16 0 # NOT NULL +END + +# imfiles is not here... is it needed? + +p3PendingExp METADATA + exp_tag STR 64 # Primary Key + p2_version S32 0 + p3_version S32 0 + label STR 64 # key +END + +p3ProcessedExp METADATA + exp_tag STR 64 # Primary Key + uri STR 255 + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + sigma_ra F32 0.0 + sigma_dec F32 0.0 + nastro S32 0 + b1_uri STR 255 + b2_uri STR 255 + zp_mean F32 0.0 + zp_stdev F32 0.0 + p2_version S32 0 + p3_version S32 0 + label STR 64 # key + fault S16 0 # NOT NULL +END + +p3Mask METADATA + label STR 64 # Primary Key +END + +detRun METADATA + det_id S32 0 # Primary Key AUTO_INCREMENT + iteration S32 0 # Key + det_type STR 64 # Key + mode STR 64 # Key + state STR 64 # Key + workdir STR 255 # destination for output files + camera STR 64 + telescope STR 64 + exp_type STR 64 # XXX this should be dropped + filter STR 64 + airmass_min F32 0.0 + airmass_max F32 0.0 + exp_time_min F32 0.0 + exp_time_max F32 0.0 + ccd_temp_min F32 0.0 + ccd_temp_max F32 0.0 + posang_min F64 0.0 + posang_max F64 0.0 + registered TAI 0001-01-01T00:00:00Z + time_begin TAI 0001-01-01T00:00:00Z + time_end TAI 0001-01-01T00:00:00Z + use_begin TAI 0001-01-01T00:00:00Z + use_end TAI 0001-01-01T00:00:00Z +END + +detInputExp METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + exp_tag STR 64 # Primary Key + include BOOL f +END + +detProcessedImfile METADATA + det_id S32 0 # Primary Key + exp_tag STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + fault S16 0 # NOT NULL +END + +detProcessedExp METADATA + det_id S32 0 # Primary Key + exp_tag STR 64 # Primary Key + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + fault S16 0 # NOT NULL +END + +detStackedImfile METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + fault S16 0 # NOT NULL +END + +detNormalizedStatImfile METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + class_id STR 64 # Primary Key + norm F32 0.0 + fault S16 0 # NOT NULL +END + +detNormalizedImfile METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + fault S16 0 # NOT NULL +END + +detNormalizedExp METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + fault S16 0 # NOT NULL +END + +#detMasterFrame METADATA +# det_id S32 0 # Primary Key +# iteration S32 0 # Primary Key +# comment STR 255 +#END +# +## drop? +#detMasterImfile METADATA +# det_id S32 0 # Primary Key +# class_id STR 64 # Primary Key +# uri STR 255 +# recipe STR 64 +#END + +detResidImfile METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + exp_tag STR 64 # Primary Key + class_id STR 64 # Primary Key + uri STR 255 + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + fault S16 0 # NOT NULL +END + +detResidExp METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + exp_tag STR 64 # Primary Key + recipe STR 64 + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + b1_uri STR 255 + b2_uri STR 255 + accept BOOL f + fault S16 0 # NOT NULL +END + +detRunSummary METADATA + det_id S32 0 # Primary Key + iteration S32 0 # Primary Key + bg F64 0.0 + bg_stdev F64 0.0 + bg_mean_stdev F64 0.0 + accept BOOL f + fault S16 0 # NOT NULL +END Index: /branches/rel-1-0/dbconfig/telescope.md =================================================================== --- /branches/rel-1-0/dbconfig/telescope.md (revision 11645) +++ /branches/rel-1-0/dbconfig/telescope.md (revision 11645) @@ -0,0 +1,18 @@ +# Table 16: Telescope Status +# Column Name Datatype Description +# Time date/time The time for which the telescope status is valid. +# Guide status enum The status of the guiding. +# Altitude float The telescope altitude. +# Azimuth float The telescope azimuth. +# RA float The telescope Right Ascension (ICRS ~ J2000). +# Dec float The telescope Declination (ICRS ~ J2000). + +telescope METADATA +# time DATETIME 2006-01-11T00:00:00 # Primary Key +# XXX there is currently no way to declare an enum - use str or int instead? + guide STR 255 + alt F32 0.0 + az F32 0.0 + ra F64 0.0 + decl F64 0.0 +END Index: /branches/rel-1-0/dbconfig/weather.md =================================================================== --- /branches/rel-1-0/dbconfig/weather.md (revision 11645) +++ /branches/rel-1-0/dbconfig/weather.md (revision 11645) @@ -0,0 +1,19 @@ +# Table 9: Weather Table: some sample weather points +# Column Name Datatype Description +# Time date/time The time the weather information was measured. +# Temperature 01 float The external temperature +# Temperature 02 float The temperature at top of the dome +# Temperature 03 float The temperature on the primary mirror +# Humidity float The relative humidity. +# Pressure float The (external) atmospheric pressure. + +weather METADATA +# time DATETIME 2006-01-10T00:00:00 # Primary Key + temp01 F32 0.0 + humi01 F32 0.0 + temp02 F32 0.0 + humi02 F32 0.0 + temp03 F32 0.0 + humi03 F32 0.0 + pressure F32 0.0 +END Index: /branches/rel-1-0/glueforge/.cvsignore =================================================================== --- /branches/rel-1-0/glueforge/.cvsignore (revision 11645) +++ /branches/rel-1-0/glueforge/.cvsignore (revision 11645) @@ -0,0 +1,11 @@ +Makefile +Makefile.in +aclocal.m4 +configure +autom4te.cache +config.log +config.status +install-sh +missing +glueforge +glueforge.1 Index: /branches/rel-1-0/glueforge/INSTALL =================================================================== --- /branches/rel-1-0/glueforge/INSTALL (revision 11645) +++ /branches/rel-1-0/glueforge/INSTALL (revision 11645) @@ -0,0 +1,95 @@ +# $Id: INSTALL,v 1.6 2006-01-11 02:12:34 jhoblitt Exp $ + +Using the glueforge meta-database API generator +-- + +- Step 0 + +Setup your shell's environment for Pan-STARRS CVS. If you need help setting up +CVS please see this article: + + http://pan-starrs.ifa.hawaii.edu/project/resources/ps-cvs.html + +- Step 1 + +Install the PS:IPP::Metadata::Config and Template modules. + +# PS:IPP::Metadata::Config +$ cvs co PS-IPP-Metadata-Config +$ cd PS-IPP-Metadata-Config +$ perl Build.PL +$ ./Build +$ ./Build test +# if you have sudo +$ sudo ./Build install +# if not +$ su +$ ./Build install + +# Template +# if you have sudo +$ sudo perl -MCPAN -e 'install Template' +# if not +$ su - +$ perl -MCPAN -e 'install Template' + +If you have never used CPAN before you will be given a series of prompts to +configure the module. + +- Step 2 + +Checkout the glueforge module from Pan-STARRS CVS and build it. If you need +help setting up CVS please see this article: + + http://pan-starrs.ifa.hawaii.edu/project/resources/ps-cvs.html + +$ cvs co glueforge +$ cd glueforge +$ ./autogen.sh +$ make + +- Step 3 + +Create a MetadataConfig file that describes the table you want to access. The +MetadataConfig file syntax is described in the pslib SDRS. glueforge specific +details are included in the "glueforge" man page. For reference, both a simple +and complex config example is provided in the examples directory. + +$ perldoc glueforge + +or + +$ man glueforge + +$ cat examples/simple.md + +or + +$ cat examples/complex.md + +- Step 4 + +Generate the database bindings with the "glueforge" executable. The example +simple.md file will generate code under the directory "foodb". + +$./glueforge -t templates/psdb -i examples/simple.md +$ cd foodb + +- Step 5 + +Package the software for distribution. This is the same as for any 'standard' +autotools package. Please note that you must have pslib installed for the step +to complete. If you need help installing pslib please see this article: + + http://pan-starrs.ifa.hawaii.edu/project/IPP/software/jhbuild/ + +$ jhbuild shell +$ ./autogen.sh && make dist + +This should generate two packages suitable for distribution. + +$ ls -la foodb-* +-rw-r--r-- 1 jhoblitt users 216367 Jul 28 17:54 foodb-0.0.1.tar.bz2 +-rw-r--r-- 1 jhoblitt users 324926 Jul 28 17:54 foodb-0.0.1.tar.gz + +The End. Index: /branches/rel-1-0/glueforge/Makefile.am =================================================================== --- /branches/rel-1-0/glueforge/Makefile.am (revision 11645) +++ /branches/rel-1-0/glueforge/Makefile.am (revision 11645) @@ -0,0 +1,37 @@ +SUBDIRS = templates/psdb templates/latex + +bin_SCRIPTS = glueforge +CLEANFILES = $(bin_SCRIPTS) + +do_subst = sed \ + -e 's,[@]datadir[@],$(datadir),g' \ + -e 's,[@]PACKAGE_NAME[@],$(PACKAGE_NAME),g' \ + -e 's,[@]PACKAGE_VERSION[@],$(PACKAGE_VERSION),g' + +glueforge: glueforge.in + $(do_subst) < $(srcdir)/glueforge.in > glueforge + chmod +x glueforge + +example_files = \ + examples/simple.md \ + examples/complex.md + +glueforgedatadir = $(datadir)/@PACKAGE_NAME@ +exampledir = $(glueforgedatadir)/examples +example_DATA = $(example_files) + +install-data-local: + -chmod 0755 $(glueforgedatadir) $(exampledir) + +man_MANS = \ + $(top_builddir)/@PACKAGE_NAME@.1 + +@PACKAGE@.1: glueforge + $(POD2MAN) glueforge > @PACKAGE@.1 + +clean-local: + -rm -rf docs @PACKAGE@.1 + +EXTRA_DIST = autogen.sh glueforge.in $(example_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/glueforge/autogen.sh =================================================================== --- /branches/rel-1-0/glueforge/autogen.sh (revision 11645) +++ /branches/rel-1-0/glueforge/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=glueforge +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOlIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +#($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $AUTOHEADER installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" +# DIE=1 +#} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +#$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL -I m4 || echo "$ACLOCAL failed" +#$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/glueforge/configure.ac =================================================================== --- /branches/rel-1-0/glueforge/configure.ac (revision 11645) +++ /branches/rel-1-0/glueforge/configure.ac (revision 11645) @@ -0,0 +1,31 @@ +AC_PREREQ(2.59) + +AC_INIT([glueforge], [1.00], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([glueforge.in]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_MAINTAINER_MODE + +AC_PROG_INSTALL + +AC_PROG_PERL_MODULES( + [PS::IPP::Metadata::Config], , + [AC_MSG_ERROR(perl module PS::IPP::Metadata::Config is required)] +) + +AC_PROG_PERL_MODULES( + [Template], , + [AC_MSG_ERROR(perl module Template is required)] +) + +AC_PATH_PROG([POD2MAN], [pod2man], [missing]) +if test "$POD2MAN" = "missing" ; then + AC_MSG_ERROR([pod2man is required]) +fi + +AC_CONFIG_FILES([ + Makefile + templates/latex/Makefile + templates/psdb/Makefile +]) +AC_OUTPUT Index: /branches/rel-1-0/glueforge/examples/complex.md =================================================================== --- /branches/rel-1-0/glueforge/examples/complex.md (revision 11645) +++ /branches/rel-1-0/glueforge/examples/complex.md (revision 11645) @@ -0,0 +1,20 @@ +glueforge METADATA + pkg_name STR foodb + pkg_namespace STR foo +END + +foo METADATA + foo STR 60 # Primary Key # the name of foo thing + bar S32 0 ## count of bar + baz F32 0.0 + boing F64 0.0 + zot BOOL t # Key +END + +bar METADATA + zot BOOL t # Key + boing F64 0.0 + baz F32 0.0 + bar S32 0 ## count of bar + foo STR 60 # Primary Key # the name of foo thing +END Index: /branches/rel-1-0/glueforge/examples/simple.md =================================================================== --- /branches/rel-1-0/glueforge/examples/simple.md (revision 11645) +++ /branches/rel-1-0/glueforge/examples/simple.md (revision 11645) @@ -0,0 +1,6 @@ +table STR foo +foo STR 60 # Primary Key # the name of foo thing +bar S32 0 ## count of bar +baz F32 0.0 +boing F64 0.0 +zot BOOL t # Key Index: /branches/rel-1-0/glueforge/glueforge.in =================================================================== --- /branches/rel-1-0/glueforge/glueforge.in (revision 11645) +++ /branches/rel-1-0/glueforge/glueforge.in (revision 11645) @@ -0,0 +1,513 @@ +#!/usr/bin/env perl + +# Copyright (C) 2005-2006 Joshua Hoblitt +# +# $Id: glueforge.in,v 1.51 2006-10-20 00:55:43 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use vars qw( $VERSION ); +$VERSION = '@PACKAGE_VERSION@'; + +use PS::IPP::Metadata::Config; +use Template; + +use Getopt::Long qw( GetOptions :config auto_help auto_version ); +use Pod::Usage qw( pod2usage ); + +my ( $input, $output, $template ); +GetOptions( + 'input=s' => \$input, + 'output=s' => \$output, + 'template=s' => \$template, +) || pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( -msg => "Required option: --input|-i", -exitval => 2 ) unless $input; + +$template = "@datadir@/@PACKAGE_NAME@/templates/psdb" unless $template; + +die "file doesn't exist: $input" unless -e $input; +die "file isn't readable: $input" unless -r $input; + +my $config = parse_config( $input ); + +die "file doesn't not contain a valid config" unless defined $config->[0]; + +# if the first hash has a class of "metadata" then this is a complex config. +# Otherwise it is a simple config. +my $complex = ($config->[0]->{class} eq "metadata") ? 1 : 0; + +my $data; # hashref +if ($complex) { + $data = complex_config($config); +} else { + $data = simple_config($config); +} + +# determine output path +$output = $data->{pkg_name} unless defined $output; + +my $ttconfig; +{ + # slurp config.pl + open(my $fh, "$template/config.pl") + or die "can't open $template/config.pl: $!"; + $ttconfig = do {local $/; <$fh>}; + close($fh) or die "can't close $template/config.pl: $!"; +} + +# configure template file paths +my %tt; +eval $ttconfig; +die "can't eval contents of $template/config.pl: $@" if $@; + +my $mangler = Template->new({ INCLUDE_PATH => $template, EVAL_PERL => 1 }); + +foreach my $t ( keys %tt ) { + $mangler->process( $t, $data, $tt{$t}) + or die $mangler->error; +} + +# make autogen.sh executable +# permissions are in octal +#chmod 0744, $tt{'autogen_sh.tt'}; + +sub parse_config { + my $file = shift; + + my $mdparser = PS::IPP::Metadata::Config->new; + $mdparser->overwrite( 1 ); + + open( my $data, $file ) or die "can't open file: $!"; + my $example = do { local $/; <$data> }; + close( $data ) or die "can't close file: $!"; + + my $config = $mdparser->parse( $example ) + or die "error parsing file: $input"; + + return $config; +} + +sub lookup_type { + my $type = shift; + + my %primitives = map { $_ => 1 } + qw( S8 S16 S32 S64 U8 U16 U32 U64 F32 F64 ); + my %times = map { $_ => 1 } qw( UTC UT1 TAI TT ); + my %mtypes = (map({ $_ => "PS_DATA_$_" } keys %primitives), + map({ $_ => "PS_DATA_TIME" } keys %times)); + my %ctypes = (map({ $_ => "ps$_" } keys %primitives), + map({ $_ => "psTime*"} keys %times)); + + my %ttypes = ( + S8 => -8, + S16 => -16, + S32 => -32, + S64 => -64, + U8 => 8, + U16 => 16, + U32 => 32, + U64 => 64, + F32 => 32.32, + F64 => 64.64, + UTC => '"0001-01-01T00:00:00Z"', + UT1 => '"0001-01-01T00:00:00Z"', + TAI => '"0001-01-01T00:00:00Z"', + TT => '"0001-01-01T00:00:00Z"', + ); + + if ($primitives{$type}) { + return { + ctype => $ctypes{$type}, + mtype => $mtypes{$type}, + test => $ttypes{$type}, + }; + } + elsif ($times{$type}) { + return { + ctype => $ctypes{$type}, + mtype => $mtypes{$type}, + test => $ttypes{$type}, + timetype => "PS_TIME_$type", + }; + } elsif ( $type =~ /STR|STRING/ ) { + return { + ctype => "char*", + mtype => 'PS_DATA_STRING', + test => '"a string"', + }; + } elsif ( $type =~ /BOOL/ ) { + return { + ctype => "bool", + mtype => 'PS_DATA_BOOL', + test => 'true', + }; + } else { + return undef; + } +} + +sub simple_config { + my $config = shift; + + # find table name + my $table_name; + for (my $i = 0; $i < @{$config}; $i++) { + my $elem = $config->[$i]; + + if ( $elem->{name} eq "table" ) { + $table_name = $elem->{value}; + + # delete element from array + splice @$config, $i, 1; + + last; + } + } + + my %data; + + # global data + $data{glueforge_version}= $VERSION; + + $data{pkg_name} = $table_name . "db"; + $data{pkg_namespace} = $table_name; + + # per table data + my %table; + $table{name} = $table_name; + $table{namespace} = $table_name; + $table{object_name} = $table{namespace} . "Row"; + + my @items; + + # setup one table per nested metadata + my @tables; + foreach my $meta (@{$config}) { + push @tables, tabledata_from_meta($meta); + } + +=for off + # setup items + for (my $i = 0; $i < @{$config}; $i++) { + my $elem = $config->[$i]; + + my $mdtypes = lookup_type( $elem->{type} ); + next if ! defined $mdtypes; + + push @items, { + %{$mdtypes}, + name => $elem->{name}, + type => $elem->{type}, +# ctype => $mdtypes->{ctype}, +# mtype => $mdtypes->{mtype}, +# test => $mdtypes->{test}, + comment => $elem->{comment}, + value => $elem->{value}, + }; + } +=cut + + $table{columns} = \@items; + $data{tables} = [ \%table ]; + + return \%data; +} + +sub complex_config { + my $config = shift; + + # find the glueforge metadata + my $glueforge_meta; + for (my $i = 0; $i < @{$config}; $i++) { + my $elem = $config->[$i]; + + if ($elem->{name} eq "glueforge") { + $glueforge_meta = $elem; + + # delete element from array + splice @$config, $i, 1; + + last; + } + } + + # global data + + my ($pkg_name, $pkg_namespace, $pkg_version); + unless (defined $glueforge_meta) { + # set default values + warn "missing glueforge METADATA, using default values"; + $pkg_name = "foodb"; + $pkg_namespace = "foo"; + } else { + # process glueforge metadata for global data + + foreach my $item (@{$glueforge_meta->{value}}) { + if ($item->{name} eq 'pkg_name') { + $pkg_name = $item->{value}; + } elsif ($item->{name} eq 'pkg_namespace') { + $pkg_namespace = $item->{value}; + } elsif ($item->{name} eq 'pkg_version') { + $pkg_version = $item->{value}; + } else { + die "invalid glueforge METADATA key: $item->{name}"; + } + } + } + + die "pkg_name is required in complex config" unless defined $pkg_name; + die "pkg_namespace is required in complex config" unless defined + $pkg_namespace; + die "pkg_version is required in complex config" unless defined + $pkg_version; + + my %data; + $data{glueforge_version}= $VERSION; + $data{pkg_name} = $pkg_name; + $data{pkg_namespace} = $pkg_namespace; + $data{pkg_version} = $pkg_version; + + # setup one table per nested metadata + my @tables; + foreach my $meta (@{$config}) { + push @tables, tabledata_from_meta($meta); + } + + $data{tables} = \@tables; + + return \%data; +} + +sub tabledata_from_meta { + my $meta = shift; # hashref + + die "not a valid metadata" unless $meta->{class} eq 'metadata'; + + my $table_name = $meta->{name}; + + my @fields; + foreach my $column (@{$meta->{value}}) { + my $mdtypes = lookup_type( $column->{type} ); + next if not defined $mdtypes; + + push @fields, { + name => $column->{name}, + type => $column->{type}, +# ctype => $mdtypes->{ctype}, +# mtype => $mdtypes->{mtype}, +# test => $mdtypes->{test}, + comment => $column->{comment}, + value => $column->{value}, + %{$mdtypes}, + }; + } + + my %table = ( + name => $table_name, + namespace => $table_name, + object_name => $table_name . "Row", + columns => \@fields, + ); + + return \%table; +} + +__END__ + +=pod + +=head1 NAME + +@PACKAGE_NAME@ - auto-generate Pan-STARRS IPP 'meta' database bindings + +=head1 SYNOPSIS + + @PACKAGE_NAME@ --input [--output ] [--template ] + +=head1 DESCRIPTION + +This program generates a set of database table format specific bindings in the +ANSI C language. The code generated is largely just wrapper functions around +the C C API. Table information is read in from a configuration +file in the pslib C format. A complete autotools configured +package that builds both static & shared libraries is generated. + +=head2 Features + +=over 4 + +=item * Generates a complete autoconf/automake/libtool package. + +=item * Generated packages include an C script. + +=item * Installs a pkgconfig C<.pc> data file. + +=item * Includes a table format specific autotest test suite. + +=item * Builds both shared and static libraries by default. + +=item * Generates man pages and HTML documentation with Doxygen. + +=item * Header files should be C++ safe. + +=item * Functions to open and close database connections. + +=item * Functions to create and destroy a table of the appropriate format. + +=item * Functions to insert and pop a row. + +=item * Functions to insert and pop a simple object representing a row. + +=item * Functions to insert, select, and pop to/from FITS files. + +=item * Functions to convert between row structs and pslib Cs. + +=back + +=head1 TABLE DESCRIPTION FORMAT + +There are two slightly different configuration file forms. A C format +for describing a single table and a C format for describing more then +one table and/or configuring specific details as to how the package is +generated. The syntax for both of these forms follows the C format as described in C. + +=head2 Simple configuration + +A simple config file specifies the name of the table, the name and types of all +columns, and can optionally specify which column(s) are to be used as indexes. + +=head2 Example simple configuration + + table STR foo + foo STR 60 # Primary Key # the name of foo thing + bar S32 0 ## count of bar + baz F32 0.0 + boing F64 0.0 + zot BOOL t # Key + +The key name C is reserved and specifies the name of the database table. +It may appear any where in the file but it is most I to have it as +the first line of the table description. + +The key name C is reserved and I be used. + +The value of a C column specifies the maximum number of characters that +the database is required to store for that field. The values for C, +C, C, & C columns are ignored. + +Indexes maybe specified as comment value of either C or C. +Only one C may be specified per table. + +Description comments may specified as either a C<# ...> after an index +specifier or as stand alone comment value of C<## ...>. + +The key name C is also reserved as it is used internally by the +database bindings. + +=head2 Complex configuration + +A complex config file consists of a series of C blocks. One (and +only one) of these blocks I be named C. This block contains +the global package configuration data. Only the keys C, +C, and C are allowed in this block and they are +both required. + +All other C blocks describe a single table with the name of the +C block being the tables name. Otherwise, the format of items +contained in a C block is the same as for the simple configuration. + +Note that nested C blocks are not allowed. + +=head2 Example complex configuration + +glueforge METADATA + pkg_name STR foodb + pkg_namespace STR foo + pkg_version STR 0.0.1 +END + +foo METADATA + foo STR 60 # Primary Key # the name of foo thing + bar S32 0 ## count of bar + baz F32 0.0 + boing F64 0.0 + zot BOOL t # Key +END + +bar METADATA + zot BOOL t # Key + boing F64 0.0 + baz F32 0.0 + bar S32 0 ## count of bar + foo STR 60 # Primary Key # the name of foo thing +END + +=head1 OPTIONS + +=over 4 + +=item * --input|-i + +File to read table description from. + +=item * --output|-o + +Send the generate files to this directory name. + +Defaults to the name of the database table. + +=item * --template|-t + +Directory to load template files from. + +Defaults to C<@datadir@/@PACKAGE_NAME@/templates/psdb>. + +=back + +=head1 BUGS + +The format of configuration files is not rigorously checked. Improperly +formatted files may not be caught and cause random (possibly silent) errors +and/or improperly code generation. + +=head1 CREDITS + +Just me, myself, and I. + +=head1 SUPPORT + +Please contact the author directly via e-mail. + +=head1 AUTHOR + +Joshua Hoblitt + +=head1 COPYRIGHT + +Copyright (C) 2005-2006 Joshua Hoblitt. All rights reserved. + +This program is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free Software +Foundation; either version 2 of the License, or (at your option) any later +version. + +This program is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A +PARTICULAR PURPOSE. See the GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along with +this program; if not, write to the Free Software Foundation, Inc., 59 Temple +Place - Suite 330, Boston, MA 02111-1307, USA. + +The full text of the license can be found in the L Pod as supplied +with Perl 5.8.1 and later. + +=head1 SEE ALSO + +C, C. L + +=cut Index: /branches/rel-1-0/glueforge/m4/ac_prog_perl_modules.m4 =================================================================== --- /branches/rel-1-0/glueforge/m4/ac_prog_perl_modules.m4 (revision 11645) +++ /branches/rel-1-0/glueforge/m4/ac_prog_perl_modules.m4 (revision 11645) @@ -0,0 +1,53 @@ +dnl @synopsis AC_PROG_PERL_MODULES([MODULES], [ACTION-IF-TRUE], [ACTION-IF-FALSE]) +dnl +dnl Checks to see if the the given perl modules are available. If true +dnl the shell commands in ACTION-IF-TRUE are executed. If not the shell +dnl commands in ACTION-IF-FALSE are run. Note if $PERL is not set (for +dnl example by calling AC_CHECK_PROG, or AC_PATH_PROG), +dnl AC_CHECK_PROG(PERL, perl, perl) will be run. +dnl +dnl Example: +dnl +dnl AC_CHECK_PERL_MODULES(Text::Wrap Net::LDAP, , +dnl AC_MSG_WARN(Need some Perl modules) +dnl +dnl @category InstalledPackages +dnl @author Dean Povey +dnl @version 2002-09-25 +dnl @license AllPermissive + +AC_DEFUN([AC_PROG_PERL_MODULES],[ + ac_perl_modules="$1" + # Make sure we have perl + if test -z "$PERL"; then + AC_CHECK_PROG(PERL,perl,perl) + fi + + if test "x$PERL" != x; then + ac_perl_modules_failed=0 + for ac_perl_module in $ac_perl_modules; do + AC_MSG_CHECKING(for perl module $ac_perl_module) + + # Would be nice to log result here, but can't rely on autoconf + # internals + $PERL "-M$ac_perl_module" -e exit > /dev/null 2>&1 + if test $? -ne 0; then + AC_MSG_RESULT(no); + ac_perl_modules_failed=1 + else + AC_MSG_RESULT(ok); + fi + done + + # Run optional shell commands + if test "$ac_perl_modules_failed" = 0; then + : + $2 + else + : + $3 + fi + else + AC_MSG_WARN(could not find perl) + fi +]) Index: /branches/rel-1-0/glueforge/templates/latex/.cvsignore =================================================================== --- /branches/rel-1-0/glueforge/templates/latex/.cvsignore (revision 11645) +++ /branches/rel-1-0/glueforge/templates/latex/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/glueforge/templates/latex/Makefile.am =================================================================== --- /branches/rel-1-0/glueforge/templates/latex/Makefile.am (revision 11645) +++ /branches/rel-1-0/glueforge/templates/latex/Makefile.am (revision 11645) @@ -0,0 +1,12 @@ +template_files = \ + config.pl \ + tables.tt + +templatedir = $(datadir)/@PACKAGE_NAME@/templates +latextemplatedir = $(templatedir)/latex +latextemplate_DATA = $(template_files) + +install-data-hook: + chmod 0755 $(templatedir) $(latextemplatedir) + +EXTRA_DIST = $(template_files) Index: /branches/rel-1-0/glueforge/templates/latex/config.pl =================================================================== --- /branches/rel-1-0/glueforge/templates/latex/config.pl (revision 11645) +++ /branches/rel-1-0/glueforge/templates/latex/config.pl (revision 11645) @@ -0,0 +1,3 @@ +%tt = ( + 'tables.tt' => "$output/tables.tex", +); Index: /branches/rel-1-0/glueforge/templates/latex/tables.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/latex/tables.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/latex/tables.tt (revision 11645) @@ -0,0 +1,21 @@ +[% FOREACH table = tables -%] +[% PERL %] print "\\clearpage\n\n" if not [% loop.count %] % 10 [% END -%] +\begin{table}[ht] +\begin{center} +\caption{\label{[% table.namespace %]}[% table.namespace.replace('_', '\\_') %]} +\begin{tabular}{|l|l|l|} +\hline +\textbf{name} & \textbf{type} & \textbf{description} \\ +\hline +[% FOREACH item = table.columns -%] +[%- IF item.type == 'STR' -%] +[% item.name.replace('_', '\\_') %] & [% item.type %] (up to [% item.value %] chars) & \parbox[t]{3.0in}{[% item.comment.replace('\#', '') %]} \\ \hline +[% ELSE -%] +[% item.name.replace('_', '\\_') %] & [% item.type %] & \parbox[t]{3.0in}{[% item.comment.replace('\#', '') %]} \\ \hline +[% END -%] +[% END -%] +\end{tabular} +\end{center} +\end{table} + +[% END %] Index: /branches/rel-1-0/glueforge/templates/psdb/.cvsignore =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/.cvsignore (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/glueforge/templates/psdb/COPYING =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/COPYING (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/COPYING (revision 11645) @@ -0,0 +1,345 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. Index: /branches/rel-1-0/glueforge/templates/psdb/Makefile.am =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/Makefile.am (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/Makefile.am (revision 11645) @@ -0,0 +1,97 @@ +template_files = \ + COPYING \ + metadata \ + metadata_from_params \ + metadata_from_struct \ + lookup_from_metadata \ + alloc.tt \ + alloc_at.tt \ + alloc_c.tt \ + alloc_h.tt \ + autogen_sh.tt \ + bootstrap_sh.tt \ + cleanup.tt \ + cleanup_at.tt \ + cleanup_c.tt \ + cleanup_h.tt \ + code.tt \ + config.pl \ + configure_ac.tt \ + createtable.tt \ + createtable_at.tt \ + createtable_c.tt \ + createtable_h.tt \ + dbcleanup_c.tt \ + dbsetup_c.tt \ + delete.tt \ + delete_h.tt \ + deleteobject.tt \ + deleteobject_h.tt \ + deleterowobjects.tt \ + deleterowobjects_h.tt \ + doxyfile_in.tt \ + droptable.tt \ + droptable_at.tt \ + droptable_c.tt \ + droptable_h.tt \ + header.tt \ + init.tt \ + init_at.tt \ + init_c.tt \ + init_h.tt \ + insert.tt \ + insert_at.tt \ + insert_c.tt \ + insert_h.tt \ + insertfits.tt \ + insertfits_at.tt \ + insertfits_c.tt \ + insertfits_h.tt \ + insertobject.tt \ + insertobject_at.tt \ + insertobject_c.tt \ + insertobject_h.tt \ + insertobjects.tt \ + insertobjects_h.tt \ + makefile.tt \ + metadatafromobject.tt \ + metadatafromobject_at.tt \ + metadatafromobject_c.tt \ + metadatafromobject_h.tt \ + object_h.tt \ + objectfrommetadata.tt \ + objectfrommetadata_at.tt \ + objectfrommetadata_c.tt \ + objectfrommetadata_h.tt \ + pkgconfig_pc_in.tt \ + printmetadata.tt \ + printmetadata_h.tt \ + printmetadataraw.tt \ + printmetadataraw_h.tt \ + printmetadatas.tt \ + printmetadatas_h.tt \ + printmetadatasraw.tt \ + printmetadatasraw_h.tt \ + printobject.tt \ + printobject_h.tt \ + printobjects.tt \ + printobjects_h.tt \ + selectrowobjects.tt \ + selectrowobjects_h.tt \ + selectrowsfits.tt \ + selectrowsfits_at.tt \ + selectrowsfits_c.tt \ + selectrowsfits_h.tt \ + src_makefile_am.tt \ + tests_makefile_am.tt \ + testsuite_at.tt \ + top_makefile_am.tt + +templatedir = $(datadir)/@PACKAGE_NAME@/templates +psdbtemplatedir = $(templatedir)/psdb +psdbtemplate_DATA = $(template_files) + +install-data-hook: + chmod 0755 $(templatedir) $(psdbtemplatedir) + +EXTRA_DIST = $(template_files) Index: /branches/rel-1-0/glueforge/templates/psdb/alloc.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/alloc.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/alloc.tt (revision 11645) @@ -0,0 +1,44 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +static void [% table.object_name %]Free([% table.object_name %] *object); + +[% table.object_name %] *[% table.object_name %]Alloc( +[%- SET i = 0 -%] +[% FOREACH item = table.columns -%] +[% IF i == 0 %][% i = 1 %][% ELSE %], [% END -%] +[% IF item.type == "STR" -%] +const char *[% item.name %] +[%- ELSE -%] +[% item.ctype %] [% item.name %] +[%- END -%] +[% END -%] +) +{ + [% indented(table.object_name, "*_object") %]; + + _object = psAlloc(sizeof([% table.object_name %])); + psMemSetDeallocator(_object, (psFreeFunc)[% table.object_name %]Free); + +[% FOREACH item = table.columns -%] +[% IF item.type == "STR" -%] + _object->[% item.name %] = psStringCopy([% item.name %]); +[% ELSIF item.mtype == "PS_DATA_TIME" -%] + _object->[% item.name %] = psTimeCopy([% item.name %]); +[% ELSE -%] + _object->[% item.name %] = [% item.name %]; +[% END -%] +[% END -%] + + return _object; +} + +static void [% table.object_name %]Free([% table.object_name %] *object) +{ +[% FOREACH item = table.columns -%] +[% IF item.type == "STR" -%] + psFree(object->[% item.name %]); +[% ELSIF item.mtype == "PS_DATA_TIME" -%] + psFree(object->[% item.name %]); +[% END -%] +[% END -%] +} Index: /branches/rel-1-0/glueforge/templates/psdb/alloc_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/alloc_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/alloc_at.tt (revision 11645) @@ -0,0 +1,6 @@ +AT_SETUP([[% table.object_name %]Alloc()]) +AT_KEYWORDS([[% table.object_name %]Alloc]) + +AT_CHECK([alloc]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/alloc_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/alloc_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/alloc_c.tt (revision 11645) @@ -0,0 +1,43 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +#include +#include <[% pkg_name %].h> +#include +#include + +#define MAX_STRING_LENGTH 1024 + +int main () +{ +[% FOREACH table = tables -%] + { + [% indented(table.object_name, "*object") %]; + + object = [% table.object_name %]Alloc( +[%- SET i = 0 -%] +[%- FOREACH item = table.columns -%] + [%- IF i == 0 -%][%- i = 1 -%][%- ELSE -%], [% END -%] [%- item.test -%] +[%- END -%] + ); + + if (!object) { + exit(EXIT_FAILURE); + } + +[% FOREACH item = table.columns -%] +[% IF item.type == "S32" or item.type == "F32" or item.type == "F64" or item.type == "BOOL" -%] + if (!object->[% item.name %] == [% item.test %]) { +[% ELSIF item.type == "STR" -%] + if (strncmp(object->[% item.name %], [% item.test %], MAX_STRING_LENGTH)) { +[% END -%] + psFree(object); + exit(EXIT_FAILURE); + } +[% END -%] + + psFree(object); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/alloc_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/alloc_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/alloc_h.tt (revision 11645) @@ -0,0 +1,18 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +/** Creates a new [% table.object_name %] object + * + * @return A new [% table.object_name %] object or NULL on failure. + */ + +[% table.object_name %] *[% table.object_name %]Alloc( +[%- SET i = 0 -%] +[% FOREACH item = table.columns -%] +[% IF i == 0 %][% i = 1 %][% ELSE %],[% END %] +[% IF item.type == "STR" -%] + [% indented("const char", "*$item.name") %] +[%- ELSE -%] + [% indented("$item.ctype", "$item.name") %] +[%- END -%] +[% END %] +); Index: /branches/rel-1-0/glueforge/templates/psdb/autogen_sh.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/autogen_sh.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/autogen_sh.tt (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=[% pkg_name %] +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL=aclocal +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $LIBTOOlIZE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" + DIE=1 +} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOHEADER installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/glueforge/templates/psdb/bootstrap_sh.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/bootstrap_sh.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/bootstrap_sh.tt (revision 11645) @@ -0,0 +1,7 @@ +#!/bin/sh + +libtoolize --automake --copy \ +&& aclocal \ +&& autoheader \ +&& automake --foreign --add-missing --copy \ +&& autoconf Index: /branches/rel-1-0/glueforge/templates/psdb/cleanup.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/cleanup.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/cleanup.tt (revision 11645) @@ -0,0 +1,4 @@ +void [% pkg_namespace %]Cleanup(psDB *dbh) +{ + psDBCleanup(dbh); +} Index: /branches/rel-1-0/glueforge/templates/psdb/cleanup_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/cleanup_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/cleanup_at.tt (revision 11645) @@ -0,0 +1,6 @@ +AT_SETUP([[% table.namespace %]Cleanup()]) +AT_KEYWORDS([[% table.namespace %]Cleanup]) + +AT_CHECK([cleanup]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/cleanup_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/cleanup_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/cleanup_c.tt (revision 11645) @@ -0,0 +1,17 @@ +#include +#include <[% pkg_name %].h> +#include + +int main () +{ + psDB *dbh; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + [% pkg_namespace %]Cleanup(dbh); + + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/cleanup_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/cleanup_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/cleanup_h.tt (revision 11645) @@ -0,0 +1,6 @@ +/** Closes a database connection + */ + +void [% pkg_namespace %]Cleanup( + psDB *dbh ///< Database handle +); Index: /branches/rel-1-0/glueforge/templates/psdb/code.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/code.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/code.tt (revision 11645) @@ -0,0 +1,61 @@ +/* + * code.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/* + * + * This file was generated by glueforge [% glueforge_version %] + * + * Do NOT directly edit this file. + * + */ + +#include +#include + +#include "[% pkg_name %].h" + +[% FOREACH table = tables -%] +#define [% table.namespace FILTER upper %]_TABLE_NAME "[% table.name %]" +[% END -%] +#define MAX_STRING_LENGTH 1024 + +[% INCLUDE init.tt %] +[% INCLUDE cleanup.tt %] +[% INCLUDE printmetadata.tt %] +[% INCLUDE printmetadataraw.tt %] +[% INCLUDE printmetadatas.tt %] +[% INCLUDE printmetadatasraw.tt %] +[% FOREACH table = tables -%] +[% INCLUDE alloc.tt %] +[% INCLUDE createtable.tt %] +[% INCLUDE droptable.tt %] +[% INCLUDE insert.tt %] +[% INCLUDE delete.tt -%] +[% INCLUDE insertobject.tt %] +[% INCLUDE insertobjects.tt %] +[% INCLUDE insertfits.tt %] +[% INCLUDE selectrowsfits.tt %] +[% INCLUDE metadatafromobject.tt %] +[% INCLUDE objectfrommetadata.tt -%] +[% INCLUDE selectrowobjects.tt -%] +[% INCLUDE deleteobject.tt -%] +[% INCLUDE deleterowobjects.tt -%] +[% INCLUDE printobjects.tt -%] +[% INCLUDE printobject.tt -%] +[% END -%] Index: /branches/rel-1-0/glueforge/templates/psdb/config.pl =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/config.pl (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/config.pl (revision 11645) @@ -0,0 +1,26 @@ +%tt = ( + 'COPYING' => "$output/COPYING", + 'autogen_sh.tt' => "$output/autogen.sh", + 'configure_ac.tt' => "$output/configure.ac", + 'pkgconfig_pc_in.tt' => "$output/$data->{pkg_name}.pc.in", + 'doxyfile_in.tt' => "$output/Doxyfile.in", + 'top_makefile_am.tt' => "$output/Makefile.am", + 'src_makefile_am.tt' => "$output/src/Makefile.am", + 'header.tt' => "$output/src/$data->{pkg_name}.h", + 'code.tt' => "$output/src/$data->{pkg_name}.c", + 'tests_makefile_am.tt' => "$output/tests/Makefile.am", + 'testsuite_at.tt' => "$output/tests/testsuite.at", + 'dbsetup_c.tt' => "$output/tests/dbsetup.c", + 'dbcleanup_c.tt' => "$output/tests/dbcleanup.c", + 'alloc_c.tt' => "$output/tests/alloc.c", + 'init_c.tt' => "$output/tests/init.c", + 'cleanup_c.tt' => "$output/tests/cleanup.c", + 'createtable_c.tt' => "$output/tests/createtable.c", + 'droptable_c.tt' => "$output/tests/droptable.c", + 'insert_c.tt' => "$output/tests/insert.c", + 'insertobject_c.tt' => "$output/tests/insertobject.c", + 'insertfits_c.tt' => "$output/tests/insertfits.c", + 'selectrowsfits_c.tt' => "$output/tests/selectrowsfits.c", + 'metadatafromobject_c.tt' => "$output/tests/metadatafromobject.c", + 'objectfrommetadata_c.tt' => "$output/tests/objectfrommetadata.c", +); Index: /branches/rel-1-0/glueforge/templates/psdb/configure_ac.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/configure_ac.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/configure_ac.tt (revision 11645) @@ -0,0 +1,46 @@ +dnl +dnl This file was generated by glueforge [% glueforge_version %] +dnl +dnl Do NOT directly edit this file. +dnl + +AC_PREREQ(2.59) + +AC_INIT([[% pkg_name %]], [[% pkg_version %]], [pan-starrs.ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([[% pkg_name %].pc.in]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([config.h]) +AM_MAINTAINER_MODE + +AC_CONFIG_TESTDIR([tests]) +AC_CONFIG_FILES([tests/Makefile]) +AM_MISSING_PROG([AUTOM4TE], [autom4te]) + +AC_LANG(C) +AC_PROG_CC +AC_PROG_INSTALL +AC_PROG_LIBTOOL + +PKG_CHECK_MODULES([PSLIB], [pslib >= 0.9.0]) + +AC_PATH_PROG([PERL], [perl], [missing]) +if test "$PERL" = "missing" ; then + AC_MSG_ERROR([perl is required]) +fi + +AC_PATH_PROG([DOXYGEN], [doxygen], []) +AM_CONDITIONAL([HAVE_DOXYGEN], [test -n "$DOXYGEN"]) + +dnl is this the best was to setup recursive CFLAGS? +IPP_STDOPTS +[% pkg_name %]_CFLAGS="-Wall -Werror -pedantic -std=c99 -fno-strict-aliasing" +AC_SUBST([[% pkg_name %]_CFLAGS]) + +AC_CONFIG_FILES([ + Makefile + src/Makefile + [% pkg_name %].pc + Doxyfile +]) +AC_OUTPUT Index: /branches/rel-1-0/glueforge/templates/psdb/createtable.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/createtable.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/createtable.tt (revision 11645) @@ -0,0 +1,10 @@ +bool [% table.namespace %]CreateTable(psDB *dbh) +{ +[% INCLUDE metadata -%] + + bool status = psDBCreateTable(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, md); + + psFree(md); + + return status; +} Index: /branches/rel-1-0/glueforge/templates/psdb/createtable_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/createtable_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/createtable_at.tt (revision 11645) @@ -0,0 +1,9 @@ +AT_SETUP([[% table.namespace %]CreateTable()]) +AT_KEYWORDS([[% table.namespace %]CreateTable]) + +# make sure the table is not there. +AT_CHECK([dbcleanup]) +AT_CHECK([createtable]) +AT_CHECK([dbcleanup]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/createtable_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/createtable_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/createtable_c.tt (revision 11645) @@ -0,0 +1,25 @@ +#include +#include <[% pkg_name %].h> +#include + +int main () +{ +[% FOREACH table = tables -%] + { + psDB *dbh; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + if(![% table.namespace %]CreateTable(dbh)) { + exit(EXIT_FAILURE); + } + + psDBCleanup(dbh); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/createtable_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/createtable_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/createtable_h.tt (revision 11645) @@ -0,0 +1,8 @@ +/** Creates a new [% table.name %] table + * + * @return true on success + */ + +bool [% table.namespace %]CreateTable( + psDB *dbh ///< Database handle +); Index: /branches/rel-1-0/glueforge/templates/psdb/dbcleanup_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/dbcleanup_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/dbcleanup_c.tt (revision 11645) @@ -0,0 +1,20 @@ +#include +#include + +int main () +{ + psDB *dbh; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + +[% FOREACH table = tables -%] + p_psDBRunQuery(dbh, "DROP TABLE IF EXISTS [% table.name %]"); +[% END -%] + + psDBCleanup(dbh); + + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/dbsetup_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/dbsetup_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/dbsetup_c.tt (revision 11645) @@ -0,0 +1,23 @@ +#include +#include <[% pkg_name %].h> +#include + +int main () +{ + psDB *dbh; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + // remove the table if it already exists +[% FOREACH table = tables -%] + p_psDBRunQuery(dbh, "DROP TABLE IF EXISTS [% table.name %]"); + [% table.namespace %]CreateTable(dbh); + +[% END -%] + psDBCleanup(dbh); + + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/delete.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/delete.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/delete.tt (revision 11645) @@ -0,0 +1,14 @@ +long long [% table.namespace %]Delete(psDB *dbh, const psMetadata *where, unsigned long long limit) +{ + long long deleted = 0; + + long long count = psDBDeleteRows(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, where, limit); + if (count < 0) { + psError(PS_ERR_UNKNOWN, true, "failed to delete row from [% table.namespace %]"); + return count; + + deleted += count; + } + + return deleted; +} Index: /branches/rel-1-0/glueforge/templates/psdb/delete_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/delete_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/delete_h.tt (revision 11645) @@ -0,0 +1,10 @@ +/** Deletes up to limit rows from the database and returns the number of rows actually deleted. + * + * @return A The number of rows removed or a negative value on error + */ + +long long [% table.namespace %]Delete( + psDB *dbh, ///< Database handle + const psMetadata *where, ///< Row match criteria + unsigned long long limit ///< Maximum number of elements to delete +); Index: /branches/rel-1-0/glueforge/templates/psdb/deleteobject.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/deleteobject.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/deleteobject.tt (revision 11645) @@ -0,0 +1,18 @@ +bool [% table.namespace %]DeleteObject(psDB *dbh, const [% table.object_name %] *object) +{ + psMetadata *where = [% table.namespace %]MetadataFromObject(object); + long long count = psDBDeleteRows(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, where, 0); + psFree(where) + if (count < 0) { + psError(PS_ERR_UNKNOWN, true, "failed to delete row from [% table.namespace %]"); + return false; + } + if (count > 1) { + // XXX should this be a psAbort() instead? It is possible that + // having an object match multiple rows was by design. + psError(PS_ERR_UNKNOWN, true, "[% table.object_name%] object matched more then one row. Check your database schema"); + return false; + } + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/deleteobject_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/deleteobject_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/deleteobject_h.tt (revision 11645) @@ -0,0 +1,12 @@ +/** Deletes a row from the database coresponding to an [% table.namespace %] + * + * Note that a 'where' search psMetadata is constructed from each object and + * used to find rows to delete. + * + * @return A The number of rows removed or a negative value on error + */ + +bool [% table.namespace %]DeleteObject( + psDB *dbh, ///< Database handle + const [% table.object_name %] *object ///< Object to delete +); Index: /branches/rel-1-0/glueforge/templates/psdb/deleterowobjects.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/deleterowobjects.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/deleterowobjects.tt (revision 11645) @@ -0,0 +1,19 @@ +long long [% table.namespace %]DeleteRowObjects(psDB *dbh, const psArray *objects, unsigned long long limit) +{ + long long deleted = 0; + + for (long long i = 0; i < objects->n; i++) { + [% table.object_name %] *object = objects->data[i]; + psMetadata *where = [% table.namespace %]MetadataFromObject(object); + long long count = psDBDeleteRows(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, where, limit); + psFree(where) + if (count < 0) { + psError(PS_ERR_UNKNOWN, true, "failed to delete row from [% table.namespace %]"); + return count; + } + + deleted += count; + } + + return deleted; +} Index: /branches/rel-1-0/glueforge/templates/psdb/deleterowobjects_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/deleterowobjects_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/deleterowobjects_h.tt (revision 11645) @@ -0,0 +1,13 @@ +/** Deletes up to limit rows from the database and returns the number of rows actually deleted. + * + * Note that a 'where' search psMetadata is constructed from each object and + * used to find rows to delete. + * + * @return A The number of rows removed or a negative value on error + */ + +long long [% table.namespace %]DeleteRowObjects( + psDB *dbh, ///< Database handle + const psArray *objects, ///< Array of objects to delete + unsigned long long limit ///< Maximum number of elements to delete +); Index: /branches/rel-1-0/glueforge/templates/psdb/doxyfile_in.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/doxyfile_in.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/doxyfile_in.tt (revision 11645) @@ -0,0 +1,1219 @@ +# Doxyfile 1.4.2 + +# This file describes the settings to be used by the documentation system +# doxygen (www.doxygen.org) for a project +# +# All text after a hash (#) is considered a comment and will be ignored +# The format is: +# TAG = value [value, ...] +# For lists items can also be appended using: +# TAG += value [value, ...] +# Values that contain spaces should be placed between quotes (" ") + +#--------------------------------------------------------------------------- +# Project related configuration options +#--------------------------------------------------------------------------- + +# The PROJECT_NAME tag is a single word (or a sequence of words surrounded +# by quotes) that should identify the project. + +PROJECT_NAME = @PACKAGE_NAME@ + +# The PROJECT_NUMBER tag can be used to enter a project or revision number. +# This could be handy for archiving the generated documentation or +# if some version control system is used. + +PROJECT_NUMBER = @PACKAGE_VERSION@ + +# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) +# base path where the generated documentation will be put. +# If a relative path is entered, it will be relative to the location +# where doxygen was started. If left blank the current directory will be used. + +# @top_builddir@ doesn't work for some reason +OUTPUT_DIRECTORY = @builddir@/docs + +# If the CREATE_SUBDIRS tag is set to YES, then doxygen will create +# 4096 sub-directories (in 2 levels) under the output directory of each output +# format and will distribute the generated files over these directories. +# Enabling this option can be useful when feeding doxygen a huge amount of +# source files, where putting all generated files in the same directory would +# otherwise cause performance problems for the file system. + +CREATE_SUBDIRS = NO + +# The OUTPUT_LANGUAGE tag is used to specify the language in which all +# documentation generated by doxygen is written. Doxygen will use this +# information to generate all constant output in the proper language. +# The default language is English, other supported languages are: +# Brazilian, Catalan, Chinese, Chinese-Traditional, Croatian, Czech, Danish, +# Dutch, Finnish, French, German, Greek, Hungarian, Italian, Japanese, +# Japanese-en (Japanese with English messages), Korean, Korean-en, Norwegian, +# Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovene, Spanish, +# Swedish, and Ukrainian. + +OUTPUT_LANGUAGE = English + +# This tag can be used to specify the encoding used in the generated output. +# The encoding is not always determined by the language that is chosen, +# but also whether or not the output is meant for Windows or non-Windows users. +# In case there is a difference, setting the USE_WINDOWS_ENCODING tag to YES +# forces the Windows encoding (this is the default for the Windows binary), +# whereas setting the tag to NO uses a Unix-style encoding (the default for +# all platforms other than Windows). + +USE_WINDOWS_ENCODING = NO + +# If the BRIEF_MEMBER_DESC tag is set to YES (the default) Doxygen will +# include brief member descriptions after the members that are listed in +# the file and class documentation (similar to JavaDoc). +# Set to NO to disable this. + +BRIEF_MEMBER_DESC = YES + +# If the REPEAT_BRIEF tag is set to YES (the default) Doxygen will prepend +# the brief description of a member or function before the detailed description. +# Note: if both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the +# brief descriptions will be completely suppressed. + +REPEAT_BRIEF = YES + +# This tag implements a quasi-intelligent brief description abbreviator +# that is used to form the text in various listings. Each string +# in this list, if found as the leading text of the brief description, will be +# stripped from the text and the result after processing the whole list, is +# used as the annotated text. Otherwise, the brief description is used as-is. +# If left blank, the following values are used ("$name" is automatically +# replaced with the name of the entity): "The $name class" "The $name widget" +# "The $name file" "is" "provides" "specifies" "contains" +# "represents" "a" "an" "the" + +ABBREVIATE_BRIEF = + +# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then +# Doxygen will generate a detailed section even if there is only a brief +# description. + +ALWAYS_DETAILED_SEC = NO + +# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all +# inherited members of a class in the documentation of that class as if those +# members were ordinary class members. Constructors, destructors and assignment +# operators of the base classes will not be shown. + +INLINE_INHERITED_MEMB = NO + +# If the FULL_PATH_NAMES tag is set to YES then Doxygen will prepend the full +# path before files name in the file list and in the header files. If set +# to NO the shortest path that makes the file name unique will be used. + +FULL_PATH_NAMES = NO + +# If the FULL_PATH_NAMES tag is set to YES then the STRIP_FROM_PATH tag +# can be used to strip a user-defined part of the path. Stripping is +# only done if one of the specified strings matches the left-hand part of +# the path. The tag can be used to show relative paths in the file list. +# If left blank the directory from which doxygen is run is used as the +# path to strip. + +STRIP_FROM_PATH = + +# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of +# the path mentioned in the documentation of a class, which tells +# the reader which header file to include in order to use a class. +# If left blank only the name of the header file containing the class +# definition is used. Otherwise one should specify the include paths that +# are normally passed to the compiler using the -I flag. + +STRIP_FROM_INC_PATH = + +# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter +# (but less readable) file names. This can be useful is your file systems +# doesn't support long names like on DOS, Mac, or CD-ROM. + +SHORT_NAMES = NO + +# If the JAVADOC_AUTOBRIEF tag is set to YES then Doxygen +# will interpret the first line (until the first dot) of a JavaDoc-style +# comment as the brief description. If set to NO, the JavaDoc +# comments will behave just like the Qt-style comments (thus requiring an +# explicit @brief command for a brief description. + +JAVADOC_AUTOBRIEF = NO + +# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make Doxygen +# treat a multi-line C++ special comment block (i.e. a block of //! or /// +# comments) as a brief description. This used to be the default behaviour. +# The new default is to treat a multi-line C++ comment block as a detailed +# description. Set this tag to YES if you prefer the old behaviour instead. + +MULTILINE_CPP_IS_BRIEF = NO + +# If the DETAILS_AT_TOP tag is set to YES then Doxygen +# will output the detailed description near the top, like JavaDoc. +# If set to NO, the detailed description appears after the member +# documentation. + +DETAILS_AT_TOP = NO + +# If the INHERIT_DOCS tag is set to YES (the default) then an undocumented +# member inherits the documentation from any documented member that it +# re-implements. + +INHERIT_DOCS = YES + +# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC +# tag is set to YES, then doxygen will reuse the documentation of the first +# member in the group (if any) for the other members of the group. By default +# all members of a group must be documented explicitly. + +DISTRIBUTE_GROUP_DOC = NO + +# If the SEPARATE_MEMBER_PAGES tag is set to YES, then doxygen will produce +# a new page for each member. If set to NO, the documentation of a member will +# be part of the file/class/namespace that contains it. + +SEPARATE_MEMBER_PAGES = NO + +# The TAB_SIZE tag can be used to set the number of spaces in a tab. +# Doxygen uses this value to replace tabs by spaces in code fragments. + +TAB_SIZE = 4 + +# This tag can be used to specify a number of aliases that acts +# as commands in the documentation. An alias has the form "name=value". +# For example adding "sideeffect=\par Side Effects:\n" will allow you to +# put the command \sideeffect (or @sideeffect) in the documentation, which +# will result in a user-defined paragraph with heading "Side Effects:". +# You can put \n's in the value part of an alias to insert newlines. + +ALIASES = + +# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C +# sources only. Doxygen will then generate output that is more tailored for C. +# For instance, some of the names that are used will be different. The list +# of all members will be omitted, etc. + +OPTIMIZE_OUTPUT_FOR_C = NO + +# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java sources +# only. Doxygen will then generate output that is more tailored for Java. +# For instance, namespaces will be presented as packages, qualified scopes +# will look different, etc. + +OPTIMIZE_OUTPUT_JAVA = NO + +# Set the SUBGROUPING tag to YES (the default) to allow class member groups of +# the same type (for instance a group of public functions) to be put as a +# subgroup of that type (e.g. under the Public Functions section). Set it to +# NO to prevent subgrouping. Alternatively, this can be done per class using +# the \nosubgrouping command. + +SUBGROUPING = YES + +#--------------------------------------------------------------------------- +# Build related configuration options +#--------------------------------------------------------------------------- + +# If the EXTRACT_ALL tag is set to YES doxygen will assume all entities in +# documentation are documented, even if no documentation was available. +# Private class members and static file members will be hidden unless +# the EXTRACT_PRIVATE and EXTRACT_STATIC tags are set to YES + +EXTRACT_ALL = NO + +# If the EXTRACT_PRIVATE tag is set to YES all private members of a class +# will be included in the documentation. + +EXTRACT_PRIVATE = NO + +# If the EXTRACT_STATIC tag is set to YES all static members of a file +# will be included in the documentation. + +EXTRACT_STATIC = NO + +# If the EXTRACT_LOCAL_CLASSES tag is set to YES classes (and structs) +# defined locally in source files will be included in the documentation. +# If set to NO only classes defined in header files are included. + +EXTRACT_LOCAL_CLASSES = YES + +# This flag is only useful for Objective-C code. When set to YES local +# methods, which are defined in the implementation section but not in +# the interface are included in the documentation. +# If set to NO (the default) only methods in the interface are included. + +EXTRACT_LOCAL_METHODS = NO + +# If the HIDE_UNDOC_MEMBERS tag is set to YES, Doxygen will hide all +# undocumented members of documented classes, files or namespaces. +# If set to NO (the default) these members will be included in the +# various overviews, but no documentation section is generated. +# This option has no effect if EXTRACT_ALL is enabled. + +HIDE_UNDOC_MEMBERS = NO + +# If the HIDE_UNDOC_CLASSES tag is set to YES, Doxygen will hide all +# undocumented classes that are normally visible in the class hierarchy. +# If set to NO (the default) these classes will be included in the various +# overviews. This option has no effect if EXTRACT_ALL is enabled. + +HIDE_UNDOC_CLASSES = NO + +# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, Doxygen will hide all +# friend (class|struct|union) declarations. +# If set to NO (the default) these declarations will be included in the +# documentation. + +HIDE_FRIEND_COMPOUNDS = NO + +# If the HIDE_IN_BODY_DOCS tag is set to YES, Doxygen will hide any +# documentation blocks found inside the body of a function. +# If set to NO (the default) these blocks will be appended to the +# function's detailed documentation block. + +HIDE_IN_BODY_DOCS = NO + +# The INTERNAL_DOCS tag determines if documentation +# that is typed after a \internal command is included. If the tag is set +# to NO (the default) then the documentation will be excluded. +# Set it to YES to include the internal documentation. + +INTERNAL_DOCS = NO + +# If the CASE_SENSE_NAMES tag is set to NO then Doxygen will only generate +# file names in lower-case letters. If set to YES upper-case letters are also +# allowed. This is useful if you have classes or files whose names only differ +# in case and if your file system supports case sensitive file names. Windows +# and Mac users are advised to set this option to NO. + +CASE_SENSE_NAMES = YES + +# If the HIDE_SCOPE_NAMES tag is set to NO (the default) then Doxygen +# will show members with their full class and namespace scopes in the +# documentation. If set to YES the scope will be hidden. + +HIDE_SCOPE_NAMES = NO + +# If the SHOW_INCLUDE_FILES tag is set to YES (the default) then Doxygen +# will put a list of the files that are included by a file in the documentation +# of that file. + +SHOW_INCLUDE_FILES = YES + +# If the INLINE_INFO tag is set to YES (the default) then a tag [inline] +# is inserted in the documentation for inline members. + +INLINE_INFO = YES + +# If the SORT_MEMBER_DOCS tag is set to YES (the default) then doxygen +# will sort the (detailed) documentation of file and class members +# alphabetically by member name. If set to NO the members will appear in +# declaration order. + +SORT_MEMBER_DOCS = YES + +# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the +# brief documentation of file, namespace and class members alphabetically +# by member name. If set to NO (the default) the members will appear in +# declaration order. + +SORT_BRIEF_DOCS = NO + +# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be +# sorted by fully-qualified names, including namespaces. If set to +# NO (the default), the class list will be sorted only by class name, +# not including the namespace part. +# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. +# Note: This option applies only to the class list, not to the +# alphabetical list. + +SORT_BY_SCOPE_NAME = NO + +# The GENERATE_TODOLIST tag can be used to enable (YES) or +# disable (NO) the todo list. This list is created by putting \todo +# commands in the documentation. + +GENERATE_TODOLIST = YES + +# The GENERATE_TESTLIST tag can be used to enable (YES) or +# disable (NO) the test list. This list is created by putting \test +# commands in the documentation. + +GENERATE_TESTLIST = YES + +# The GENERATE_BUGLIST tag can be used to enable (YES) or +# disable (NO) the bug list. This list is created by putting \bug +# commands in the documentation. + +GENERATE_BUGLIST = YES + +# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or +# disable (NO) the deprecated list. This list is created by putting +# \deprecated commands in the documentation. + +GENERATE_DEPRECATEDLIST= YES + +# The ENABLED_SECTIONS tag can be used to enable conditional +# documentation sections, marked by \if sectionname ... \endif. + +ENABLED_SECTIONS = + +# The MAX_INITIALIZER_LINES tag determines the maximum number of lines +# the initial value of a variable or define consists of for it to appear in +# the documentation. If the initializer consists of more lines than specified +# here it will be hidden. Use a value of 0 to hide initializers completely. +# The appearance of the initializer of individual variables and defines in the +# documentation can be controlled using \showinitializer or \hideinitializer +# command in the documentation regardless of this setting. + +MAX_INITIALIZER_LINES = 30 + +# Set the SHOW_USED_FILES tag to NO to disable the list of files generated +# at the bottom of the documentation of classes and structs. If set to YES the +# list will mention the files that were used to generate the documentation. + +SHOW_USED_FILES = YES + +# If the sources in your project are distributed over multiple directories +# then setting the SHOW_DIRECTORIES tag to YES will show the directory hierarchy +# in the documentation. + +SHOW_DIRECTORIES = YES + +# The FILE_VERSION_FILTER tag can be used to specify a program or script that +# doxygen should invoke to get the current version for each file (typically from the +# version control system). Doxygen will invoke the program by executing (via +# popen()) the command , where is the value of +# the FILE_VERSION_FILTER tag, and is the name of an input file +# provided by doxygen. Whatever the progam writes to standard output +# is used as the file version. See the manual for examples. + +FILE_VERSION_FILTER = + +#--------------------------------------------------------------------------- +# configuration options related to warning and progress messages +#--------------------------------------------------------------------------- + +# The QUIET tag can be used to turn on/off the messages that are generated +# by doxygen. Possible values are YES and NO. If left blank NO is used. + +QUIET = NO + +# The WARNINGS tag can be used to turn on/off the warning messages that are +# generated by doxygen. Possible values are YES and NO. If left blank +# NO is used. + +WARNINGS = YES + +# If WARN_IF_UNDOCUMENTED is set to YES, then doxygen will generate warnings +# for undocumented members. If EXTRACT_ALL is set to YES then this flag will +# automatically be disabled. + +WARN_IF_UNDOCUMENTED = YES + +# If WARN_IF_DOC_ERROR is set to YES, doxygen will generate warnings for +# potential errors in the documentation, such as not documenting some +# parameters in a documented function, or documenting parameters that +# don't exist or using markup commands wrongly. + +WARN_IF_DOC_ERROR = YES + +# This WARN_NO_PARAMDOC option can be abled to get warnings for +# functions that are documented, but have no documentation for their parameters +# or return value. If set to NO (the default) doxygen will only warn about +# wrong or incomplete parameter documentation, but not about the absence of +# documentation. + +WARN_NO_PARAMDOC = NO + +# The WARN_FORMAT tag determines the format of the warning messages that +# doxygen can produce. The string should contain the $file, $line, and $text +# tags, which will be replaced by the file and line number from which the +# warning originated and the warning text. Optionally the format may contain +# $version, which will be replaced by the version of the file (if it could +# be obtained via FILE_VERSION_FILTER) + +WARN_FORMAT = "$file:$line: $text" + +# The WARN_LOGFILE tag can be used to specify a file to which warning +# and error messages should be written. If left blank the output is written +# to stderr. + +WARN_LOGFILE = + +#--------------------------------------------------------------------------- +# configuration options related to the input files +#--------------------------------------------------------------------------- + +# The INPUT tag can be used to specify the files and/or directories that contain +# documented source files. You may enter file names like "myfile.cpp" or +# directories like "/usr/src/myproject". Separate the files or directories +# with spaces. + +INPUT = @top_srcdir@/src + +# If the value of the INPUT tag contains directories, you can use the +# FILE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp +# and *.h) to filter out the source-files in the directories. If left +# blank the following patterns are tested: +# *.c *.cc *.cxx *.cpp *.c++ *.java *.ii *.ixx *.ipp *.i++ *.inl *.h *.hh *.hxx +# *.hpp *.h++ *.idl *.odl *.cs *.php *.php3 *.inc *.m *.mm + +FILE_PATTERNS = *.h + +# The RECURSIVE tag can be used to turn specify whether or not subdirectories +# should be searched for input files as well. Possible values are YES and NO. +# If left blank NO is used. + +RECURSIVE = YES + +# The EXCLUDE tag can be used to specify files and/or directories that should +# excluded from the INPUT source files. This way you can easily exclude a +# subdirectory from a directory tree whose root is specified with the INPUT tag. + +EXCLUDE = + +# The EXCLUDE_SYMLINKS tag can be used select whether or not files or +# directories that are symbolic links (a Unix filesystem feature) are excluded +# from the input. + +EXCLUDE_SYMLINKS = NO + +# If the value of the INPUT tag contains directories, you can use the +# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude +# certain files from those directories. + +EXCLUDE_PATTERNS = + +# The EXAMPLE_PATH tag can be used to specify one or more files or +# directories that contain example code fragments that are included (see +# the \include command). + +EXAMPLE_PATH = + +# If the value of the EXAMPLE_PATH tag contains directories, you can use the +# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp +# and *.h) to filter out the source-files in the directories. If left +# blank all files are included. + +EXAMPLE_PATTERNS = + +# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be +# searched for input files to be used with the \include or \dontinclude +# commands irrespective of the value of the RECURSIVE tag. +# Possible values are YES and NO. If left blank NO is used. + +EXAMPLE_RECURSIVE = NO + +# The IMAGE_PATH tag can be used to specify one or more files or +# directories that contain image that are included in the documentation (see +# the \image command). + +IMAGE_PATH = + +# The INPUT_FILTER tag can be used to specify a program that doxygen should +# invoke to filter for each input file. Doxygen will invoke the filter program +# by executing (via popen()) the command , where +# is the value of the INPUT_FILTER tag, and is the name of an +# input file. Doxygen will then use the output that the filter program writes +# to standard output. If FILTER_PATTERNS is specified, this tag will be +# ignored. + +INPUT_FILTER = + +# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern +# basis. Doxygen will compare the file name with each pattern and apply the +# filter if there is a match. The filters are a list of the form: +# pattern=filter (like *.cpp=my_cpp_filter). See INPUT_FILTER for further +# info on how filters are used. If FILTER_PATTERNS is empty, INPUT_FILTER +# is applied to all files. + +FILTER_PATTERNS = + +# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using +# INPUT_FILTER) will be used to filter the input files when producing source +# files to browse (i.e. when SOURCE_BROWSER is set to YES). + +FILTER_SOURCE_FILES = NO + +#--------------------------------------------------------------------------- +# configuration options related to source browsing +#--------------------------------------------------------------------------- + +# If the SOURCE_BROWSER tag is set to YES then a list of source files will +# be generated. Documented entities will be cross-referenced with these sources. +# Note: To get rid of all source code in the generated output, make sure also +# VERBATIM_HEADERS is set to NO. + +SOURCE_BROWSER = NO + +# Setting the INLINE_SOURCES tag to YES will include the body +# of functions and classes directly in the documentation. + +INLINE_SOURCES = NO + +# Setting the STRIP_CODE_COMMENTS tag to YES (the default) will instruct +# doxygen to hide any special comment blocks from generated source code +# fragments. Normal C and C++ comments will always remain visible. + +STRIP_CODE_COMMENTS = YES + +# If the REFERENCED_BY_RELATION tag is set to YES (the default) +# then for each documented function all documented +# functions referencing it will be listed. + +REFERENCED_BY_RELATION = YES + +# If the REFERENCES_RELATION tag is set to YES (the default) +# then for each documented function all documented entities +# called/used by that function will be listed. + +REFERENCES_RELATION = YES + +# If the VERBATIM_HEADERS tag is set to YES (the default) then Doxygen +# will generate a verbatim copy of the header file for each class for +# which an include is specified. Set to NO to disable this. + +VERBATIM_HEADERS = YES + +#--------------------------------------------------------------------------- +# configuration options related to the alphabetical class index +#--------------------------------------------------------------------------- + +# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index +# of all compounds will be generated. Enable this if the project +# contains a lot of classes, structs, unions or interfaces. + +ALPHABETICAL_INDEX = NO + +# If the alphabetical index is enabled (see ALPHABETICAL_INDEX) then +# the COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns +# in which this list will be split (can be a number in the range [1..20]) + +COLS_IN_ALPHA_INDEX = 5 + +# In case all classes in a project start with a common prefix, all +# classes will be put under the same header in the alphabetical index. +# The IGNORE_PREFIX tag can be used to specify one or more prefixes that +# should be ignored while generating the index headers. + +IGNORE_PREFIX = + +#--------------------------------------------------------------------------- +# configuration options related to the HTML output +#--------------------------------------------------------------------------- + +# If the GENERATE_HTML tag is set to YES (the default) Doxygen will +# generate HTML output. + +GENERATE_HTML = YES + +# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `html' will be used as the default path. + +HTML_OUTPUT = html + +# The HTML_FILE_EXTENSION tag can be used to specify the file extension for +# each generated HTML page (for example: .htm,.php,.asp). If it is left blank +# doxygen will generate files with .html extension. + +HTML_FILE_EXTENSION = .html + +# The HTML_HEADER tag can be used to specify a personal HTML header for +# each generated HTML page. If it is left blank doxygen will generate a +# standard header. + +HTML_HEADER = + +# The HTML_FOOTER tag can be used to specify a personal HTML footer for +# each generated HTML page. If it is left blank doxygen will generate a +# standard footer. + +HTML_FOOTER = + +# The HTML_STYLESHEET tag can be used to specify a user-defined cascading +# style sheet that is used by each HTML page. It can be used to +# fine-tune the look of the HTML output. If the tag is left blank doxygen +# will generate a default style sheet. Note that doxygen will try to copy +# the style sheet file to the HTML output directory, so don't put your own +# stylesheet in the HTML output directory as well, or it will be erased! + +HTML_STYLESHEET = + +# If the HTML_ALIGN_MEMBERS tag is set to YES, the members of classes, +# files or namespaces will be aligned in HTML using tables. If set to +# NO a bullet list will be used. + +HTML_ALIGN_MEMBERS = YES + +# If the GENERATE_HTMLHELP tag is set to YES, additional index files +# will be generated that can be used as input for tools like the +# Microsoft HTML help workshop to generate a compressed HTML help file (.chm) +# of the generated HTML documentation. + +GENERATE_HTMLHELP = NO + +# If the GENERATE_HTMLHELP tag is set to YES, the CHM_FILE tag can +# be used to specify the file name of the resulting .chm file. You +# can add a path in front of the file if the result should not be +# written to the html output directory. + +CHM_FILE = + +# If the GENERATE_HTMLHELP tag is set to YES, the HHC_LOCATION tag can +# be used to specify the location (absolute path including file name) of +# the HTML help compiler (hhc.exe). If non-empty doxygen will try to run +# the HTML help compiler on the generated index.hhp. + +HHC_LOCATION = + +# If the GENERATE_HTMLHELP tag is set to YES, the GENERATE_CHI flag +# controls if a separate .chi index file is generated (YES) or that +# it should be included in the master .chm file (NO). + +GENERATE_CHI = NO + +# If the GENERATE_HTMLHELP tag is set to YES, the BINARY_TOC flag +# controls whether a binary table of contents is generated (YES) or a +# normal table of contents (NO) in the .chm file. + +BINARY_TOC = NO + +# The TOC_EXPAND flag can be set to YES to add extra items for group members +# to the contents of the HTML help documentation and to the tree view. + +TOC_EXPAND = NO + +# The DISABLE_INDEX tag can be used to turn on/off the condensed index at +# top of each HTML page. The value NO (the default) enables the index and +# the value YES disables it. + +DISABLE_INDEX = NO + +# This tag can be used to set the number of enum values (range [1..20]) +# that doxygen will group on one line in the generated HTML documentation. + +ENUM_VALUES_PER_LINE = 4 + +# If the GENERATE_TREEVIEW tag is set to YES, a side panel will be +# generated containing a tree-like index structure (just like the one that +# is generated for HTML Help). For this to work a browser that supports +# JavaScript, DHTML, CSS and frames is required (for instance Mozilla 1.0+, +# Netscape 6.0+, Internet explorer 5.0+, or Konqueror). Windows users are +# probably better off using the HTML help feature. + +GENERATE_TREEVIEW = NO + +# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be +# used to set the initial width (in pixels) of the frame in which the tree +# is shown. + +TREEVIEW_WIDTH = 250 + +#--------------------------------------------------------------------------- +# configuration options related to the LaTeX output +#--------------------------------------------------------------------------- + +# If the GENERATE_LATEX tag is set to YES (the default) Doxygen will +# generate Latex output. + +GENERATE_LATEX = NO + +# The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `latex' will be used as the default path. + +LATEX_OUTPUT = latex + +# The LATEX_CMD_NAME tag can be used to specify the LaTeX command name to be +# invoked. If left blank `latex' will be used as the default command name. + +LATEX_CMD_NAME = latex + +# The MAKEINDEX_CMD_NAME tag can be used to specify the command name to +# generate index for LaTeX. If left blank `makeindex' will be used as the +# default command name. + +MAKEINDEX_CMD_NAME = makeindex + +# If the COMPACT_LATEX tag is set to YES Doxygen generates more compact +# LaTeX documents. This may be useful for small projects and may help to +# save some trees in general. + +COMPACT_LATEX = NO + +# The PAPER_TYPE tag can be used to set the paper type that is used +# by the printer. Possible values are: a4, a4wide, letter, legal and +# executive. If left blank a4wide will be used. + +PAPER_TYPE = a4wide + +# The EXTRA_PACKAGES tag can be to specify one or more names of LaTeX +# packages that should be included in the LaTeX output. + +EXTRA_PACKAGES = + +# The LATEX_HEADER tag can be used to specify a personal LaTeX header for +# the generated latex document. The header should contain everything until +# the first chapter. If it is left blank doxygen will generate a +# standard header. Notice: only use this tag if you know what you are doing! + +LATEX_HEADER = + +# If the PDF_HYPERLINKS tag is set to YES, the LaTeX that is generated +# is prepared for conversion to pdf (using ps2pdf). The pdf file will +# contain links (just like the HTML output) instead of page references +# This makes the output suitable for online browsing using a pdf viewer. + +PDF_HYPERLINKS = NO + +# If the USE_PDFLATEX tag is set to YES, pdflatex will be used instead of +# plain latex in the generated Makefile. Set this option to YES to get a +# higher quality PDF documentation. + +USE_PDFLATEX = NO + +# If the LATEX_BATCHMODE tag is set to YES, doxygen will add the \\batchmode. +# command to the generated LaTeX files. This will instruct LaTeX to keep +# running if errors occur, instead of asking the user for help. +# This option is also used when generating formulas in HTML. + +LATEX_BATCHMODE = NO + +# If LATEX_HIDE_INDICES is set to YES then doxygen will not +# include the index chapters (such as File Index, Compound Index, etc.) +# in the output. + +LATEX_HIDE_INDICES = NO + +#--------------------------------------------------------------------------- +# configuration options related to the RTF output +#--------------------------------------------------------------------------- + +# If the GENERATE_RTF tag is set to YES Doxygen will generate RTF output +# The RTF output is optimized for Word 97 and may not look very pretty with +# other RTF readers or editors. + +GENERATE_RTF = NO + +# The RTF_OUTPUT tag is used to specify where the RTF docs will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `rtf' will be used as the default path. + +RTF_OUTPUT = rtf + +# If the COMPACT_RTF tag is set to YES Doxygen generates more compact +# RTF documents. This may be useful for small projects and may help to +# save some trees in general. + +COMPACT_RTF = NO + +# If the RTF_HYPERLINKS tag is set to YES, the RTF that is generated +# will contain hyperlink fields. The RTF file will +# contain links (just like the HTML output) instead of page references. +# This makes the output suitable for online browsing using WORD or other +# programs which support those fields. +# Note: wordpad (write) and others do not support links. + +RTF_HYPERLINKS = NO + +# Load stylesheet definitions from file. Syntax is similar to doxygen's +# config file, i.e. a series of assignments. You only have to provide +# replacements, missing definitions are set to their default value. + +RTF_STYLESHEET_FILE = + +# Set optional variables used in the generation of an rtf document. +# Syntax is similar to doxygen's config file. + +RTF_EXTENSIONS_FILE = + +#--------------------------------------------------------------------------- +# configuration options related to the man page output +#--------------------------------------------------------------------------- + +# If the GENERATE_MAN tag is set to YES (the default) Doxygen will +# generate man pages + +GENERATE_MAN = YES + +# The MAN_OUTPUT tag is used to specify where the man pages will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `man' will be used as the default path. + +MAN_OUTPUT = man + +# The MAN_EXTENSION tag determines the extension that is added to +# the generated man pages (default is the subroutine's section .3) + +MAN_EXTENSION = .3 + +# If the MAN_LINKS tag is set to YES and Doxygen generates man output, +# then it will generate one additional man file for each entity +# documented in the real man page(s). These additional files +# only source the real man page, but without them the man command +# would be unable to find the correct page. The default is NO. + +MAN_LINKS = NO + +#--------------------------------------------------------------------------- +# configuration options related to the XML output +#--------------------------------------------------------------------------- + +# If the GENERATE_XML tag is set to YES Doxygen will +# generate an XML file that captures the structure of +# the code including all documentation. + +GENERATE_XML = NO + +# The XML_OUTPUT tag is used to specify where the XML pages will be put. +# If a relative path is entered the value of OUTPUT_DIRECTORY will be +# put in front of it. If left blank `xml' will be used as the default path. + +XML_OUTPUT = xml + +# The XML_SCHEMA tag can be used to specify an XML schema, +# which can be used by a validating XML parser to check the +# syntax of the XML files. + +XML_SCHEMA = + +# The XML_DTD tag can be used to specify an XML DTD, +# which can be used by a validating XML parser to check the +# syntax of the XML files. + +XML_DTD = + +# If the XML_PROGRAMLISTING tag is set to YES Doxygen will +# dump the program listings (including syntax highlighting +# and cross-referencing information) to the XML output. Note that +# enabling this will significantly increase the size of the XML output. + +XML_PROGRAMLISTING = YES + +#--------------------------------------------------------------------------- +# configuration options for the AutoGen Definitions output +#--------------------------------------------------------------------------- + +# If the GENERATE_AUTOGEN_DEF tag is set to YES Doxygen will +# generate an AutoGen Definitions (see autogen.sf.net) file +# that captures the structure of the code including all +# documentation. Note that this feature is still experimental +# and incomplete at the moment. + +GENERATE_AUTOGEN_DEF = NO + +#--------------------------------------------------------------------------- +# configuration options related to the Perl module output +#--------------------------------------------------------------------------- + +# If the GENERATE_PERLMOD tag is set to YES Doxygen will +# generate a Perl module file that captures the structure of +# the code including all documentation. Note that this +# feature is still experimental and incomplete at the +# moment. + +GENERATE_PERLMOD = NO + +# If the PERLMOD_LATEX tag is set to YES Doxygen will generate +# the necessary Makefile rules, Perl scripts and LaTeX code to be able +# to generate PDF and DVI output from the Perl module output. + +PERLMOD_LATEX = NO + +# If the PERLMOD_PRETTY tag is set to YES the Perl module output will be +# nicely formatted so it can be parsed by a human reader. This is useful +# if you want to understand what is going on. On the other hand, if this +# tag is set to NO the size of the Perl module output will be much smaller +# and Perl will parse it just the same. + +PERLMOD_PRETTY = YES + +# The names of the make variables in the generated doxyrules.make file +# are prefixed with the string contained in PERLMOD_MAKEVAR_PREFIX. +# This is useful so different doxyrules.make files included by the same +# Makefile don't overwrite each other's variables. + +PERLMOD_MAKEVAR_PREFIX = + +#--------------------------------------------------------------------------- +# Configuration options related to the preprocessor +#--------------------------------------------------------------------------- + +# If the ENABLE_PREPROCESSING tag is set to YES (the default) Doxygen will +# evaluate all C-preprocessor directives found in the sources and include +# files. + +ENABLE_PREPROCESSING = YES + +# If the MACRO_EXPANSION tag is set to YES Doxygen will expand all macro +# names in the source code. If set to NO (the default) only conditional +# compilation will be performed. Macro expansion can be done in a controlled +# way by setting EXPAND_ONLY_PREDEF to YES. + +MACRO_EXPANSION = NO + +# If the EXPAND_ONLY_PREDEF and MACRO_EXPANSION tags are both set to YES +# then the macro expansion is limited to the macros specified with the +# PREDEFINED and EXPAND_AS_PREDEFINED tags. + +EXPAND_ONLY_PREDEF = NO + +# If the SEARCH_INCLUDES tag is set to YES (the default) the includes files +# in the INCLUDE_PATH (see below) will be search if a #include is found. + +SEARCH_INCLUDES = YES + +# The INCLUDE_PATH tag can be used to specify one or more directories that +# contain include files that are not input files but should be processed by +# the preprocessor. + +INCLUDE_PATH = + +# You can use the INCLUDE_FILE_PATTERNS tag to specify one or more wildcard +# patterns (like *.h and *.hpp) to filter out the header-files in the +# directories. If left blank, the patterns specified with FILE_PATTERNS will +# be used. + +INCLUDE_FILE_PATTERNS = + +# The PREDEFINED tag can be used to specify one or more macro names that +# are defined before the preprocessor is started (similar to the -D option of +# gcc). The argument of the tag is a list of macros of the form: name +# or name=definition (no spaces). If the definition and the = are +# omitted =1 is assumed. To prevent a macro definition from being +# undefined via #undef or recursively expanded use the := operator +# instead of the = operator. + +PREDEFINED = + +# If the MACRO_EXPANSION and EXPAND_ONLY_PREDEF tags are set to YES then +# this tag can be used to specify a list of macro names that should be expanded. +# The macro definition that is found in the sources will be used. +# Use the PREDEFINED tag if you want to use a different macro definition. + +EXPAND_AS_DEFINED = + +# If the SKIP_FUNCTION_MACROS tag is set to YES (the default) then +# doxygen's preprocessor will remove all function-like macros that are alone +# on a line, have an all uppercase name, and do not end with a semicolon. Such +# function macros are typically used for boiler-plate code, and will confuse +# the parser if not removed. + +SKIP_FUNCTION_MACROS = YES + +#--------------------------------------------------------------------------- +# Configuration::additions related to external references +#--------------------------------------------------------------------------- + +# The TAGFILES option can be used to specify one or more tagfiles. +# Optionally an initial location of the external documentation +# can be added for each tagfile. The format of a tag file without +# this location is as follows: +# TAGFILES = file1 file2 ... +# Adding location for the tag files is done as follows: +# TAGFILES = file1=loc1 "file2 = loc2" ... +# where "loc1" and "loc2" can be relative or absolute paths or +# URLs. If a location is present for each tag, the installdox tool +# does not have to be run to correct the links. +# Note that each tag file must have a unique name +# (where the name does NOT include the path) +# If a tag file is not located in the directory in which doxygen +# is run, you must also specify the path to the tagfile here. + +TAGFILES = + +# When a file name is specified after GENERATE_TAGFILE, doxygen will create +# a tag file that is based on the input files it reads. + +GENERATE_TAGFILE = + +# If the ALLEXTERNALS tag is set to YES all external classes will be listed +# in the class index. If set to NO only the inherited external classes +# will be listed. + +ALLEXTERNALS = NO + +# If the EXTERNAL_GROUPS tag is set to YES all external groups will be listed +# in the modules index. If set to NO, only the current project's groups will +# be listed. + +EXTERNAL_GROUPS = YES + +# The PERL_PATH should be the absolute path and name of the perl script +# interpreter (i.e. the result of `which perl'). + +PERL_PATH = @PERL@ + +#--------------------------------------------------------------------------- +# Configuration options related to the dot tool +#--------------------------------------------------------------------------- + +# If the CLASS_DIAGRAMS tag is set to YES (the default) Doxygen will +# generate a inheritance diagram (in HTML, RTF and LaTeX) for classes with base +# or super classes. Setting the tag to NO turns the diagrams off. Note that +# this option is superseded by the HAVE_DOT option below. This is only a +# fallback. It is recommended to install and use dot, since it yields more +# powerful graphs. + +CLASS_DIAGRAMS = YES + +# If set to YES, the inheritance and collaboration graphs will hide +# inheritance and usage relations if the target is undocumented +# or is not a class. + +HIDE_UNDOC_RELATIONS = YES + +# If you set the HAVE_DOT tag to YES then doxygen will assume the dot tool is +# available from the path. This tool is part of Graphviz, a graph visualization +# toolkit from AT&T and Lucent Bell Labs. The other options in this section +# have no effect if this option is set to NO (the default) + +HAVE_DOT = NO + +# If the CLASS_GRAPH and HAVE_DOT tags are set to YES then doxygen +# will generate a graph for each documented class showing the direct and +# indirect inheritance relations. Setting this tag to YES will force the +# the CLASS_DIAGRAMS tag to NO. + +CLASS_GRAPH = YES + +# If the COLLABORATION_GRAPH and HAVE_DOT tags are set to YES then doxygen +# will generate a graph for each documented class showing the direct and +# indirect implementation dependencies (inheritance, containment, and +# class references variables) of the class with other documented classes. + +COLLABORATION_GRAPH = YES + +# If the GROUP_GRAPHS and HAVE_DOT tags are set to YES then doxygen +# will generate a graph for groups, showing the direct groups dependencies + +GROUP_GRAPHS = YES + +# If the UML_LOOK tag is set to YES doxygen will generate inheritance and +# collaboration diagrams in a style similar to the OMG's Unified Modeling +# Language. + +UML_LOOK = NO + +# If set to YES, the inheritance and collaboration graphs will show the +# relations between templates and their instances. + +TEMPLATE_RELATIONS = NO + +# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDE_GRAPH, and HAVE_DOT +# tags are set to YES then doxygen will generate a graph for each documented +# file showing the direct and indirect include dependencies of the file with +# other documented files. + +INCLUDE_GRAPH = YES + +# If the ENABLE_PREPROCESSING, SEARCH_INCLUDES, INCLUDED_BY_GRAPH, and +# HAVE_DOT tags are set to YES then doxygen will generate a graph for each +# documented header file showing the documented files that directly or +# indirectly include this file. + +INCLUDED_BY_GRAPH = YES + +# If the CALL_GRAPH and HAVE_DOT tags are set to YES then doxygen will +# generate a call dependency graph for every global function or class method. +# Note that enabling this option will significantly increase the time of a run. +# So in most cases it will be better to enable call graphs for selected +# functions only using the \callgraph command. + +CALL_GRAPH = NO + +# If the GRAPHICAL_HIERARCHY and HAVE_DOT tags are set to YES then doxygen +# will graphical hierarchy of all classes instead of a textual one. + +GRAPHICAL_HIERARCHY = YES + +# If the DIRECTORY_GRAPH, SHOW_DIRECTORIES and HAVE_DOT tags are set to YES +# then doxygen will show the dependencies a directory has on other directories +# in a graphical way. The dependency relations are determined by the #include +# relations between the files in the directories. + +DIRECTORY_GRAPH = YES + +# The DOT_IMAGE_FORMAT tag can be used to set the image format of the images +# generated by dot. Possible values are png, jpg, or gif +# If left blank png will be used. + +DOT_IMAGE_FORMAT = png + +# The tag DOT_PATH can be used to specify the path where the dot tool can be +# found. If left blank, it is assumed the dot tool can be found in the path. + +DOT_PATH = + +# The DOTFILE_DIRS tag can be used to specify one or more directories that +# contain dot files that are included in the documentation (see the +# \dotfile command). + +DOTFILE_DIRS = + +# The MAX_DOT_GRAPH_WIDTH tag can be used to set the maximum allowed width +# (in pixels) of the graphs generated by dot. If a graph becomes larger than +# this value, doxygen will try to truncate the graph, so that it fits within +# the specified constraint. Beware that most browsers cannot cope with very +# large images. + +MAX_DOT_GRAPH_WIDTH = 1024 + +# The MAX_DOT_GRAPH_HEIGHT tag can be used to set the maximum allows height +# (in pixels) of the graphs generated by dot. If a graph becomes larger than +# this value, doxygen will try to truncate the graph, so that it fits within +# the specified constraint. Beware that most browsers cannot cope with very +# large images. + +MAX_DOT_GRAPH_HEIGHT = 1024 + +# The MAX_DOT_GRAPH_DEPTH tag can be used to set the maximum depth of the +# graphs generated by dot. A depth value of 3 means that only nodes reachable +# from the root by following a path via at most 3 edges will be shown. Nodes +# that lay further from the root node will be omitted. Note that setting this +# option to 1 or 2 may greatly reduce the computation time needed for large +# code bases. Also note that a graph may be further truncated if the graph's +# image dimensions are not sufficient to fit the graph (see MAX_DOT_GRAPH_WIDTH +# and MAX_DOT_GRAPH_HEIGHT). If 0 is used for the depth value (the default), +# the graph is not depth-constrained. + +MAX_DOT_GRAPH_DEPTH = 0 + +# Set the DOT_TRANSPARENT tag to YES to generate images with a transparent +# background. This is disabled by default, which results in a white background. +# Warning: Depending on the platform used, enabling this option may lead to +# badly anti-aliased labels on the edges of a graph (i.e. they become hard to +# read). + +DOT_TRANSPARENT = NO + +# Set the DOT_MULTI_TARGETS tag to YES allow dot to generate multiple output +# files in one run (i.e. multiple -o and -T options on the command line). This +# makes dot run faster, but since only newer versions of dot (>1.8.10) +# support this, this feature is disabled by default. + +DOT_MULTI_TARGETS = NO + +# If the GENERATE_LEGEND tag is set to YES (the default) Doxygen will +# generate a legend page explaining the meaning of the various boxes and +# arrows in the dot generated graphs. + +GENERATE_LEGEND = YES + +# If the DOT_CLEANUP tag is set to YES (the default) Doxygen will +# remove the intermediate dot files that are used to generate +# the various graphs. + +DOT_CLEANUP = YES + +#--------------------------------------------------------------------------- +# Configuration::additions related to the search engine +#--------------------------------------------------------------------------- + +# The SEARCHENGINE tag specifies whether or not a search engine should be +# used. If set to NO the values of all tags below this one will be ignored. + +SEARCHENGINE = NO Index: /branches/rel-1-0/glueforge/templates/psdb/droptable.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/droptable.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/droptable.tt (revision 11645) @@ -0,0 +1,4 @@ +bool [% table.namespace %]DropTable(psDB *dbh) +{ + return psDBDropTable(dbh, [% table.namespace FILTER upper %]_TABLE_NAME); +} Index: /branches/rel-1-0/glueforge/templates/psdb/droptable_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/droptable_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/droptable_at.tt (revision 11645) @@ -0,0 +1,9 @@ +AT_SETUP([[% table.namespace %]DropTable()]) +AT_KEYWORDS([[% table.namespace %]DropTable]) + +# make sure the table is not there. +AT_CHECK([dbsetup]) +AT_CHECK([droptable]) +AT_CHECK([dbcleanup]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/droptable_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/droptable_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/droptable_c.tt (revision 11645) @@ -0,0 +1,25 @@ +#include +#include <[% pkg_name %].h> +#include + +int main () +{ +[% FOREACH table = tables -%] + { + psDB *dbh; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + if (![% table.namespace %]DropTable(dbh)) { + exit(EXIT_FAILURE); + } + + psDBCleanup(dbh); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/droptable_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/droptable_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/droptable_h.tt (revision 11645) @@ -0,0 +1,8 @@ +/** Deletes a [% table.name %] table + * + * @return true on success + */ + +bool [% table.namespace %]DropTable( + psDB *dbh ///< Database handle +); Index: /branches/rel-1-0/glueforge/templates/psdb/header.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/header.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/header.tt (revision 11645) @@ -0,0 +1,72 @@ +/* + * header.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +/* + * + * This file was generated by glueforge [% glueforge_version %] + * + * Do NOT directly edit this file. + * + */ + +#ifndef [% pkg_name FILTER upper %]_H +#define [% pkg_name FILTER upper %]_H 1 + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/// @addtogroup [% pkg_name %] +/// @{ + +[% INCLUDE init_h.tt %] +[% INCLUDE cleanup_h.tt %] +[% INCLUDE printmetadata_h.tt %] +[% INCLUDE printmetadataraw_h.tt %] +[% INCLUDE printmetadatas_h.tt %] +[% INCLUDE printmetadatasraw_h.tt %] +[% FOREACH table = tables -%] +[% INCLUDE object_h.tt %] +[% INCLUDE alloc_h.tt %] +[% INCLUDE createtable_h.tt %] +[% INCLUDE droptable_h.tt %] +[% INCLUDE insert_h.tt %] +[% INCLUDE delete_h.tt %] +[% INCLUDE insertobject_h.tt %] +[% INCLUDE insertobjects_h.tt %] +[% INCLUDE insertfits_h.tt %] +[% INCLUDE selectrowsfits_h.tt %] +[% INCLUDE metadatafromobject_h.tt %] +[% INCLUDE objectfrommetadata_h.tt -%] +[% INCLUDE selectrowobjects_h.tt -%] +[% INCLUDE deleteobject_h.tt -%] +[% INCLUDE deleterowobjects_h.tt -%] +[% INCLUDE printobjects_h.tt -%] +[% INCLUDE printobject_h.tt -%] +[% END -%] + +/// @} + +#ifdef __cplusplus +} +#endif + +#endif // [% table.namespace FILTER upper %]_DB_H Index: /branches/rel-1-0/glueforge/templates/psdb/init.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/init.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/init.tt (revision 11645) @@ -0,0 +1,4 @@ +psDB *[% pkg_namespace %]Init(const char *host, const char *user, const char *passwd, const char *dbname, unsigned int port) +{ + return psDBInit(host, user, passwd, dbname, port); +} Index: /branches/rel-1-0/glueforge/templates/psdb/init_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/init_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/init_at.tt (revision 11645) @@ -0,0 +1,6 @@ +AT_SETUP([[% table.namespace %]Init()]) +AT_KEYWORDS([[% table.namespace %]Init]) + +AT_CHECK([init]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/init_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/init_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/init_c.tt (revision 11645) @@ -0,0 +1,17 @@ +#include +#include <[% pkg_name %].h> +#include + +int main () +{ + psDB *dbh; + + dbh = [% pkg_namespace %]Init("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + psDBCleanup(dbh); + + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/init_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/init_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/init_h.tt (revision 11645) @@ -0,0 +1,13 @@ +/** Opens a new database connection + * + * @return A new psDB object if the database connection is successful or NULL + * on failure. + */ + +psDB *[% pkg_namespace %]Init( + const char *host, ///< Database server hostname + const char *user, ///< Database username + const char *passwd, ///< Database password + const char *dbname, ///< Database table.namespace + unsigned int port ///< Database port +); Index: /branches/rel-1-0/glueforge/templates/psdb/insert.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insert.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insert.tt (revision 11645) @@ -0,0 +1,19 @@ +bool [% table.namespace %]Insert(psDB * dbh, +[%- SET i = 0 -%] +[% FOREACH item = table.columns -%] +[% IF i == 0 %][% i = 1 %] [% ELSE %], [% END -%] +[% IF item.type == "STR" -%] +const char *[% item.name %] +[%- ELSE -%] +[% item.ctype %] [% item.name %] +[%- END -%] +[% END -%] +) +{ +[% INCLUDE metadata_from_params -%] + + bool status = psDBInsertOneRow(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, md); + psFree(md); + + return status; +} Index: /branches/rel-1-0/glueforge/templates/psdb/insert_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insert_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insert_at.tt (revision 11645) @@ -0,0 +1,8 @@ +AT_SETUP([[% table.namespace %]Insert()]) +AT_KEYWORDS([[% table.namespace %]Insert]) + +AT_CHECK([dbsetup]) +AT_CHECK([insert]) +AT_CHECK([dbcleanup]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/insert_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insert_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insert_c.tt (revision 11645) @@ -0,0 +1,32 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +#include +#include <[% pkg_name %].h> +#include + +int main () +{ +[% FOREACH table = tables -%] + { + psDB *dbh; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + if (![% table.namespace %]Insert(dbh, +[%- SET i = 0 -%] +[%- FOREACH item = table.columns -%] +[%- IF i == 0 -%][%- i = 1 -%][%- ELSE -%], [% END -%] [%- item.test -%] +[%- END -%] +)) { + exit(EXIT_FAILURE); + } + + psDBCleanup(dbh); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/insert_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insert_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insert_h.tt (revision 11645) @@ -0,0 +1,21 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +/** Insert a single row into a table + * + * This function constructs and inserts a single row based on it's parameters. + * + * @return true on success + */ + +bool [% table.namespace %]Insert( + psDB *dbh, ///< Database handle +[%- SET i = 0 -%] +[% FOREACH item = table.columns -%] +[% IF i == 0 %][% i = 1 %][% ELSE %],[% END %] +[% IF item.type == "STR" -%] + [% indented("const char", "*$item.name") -%] +[% ELSE -%] + [% indented("$item.ctype", "$item.name") -%] +[% END -%] +[% END %] +); Index: /branches/rel-1-0/glueforge/templates/psdb/insertfits.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertfits.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertfits.tt (revision 11645) @@ -0,0 +1,34 @@ +bool [% table.namespace %]InsertFits(psDB *dbh, const psFits *fits) +{ + psArray *rowSet; + + // move to (the first?) extension named [% table.namespace FILTER upper %]_TABLE_NAME + if (!psFitsMoveExtName(fits, [% table.namespace FILTER upper %]_TABLE_NAME)) { + psError(PS_ERR_UNKNOWN, true, "failed to find FITS extension %s", [% table.namespace FILTER upper %]_TABLE_NAME); + return false; + } + + // check HDU type + if (psFitsGetExtType(fits) != PS_FITS_TYPE_BINARY_TABLE) { + psError(PS_ERR_UNKNOWN, true, "FITS HDU type is not PS_FITS_TYPE_BINARY_TABLE"); + return false; + } + + // read fits table + rowSet = psFitsReadTable(fits); + if (!rowSet) { + psError(PS_ERR_UNKNOWN, true, "FITS read error or FITS table is empty"); + psFree(rowSet); + return false; + } + + if (!psDBInsertRows(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, rowSet)) { + psError(PS_ERR_UNKNOWN, false, "databse insert failed"); + psFree(rowSet); + return false; + } + + psFree(rowSet); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/insertfits_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertfits_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertfits_at.tt (revision 11645) @@ -0,0 +1,15 @@ +AT_SETUP([[% table.namespace %]InsertFits()]) +AT_KEYWORDS([[% table.namespace %]InsertFits]) + +AT_CHECK([dbsetup]) +# run insert so there is a row in the table +AT_CHECK([insert]) +# run popfis so there is afits file to read +AT_CHECK([selectrowsfits]) +AT_CHECK([insertfits]) +AT_CHECK([dbcleanup]) +if [ test -f "blargh" ] ; then + rm -f "blargh" +fi + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/insertfits_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertfits_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertfits_c.tt (revision 11645) @@ -0,0 +1,40 @@ +#include +#include <[% pkg_name %].h> +#include +#include +#include + +#define TMP_FILENAME "./blargh" + +int main () +{ +[% FOREACH table = tables -%] + { + psDB *dbh; + psFits *fits; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + // open a temp + fits = psFitsOpen(TMP_FILENAME, "r"); + if (!fits) { + exit(EXIT_FAILURE); + } + + if (![% table.namespace %]InsertFits(dbh, fits)) { + exit(EXIT_FAILURE); + } + + if (!psFitsClose(fits)) { + exit(EXIT_FAILURE); + } + + psDBCleanup(dbh); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/insertfits_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertfits_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertfits_h.tt (revision 11645) @@ -0,0 +1,14 @@ +/** Insert data from a binary FITS table [% table.object_name %] into the database + * + * This function expects a psFits object with a FITS table as the first + * extension. The table must have at least one row of data in it, that is of + * the appropriate format (number of columns and their type). All other + * extensions are ignored. + * + * @return true on success + */ + +bool [% table.namespace %]InsertFits( + psDB *dbh, ///< Database handle + const psFits *fits ///< psFits object +); Index: /branches/rel-1-0/glueforge/templates/psdb/insertobject.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertobject.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertobject.tt (revision 11645) @@ -0,0 +1,10 @@ +bool [% table.namespace %]InsertObject(psDB *dbh, [% table.object_name %] *object) +{ + return [% table.namespace %]Insert(dbh, +[%- SET i = 0 -%] +[%- FOREACH item = table.columns -%] +[%- IF i == 0 -%][%- i = 1 -%] [% ELSE %], [% END -%] +object->[% item.name -%] +[%- END -%] +); +} Index: /branches/rel-1-0/glueforge/templates/psdb/insertobject_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertobject_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertobject_at.tt (revision 11645) @@ -0,0 +1,8 @@ +AT_SETUP([[% table.namespace %]InsertObject()]) +AT_KEYWORDS([[% table.namespace %]InsertObject]) + +AT_CHECK([dbsetup]) +AT_CHECK([insertobject]) +AT_CHECK([dbcleanup]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/insertobject_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertobject_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertobject_c.tt (revision 11645) @@ -0,0 +1,39 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +#include +#include <[% pkg_name %].h> +#include + +int main () +{ +[% FOREACH table = tables -%] + { + psDB *dbh; + [% indented(table.object_name, "*object") %]; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + object = [% table.object_name %]Alloc( +[%- SET i = 0 -%] +[%- FOREACH item = table.columns -%] + [%- IF i == 0 -%][%- i = 1 -%][%- ELSE -%], [% END -%] [%- item.test -%] +[%- END -%] +); + if (!object) { + exit(EXIT_FAILURE); + } + + if (![% table.namespace %]InsertObject(dbh, object)) { + exit(EXIT_FAILURE); + } + + psFree(object); + psDBCleanup(dbh); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/insertobject_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertobject_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertobject_h.tt (revision 11645) @@ -0,0 +1,13 @@ +[% USE format -%] +[% indented = format('%-15s %-20s') -%] +/** Insert a single [% table.object_name %] object into a table + * + * This function constructs and inserts a single row based on it's parameters. + * + * @return true on success + */ + +bool [% table.namespace %]InsertObject( + psDB *dbh, ///< Database handle + [% indented(table.object_name, "*object") %]///< [% table.object_name %] object +); Index: /branches/rel-1-0/glueforge/templates/psdb/insertobjects.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertobjects.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertobjects.tt (revision 11645) @@ -0,0 +1,10 @@ +bool [% table.namespace %]InsertObjects(psDB *dbh, psArray *objects) +{ + for (long i = 0; i < psArrayLength(objects); i++) { + if (![% table.namespace %]InsertObject(dbh, objects->data[i])) { + return false; + } + } + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/insertobjects_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/insertobjects_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/insertobjects_h.tt (revision 11645) @@ -0,0 +1,11 @@ +/** Insert an array of [% table.object_name %] object into a table + * + * This function constructs and inserts multiple rows based on it's parameters. + * + * @return true on success + */ + +bool [% table.namespace %]InsertObjects( + psDB *dbh, ///< Database handle + psArray *objects ///< array of [% table.object_name %] objects +); Index: /branches/rel-1-0/glueforge/templates/psdb/lookup_from_metadata =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/lookup_from_metadata (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/lookup_from_metadata (revision 11645) @@ -0,0 +1,14 @@ +bool status = false; +[% FOREACH item = table.columns -%] +[% IF item.type == "STR" OR item.mtype == "PS_DATA_TIME" -%] + [% item.ctype %] [% item.name %] = psMetadataLookupPtr(&status, md, "[% item.name %]"); +[% ELSIF item.type == "BOOL" -%] + [% item.ctype %] [% item.name %] = psMetadataLookupBool(&status, md, "[% item.name %]"); +[% ELSE -%] + [% item.ctype %] [% item.name %] = psMetadataLookup[% item.type %](&status, md, "[% item.name %]"); +[% END -%] + if (!status) { + psError(PS_ERR_UNKNOWN, true, "failed to lookup value for item [% item.name %]"); + return false; + } +[% END -%] Index: /branches/rel-1-0/glueforge/templates/psdb/makefile.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/makefile.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/makefile.tt (revision 11645) @@ -0,0 +1,19 @@ +SHELL = /bin/sh +CFLAGS = -g -O0 -pipe -fpic -Wall -pedantic -std=c99 +LIBS = -lpslib +CFLAGS += `mysql_config --cflags` +LIBS += `mysql_config --libs` + + +objects=[% table.namespace %]db.o + +all: lib[% table.namespace %]db.so + +lib[% table.namespace %]db.so: $(objects) + $(CC) $(CFLAGS) -shared -o lib[% table.namespace %]db.so $(objects) $(LIBS) + +$(objects): %.o : %.c + $(CC) $(CFLAGS) -o $@ -c $< + +clean: + $(RM) *.so *.o core test Index: /branches/rel-1-0/glueforge/templates/psdb/metadata =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadata (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadata (revision 11645) @@ -0,0 +1,29 @@ + psMetadata *md = psMetadataAlloc(); +[% FOREACH item = table.columns -%] +[% IF item.comment; + SET item.comment = "\"$item.comment\""; +ELSE; + SET item.comment = 'NULL'; +END; +-%] +[% IF item.type == "STR" -%] + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", [% item.mtype %], [% item.comment %], "[% item.value %]")) { +[% ELSIF item.mtype == "PS_DATA_TIME" -%] +[% IF DEFINED(item.value) -%] + [% item.ctype %] [% item.name %] = psTimeFromISO("[% item.value %]", [% item.timetype %]); + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", [% item.mtype %], [% item.comment %], [% item.name %])) { + psFree([% item.name %]); +[% ELSE -%] + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", [% item.mtype %], [% item.comment %], NULL)) { +[% END -%] +[% ELSE -%] + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", [% item.mtype %], [% item.comment %], [% item.value %])) { +[% END -%] + psError(PS_ERR_UNKNOWN, false, "failed to add item [% item.name %]"); + psFree(md); + return false; + } +[% IF item.mtype == "PS_DATA_TIME" AND DEFINED(item.value) -%] + psFree([% item.name %]); +[% END -%] +[% END -%] Index: /branches/rel-1-0/glueforge/templates/psdb/metadata_from_params =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadata_from_params (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadata_from_params (revision 11645) @@ -0,0 +1,8 @@ + psMetadata *md = psMetadataAlloc(); +[% FOREACH item = table.columns -%] + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", [% item.mtype %], NULL, [% item.name %])) { + psError(PS_ERR_UNKNOWN, false, "failed to add item [% item.name %]"); + psFree(md); + return false; + } +[% END -%] Index: /branches/rel-1-0/glueforge/templates/psdb/metadata_from_struct =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadata_from_struct (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadata_from_struct (revision 11645) @@ -0,0 +1,8 @@ + psMetadata *md = psMetadataAlloc(); +[% FOREACH item = table.columns -%] + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", [% item.mtype %], NULL, object->[% item.name %])) { + psError(PS_ERR_UNKNOWN, false, "failed to add item [% item.name %]"); + psFree(md); + return false; + } +[% END -%] Index: /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject.tt (revision 11645) @@ -0,0 +1,6 @@ +psMetadata *[% table.namespace %]MetadataFromObject(const [% table.object_name %] *object) +{ +[% INCLUDE metadata_from_struct %] + + return md; +} Index: /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_at.tt (revision 11645) @@ -0,0 +1,6 @@ +AT_SETUP([[% table.namespace %]MetadataFromObject()]) +AT_KEYWORDS([[% table.namespace %]MetadataFromObject]) + +AT_CHECK([metadatafromobject]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_c.tt (revision 11645) @@ -0,0 +1,53 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +#include +#include <[% pkg_name %].h> +#include +#include + +#define MAX_STRING_LENGTH 1024 + +int main () +{ +[% FOREACH table = tables -%] + { + psMetadata *md; + [% indented(table.object_name, "*object") %]; + bool status; + + object = [% table.object_name %]Alloc( +[%- SET i = 0 -%] +[%- FOREACH item = table.columns -%] + [%- IF i == 0 -%][%- i = 1 -%][%- ELSE -%], [% END -%] [%- item.test -%] +[%- END -%] +); + if (!object) { + exit(EXIT_FAILURE); + } + + md = [% table.namespace %]MetadataFromObject(object); + if (!md) { + exit(EXIT_FAILURE); + } + + psFree(object); + +[% FOREACH item = table.columns -%] +[% IF item.type == "S32" or item.type == "F32" or item.type == "F64" -%] + if (!psMetadataLookup[% item.type %](&status, md, "[% item.name %]") == [% item.test %]) { +[% ELSIF item.type == "STR" -%] + if (strncmp(psMetadataLookupPtr(&status, md, "[% item.name %]"), [% item.test %], MAX_STRING_LENGTH)) { +[% ELSIF item.type == "BOOL" -%] + if (!psMetadataLookupBool(&status, md, "[% item.name %]") == [% item.test %]) { +[% END -%] + psFree(md); + exit(EXIT_FAILURE); + } +[% END -%] + + psFree(md); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/metadatafromobject_h.tt (revision 11645) @@ -0,0 +1,10 @@ +[% USE format -%] +[% indented = format('%-15s %-20s') -%] +/** Convert a [% table.object_name %] into an equivalent psMetadata + * + * @return A psMetadata pointer or NULL on error + */ + +psMetadata *[% table.namespace %]MetadataFromObject( + [% indented("const $table.object_name", "*object") %]///< fooRow to convert into a psMetadata +); Index: /branches/rel-1-0/glueforge/templates/psdb/object_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/object_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/object_h.tt (revision 11645) @@ -0,0 +1,17 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +/** [% table.object_name %] data structure + * + * Structure for representing a single row of [% table.name %] table data. + */ + +typedef struct { +[% SET i = 0 -%] +[% FOREACH item = table.columns -%] +[% IF item.type == "STR" -%] + [% indented("char", "*$item.name") %]; +[% ELSE -%] + [% indented("$item.ctype", "$item.name") %]; +[% END -%] +[% END -%] +} [% table.object_name %]; Index: /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata.tt (revision 11645) @@ -0,0 +1,14 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +[% table.object_name %] *[% table.namespace %]ObjectFromMetadata(psMetadata *md) +{ + +[% INCLUDE lookup_from_metadata -%] + + return [% table.object_name %]Alloc( +[%- SET i = 0 -%] +[% FOREACH item = table.columns -%] +[% IF i == 0 -%][%- i = 1 -%][% ELSE %], [% END -%] +[% item.name %] +[%- END -%]); +} Index: /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_at.tt (revision 11645) @@ -0,0 +1,6 @@ +AT_SETUP([[% table.namespace %]ObjectFromMetadata()]) +AT_KEYWORDS([[% table.namespace %]ObjectFromMetadata]) + +AT_CHECK([objectfrommetadata]) + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_c.tt (revision 11645) @@ -0,0 +1,55 @@ +[% USE format -%] +[% indented = format('%-15s %s') -%] +#include +#include <[% pkg_name %].h> +#include +#include + +#define MAX_STRING_LENGTH 1024 + +int main () +{ +[% FOREACH table = tables -%] + { + psMetadata *md; + [% indented(table.object_name, "*object") %]; + + md = psMetadataAlloc(); +[% FOREACH item = table.columns -%] +[% IF item.type == "S32" or item.type == "F32" or item.type == "F64" -%] + if (!psMetadataAdd[% item.type %](md, PS_LIST_TAIL, "[% item.name %]", 0, NULL, [% item.test %])) { +[% ELSIF item.type == "STR" -%] + if (!psMetadataAddStr(md, PS_LIST_TAIL, "[% item.name %]", 0, NULL, [% item.test %])) { +[% ELSIF item.type == "BOOL" -%] + if (!psMetadataAdd(md, PS_LIST_TAIL, "[% item.name %]", PS_DATA_BOOL, NULL, [% item.test %])) { +[% END -%] + psFree(md); + exit(EXIT_FAILURE); + } +[% END -%] + + object = [% table.namespace %]ObjectFromMetadata(md); + if (!object) { + psFree(md); + exit(EXIT_FAILURE); + } + + psFree(md); + +[% FOREACH item = table.columns -%] +[% IF item.type == "S32" or item.type == "F32" or item.type == "F64" or item.type == "BOOL" -%] + if (!object->[% item.name %] == [% item.test %]) { +[% ELSIF item.type == "STR" -%] + if (strncmp(object->[% item.name %], [% item.test %], MAX_STRING_LENGTH)) { +[% END -%] + psFree(object); + exit(EXIT_FAILURE); + } +[% END -%] + + psFree(object); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/objectfrommetadata_h.tt (revision 11645) @@ -0,0 +1,8 @@ +/** Convert a psMetadata into an equivalent fooRow + * + * @return A [% table.object_name %] pointer or NULL on error + */ + +[% table.object_name %] *[% table.namespace %]ObjectFromMetadata( + psMetadata *md ///< psMetadata to convert into a fooRow +); Index: /branches/rel-1-0/glueforge/templates/psdb/pkgconfig_pc_in.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/pkgconfig_pc_in.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/pkgconfig_pc_in.tt (revision 11645) @@ -0,0 +1,11 @@ +prefix=@prefix@ +exec_prefix=@exec_prefix@ +libdir=@libdir@ +includedir=@includedir@ + +Name: @PACKAGE@ +Description: some library +Version: @VERSION@ +Requires: pslib +Libs: -L${libdir} -l@PACKAGE@ +Cflags: -I${includedir} Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadata.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadata.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadata.tt (revision 11645) @@ -0,0 +1,15 @@ +bool [% pkg_namespace %]PrintMetadata(FILE *stream, psMetadata *md, bool mdcf) +{ + PS_ASSERT_PTR_NON_NULL(md, false); + + psMetadata *clean = psMetadataCopy(NULL, md); + + if (![% pkg_namespace %]PrintMetadataRaw(stream, clean, mdcf)) { + psError(PS_ERR_UNKNOWN, false, "failed to print metadata"); + psFree(clean); + return false; + } + psFree(clean); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadata_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadata_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadata_h.tt (revision 11645) @@ -0,0 +1,15 @@ +/** Formats and prints a metadata + * + * Any internal use fields are stripped. + * + * When mdcf is set the formated output is in psMetadataConfig + * format, otherwise it is in a simple tabular format. + * + * @return true on success + */ + +bool [% pkg_namespace %]PrintMetadata( + FILE *stream, ///< a stream + psMetadata *md, ///< An array of metadata + bool mdcf ///< format as mdconfig or simple +); Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadataraw.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadataraw.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadataraw.tt (revision 11645) @@ -0,0 +1,92 @@ +bool [% pkg_namespace %]PrintMetadataRaw(FILE *stream, psMetadata *md, bool mdcf) +{ + PS_ASSERT_PTR_NON_NULL(md, false); + + if (mdcf) { + psString str = psMetadataConfigFormat(md); + if (!str) { + psError(PS_ERR_UNKNOWN, false, "failed to format data into a string"); + psFree(str); + return false; + } + fprintf(stream, "%s\n", str); + psFree(str); + + return true; + } + +#define METADATAITEM_STRIFY_CASE(ptype, format, type) \ +case ptype: \ + psStringAppend(&str, format, item->data.type); \ + break; + + // else + // flatten the metadata into | separated values + psString str = NULL; + psMetadataItem *item = NULL; + psMetadataIterator *iter = psMetadataIteratorAlloc(md, 0, NULL); + while ((item = psMetadataGetAndIncrement(iter))) { + switch (item->type) { + METADATAITEM_STRIFY_CASE(PS_DATA_S8, "%hhd", S8); + METADATAITEM_STRIFY_CASE(PS_DATA_S16, "%hd", S16); + METADATAITEM_STRIFY_CASE(PS_DATA_S32, "%d", S32); + METADATAITEM_STRIFY_CASE(PS_DATA_S64, "%" PRId64, S64); + METADATAITEM_STRIFY_CASE(PS_DATA_U8, "%hhu", U8); + METADATAITEM_STRIFY_CASE(PS_DATA_U16, "%hu", U16); + METADATAITEM_STRIFY_CASE(PS_DATA_U32, "%u", U32); + METADATAITEM_STRIFY_CASE(PS_DATA_U64, "%" PRIu64, U64); + METADATAITEM_STRIFY_CASE(PS_DATA_F32, "%f", F32); + METADATAITEM_STRIFY_CASE(PS_DATA_F64, "%f", F64); + case PS_DATA_STRING: + if (item->data.str) { + psString tmpStr = psStringCopy(item->data.str); + psStringSubstitute(&tmpStr, "_", " "); + psStringAppend(&str, "%s", tmpStr); + psFree(tmpStr); + } + break; + case PS_DATA_BOOL: + if (item->data.B) { + psStringAppend(&str, "T"); + } else { + psStringAppend(&str, "F"); + } + break; + case PS_DATA_METADATA: + if (![% pkg_namespace %]PrintMetadataRaw(stream, item->data.md, mdcf)) { + psError(PS_ERR_UNKNOWN, false ,"failed to print nested metadata"); + } + // a metadata is a special case. We don't want a | seperating + // the metadata from any other output so we need to skip the + // !->offEnd test + continue; + case PS_DATA_TIME: + // pass through NULLs as "NULL" + if (item->data.V) { + psString time = psTimeToISO(item->data.V); + psStringAppend(&str, "%s", time); + psFree(time); + } else { + psStringAppend(&str, "NULL"); + } + break; + default: + psError(PS_ERR_UNKNOWN, true,"unsupported psMetadataItem type"); + psFree(iter); + psFree(str); + return false; + } + if (!iter->iter->offEnd) { + psStringAppend(&str, " "); + } + } + psFree(iter); + + // if we did nothing but handle recursive metadatas str will be NULL + if (str) { + fprintf(stream, "%s\n", str); + psFree(str); + } + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadataraw_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadataraw_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadataraw_h.tt (revision 11645) @@ -0,0 +1,15 @@ +/** Formats and prints a metadata + * + * The metadata is printed verbatium without removing any fields. + * + * When mdcf is set the formated output is in psMetadataConfig + * format, otherwise it is in a simple tabular format. + * + * @return true on success + */ + +bool [% pkg_namespace %]PrintMetadataRaw( + FILE *stream, ///< a stream + psMetadata *md, ///< An array of metadata + bool mdcf ///< format as mdconfig or simple +); Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadatas.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadatas.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadatas.tt (revision 11645) @@ -0,0 +1,28 @@ +bool [% pkg_namespace %]PrintMetadatas(FILE *stream, psArray *mds, const char *mdname, bool mdcf) +{ + PS_ASSERT_PTR_NON_NULL(mds, false); + PS_ASSERT_PTR_NON_NULL(mdname, false); + + psMetadata *output = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(mds); i++) { + psMetadata *md = psMetadataCopy(NULL, mds->data[i]); + if (!psMetadataAddMetadata(output, PS_LIST_TAIL, mdname, PS_META_DUPLICATE_OK, NULL, md)) { + psError(PS_ERR_UNKNOWN, false, "failed to add a metadata item"); + psFree(md); + psFree(output); + return false; + } + + psFree(md); + } + + if (![% pkg_namespace %]PrintMetadata(stream, output, mdcf)) { + psError(PS_ERR_UNKNOWN, false, "failed to print metadata"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadatas_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadatas_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadatas_h.tt (revision 11645) @@ -0,0 +1,16 @@ +/** Formats and prints an array of metadata + * + * Any internal use fields are stripped. + * + * When mdcf is set the formated output is in psMetadataConfig + * format, otherwise it is in a simple tabular format. + * + * @return true on success + */ + +bool [% pkg_namespace %]PrintMetadatas( + FILE *stream, ///< a stream + psArray *mds, ///< An array of metadata + const char *mdname, ///< name of the metadata(s) + bool mdcf ///< format as mdconfig or simple +); Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadatasraw.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadatasraw.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadatasraw.tt (revision 11645) @@ -0,0 +1,25 @@ +bool [% pkg_namespace %]PrintMetadatasRaw(FILE *stream, psArray *mds, const char *mdname, bool mdcf) +{ + PS_ASSERT_PTR_NON_NULL(mds, false); + PS_ASSERT_PTR_NON_NULL(mdname, false); + + psMetadata *output = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(mds); i++) { + psMetadata *md = mds->data[i]; + if (!psMetadataAddMetadata(output, PS_LIST_TAIL, mdname, PS_META_DUPLICATE_OK, NULL, md)) { + psError(PS_ERR_UNKNOWN, false, "failed to add a metadata item"); + psFree(output); + return false; + } + } + + if (![% pkg_namespace %]PrintMetadataRaw(stream, output, mdcf)) { + psError(PS_ERR_UNKNOWN, false, "failed to print metadata"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/printmetadatasraw_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printmetadatasraw_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printmetadatasraw_h.tt (revision 11645) @@ -0,0 +1,16 @@ +/** Formats and prints an array of metadata + * + * The metadatas are printed verbatium without removing any fields. + * + * When mdcf is set the formated output is in psMetadataConfig + * format, otherwise it is in a simple tabular format. + * + * @return true on success + */ + +bool [% pkg_namespace %]PrintMetadatasRaw( + FILE *stream, ///< a stream + psArray *mds, ///< An array of metadata + const char *mdname, ///< name of the metadata(s) + bool mdcf ///< format as mdconfig or simple +); Index: /branches/rel-1-0/glueforge/templates/psdb/printobject.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printobject.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printobject.tt (revision 11645) @@ -0,0 +1,15 @@ +bool [% table.namespace %]PrintObject(FILE *stream, [% table.object_name %] *object, bool mdcf) +{ + PS_ASSERT_PTR_NON_NULL(object, false); + + psMetadata *md = [% table.namespace %]MetadataFromObject(object); + + if (![% pkg_namespace %]PrintMetadataRaw(stream, md, mdcf)) { + psError(PS_ERR_UNKNOWN, false, "failed to print metadata"); + psFree(md); + } + + psFree(md); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/printobject_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printobject_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printobject_h.tt (revision 11645) @@ -0,0 +1,13 @@ +/** Formats and prints an [% table.object_name %] object + * + * When mdcf is set the formated output is in psMetadataConfig + * format, otherwise it is in a simple tabular format. + * + * @return true on success + */ + +bool [% table.namespace %]PrintObject( + FILE *stream, ///< a stream + [% table.object_name %] *object, ///< an [% table.object_name %] object + bool mdcf ///< format as mdconfig or simple +); Index: /branches/rel-1-0/glueforge/templates/psdb/printobjects.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printobjects.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printobjects.tt (revision 11645) @@ -0,0 +1,31 @@ +bool [% table.namespace %]PrintObjects(FILE *stream, psArray *objects, bool mdcf) +{ + PS_ASSERT_PTR_NON_NULL(objects, false); + + psMetadata *output = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(objects); i++) { + psMetadata *md = [% table.namespace %]MetadataFromObject(objects->data[i]); + if (!psMetadataAddMetadata( + output, + PS_LIST_TAIL, + [% table.namespace FILTER upper %]_TABLE_NAME, + PS_META_DUPLICATE_OK, + NULL, + md + )) { + psError(PS_ERR_UNKNOWN, false, "failed to add metadata"); + psFree(md); + psFree(output); + return false; + } + psFree(md); + } + + if (![% pkg_namespace %]PrintMetadataRaw(stream, output, mdcf)) { + psError(PS_ERR_UNKNOWN, false, "failed to print metadata"); + psFree(output); + } + psFree(output); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/printobjects_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/printobjects_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/printobjects_h.tt (revision 11645) @@ -0,0 +1,13 @@ +/** Formats and prints an array of [% table.object_name %] objects + * + * When mdcf is set the formated output is in psMetadataConfig + * format, otherwise it is in a simple tabular format. + * + * @return true on success + */ + +bool [% table.namespace %]PrintObjects( + FILE *stream, ///< a stream + psArray *objects, ///< An array of [% table.object_name %] objects + bool mdcf ///< format as mdconfig or simple +); Index: /branches/rel-1-0/glueforge/templates/psdb/selectrowobjects.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/selectrowobjects.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/selectrowobjects.tt (revision 11645) @@ -0,0 +1,25 @@ +psArray *[% table.namespace %]SelectRowObjects(psDB *dbh, const psMetadata *where, unsigned long long limit) +{ + psArray *rowSet; + psArray *returnSet; + psU64 i; + + rowSet = psDBSelectRows(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, where, limit); + if (!rowSet) { + return NULL; + } + + // convert psMetadata rows to row objects + + returnSet = psArrayAllocEmpty(rowSet->n); + + for (i = 0; i < rowSet->n; i++) { + [% table.object_name %] *object = [% table.namespace %]ObjectFromMetadata(rowSet->data[i]); + psArrayAdd(returnSet, 0, object); + psFree(object); + } + + psFree(rowSet); + + return returnSet; +} Index: /branches/rel-1-0/glueforge/templates/psdb/selectrowobjects_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/selectrowobjects_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/selectrowobjects_h.tt (revision 11645) @@ -0,0 +1,12 @@ +/** Selects up to limit rows from the database and returns as [% table.object_name %] objects in a psArray + * + * See psDBSelectRows() for documentation on the format of where. + * + * @return A psArray pointer or NULL on error + */ + +psArray *[% table.namespace %]SelectRowObjects( + psDB *dbh, ///< Database handle + const psMetadata *where, ///< Row match criteria + unsigned long long limit ///< Maximum number of elements to return +); Index: /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits.tt (revision 11645) @@ -0,0 +1,20 @@ +bool [% table.namespace %]SelectRowsFits(psDB *dbh, psFits *fits, const psMetadata *where, unsigned long long limit) +{ + psArray *rowSet; + + rowSet = psDBSelectRows(dbh, [% table.namespace FILTER upper %]_TABLE_NAME, where, limit); + if (!rowSet) { + return false; + } + + // output to fits + if (!psFitsWriteTable(fits, NULL, rowSet, [% table.namespace FILTER upper %]_TABLE_NAME)) { + psError(PS_ERR_UNKNOWN, false, "FITS table write failed"); + psFree(rowSet); + return false; + } + + psFree(rowSet); + + return true; +} Index: /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_at.tt (revision 11645) @@ -0,0 +1,13 @@ +AT_SETUP([[% table.namespace %]SelectRowsFits()]) +AT_KEYWORDS([[% table.namespace %]SelectRowsFits]) + +AT_CHECK([dbsetup]) +# run insert so there is a row in the table +AT_CHECK([insert]) +AT_CHECK([selectrowsfits]) +AT_CHECK([dbcleanup]) +if [ test -f "blargh" ] ; then + rm -f "blargh" +fi + +AT_CLEANUP Index: /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_c.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_c.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_c.tt (revision 11645) @@ -0,0 +1,34 @@ +#include +#include <[% pkg_name %].h> +#include + +#define TMP_FILENAME "./blargh" + +int main () +{ +[% FOREACH table = tables -%] + { + psDB *dbh; + psFits *fits; + + dbh = psDBInit("localhost", "test", NULL, "test"); + if (!dbh) { + exit(EXIT_FAILURE); + } + + fits = psFitsOpen(TMP_FILENAME, "w"); + if (!fits) { + exit(EXIT_FAILURE); + } + + if (![% table.namespace %]SelectRowsFits(dbh, fits, NULL, 1)) { + exit(EXIT_FAILURE); + } + + psFree(fits); + psDBCleanup(dbh); + } + +[% END -%] + exit(EXIT_SUCCESS); +} Index: /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_h.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_h.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/selectrowsfits_h.tt (revision 11645) @@ -0,0 +1,16 @@ +/** Selects up to limit from the database and returns them in a binary FITS table + * + * This function assumes an empty psFits object and will create a FITS table + * as the first extension. + * + * See psDBSelectRows() for documentation on the format of where. + * + * @return true on success + */ + +bool [% table.namespace %]SelectRowsFits( + psDB *dbh, ///< Database handle + psFits *fits, ///< psFits object + const psMetadata *where, ///< Row match criteria + unsigned long long limit ///< Maximum number of elements to return +); Index: /branches/rel-1-0/glueforge/templates/psdb/src_makefile_am.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/src_makefile_am.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/src_makefile_am.tt (revision 11645) @@ -0,0 +1,10 @@ +AM_CFLAGS = @[% pkg_name %]_CFLAGS@ + +include_HEADERS = [% pkg_name %].h +lib_LTLIBRARIES = lib[% pkg_name %].la +lib[% pkg_name %]_la_CFLAGS = $(PSLIB_CFLAGS) $(AM_CFLAGS) +lib[% pkg_name %]_la_LIBS = $(PSLIB_LIBS) $(AM_LIBS) +lib[% pkg_name %]_la_LDFLAGS = -release $(PACKAGE_VERSION) +lib[% pkg_name %]_la_SOURCES = \ + [% pkg_name %].h \ + [% pkg_name %].c Index: /branches/rel-1-0/glueforge/templates/psdb/tests_makefile_am.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/tests_makefile_am.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/tests_makefile_am.tt (revision 11645) @@ -0,0 +1,64 @@ +# copied from bison-1.875d + +EXTRA_DIST = $(TESTSUITE_AT) testsuite package.m4 + +DISTCLEANFILES = atconfig $(check_SCRIPTS) +MAINTAINERCLEANFILES = Makefile.in $(TESTSUITE) + +## ------------ ## +## package.m4. ## +## ------------ ## + +$(srcdir)/package.m4: $(top_srcdir)/configure.ac + { \ + echo '# Signature of the current package.'; \ + echo 'm4_define([AT_PACKAGE_NAME], [@PACKAGE_NAME@])'; \ + echo 'm4_define([AT_PACKAGE_TARNAME], [@PACKAGE_TARNAME@])'; \ + echo 'm4_define([AT_PACKAGE_VERSION], [@PACKAGE_VERSION@])'; \ + echo 'm4_define([AT_PACKAGE_STRING], [@PACKAGE_STRING@])'; \ + echo 'm4_define([AT_PACKAGE_BUGREPORT], [@PACKAGE_BUGREPORT@])'; \ + } >$(srcdir)/package.m4 + +## ------------ ## +## Test suite. ## +## ------------ ## + +TESTSUITE = $(srcdir)/testsuite + +AUTOTEST = $(AUTOM4TE) --language=autotest +$(TESTSUITE): package.m4 $(TESTSUITE_AT) + $(AUTOTEST) -I $(srcdir) testsuite.at -o $@.tmp + mv $@.tmp $@ + +atconfig: $(top_builddir)/config.status + cd $(top_builddir) && ./config.status tests/$@ + +clean-local: + test ! -f $(TESTSUITE) || $(SHELL) $(TESTSUITE) --clean + +check-local: atconfig $(TESTSUITE) + $(SHELL) $(TESTSUITE) + +# +# END BISON +# + +UNITS = \ + alloc \ + init \ + cleanup \ + createtable \ + droptable \ + insert \ + insertobject \ + insertfits \ + selectrowsfits \ + metadatafromobject \ + objectfrommetadata + +AM_CPPFLAGS = -I$(top_srcdir)/src$ +AM_CFLAGS = @[% pkg_name %]_CFLAGS@ $(PSLIB_CFLAGS)$ +AM_LDFLAGS = $(PSLIB_LIBS)$ +LDADD = $(top_builddir)/src/lib[% pkg_name %].la$ + +check_PROGRAMS = dbsetup dbcleanup $(UNITS) Index: /branches/rel-1-0/glueforge/templates/psdb/testsuite_at.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/testsuite_at.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/testsuite_at.tt (revision 11645) @@ -0,0 +1,27 @@ +m4_version_prereq([2.59]) + +AT_INIT + +### +[% INCLUDE alloc_at.tt %] +### +[% INCLUDE init_at.tt %] +### +[% INCLUDE cleanup_at.tt %] +### +[% INCLUDE createtable_at.tt %] +### +[% INCLUDE droptable_at.tt %] +### +[% INCLUDE insert_at.tt %] +### +[% INCLUDE insertobject_at.tt %] +### +[% INCLUDE insertfits_at.tt %] +### +[% INCLUDE selectrowsfits_at.tt %] +### +[% INCLUDE metadatafromobject_at.tt %] +### +[% INCLUDE objectfrommetadata_at.tt %] +### Index: /branches/rel-1-0/glueforge/templates/psdb/top_makefile_am.tt =================================================================== --- /branches/rel-1-0/glueforge/templates/psdb/top_makefile_am.tt (revision 11645) +++ /branches/rel-1-0/glueforge/templates/psdb/top_makefile_am.tt (revision 11645) @@ -0,0 +1,30 @@ +SUBDIRS = src tests + +pkgconfigdir = $(libdir)/pkgconfig +pkgconfig_DATA= [% pkg_name %].pc + +EXTRA_DIST = [% pkg_name %].pc.in Doxyfile.in + +if HAVE_DOXYGEN + +man_MANS = \ + $(top_builddir)/docs/man/man3/[% pkg_name %].3 \ +[%- FOREACH table = tables %] +[% IF NOT loop.last -%] + $(top_builddir)/docs/man/man3/[% table.object_name %].3 \ +[%- ELSE -%] + $(top_builddir)/docs/man/man3/[% table.object_name %].3 +[% END -%] +[% END %] + +docs/man/man3/[% pkg_name %].3 +[%- FOREACH table = tables -%] + docs/man/man3/[% table.object_name %].3 +[%- IF loop.last %]:[% END -%] +[%- END %] + $(DOXYGEN) + +endif + +clean-local: + -rm -rf docs Index: /branches/rel-1-0/ippMonitor/.cvsignore =================================================================== --- /branches/rel-1-0/ippMonitor/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippMonitor/.cvsignore (revision 11645) @@ -0,0 +1,1 @@ +src Index: /branches/rel-1-0/ippMonitor/Makefile =================================================================== --- /branches/rel-1-0/ippMonitor/Makefile (revision 11645) +++ /branches/rel-1-0/ippMonitor/Makefile (revision 11645) @@ -0,0 +1,157 @@ +default: php +help: + @echo "USAGE: make php" + +SRC = src +DEF = def +RAW = raw +DESTBIN = /var/www/kiawe/ippMonitor + +RAWSRC = \ +$(DESTBIN)/ipp.css \ +$(DESTBIN)/ipp.php \ +$(DESTBIN)/ipp.menu.dat \ +$(DESTBIN)/ipp.copy.dat \ +$(DESTBIN)/ipp.science.dat \ +$(DESTBIN)/ipp.detrend.dat \ +$(DESTBIN)/ipp.imfiles.dat \ +$(DESTBIN)/Login.php \ +$(DESTBIN)/SelectProject.php \ +$(DESTBIN)/LoadSetup.php \ +$(DESTBIN)/ScienceSteps.php \ +$(DESTBIN)/DetrendSteps.php \ +$(DESTBIN)/ImfileTables.php \ +$(DESTBIN)/phptest.php \ +$(DESTBIN)/getimage.php + +DEFSRC = \ +$(DESTBIN)/masterDetrendFrames.php \ +$(DESTBIN)/masterDetrendImfiles.php \ +$(DESTBIN)/detInputExp.php \ +$(DESTBIN)/detMasterFrame.php \ +$(DESTBIN)/detMasterImfile.php \ +$(DESTBIN)/detNormalizedExp.php \ +$(DESTBIN)/detNormalizedImfile.php \ +$(DESTBIN)/detNormalizedStatImfile.php \ +$(DESTBIN)/detProcessedImfile.php \ +$(DESTBIN)/detProcessedExp.php \ +$(DESTBIN)/detResidExp.php \ +$(DESTBIN)/detResidImfile.php \ +$(DESTBIN)/detRun.php \ +$(DESTBIN)/detRunSummary.php \ +$(DESTBIN)/detStackedImfile.php \ +$(DESTBIN)/newExp.php \ +$(DESTBIN)/newImfile.php \ +$(DESTBIN)/p1PendingExp.php \ +$(DESTBIN)/p2PendingExp.php \ +$(DESTBIN)/p2PendingImfile.php \ +$(DESTBIN)/p2ProcessedExp.php \ +$(DESTBIN)/p2ProcessedImfile.php \ +$(DESTBIN)/p3PendingExp.php \ +$(DESTBIN)/p3ProcessedExp.php \ +$(DESTBIN)/pzPendingExp.php \ +$(DESTBIN)/pzPendingImfile.php \ +$(DESTBIN)/rawUnknownExp.php \ +$(DESTBIN)/rawDetrendExp.php \ +$(DESTBIN)/rawDetrendExp_detrend.php \ +$(DESTBIN)/rawImfile.php \ +$(DESTBIN)/rawImfile_failed.php \ +$(DESTBIN)/rawExp_failed.php \ +$(DESTBIN)/rawScienceExp.php \ +$(DESTBIN)/summitExp.php + +PICTURES = \ +$(DESTBIN)/PScolorlogo2.jpg \ +$(DESTBIN)/missing.png + +DESTLINK = $(DESTBIN)/index.php + +php: $(RAWSRC) $(DEFSRC) $(PICTURES) $(DESTLINK) + +# dependancy rules for binary code ######################### +# .PRECIOUS: %.$(ARCH).o +# .PRECIOUS: $(BIN)/%.$(ARCH) + +$(DESTLINK): $(DESTBIN)/Login.php + rm -f $@ + ln -s $(DESTBIN)/Login.php $@ + +$(SRC)/%.php: $(DEF)/%.d $(DEF)/autocode.php ./generate + @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi + ./generate $< $(DEF)/autocode.php $@ + +# $(SRC)/%.php: $(RAW)/%.php +# @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi +# cp $< $@ +# +# $(SRC)/%.css: $(RAW)/%.css +# @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi +# cp $< $@ +# +# $(SRC)/%.dat: $(RAW)/%.dat +# @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi +# cp $< $@ +# +# $(SRC)/%.jpg: $(RAW)/%.jpg +# @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi +# cp $< $@ +# +# $(SRC)/%.png: $(RAW)/%.png +# @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi +# cp $< $@ +# +# $(SRC)/%.jpeg: $(RAW)/%.jpeg +# @if [ ! -d $(SRC) ]; then mkdir -p $(SRC); fi +# cp $< $@ + +# generated php code is built into SRC first +$(DESTBIN)/%.php: $(SRC)/%.php + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.php || exit + cp $(SRC)/$*.php $(DESTBIN)/$*.php || exit + +# non-generated php code is copied directly +$(DESTBIN)/%.php: $(RAW)/%.php + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.php || exit + cp $(RAW)/$*.php $(DESTBIN)/$*.php || exit + +# all other files are copied directly +$(DESTBIN)/%.css: $(RAW)/%.css + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.css || exit + cp $(RAW)/$*.css $(DESTBIN)/$*.css || exit + +$(DESTBIN)/%.dat: $(RAW)/%.dat + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.dat || exit + cp $(RAW)/$*.dat $(DESTBIN)/$*.dat || exit + +$(DESTBIN)/%.jpg: $(RAW)/%.jpg + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.jpg || exit + cp $(RAW)/$*.jpg $(DESTBIN)/$*.jpg || exit + +$(DESTBIN)/%.jpeg: $(RAW)/%.jpeg + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.jpeg || exit + cp $(RAW)/$*.jpeg $(DESTBIN)/$*.jpeg || exit + +$(DESTBIN)/%.png: $(RAW)/%.png + @if [ ! -d $(DESTBIN) ]; then mkdir -p $(DESTBIN) || exit; fi + rm -f $(DESTBIN)/$*.png || exit + cp $(RAW)/$*.png $(DESTBIN)/$*.png || exit + +# utilities ################################################# + +clean: + rm -f `find . -name "*.o"` + rm -f `find . -name "*~"` + rm -f `find . -name "#*"` + +.PRECIOUS: $(SRC)/%.php +.PRECIOUS: $(SRC)/%.css +.PRECIOUS: $(SRC)/%.dat +.PRECIOUS: $(SRC)/%.png +.PRECIOUS: $(SRC)/%.jpg +.PRECIOUS: $(SRC)/%.jpeg Index: /branches/rel-1-0/ippMonitor/dbadmin =================================================================== --- /branches/rel-1-0/ippMonitor/dbadmin (revision 11645) +++ /branches/rel-1-0/ippMonitor/dbadmin (revision 11645) @@ -0,0 +1,69 @@ +#!/bin/csh -f +## this script creates the tables needed by ippMonitor for user administration + +if ($#argv < 4) goto usage; + +if ("$1" == "init") then + if ($#argv != 4) goto usage; + set dbserver = $2 + set dbuser = $3 + set dbpass = $4 + + mysql -h $dbserver -u root -p < $TITLE

"; + +// set up the form +echo "
\n"; + +// define restrictiosn to the queries +// ** TABLE RESTRICTIONS ** + +// get the result table count +if ($WHERE) { + $sql = "SELECT count(*) FROM $TABLE WHERE $WHERE"; +} else { + $sql = "SELECT count(*) FROM $TABLE"; +} +$qry = $db->query($sql); +if (DB::iserror($db)) { + echo "error reading $TABLE table count
\n"; + menu_end(); +} +if (!$qry->fetchInto($row)) { + echo "error reading $TABLE table count
\n"; + menu_end(); +} +// set up the row counter variables +if ($ID['from']) { + $rowStart = $ID['from']; +} else { + $rowStart = 0; +} +$rowLast = $rowStart + $dTABLE; +$rowTotal = $row[0]; +if ($rowLast > $rowTotal) { $rowLast = $rowTotal; } +echo " $rowStart to $rowLast of $rowTotal items
\n"; + +// query the database +if ($WHERE) { + $sql = "SELECT $FIELDS FROM $TABLE WHERE $WHERE LIMIT $dTABLE OFFSET $rowStart"; +} else { + $sql = "SELECT $FIELDS FROM $TABLE LIMIT $dTABLE OFFSET $rowStart"; +} + +$qry = $db->query($sql); +if (DB::iserror($db)) { + echo "error reading $TABLE table
\n"; + menu_end(); +} + +// ** HEAD CODE ** + +if ($rowStart > 0) { + $value = 0; + $link = "$FILE" . "?" . $ID['link'] . "&from=$value"; + echo "<<\n"; +} else { + echo "<<\n"; +} +if ($rowStart >= $dTABLE) { + $value = $rowStart - $dTABLE; + $link = "$FILE" . "?" . $ID['link'] . "&from=$value"; + echo "<\n"; +} else { + echo "<\n"; +} + +if ($rowLast < $rowTotal) { + $value = $rowStart + $dTABLE; + $link = "$FILE" . "?" . $ID['link'] . "&from=$value"; + echo ">\n"; +} else { + echo ">\n"; +} + +if ($rowLast < $rowTotal) { + $value = $rowTotal - $dTABLE; + $link = "$FILE" . "?" . $ID['link'] . "&from=$value"; + echo ">>\n"; +} else { + echo ">>\n"; +} + +echo "  :   or enter start row: \n"; + +// set up the table +echo "
\n"; +echo "\n"; +// ** TABLE HEADER ** +// echo "\n"; +echo "\n"; + +// query restriction form +echo "\n"; +// echo "\n"; +// ** TABLE QUERY ** +echo "\n"; + +// list the results +while ($qry->fetchInto($row)) { + // $link = "$FILE" . "?expID=" . $row[0] . "&" . $ID['link']; + + echo "\n"; + // ** TABLE DATA ** + // echo "\n"; + // echo "\n"; + echo "\n"; +} + +// query restriction form +// echo "\n"; +// echo "\n"; +// TABLE QUERY +// echo "\n"; + +// close the table and form +echo "
FIELD
$row[0] $row[1]
\n"; +echo "\n"; +$pass = $ID['pass']; +$proj = $ID['proj']; +$menu = $ID['menu']; +echo "\n"; +echo "\n"; +echo "\n"; +echo "\n"; + +// ** TAIL CODE ** + +echo " WHERE: $WHERE

\n"; +echo " SQL: $sql

\n"; + +menu_end(); + +?> Index: /branches/rel-1-0/ippMonitor/def/detInputExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detInputExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detInputExp.d (revision 11645) @@ -0,0 +1,16 @@ +TABLE detInputExp +TITLE detInputExp +FILE detInputExp.php +MENU ipp.detrend.dat + +# HEAD HTML test head 1
+ +# field width name show link to extras +FIELD det_id, 7, det_id +FIELD exp_tag, 20, exposure, value, rawImfile.php, exp_tag=$exp_tag +FIELD iteration, 5, iteration +FIELD include, 7, include +FIELD *, 5, resid, value=goto, detResidImfile.php, exp_tag=$exp_tag,det_id=$det_id,iteration=$iteration +FIELD *, 5, proc, value=goto, detProcessedImfile.php, exp_tag=$exp_tag,det_id=$det_id,iteration=$iteration + +# TAIL HTML test tail 1
Index: /branches/rel-1-0/ippMonitor/def/detMasterFrame.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detMasterFrame.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detMasterFrame.d (revision 11645) @@ -0,0 +1,9 @@ +TABLE detMasterFrame +TITLE detMasterFrame +FILE detMasterFrame.php +MENU ipp.detrend.dat + +# field name show link to extras +FIELD det_id, 20, det_id +FIELD iteration, 20, iteration +FIELD comment, 20, comment Index: /branches/rel-1-0/ippMonitor/def/detMasterImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detMasterImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detMasterImfile.d (revision 11645) @@ -0,0 +1,10 @@ +TABLE detMasterImfile +TITLE detMasterImfile +FILE detMasterImfile.php +MENU ipp.imfiles.dat + +# field name show link to extras +FIELD det_id, 20, det_id +FIELD class_id, 20, class_id +FIELD uri, 20, uri +FIELD recipe, 20, recipe Index: /branches/rel-1-0/ippMonitor/def/detNormalizedExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detNormalizedExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detNormalizedExp.d (revision 11645) @@ -0,0 +1,14 @@ +TABLE detNormalizedExp +TITLE detNormalizedExp +FILE detNormalizedExp.php +MENU ipp.detrend.dat + +# field size name show link to extras +FIELD det_id, 7, det_id +FIELD bg, 8, backgnd +FIELD bg_mean_stdev, 8, [stdev] +FIELD bg_stdev, 8, stdev +FIELD iteration, 5, iteration +FIELD b2_uri, 8, image, image=$b1_uri@PPIMAGE.JPEG2, detNormalizedImfile.php, det_id=$det_id,iteration=$iteration,image=$b1_uri@PPIMAGE.JPEG1 +#FIELD b1_uri, 20, b1_uri +#FIELD recipe, 20, recipe Index: /branches/rel-1-0/ippMonitor/def/detNormalizedImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detNormalizedImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detNormalizedImfile.d (revision 11645) @@ -0,0 +1,17 @@ +TABLE detNormalizedImfile +TITLE detNormalizedImfile +FILE detNormalizedImfile.php +MENU ipp.imfiles.dat + +# field size name show link to extras +FIELD det_id, 7, det_id +FIELD class_id, 8, class_id +FIELD bg, 8, backgnd +FIELD bg_stdev, 8, stdev +FIELD bg_mean_stdev, 8, [stdev] +FIELD iteration, 5, iteration +#FIELD uri, 20, uri +#FIELD b1_uri, 20, b1_uri +#FIELD b2_uri, 20, b2_uri + +TAIL PHP insert_image ('image'); Index: /branches/rel-1-0/ippMonitor/def/detNormalizedStatImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detNormalizedStatImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detNormalizedStatImfile.d (revision 11645) @@ -0,0 +1,10 @@ +TABLE detNormalizedStatImfile +TITLE detNormalizedStatImfile +FILE detNormalizedStatImfile.php +MENU ipp.imfiles.dat + +# field name show link to extras +FIELD det_id, 7, det_id +FIELD class_id, 8, class_id +FIELD iteration, 5, iteration +FIELD norm, 5, norm Index: /branches/rel-1-0/ippMonitor/def/detProcessedExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detProcessedExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detProcessedExp.d (revision 11645) @@ -0,0 +1,14 @@ +TABLE detProcessedExp +TITLE detProcessedExp +FILE detProcessedExp.php +MENU ipp.detrend.dat + +# field size name show link to extras +FIELD det_id, 7, Det ID, value +FIELD exp_tag, 15, Exposure, value +FIELD b2_uri, 8, image, image=$b1_uri@PPIMAGE.JPEG2, detProcessedImfile.php, det_id=$det_id,exp_tag=$exp_tag,image=$b1_uri@PPIMAGE.JPEG1 +FIELD bg, 8, backgnd, value +FIELD bg_mean_stdev, 8, [stdev], value +FIELD bg_stdev, 8, stdev, value +FIELD recipe, 15, Recipe, value +FIELD b1_uri, 20, b1_uri, none Index: /branches/rel-1-0/ippMonitor/def/detProcessedImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detProcessedImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detProcessedImfile.d (revision 11645) @@ -0,0 +1,18 @@ +TABLE detProcessedImfile +TITLE detProcessedImfile +FILE detProcessedImfile.php +MENU ipp.imfiles.dat + +# field size name show link to extras +FIELD det_id, 7, Det ID +FIELD exp_tag, 15, Exposure +FIELD class_id, 8, Chip ID +FIELD bg, 8, backgnd +FIELD bg_stdev, 8, stdev +FIELD bg_mean_stdev, 8, [stdev] +# FIELD recipe, 10, recipe +# FIELD uri, 20, uri +# FIELD b1_uri, 20, b1_uri +# FIELD b2_uri, 20, b2_uri + +TAIL PHP insert_image ('image'); Index: /branches/rel-1-0/ippMonitor/def/detResidExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detResidExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detResidExp.d (revision 11645) @@ -0,0 +1,16 @@ +TABLE detResidExp +TITLE detResidExp +FILE detResidExp.php +MENU ipp.detrend.dat + +# field size name show link to extras +FIELD det_id, 7, Det ID, value +FIELD exp_tag, 15, Exposure, value, rawImfile.php, exp_tag=$exp_tag +FIELD b2_uri, 8, image, image=$b1_uri@PPIMAGE.JPEG2, detResidImfile.php, det_id=$det_id,iteration=$iteration,exp_tag=$exp_tag,image=$b1_uri@PPIMAGE.JPEG1 +FIELD bg, 8, backgnd, value +FIELD bg_mean_stdev, 8, [stdev] +FIELD bg_stdev, 8, stdev, value +FIELD iteration, 5, iter, value +FIELD accept, 4, keep, value +FIELD b1_uri, 20, b1_uri, none +# FIELD recipe, 15, recipe, value Index: /branches/rel-1-0/ippMonitor/def/detResidImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detResidImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detResidImfile.d (revision 11645) @@ -0,0 +1,19 @@ +TABLE detResidImfile +TITLE detResidImfile +FILE detResidImfile.php +MENU ipp.imfiles.dat + +# field size name show link to extras +FIELD det_id, 7, Det ID +FIELD exp_tag, 15, Exposure +FIELD class_id, 8, Chip ID +FIELD bg, 8, backgnd +FIELD bg_stdev, 8, bg stdev +FIELD bg_mean_stdev, 8, [stdev] +FIELD iteration, 5, iter +# FIELD uri, 20, uri +# FIELD recipe, 20, recipe +# FIELD b1_uri, 20, b1_uri +# FIELD b2_uri, 20, b2_uri + +TAIL PHP insert_image ('image'); Index: /branches/rel-1-0/ippMonitor/def/detRun.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detRun.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detRun.d (revision 11645) @@ -0,0 +1,21 @@ +TABLE detRun +TITLE detRun +FILE detRun.php +MENU ipp.detrend.dat + +# field width name show link to extras +FIELD det_id, 7, det ID, value, detRunSummary.php, det_id=$det_id +FIELD iteration, 5, iteration +FIELD det_type, 8, det_type +FIELD state, 5, state +FIELD mode, 5, mode +FIELD exp_type, 8, exp_type (drop) +FIELD filter, 10, filter +FIELD airmass, 10, airmass +FIELD exp_time, 10, exp_time +FIELD ccd_temp, 10, ccd_temp +FIELD posang, 10, posang +FIELD object, 10, object +FIELD registered, 10, registered +FIELD use_begin, 10, use_begin +FIELD use_end, 10, use_end Index: /branches/rel-1-0/ippMonitor/def/detRunSummary.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detRunSummary.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detRunSummary.d (revision 11645) @@ -0,0 +1,14 @@ +TABLE detRunSummary +TITLE detRunSummary +FILE detRunSummary.php +MENU ipp.detrend.dat + +# field name show link to extras +FIELD det_id, 7, det_id +FIELD iteration, 5, iter +FIELD accept, 4, keep +FIELD *, 20, choose, value=input, detInputExp.php, det_id=$det_id,iteration=$iteration +FIELD *, 20, choose, value=resid, detResidExp.php, det_id=$det_id,iteration=$iteration +FIELD bg, 20, bg +FIELD bg_stdev, 20, bg_stdev +# FIELD bg_mean_stdev, 20, bg_mean_stdev Index: /branches/rel-1-0/ippMonitor/def/detStackedImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/detStackedImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/detStackedImfile.d (revision 11645) @@ -0,0 +1,14 @@ +TABLE detStackedImfile +TITLE detStackedImfile +FILE detStackedImfile.php +MENU ipp.imfiles.dat + +# field name show link to extras +FIELD det_id, 7, det_id +FIELD class_id, 8, class_id +FIELD bg, 8, backgnd +FIELD bg_stdev, 8, stdev +FIELD bg_mean_stdev, 8, [stdev] +FIELD iteration, 5, iteration +# FIELD uri, 20, uri +# FIELD recipe, 20, recipe Index: /branches/rel-1-0/ippMonitor/def/example.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/example.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/example.d (revision 11645) @@ -0,0 +1,29 @@ +# this file il +# master detrend runs which have been accepted + +TABLE detRunSummary, detRun +TITLE Master Detrend Images +FILE masterDetrendImages.php +MENU ipp.detrend.dat + +# the following WHERE clauses are added to all queries joined by AND +WHERE detRunSummary.accept = 1 +WHERE detRun.det_it = detRunSummary.det_id +WHERE detRun.iteration = detRunSummary.iteration + +# field width name show link to extras +FIELD detRunSummary.det_id, 7, det_id +FIELD detRun.det_type, 5, type +FIELD detRunSummary.iteration, 5, iter +FIELD *, 20, choose, value=input, detInputExp.php, det_id=$detRunSummary.det_id,iteration=$detRunSummary.iteration +FIELD *, 20, choose, value=resid, detResidExp.php, det_id=$detRunSummary.det_id,iteration=$detRunSummary.iteration +FIELD detRunSummary.bg, 20, bg +FIELD detRunSummary.bg_stdev, 20, bg_stdev +# FIELD bg_mean_stdev, 20, bg_mean_stdev + +# this file illustrates a simple inner join to generate the output table +# assuming no other dynamic restrictions are added, this generates the following query: +# SELECT detRunSummary.det_id,detRunSummary.iteration,detRunSummary.iteration,detRun.det_type,detRunSummary.bg,detRunSummary.bg_stdev \ +# FROM detRunSummary, detRun \ +# WHERE detRunSummary.accept = 1 AND detRun.det_id = detRunSummary.det_id AND detRun.iteration = detRunSummary.iteration \ +# LIMIT 20 Index: /branches/rel-1-0/ippMonitor/def/masterDetrendFrames.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/masterDetrendFrames.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/masterDetrendFrames.d (revision 11645) @@ -0,0 +1,30 @@ +# this file is based on the detRunSummary.d file: it uses the same tables restricted to the +# master detrend runs which have been accepted + +TABLE detRunSummary, detRun +TITLE Master Detrend Frames +FILE masterDetrendFrames.php +MENU ipp.detrend.dat + +# the following WHERE clauses are added to all queries joined by AND +WHERE detRunSummary.accept = 1 +WHERE detRun.det_id = detRunSummary.det_id +WHERE detRun.iteration = detRunSummary.iteration + +# field width name show link to extras +FIELD detRunSummary.det_id, 7, det_id, value, detRunSummary.php, det_id=$detRunSummary.det_id +FIELD detRunSummary.iteration, 5, iter +FIELD detRun.det_type, 5, type +FIELD *, 20, choose, value=files, detNormalizedImfiles.php, det_id=$detRunSummary.det_id,iteration=$detRunSummary.iteration +FIELD *, 20, choose, value=input, detInputExp.php, det_id=$detRunSummary.det_id,iteration=$detRunSummary.iteration +FIELD *, 20, choose, value=resid, detResidExp.php, det_id=$detRunSummary.det_id,iteration=$detRunSummary.iteration +FIELD detRunSummary.bg, 20, bg +FIELD detRunSummary.bg_stdev, 20, bg_stdev +# FIELD bg_mean_stdev, 20, bg_mean_stdev + +# this file illustrates a simple inner join to generate the output table +# assuming no other dynamic restrictions are added, this generates the following query: +# SELECT detRunSummary.det_id,detRunSummary.iteration,detRunSummary.iteration,detRun.det_type,detRunSummary.bg,detRunSummary.bg_stdev \ +# FROM detRunSummary, detRun \ +# WHERE detRunSummary.accept = 1 AND detRun.det_id = detRunSummary.det_id AND detRun.iteration = detRunSummary.iteration \ +# LIMIT 20 Index: /branches/rel-1-0/ippMonitor/def/masterDetrendImfiles.d.src =================================================================== --- /branches/rel-1-0/ippMonitor/def/masterDetrendImfiles.d.src (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/masterDetrendImfiles.d.src (revision 11645) @@ -0,0 +1,19 @@ +# this file is based on the detRunSummary.d file: it uses the same tables restricted to the +# master detrend runs which have been accepted + +TABLE detStackedImfile +TITLE Master Detrend Imfiles +FILE masterDetrendImfiles.php +MENU ipp.detrend.dat + +# field width name show link to extras +FIELD det_id, 7, det_id +FIELD iteration, 5, iter +FIELD uri, 5, uri + +# this file illustrates a simple inner join to generate the output table +# assuming no other dynamic restrictions are added, this generates the following query: +# SELECT detRunSummary.det_id,detRunSummary.iteration,detRunSummary.iteration,detRun.det_type,detRunSummary.bg,detRunSummary.bg_stdev \ +# FROM detRunSummary, detRun \ +# WHERE detRunSummary.accept = 1 AND detRun.position = detRunSummary.det_id AND detRun.iteration = detRunSummary.iteration \ +# LIMIT 20 Index: /branches/rel-1-0/ippMonitor/def/newExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/newExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/newExp.d (revision 11645) @@ -0,0 +1,11 @@ +TABLE newExp +TITLE New Exposures +FILE newExp.php +MENU ipp.copy.dat + +# field name show link to extras +FIELD exp_id, 20, exposure ID +FIELD exp_tag, 20, exposure Tag +FIELD camera, 20, camera +FIELD telescope, 20, telescope +FIELD imfiles, 20, N imfiles Index: /branches/rel-1-0/ippMonitor/def/newImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/newImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/newImfile.d (revision 11645) @@ -0,0 +1,10 @@ +TABLE newImfile +TITLE Raw Science Exposures +FILE newImfile.php +MENU ipp.imfiles.dat + +# field name show link to extras +FIELD exp_tag, 20, exposure Tag +FIELD class, 20, class +FIELD class_id, 20, class ID +FIELD uri, 20, URI Index: /branches/rel-1-0/ippMonitor/def/p1PendingExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p1PendingExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p1PendingExp.d (revision 11645) @@ -0,0 +1,9 @@ +TABLE p1PendingExp +TITLE p1PendingExp +FILE p1PendingExp.php +MENU ipp.science.dat + +# field name show link to extras +FIELD exp_tag, 20, exposure ID, value, rawImfile.php, exp_tag=$exp_tag +FIELD recipe, 20, recipe +FIELD p1_version, 20, p1 version Index: /branches/rel-1-0/ippMonitor/def/p2PendingExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p2PendingExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p2PendingExp.d (revision 11645) @@ -0,0 +1,10 @@ +TABLE p2PendingExp +TITLE p2PendingExp +FILE p2PendingExp.php +MENU ipp.science.dat + +# field name show link to extras +FIELD exp_tag, 20, exposure ID, value, rawImfile.php, exp_tag=$exp_tag +FIELD recipe, 20, recipe +FIELD p1_version, 20, p1 version +FIELD p2_version, 20, p2 version Index: /branches/rel-1-0/ippMonitor/def/p2PendingImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p2PendingImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p2PendingImfile.d (revision 11645) @@ -0,0 +1,12 @@ +TABLE p2PendingImfile +TITLE p2PendingImfile +FILE p2PendingImfile.php +MENU ipp.imfiles.dat + +# field name show link to extras +FIELD exp_tag, 20, exposure Tag +FIELD class_id, 20, class ID +FIELD uri, 20, URI +FIELD recipe, 20, recipe +FIELD p1_version, 20, p1 version +FIELD p2_version, 20, p2 version Index: /branches/rel-1-0/ippMonitor/def/p2ProcessedExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p2ProcessedExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p2ProcessedExp.d (revision 11645) @@ -0,0 +1,9 @@ +TABLE p2ProcessedExp +TITLE p2ProcessedExp +FILE p2ProcessedExp.php +MENU ipp.science.dat + +# field name show link to extras +FIELD exp_tag, 20, exposure ID, value, p2ProcessedImfile.php, exp_tag=$exp_tag +FIELD p1_version, 20, p1 version, value, p1PendingExp.php exp_tag=$exp_tag +FIELD p2_version, 20, p2 version Index: /branches/rel-1-0/ippMonitor/def/p2ProcessedImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p2ProcessedImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p2ProcessedImfile.d (revision 11645) @@ -0,0 +1,19 @@ +TABLE p2ProcessedImfile +TITLE p2ProcessedImfile +FILE p2ProcessedImfile.php +MENU ipp.imfiles.dat + +# field size name show link to extras +FIELD exp_tag, 15, Exposure +FIELD class_id, 8, Chip ID +FIELD bg, 8, backgnd +FIELD bg_mean_stdev, 8, [stdev] +FIELD bg_stdev, 8, stdev +#FIELD p1_version, 20, p1 version +#FIELD p2_version, 20, p2 version +#FIELD recipe, 20, recipe +#FIELD uri, 20, URI +#FIELD b1_uri, 20, URI small binned +#FIELD b2_uri, 20, URI large binned + +TAIL PHP insert_image ('image'); Index: /branches/rel-1-0/ippMonitor/def/p3PendingExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p3PendingExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p3PendingExp.d (revision 11645) @@ -0,0 +1,9 @@ +TABLE p3PendingExp +TITLE p3PendingExp +FILE p3PendingExp.php +MENU ipp.science.dat + +# field name show link to extras +FIELD exp_tag, 20, exposure ID, value, p2ProcessedImfile.php, exp_tag=$exp_tag +FIELD p2_version, 20, p2 version, value, p2ProcessedExp.php, exp_tag=$exp_tag +FIELD p3_version, 20, p3 version Index: /branches/rel-1-0/ippMonitor/def/p3ProcessedExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/p3ProcessedExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/p3ProcessedExp.d (revision 11645) @@ -0,0 +1,13 @@ +TABLE p3ProcessedExp +TITLE p3ProcessedExp +FILE p3ProcessedExp.php +MENU ipp.science.dat + +# field size name show link to extras +FIELD exp_tag, 15, Exposure, value +FIELD b2_uri, 8, image, image=$b1_uri@PPIMAGE.JPEG2, p2ProcessedImfile.php, exp_tag=$exp_tag,image=$b1_uri@PPIMAGE.JPEG1 +FIELD bg, 8, backgnd, value +FIELD bg_mean_stdev, 8, [stdev], value +FIELD bg_stdev, 8, stdev, value +FIELD b1_uri, 20, b1_uri, none +#FIELD recipe, 15, Recipe, value Index: /branches/rel-1-0/ippMonitor/def/pzPendingExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/pzPendingExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/pzPendingExp.d (revision 11645) @@ -0,0 +1,9 @@ +TABLE pzPendingExp +TITLE pending exposures +FILE pzPendingExp.php +MENU ipp.copy.dat + +# field name show link to extras +FIELD exp_id, 20, exposures ID +FIELD camera, 20, camera +FIELD telescope, 20, telescope Index: /branches/rel-1-0/ippMonitor/def/pzPendingImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/pzPendingImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/pzPendingImfile.d (revision 11645) @@ -0,0 +1,11 @@ +TABLE pzPendingImfile +TITLE Pending Image Files +FILE pzPendingImfile.php +MENU ipp.imfiles.dat + +# field name show link to extras +FIELD exp_id, 20, exposure ID +FIELD camera, 20, camera +FIELD telescope, 20, telescope +FIELD class, 20, class +FIELD class_id, 20, class ID Index: /branches/rel-1-0/ippMonitor/def/rawDetrendExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawDetrendExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawDetrendExp.d (revision 11645) @@ -0,0 +1,30 @@ +TABLE rawExp +TITLE Raw Detrend Exposures +FILE rawDetrendExp.php +MENU ipp.copy.dat + +# limit this table to non-OBJECT types of images +WHERE rawExp.exp_type != 'OBJECT' + +# field name show link to extras +FIELD exp_tag, 10, exposure ID, value, rawImfile.php, exp_tag=$exp_tag +FIELD telescope, 10, Telescope +FIELD camera, 10, Camera +FIELD exp_type, 10, Type +FIELD filter, 10, FILTER +FIELD dateobs, 20, Date/Time +FIELD exp_time, 5, exp_time +FIELD airmass, 5, airmass +FIELD bg, 5, backgnd +FIELD bg_stdev, 5, stdev +FIELD bg_mean_stdev, 5, <backgnd> +# FIELD ra, 8, ra +# FIELD decl, 8, decl +# FIELD object, 10, object +# FIELD imfiles, 20, imfiles +# FIELD airmass, 20, airmass +# FIELD bg_mean_stdev, 20, bg_mean_stdev +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang Index: /branches/rel-1-0/ippMonitor/def/rawDetrendExp_detrend.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawDetrendExp_detrend.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawDetrendExp_detrend.d (revision 11645) @@ -0,0 +1,32 @@ +TABLE rawExp +TITLE Raw Detrend Exposures +FILE rawDetrendExp.php +MENU ipp.detrend.dat + +# limit this table to non-OBJECT types of images +WHERE rawExp.exp_type != 'OBJECT' + +# field name show link to extras +FIELD exp_tag, 10, exposure ID, value, rawImfile.php, exp_tag=$exp_tag +FIELD telescope, 10, Telescope +FIELD camera, 10, Camera +FIELD exp_type, 10, Type +FIELD filter, 10, FILTER +FIELD dateobs, 20, Date/Time +FIELD exp_time, 5, exp_time +FIELD airmass, 5, airmass +FIELD bg, 5, backgnd +FIELD bg_stdev, 5, stdev +FIELD bg_mean_stdev, 5, <backgnd> +# FIELD imfiles, 20, imfiles +# FIELD airmass, 20, airmass +# FIELD ra, 20, ra +# FIELD decl, 20, decl +# FIELD exp_time, 20, exp_time +# FIELD bg, 20, bg +# FIELD bg_stdev, 20, bg_stdev +# FIELD bg_mean_stdev, 20, bg_mean_stdev +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang Index: /branches/rel-1-0/ippMonitor/def/rawExp_failed.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawExp_failed.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawExp_failed.d (revision 11645) @@ -0,0 +1,30 @@ +TABLE rawExp +TITLE Raw Detrend Exposures +FILE rawExp_failed.php +MENU ipp.copy.dat + +# limit this table to non-OBJECT types of images +WHERE fault > 0 + +# field name show link to extras +FIELD exp_tag, 10, exposure ID, value, rawImfile.php, exp_tag=$exp_tag +FIELD telescope, 10, Telescope +FIELD camera, 10, Camera +FIELD exp_type, 10, Type +FIELD filter, 10, FILTER +FIELD dateobs, 20, Date/Time +FIELD exp_time, 5, exp_time +FIELD airmass, 5, airmass +FIELD bg, 5, backgnd +FIELD bg_stdev, 5, stdev +FIELD bg_mean_stdev, 5, <backgnd> +# FIELD ra, 8, ra +# FIELD decl, 8, decl +# FIELD object, 10, object +# FIELD imfiles, 20, imfiles +# FIELD airmass, 20, airmass +# FIELD bg_mean_stdev, 20, bg_mean_stdev +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang Index: /branches/rel-1-0/ippMonitor/def/rawImfile.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawImfile.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawImfile.d (revision 11645) @@ -0,0 +1,27 @@ +TABLE rawImfile +TITLE Raw Image File +FILE rawImfile.php +MENU ipp.imfiles.dat + +HEAD PHP insert_backref ("rawUnknownExp.php", "exp_tag", $ID['link']); + +# field name show link to extras +FIELD exp_tag, 15, Exposure +FIELD class_id, 8, Chip ID +FIELD bg, 8, backgnd +FIELD bg_stdev, 8, stdev +FIELD bg_mean_stdev, 8, [stdev] +FIELD uri, 10, uri +# FIELD class, 20, Class +# FIELD uri, 20, uri +# FIELD exp_type, 20, exp_type +# FIELD filter, 20, filter +# FIELD airmass, 20, airmass +# FIELD ra, 20, ra +# FIELD decl, 20, decl +# FIELD exp_time, 20, exp_time +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang +# FIELD object, 20, object Index: /branches/rel-1-0/ippMonitor/def/rawImfile_failed.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawImfile_failed.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawImfile_failed.d (revision 11645) @@ -0,0 +1,28 @@ +TABLE rawImfile +TITLE Raw Image File +FILE rawImfile.php +MENU ipp.copy.dat + +HEAD PHP insert_backref ("rawUnknownExp.php", "exp_tag", $ID['link']); +WHERE fault > 0 + +# field name show link to extras +FIELD exp_tag, 15, Exposure +FIELD class_id, 8, Chip ID +FIELD bg, 8, backgnd +FIELD bg_stdev, 8, stdev +FIELD bg_mean_stdev, 8, [stdev] +FIELD uri, 10, uri +# FIELD class, 20, Class +# FIELD uri, 20, uri +# FIELD exp_type, 20, exp_type +# FIELD filter, 20, filter +# FIELD airmass, 20, airmass +# FIELD ra, 20, ra +# FIELD decl, 20, decl +# FIELD exp_time, 20, exp_time +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang +# FIELD object, 20, object Index: /branches/rel-1-0/ippMonitor/def/rawScienceExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawScienceExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawScienceExp.d (revision 11645) @@ -0,0 +1,27 @@ +TABLE rawExp +TITLE Raw Science Exposures +FILE rawScienceExp.php +MENU ipp.copy.dat + +# limit this table to OBJECT types of images +WHERE rawExp.exp_type = 'OBJECT' + +# field name show link to extras +FIELD exp_tag, 10, exposure Tag, value, rawImfile.php, exp_tag=$exp_tag,camera=$camera,foo=bar +FIELD telescope, 10, Telescope, value, telescopes.php +FIELD camera, 10, Camera +FIELD exp_type, 8, Type +FIELD filter, 10, FILTER +FIELD ra, 8, ra +FIELD decl, 8, decl +FIELD object, 10, object +FIELD airmass, 5, airmass +FIELD exp_time, 5, exp_time +FIELD bg, 5, bg +FIELD bg_stdev, 5, bg_stdev +# FIELD imfiles, 20, imfiles +# FIELD bg_mean_stdev, 20, bg_mean_stdev +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang Index: /branches/rel-1-0/ippMonitor/def/rawUnknownExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/rawUnknownExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/rawUnknownExp.d (revision 11645) @@ -0,0 +1,30 @@ +TABLE rawExp +TITLE Raw Detrend Exposures +FILE rawDetrendExp.php +MENU ipp.copy.dat + +# limit this table to non-OBJECT types of images +WHERE rawExp.exp_type is NULL + +# field name show link to extras +FIELD exp_tag, 10, exposure ID, value, rawImfile.php, exp_tag=$exp_tag +FIELD telescope, 10, Telescope +FIELD camera, 10, Camera +FIELD exp_type, 10, Type +FIELD filter, 10, FILTER +FIELD dateobs, 20, Date/Time +FIELD exp_time, 5, exp_time +FIELD airmass, 5, airmass +FIELD bg, 5, backgnd +FIELD bg_stdev, 5, stdev +FIELD bg_mean_stdev, 5, <backgnd> +# FIELD ra, 8, ra +# FIELD decl, 8, decl +# FIELD object, 10, object +# FIELD imfiles, 20, imfiles +# FIELD airmass, 20, airmass +# FIELD bg_mean_stdev, 20, bg_mean_stdev +# FIELD alt, 20, alt +# FIELD az, 20, az +# FIELD ccd_temp, 20, ccd_temp +# FIELD posang, 20, posang Index: /branches/rel-1-0/ippMonitor/def/summitExp.d =================================================================== --- /branches/rel-1-0/ippMonitor/def/summitExp.d (revision 11645) +++ /branches/rel-1-0/ippMonitor/def/summitExp.d (revision 11645) @@ -0,0 +1,11 @@ +TABLE summitExp +TITLE Pending Summit Exposures +FILE summitExp.php +MENU ipp.copy.dat + +# field name show link to extras +FIELD exp_id, 20, exposure ID +FIELD camera, 20, camera +FIELD telescope, 20, telescope +FIELD exp_type, 20, type +FIELD uri, 20, URI Index: /branches/rel-1-0/ippMonitor/doc/notes.txt =================================================================== --- /branches/rel-1-0/ippMonitor/doc/notes.txt (revision 11645) +++ /branches/rel-1-0/ippMonitor/doc/notes.txt (revision 11645) @@ -0,0 +1,35 @@ + +Installation: + +the code currently requires three (3) hardwired paths to be set before +installation: + +- the DESTBIN directory in the Makefile must be set to the directory + in your apache installation where the package will reside + +- the database hostname, user, and password must be set in ipp.php + +- the location of the ipprc.config file must be specific in + getimage.php + +** TODO: set these with a configure script. + +- to update the mysql database from a dump: + * mysql -u root -p + create database NAME; + grant all on NAME.* to ipp@localhost; + * mysql -u ipp NAME -p < DUMPFILE + +- norm exp need to use dettool (-toresid ?) to get the list. + +- select * from detNormalizedImfile where detid,iteration + union select from detStackedImfile where detid,iteration,normalize=0 + +- create users table: + create database ippadmin; + use ippadmin; + create table users (username char(20) not null, password varchar(255)); + insert into users (username, password) values ('eugene', 'test1'); + + create table cookies (username char(20) not null, cookie varchar(255)); + insert into cookies (username, cookie) values ('eugene', 'foobar'); Index: /branches/rel-1-0/ippMonitor/generate =================================================================== --- /branches/rel-1-0/ippMonitor/generate (revision 11645) +++ /branches/rel-1-0/ippMonitor/generate (revision 11645) @@ -0,0 +1,418 @@ +#!/usr/bin/env perl + +$VERBOSE = 0; + +if ($ARGV[0] eq "-v") { $VERBOSE = 1; shift @ARGV; } +if (@ARGV != 3) { die "generate (schema) (template) (output)\n"; } + +$schema = $ARGV[0]; +$template = $ARGV[1]; +$output = $ARGV[2]; + +# global variables: +# @key, @value : paired list of keys and values +# @headcode, @tailcode : additional HTML or PHP code before or after table +# @field, @width, @name, @show, @link, @extras : properties of each table column + +&parse_schema; +&parse_template; +exit 0; + +sub parse_schema { + open (FILE, "$schema"); + @list = ; + close (FILE); + + &init_key ("MENU"); + &init_key ("STYLE"); + &init_key ("TABLE"); + &init_key ("TITLE"); + &init_key ("FILE"); + + &init_key ("FIELDS"); + + foreach $line (@list) { + chop $line; + ($key, $value) = split (" ", $line, 2); + + # strip leading and trailing white space from the following + if ($key eq "TABLE") { ($value) = $value =~ m|^\s*(.+)\s*$|; } + if ($key eq "TYPE") { ($value) = $value =~ m|^\s*(.+)\s*$|; } + + &set_keypair ($key, $value); + + # list of the table fields + if ($key eq "FIELD") { + ($field, $width, $name, $show, $link, $extras) = split (/,\s+/, $value, 6); + if (! $name) { die "table $ARGV[0] missing required elements\n"; } + # ($field) = $field =~ m|\s*(\S+)\s*|; + + if ($VERBOSE) { printf "%-20s %-20s %-20s %-20s %-20s\n", $field, $name, $show, $link, $extras; } + if ($show eq "") {$show = "value";} + push @field, $field; + push @width, $width; + push @name, $name; + push @show, $show; + push @link, $link; + push @extras, $extras; + } + + # list of the table fields + if ($key eq "HEAD") { + push @headcode, $value; + } + + # list of the table fields + if ($key eq "TAIL") { + push @tailcode, $value; + } + + # list of the table restrictions + # replace this with a flexible where statement? + if ($key eq "WHERE") { + ($field) = split (/,\s+/, $value, 1); + if ($VERBOSE) { printf "%-20s\n", $field; } + push @where, $field; + } + } + + # check on keys + &check_key ("MENU", "ipp.menu.dat"); + &check_key ("STYLE", "ipp.css"); + &check_key ("TABLE", ""); + &check_key ("TITLE", ""); + &check_key ("FILE", ""); + + # these are used internally (not just a replacement) + for ($i = 0; $i < @key; $i++) { + if ($key[$i] eq "MENU") { $MENU = $value[$i]; } + } + + # define FIELDS and WHERE strings, add to keypairs + &define_fields_count; + &define_fields_string (@field); + &define_where_string (@where); + + if (0) { + print STDERR "show: $show[0]\n"; + ($a, $b, $c) = &parse_label ($show[0]); + print STDERR "a: $a, b: $b, c: $c\n"; + + print STDERR "extras: $extras[0]\n"; + ($a, $b, $c) = &parse_label ($extras[0]); + print STDERR "a: $a, b: $b, c: $c\n"; + + print STDERR "show: $show[1]\n"; + ($a, $b, $c) = &parse_label ($show[1]); + print STDERR "a: $a, b: $b, c: $c\n"; + + print STDERR "extras: $extras[1]\n"; + ($a, $b, $c) = &parse_label ($extras[1]); + print STDERR "a: $a, b: $b, c: $c\n"; + exit 0; + } +} + +sub parse_template { + open (FILE, $template); + @list = ; + close (FILE); + + open (FILE, ">$output"); + + foreach $line (@list) { + + &check_keypairs; + + print FILE $line; + + # fill in table header + if ($line =~ m|// \*\* TABLE HEADER \*\*|) { + &write_table_header; + } + + # fill in table data + if ($line =~ m|// \*\* TABLE DATA \*\*|) { + &write_table_data; + } + + # fill in table query + if ($line =~ m|// \*\* TABLE QUERY \*\*|) { + &write_table_query; + } + + # fill in table restricts + if ($line =~ m|// \*\* TABLE RESTRICTIONS \*\*|) { + &write_table_restrict; + } + + # fill in head HTML or PHP code + if ($line =~ m|// \*\* HEAD CODE \*\*|) { + &write_inline_code (@headcode); + } + + # fill in tail HTML or PHP code + if ($line =~ m|// \*\* TAIL CODE \*\*|) { + &write_inline_code (@tailcode); + } + } + close (FILE); +} + +sub write_inline_code { + my (@code) = @_; + my ($i, $code, $type); + + for ($i = 0; $i < @code; $i++) { + ($type, $line) = split (" ", $code[$i], 2); + if ($type eq "PHP") { + printf FILE "$line\n"; + } + if ($type eq "HTML") { + printf FILE "?> $line ) for each field (show != none) +sub write_table_header { + + # add column sorting elements here + for ($i = 0; $i < @field; $i++) { + if ($show[$i] eq "none") { next; } + printf FILE "echo \" $name[$i] \\n\";\n"; + } +} + +# generate a WHERE test for each field (field != *) +sub write_table_restrict { + + if ($WHERE) { + printf FILE "\$WHERE = \"$WHERE\";\n"; + } + for ($i = 0; $i < @field; $i++) { + $value = $field[$i]; + if ($value eq "*") { next; } + printf FILE "\$WHERE = check_restrict ('$value', \$WHERE);\n"; + } +} + +sub write_table_data { + my ($i, $Nrow); + my ($label, $value, $string); + + for ($i = 0; $i < @field; $i++) { + ($label, $value, $string) = &parse_label ($show[$i]); + $Nrow = $count[$i]; + + # create the link variable if this entry should be linked + if ($label eq "none") { next; } + + if (($label eq "value") && !$link[$i]) { + # print value without link or image + if (! $value && ($field[$i] ne "*")) { $value = "\$row[$Nrow]"; } + print FILE " echo \" $value \\n\";\n"; + next; + } + + if (($label eq "value") && $link[$i]) { + # create the basic link variable + printf FILE " \$link = \"$link[$i]\" . \"?menu=$MENU&\" . \$ID['link'];\n"; + # add extra GET data to target + if ($extras[$i]) { &parse_extras ($extras[$i]); } + # print the actual table cell line with the link... + if (! $value && ($field[$i] ne "*")) { $value = "\$row[$Nrow]"; } + printf FILE " echo \" $value \\n\";\n"; + next; + } + + if (($label eq "image") && !$link[$i]) { + # print the actual table cell line with the image... + printf FILE " echo \"\\n\";\n"; + next; + } + + if (($label eq "image") && $link[$i]) { + # create the basic link + printf FILE " \$link = \"$link[$i]\" . \"?menu=$MENU&\" . \$ID['link'];\n"; + # add extra php data to target + if ($extras[$i]) { &parse_extras ($extras[$i]); } + # print the actual table cell line with the link... + printf FILE " echo \" \\n\";\n"; + next; + } + } +} + +sub write_table_query { + my ($i); + for ($i = 0; $i < @field; $i++) { + if ($show[$i] eq "none") { next; } + if ($field[$i] eq "*") { + # * fields create an empty cell + printf FILE "echo \"   \\n\";\n"; + } else { + printf FILE "write_query_row ('$field[$i]', $width[$i]);\n"; + } + } +} + +sub define_fields_string { + my (@array) = @_; + my ($i, $FIELDS); + + $FIELDS = "$array[0]"; + for ($i = 1; $i < @array; $i++) { + # skip * fields (not a valid query element + if ($array[$i] eq "*") { next; } + $FIELDS = "$FIELDS,$array[$i]"; + } + set_keypair ("FIELDS", $FIELDS); +} + +sub define_fields_count { + my ($i, $Nrow); + + @count = (); + + $Nrow = 0; + for ($i = 0; $i < @field; $i++) { + # skip * fields (not a valid query element + if ($field[$i] eq "*") { + $count[$i] = -1; + } else { + $count[$i] = $Nrow; + $Nrow ++; + } + } +} + +sub define_where_string { + my (@array) = @_; + my ($i, $FIELDS); + + if (@array == 0) { return; } + + $WHERE = "$array[0]"; + for ($i = 1; $i < @array; $i++) { + $WHERE = "$WHERE AND $array[$i]"; + } + + set_keypair ("WHERE", $WHERE); +} + +sub parse_extras { + my ($extras) = $_[0]; + my (@extfields); + my ($label, $value, $outline); + my ($i); + + # examine the extras and parse the embedded fields (comma-separated) + @extfields = split (/,/, $extras); + for ($i = 0; $i < @extfields; $i++) { + ($label, $value, $outline) = &parse_label ($extfields[$i]); + printf FILE " \$link = \$link . \"&$outline\";\n"; + } +} + +# parse strings of the form label=value +# if 'value' is of the form $field, try to match with +# a supplied DB field, and replace with $row[$n] if found +sub parse_label { + my ($string) = $_[0]; + my ($label, $value, $outline); + my ($Nrow, $i); + + # unless we replace the value with the db row element, return the original string + $outline = $string; + $label = $string; + $value = ""; + + # search for embedded table fields and replace with row[N] + if ($VERBOSE) { print STDERR "string: $string\n"; } + if ($string =~ m|\S+=\S+|) { + ($label, $value) = $string =~ m|(\S+)=(\S+)|; + if ($value =~ m|^\$|) { + MATCH_LABEL: + for ($i = 0; $i < @field; $i++) { + if ($field[$i] eq "*") { next; } + if ($value =~ m|^\$$field[$i]|) { + # print "field: $field[$i]\n"; + ($name, $extra) = $value =~ m|^\$($field[$i])\@(\S+)|; + # print "name: $name\n"; + # print "extra: $extra\n"; + # print "value: $value\n"; + + $Nrow = $count[$i]; + if ($extra) { + $value = "\$row[$Nrow]\@$extra"; + } else { + $value = "\$row[$Nrow]"; + } + # print "outval: $value\n"; + last MATCH_LABEL; + } + } + $outline = "$label=$value"; + } + } + return ($label, $value, $outline); +} + +sub init_key { + my ($key) = $_[0]; + + push @key, $key; + push @value, ""; +} + +sub set_keypair { + my ($i); + my ($key) = $_[0]; + my ($value) = $_[1]; + + for ($i = 0; $i < @key; $i++) { + if ($key eq $key[$i]) { + if ($value[$i] ne "") { die "key is multiply defined\n"; } + $value[$i] = $value; + if ($VERBOSE) { print STDERR "setting $key = $value\n"; } + return; + } + } +} + +sub check_key { + my ($i); + my ($key) = $_[0]; + my ($default) = $_[1]; + + for ($i = 0; $i < @key; $i++) { + if ($key eq $key[$i]) { + if ($VERBOSE) { print "found $key: $key[$i] -- $value[$i] (def: $default)\n"; } + if (($default eq "") && ($value[$i] eq "")) { die "missing value for required key $key[$i]\n"; } + if ($value[$i] eq "") { $value[$i] = $default; } + return; + } + } + die "unknown key $key\n"; +} + +sub check_keypairs { + my ($i); + for ($i = 0; $i < @key; $i++) { + if ($VERBOSE) { print "$key[$i] -- $value[$i]\n"; } + if ($line =~ m|\$$key[$i]|) { + if ($value[$i] eq "") { die "missing value for required key $key[$i]\n"; } + $line =~ s|\$$key[$i]|$value[$i]|g; + } + } +} + +# we need to find the structure size, including padding +# i'm not sure I know the answer to this: it is probably +# the total number of bytes rounded up to the largest +# data item in the structure (ie, 8 for a double, etc) +# if we have the size, then we can double check the structure +# against the expectation at runtime. for the moment, +# calculate by hand and add to def.d file + Index: /branches/rel-1-0/ippMonitor/raw/DetrendSteps.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/DetrendSteps.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/DetrendSteps.php (revision 11645) @@ -0,0 +1,24 @@ +\n + +The pages in this section allow you to examine the master detrend +images and the processing steps involved in their creation. You may +search through the detrend images which have been accepted by +the detrend analysis system by selecting the Master Detrend Images tab. + +"; + +menu_end(); + +?> Index: /branches/rel-1-0/ippMonitor/raw/ImfileTables.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ImfileTables.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ImfileTables.php (revision 11645) @@ -0,0 +1,17 @@ +\n"; + +menu_end(); + +?> Index: /branches/rel-1-0/ippMonitor/raw/LoadSetup.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/LoadSetup.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/LoadSetup.php (revision 11645) @@ -0,0 +1,17 @@ +\n"; + +menu_end(); + +?> Index: /branches/rel-1-0/ippMonitor/raw/Login.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/Login.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/Login.php (revision 11645) @@ -0,0 +1,61 @@ +\n"; + menu_end (); + exit (); +} + +if ($_SERVER[REQUEST_METHOD] == 'GET') { + + $ID = checkID (); + + menu ('ipp.menu.dat', 'Login', 'ipp.css', $ID['link'], $ID['proj']); + + $pass = $ID['pass']; + $name = $ID['name']; + echo "

You are currently logged in as $name.
\n"; + echo "Click here to log out.\n"; + echo "

\n"; + echo "\n"; + echo "\n"; + echo "

\n\n"; + + menu_end (); + exit (); +} + +if ($_SERVER[REQUEST_METHOD] == 'POST') { + if (key_exists (login, $_POST)) { + + $ID = checkLogin (); + + menu ('ipp.menu.dat', 'Login', 'ipp.css', $ID['link'], $ID['proj']); + echo "Login Accepted\n"; + menu_end(); + exit (); + } + + if (key_exists (logout, $_POST)) { + menu ('ipp.menu.dat', 'Login', 'ipp.css', '', ''); + echo "You are now logged out
\n"; + logintext (); + loginform (); + menu_end (); + exit (); + } + + // missing a valid POST state (login or logout) + menu ('ipp.menu.dat', 'Login', 'ipp.css', '', ''); + echo "Invalid Client Post Request
\n"; + foreach ($_POST as $key => $value) { + echo "$key : $value
\n"; + } + menu_end (); + exit (); +} + +?> Index: /branches/rel-1-0/ippMonitor/raw/ScienceSteps.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ScienceSteps.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ScienceSteps.php (revision 11645) @@ -0,0 +1,17 @@ +\n"; + +menu_end(); + +?> Index: /branches/rel-1-0/ippMonitor/raw/SelectProject.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/SelectProject.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/SelectProject.php (revision 11645) @@ -0,0 +1,37 @@ +\n"; + menu_end (); + exit (); +} + +$ID = checkID (); + +if ($_SERVER[REQUEST_METHOD] == 'GET') { + projectform ($ID); +} + +if ($_SERVER[REQUEST_METHOD] == 'POST') { + + if (key_exists (project, $_POST)) { + $myProj = $_POST[proj]; + $ID['proj'] = $myProj; + // validate the existence of the project + + // update ID['link'] + // $link is attached to every nav link + $ID['link'] = "pass=" . $ID['pass']; + if ($ID['proj']) { + $ID['link'] = $ID['link'] . "&proj=" . $ID['proj']; + } + + menu ('ipp.menu.dat', 'Select Project', 'ipp.css', $ID['link'], $ID['proj']); + echo "New project is : $myProj
\n"; + menu_end(); + } +} +?> Index: /branches/rel-1-0/ippMonitor/raw/getimage.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/getimage.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/getimage.php (revision 11645) @@ -0,0 +1,65 @@ +"; + +# $basename may contain filename@filerule +$basename = strtok($_GET[name],"@"); +$filerule = strtok("@"); + +# echo "basename: $basename
"; +# echo "filerule: $filerule
"; + +# need to supply the camera as well... +# exec ("ipp_filerule.pl --site=$SITE $basename", $output, $status); + +if ($filerule) { + if ($filerule == "PPIMAGE.JPEG1") { + $basename = "$basename.b1.jpg"; + } + if ($filerule == "PPIMAGE.JPEG2") { + $basename = "$basename.b2.jpg"; + } +} + +$basename = escapeshellarg($basename); +$basename = str_replace ('..','',$basename); + +exec ("ipp_datapath.pl --site=$SITE $basename", $output, $status); + +$filename = $output[0]; +if ($status) { + $filename = $MISSING; +} + +if (0) { + echo "basename: $basename
"; + echo "result: $result
"; + echo "output: $output[0]
"; + echo "status: $status
"; + echo "filename: $filename
"; + exit (); +} + +$file = fopen ($filename, "r"); +if ($file) { + header ('Content-Type: image/jpg'); + fpassthru ($file); +} +exit (); +?> Index: /branches/rel-1-0/ippMonitor/raw/ipp.copy.dat =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.copy.dat (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.copy.dat (revision 11645) @@ -0,0 +1,35 @@ +# style | select-style | type | menu line | link page + +# we have four valid menu types: picture, piclink, link, plain +# picture | picture | picture | PScolorlogo2.jpg | none +# piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +# menulink | menuselect | link | other page | notest.php +# menutop | menutop | plain | foo bar | none + +piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS public | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS project | http://panstarrs.ifa.hawaii.edu/project +menuext | menuext | link | Pan-STARRS IPP | http://panstarrs.ifa.hawaii.edu/project/IPP + +menutop | menutop | plain |   | +menulink | menuselect | link | Login | Login.php +menulink | menuselect | link | Select Project | SelectProject.php +menulink | menuselect | link | [$PROJECT] | SelectProject.php + +menutop | menutop | plain |   | +menutops | menutops | link | Load and Setup | LoadSetup.php +menutop | menutop | link | Science Steps | ScienceSteps.php +menutop | menutop | link | Detrend Steps | DetrendSteps.php +menutop | menutop | link | Imfile Tables | ImfileTables.php + +menutop | menutop | plain |   | +menulink | menuselect | link | Raw Detrend Exp | rawDetrendExp.php +menulink | menuselect | link | Raw Science Exp | rawScienceExp.php +menulink | menuselect | link | Raw Unknown Exp | rawUnknownExp.php +menutop | menutop | plain |   | +menulink | menuselect | link | Summit Exp | summitExp.php +menulink | menuselect | link | Copy Pending Exp | pzPendingExp.php +menulink | menuselect | link | New Exp | newExp.php +menutop | menutop | plain |   | +menulink | menuselect | link | Failed Raw Imfiles | rawImfile_failed.php +menulink | menuselect | link | Failed Raw Exp | rawExp_failed.php Index: /branches/rel-1-0/ippMonitor/raw/ipp.css =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.css (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.css (revision 11645) @@ -0,0 +1,81 @@ + +/* internal link line: c0d0d0 : d8d0f0 + external link line: 303070 + border: 0080c0 + */ + +body { bgcolor: #a0d0e0; background-color: #a0d0e0; padding: 5px; margin: 5px } + +a:link, a:visited, a:active { text-decoration: underline; font-weight: bold; color: #ff0000 } + +a.button { text-decoration: none; font-weight: bold; color: #000000; background: #0080c0} +a.menutop { text-decoration: none; font-weight: bold; color: #ffffff} +a.menutops { text-decoration: none; font-weight: bold; color: #80f0ff} +a.menuext { text-decoration: none; font-weight: bold; color: #ffffff} +a.menulink { text-decoration: none; font-weight: normal; color: #000000} +a.menuselect { text-decoration: none; font-weight: normal; color: #0000ff} +a.piclink { text-decoration: none; font-weight: normal; color: #0080c0} + +td.menutop { text-align: left; background: #0080c0; font-size: small; color: #ffffff; padding: 2px; } +td.menutops { text-align: left; background: #0080c0; font-size: small; color: #000000; padding: 2px; } +td.menuext { text-align: left; background: #303070; font-size: small; color: #ffffff; padding: 2px; } +td.menulink { text-align: left; background: #d0d0ff; font-size: small; color: #000000; padding: 2px; font-weight: normal; text-indent: 0px; } +td.menuselect { text-align: left; background: #d0d0ff; font-size: small; color: #ff0000; padding: 2px; font-weight: normal; text-indent: 0px; } +td.menuline { text-align: left; background: #d0d0ff; font-size: small; color: #000000; padding: 2px; font-weight: normal; text-indent: 0px; } + +td.picture { text-align: left; background: #00ffff; font-size: small; color: #ff0000; padding: 0px; text-indent: 2px; font-weight: normal; } +td.piclink { text-align: left; background: #0080c0; font-size: small; color: #ff0000; padding: 0px; text-indent: 0px; font-weight: normal; } + +table.page { border: 0px solid #0080c0; background: #a0d0e0; padding: 0px} + +td.title { background-color: #ff0000; padding: 5px; font-size: x-large; text-align: left; vertical-align: top} + +td.body { + text-align: left; + font-size: normal; + font-weight: normal; + vertical-align: top; + background: #d0d0ff; + background-color: #d0d0ff; + border: 3px solid #0080c0; + padding: 5px; +} + +table.menu { text-align: left; + font-size: small; + font-weight: normal; + background: #ff0000; + border: 3px solid #0080c0; + padding: 0px; +} + +table.list { text-align: left; + font-size: small; + font-weight: normal; + background: #d0d0ff; + border: 2px solid #ffffff; + padding: 0px; +} + +td.list { + text-align: left; + font-size: normal; + font-weight: normal; + vertical-align: top; + background: #d0d0ff; + background-color: #d0d0ff; + border: 2px solid #000000; + padding: 2px; +} + +th.list { + text-align: left; + font-size: normal; + font-weight: bold; + vertical-align: top; + background: #d0d0ff; + background-color: #d0d0ff; + border: 2px solid #000000; + padding: 2px; +} + Index: /branches/rel-1-0/ippMonitor/raw/ipp.detrend.dat =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.detrend.dat (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.detrend.dat (revision 11645) @@ -0,0 +1,34 @@ +# style | select-style | type | menu line | link page + +# we have four valid menu types: picture, piclink, link, plain +# picture | picture | picture | PScolorlogo2.jpg | none +# piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +# menulink | menuselect | link | other page | notest.php +# menutop | menutop | plain | foo bar | none + +piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS public | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS project | http://panstarrs.ifa.hawaii.edu/project +menuext | menuext | link | Pan-STARRS IPP | http://panstarrs.ifa.hawaii.edu/project/IPP + +menutop | menutop | plain |   | +menulink | menuselect | link | Login | Login.php +menulink | menuselect | link | Select Project | SelectProject.php +menulink | menuselect | link | [$PROJECT] | SelectProject.php + +menutop | menutop | plain |   | +menutop | menutop | link | Load and Setup | LoadSetup.php +menutop | menutop | link | Science Steps | ScienceSteps.php +menutops | menutops | link | Detrend Steps | DetrendSteps.php +menutop | menutop | link | Imfile Tables | ImfileTables.php + +menutop | menutop | plain |   | +menulink | menuselect | link | Detrend Frames | masterDetrendFrames.php +menulink | menuselect | link | Detrend Runs | detRun.php +menulink | menuselect | link | Detrend Iterations | detRunSummary.php +menulink | menulink | plain |   | +menulink | menuselect | link | Raw Detrend Exp | rawDetrendExp_detrend.php +menulink | menuselect | link | Input Detrend Exp | detInputExp.php +menulink | menuselect | link | Proc Detrend Exp | detProcessedExp.php +menulink | menuselect | link | Norm Detrend Exp | detNormalizedExp.php +menulink | menuselect | link | Resid Detrend Exp | detResidExp.php Index: /branches/rel-1-0/ippMonitor/raw/ipp.imfiles.dat =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.imfiles.dat (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.imfiles.dat (revision 11645) @@ -0,0 +1,40 @@ +# style | select-style | type | menu line | link page + +# we have four valid menu types: picture, piclink, link, plain +# picture | picture | picture | PScolorlogo2.jpg | none +# piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +# menulink | menuselect | link | other page | notest.php +# menutop | menutop | plain | foo bar | none + +piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS public | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS project | http://panstarrs.ifa.hawaii.edu/project +menuext | menuext | link | Pan-STARRS IPP | http://panstarrs.ifa.hawaii.edu/project/IPP + +menutop | menutop | plain |   | +menulink | menuselect | link | Login | Login.php +menulink | menuselect | link | Select Project | SelectProject.php +menulink | menuselect | link | [$PROJECT] | SelectProject.php + +menutop | menutop | plain |   | +menutop | menutop | link | Load and Setup | LoadSetup.php +menutop | menutop | link | Science Steps | ScienceSteps.php +menutop | menutop | link | Detrend Steps | DetrendSteps.php +menutops | menutops | link | Imfile Tables | ImfileTables.php + +menutop | menutop | plain |   | +menulink | menuselect | link | newImfile | newImfile.php +menulink | menuselect | link | rawImfile | rawImfile.php +menulink | menuselect | link | detProcessedImfile | detProcessedImfile.php +menulink | menuselect | link | detStackedImfile | detStackedImfile.php +menulink | menuselect | link | detNormalizedImfile | detNormalizedImfile.php +menulink | menuselect | link | detNormalizedStatImfile | detNormalizedStatImfile.php +menulink | menuselect | link | detResidImfile | detResidImfile.php +menulink | menuselect | link | masterDetrendImfiles | masterDetrendImfiles.php +menulink | menuselect | link | pzPendingImfile | pzPendingImfile.php +menulink | menuselect | link | p2PendingImfile | p2PendingImfile.php +menulink | menuselect | link | p2ProcessedImfile | p2ProcessedImfile.php + +menutop | menutop | plain |   | +menutop | menutop | plain | External Links | none +menutop | menutop | link | test page | phptest.php Index: /branches/rel-1-0/ippMonitor/raw/ipp.menu.dat =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.menu.dat (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.menu.dat (revision 11645) @@ -0,0 +1,23 @@ +# style | select-style | type | menu line | link page + +# we have four valid menu types: picture, piclink, link, plain +# picture | picture | picture | PScolorlogo2.jpg | none +# piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +# menulink | menuselect | link | other page | notest.php +# menutop | menutop | plain | foo bar | none + +piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS public | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS project | http://panstarrs.ifa.hawaii.edu/project +menuext | menuext | link | Pan-STARRS IPP | http://panstarrs.ifa.hawaii.edu/project/IPP + +menutop | menutop | plain |   | +menulink | menuselect | link | Login | Login.php +menulink | menuselect | link | Select Project | SelectProject.php +menulink | menuselect | link | [$PROJECT] | SelectProject.php + +menutop | menutop | plain |   | +menutop | menutop | link | Load and Setup | LoadSetup.php +menutop | menutop | link | Science Steps | ScienceSteps.php +menutop | menutop | link | Detrend Steps | DetrendSteps.php +menutop | menutop | link | Imfile Tables | ImfileTables.php Index: /branches/rel-1-0/ippMonitor/raw/ipp.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.php (revision 11645) @@ -0,0 +1,326 @@ +Welcome to the IPP status pages.

', "\n"; + echo 'From these pages, you may examine summary information about the IPP processing results.
', "\n"; + echo 'In order to continue, it is necessary to login as an authorized IPP user.

', "\n\n"; +} + +function loginform () { + echo '
', "\n"; + echo 'Username:
', "\n"; + echo 'Password:
', "\n"; + echo '', "\n"; + echo '
', "\n\n"; +} + +function projectform ($ID) { + menu ('ipp.menu.dat', 'Select Project', 'ipp.css', $ID['link'], $ID['proj']); + echo "

Please select the project of interest

\n"; + echo "
\n"; + echo "Project:
\n"; + echo "\n"; + + $pass = $ID['pass']; + echo "\n"; + echo "
\n"; + + menu_end (); +} + +function dbconnect ($database) { + + // connect to the database + require_once('DB.php'); + $db = DB::connect("mysql://ipp:ipp@alala/$database"); + if (DB::iserror($db)) { + echo "error accessing database
\n"; + echo "tried mysql://ipp:ipp@localhost/ipp
\n"; + menu_end(); + } + return $db; +} + +// there are three pieces of data that are used to define a user: +// - name (the username: available somewhere: db table?) +// - proj (defines which database is being used) +// - pass (NOT the same as the user password: randomly generated for the session and given a timeout) + +// have this function take the current menu and style? +function checkLogin () { + + $username = $_POST[username]; + $password = $_POST[password]; + + $db = dbconnect('ippadmin'); + + // make this a DB lookup + $sql = "SELECT username FROM users WHERE username = '$username' AND password = '$password'"; + + $qry = $db->query($sql); + if (DB::iserror($db)) { + echo "error reading users table
\n"; + echo "Login Failed, please try again
\n"; + loginform (); + menu_end(); + } + + $success = $qry->fetchInto($row); + if (!$success) { + menu ('ipp.menu.dat', 'Login', 'ipp.css', '', ''); + echo "Login Failed, please try again
\n"; + loginform (); + menu_end (); + exit (); + } + + // generate an random ephemeral ID + $ID['name'] = $username; + $ID['proj'] = ""; + + // generate a cookie for the user + $sql = "SELECT username,cookie FROM cookies WHERE username = '$username'"; + $qry = $db->query($sql); + if (DB::iserror($db)) { + echo "Cookie Failed, please try again
\n"; + menu_end(); + } + + if ($qry->fetchInto($row)) { + $ID['pass'] = $row[1]; + } else { + $cookie = rand(0,100000); + $ID['pass'] = $cookie; + $sql = "INSERT into cookies (username, cookie) values ('$username', '$cookie')"; + $qry = $db->query($sql); + if (DB::iserror($db)) { + echo "Cookie Failed, please try again
\n"; + menu_end(); + } + } + + // the link is attached to every nav link + $ID['link'] = "pass=" . $ID['pass']; + if ($ID['proj']) { + $ID['link'] = $link . "&proj=" . $ID['proj']; + } + + return $ID; +} + +function checkID () { + + $VERBOSE = 0; + + // check for valid server method + if (($_SERVER[REQUEST_METHOD] != 'POST') && ($_SERVER[REQUEST_METHOD] != 'GET')) { + menu ('ipp.menu.dat', 'Login', 'ipp.css', '', ''); + echo "Invalid Client Request
\n"; + menu_end (); + exit (); + } + + // look for pass & proj in appropriate location + if ($_SERVER[REQUEST_METHOD] == 'GET') { + $ID['pass'] = $_GET[pass]; + $ID['proj'] = $_GET[proj]; + $ID['menu'] = $_GET[menu]; + $ID['from'] = $_GET[from]; + + if ($VERBOSE) { + echo "using GET: ", $ID['pass'], $ID['proj'], "
\n"; + echo "GET list
\n"; + foreach ($_GET as $key => $value) { + echo "$key : $value
\n"; + } + } + } + if ($_SERVER[REQUEST_METHOD] == 'POST') { + $ID['pass'] = $_POST[pass]; + $ID['proj'] = $_POST[proj]; + $ID['menu'] = $_POST[menu]; + $ID['from'] = $_POST[from]; + + if ($VERBOSE) { + echo "using POST: ", $ID['pass'], $ID['proj'], "
\n"; + echo "POST list
\n"; + foreach ($_POST as $key => $value) { + echo "$key : $value
\n"; + } + } + } + + // user is not logged in at all + if ($ID['pass'] == "") { + menu('ipp.menu.dat', 'Login', 'ipp.css', '', ''); + logintext (); + loginform (); + menu_end (); + } + + $db = dbconnect ('ippadmin'); + + // make this a DB lookup + $pass = $ID['pass']; + $sql = "SELECT username FROM cookies WHERE cookie = '$pass'"; + $qry = $db->query($sql); + if (DB::iserror($db)) { + echo "error reading users table
\n"; + echo "Login Failed, please try again
\n"; + loginform (); + menu_end(); + } + + # $success = (($username == "eugene") && ($password == "test")); + $success = $qry->fetchInto($row); + if (! $success) { + menu('ipp.menu.dat', 'Login', 'ipp.css', '', ''); + echo "unknown user, please login again
\n"; + loginform (); + menu_end (); + } + // use the pass value to find the corresponding user + $ID['name'] = $row[0]; + + // $link is attached to every nav link + $ID['link'] = "pass=" . $ID['pass']; + if ($ID['proj']) { + $ID['link'] = $ID['link'] . "&proj=" . $ID['proj']; + } + + return $ID; +} + +function menu ($source, $title, $sheet, $append, $project) { + + echo " $title \n\n"; + echo "\n"; + echo "\n"; + + $root = "/ippMonitor"; + + if (! $project) { $project = "none"; } + + $file = fopen ($source, "r"); + + echo "
\n"; + echo "\n"; + + while ($line = fgetcsv ($file, 1024, "|")) { + if (count($line) != 5) continue; + if (ereg ('^[:blank:]*#', $line[0])) continue; + + $type = trim($line[2]); + $name = trim($line[3]); + $base = trim($line[4]); + + if (preg_match ('|\$PROJECT|', $name)) { + $name = preg_replace ('|\$PROJECT|', $project, $name); + } + + if ($append) { + $link = $base . "?" . $append; + } else { + $link = $base; + } + + $this = $_SERVER[SCRIPT_NAME]; + $page = "$root/" . $base; + if ($page == $this) { + $style = trim($line[1]); + } else { + $style = trim($line[0]); + } + + switch ($type) { + case 'plain': + echo "\n"; + break; + case 'picture': + echo "\n"; + break; + case 'piclink': + echo "\n"; + break; + default: + echo "\n"; + break; + } + } + fclose ($file); + echo '
$name
$name
', "\n\n"; +} + +function menu_end () { + echo "
\n"; + echo "\n\n"; + exit (); +} + +function check_restrict ($key, $where) { + $htmlkey = preg_replace ('|\.|', '_', $key); + if ($_SERVER[REQUEST_METHOD] == 'GET') { + $value = $_GET[$htmlkey]; + // echo "using GET: $value
\n"; + } else { + $value = $_POST[$htmlkey]; + // echo "using POST: $value
\n"; + } + // echo "value: $value
\n"; + if ($value != "") { + if ($where) { + $where = $where . " AND $key = '$value'"; + // echo "where 1: $where
\n"; + } else { + $where = "$key = '$value'"; + // echo "where 2: $where
\n"; + } + } + return $where; +} + +function write_query_row ($key, $width) { + $htmlkey = preg_replace ('|\.|', '_', $key); + $value = $_POST[$htmlkey]; + if (($_SERVER[REQUEST_METHOD] == 'POST') && $value) { + echo " \n"; + } else { + echo " \n"; + } +} + +// expect image supplied with key=name +// this function should define matching hidden inputs in the form +function insert_image ($key) { + + if ($_SERVER[REQUEST_METHOD] == 'GET') { + $name = $_GET[$key]; + } else { + $name = $_POST[$key]; + } + + if ($name) { + echo "\n"; + } +} + +// expect image supplied with key=name +// this function should define matching hidden inputs in the form +function insert_backref ($page, $key, $link) { + + if ($_SERVER[REQUEST_METHOD] == 'GET') { + $name = $_GET[$key]; + } else { + $name = $_POST[$key]; + } + + if ($name) { + echo "back to $name
\n"; + } +} + +// $myPage = $_SERVER[SCRIPT_NAME] . "?pass=$pass"; + +?> Index: /branches/rel-1-0/ippMonitor/raw/ipp.science.dat =================================================================== --- /branches/rel-1-0/ippMonitor/raw/ipp.science.dat (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/ipp.science.dat (revision 11645) @@ -0,0 +1,32 @@ +# style | select-style | type | menu line | link page + +# we have four valid menu types: picture, piclink, link, plain +# picture | picture | picture | PScolorlogo2.jpg | none +# piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +# menulink | menuselect | link | other page | notest.php +# menutop | menutop | plain | foo bar | none + +piclink | piclink | piclink | PScolorlogo2.jpg | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS public | http://panstarrs.ifa.hawaii.edu +menuext | menuext | link | Pan-STARRS project | http://panstarrs.ifa.hawaii.edu/project +menuext | menuext | link | Pan-STARRS IPP | http://panstarrs.ifa.hawaii.edu/project/IPP + +menutop | menutop | plain |   | +menulink | menuselect | link | Login | Login.php +menulink | menuselect | link | Select Project | SelectProject.php +menulink | menuselect | link | [$PROJECT] | SelectProject.php + +menutop | menutop | plain |   | +menutop | menutop | link | Load and Setup | LoadSetup.php +menutops | menutops | link | Science Steps | ScienceSteps.php +menutop | menutop | link | Detrend Steps | DetrendSteps.php +menutop | menutop | link | Imfile Tables | ImfileTables.php + +menutop | menutop | plain |   | +menulink | menuselect | plain | P1 Processed Exp | +menulink | menuselect | link | P2 Processed Exp | p2ProcessedExp.php +menulink | menuselect | link | P3 Processed Exp | p3ProcessedExp.php +menutop | menutop | plain |   | +menulink | menuselect | link | P1 Pending Exp | p1PendingExp.php +menulink | menuselect | link | P2 Pending Exp | p2PendingExp.php +menulink | menuselect | link | P3 Pending Exp | p3PendingExp.php Index: /branches/rel-1-0/ippMonitor/raw/masterDetrendImfiles.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/masterDetrendImfiles.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/masterDetrendImfiles.php (revision 11645) @@ -0,0 +1,95 @@ + Master Detrend Imfiles

"; + +// define restrictiosn to the queries +// ** TABLE RESTRICTIONS ** +$WHERE = check_restrict ('det_id', $WHERE); +$WHERE = check_restrict ('iteration', $WHERE); +$WHERE = check_restrict ('uri', $WHERE); + +// query the database +if ($WHERE) { + $sql = "SELECT det_id,iteration,uri FROM detNormalizedImfile WHERE $WHERE UNION SELECT det_id,iteration,uri FROM detStackedImfile WHERE normalize = 0 AND $WHERE LIMIT 20"; +} else { + $sql = "SELECT det_id,iteration,uri FROM detNormalizedImfile UNION SELECT det_id,iteration,uri FROM detStackedImfile WHERE normalize = 0 LIMIT 20"; +} + +$qry = $db->query($sql); +if (DB::iserror($db)) { + echo "error reading detStackedImfile table
\n"; + menu_end(); +} + +// ** HEAD CODE ** + +// set up the table and form +echo "
\n"; + +echo "\n"; +echo "\n"; +// ** TABLE HEADER ** +echo "\n"; +echo "\n"; +echo "\n"; +// echo "\n"; +echo "\n"; + +// list the results +while ($qry->fetchInto($row)) { + // $link = "masterDetrendImfiles.php" . "?expID=" . $row[0] . "&" . $ID['link']; + + echo "\n"; + // ** TABLE DATA ** + echo "\n"; + echo "\n"; + echo "\n"; + // echo "\n"; + // echo "\n"; + echo "\n"; +} + +// query restriction form +echo "\n"; +// echo "\n"; +// ** TABLE QUERY ** +write_query_row ('det_id', 7); +write_query_row ('iteration', 5); +write_query_row ('uri', 5); +echo "\n"; + +// close the table and form +echo "
det_id iter uri FIELD
$row[0] $row[1] $row[2] $row[0] $row[1]
\n"; +echo "\n"; +$pass = $ID['pass']; +$proj = $ID['proj']; +$menu = $ID['menu']; +echo "\n"; +echo "\n"; +echo "\n"; +echo "
\n"; + +// ** TAIL CODE ** + +echo " WHERE: $WHERE

\n"; +echo " SQL: $sql

\n"; + +menu_end(); + +?> Index: /branches/rel-1-0/ippMonitor/raw/phptest.php =================================================================== --- /branches/rel-1-0/ippMonitor/raw/phptest.php (revision 11645) +++ /branches/rel-1-0/ippMonitor/raw/phptest.php (revision 11645) @@ -0,0 +1,25 @@ +\n"; +} + +echo "get list
\n"; +foreach ($_GET as $key => $value) { + echo "$key : $value
\n"; +} + +$value = system ("ls /etc", $status); +echo "status: $status\n"; + +menu_end(); +?> Index: /branches/rel-1-0/ippScripts/.cvsignore =================================================================== --- /branches/rel-1-0/ippScripts/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippScripts/.cvsignore (revision 11645) @@ -0,0 +1,6 @@ +Build +META.yml +Makefile +Makefile.PL +_build +blib Index: /branches/rel-1-0/ippScripts/Build.PL =================================================================== --- /branches/rel-1-0/ippScripts/Build.PL (revision 11645) +++ /branches/rel-1-0/ippScripts/Build.PL (revision 11645) @@ -0,0 +1,44 @@ +use Module::Build; +# See perldoc Module::Build for details of how this works + +Module::Build->new( + module_name => 'ippscripts', + dist_version => '0.01', + author => 'Joshua Hoblitt ', + license => 'gpl', + create_makefile_pl => 'passthrough', + requires => { + 'IPC::Cmd' => 0, + 'IPC::Run' => 0, + 'PS::IPP::Config' => 0, + 'PS::IPP::Metadata::Config' => 0, + 'Statistics::Descriptive' => 0, + 'Cache::File' => 0, + 'Storable' => 0, + }, + recommends => { + 'Test::Distribution' => '1.22', + }, + script_files => [qw( + scripts/ipp_serial_inject.pl + scripts/detrend_norm_apply.pl + scripts/detrend_norm_calc.pl + scripts/detrend_norm_exp.pl + scripts/detrend_process_exp.pl + scripts/detrend_process_imfile.pl + scripts/detrend_reject_exp.pl + scripts/detrend_reject_imfile.pl + scripts/detrend_resid.pl + scripts/detrend_stack.pl + scripts/ipp_datapath.pl + scripts/ipp_filename.pl + scripts/mdc2list.pl + scripts/phase0_exp.pl + scripts/phase0_imfile.pl + scripts/phase2.pl + scripts/phase3.pl + scripts/ipp_serial_detrend.pl + scripts/ipp_serial_phase0.pl + scripts/ipp_serial_inject_mosaic.pl + )], + )->create_build_script; Index: /branches/rel-1-0/ippScripts/Changes =================================================================== --- /branches/rel-1-0/ippScripts/Changes (revision 11645) +++ /branches/rel-1-0/ippScripts/Changes (revision 11645) @@ -0,0 +1,6 @@ +Revision history for Perl module ippScripts + +0.01 Thu Feb 23 13:55:57 2006 + - help scripts split out of the ippTools package + + Index: /branches/rel-1-0/ippScripts/LICENSE =================================================================== --- /branches/rel-1-0/ippScripts/LICENSE (revision 11645) +++ /branches/rel-1-0/ippScripts/LICENSE (revision 11645) @@ -0,0 +1,260 @@ +The General Public License (GPL) +Version 2, June 1991 + +Copyright (C) 1989, 1991 Free Software Foundation, Inc. 675 Mass Ave, +Cambridge, MA 02139, USA. Everyone is permitted to copy and distribute +verbatim copies of this license document, but changing it is not allowed. + +Preamble + +The licenses for most software are designed to take away your freedom to share +and change it. By contrast, the GNU General Public License is intended to +guarantee your freedom to share and change free software--to make sure the +software is free for all its users. This General Public License applies to most of +the Free Software Foundation's software and to any other program whose +authors commit to using it. (Some other Free Software Foundation software is +covered by the GNU Library General Public License instead.) You can apply it to +your programs, too. + +When we speak of free software, we are referring to freedom, not price. Our +General Public Licenses are designed to make sure that you have the freedom +to distribute copies of free software (and charge for this service if you wish), that +you receive source code or can get it if you want it, that you can change the +software or use pieces of it in new free programs; and that you know you can do +these things. + +To protect your rights, we need to make restrictions that forbid anyone to deny +you these rights or to ask you to surrender the rights. These restrictions +translate to certain responsibilities for you if you distribute copies of the +software, or if you modify it. + +For example, if you distribute copies of such a program, whether gratis or for a +fee, you must give the recipients all the rights that you have. You must make +sure that they, too, receive or can get the source code. And you must show +them these terms so they know their rights. + +We protect your rights with two steps: (1) copyright the software, and (2) offer +you this license which gives you legal permission to copy, distribute and/or +modify the software. + +Also, for each author's protection and ours, we want to make certain that +everyone understands that there is no warranty for this free software. If the +software is modified by someone else and passed on, we want its recipients to +know that what they have is not the original, so that any problems introduced by +others will not reflect on the original authors' reputations. + +Finally, any free program is threatened constantly by software patents. We wish +to avoid the danger that redistributors of a free program will individually obtain +patent licenses, in effect making the program proprietary. To prevent this, we +have made it clear that any patent must be licensed for everyone's free use or +not licensed at all. + +The precise terms and conditions for copying, distribution and modification +follow. + +GNU GENERAL PUBLIC LICENSE +TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND +MODIFICATION + +0. This License applies to any program or other work which contains a notice +placed by the copyright holder saying it may be distributed under the terms of +this General Public License. The "Program", below, refers to any such program +or work, and a "work based on the Program" means either the Program or any +derivative work under copyright law: that is to say, a work containing the +Program or a portion of it, either verbatim or with modifications and/or translated +into another language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not covered by +this License; they are outside its scope. The act of running the Program is not +restricted, and the output from the Program is covered only if its contents +constitute a work based on the Program (independent of having been made by +running the Program). Whether that is true depends on what the Program does. + +1. You may copy and distribute verbatim copies of the Program's source code as +you receive it, in any medium, provided that you conspicuously and appropriately +publish on each copy an appropriate copyright notice and disclaimer of warranty; +keep intact all the notices that refer to this License and to the absence of any +warranty; and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and you may at +your option offer warranty protection in exchange for a fee. + +2. You may modify your copy or copies of the Program or any portion of it, thus +forming a work based on the Program, and copy and distribute such +modifications or work under the terms of Section 1 above, provided that you also +meet all of these conditions: + +a) You must cause the modified files to carry prominent notices stating that you +changed the files and the date of any change. + +b) You must cause any work that you distribute or publish, that in whole or in +part contains or is derived from the Program or any part thereof, to be licensed +as a whole at no charge to all third parties under the terms of this License. + +c) If the modified program normally reads commands interactively when run, you +must cause it, when started running for such interactive use in the most ordinary +way, to print or display an announcement including an appropriate copyright +notice and a notice that there is no warranty (or else, saying that you provide a +warranty) and that users may redistribute the program under these conditions, +and telling the user how to view a copy of this License. (Exception: if the +Program itself is interactive but does not normally print such an announcement, +your work based on the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If identifiable +sections of that work are not derived from the Program, and can be reasonably +considered independent and separate works in themselves, then this License, +and its terms, do not apply to those sections when you distribute them as +separate works. But when you distribute the same sections as part of a whole +which is a work based on the Program, the distribution of the whole must be on +the terms of this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest your rights to +work written entirely by you; rather, the intent is to exercise the right to control +the distribution of derivative or collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program with the +Program (or with a work based on the Program) on a volume of a storage or +distribution medium does not bring the other work under the scope of this +License. + +3. You may copy and distribute the Program (or a work based on it, under +Section 2) in object code or executable form under the terms of Sections 1 and 2 +above provided that you also do one of the following: + +a) Accompany it with the complete corresponding machine-readable source +code, which must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange; or, + +b) Accompany it with a written offer, valid for at least three years, to give any +third party, for a charge no more than your cost of physically performing source +distribution, a complete machine-readable copy of the corresponding source +code, to be distributed under the terms of Sections 1 and 2 above on a medium +customarily used for software interchange; or, + +c) Accompany it with the information you received as to the offer to distribute +corresponding source code. (This alternative is allowed only for noncommercial +distribution and only if you received the program in object code or executable +form with such an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for making +modifications to it. For an executable work, complete source code means all the +source code for all modules it contains, plus any associated interface definition +files, plus the scripts used to control compilation and installation of the +executable. However, as a special exception, the source code distributed need +not include anything that is normally distributed (in either source or binary form) +with the major components (compiler, kernel, and so on) of the operating system +on which the executable runs, unless that component itself accompanies the +executable. + +If distribution of executable or object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the source +code from the same place counts as distribution of the source code, even though +third parties are not compelled to copy the source along with the object code. + +4. You may not copy, modify, sublicense, or distribute the Program except as +expressly provided under this License. Any attempt otherwise to copy, modify, +sublicense or distribute the Program is void, and will automatically terminate +your rights under this License. However, parties who have received copies, or +rights, from you under this License will not have their licenses terminated so long +as such parties remain in full compliance. + +5. You are not required to accept this License, since you have not signed it. +However, nothing else grants you permission to modify or distribute the Program +or its derivative works. These actions are prohibited by law if you do not accept +this License. Therefore, by modifying or distributing the Program (or any work +based on the Program), you indicate your acceptance of this License to do so, +and all its terms and conditions for copying, distributing or modifying the +Program or works based on it. + +6. Each time you redistribute the Program (or any work based on the Program), +the recipient automatically receives a license from the original licensor to copy, +distribute or modify the Program subject to these terms and conditions. You +may not impose any further restrictions on the recipients' exercise of the rights +granted herein. You are not responsible for enforcing compliance by third parties +to this License. + +7. If, as a consequence of a court judgment or allegation of patent infringement +or for any other reason (not limited to patent issues), conditions are imposed on +you (whether by court order, agreement or otherwise) that contradict the +conditions of this License, they do not excuse you from the conditions of this +License. If you cannot distribute so as to satisfy simultaneously your obligations +under this License and any other pertinent obligations, then as a consequence +you may not distribute the Program at all. For example, if a patent license would +not permit royalty-free redistribution of the Program by all those who receive +copies directly or indirectly through you, then the only way you could satisfy +both it and this License would be to refrain entirely from distribution of the +Program. + +If any portion of this section is held invalid or unenforceable under any particular +circumstance, the balance of the section is intended to apply and the section as +a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any patents or other +property right claims or to contest validity of any such claims; this section has +the sole purpose of protecting the integrity of the free software distribution +system, which is implemented by public license practices. Many people have +made generous contributions to the wide range of software distributed through +that system in reliance on consistent application of that system; it is up to the +author/donor to decide if he or she is willing to distribute software through any +other system and a licensee cannot impose that choice. + +This section is intended to make thoroughly clear what is believed to be a +consequence of the rest of this License. + +8. If the distribution and/or use of the Program is restricted in certain countries +either by patents or by copyrighted interfaces, the original copyright holder who +places the Program under this License may add an explicit geographical +distribution limitation excluding those countries, so that distribution is permitted +only in or among countries not thus excluded. In such case, this License +incorporates the limitation as if written in the body of this License. + +9. The Free Software Foundation may publish revised and/or new versions of the +General Public License from time to time. Such new versions will be similar in +spirit to the present version, but may differ in detail to address new problems or +concerns. + +Each version is given a distinguishing version number. If the Program specifies a +version number of this License which applies to it and "any later version", you +have the option of following the terms and conditions either of that version or of +any later version published by the Free Software Foundation. If the Program does +not specify a version number of this License, you may choose any version ever +published by the Free Software Foundation. + +10. If you wish to incorporate parts of the Program into other free programs +whose distribution conditions are different, write to the author to ask for +permission. For software which is copyrighted by the Free Software Foundation, +write to the Free Software Foundation; we sometimes make exceptions for this. +Our decision will be guided by the two goals of preserving the free status of all +derivatives of our free software and of promoting the sharing and reuse of +software generally. + +NO WARRANTY + +11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS +NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE +COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM +"AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR +IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE +ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, +YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED +TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY +WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS +PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM +(INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY +OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS +BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +END OF TERMS AND CONDITIONS Index: /branches/rel-1-0/ippScripts/MANIFEST =================================================================== --- /branches/rel-1-0/ippScripts/MANIFEST (revision 11645) +++ /branches/rel-1-0/ippScripts/MANIFEST (revision 11645) @@ -0,0 +1,26 @@ +Build.PL +Changes +LICENSE +MANIFEST +README +scripts/ipp_serial_inject.pl +scripts/detrend_norm_apply.pl +scripts/detrend_norm_calc.pl +scripts/detrend_norm_exp.pl +scripts/detrend_process_exp.pl +scripts/detrend_process_imfile.pl +scripts/detrend_reject_exp.pl +scripts/detrend_reject_imfile.pl +scripts/detrend_resid.pl +scripts/detrend_stack.pl +scripts/ipp_datapath.pl +scripts/ipp_filename.pl +scripts/mdc2list.pl +scripts/phase0_exp.pl +scripts/phase0_imfile.pl +scripts/phase2.pl +scripts/phase3.pl +scripts/ipp_serial_detrend.pl +scripts/ipp_serial_phase0.pl +scripts/ipp_serial_inject_mosaic.pl +t/00_distribution.t Index: /branches/rel-1-0/ippScripts/README =================================================================== --- /branches/rel-1-0/ippScripts/README (revision 11645) +++ /branches/rel-1-0/ippScripts/README (revision 11645) @@ -0,0 +1,15 @@ +pod2text foo.pm > README + +If this is still here it means the programmer was too lazy to create the readme file. + +You can create it now by using the command shown above from this directory. + +At the very least you should be able to use this set of instructions +to install the module... + +perl Build.PL +./Build +./Build test +./Build install + +If you are on a windows box you should use 'nmake' rather than 'make'. Index: /branches/rel-1-0/ippScripts/scripts/detrend_norm_apply.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_norm_apply.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_norm_apply.pl (revision 11645) @@ -0,0 +1,186 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use PS::IPP::Metadata::Stats; +use IPC::Cmd qw( can_run run ); +use Data::Dumper; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +# Parse the command-line +my ($det_id, # Detrend ID + $iter, # Iteration + $class_id, # Class ID + $value, # Value to multiple (for normalisation) + $input, # Input file + $camera, # Camera + $det_type, # Detrend type + $dbname, # Database name + $workdir, # Working directory, for output files + $no_update # Don't update the database + ); +GetOptions( + 'det_id|d=s' => \$det_id, + 'iteration|n=s' => \$iter, + 'class_id|i=s' => \$class_id, + 'value|v=s' => \$value, + 'input_uri|u=s' => \$input, + 'camera|c=s' => \$camera, + 'det_type|t=s' => \$det_type, + 'dbname|d=s' => \$dbname, + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update + ) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( -msg => "Required options: --det_id --iteration --class_id --value --input_uri --camera --det_type", + -exitval => 3, + ) unless defined $det_id + and defined $iter + and defined $class_id + and defined $value + and defined $input + and defined $camera + and defined $det_type; + +$ipprc->define_camera($camera); + + +use constant RECIPE => 'PPIMAGE_N'; # Recipe to use with ppImage + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); + +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my ($vol, $dir, $file) = File::Spec->splitpath( $input ); + $workdir = $dir; +} +$input = $ipprc->convert_filename_absolute( $input ); + +# Output name +my $outputFile = "$camera.$det_type.norm.$det_id.$iter"; # Root name +my $outputSubD = "$camera.$det_type.$det_id"; +my $outputRoot = File::Spec->catfile( $workdir, $outputSubD, $outputFile ); + +my $output = $ipprc->filename("PPIMAGE.OUTPUT", $outputRoot, $class_id); +my $b1name = $ipprc->filename("PPIMAGE.BIN1", $outputRoot, $class_id); +my $b2name = $ipprc->filename("PPIMAGE.BIN2", $outputRoot, $class_id); + +my $statsName = $outputRoot . '.' . $class_id . '.stats'; # Statistics file + +# Run normalisation +{ + my $command = "$ppImage -file $input $outputRoot -norm $value -stat $statsName -recipe PPIMAGE " . RECIPE(); # Command to run + $command .= ' -isfringe' if lc($det_type) eq 'fringe'; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $det_id, $iter, $class_id, $error_code); + } + &my_die("Can't find expected output file: $output", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR) unless -e $output; + &my_die("Can't find expected output file: $b1name", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR) unless -e $b1name; + &my_die("Can't find expected output file: $b2name", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR) unless -e $b2name; + &my_die("Can't find expected output file: $statsName", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR) unless -e $statsName; +} + +# Get the statistics on the normalised image +my $stats; # Statistics from ppImage +{ + my $statsFile; # File handle + open $statsFile, $statsName or &my_die("Can't open statistics file $statsName: $!\n", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR); + my @contents = <$statsFile>; # Contents of file + close $statsFile; + my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + my $metadata = $mdcParser->parse(join "", @contents) + or &my_die("Unable to parse metadata config", $det_id, $iter, $class_id, $PS_EXIT_PROG_ERROR); + $stats = PS::IPP::Metadata::Stats->new(); # Stats parser + $stats->parse($metadata) or &my_die("Unable to find all values in statistics output.", $det_id, $iter, $class_id, $PS_EXIT_PROG_ERROR); +} + +# Update the database +$output = $ipprc->convert_filename_relative( $output ); +$b1name = $ipprc->convert_filename_relative( $b1name ); +$b2name = $ipprc->convert_filename_relative( $b2name ); +unless ($no_update) { + my $command = "$dettool -addnormalizedimfile -det_id $det_id -iteration $iter -class_id $class_id ". + "-uri $output -b1_uri $outputRoot"; # Command to run + # Add the statistics triplet + $command .= " -bg " . $stats->bg_mean(); + if (defined($stats->bg_stdev())) { + $command .= " -bg_stdev " . $stats->bg_stdev(); + } else { + # May be undefined if there's only a single imfile + $command .= " -bg_stdev 0"; + } + $command .= " -bg_mean_stdev " . $stats->bg_mean_stdev(); + + $command .= " -dbname $dbname" if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addnormalizedimfile: $error_code\n"); + exit($error_code); + } + + unlink $statsName; +} + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $iter = shift; # Iteration + my $class_id = shift; # Class identifier + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $iter and $class_id) { + my $command = "$dettool -addnormalizedimfile -det_id $det_id -iteration $iter -class_id $class_id -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_norm_calc.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_norm_calc.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_norm_calc.pl (revision 11645) @@ -0,0 +1,208 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use IPC::Cmd qw( can_run ); +use IPC::Run qw ( run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Data::Dumper; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + + +# Parse command-line arguments +my ($detId, # Detrend id + $iter, # Iteration + $detType, # Detrend type + $workdir, # Working directory for output files + $dbname, # Database name + $no_update # Don't update the database? + ); +GetOptions( + 'det_id|d=s' => \$detId, + 'iteration|i=s' => \$iter, + 'det_type|t=s' => \$detType, + 'workdir|w=s' => \$workdir, + 'dbname|d=s' => \$dbname,# Database name + 'no-update' => \$no_update + ) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( -msg => "Required options --det_id --iteration --det_type", + -exitval => 3, + ) unless defined $detId + and defined $iter + and defined $detType; + + +use constant STATISTIC => 'bg'; # Background statistic to use from the database +# Define which detrend types we normalise +use constant NORMALIZE => { + 'bias' => 0, + 'dark' => 0, + 'shutter' => 0, + 'flat' => 1, + 'domeflat' => 1, + 'skyflat' => 1, + 'fringe' => 0, + }; + + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppNormCalc = can_run('ppNormCalc') or (warn "Can't find ppNormCalc" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +unless (exists NORMALIZE()->{lc($detType)}) { + warn("Unrecognised detrend type: $detType"); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get the list of inputs +my @files; # The input files +{ + my $command = "$dettool -processedimfile -det_id $detId"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my @command = split /\s+/, $command; + my ( $stdin, $stdout, $stderr ); # Buffers for running program + print "Running [$command]...\n"; + if (not run(\@command, \$stdin, \$stdout, \$stderr)) { + warn("Unable to perform dettool -processedimfile on detrend $detId/$iter: $?"); + exit($PS_EXIT_SYS_ERROR); + } +# print $stdout . "\n"; + + # Because of the length, need to split into individual metadatas --- it parses SO much quicker! + my @whole = split /\n/, $stdout; + my @single = (); + while ( scalar @whole > 0 ) { + my $value = shift @whole; + push @single, $value; + if ($value =~ /^\s*END\s*$/) { + push @single, "\n"; + + my $list = parse_md_list( $mdcParser->parse( join( "\n", @single ) ) ); + unless (defined $list) { + warn("Unable to parse output from dettool."); + exit($PS_EXIT_PROG_ERROR); + } + push @files, @$list; + @single = (); + } + } +} + +my $norms; # MDC with normalisations +if (NORMALIZE()->{lc($detType)}) { + + my %matrix; # Matrix of statistics as a function of exposures and classes + foreach my $file (@files) { + my $expTag = $file->{'exp_tag'}; # Exposure ID + my $classId = $file->{'class_id'}; # Class ID + my $stat = $file->{STATISTIC()}; # Statistic of interest + + # Create matrix elements + $matrix{$expTag} = {} if not defined $matrix{$expTag}; + $matrix{$expTag}->{$classId} = $stat; + } + + # Generate the input for ppNormCalc + my $normData; # Normalisation data + foreach my $expTag (keys %matrix) { + $normData .= "$expTag\tMETADATA\n"; + foreach my $classId (keys %{$matrix{$expTag}}) { + $normData .= "\t" . $classId . "\tF32\t" . $matrix{$expTag}->{$classId} . "\n"; + } + $normData .= "END\n\n"; + } + + # Run ppNormCalc + { + my ( $stdout, $stderr ); # Buffers for running program + my @command = split /\s+/, $ppNormCalc; + print "Running [$ppNormCalc]...\n"; + if (not run(\@command, \$normData, \$stdout, \$stderr)) { + warn("Unable to perform ppNormCalc: $?"); + exit($PS_EXIT_SYS_ERROR); + } + print $stdout . "\n"; + + # Parse the output + $norms = $mdcParser->parse($stdout); + unless ($norms) { + warn("Unable to parse metadata config doc"); + exit($PS_EXIT_PROG_ERROR); + } + } + +} else { + # It's something that doesn't need normalisation --- just push in a normalisation of 1 + my %classes; # List of unique classes + foreach my $file (@files) { + my $classId = $file->{'class_id'}; # Class Id + $classes{$classId} = 1; + } + + foreach my $classId (keys %classes) { + my %mdValue; # Metadata value for this class id + $mdValue{name} = $classId; + $mdValue{value} = 1.0; + push @$norms, \%mdValue; + } +} + + +# Process output normalisations +unless ($no_update) { + foreach my $normItem (@$norms) { + my $className = $normItem->{name}; # Name of component + my $normalisation = $normItem->{value}; # Normalisation for component + + my $command = "$dettool -addnormalizedstat -det_id $detId -iteration $iter -class_id $className ". + "-norm $normalisation"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my @command = split /\s+/, $command; + + my ( $stdin, $stdout, $stderr ); # Buffers for running program + print "Running [$command]...\n"; + if (not run \@command, \$stdin, \$stdout, \$stderr) { + warn("Unable to perform dettool -addnormstat for $className: $?"); + exit($PS_EXIT_SYS_ERROR); + } + print $stdout . "\n"; + } +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_norm_exp.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_norm_exp.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_norm_exp.pl (revision 11645) @@ -0,0 +1,208 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Statistics::Descriptive; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $iter, $det_type, $camera, $dbname, $workdir, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'iteration|i=s' => \$iter, + 'camera|c=s' => \$camera, + 'det_type|t=s' => \$det_type, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --iteration --camera --det_type", + -exitval => 3, +) unless defined $det_id + and defined $iter + and defined $camera + and defined $det_type; + +$ipprc->define_camera($camera); + +use constant RECIPE1 => 'PPIMAGE_J1'; # Recipe to use for ppImage to make JPEGs +use constant RECIPE2 => 'PPIMAGE_J2'; # Recipe to use for ppImage to make JPEGs + + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get list of component files +my $files; # Array of component files +{ + my $command = "$dettool -normalizedimfile -det_id $det_id -iteration $iter"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $det_id, $iter, $error_code); + } + my $metadata = $mdcParser->parse(join "", @$stdout_buf) or + &my_die("Unable to parse metadata config doc", $det_id, $iter, $PS_EXIT_PROG_ERROR); + $files = parse_md_list($metadata) or + &my_die("Unable to parse metadata list", $det_id, $iter, $PS_EXIT_PROG_ERROR); +} + +# Gather the statistics +my ($bg, $bg_stdev, $bg_mean_stdev); # The statistics triplet +{ + my @backgrounds; # Array of backgrounds in each component + my @variances; # Array of variances for each component + foreach my $file (@$files) { + &my_die("Unable to find class id", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $file->{class_id}; + my $class_id = $file->{class_id}; + &my_die("Unable to find bg for class_id=$class_id", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $file->{bg}; + &my_die("Unable to find bg_mean_stdev for class_id=$class_id", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $file->{bg_mean_stdev}; + push @backgrounds, $file->{bg}; + push @variances, $file->{bg_stdev}**2; + } + + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@backgrounds); + $bg = $stats->mean(); + $bg_mean_stdev = $stats->standard_deviation() || 0.0; + } + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@variances); + $bg_stdev = sqrt( $stats->mean() ); + } +} + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my $example = ${$files}[0]->{b1_uri}; # Example file, for path + my ($vol, $dir, $file) = File::Spec->splitpath( $example ); + $workdir = $dir; +} + +# Generate the file list, and get the statistics +my $outputFile = "$camera.$det_type.norm.$det_id.$iter"; # Root output name +my $outputSubD = "$camera.$det_type.$det_id"; +my $outputRoot = File::Spec->catfile( $workdir, $outputSubD, $outputFile ); + +my $list1Name = $outputRoot . '.b1.list'; # Name for the input file list for binning 1 +my $list2Name = $outputRoot . '.b2.list'; # Name for the input file list for binning 2 +my @means; # Array of means +my @stdevs; # Array of stdevs +open my $list1File, '>' . $list1Name; +open my $list2File, '>' . $list2Name; +foreach my $file (@$files) { + my $origRoot = $ipprc->convert_filename_absolute( $file->{b1_uri} ); # Original root name + print $list1File ( $ipprc->filename( "PPIMAGE.BIN1", $origRoot, $file->{class_id} ) . "\n"); + print $list2File ( $ipprc->filename( "PPIMAGE.BIN2", $origRoot, $file->{class_id} ) . "\n"); + push @means, $file->{bg}; + push @stdevs, $file->{bg_stdev}; +} +close $list1File; +close $list2File; + +# Output products +my $jpeg1Name = $ipprc->filename("PPIMAGE.JPEG1", $outputRoot); # Binned JPEG #1 +my $jpeg2Name = $ipprc->filename("PPIMAGE.JPEG2", $outputRoot); # Binned JPEG #2 + +# Make the jpeg for binning 1 +{ + my $command = "$ppImage -list $list1Name $outputRoot -recipe PPIMAGE " . RECIPE1; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + &my_die("Unable to find expected output file: $jpeg1Name", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless -f $jpeg1Name; +} + +# Make the jpeg for binning 2 +{ + my $command = "$ppImage -list $list2Name $outputRoot -recipe PPIMAGE " . RECIPE2; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + &my_die("Unable to find expected output file: $jpeg2Name", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless -f $jpeg2Name; +} + +# Add the result into the database +$jpeg1Name = $ipprc->convert_filename_relative( $jpeg1Name ); +$jpeg2Name = $ipprc->convert_filename_relative( $jpeg2Name ); +unless ($no_update) { + my $command = "$dettool -addnormalizedexp -det_id $det_id -iteration $iter " . + "-recip " . RECIPE1() . "," . RECIPE2() . " -b1_uri $outputRoot " . + "-bg $bg -bg_stdev $bg_stdev -bg_mean_stdev $bg_mean_stdev"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform dettool -addnormalizedexp: $error_code", $det_id, $iter, $error_code); + } + + unlink $list1Name; + unlink $list2Name; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $iter = shift; # Iteration + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $iter) { + my $command = "$dettool -addprocessedimfile -det_id $det_id -iter $iter -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_process_exp.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_process_exp.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_process_exp.pl (revision 11645) @@ -0,0 +1,215 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Statistics::Descriptive; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $exp_tag, $camera, $dbname, $workdir, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'exp_tag|e=s' => \$exp_tag, + 'camera|c=s' => \$camera, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --exp_tag --camera", + -exitval => 3, +) unless defined $det_id + and defined $exp_tag, + and defined $camera; + +$ipprc->define_camera($camera); + +use constant RECIPE1 => 'PPIMAGE_J1'; # Recipe to use for ppImage to make JPEGs +use constant RECIPE2 => 'PPIMAGE_J2'; # Recipe to use for ppImage to make JPEGs + + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get list of component files +my $files; # Array of component files +{ + my $command = "$dettool -processedimfile -det_id $det_id -exp_tag $exp_tag"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform dettool -processedimfile: $error_code", $det_id, $exp_tag, $error_code); + } + my $metadata = $mdcParser->parse(join "", @$stdout_buf) or + &my_die("Unable to parse metadata config doc", $det_id, $exp_tag, $PS_EXIT_PROG_ERROR); + $files = parse_md_list($metadata) or + &my_die("Unable to parse metadata list", $det_id, $exp_tag, $PS_EXIT_PROG_ERROR); +} + +# Gather the statistics +my ($bg, $bg_stdev, $bg_mean_stdev); # The statistics triplet +{ + my @backgrounds; # Array of backgrounds in each component + my @variances; # Array of variances for each component + foreach my $file (@$files) { + &my_die("Unable to find class id", $det_id, $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{class_id}; + my $class_id = $file->{class_id}; + &my_die("Unable to find bg for class_id=$class_id", $det_id, $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{bg}; + &my_die("Unable to find bg_mean_stdev for class_id=$class_id", $det_id, $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{bg_mean_stdev}; + push @backgrounds, $file->{bg}; + push @variances, $file->{bg_stdev}**2; + } + + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@backgrounds); + $bg = $stats->mean(); + $bg_mean_stdev = $stats->standard_deviation() || 0.0; + } + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@variances); + $bg_stdev = sqrt( $stats->mean() ); + } +} + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my $example = ${$files}[0]->{b1_uri}; # Example original root name + my ($vol, $dir, $file) = File::Spec->splitpath( $example ); + $workdir = $dir; +} +my $outputFile = $exp_tag . '.detproc.' . $det_id; # Root name +my $outputRoot = File::Spec->catfile( $workdir, $exp_tag, $outputFile ); + +# output files: +my $list1Name = $outputRoot . '.b1.list'; # Name for the input file list for binning 1 +my $list2Name = $outputRoot . '.b2.list'; # Name for the input file list for binning 2 +my @means; # Array of means +my @stdevs; # Array of stdevs +open my $list1File, '>' . $list1Name; +open my $list2File, '>' . $list2Name; +foreach my $file (@$files) { + my $origRoot = $ipprc->convert_filename_absolute( $file->{b1_uri} ); # Original root name + print $list1File ($ipprc->filename( "PPIMAGE.BIN1", $origRoot, $file->{class_id} ) . "\n"); + print $list2File ($ipprc->filename( "PPIMAGE.BIN2", $origRoot, $file->{class_id} ) . "\n"); + push @means, $file->{bg}; + push @stdevs, $file->{bg_stdev}; +} +close $list1File; +close $list2File; + +# Output products +my $jpeg1 = $ipprc->filename("PPIMAGE.JPEG1", $outputRoot); # Binned JPEG #1 +my $jpeg2 = $ipprc->filename("PPIMAGE.JPEG2", $outputRoot); # Binned JPEG #2 + +# Make the jpeg for binning 1 +{ + my $command = "$ppImage -list $list1Name $outputRoot -recipe PPIMAGE " . RECIPE1; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $det_id, $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $jpeg1", $det_id, $exp_tag, $PS_EXIT_SYS_ERROR) unless -f $jpeg1; +} + +# Make the jpeg for binning 2 +{ + my $command = "$ppImage -list $list2Name $outputRoot -recipe PPIMAGE " . RECIPE2; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $det_id, $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $jpeg2", $det_id, $exp_tag, $PS_EXIT_SYS_ERROR) unless -f $jpeg2; +} + + +# Add the result into the database +$outputRoot = $ipprc->convert_filename_relative( $outputRoot ); +$jpeg1 = $ipprc->convert_filename_relative( $jpeg1 ); +$jpeg2 = $ipprc->convert_filename_relative( $jpeg2 ); +unless ($no_update) { + my $command = "$dettool -addprocessedexp -det_id $det_id -exp_tag $exp_tag " . + "-recip " . RECIPE1() . "," . RECIPE2() . " -b1_uri $outputRoot " . + "-bg $bg -bg_stdev $bg_stdev -bg_mean_stdev $bg_mean_stdev"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addprocessedexp: $error_code"); + exit($error_code); + } + + unlink $list1Name; + unlink $list2Name; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $exp_tag = shift; # Exposure tag + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id && $exp_tag) { + my $command = "$dettool -addprocessedexp -det_id $det_id -exp_tag $exp_tag -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_process_imfile.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_process_imfile.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_process_imfile.pl (revision 11645) @@ -0,0 +1,190 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::Stats; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $exp_tag, $class_id, $det_type, $input_uri, $camera, $dbname, $workdir, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'exp_tag|e=s' => \$exp_tag, + 'class_id|i=s' => \$class_id, + 'det_type|t=s' => \$det_type, + 'input_uri|u=s' => \$input_uri, + 'camera|c=s' => \$camera, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --exp_tag --class_id --det_type --input_uri --camera", + -exitval => 3, +) unless defined $det_id + and defined $exp_tag + and defined $class_id + and defined $det_type + and defined $input_uri + and defined $camera; + +# XXX this exits with status = 0 on failure +$ipprc->define_camera($camera); + +# Recipes to use, as a function of the detrend type +use constant RECIPES => { + 'bias' => 'PPIMAGE_O', # Overscan only + 'dark' => 'PPIMAGE_OB', # Overscan and bias only + 'shutter' => 'PPIMAGE_OBD', # Overscan, bias and dark only + 'flat' => 'PPIMAGE_OBDS', # Overscan, bias, dark and shutter only + 'domeflat' => 'PPIMAGE_OBDS', # Overscan, bias, dark and shutter only + 'skyflat' => 'PPIMAGE_OBDS', # Overscan, bias, dark and shutter only + 'fringe' => 'PPIMAGE_OBDSF',# Overscan, bias, dark, shutter and flat only +}; + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} +$ppImage .= " -dbname $dbname" if defined $dbname; + +# Recipe to use in processing +my $recipe = RECIPES->{lc($det_type)}; +unless (defined $recipe) { + warn("Unrecognised detrend type: $det_type"); + exit($PS_EXIT_CONFIG_ERROR); +} + +### Output file name +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my ($vol, $dir, $file) = File::Spec->splitpath( $input_uri ); + $workdir = $dir; +} +my $outputFile = "$exp_tag.detproc.$det_id"; # Root name +my $outputRoot = File::Spec->catfile( $workdir, $exp_tag, $outputFile ); +my $outputImage = $ipprc->filename("PPIMAGE.OUTPUT", $outputRoot, $class_id); +my $outputBin1 = $ipprc->filename("PPIMAGE.BIN1", $outputRoot, $class_id); +my $outputBin2 = $ipprc->filename("PPIMAGE.BIN2", $outputRoot, $class_id); + +my $outputStats = $outputRoot . '.' . $class_id . '.stats'; + +$input_uri = $ipprc->convert_filename_absolute( $input_uri ); + +# Run ppImage +{ + my $command = "$ppImage -file $input_uri $outputRoot -recipe PPIMAGE $recipe" . + " -stat $outputStats"; # Command to run ppImage + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $det_id, $exp_tag, $class_id, $error_code); + } + &my_die("Couldn't find expected output file: $outputImage", $det_id, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputImage; + &my_die("Couldn't find expected output file: $outputStats", $det_id, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputStats; + &my_die("Couldn't find expected output file: $outputBin1", $det_id, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputBin1; + &my_die("Couldn't find expected output file: $outputBin2", $det_id, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputBin2; +} + +# Get the statistics on the processed image +my $stats; # Statistics from ppImage +{ + my $statsFile; # File handle + open $statsFile, "$outputStats" or die "Can't open statistics file $outputStats: $!\n"; + my @contents = <$statsFile>; # Contents of file + close $statsFile; + my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + my $metadata = $mdcParser->parse(join "", @contents) + or &my_die("Unable to parse metadata config", $det_id, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR); + $stats = PS::IPP::Metadata::Stats->new(); # Stats parser + $stats->parse($metadata) or &my_die("Unable to find all values in statistics output.", $det_id, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR); +} + +# Take off the absolute path, to stuff into the database +$outputImage = $ipprc->convert_filename_relative( $outputImage ); +$outputBin1 = $ipprc->convert_filename_relative( $outputBin1 ); +$outputBin2 = $ipprc->convert_filename_relative( $outputBin2 ); + +# Add the processed file to the database +unless ($no_update) { + my $command = "$dettool -addprocessedimfile -det_id $det_id -exp_tag $exp_tag " . + "-class_id $class_id -recip $recipe -uri $outputImage -b1_uri $outputRoot"; # Command to run dettool + $command .= " -bg " . $stats->bg_mean(); + if (defined($stats->bg_stdev())) { + $command .= " -bg_stdev " . $stats->bg_stdev(); + } else { + # May be undefined if there's only a single imfile + $command .= " -bg_stdev 0"; + } + $command .= " -bg_mean_stdev " . $stats->bg_mean_stdev(); + $command .= " -dbname $dbname" if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addprocessedimfile: $error_code\n"); + exit($error_code); + } + + unlink $outputStats; +} + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $exp_tag = shift; # Exposure tag + my $class_id = shift; # Class identifier + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $exp_tag and $class_id) { + my $command = "$dettool -addprocessedimfile -det_id $det_id -exp_tag $exp_tag -class_id $class_id -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_reject_exp.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_reject_exp.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_reject_exp.pl (revision 11645) @@ -0,0 +1,305 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Statistics::Descriptive; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $iter, $det_type, $camera, $filter, $workdir, $dbname, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'iteration=s' => \$iter, + 'det_type|t=s' => \$det_type, + 'camera=s' => \$camera, + 'filter=s' => \$filter, + 'workdir|w=s' => \$workdir, # Working directory for output files + 'dbname|d=s' => \$dbname, # Database name + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --iteration --det_type --camera", + -exitval => 3, +) unless defined $det_id + and defined $iter + and defined $det_type + and defined $camera; + + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get list of component files +my $exposures; # Array of exposures +{ + my $command = "$dettool -residexp -det_id $det_id -iteration $iter"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform dettool -residexp: $error_code", $det_id, $iter, $error_code); + } + + my $metadata = $mdcParser->parse(join "", @$stdout_buf) or + &my_die("Unable to parse metadata config doc", $det_id, $iter, $PS_EXIT_PROG_ERROR); + # XXX need to test that the result list is 0 length and return with a different error + $exposures = parse_md_list($metadata) or + &my_die("Unable to parse metadata list", $det_id, $iter, $PS_EXIT_PROG_ERROR); +} + +my @expTags; # Array of exposure IDs +my @means; # Array of means +my @stdevs; # Array of standard deviations +my @meanStdevs; # Array of standard deviations of the mean (normalised by the mean) +my @variances; # Array of variances +my @accept; # Array of accept flags +my @include; # Array of include flags +foreach my $exposure (@$exposures) { + &my_die("Unable to find exposure id.\n", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $exposure->{exp_tag}; + &my_die("Unable to find mean.\n", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $exposure->{bg}; + &my_die("Unable to find stdev.\n", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $exposure->{bg_stdev}; + &my_die("Unable to find mean stdev.\n", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $exposure->{bg_mean_stdev}; + &my_die("Unable to find accept.\n", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $exposure->{accept}; + &my_die("Unable to find include.\n", $det_id, $iter, $PS_EXIT_SYS_ERROR) unless defined $exposure->{include}; + push @expTags, $exposure->{exp_tag}; + push @means, $exposure->{bg}; + push @stdevs, $exposure->{bg_stdev}; + # XXX why are we using mean_stdev / bd? + # push @meanStdevs, $exposure->{bg_mean_stdev} / $exposure->{bg}; + push @meanStdevs, $exposure->{bg_mean_stdev}; + push @variances, $exposure->{bg_stdev}**2; + push @accept, $exposure->{accept}; + push @include, $exposure->{include}; +} +my $meanStats = Statistics::Descriptive::Sparse->new(); # Statistics calculator +$meanStats->add_data(@means); +my $stdevStats = Statistics::Descriptive::Sparse->new(); # Statistics calculator +$stdevStats->add_data(@stdevs); +my $meanStdevStats = Statistics::Descriptive::Sparse->new(); # Statistics calculator +$meanStdevStats->add_data(@meanStdevs); + +$ipprc->define_camera($camera); +# Rejection thresholds +my $reject_mean = rejection_limit( 'ENSEMBLE.MEAN', $det_type, $filter ); +my $reject_stdev = rejection_limit( 'ENSEMBLE.STDEV', $det_type, $filter ); +my $reject_meanstdev = rejection_limit( 'ENSEMBLE.MEANSTDEV', $det_type, $filter ); + +print "Ensemble mean " . $meanStats->mean() . " +/- " . $meanStats->standard_deviation . + ", stdev " . $stdevStats->mean() . " +/- " . $stdevStats->standard_deviation() . "\n\n"; + +# Go through again to do rejection, and update the database for each exposure +my $numChanges = 0; # Number of exposures with changed status +my $numReject = 0; # Number of exposures rejected +my $command; +my $reject; +my $expTag; +for (my $i = 0; $i < scalar @means; $i++) { + $expTag = $expTags[$i]; # Exposure ID + $command = "$dettool -updateresidexp -det_id $det_id -iteration $iter -exp_tag $expTag"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + + $reject = 0; # Reject this exposure? + + if (not $accept[$i]) { + # Rejected this at an earlier stage + print "Rejecting $expTag based on earlier determination.\n"; + $reject = 1; + goto UPDATE; + } + + if ($reject_mean and defined $meanStats->standard_deviation() ) { + my $dMean = abs($means[$i] - $meanStats->mean()) ; + if ($dMean > ($reject_mean * $meanStats->standard_deviation())) { + print "Rejecting $expTag based on outlier mean value: " . + "$means[$i] --> $dMean vs " . $reject_mean * $meanStats->standard_deviation() . "\n"; + $reject = 1; + goto UPDATE; + } + } else { + print "no rejection for exposure mean\n"; + } + + if ($reject_stdev and defined $stdevStats->standard_deviation() > 0) { + my $dMean = abs($stdevs[$i] - $stdevStats->mean()); + if ($dMean > ($reject_stdev * $stdevStats->standard_deviation())) { + print "Rejecting $expTag based on outlier stdev: " . + $stdevs[$i] . " --> $dMean sigma vs " . $reject_stdev . "\n"; + $reject = 1; + goto UPDATE; + } + } else { + print "no rejection for exposure stdev\n"; + } + + if ($reject_meanstdev and defined $meanStdevStats->standard_deviation() > 0) { + my $dMean = abs($meanStdevs[$i] - $meanStdevStats->mean()); + if ($dMean > ($reject_meanstdev * $meanStdevStats->standard_deviation())) { + print "Rejecting $expTag based on outlier mean stdev: " . + $meanStdevs[$i] . " --> $dMean sigma vs " . $reject_meanstdev . "\n"; + $reject = 1; + goto UPDATE; + } + } else { + print "no rejection for exposure mean stdev\n"; + } + + UPDATE: + if ($reject) { + $command .= ' -reject'; + $numReject++; + } + + # Check for status changes + if ((not $include[$i] and not $reject) or ($include[$i] and $reject)) { + print "Status of $expTag has changed.\n"; + $numChanges++; + } + + unless ($no_update) { + # Update + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform dettool -updateresidexp: $error_code", $det_id, $iter, $error_code); + } + } +} + +# Decide if the current is sufficient to use as a master, and if we can stop iterating +my $master = 1; # This is good enough for a master +my $stop = 1; # Stop iterating + +if ($numChanges > 0) { + $master = 0; + $stop = 0; +} + +# Rejecting everything --- stop before something bad happens! +if ($numReject == scalar @means) { + $master = 0; + $stop = 1; +} + +print "Master: $master\n"; +print "Stop: $stop\n"; + +# Put the result into the database +unless ($no_update) { + my $varianceStats = Statistics::Descriptive::Sparse->new(); # Statistics calculator + $varianceStats->add_data(@variances); + + my $command = "$dettool -adddetrunsummary -det_id $det_id -iteration $iter" . + " -bg " . $meanStats->mean() . " -bg_stdev " . sqrt($varianceStats->mean()) . + " -bg_mean_stdev " . $meanStats->standard_deviation(); + $command .= " -accept" if $master; + $command .= " -dbname $dbname" if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -adddetrunsummary: $error_code"); + exit($error_code); + } +} + +# Re-run processing if required +unless ($no_update) { + my $command = "$dettool -updatedetrun -det_id $det_id"; + if ($stop) { + $command .= ' -state stop'; + } else { + $command .= ' -again'; + } + + $command .= " -dbname $dbname" if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -updatedetrun: $error_code"); + exit($error_code); + } +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $iter = shift; # Iteration + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $iter) { + my $command = "$dettool -adddetrunsummary -det_id $det_id -iteration $iter -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + + + +# Retrieve the requested rejection limit, dying unless extant +sub rejection_limit +{ + my $name = shift; # Rejection limit to + my $type = shift; # Type of exposure + my $filter = shift; # Filter + + my $value = $ipprc->rejection( $name, $det_type, $filter ); + if (not defined $value) { + $filter = "(no filter)" unless defined $filter; + die "Unable to determine $name rejection limit for $det_type with $filter.\n"; + } + + return $value; +} + + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_reject_imfile.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_reject_imfile.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_reject_imfile.pl (revision 11645) @@ -0,0 +1,376 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Statistics::Descriptive; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $iter, $exp_tag, $det_type, $camera, $filter, $reject, $dbname, $workdir, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'iteration=s' => \$iter, + 'exp_tag|e=s' => \$exp_tag, + 'det_type|t=s' => \$det_type, + 'camera=s' => \$camera, + 'filter=s' => \$filter, + 'reject' => \$reject, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update, + ) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --iteration --exp_tag --det_type --camera", + -exitval => 3, + ) unless defined $det_id + and defined $iter + and defined $exp_tag + and defined $det_type + and defined $camera; + +$ipprc->define_camera($camera); + +use constant RECIPE1 => 'PPIMAGE_J1'; # Recipe to use for ppImage to make JPEGs +use constant RECIPE2 => 'PPIMAGE_J2'; # Recipe to use for ppImage to make JPEGs + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get list of imfile files +my $files; # Array of imfile files +{ + my $command = "$dettool -residimfile -det_id $det_id -iteration $iter -exp_tag $exp_tag"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -residimfile: $error_code\n"); + exit($error_code); + } + + my $metadata = $mdcParser->parse(join "", @$stdout_buf) or + &my_die("Unable to parse metadata config doc", $det_id, $iter, $exp_tag, $PS_EXIT_PROG_ERROR); + $files = parse_md_list($metadata) or + &my_die("Unable to parse metadata list", $det_id, $iter, $exp_tag, $PS_EXIT_PROG_ERROR); +} + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my $example = ${$files}[0]->{b1_uri}; # Example original root name + my ($vol, $dir, $file) = File::Spec->splitpath( $example ); + $workdir = $dir; +} + +# Generate the file list, and get the statistics +my $outputFile = "$exp_tag.detresid.$det_id.$iter"; # Root name +my $outputRoot = File::Spec->catfile( $workdir, $exp_tag, $outputFile ); + +my $list1Name = $outputRoot . '.b1.list'; # Name for the input file list for binning 1 +my $list2Name = $outputRoot . '.b2.list'; # Name for the input file list for binning 2 + +my @means; # Array of means +my @variances; # Array of variances +my @meanStdevs; # Array of mean stdevs +open my $list1File, '>' . $list1Name; +open my $list2File, '>' . $list2Name; +foreach my $file (@$files) { + my $origRoot = $ipprc->convert_filename_absolute( $file->{b1_uri} ); # Original root name + print $list1File ($ipprc->filename( "PPIMAGE.BIN1", $origRoot, $file->{class_id} ) . "\n"); + print $list2File ($ipprc->filename( "PPIMAGE.BIN2", $origRoot, $file->{class_id} ) . "\n"); + push @means, $file->{bg}; + push @meanStdevs, $file->{bg_mean_stdev}; + ## calculate the root-mean-square of the bd_stdevs + push @variances, $file->{bg_stdev}*$file->{bg_stdev}; + +} +close $list1File; +close $list2File; + +# Output products --- need to synch with the camera configuration! +my $jpeg1Name = $ipprc->filename("PPIMAGE.JPEG1", $outputRoot); # Binned JPEG #1 +my $jpeg2Name = $ipprc->filename("PPIMAGE.JPEG2", $outputRoot); # Binned JPEG #2 + +# Make the jpeg for binning 1 +{ + my $command = "$ppImage -list $list1Name $outputRoot -recipe PPIMAGE " . RECIPE1; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to run ppImage: $error_code", $det_id, $iter, $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $jpeg1Name", $det_id, $iter, $exp_tag, $PS_EXIT_SYS_ERROR) unless -f $jpeg1Name; +} + +# Make the jpeg for binning 2 +{ + my $command = "$ppImage -list $list2Name $outputRoot -recipe PPIMAGE " . RECIPE2; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to run ppImage: $error_code", $det_id, $iter, $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $jpeg2Name", $det_id, $iter, $exp_tag, $PS_EXIT_SYS_ERROR) unless -f $jpeg2Name; +} + +my $expected = rejection_limit( 'EXPECTED', $det_type, $filter ); # Expected mean +# Rejection thresholds +my $reject_imfile_mean = rejection_limit( 'IMFILE.MEAN', $det_type, $filter ); +my $reject_imfile_stdev = rejection_limit( 'IMFILE.STDEV', $det_type, $filter ); +my $reject_imfile_meanstdev = rejection_limit( 'IMFILE.MEANSTDEV', $det_type, $filter ); +my $reject_imfile_sn = rejection_limit( 'IMFILE.SN', $det_type, $filter ); +my $reject_exp_mean = rejection_limit( 'EXP.MEAN', $det_type, $filter ); +my $reject_exp_stdev = rejection_limit( 'EXP.STDEV', $det_type, $filter ); +my $reject_exp_meanstdev = rejection_limit( 'EXP.MEANSTDEV', $det_type, $filter ); +my $reject_exp_sn = rejection_limit( 'EXP.SN', $det_type, $filter ); + +# Reject based on the stats of the imfiles +# it is VALID to reject on more than one criterion +&my_die("Number of means and number of variances differ!", $det_id, $iter, $exp_tag, $PS_EXIT_PROG_ERROR) unless scalar @means == scalar @variances; +for (my $i = 0; $i < scalar @means; $i++) { + my $mean = $means[$i]; # Mean for this imfile + $mean -= $expected; + my $stdev = sqrt($variances[$i]); # Stdev for this imfile + + if ($reject_imfile_mean) { + if (abs($mean) > $reject_imfile_mean * $stdev) { + print "Rejecting exposure based on bad imfile mean for imfile $i: " . + $mean . " vs " . $reject_imfile_mean . "\n"; + $reject = 1; + last; + } + } else { + print "no rejection for imfile mean\n"; + } + if ($reject_imfile_stdev) { + if ($stdev > $reject_imfile_stdev) { + print "Rejecting exposure based on bad imfile stdev for imfile $i: " . + $stdev . " vs " . $reject_imfile_stdev . "\n"; + $reject = 1; + last; + } + } else { + print "no rejection for imfile stdev\n"; + } + if ($reject_imfile_meanstdev) { + if ($meanStdevs[$i] > $reject_imfile_meanstdev) { + print "Rejecting exposure based on bad imfile mean stdev for imfile $i: " . + $meanStdevs[$i] . " vs " . $reject_imfile_meanstdev . "\n"; + $reject = 1; + last; + } + } else { + print "no rejection for imfile mean stdev\n"; + } + if ($reject_imfile_sn) { + if ($mean < $stdev * $reject_imfile_sn) { + print "Rejecting exposure based on bad imfile S/N for imfile $i: \n"; + print "mean: " . $mean . " vs " . "stdev*SNlimit: " . $stdev * $reject_imfile_sn . "\n"; + $reject = 1; + last; + } + } else { + print "no rejection for imfile S/N\n"; + } + +} + +# calculate the exposure ensemble statistics +my $meanStats = Statistics::Descriptive::Sparse->new(); # Statistics calculator for means +$meanStats->add_data(@means); +my $varianceStats = Statistics::Descriptive::Sparse->new(); # Statistics calculator for variances +$varianceStats->add_data(@variances); + +my $mean = $meanStats->mean(); # Mean of the imfile means +my $meanStdev = $meanStats->standard_deviation(); # Stdev of the imfile means +if (not defined $meanStdev) { + # this is the case for Nimfile == 1 + $meanStdev = 0; +} +my $stdev = sqrt($varianceStats->mean()); # Root-Mean-Square of the imfile stdevs (root mean of variances) +my $exp_sn = 0.0; +if ($stdev > 0) { $exp_sn = $mean / $stdev; } + +print "exposure mean $mean, stdev $stdev, mean stdev $meanStdev, exp s/n: $exp_sn\n"; + +## Reject based on the exposure ensemble stats +# reject if the exposure ensemble mean is deviant +if ($reject_exp_mean) { + if (abs($mean) > $reject_exp_mean * $stdev) { + print "Rejecting exposure based on bad mean: \n"; + print "mean: $mean, stdev: $stdev (limit is: % $reject_exp_mean\n"; + $reject = 1; + } +} else { + print "no rejection for exp mean\n"; +} +# reject if the exposure ensemble stdev is deviant +if ($reject_exp_stdev) { + if ($stdev > $reject_exp_stdev) { + print "Rejecting exposure based on bad mean stdev: " . $stdev . " vs " . + $reject_exp_stdev . "\n"; + $reject = 1; + } +} else { + print "no rejection for exp stdev\n"; +} +# reject if the exposure ensemble mean stdev is deviant +if ($reject_exp_meanstdev) { + if ($meanStdev > $reject_exp_meanstdev) { + print "Rejecting exposure based on bad mean stdev: " . $meanStdev . " vs " . + $reject_exp_meanstdev . "\n"; + $reject = 1; + } +} else { + print "no rejection for exp mean stdev\n"; +} +# reject if the signal-to-noise is insufficient +if ($reject_exp_sn) { + if (abs($mean) < abs($stdev * $reject_exp_sn)) { + print "Rejecting exposure based on poor S/N: \n"; + print "signal: $mean vs noise: $stdev (s/n limit is: $reject_exp_sn)\n"; + $reject = 1; + } +} else { + print "no rejection for exp S/N\n"; +} + +# Add the result into the database +$jpeg1Name = $ipprc->convert_filename_relative( $jpeg1Name ); +$jpeg2Name = $ipprc->convert_filename_relative( $jpeg2Name ); +unless ($no_update) { + my $command = "$dettool -addresidexp -det_id $det_id -iteration $iter -exp_tag $exp_tag " . + "-recip " . RECIPE1() . "," . RECIPE2() . " -b1_uri $outputRoot " . + "-bg $mean -bg_stdev $stdev -bg_mean_stdev $meanStdev"; # Command to run + $command .= ' -reject' if $reject; + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addresidexp: $error_code\n"); + exit($error_code); + } + + unlink $list1Name; + unlink $list2Name; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $iter = shift; # Iteration + my $exp_tag = shift; # Exposure tag + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $iter and $exp_tag) { + my $command = "$dettool -addresidexp -det_id $det_id -iteration $iter -exp_tag $exp_tag -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + + +# Retrieve the requested rejection limit, dying if not extant +sub rejection_limit +{ + my $name = shift; # Rejection limit to + my $type = shift; # Type of exposure + my $filter = shift; # Filter + + my $value = $ipprc->rejection( $name, $det_type, $filter ); + if (not defined $value) { + $filter = "(no filter)" if not defined $filter; + die "Unable to determine $name rejection limit for $det_type with $filter.\n"; + } + + return $value; +} + + + + +__END__ + +#### + +## this function is not well though out. it loads a collection of +## values (background mean and stdev) for a collection of imfile +## subcomponents. it then measures the mean background, the +## background stdev (rms of ensemble stdevs), and the background mean +## stdev (rms of the means). it then tries to reject based on the +## following criteria: +## (background / stdev > limit) for a single component +## ---> this will ACCEPT for a poor images (large stdev) +## (stdev > limit) +## (mean background / stdev > limit) same problem +## (mean stdev > limit) + +### I would suggest the following: +## 1) calculate the ensemble mean and stdev +## 2) for each component, is the (back - mean) / mean stdev > limit? +## (ie, is it an outlier?) +## 3) is the component stdev > limit? (absolute test) +## 4) calculate the background stdev for the ensemble (excluding rejects) +## 5) is the mean stdev > limit (poor images) +## 6) calculate the ensemble stdev (rms) of remaining (excluding rejects) +## 7) is the ensemble stdev > limit + +## for flats, the value used for stdev should be the fractional stdev +## (stdev/mean) and no rejection should be performed on the basis of +## the mean value, but could still be done on the mean stdev + +## the same logical cuts above can be applied to components in an +## imfile, imfiles in an exposure, and exposures in a stack + +### this function needs to avoid div by zero: compare Signal vs Noise*limit Index: /branches/rel-1-0/ippScripts/scripts/detrend_resid.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_resid.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_resid.pl (revision 11645) @@ -0,0 +1,233 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::Stats; +use Data::Dumper; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $iter, $exp_tag, $class_id, $det_type, $detrend, + $input_uri, $camera, $mode, $dbname, $workdir, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'iteration=s' => \$iter, + 'exp_tag|e=s' => \$exp_tag, + 'class_id|i=s' => \$class_id, + 'det_type|t=s' => \$det_type, + 'detrend=s' => \$detrend, + 'input_uri|u=s' => \$input_uri, + 'camera|c=s' => \$camera, + 'mode|m=s' => \$mode, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update, +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --iteration --exp_tag --class_id --det_type --camera --input_uri --mode --detrend (not for 'verify' mode)", + -exitval => 3, +) unless defined $det_id + and defined $iter + and defined $exp_tag + and defined $class_id + and defined $det_type + and defined $input_uri + and defined $camera + and defined $mode + and (defined $detrend or lc($mode) eq 'verify'); + +$ipprc->define_camera($camera); + +# Recipes to use, as a function of the detrend type +use constant RECIPES => { + 'master' => { # We're creating a master --- already processed the input + 'bias' => 'PPIMAGE_B', # Bias only + 'dark' => 'PPIMAGE_D', # Dark only + 'shutter' => 'PPIMAGE_S', # Shutter only + 'flat' => 'PPIMAGE_F', # Flat-field only + 'domeflat' => 'PPIMAGE_F', # Flat-field only + 'skyflat' => 'PPIMAGE_F', # Flat-field only + 'fringe' => 'PPIMAGE_R', # Fringe only + }, + 'verify' => { # We're checking the master --- input is not already processed + 'bias' => 'PPIMAGE_OB', # Bias only + 'dark' => 'PPIMAGE_OBD', # Dark only + 'shutter' => 'PPIMAGE_OBDS', # Shutter only + 'flat' => 'PPIMAGE_OBDSF', # Flat-field only + 'domeflat' => 'PPIMAGE_OBDSF', # Flat-field only + 'skyflat' => 'PPIMAGE_OBDSF', # Flat-field only + 'fringe' => 'PPIMAGE_OBDSFR', # Fringe only + }, +}; + +# Flags to specify the particular detrend to use +use constant DETRENDS => { + 'bias' => '-bias', # Specify the bias frame + 'dark' => '-dark', # Specify the dark frame + 'shutter' => '-shutter', # Specify the shutter frame + 'flat' => '-flat', # Specify the flat frame + 'domeflat' => '-flat', # Specify the flat frame + 'skyflat' => '-flat', # Specify the flat frame + 'fringe' => '-fringe', # Specify the fringe frame +}; + + +use constant DELETE_STATS => 0; # Delete the statistics file when done? + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} +$ppImage .= " -dbname $dbname" if defined $dbname; + +# Recipe to use in processing +&my_die("Unrecognised mode: $mode", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR) unless defined RECIPES->{$mode}; +my $recipe = RECIPES->{$mode}->{lc($det_type)}; +&my_die("Unrecognised detrend type: $det_type", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR) unless defined $recipe; + +### Output file names --- must match camera configuration! +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my ($vol, $dir, $file) = File::Spec->splitpath( $input_uri ); + $workdir = $dir; +} + +my $outputFile = "$exp_tag.detresid.$det_id.$iter"; # Root name +my $outputRoot = File::Spec->catfile( $workdir, $exp_tag, $outputFile ); + +my $outputName = $ipprc->filename("PPIMAGE.OUTPUT", $outputRoot, $class_id); +my $bin1Name = $ipprc->filename("PPIMAGE.BIN1", $outputRoot, $class_id); +my $bin2Name = $ipprc->filename("PPIMAGE.BIN2", $outputRoot, $class_id); + +my $outputStats = $outputRoot . '.' . $class_id . '.stats'; + +$detrend = $ipprc->convert_filename_absolute( $detrend ) if defined $detrend; +$input_uri = $ipprc->convert_filename_absolute( $input_uri ); + +# Run ppImage +{ + my $command = "$ppImage -file $input_uri $outputRoot -recipe PPIMAGE $recipe" . + " -stat $outputStats "; # Command to run ppImage + + # Detrend to use in processing + if (lc($mode) ne 'verify') { + my $detFlag = DETRENDS->{lc($det_type)}; + &my_die("Unrecognised detrend type: $det_type", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR) unless defined $detFlag; + $command .= "$detFlag $detrend"; + } + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $det_id, $iter, $exp_tag, $class_id, $error_code); + } + &my_die("Couldn't find expected output file: $outputName", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputName; + &my_die("Couldn't find expected output file: $outputStats", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputStats; + &my_die("Couldn't find expected output file: $bin1Name", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $bin1Name; + &my_die("Couldn't find expected output file: $bin2Name", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $bin2Name; +} + +# Get the statistics on the residual image +my $stats; # Statistics from ppImage +{ + my $statsFile; # File handle + open $statsFile, $outputStats or &my_die("Can't open statistics file $outputStats: $!", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_SYS_ERROR); + my @contents = <$statsFile>; # Contents of file + close $statsFile; + my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + my $metadata = $mdcParser->parse(join "", @contents) or + &my_die("Unable to parse metadata config doc", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR); + $stats = PS::IPP::Metadata::Stats->new(); # Stats parser + $stats->parse($metadata) or &my_die("Unable to find all values in statistics output.", $det_id, $iter, $exp_tag, $class_id, $PS_EXIT_PROG_ERROR); +} + +# Add the processed file to the database +$outputName = $ipprc->convert_filename_relative( $outputName ); +$bin1Name = $ipprc->convert_filename_relative( $bin1Name ); +$bin2Name = $ipprc->convert_filename_relative( $bin2Name ); +unless ($no_update) { + my $command = "$dettool -addresidimfile -det_id $det_id -iteration $iter -exp_tag $exp_tag " . + "-class_id $class_id -recip $recipe -uri $outputName -b1_uri $outputRoot"; # Command to run dettool + $command .= " -bg " . $stats->bg_mean(); + $command .= " -dbname $dbname" if defined $dbname; + + if (defined($stats->bg_mean_stdev())) { + $command .= " -bg_mean_stdev " . $stats->bg_mean_stdev(); + } else { + # May be undefined if there is only a single imfile + $command .= ' -bg_stdev 0'; + } + + $command .= " -bg_stdev " . $stats->bg_stdev(); + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addresidimfile: $error_code\n"); + exit($error_code); + } + + unlink $outputStats; +} + + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $iter = shift; # Iteration + my $exp_tag = shift; # Exposure tag + my $class_id = shift; # Class identifier + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $iter and $exp_tag) { + my $command = "$dettool -addresidimfile -det_id $det_id -iteration $iter -exp_tag $exp_tag -class_id $class_id -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +END { + my $exit = $?; + system("sync") == 0 or die "failed to execute sync: $!"; + $? = $exit; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/detrend_stack.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/detrend_stack.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/detrend_stack.pl (revision 11645) @@ -0,0 +1,201 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use PS::IPP::Metadata::Stats; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($det_id, $iter, $class_id, $det_type, $camera, $dbname, $workdir, $no_update); +GetOptions( + 'det_id|d=s' => \$det_id, + 'iteration=s' => \$iter, + 'class_id|i=s' => \$class_id, + 'det_type|t=s' => \$det_type, + 'camera|c=s' => \$camera, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, # Working directory, for output files + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --det_id --iteration --class_id --det_type --camera", + -exitval => 3, +) unless defined $det_id + and defined $iter + and defined $class_id + and defined $det_type + and defined $camera; + +# Recipes to use as a function of detrend type +use constant RECIPES => { + 'bias' => 'PPMERGE_BIAS', + 'dark' => 'PPMERGE_DARK', + 'shutter' => 'PPMERGE_SHUTTER', + 'flat' => 'PPMERGE_FLAT', + 'domeflat' => 'PPMERGE_FLAT', + 'skyflat' => 'PPMERGE_FLAT', + 'fringe' => 'PPMERGE_FRINGE', + }; + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $ppMerge = can_run('ppMerge') or (warn "Can't find ppMerge" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $recipe = RECIPES()->{lc($det_type)}; # Recipe to use in stacking +&my_die("Unrecognised detrend type: $det_type", $det_id, $iter, $class_id, $PS_EXIT_PROG_ERROR) unless defined $recipe; + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get list of files to stack +my $files; # Array of files to be stacked +{ + my $command = "$dettool -processedimfile -det_id $det_id -class_id $class_id -included"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform dettool -processedimfile: $error_code", $det_id, $iter, $class_id, $error_code); + } + my $metadata = $mdcParser->parse(join "", @$stdout_buf) or + &my_die("Unable to parse metadata config doc", $det_id, $iter, $class_id, $PS_EXIT_PROG_ERROR); + $files = parse_md_list($metadata) or + &my_die("Unable to parse metadata list", $det_id, $iter, $class_id, $PS_EXIT_PROG_ERROR); +} + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my $example = ${$files}[0]->{uri}; # Example file + my ($vol, $dir, $file) = File::Spec->splitpath( $example ); + $workdir = $dir; +} + +# Stack the files +my $outputFile = "$camera.$det_type.$det_id.$iter.$class_id"; # Root name +my $outputSubD = "$camera.$det_type.$det_id"; +my $outputRoot = File::Spec->catfile( $workdir, $outputSubD, $outputFile ); +my $outputStack = $outputRoot . '.fits'; # Output name +my $outputStats = $outputRoot . '.stats'; # Statistics name + +{ + my $command = "$ppMerge $outputStack"; # Command to run + foreach my $file (@$files) { + my $uri = $file->{uri}; # URI for input file + $uri = $ipprc->convert_filename_absolute($uri); + $command .= ' ' . $uri; + } + $command .= " -recipe PPMERGE $recipe"; + $command .= ' -type ' . uc($det_type); # Type of stacking to perform + $command .= " -stats $outputStats"; # Statistics output filename + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppMerge: $error_code", $det_id, $iter, $class_id, $error_code); + } + &my_die("Unable to find expected output file: $outputStack\n", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputStack; + &my_die("Unable to find expected output file: $outputStats\n", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputStats; +} + +# Get the statistics on the stacked image +my $stats; # Statistics from ppMerge +{ + open(my $statsFile, "$outputStats") or + &my_die("Can't open statistics file $outputStats: $!", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR); + my $contents = do { local $/; <$statsFile> }; # Contents of file + close($statsFile); + + my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + my $metadata = $mdcParser->parse($contents) or + &my_die("Unable to parse metadata config doc", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR); + $stats = PS::IPP::Metadata::Stats->new(); # Stats parser + $stats->parse($metadata) or + &my_die("Unable to find all values in statistics output.", $det_id, $iter, $class_id, $PS_EXIT_SYS_ERROR); +} + +# Add the resultant into the database +$outputStack = $ipprc->convert_filename_relative($outputStack); +unless ($no_update) { + my $command = "$dettool -addstacked -det_id $det_id -iteration $iter -class_id $class_id" . + " -uri $outputStack -recip $recipe"; # Command to run + $command .= ' -bg ' . $stats->bg_mean(); + if (defined($stats->bg_stdev())) { + $command .= ' -bg_stdev ' . $stats->bg_stdev(); + } else { + # May be undefined if there is only a single imfile + $command .= ' -bg_stdev 0'; + } + $command .= ' -bg_mean_stdev ' . $stats->bg_mean_stdev(); + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addstacked: $error_code\n"); + exit($error_code); + } + + unlink $outputStats; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $det_id = shift; # Detrend identifier + my $iter = shift; # Iteration + my $class_id = shift; # Class identifier + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($det_id and $iter and $class_id) { + my $command = "$dettool -addstacked -det_id $det_id -iteration $iter -class_id $class_id -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/ipp_datapath.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/ipp_datapath.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/ipp_datapath.pl (revision 11645) @@ -0,0 +1,15 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +use PS::IPP::Config; +my $ipprc = PS::IPP::Config->new(); + +die "No filename specified.\n" if scalar @ARGV != 1; + +print $ipprc->convert_filename_absolute(shift @ARGV) . "\n"; + +1; + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/ipp_filename.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/ipp_filename.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/ipp_filename.pl (revision 11645) @@ -0,0 +1,19 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +use PS::IPP::Config; +my $ipprc = PS::IPP::Config->new(); + +die "USAGE: ipp_filename.pl (filerule) (output) (classID)\n" if scalar @ARGV != 3; + +my $filerule = $ARGV[0]; +my $output = $ARGV[1]; +my $classID = $ARGV[2]; + +print $ipprc->convert_filename_absolute($filerule, $output, $classID) . "\n"; + +1; + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/ipp_serial_detrend.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/ipp_serial_detrend.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/ipp_serial_detrend.pl (revision 11645) @@ -0,0 +1,274 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Data::Dumper; + +my $mdcParser = PS::IPP::Metadata::Config->new; # Metadata config parser + +# Look for programs we need +my $missing_tools; +my $dettool = can_run('dettool') or (warn "Can't find dettool" and $missing_tools = 1); +my $detrend_process_imfile = can_run('detrend_process_imfile.pl') or (warn "Can't find detrend_process_imfile.pl" and $missing_tools = 1); +my $detrend_process_exp = can_run('detrend_process_exp.pl') or (warn "Can't find detrend_process_exp.pl" and $missing_tools = 1); +my $detrend_stack = can_run('detrend_stack.pl') or (warn "Can't find detrend_stack.pl" and $missing_tools = 1); +my $detrend_norm_calc = can_run('detrend_norm_calc.pl') or (warn "Can't find detrend_norm_calc.pl" and $missing_tools = 1); +my $detrend_norm_apply = can_run('detrend_norm_apply.pl') or (warn "Can't find detrend_norm_apply.pl" and $missing_tools = 1); +my $detrend_norm_exp = can_run('detrend_norm_exp.pl') or (warn "Can't find detrend_norm_exp.pl" and $missing_tools = 1); +my $detrend_resid = can_run('detrend_resid.pl') or (warn "Can't find detrend_resid.pl" and $missing_tools = 1); +my $detrend_reject_imfile = can_run('detrend_reject_imfile.pl') or (warn "Can't find detrend_reject_imfile.pl" and $missing_tools = 1); +my $detrend_reject_exp = can_run('detrend_reject_exp.pl') or (warn "Can't find detrend_reject_exp.pl" and $missing_tools = 1); +die "Can't find required tools.\n" if $missing_tools; + +# Process raw imfiles +{ + my $command = "$dettool -toprocessedimfile"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get detrend raw list: $error_code\n" if not $success; + + my @whole = split /\n/, join( '', @$stdout_buf ); + my @single = (); + + while ( scalar @whole > 0 ) { + my $value = shift @whole; + push @single, $value; + if ($value =~ /^\s*END\s*$/) { + push @single, "\n"; + + my $list = parse_md_list( $mdcParser->parse( join( "\n", @single ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $det_id = $item->{det_id}; + my $det_type = $item->{det_type}; + my $exp_tag = $item->{exp_tag}; + my $class = $item->{class}; + my $class_id = $item->{class_id}; + my $uri = $item->{uri}; + my $camera = $item->{camera}; + + my $command = "$detrend_process_imfile --det_id $det_id --exp_tag $exp_tag --class $class --class_id $class_id --det_type $det_type --input_uri $uri --camera $camera"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do raw imfile processing on $exp_tag $class_id: $error_code\n" if not $success; + } + + @single = (); + + } + } +} + +# Process raw exposures +{ + my $command = "$dettool -toprocessedexp"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get detrend raw list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $exp_tag = $item->{exp_tag}; + my $camera = $item->{camera}; + my $det_id = $item->{det_id}; + + my $command = "$detrend_process_exp --det_id $det_id --exp_tag $exp_tag --camera $camera"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do raw exposure processing on $det_id $exp_tag: $error_code\n" if not $success; + } +} + +# Stack +{ + my $command = "$dettool -tostacked"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get stack list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $class_id = $item->{class_id}; + my $det_type = $item->{det_type}; + my $camera = $item->{camera}; + + my $command = "$detrend_stack --det_id $det_id --iteration $iteration --class_id $class_id --det_type $det_type --camera $camera"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to stack detrend $det_id $iteration $class_id: $error_code\n" if not $success; + } +} + +# Calculate normalisation +{ + my $command = "$dettool -tonormalizedstat"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get normalise calculation list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $det_type = $item->{det_type}; + + my $command = "$detrend_norm_calc --det_id $det_id --iteration $iteration --det_type $det_type"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to calculate normalisation for $det_id $iteration: $error_code\n" if not $success; + } +} + +# Apply normalisation +{ + my $command = "$dettool -tonormalize"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get normalisation list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $det_type = $item->{det_type}; + my $class_id = $item->{class_id}; + my $value = $item->{norm}; + my $uri = $item->{uri}; + my $camera = $item->{camera}; + + my $command = "$detrend_norm_apply --det_id $det_id --iteration $iteration --class_id $class_id --value $value --input_uri $uri --camera $camera --det_type $det_type"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to apply normalisation for $det_id $iteration $class_id: $error_code\n" if not $success; + } +} + +# Examine normalised exposure +{ + my $command = "$dettool -tonormalizedexp"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get normalised exposures list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $det_type = $item->{det_type}; + my $class_id = $item->{class_id}; + my $value = $item->{norm}; + my $uri = $item->{uri}; + my $camera = $item->{camera}; + + my $command = "$detrend_norm_exp --det_id $det_id --iteration $iteration --camera $camera --det_type $det_type"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to examine normalised exposure for $det_id $iteration: $error_code\n" if not $success; + } +} + +# Get residuals +{ + my $command = "$dettool -toresidimfile"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get residual processing list: $error_code\n" if not $success; + + my @whole = split /\n/, join( '', @$stdout_buf ); + my @single = (); + + while ( scalar @whole > 0 ) { + my $value = shift @whole; + push @single, $value; + if ($value =~ /^\s*END\s*$/) { + push @single, "\n"; + + my $list = parse_md_list( $mdcParser->parse( join( "\n", @single ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $exp_tag = $item->{exp_tag}; + my $camera = $item->{camera}; + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $class_id = $item->{class_id}; + my $det_type = $item->{det_type}; + my $detrend = $item->{det_uri}; + my $uri = $item->{uri}; + my $mode = $item->{mode}; + + my $command = "$detrend_resid --det_id $det_id --iteration $iteration --exp_tag $exp_tag --class_id $class_id --det_type $det_type --camera $camera --input_uri $uri --mode $mode"; + $command .= " --detrend $detrend" if defined $detrend; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do residual processing on $exp_tag $class_id: $error_code\n" if not $success; + } + + @single = (); + + } + } +} + +# Reject based on imfiles +{ + my $command = "$dettool -toresidexp"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get residual imfile list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $exp_tag = $item->{exp_tag}; + my $camera = $item->{camera}; + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $det_type = $item->{det_type}; + + my $command = "$detrend_reject_imfile --det_id $det_id --iteration $iteration --exp_tag $exp_tag --det_type $det_type --camera $camera"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do imfile rejection on $exp_tag $det_id $iteration: $error_code\n" if not $success; + } +} + +# Reject based on exposures +{ + my $command = "$dettool -residdetrun"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get residual exposure list: $error_code\n" if not $success; + my $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from dettool.\n"; + + foreach my $item (@$list) { + my $camera = $item->{camera}; + my $det_id = $item->{det_id}; + my $iteration = $item->{iteration}; + my $det_type = $item->{det_type}; + + my $command = "$detrend_reject_exp --det_id $det_id --iteration $iteration --det_type $det_type --camera $camera"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do exposure rejection on $det_id $iteration: $error_code\n" if not $success; + } +} + + +__END__ + + Index: /branches/rel-1-0/ippScripts/scripts/ipp_serial_inject.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/ipp_serial_inject.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/ipp_serial_inject.pl (revision 11645) @@ -0,0 +1,98 @@ +#!/usr/bin/env perl + +# this program injects a list of single-file exposures into the db, taking the +# filename (without .fits) as the exp_tag. + +use warnings; +use strict; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use File::Spec; +use File::Find::Rule; +use PS::IPP::Config; +my $ipprc = PS::IPP::Config->new(); +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +# Parse the command-line arguments +my ($workdir, $dbname); +GetOptions( + 'workdir|w=s' => \$workdir, # working directory for output files + 'dbname|d=s' => \$dbname # Database name +) or pod2usage( 2 ); + +pod2usage( -msg => "Usage: $0 [--workdir path] [--dbname dbname] (files)", -exitval => 2 ) if scalar @ARGV == 0; + +my $pxinject = can_run('pxinject') or die "Can't find pxinject\n"; +my $ppStats = can_run('ppStats') or die "Can't find ppStats\n"; + +# if workdir is not defined, assign the current path +# XXX we need to handle relative paths for workdir (not allowed) +if (! $workdir) { + $workdir = File::Spec->rel2abs( "." ); +} + +my $num = 0; +foreach my $file ( @ARGV ) { + my $absfile = File::Spec->rel2abs( $file ); + inject($absfile, $workdir, $dbname); + $num ++; +} + +print "$num files injected.\n"; + +sub inject +{ + my $absfile = shift; # absolute path for this file + my $workdir = shift; # absolute path for output directory + my $dbname = shift; + + my ( $vol, $path, $name ) = File::Spec->splitpath( $absfile ); + my ( $exp ) = $name =~ /(.*)\.fits/; + + my $relfile = $ipprc->convert_filename_relative( $absfile ); + + my $command_type = "$ppStats -concept FPA.OBSTYPE -concept FPA.TELESCOPE -concept FPA.INSTRUMENT $absfile"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = run(command => $command_type, verbose => 1); + die "Unable to perform ppStats: $error_code\n" if not $success; + my $metadata = $mdcParser->parse(join "", @$stdout_buf) + or die "unable to parse metadata config doc"; + + my $type; + my $telescope; + my $instrument; + foreach my $row (@$metadata) { + if ($row->{name} eq "FPA.TELESCOPE") { $telescope = $row->{value}; } + if ($row->{name} eq "FPA.INSTRUMENT") { $instrument = $row->{value}; } # this entry must return the camera in the ipp/config system which we load +# if ($row->{name} eq "FPA.OBSTYPE") { $type = $row->{value}; } + } + + my $command_exp = "$pxinject -newExp -exp_id $exp -inst $instrument -telescope $telescope -imfiles 1 -workdir $workdir" ; # Command to run + if ($dbname) { + $command_exp = "$command_exp -dbname $dbname"; + } + + my ( $success_exp, $error_code_exp, $full_buf_exp, $stdout_buf_exp, $stderr_buf_exp ) = + run( command => $command_exp, verbose => 1 ); + die "Unable to inject $exp: $error_code_exp\n" if not $success_exp; + + my @line = split(/\s+/, $$stdout_buf_exp[0]); # The output line, containing the exposure tag + my $exp_tag = $line[2]; # The exposure tag + + my $command_imfile = "$pxinject -newImfile -exp_tag $exp_tag -class fpa -class_id fpa -uri $relfile"; # Command to run + if ($dbname) { + $command_imfile = "$command_imfile -dbname $dbname"; + } + + my ( $success_imfile, $error_code_imfile, $full_buf_imfile, $stdout_buf_imfile, $stderr_buf_imfile ) = run( command => $command_imfile, verbose => 1 ); + die "Unable to inject $exp imfile: $error_code_imfile\n" if not $success_imfile; + + return 1; +} + + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/ipp_serial_inject_mosaic.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/ipp_serial_inject_mosaic.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/ipp_serial_inject_mosaic.pl (revision 11645) @@ -0,0 +1,91 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Data::Dumper; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($camera, # Camera used + $telescope, # Telescope used + $exp_type, # Type of exposure + $path, # Path to use + ); +GetOptions( + 'camera|c=s' => \$camera, + 'telescope|t=s' => \$telescope, + 'exp_type|e=s' => \$exp_type, + 'path=s' => \$path, +) or pod2usage( 2 ); + +pod2usage( + -msg => "Required options: --camera --telescope --exp_type --path", + -exitval => 3, + ) unless scalar @ARGV == 0 or (defined $camera + and defined $telescope + and defined $exp_type + and defined $path); + +my $mdcParser = PS::IPP::Metadata::Config->new; # Metadata config parser + +# Look for programs we need +my $missing_tools; +my $pxinject = can_run('pxinject') or (warn "Can't find pxinject" and $missing_tools = 1); + +# Inject new data into the database +if (scalar @ARGV != 0) { + my @classes; + my $imfiles; + if ($camera eq "MEGACAM") { + for (my $i = 0; $i < 36; $i++) { + push @classes, sprintf("ccd%02d", $i); + } + } elsif ($camera eq "MCSHORT") { + @classes = ( 'ccd12', 'ccd13', 'ccd14', 'ccd21', 'ccd22', 'ccd23' ); + } elsif ($camera eq "CTIO_MOSAIC2") { + @classes = (); + } else { + die "Unrecognised camera name: $camera.\n"; + } + + foreach my $exp ( @ARGV ) { + my $command = "$pxinject -newExp -exp_id $exp -inst $camera -telescope $telescope -exp_type $exp_type -imfiles " . (scalar @classes or 1) ; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to inject $exp: $error_code\n" if not $success; + + my @line = split(/\s+/, $$stdout_buf[0]); # The output line, containing the exposure tag + my $exp_tag = $line[2]; # The exposure tag + foreach my $class_id (@classes) { + my $command = "$pxinject -newImfile -exp_tag $exp_tag -class chip -class_id $class_id -uri $path/$exp.$class_id.fits"; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to inject $exp $class_id: $error_code\n" if not $success; + } + + if (scalar @classes == 0) { + my $command = "$pxinject -newImfile -exp_tag $exp_tag -class fpa -class_id fpa -uri $path/$exp.fits"; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to inject $exp imfile: $error_code\n" if not $success; + } + + } +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + + +__END__ + + Index: /branches/rel-1-0/ippScripts/scripts/ipp_serial_phase0.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/ipp_serial_phase0.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/ipp_serial_phase0.pl (revision 11645) @@ -0,0 +1,86 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Data::Dumper; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($dbname # Database name to use + ); +GetOptions( + 'dbname|d=s' => \$dbname, +) or pod2usage( 2 ); + +pod2usage( + -msg => "Required options: --dbname", + -exitval => 3, + ) unless defined $dbname; + +my $mdcParser = PS::IPP::Metadata::Config->new; # Metadata config parser + +# Look for programs we need +my $missing_tools; +my $p0tool = can_run('p0tool') or (warn "Can't find p0tool" and $missing_tools = 1); +my $phase0_imfile = can_run('phase0_imfile.pl') or (warn "Can't find phase0_imfile.pl" and $missing_tools = 1); +my $phase0_exp = can_run('phase0_exp.pl') or (warn "Can't find phase0_exp.pl" and $missing_tools = 1); +die "Can't find required tools.\n" if $missing_tools; + +# Phase 0 imfile processing +my $list; +{ + my $command = "$p0tool -pendingimfile -dbname $dbname"; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get phase 0 imfile list: $error_code\n" if not $success; + $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from p0tool.\n"; +} + +foreach my $item (@$list) { + my $exp_tag = $item->{exp_tag}; + my $class = $item->{class}; + my $class_id = $item->{class_id}; + my $uri = $item->{uri}; + + my $command = "$phase0_imfile --exp_tag $exp_tag --class $class --class_id $class_id --uri $uri --dbname $dbname"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do phase 0 imfile processing on $exp_tag $class_id: $error_code\n" if not $success; +} + +# Phase 0 exposure processing +{ + my $command = "$p0tool -pendingexp -dbname $dbname"; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to get phase 0 exposure list: $error_code\n" if not $success; + $list = parse_md_list( $mdcParser->parse( join( '', @$stdout_buf ) ) ) or + die "Unable to parse output from p0tool.\n"; +} + +foreach my $item (@$list) { + my $exp_tag = $item->{exp_tag}; + + my $command = "$phase0_exp --exp_tag $exp_tag --dbname $dbname"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run( command => $command, verbose => 1 ); + die "Unable to do phase 0 exposure processing on $exp_tag: $error_code\n" if not $success; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + + +__END__ + + Index: /branches/rel-1-0/ippScripts/scripts/mdc2list.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/mdc2list.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/mdc2list.pl (revision 11645) @@ -0,0 +1,77 @@ +#!/usr/bin/env perl + +# Program to convert a PS metadata config syntax (usually from one of +# the IPP tools) into a flat tab-delimited list suitable for parsing +# using "split". + +# Copyright (C) 2006 Joshua Hoblitt, Paul A. Price. + +use strict; +use warnings; + +use PS::IPP::Metadata::Config; # Supplies the metadata config parser. + +die "Program to convert a PS metadata config file (usually from one of the IPP\n" . + "tools) into a flat tab-delimited list suitable for parsing using \'split\'.\n\n" . + "Usage: $0 [INFILE [OUTFILE]]\n" if ((join "", @ARGV) =~ /--help/ or scalar @ARGV > 2); + +my $inFile; # Input file +my $outFile; # Output file +if (scalar @ARGV >= 1) { + my $inName = shift @ARGV; + open $inFile, $inName or die "Can't open $inName: $!\n"; + if (scalar @ARGV == 1) { + my $outName = shift @ARGV; + open $outFile, ">", $outName or die "Can't open $outName: $!\n"; + } else { + $outFile = *STDOUT; + } +} else { + $inFile = *STDIN; + $outFile = *STDOUT; +} + +my @input = <$inFile>; # Contents of the metadata config file +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files +my $md = $mdcParser->parse(join "", @input) + or die "unable to parse metadata config doc"; +my $hashes = mds2hashes($md); # An array of hashes +foreach my $pending (@$hashes) { + foreach my $key (keys %$pending) { + print $outFile ( $pending->{$key} . "\t"); + } + print $outFile "\n"; +} + +### Pau. + + +# Given an array of MDs, return an array of hashes +sub mds2hashes +{ + my $mds = shift; # Reference to the metadatas + my @array; # The array of hashes, to be returned + foreach my $md (@$mds) { + my $values = md2hash($md->{value}); + push @array, $values; + } + return \@array; +} + +# Convert the metadata to a hash; in effect, strips out the comment, type and class fields. +sub md2hash +{ + my $values = shift; # Reference to the metadata + my %hash; # Hash, to be returned + foreach my $data (@$values) { + $hash{$data->{name}} = $data->{value}; + } + return \%hash; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} Index: /branches/rel-1-0/ippScripts/scripts/phase0_exp.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/phase0_exp.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/phase0_exp.pl (revision 11645) @@ -0,0 +1,267 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use Cache::File; +use Storable qw( freeze thaw ); +use File::Basename qw( basename ); +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Statistics::Descriptive; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); + +my ($cache, $exptag, $workdir, $dbname, $no_update); + +GetOptions( + 'caches' => \$cache, + 'exp_tag|e=s' => \$exptag, + 'workdir|w=s' => \$workdir, # Working directory for output files + 'dbname|d=s' => \$dbname, # Database name + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --exp_tag", + -exitval => 3, +) unless defined $exptag; + +# Define setup +use constant TYPE => "exp_type"; # Observation type keyword +use constant DETRENDS => [ "bias", "zero", "dark", "shutter", "flat", "fringe", "domeflat", "skyflat" ]; # Observation types to mark as detrend +use constant DETREND_FLAG => "-detrend"; # Flag to use to mark detrend exposure + +use constant PHASE0_URI => 'uri'; # Key for the URI +use constant PHASE0_CLASSID => 'class_id'; # Key for the class id +use constant PHASE0_BG => 'bg'; # Key for the background +use constant PHASE0_BG_STDEV => 'bg_stdev'; # Key for the background standard deviation +use constant PHASE0_BG_MEAN_STDEV => 'bg_mean_stdev'; # Key for the mean of the background standard deviation + +# These values should be constant for all components +use constant CONSTANTS => [ + "object", # Object + "exp_type", # Exposure type + "filter", # Filter used + "airmass", # Airmass + "ra", # Right ascension + "decl", # Declination + "posang", # Position angle + "alt", # Altitude + "az", # Azimuth + "ccd_temp", # CCD temperature + "dateobs", # Time of exposure + ]; + +# These values may vary across components; we will take the average +use constant VARIABLES => [ + "exp_time", # Exposure time + ]; + + +# Look for commands we need +my $missing_tools; +my $p0tool = can_run('p0tool') + or (warn "can't find p0tool" and $missing_tools = 1); +my $ppStats = can_run('ppStats') + or (warn "can't find ppStats" and $missing_tools = 1); + +if ($missing_tools) { + warn ("Can't find required tools"); + exit($PS_EXIT_CONFIG_ERROR); +} + +# setup cache interface + +my $c = Cache::File->new( + cache_root => File::Spec->catdir($ENV{'HOME'}, '.pxtools', basename($0)), + default_expires => '7200 sec', +); + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get the list of imfiles +my $imfiles; +{ + my $command = "$p0tool -processedimfile -exp_tag $exptag"; + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + cache_run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn ("Unable to perform p0tool on exposure id $exptag: $error_code"); + exit ($error_code); + } + + my $metadata = $mdcParser->parse(join "", @$stdout_buf); + unless ($metadata) { + warn ("Unable to parse metadata config doc"); + &my_die ($exptag, $PS_EXIT_PROG_ERROR); + } + $imfiles = parse_md_list($metadata); # Data for imfiles +} + +# Process the values +my %values; # Values to update +foreach my $variable (@{VARIABLES()}) { + $values{$variable} = []; +} +my @backgrounds; # Array of backgrounds +my @variances; # Array of background standard deviations\ +my $obsType; # Observation type + +foreach my $imfile (@$imfiles) { + foreach my $constant (@{CONSTANTS()}) { + my $value = get_value($imfile, $constant); # Value for imfile + if (not defined $values{$constant}) { + $values{$constant} = $value; + } + if ($values{$constant} ne $value) { + warn ("Value of $constant for $imfile->{PHASE0_CLASSID} doesn't match previous value"); + &my_die ($exptag, $PS_EXIT_PROG_ERROR); + } + } + + foreach my $variable (@{VARIABLES()}) { + my $value = get_value($imfile, $variable); # Value for imfile + my $array = $values{$variable}; # Array of data + push @$array, $value; + } + + my $bg = get_value($imfile, PHASE0_BG()); + push @backgrounds, $bg; + + my $stdev = get_value($imfile, PHASE0_BG_MEAN_STDEV()); + push @variances, $stdev**2; +} + +if (0) { +# XXX for a test, randomly declare a failure and return to pantasks +my $rnd = rand(1); +if ($rnd > 0.5) { + warn ("random failure"); + &my_die ($exptag, $PS_EXIT_DATA_ERROR); +} +} + +# Output results to the database +unless ($no_update) { + my @command; + push @command, $p0tool, '-addprocessedexp', '-exp_tag', $exptag; # Command to execute to update exposure parameters + + # Add the values of interest + foreach my $constant (@{CONSTANTS()}) { + push @command, ( '-' . $constant ), $values{$constant}; + } + foreach my $variable (@{VARIABLES()}) { + my $array = $values{$variable}; # Array of values + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@$array); + push @command, ( '-' . $variable ), $stats->mean(); + } + + # Add the statistics + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@backgrounds); + push @command, ( '-' . PHASE0_BG() ), $stats->mean(); + push @command, ( '-' . PHASE0_BG_MEAN_STDEV() ), ( $stats->standard_deviation() || 0 ) + } + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@variances); + push @command, ( '-' . PHASE0_BG_STDEV() ), sqrt( $stats->mean() ); + } + + # Add the detrend flag + foreach my $detrendType (@{DETRENDS()}) { + if (lc($values{TYPE()}) =~ /$detrendType/) { + push @command, DETREND_FLAG; + last; + } + } + + # Quote arguments with whitespace + for (my $i = 0; $i < scalar @command; $i++) { + if ($command[$i] =~ /\s/) { + $command[$i] = "\'$command[$i]\'"; + } + } + + push @command, "-dbname", $dbname if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + cache_run(command => \@command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn ("Unable to run phase0 update for $exptag: $error_code"); + exit($error_code); + } +} + +### Pau. + +# Get the value for a particular imfile, along with a strong check for its existence +sub get_value { + my $imfile = shift; # The hash of values + my $name = shift; # The name of the value to check + + my $source = $imfile->{PHASE0_CLASSID()}; # Where it comes from + + unless (defined $imfile->{$name}) { + warn ("Couldn't find value of $name for class_id=$source"); + &my_die ($exptag, $PS_EXIT_PROG_ERROR); + } + return $imfile->{$name}; +} + +sub cache_run +{ + my %p = @_; + + my $cmd_output = $c->get($p{command}) if $cache; + if (defined $cmd_output) { + return @{thaw $cmd_output}; + } else { + my @output = run(%p); + $c->set($p{command}, freeze \@output) if $cache; + return @output; + } +} + +sub my_die +{ + my $exp_tag = $_[0]; + my $exit_code = $_[1]; + if ($exp_tag) { + my $command = "$p0tool -addprocessedexp -exp_tag $exp_tag -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system($command); + } + exit $exit_code; +} + +END { + my $exit = $?; + system("sync") == 0 or die "failed to execute sync: $!"; + $? = $exit; +} Index: /branches/rel-1-0/ippScripts/scripts/phase0_imfile.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/phase0_imfile.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/phase0_imfile.pl (revision 11645) @@ -0,0 +1,250 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use Cache::File; +use Storable qw(freeze thaw); +use File::Basename qw( basename); +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::Stats; +use Data::Dumper; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); + +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($cache, $exp_tag, $class_id, $uri, $workdir, $dbname, $no_update); + +# print "args: @ARGV\n"; +# for (my $i = 0; $i < @ARGV; $i++) { +# print "arg $i: ...$ARGV[$i]...\n"; +# } + +GetOptions( + 'caches' => \$cache, + 'exp_tag|e=s' => \$exp_tag, + 'class_id|i=s' => \$class_id, + 'uri|u=s' => \$uri, + 'workdir|w=s' => \$workdir, # Working directory for output files + 'dbname|d=s' => \$dbname,# Database name + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --exp_tag --class_id --uri", + -exitval => 3, +) unless defined $exp_tag + and defined $class_id + and defined $uri; + +my $RECIPE = "PPSTATS_PHASE0"; # Recipe to use for ppStats +# use constant RECIPE => "PPSTATS_PHASE0"; # Recipe to use for ppStats + +# These values should be constant for all components +# The key is the name in the ppStats output; the value is the switch for "phase0 -updateexp" +use constant CONSTANTS => { + "FPA.OBJECT" => "-object", # Object + "FPA.OBSTYPE" => "-exp_type", # Exposure type + "FPA.FILTER" => "-filter", # Filter used + "FPA.AIRMASS" => "-airmass", # Airmass + "FPA.RA" => "-ra", # Right ascension + "FPA.DEC" => "-decl", # Declination + "FPA.ALT" => "-alt", # Altitude + "FPA.AZ" => "-az", # Azimuth + "FPA.POSANGLE" => "-posang", # Position angle + "FPA.TIME" => "-dateobs", # Date of observation (UTC) + }; + +# These values may vary across components; we will take the average +# The key is the name in the ppStats output; the value is the switch for "phase0 -updateexp" +use constant VARIABLES => { + "CHIP.TEMP" => "-ccd_temp", # CCD temperature + "CELL.EXPOSURE" => "-exp_time" # Exposure time + }; + + +# Switches for p0tool +use constant P0TOOL_MODE => '-addprocessedimfile'; # Mode for p0tool +use constant P0TOOL_EXPTAG => '-exp_tag'; # Switch to specify the exposure id +use constant P0TOOL_CLASSID => '-class_id'; # Switch to specify the class id +use constant P0TOOL_BG_MEAN => '-bg'; # Switch to add the background +use constant P0TOOL_BG_STDEV => '-bg_stdev'; # Switch to add the bg stdev +use constant P0TOOL_BG_MEAN_STDEV => '-bg_mean_stdev'; # Switch to add the bg mean stdev + +# Look for programs we need +my $missing_tools; +my $p0tool = can_run('p0tool') + or (warn "Can't find p0tool" and $missing_tools = 1); +my $ppStats = can_run('ppStats') + or (warn "Can't find ppStats" and $missing_tools = 1); + +if ($missing_tools) { + warn ("Can't find required tools"); + exit($PS_EXIT_CONFIG_ERROR); +} + +# setup cache interface + +my $c = Cache::File->new( + cache_root => File::Spec->catdir($ENV{'HOME'}, '.pxtools', basename($0)), + default_expires => '7200 sec', +); + +# Resolve the input URI +$uri = $ipprc->convert_filename_absolute($uri); + +# Run ppStats on the input file +my $stats; +{ + my $command = "$ppStats $uri -recipe PPSTATS $RECIPE"; # Command to run ppStats + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + cache_run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn ("Unable to perform ppStats on exposure id $exp_tag: $error_code"); + &my_die ($exp_tag, $class_id, $error_code); + } + + # Parse the output + my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + my $metadata = $mdcParser->parse(join "", @$stdout_buf); # XXX is this join necessary? + unless ($metadata) { + warn ("Unable to parse metadata config doc"); + &my_die ($exp_tag, $class_id, $PS_EXIT_PROG_ERROR); + } + my @constants = keys %{CONSTANTS()}; # List of constants to parse out + my @variables = keys %{VARIABLES()}; # List of variables to parse out + $stats = PS::IPP::Metadata::Stats->new(\@constants, \@variables); # Stats parser + unless ($stats->parse($metadata)) { + warn ("Unable to find all values"); + &my_die ($exp_tag, $class_id, $PS_EXIT_PROG_ERROR); + # XXX is this a programming or a config error? + } + + if (0) { + # XXX for a test, randomly declare a failure and return to pantasks + my $rnd = rand(1); + if ($rnd > 0.5) { + warn ("random failure"); + &my_die ($exp_tag, $class_id, $PS_EXIT_DATA_ERROR); + } +} +} + +# Push the results into the database +unless ($no_update) { + my @command; + push @command, $p0tool, P0TOOL_MODE(), P0TOOL_EXPTAG(), $exp_tag, P0TOOL_CLASSID(), $class_id; # Command to run p0tool + + foreach my $constant (keys %{CONSTANTS()}) { + push @command, CONSTANTS->{$constant}, ($stats->data($constant))->{value}; + + } + foreach my $variable (keys %{VARIABLES()}) { + # Just use the mean value + push @command, VARIABLES->{$variable}, ($stats->data($variable))->{mean}; + } + + push @command, P0TOOL_BG_MEAN(); + if (defined $stats->bg_mean()) { + push @command, $stats->bg_mean(); + } else { + push @command, "NAN"; + } + push @command, P0TOOL_BG_STDEV(); + if (defined($stats->bg_stdev())) { + push @command, $stats->bg_stdev(); + } else { + # Will not be defined if there is only a single imfile + push @command, 0; + } + push @command, P0TOOL_BG_MEAN_STDEV(); + if (defined $stats->bg_mean_stdev()) { + push @command, $stats->bg_mean_stdev(); + } else { + push @command, "NAN"; + } + + # Quote arguments with whitespace + for (my $i = 0; $i < scalar @command; $i++) { + if ($command[$i] =~ /\s/) { + $command[$i] = "\'$command[$i]\'"; + } + } + + push @command, "-dbname", $dbname if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => \@command, verbose => 1); + + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + # XXX if we can't run -addprocessedimfile, we can't actually set the + # error code. + # XXX if this is not a database error, it is probably a + # programming error (in p0tool or the passed args) + warn ("Unable to perform p0tool -addprocessedimfile: $error_code"); + exit($error_code); + } +} + +sub cache_run +{ + my %p = @_; + + my $cmd_output = $c->get($p{command}) if $cache; + if (defined $cmd_output) { + return @{thaw $cmd_output}; + } else { + my @output = run(%p); + $c->set($p{command}, freeze \@output) if $cache; + return @output; + } +} + +sub my_die +{ + my $exp_tag = $_[0]; + my $class_id = $_[1]; + my $exit_code = $_[2]; + if ($exp_tag && $class_id) { + my $command = "$p0tool -addprocessedimfile -exp_tag $exp_tag -class_id $class_id -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +# Pau. + +END { + my $exit = $?; + system("sync") == 0 or die "failed to execute sync: $!"; + $? = $exit; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/phase2.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/phase2.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/phase2.pl (revision 11645) @@ -0,0 +1,182 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::Stats; +use Data::Dumper; +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +use constant RECIPE => 'PPIMAGE_OBDSFRA'; # Recipe to use + +# Parse the command-line arguments +my ($exp_tag, # Exposure tag + $class_id, # Class Id + $input, # Input FITS file + $camera, # Camera + $dbname, # Database name + $workdir, # Working directory, for output files + $no_update # Don't update the database? + ); +GetOptions( + 'exp_tag|e=s' => \$exp_tag, + 'class_id|i=s' => \$class_id, + 'uri|u=s' => \$input, + 'camera|c=s' => \$camera, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --exp_tag --class_id --uri --camera --workdir", + -exitval => 3, +) unless defined $exp_tag + and defined $class_id + and defined $input + and defined $camera; + +$ipprc->define_camera($camera); + +# Look for programs we need +my $missing_tools; +my $p2tool = can_run('p2tool') or (warn "Can't find p2tool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} +$ppImage .= " -dbname $dbname" if defined $dbname; + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my ($vol, $dir, $file) = File::Spec->splitpath( $input ); + $workdir = $dir; +} + +### Output file name --- must match camera configuration! +my $outputFile = "$exp_tag.p2"; +my $outputRoot = File::Spec->catfile( $workdir, $exp_tag, $outputFile ); +my $outputImage = $ipprc->filename("PPIMAGE.OUTPUT", $outputRoot, $class_id); +my $outputBin1 = $ipprc->filename("PPIMAGE.BIN1", $outputRoot, $class_id); +my $outputBin2 = $ipprc->filename("PPIMAGE.BIN2", $outputRoot, $class_id); +my $outputStats = $outputRoot . '.' . $class_id . '.stats'; + +$input = $ipprc->convert_filename_absolute( $input ); + +# Run ppImage +{ + print "outputImage: $outputImage\n"; + print "outputBin1: $outputBin1\n"; + print "outputStats: $outputStats\n"; + + my $command = "$ppImage -file $input $outputRoot -recipe PPIMAGE " . RECIPE . + " -stat $outputStats"; # Command to run ppImage + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $exp_tag, $class_id, $error_code); + } + &my_die("Couldn't find expected output file: $outputImage\n", $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputImage; + &my_die("Couldn't find expected output file: $outputBin1\n", $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputBin1; + &my_die("Couldn't find expected output file: $outputBin2\n", $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputBin2; + &my_die("Couldn't find expected output file: $outputStats\n", $exp_tag, $class_id, $PS_EXIT_SYS_ERROR) unless -f $outputStats; +} + +# Get the statistics on the processed image +my $stats; # Statistics from ppImage +{ + my $statsFile; # File handle + open $statsFile, $outputStats or &my_die("Can't open statistics file $outputStats: $!", $exp_tag, $class_id, $PS_EXIT_SYS_ERROR); + my @contents = <$statsFile>; # Contents of file + close $statsFile; + my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + my $metadata = $mdcParser->parse(join "", @contents) or + &my_die("Unable to parse metadata config doc", $exp_tag, $class_id, $PS_EXIT_PROG_ERROR); + $stats = PS::IPP::Metadata::Stats->new(); # Stats parser + $stats->parse($metadata) or + &my_die("Unable to find all values in statistics output.\n", $exp_tag, $class_id, $PS_EXIT_PROG_ERROR); +} + +# Add the processed file to the database +$outputImage = $ipprc->convert_filename_relative( $outputImage); +$outputBin1 = $ipprc->convert_filename_relative( $outputBin1 ); +$outputBin2 = $ipprc->convert_filename_relative( $outputBin2 ); +unless ($no_update) { + # Command to run p2tool + my $command = "$p2tool -addprocessedimfile"; + $command .= " -exp_tag $exp_tag"; + $command .= " -class_id $class_id"; + $command .= " -recip " . RECIPE; + $command .= " -uri $outputImage"; + $command .= " -b1_uri $outputRoot"; + $command .= " -bg " . $stats->bg_mean(); + $command .= " -bg_stdev " . $stats->bg_stdev(); + $command .= " -bg_mean_stdev " . $stats->bg_mean_stdev(); + $command .= " -dbname $dbname" if defined $dbname; + + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform p2tool -addprocessedimfile: $error_code\n"); + exit($error_code); + } + + unlink $outputStats; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $exp_tag = shift; # Exposure tag + my $class_id = shift; # Class identifier + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($exp_tag and $class_id) { + my $command = "$p2tool -addprocessedimfile"; + $command .= " -exp_tag $exp_tag"; + $command .= " -class_id $class_id"; + $command .= " -code $exit_code"; + $command .= " -uri $outputImage"; + $command .= " -recip " . RECIPE; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/scripts/phase3.pl =================================================================== --- /branches/rel-1-0/ippScripts/scripts/phase3.pl (revision 11645) +++ /branches/rel-1-0/ippScripts/scripts/phase3.pl (revision 11645) @@ -0,0 +1,270 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +## report the program and machine +use Sys::Hostname; +my $host = hostname(); +print "\n\n"; +print "Starting script $0 on $host\n\n"; + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use IPC::Cmd qw( can_run run ); +use PS::IPP::Metadata::Config; +use PS::IPP::Metadata::List qw( parse_md_list ); +use Statistics::Descriptive; + +use PS::IPP::Config qw( + $PS_EXIT_SUCCESS + $PS_EXIT_UNKNOWN_ERROR + $PS_EXIT_SYS_ERROR + $PS_EXIT_CONFIG_ERROR + $PS_EXIT_PROG_ERROR + $PS_EXIT_DATA_ERROR + $PS_EXIT_TIMEOUT_ERROR + ); +my $ipprc = PS::IPP::Config->new(); # IPP configuration +use File::Spec; + +use Getopt::Long qw( GetOptions :config auto_help auto_version gnu_getopt ); +use Pod::Usage qw( pod2usage ); + +my ($exp_tag, $camera, $dbname, $workdir, $no_update); +GetOptions( + 'exp_tag|e=s' => \$exp_tag, + 'camera|c=s' => \$camera, + 'dbname|d=s' => \$dbname, # Database name + 'workdir|w=s' => \$workdir, + 'no-update' => \$no_update +) or pod2usage( 2 ); + +pod2usage( -msg => "Unknown option: @ARGV", -exitval => 2 ) if @ARGV; +pod2usage( + -msg => "Required options: --exp_tag --camera --workdir", + -exitval => 3, +) unless defined $exp_tag + and defined $camera; + +$ipprc->define_camera($camera); + +use constant RECIPE1 => 'PPIMAGE_J1'; # Recipe to use for ppImage to make JPEGs +use constant RECIPE2 => 'PPIMAGE_J2'; # Recipe to use for ppImage to make JPEGs + + +# Look for programs we need +my $missing_tools; +my $p3tool = can_run('p3tool') or (warn "Can't find p3tool" and $missing_tools = 1); +my $ppImage = can_run('ppImage') or (warn "Can't find ppImage" and $missing_tools = 1); + +# test for addstar and psastro: +my $psastro = can_run('psastro') or (warn "Can't find psastro" and $missing_tools = 1); +my $addstar = can_run('addstar') or (warn "Can't find addstar" and $missing_tools = 1); + +if ($missing_tools) { + warn("Can't find required tools."); + exit($PS_EXIT_CONFIG_ERROR); +} + +my $mdcParser = PS::IPP::Metadata::Config->new; # Parser for metadata config files + +# Get list of component files +my $files; # Array of component files +{ + my $command = "$p3tool -pendingimfile -exp_tag $exp_tag"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform p3tool: $error_code", $exp_tag, $error_code); + } + my $metadata = $mdcParser->parse(join "", @$stdout_buf) or + &my_die("Unable to parse metadata config doc", $exp_tag, $PS_EXIT_PROG_ERROR); + $files = parse_md_list($metadata) or + &my_die("Unable to parse metadata list", $exp_tag, $PS_EXIT_PROG_ERROR); +} + +# Gather the statistics +my ($bg, $bg_stdev, $bg_mean_stdev); # The statistics triplet +{ + my @backgrounds; # Array of backgrounds in each component + my @variances; # Array of variances for each component +# my @ra; # Array of ra errors +# my @dec; # Array of dec errors +# my @zp; # Array of photometric zero points + foreach my $file (@$files) { + &my_die("Unable to find class id", $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{class_id}; + my $class_id = $file->{class_id}; + &my_die("Unable to find bg for class_id=$class_id", $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{bg}; + &my_die("Unable to find bg_mean_stdev for class_id=$class_id", $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{bg_mean_stdev}; +# &my_die("Unable to find sigma_ra for class_id=$class_id", $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{sigma_ra}; +# &my_die("Unable to find sigma_dec for class_id=$class_id", $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{sigma_dec}; +# &my_die("Unable to find zp for class_id=$class_id", $exp_tag, $PS_EXIT_SYS_ERROR) unless defined $file->{zp}; + push @backgrounds, $file->{bg}; + push @variances, $file->{bg_stdev}**2; + } + + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@backgrounds); + $bg = $stats->mean(); + $bg_mean_stdev = $stats->standard_deviation() || 0.0; + } + { + my $stats = Statistics::Descriptive::Sparse->new; # Statistics calculator + $stats->add_data(@variances); + $bg_stdev = sqrt( $stats->mean() ); + } +} + +if (defined $workdir) { + $workdir = $ipprc->convert_filename_absolute( $workdir ); +} else { + my $example = ${$files}[0]->{b1_uri}; # Example original root name + my ($vol, $dir, $file) = File::Spec->splitpath( $example ); + $workdir = $dir; +} + +# make this a function which generates a specific list file for a specific filename + +# Generate the file list, and get the statistics +my $outputFile = "$exp_tag.p3"; +my $outputRoot = File::Spec->catfile( $workdir, $exp_tag, $outputFile ); + +my $list1Name = $outputRoot . '.b1.list'; # Name for the input file list for binning 1 +my $list2Name = $outputRoot . '.b2.list'; # Name for the input file list for binning 2 +my $list3Name = $outputRoot . '.b3.list'; # Name for the input file list for chip astrometry + +my @means; # Array of means +my @stdevs; # Array of stdevs +open my $list1File, '>' . $list1Name; +open my $list2File, '>' . $list2Name; +open my $list3File, '>' . $list3Name; + +my $chipObjects; +foreach my $file (@$files) { + # use the b1_uri as OUTPUT root and convert the filenames + # with ipprc->filename: + my $class_id = $file->{class_id}; + my $origRoot = $ipprc->convert_filename_absolute( $file->{b1_uri} ); # Original root name + + # if there is only one chip, we use this name for the input to addstar + $chipObjects = $ipprc->filename("PSASTRO.OUTPUT", $origRoot, $class_id); + print $list1File $ipprc->filename("PPIMAGE.BIN1", $origRoot, $class_id); + print $list2File $ipprc->filename("PPIMAGE.BIN2", $origRoot, $class_id); + print $list3File $chipObjects; + push @means, $file->{bg}; + push @stdevs, $file->{bg_stdev}; +} +close $list1File; +close $list2File; +close $list3File; + +# Output products --- need to synch with the camera configuration! +my $jpeg1 = $ipprc->filename("PPIMAGE.JPEG1", $outputRoot); # Binned JPEG #1 +my $jpeg2 = $ipprc->filename("PPIMAGE.JPEG2", $outputRoot); # Binned JPEG #2 +my $fpaObjects = $ipprc->filename("PSASTRO.OUTPUT", $outputRoot); # MEF psastro output + +# run psastro +mosastro on the set of chips +# XXX note that this is wrong if imfiles are cells +if (@$files > 1) { + my $command = "$psastro -list $list3Name $outputRoot +mosastro -chipastro"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform psastro: $error_code", $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $fpaObjects", $exp_tag, $PS_EXIT_PROG_ERROR) unless -f $fpaObjects; +} else { + $fpaObjects = $chipObjects; +} + +# run addstar on either the single chip output or the single fpa output +# XXX this construct requires the user to have a valid .ptolemyrc +# XXX which in turn points at ippconfig/dvo.site +{ + my $command = "addstar -D CAMERA $camera $fpaObjects"; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform addstar: $error_code", $exp_tag, $error_code); + } +} + +# Make the jpeg for binning 1 +{ + my $command = "$ppImage -list $list1Name $outputRoot -recipe PPIMAGE " . RECIPE1; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $jpeg1", $exp_tag, $PS_EXIT_PROG_ERROR) unless -f $jpeg1; +} + +# Make the jpeg for binning 2 +{ + my $command = "$ppImage -list $list2Name $outputRoot -recipe PPIMAGE " . RECIPE2; # Command to run + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + &my_die("Unable to perform ppImage: $error_code", $exp_tag, $error_code); + } + &my_die("Unable to find expected output file: $jpeg2", $exp_tag, $PS_EXIT_PROG_ERROR) unless -f $jpeg2; +} + +# XXX keep the same outroot as the input +# Add the result into the database +$outputRoot = $ipprc->convert_filename_relative($outputRoot); +$jpeg1 = $ipprc->convert_filename_relative($jpeg1); +$jpeg2 = $ipprc->convert_filename_relative($jpeg2); +unless ($no_update) { + my $command = "$p3tool -addprocessedexp -exp_tag $exp_tag -uri UNKNOWN " . + "-recip " . RECIPE1() . "," . RECIPE2() . " -b1_uri $outputRoot " . + "-bg $bg -bg_stdev $bg_stdev -bg_mean_stdev $bg_mean_stdev " . + "-sigma_ra 0.0 -sigma_dec 0.0 -nastro 0 -zp_mean 0.0 -zp_stdev 0.0"; # Command to run + $command .= " -dbname $dbname" if defined $dbname; + my ( $success, $error_code, $full_buf, $stdout_buf, $stderr_buf ) = + run(command => $command, verbose => 1); + unless ($success) { + $error_code = (($error_code >> 8) or $PS_EXIT_PROG_ERROR); + warn("Unable to perform dettool -addprocessedexp: $error_code\n"); + exit($error_code); + } + + unlink $list1Name; + unlink $list2Name; +} + + +sub my_die +{ + my $msg = shift; # Warning message on die + my $exp_tag = shift; # Exposure tag + my $exit_code = shift; # Exit code to add + + warn($msg); + if ($exp_tag) { + my $command = "$p3tool -addprocessedexp -exp_tag $exp_tag -code $exit_code"; + $command .= " -dbname $dbname" if defined $dbname; + system ($command); + } + exit $exit_code; +} + + +END { + my $status = $?; + system("sync") == 0 + or die "failed to execute sync: $!" ; + $? = $status; +} + +__END__ Index: /branches/rel-1-0/ippScripts/t/00_distribution.t =================================================================== --- /branches/rel-1-0/ippScripts/t/00_distribution.t (revision 11645) +++ /branches/rel-1-0/ippScripts/t/00_distribution.t (revision 11645) @@ -0,0 +1,25 @@ +#!/usr/bin/env perl + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: 00_distribution.t,v 1.1 2007-01-30 01:52:16 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use lib qw( ./lib ./t ); + +use Test::More; + +# example taken from Test::Distribution Pod + +BEGIN { + eval { + require Test::Distribution; + }; + if($@) { + plan skip_all => 'Test::Distribution not installed'; + } else { + import Test::Distribution; # not => qw( podcover ); + } +} Index: /branches/rel-1-0/ippTasks/.cvsignore =================================================================== --- /branches/rel-1-0/ippTasks/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippTasks/.cvsignore (revision 11645) @@ -0,0 +1,9 @@ +Makefile +Makefile.in +aclocal.m4 +autom4te.cache +config.log +config.status +configure +install-sh +missing Index: /branches/rel-1-0/ippTasks/Makefile.am =================================================================== --- /branches/rel-1-0/ippTasks/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippTasks/Makefile.am (revision 11645) @@ -0,0 +1,25 @@ +task_files = \ + detrend.mkruns.pro \ + detrend.norm.pro \ + detrend.process.pro \ + detrend.reject.pro \ + detrend.resid.pro \ + detrend.stack.pro \ + mkdetrend.pro \ + pantasks.pro \ + phase0.pro \ + phase2.pro \ + phase3.pro \ + summit.copy.pro \ + testscript.pro + +pantasksdir = $(datadir)/pantasks +ipptasksdir = $(pantasksdir)/modules +ipptasks_DATA = $(task_files) + +install-data-hook: + chmod 0755 $(panstasks) $(ipptasksdir) + +EXTRA_DIST = $(task_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippTasks/autogen.sh =================================================================== --- /branches/rel-1-0/ippTasks/autogen.sh (revision 11645) +++ /branches/rel-1-0/ippTasks/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=ippTasks +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOlIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +#($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $AUTOHEADER installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" +# DIE=1 +#} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +#$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +#$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ippTasks/configure.ac =================================================================== --- /branches/rel-1-0/ippTasks/configure.ac (revision 11645) +++ /branches/rel-1-0/ippTasks/configure.ac (revision 11645) @@ -0,0 +1,14 @@ +AC_PREREQ(2.59) + +AC_INIT([ippTasks], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([phase0.pro]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_MAINTAINER_MODE + +AC_PROG_INSTALL + +AC_CONFIG_FILES([ + Makefile +]) +AC_OUTPUT Index: /branches/rel-1-0/ippTasks/detrend.mkruns.pro =================================================================== --- /branches/rel-1-0/ippTasks/detrend.mkruns.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/detrend.mkruns.pro (revision 11645) @@ -0,0 +1,97 @@ +## XXX this script needs to be updated to use books, not queues +## XXX this script needs to be reconsidered... + +## This script defines tasks which regularly initiate the detrend +## runs. Each detrend type is given its own task, which are run once +## (or twice) per day to build the relevant detrend data + +## when one of these tasks is run, it creates a new detrend run ID in +## the database, and selects the exposures for that run on the basis +## of the specified filters + +# create a new detrend bias-creation run +task mkdetruns.megacam + # command dettool -select -time NOW -24h -type bias -camera megacam + command mkdetruns.megacam + host local + + periods -poll 1 + periods -exec 60 + periods -timeout 30 + trange 18:00 19:00 -nmax 1 + + # success: a new detrend run was created + task.exit 0 + echo "new MEGACAM detrend runs" + queueprint stdout + end + + # default exit status + task.exit default + echo "unable to define a new detrend run" + end + + # operation timed out? is the database down? + task.exit timeout + # send someone email? + echo "unable to define a new detrend run" + end +end + +# create a new detrend bias-creation run +task mkdetruns.isp + # command dettool -select -time NOW -24h -type bias -camera megacam + command mkdetruns.isp + host local + + periods -poll 1 + periods -exec 60 + periods -timeout 30 + trange 18:00 19:00 -nmax 1 + + # success: a new detrend run was created + task.exit 0 + echo "new ISP detrend runs" + queueprint stdout + end + + # default exit status + task.exit default + echo "unable to define a new detrend run" + end + + # operation timed out? is the database down? + task.exit timeout + # send someone email? + echo "unable to define a new detrend run" + end +end + +# create a new detrend bias-creation run +task mkdetruns.gpc + # command dettool -select -time NOW -24h -type bias -camera megacam + command mkdetruns.gpc + host local + + periods -poll 1 + periods -exec 60 + periods -timeout 30 + trange 18:00 19:00 -nmax 1 + + # success: a new detrend run was created + task.exit 0 + echo "new GPC detrend runs" + queueprint stdout + end + + # default exit status + task.exit default + echo "unable to define a new detrend run" + end + + # operation timed out? is the database down? + task.exit timeout + # send someone email? + echo "unable to define a new detrend run" + end +end Index: /branches/rel-1-0/ippTasks/detrend.norm.pro =================================================================== --- /branches/rel-1-0/ippTasks/detrend.norm.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/detrend.norm.pro (revision 11645) @@ -0,0 +1,420 @@ +## this file contains the tasks for running the detrend normalization stages +## these tasks use the books detPendingNormStatImfile detPendingNormImfile detPendingNormExp + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/detrend +exec mkdir -p $LOGSUBDIR + +book init detPendingNormStatImfile +book init detPendingNormImfile +book init detPendingNormExp + +macro detnorm.reset + book init detPendingNormStatImfile + book init detPendingNormImfile + book init detPendingNormExp +end + +macro detnorm.status + book listbook detPendingNormStatImfile + book listbook detPendingNormImfile + book listbook detPendingNormExp +end + +macro detnorm.on + task dettool.norm.load + active true + end + task dettool.norm.run + active true + end + task dettool.normexp.load + active true + end + task dettool.normexp.run + active true + end + task dettool.normstat.load + active true + end + task dettool.normstat.run + active true + end +end + +macro detnorm.off + task dettool.norm.load + active false + end + task dettool.norm.run + active false + end + task dettool.normexp.load + active false + end + task dettool.normexp.run + active false + end + task dettool.normstat.load + active false + end + task dettool.normstat.run + active false + end +end + +# these variables will cycle through the known database names +$detPendingNormStatImfile_DB = 0 +$detPendingNormImfile_DB = 0 +$detPendingNormExp_DB = 0 + +# select images ready for copy +# new entries are added to detPendingNormStatImfile +# compare the new list with the ones already selected +task dettool.normstat.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/dettool.normstat.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -tonormalizedstat -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingNormStatImfile_DB + command dettool -tonormalizedstat -limit 20 -dbname $DB:$detPendingNormStatImfile_DB + $detPendingNormStatImfile_DB ++ + if ($detPendingNormStatImfile_DB >= $DB:n) set detPendingNormStatImfile_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingNormStatImfile -key det_id:iteration -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingNormStatImfile + end + + # delete existing entries in the appropriate states + process_cleanup detPendingNormStatImfile + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.normstat.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingNormStatImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingNormStatImfile + book getpage detPendingNormStatImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingNormStatImfile $pageName state RUN + book getword detPendingNormStatImfile $pageName det_id -var DET_ID + book getword detPendingNormStatImfile $pageName det_type -var DET_TYPE + book getword detPendingNormStatImfile $pageName iteration -var ITERATION + book getword detPendingNormStatImfile $pageName camera -var CAMERA + book getword detPendingNormStatImfile $pageName workdir -var WORKDIR + book getword detPendingNormStatImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX add $WORKDIR/$LOG_DIR + # XXX use ipp_filename.pl to lookup output file names + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$CAMERA.$DET_TYPE.$DET_ID + $logfile = $outroot/$CAMERA.$DET_TYPE.reject.$DET_ID.$ITERATION.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_norm_calc.pl --det_id $DET_ID --iteration $ITERATION --det_type $DET_TYPE $ARGS + end + command detrend_norm_calc.pl --det_id $DET_ID --iteration $ITERATION --det_type $DET_TYPE $ARGS + end + + # default exit status + task.exit default + process_exit detPendingNormStatImfile $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingNormStatImfile $options:0 state TIMEOUT + end +end + +# select images ready for copy +# new entries are added to detPendingNormImfile +# compare the new list with the ones already selected +task dettool.norm.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/dettool.norm.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -tonormalize -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingNormImfile_DB + command dettool -tonormalize -limit 20 -dbname $DB:$detPendingNormImfile_DB + $detPendingNormImfile_DB ++ + if ($detPendingNormImfile_DB >= $DB:n) set detPendingNormImfile_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingNormImfile -key det_id:iteration -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingNormImfile + end + + # delete existing entries in the appropriate states + process_cleanup detPendingNormImfile + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.norm.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingNormImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingNormImfile + book getpage detPendingNormImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingNormImfile $pageName state RUN + book getword detPendingNormImfile $pageName det_type -var DET_TYPE + book getword detPendingNormImfile $pageName camera -var CAMERA + book getword detPendingNormImfile $pageName uri -var URI + book getword detPendingNormImfile $pageName det_id -var DET_ID + book getword detPendingNormImfile $pageName iteration -var ITERATION + book getword detPendingNormImfile $pageName class_id -var CLASS_ID + book getword detPendingNormImfile $pageName norm -var NORM + book getword detPendingNormImfile $pageName workdir -var WORKDIR + book getword detPendingNormImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX use ipp_filename.pl to lookup output file names + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$CAMERA.$DET_TYPE.$DET_ID + $logfile = $outroot/$CAMERA.$DET_TYPE.$DET_ID.$ITERATION.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_norm_apply.pl --det_id $DET_ID --iteration $ITERATION --class_id $CLASS_ID --value $NORM --input_uri $URI --camera $CAMERA --det_type $DET_TYPE $ARGS + end + command detrend_norm_apply.pl --det_id $DET_ID --iteration $ITERATION --class_id $CLASS_ID --value $NORM --input_uri $URI --camera $CAMERA --det_type $DET_TYPE $ARGS + end + + # default exit status + task.exit default + process_exit detPendingNormImfile $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingNormImfile $options:0 state TIMEOUT + end +end + +# select images ready for copy +# new entries are added to detPendingNormExp +# compare the new list with the ones already selected +task dettool.normexp.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/dettool.normexp.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -tonormalizedexp -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingNormExp_DB + command dettool -tonormalizedexp -limit 20 -dbname $DB:$detPendingNormExp_DB + $detPendingNormExp_DB ++ + if ($detPendingNormExp_DB >= $DB:n) set detPendingNormExp_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingNormExp -key det_id:iteration -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingNormExp + end + + # delete existing entries in the appropriate states + process_cleanup detPendingNormExp + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.normexp.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingNormExp -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingNormExp + book getpage detPendingNormExp 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingNormExp $pageName state RUN + book getword detPendingNormExp $pageName det_id -var DET_ID + book getword detPendingNormExp $pageName iteration -var ITERATION + book getword detPendingNormExp $pageName det_type -var DET_TYPE + book getword detPendingNormExp $pageName camera -var CAMERA + book getword detPendingNormExp $pageName workdir -var WORKDIR + book getword detPendingNormExp $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX add $WORKDIR/$LOG_DIR + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$CAMERA.$DET_TYPE.$DET_ID + $logfile = $outroot/$CAMERA.$DET_TYPE.$DET_ID.$ITERATION.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_norm_exp.pl --det_id $DET_ID --iteration $ITERATION --camera $CAMERA --det_type $DET_TYPE $ARGS + end + command detrend_norm_exp.pl --det_id $DET_ID --iteration $ITERATION --camera $CAMERA --det_type $DET_TYPE $ARGS + end + + # default exit status + task.exit default + process_exit detPendingNormExp $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingNormExp $options:0 state TIMEOUT + end +end Index: /branches/rel-1-0/ippTasks/detrend.process.pro =================================================================== --- /branches/rel-1-0/ippTasks/detrend.process.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/detrend.process.pro (revision 11645) @@ -0,0 +1,297 @@ +## this file contains the tasks for running the detrend processing stage +## these tasks use the books detPendingProcessedImfile and detPendingProcessedExp + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/detrend +exec mkdir -p $LOGSUBDIR + +book init detPendingProcessedImfile +book init detPendingProcessedExp + +macro detproc.reset + book init detPendingProcessedImfile + book init detPendingProcessedExp +end + +macro detproc.status + echo detPendingProcessedImfile + book listbook detPendingProcessedImfile + echo detPendingProcessedExp + book listbook detPendingProcessedExp +end + +macro detproc.on + task dettool.process.load + active true + end + task dettool.process.run + active true + end + task dettool.processexp.load + active true + end + task dettool.processexp.run + active true + end +end + +macro detproc.off + task dettool.process.load + active false + end + task dettool.process.run + active false + end + task dettool.processexp.load + active false + end + task dettool.processexp.run + active false + end +end + + +# these variables will cycle through the known database names +$detPendingProcessedImfile_DB = 0 +$detPendingProcessedExp_DB = 0 + +# select images ready for copy +# new entries are added to detPendingProcessedImfile +# compare the new list with the ones already selected +task dettool.process.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/detproc.imfile.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -toprocessedimfile -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingProcessedImfile_DB + command dettool -toprocessedimfile -limit 20 -dbname $DB:$detPendingProcessedImfile_DB + $detPendingProcessedImfile_DB ++ + if ($detPendingProcessedImfile_DB >= $DB:n) set detPendingProcessedImfile_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingProcessedImfile -key det_id:exp_tag:class_id -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingProcessedImfile + end + + # delete existing entries in the appropriate states + process_cleanup detPendingProcessedImfile + end + + # error + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand + end +end + +# run detrend_process_imfile.ps on pending images +task dettool.process.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingProcessedImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingProcessedImfile + book getpage detPendingProcessedImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingProcessedImfile $pageName state RUN + book getword detPendingProcessedImfile $pageName det_id -var DET_ID + book getword detPendingProcessedImfile $pageName det_type -var DET_TYPE + book getword detPendingProcessedImfile $pageName exp_tag -var EXP_TAG + book getword detPendingProcessedImfile $pageName class -var CLASS + book getword detPendingProcessedImfile $pageName class_id -var CLASS_ID + book getword detPendingProcessedImfile $pageName uri -var URI + book getword detPendingProcessedImfile $pageName camera -var CAMERA + book getword detPendingProcessedImfile $pageName workdir -var WORKDIR + book getword detPendingProcessedImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + ## output log filename + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + $logfile = $outroot/$EXP_TAG.detproc.$DET_ID.$CLASS_ID.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_process_imfile.pl --det_id $DET_ID --det_type $DET_TYPE --exp_tag $EXP_TAG --class $CLASS --class_id $CLASS_ID --input_uri $URI --camera $CAMERA $ARGS + end + command detrend_process_imfile.pl --det_id $DET_ID --det_type $DET_TYPE --exp_tag $EXP_TAG --class $CLASS --class_id $CLASS_ID --input_uri $URI --camera $CAMERA $ARGS + end + + # default exit status + task.exit default + process_exit detPendingProcessedImfile $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingProcessedImfile $options:0 state TIMEOUT + end +end + +# select images ready for copy +# new entries are added to detPendingProcessedExp +# compare the new list with the ones already selected +task dettool.processexp.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/detproc.exp.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -toprocessedexp -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingProcessedExp_DB + command dettool -toprocessedexp -limit 20 -dbname $DB:$detPendingProcessedExp_DB + $detPendingProcessedExp_DB ++ + if ($detPendingProcessedExp_DB >= $DB:n) set detPendingProcessedExp_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingProcessedExp -key det_id:iteration:exp_tag -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingProcessedExp + end + + # delete existing entries in the appropriate states + process_cleanup detPendingProcessedExp + end + + # error + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.processexp.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingProcessedExp -var N + if ($N == 0) break + if ($network == 0) break + + # look for new exposures in detPendingProcessedExp + book getpage detPendingProcessedExp 0 -var pageName -key state NULL + if ($pageName == NULL) break + + # XXX is ITERATION relevant here?? + book setword detPendingProcessedExp $pageName state RUN + book getword detPendingProcessedExp $pageName det_id -var DET_ID + book getword detPendingProcessedExp $pageName iteration -var ITERATION + book getword detPendingProcessedExp $pageName det_type -var DET_TYPE + book getword detPendingProcessedExp $pageName exp_tag -var EXP_TAG + book getword detPendingProcessedExp $pageName camera -var CAMERA + book getword detPendingProcessedExp $pageName workdir -var WORKDIR + book getword detPendingProcessedExp $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX push this into the filerules? + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + $logfile = $outroot/$EXP_TAG.detproc.$DET_ID.$ITERATION.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_process_exp.pl --det_id $DET_ID --exp_tag $EXP_TAG --camera $CAMERA $ARGS + end + command detrend_process_exp.pl --det_id $DET_ID --exp_tag $EXP_TAG --camera $CAMERA $ARGS + end + + # success + task.exit default + process_exit detPendingProcessedExp $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingProcessedExp $options:0 state TIMEOUT + end +end Index: /branches/rel-1-0/ippTasks/detrend.reject.pro =================================================================== --- /branches/rel-1-0/ippTasks/detrend.reject.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/detrend.reject.pro (revision 11645) @@ -0,0 +1,165 @@ +## this file contains the tasks for running the detrend processing stage +## these tasks use the book detRejectExp + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/detrend +exec mkdir -p $LOGSUBDIR + +book init detRejectExp + +macro detreject.reset + book init detRejectExp +end + +macro detreject.status + book listbook detRejectExp +end + +macro detreject.on + task dettool.reject.load + active true + end + task dettool.reject.run + active true + end +end + +macro detreject.off + task dettool.reject.load + active false + end + task dettool.reject.run + active false + end +end + +# this variable will cycle through the known database names +$detRejectExp_DB = 0 + +# select images ready for copy +# new entries are added to detRejectExp +# compare the new list with the ones already selected +task dettool.reject.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/dettool.reject.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -residdetrun -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detRejectExp_DB + command dettool -residdetrun -limit 20 -dbname $DB:$detRejectExp_DB + $detRejectExp_DB ++ + if ($detRejectExp_DB >= $DB:n) set detRejectExp_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detRejectExp -key det_id:iteration -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detRejectExp + end + + # delete existing entries in the appropriate states + process_cleanup detRejectExp + end + + # locked list + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand failure + end +end + +# copy new images, sending job to desired host +task dettool.reject.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detRejectExp -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detRejectExp + book getpage detRejectExp 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detRejectExp $pageName state RUN + book getword detRejectExp $pageName det_id -var DET_ID + book getword detRejectExp $pageName iteration -var ITERATION + book getword detRejectExp $pageName det_type -var DET_TYPE + book getword detRejectExp $pageName mode -var MODE + book getword detRejectExp $pageName camera -var CAMERA + # book getword detRejectExp $pageName workdir -var WORKDIR + # XXX workdir is not being returned here + $WORKDIR = NONE + book getword detRejectExp $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # I'd like to add CAMERA to the log file... + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$CAMERA.$DET_TYPE.$DET_ID + $logfile = $outroot/$CAMERA.$DET_TYPE.reject.$DET_ID.$ITERATION.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_reject_exp.pl --det_id $DET_ID --iteration $ITERATION --det_type $DET_TYPE --camera $CAMERA $ARGS + end + command detrend_reject_exp.pl --det_id $DET_ID --iteration $ITERATION --det_type $DET_TYPE --camera $CAMERA $ARGS + end + + # default exit status + task.exit default + process_exit detRejectExp $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand failure + book setword detRejectExp $options:0 state TIMEOUT + end +end Index: /branches/rel-1-0/ippTasks/detrend.resid.pro =================================================================== --- /branches/rel-1-0/ippTasks/detrend.resid.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/detrend.resid.pro (revision 11645) @@ -0,0 +1,297 @@ +## this file contains the tasks for running the detrend processing stage +## these tasks use the books detPendingResidImfile and detPendingResidExp + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/detrend +exec mkdir -p $LOGSUBDIR + +book init detPendingResidImfile +book init detPendingResidExp + +macro detresid.reset + book init detPendingResidImfile + book init detPendingResidExp +end + +macro detresid.status + book listbook detPendingResidImfile + book listbook detPendingResidExp +end + +macro detresid.on + task dettool.resid.load + active true + end + task dettool.resid.run + active true + end + task dettool.residexp.load + active true + end + task dettool.residexp.run + active true + end +end + +macro detresid.off + task dettool.resid.load + active false + end + task dettool.resid.run + active false + end + task dettool.residexp.load + active false + end + task dettool.residexp.run + active false + end +end + +# these variables will cycle through the known database names +$detPendingResidImfile_DB = 0 +$detPendingResidExp_DB = 0 + +# select images ready for copy +# new entries are added to detPendingResidImfile +# compare the new list with the ones already selected +task dettool.resid.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/detresid.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -toresidimfile -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingResidImfile_DB + command dettool -toresidimfile -limit 20 -dbname $DB:$detPendingResidImfile_DB + $detPendingResidImfile_DB ++ + if ($detPendingResidImfile_DB >= $DB:n) set detPendingResidImfile_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingResidImfile -key det_id:iteration:exp_tag:class_id -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingResidImfile + end + + # delete existing entries in the appropriate states + process_cleanup detPendingResidImfile + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.resid.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingResidImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingResidImfile + book getpage detPendingResidImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingResidImfile $pageName state RUN + book getword detPendingResidImfile $pageName det_id -var DET_ID + book getword detPendingResidImfile $pageName iteration -var ITERATION + book getword detPendingResidImfile $pageName det_type -var DET_TYPE + book getword detPendingResidImfile $pageName mode -var MODE + book getword detPendingResidImfile $pageName exp_tag -var EXP_TAG + book getword detPendingResidImfile $pageName class_id -var CLASS_ID + book getword detPendingResidImfile $pageName uri -var URI + book getword detPendingResidImfile $pageName det_uri -var DET_URI + book getword detPendingResidImfile $pageName camera -var CAMERA + book getword detPendingResidImfile $pageName workdir -var WORKDIR + book getword detPendingResidImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX use ipp_filename.pl to lookup output file names + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + $logfile = $outroot/$EXP_TAG.detresid.$DET_ID.$ITERATION.$CLASS_ID.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create command + if ($VERBOSE > 1) + echo command detrend_resid.pl --det_id $DET_ID --iteration $ITERATION --exp_tag $EXP_TAG --class_id $CLASS_ID --det_type $DET_TYPE --detrend $DET_URI --input_uri $URI --camera $CAMERA --mode $MODE $ARGS + end + command detrend_resid.pl --det_id $DET_ID --iteration $ITERATION --exp_tag $EXP_TAG --class_id $CLASS_ID --det_type $DET_TYPE --detrend $DET_URI --input_uri $URI --camera $CAMERA --mode $MODE $ARGS + end + + # default exit status + task.exit default + process_exit detPendingResidImfile $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingResidImfile $options:0 state TIMEOUT + end +end + +# select images ready for copy +# new entries are added to detPendingResidExp +# compare the new list with the ones already selected +task dettool.residexp.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/detresid.exp.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -toresidexp -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingResidExp_DB + command dettool -toresidexp -limit 20 -dbname $DB:$detPendingResidExp_DB + $detPendingResidExp_DB ++ + if ($detPendingResidExp_DB >= $DB:n) set detPendingResidExp_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingResidExp -key det_id:iteration:exp_tag -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingResidExp + end + + # delete existing entries in the appropriate states + process_cleanup detPendingResidExp + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.residexp.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages detPendingResidExp -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingResidExp + book getpage detPendingResidExp 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingResidExp $pageName state RUN + book getword detPendingResidExp $pageName det_id -var DET_ID + book getword detPendingResidExp $pageName iteration -var ITERATION + book getword detPendingResidExp $pageName det_type -var DET_TYPE + book getword detPendingResidExp $pageName mode -var MODE + book getword detPendingResidExp $pageName exp_tag -var EXP_TAG + book getword detPendingResidExp $pageName include -var INCLUDE + book getword detPendingResidExp $pageName camera -var CAMERA + book getword detPendingResidExp $pageName workdir -var WORKDIR + book getword detPendingResidExp $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX use ipp_filename.pl to lookup output file names + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + $logfile = $outroot/$EXP_TAG.detresid.$DET_ID.$ITERATION.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create example job options as a demonstration + if ($VERBOSE > 1) + echo command detrend_reject_imfile.pl --det_id $DET_ID --iteration $ITERATION --exp_tag $EXP_TAG --det_type $DET_TYPE --camera $CAMERA $ARGS + end + command detrend_reject_imfile.pl --det_id $DET_ID --iteration $ITERATION --exp_tag $EXP_TAG --det_type $DET_TYPE --camera $CAMERA $ARGS + end + + # default exit status + task.exit default + process_exit detPendingResidExp $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingResidExp $options:0 state TIMEOUT + end +end Index: /branches/rel-1-0/ippTasks/detrend.stack.pro =================================================================== --- /branches/rel-1-0/ippTasks/detrend.stack.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/detrend.stack.pro (revision 11645) @@ -0,0 +1,165 @@ +## this file contains the tasks for running the detrend stacking stage +## these tasks use the book detPendingStackedImfile + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/detrend +exec mkdir -p $LOGSUBDIR + +book init detPendingStackedImfile + +macro detstack.reset + book init detPendingStackedImfile +end + +macro detstack.status + echo detPendingStackedImfile + book listbook detPendingStackedImfile +end + +macro detstack.on + task dettool.stack.load + active true + end + task dettool.stack.run + active true + end +end + +macro detstack.off + task dettool.stack.load + active false + end + task dettool.stack.run + active false + end +end + +# this variable will cycle through the known database names +$detPendingStackedImfile_DB = 0 + +# select images ready for detrend_stack.pl +# new entries are added to detPendingStackedImfile +# compare the new list with the ones already selected +task dettool.stack.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/detstack.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command dettool -tostacked -limit 20 + else + # save the DB name for the exit tasks + option $DB:$detPendingStackedImfile_DB + command dettool -tostacked -limit 20 -dbname $DB:$detPendingStackedImfile_DB + $detPendingStackedImfile_DB ++ + if ($detPendingStackedImfile_DB >= $DB:n) set detPendingStackedImfile_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout detPendingStackedImfile -key det_id:iteration:class_id -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook detPendingStackedImfile + end + + # delete existing entries in the appropriate states + process_cleanup detPendingStackedImfile + end + + # locked list + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# copy new images, sending job to desired host +task dettool.stack.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 120 + + task.exec + book npages detPendingStackedImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in detPendingStackedImfile + book getpage detPendingStackedImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword detPendingStackedImfile $pageName state RUN + book getword detPendingStackedImfile $pageName det_id -var DET_ID + book getword detPendingStackedImfile $pageName iteration -var ITERATION + book getword detPendingStackedImfile $pageName det_type -var DET_TYPE + book getword detPendingStackedImfile $pageName class_id -var CLASS_ID + book getword detPendingStackedImfile $pageName camera -var CAMERA + book getword detPendingStackedImfile $pageName workdir -var WORKDIR + book getword detPendingStackedImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host: + if ($parallel) + host anyhost + else + host local + end + + # XXX add $WORKDIR/$LOG_DIR + # XXX use ipp_filename.pl to lookup output file names + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$CAMERA.$DET_TYPE.$DET_ID + $logfile = $outroot/$CAMERA.$DET_TYPE.stack.$DET_ID.$ITERATION.$CLASS_ID.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create the command line + if ($VERBOSE > 1) + echo command detrend_stack.pl --det_id $DET_ID --iteration $ITERATION --class_id $CLASS_ID --det_type $DET_TYPE --camera $CAMERA $ARGS + end + command detrend_stack.pl --det_id $DET_ID --iteration $ITERATION --class_id $CLASS_ID --det_type $DET_TYPE --camera $CAMERA $ARGS + end + + # default exit status + task.exit default + process_exit detPendingStackedImfile $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword detPendingStackedImfile $options:0 state TIMEOUT + end +end Index: /branches/rel-1-0/ippTasks/mkdetrend.pro =================================================================== --- /branches/rel-1-0/ippTasks/mkdetrend.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/mkdetrend.pro (revision 11645) @@ -0,0 +1,8 @@ + +input detrend.mkruns.pro +input detrend.process.pro +input detrend.stack.pro +input detrend.norm.pro +input detrend.resid.pro +input detrend.assess.pro + Index: /branches/rel-1-0/ippTasks/notes.txt =================================================================== --- /branches/rel-1-0/ippTasks/notes.txt (revision 11645) +++ /branches/rel-1-0/ippTasks/notes.txt (revision 11645) @@ -0,0 +1,192 @@ + +2007.01.08 + + I have updated pantasks to used named tags in named groups to make + life much easier when changing the schema of the db tables. The + function ipptool2book convernts the output from an ippTool query in + the form of a set of metadata structures (saved in stdout) into a + book format. + + dettool -tonormalizedstat DetrendNormStat + dettool -tonormalize DetrendNorm + dettool -tonormalizedexp DetrendNormExp + dettool -toprocessedimfile DetrendProcessImfiles + dettool -toprocessedexp DetrendProcessExposures + dettool -residdetrun DetrendRejectExp + dettool -toresidimfile DetrendResidImfiles + dettool -toresidexp DetrendResidExposures + dettool -tostacked DetrendStackClass + p0tool -pendingimfile Phase0Imfiles + p0tool -pendingexp Phase0Exposures + p2tool -pendingimfile Phase2Imfiles + p3tool -pendingexp Phase3Exposures + +2006.08.28 + +- ipptools need to use expID:camera:telescope as unique key (ie, not expID alone) +- p0tool -pendingimfile -simple needs to report the expKey (ie, not expID) +- + + +This directory contains the PanTasks scripts which define the steps +performed by the IPP. These scripts use the ippTools to decide what +images need to be processed, and to update the Metadata Database with +the current state of the analysis. The analysis is performed using +IPP programs which are distributed to the cluster computers as needed. + +1. The Detrend Creation Scripts + +The detrend creation process is among the more complex elements of the +IPP. There are N major steps which must be performed: + +1.1. Define New Detrend Runs + +The script 'detrend.mkruns.pro' defines tasks which initiate new +Detrend Runs. There is a separate task for each camera automatically +processed by the system (mkdetruns.CAMERA). These tasks are executed +on a regular, but infrequent basis. The tasks call external helper +programs which actually run the command dettools -define. The helper +programs are thus responsible for identifying all of the types of +detrend images to be run for a given camera, as well as the correct +set of filters to apply. These tasks are run locally and only affect +the database. PanTasks does not track the detrend runs which are +created. + +- we may need to define these within the PanTasks scripts +- how frequently are these tasks run (1/day?) +- if no raw images are selected by the query, no new detrun should be + created. +- we may need different schedules for the dark (bias/dark) and the + light (flat/fringe) data. And yet another schedule for sky data + (skyflat/skyfringe). + +1.2. Process new raw imfiles + +The script 'detrend.process.pro' defines tasks which processes the new +raw imfiles for any detrend run. This script defines two tasks, +'dettool.raw.load' and 'dettool.raw.process'. The first script +occasionally polls the Metadata Database to select raw imfiles which +have been identified as part of a detrend run. The identified raw +imfiles are added to an internal queue, DetrendImfilesToProcess. Any +raw imfile already in the list is ignored. The imfiles are added to +the queue with a state of NEW. The second task examines the queue for +imfiles with state NEW and defines a ppImage analysis job for that +imfile. The job is sent to pcontrol for processing on the distributed +cluster. At this point, the corresponding queue entry is set to the +state RUN. Upon success, the queue entry is set to DONE and the +database is notified of the successful operation. The DONE entries +are occasionally pruned from the queue; this must be done in the same +task as the initial query and the update of the list based on the +database results. + +- we need to define the columns in the output of dettool -raw +- we need a method to associate an imfile with a node in the cluster +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs +- we need a method to identify the correct recipe (function of type?) + +1.3. Stack new processed imfiles + +The script 'detrend.stack.pro' defines tasks which stack the newly +processed imfiles for any detrend run iteration. This script defines +two tasks, 'dettool.stack.load and 'dettool.stack.process. The first +script occasionally polls the Metadata Database to select class IDs +for detrend runs for which the input imfiles have all been processed. +The identified detRun/classID combination is added to an internal +queue, DetrendClassIDtoStack. The entries are added to the queue with +a state of NEW. The second task examines the queue for entries with a +state of NEW. It defines a ppMerge analysis job for that +detID/classID. The helper script selects the corresponding input +imfiles and passes them to ppMerge. The job is sent to pcontrol for +processing on the distributed cluster. At this point, the +corresponding queue entry is set to the state RUN. Upon success, the +queue entry is set to DONE and the database is notified of the +successful operation. The DONE entries are occasionally pruned from +the queue; this must be done in the same task as the initial query and +the update of the list based on the database results. + +- we need to define the columns in the output of dettool -chip -processed +- we need a method to associate a classID with a node in the cluster +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) +- we need a method to identify the correct recipe (function of type?) + + +1.4. Normalize stacked masters + +The script 'detrend.norm.pro' defines tasks which normalize the newly +processed stacks for any detrend run iteration. This script defines +two tasks, 'dettool.norm.load and 'dettool.norm.process. The first +script occasionally polls the Metadata Database to select detrun IDs +for which the classes have all been stacked. The identified +detRunID/iteration combination is added to an internal queue, +DetrendRunToNormalize. The entries are added to the queue with a +state of NEW. The second task examines the queue for entries with a +state of NEW. It defines a ppNorm analysis job for that +detID/iteration. The job is sent to pcontrol for processing on the +distributed cluster. At this point, the corresponding queue entry is +set to the state RUN. Upon success, the queue entry is set to DONE and +the database is notified of the successful operation. The DONE +entries are occasionally pruned from the queue; this must be done in +the same task as the initial query and the update of the list based on +the database results. + +- we need to define the columns in the output of dettool +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) +- we need a method to identify the correct recipe (function of type?) +- we need to define the ppNorm operation inputs and outputs better + + +1.5. Generate Resid Imfiles + +The script 'detrend.resid.pro' defines tasks which apply the newly +generated detrend image to the corresponding input images to generate +the residual images. This script defines two tasks, +'dettool.resid.load and 'dettool.resid.process. The first script +occasionally polls the Metadata Database to select imfiles for which +the master detrend images has been normalized. The identified imfiles +are added to an internal queue, DetrendResidImfiles. The entries are +added to the queue with a state of NEW. The second task examines the +queue for entries with a state of NEW. It defines a ppImage analysis +job for that imfile. The job is sent to pcontrol for processing on +the distributed cluster. At this point, the corresponding queue entry +is set to the state RUN. Upon success, the queue entry is set to DONE +and the database is notified of the successful operation. The DONE +entries are occasionally pruned from the queue; this must be done in +the same task as the initial query and the update of the list based on +the database results. + +- we need to define the columns in the output of dettool +- can we select the imfiles here without apriori knowledge that the + master has been normalized? +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) +- we need a method to identify the correct recipe (function of type?) +- we need a method to pass the matching detrend image to ppImage in + the correct context (ie, a bias needs to be supplied to the -bias element...) + +1.6. Assess Detrend Results + +The script 'detrend.assess.pro' defines tasks which assess the +residual statistics for completed detrend runs. This script defines +two tasks, 'dettool.assess.load and 'dettool.assess.process. The +first script occasionally polls the Metadata Database to select +detrunIDs/iterations for which the resid imfiles have all been +generated. The identified detruns are added to an internal queue, +DetrendRunToAssess. The entries are added to the queue with a state +of NEW. The second task examines the queue for entries with a state +of NEW. It defines an analysis job for that collection. The job is +sent to pcontrol for processing on the distributed cluster. At this +point, the corresponding queue entry is set to the state RUN. Upon +success, the queue entry is set to DONE and the database is notified +of the successful operation. The DONE entries are occasionally pruned +from the queue; this must be done in the same task as the initial +query and the update of the list based on the database results. + +- the assess tool needs to trigger a new run, if needed. +- what is the command to run on the resid statistics? +- we need to define the columns in the output of dettool +- can we select the detrend runs here? +- we need to perform some appropriate action when the queries fail. +- we need a method to generate the output URIs (not a function of the imfiles...) Index: /branches/rel-1-0/ippTasks/p0tool.sh =================================================================== --- /branches/rel-1-0/ippTasks/p0tool.sh (revision 11645) +++ /branches/rel-1-0/ippTasks/p0tool.sh (revision 11645) @@ -0,0 +1,7 @@ +#!/bin/csh -f +if ($#argv != 1) then + echo "USAGE: p0tool.sh (Nselect)" + exit 2 +endif + +p0tool -pendingimfile -simple | head -n $1 Index: /branches/rel-1-0/ippTasks/pantasks.pro =================================================================== --- /branches/rel-1-0/ippTasks/pantasks.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/pantasks.pro (revision 11645) @@ -0,0 +1,255 @@ + +# XXX make these upper-case +$network = 1 +$parallel = 1 +$VERBOSE = 1 +$LOGDIR = `pwd` +$scripts = . + +$loadpoll = 1 +$loadexec = 5 +$runpoll = 0.5 +$runexec = 1 + +$EXIT_SUCCESS = 0 +$EXIT_UNKNOWN_ERR = 1 +$EXIT_SYS_ERR = 2 +$EXIT_CONFIG_ERR = 3 +$EXIT_PROG_ERR = 4 +$EXIT_DATA_ERR = 5 + +# DB lists the database names in use; the default ipprc database +# is only used if DB:n is 0 +$DB:n = 0 + +macro init.cluster.po + $parallel = 1 + controller exit true + + # po05 -- using for pantasks + # po23 -- broken perl + + controller host add po02 + controller host add po03 + controller host add po04 +# controller host add po05 + controller host add po06 + controller host add po07 +# controller host add po08 +# controller host add po09 +# controller host add po10 +# controller host add po11 +# controller host add po12 +# controller host add po13 +# controller host add po14 +# controller host add po15 +# controller host add po16 +# controller host add po17 +# controller host add po18 +# controller host add po19 +# controller host add po20 +# controller host add po21 +# controller host add po22 +# controller host add po23 +# controller host add po24 +end + +macro init.cluster.sn + $parallel = 1 + controller exit true + controller host add sn2 + controller host add sn3 + controller host add sn4 + controller host add sn5 +end + +macro init.isp + list DB -add isp +end + +macro init.essence + list DB -add essence_v2 +end + +macro init.simtest.local + list DB -add simtest + $parallel = 0 + controller exit true +end + +macro init.simtest.default.db + $host = `hostname` + if ($parallel) + controller exit true + controller host add $host + end +end + +macro init.simtest + list DB -add simtest + + $host = `hostname` + if ($parallel) + controller exit true + controller host add $host + end +end + +# need the ability to activate or deactivate specific tasks... + +macro detrend.on + detproc.on + detstack.on + detnorm.on + detresid.on + detreject.on +end + +macro detrend.off + detproc.off + detstack.off + detnorm.off + detresid.off + detreject.off +end + +macro detrend.reset + detproc.reset + detstack.reset + detnorm.reset + detresid.reset + detreject.reset +end + + +macro module.tasks + module phase0.pro +# module $scripts/phase2.pro +# module $scripts/phase3.pro + module detrend.process.pro + module detrend.stack.pro + module detrend.norm.pro + module detrend.resid.pro + module detrend.reject.pro +end + +macro showcommand + local n + + if ($VERBOSE) + $command = $taskarg:0 + for n 1 $taskarg:n + $command = $command $taskarg:$n + end + if ($0 == 2) + echo "$1 for: $command" + echo "job exit status: $JOB_STATUS" + echo "job dtime: $JOB_DTIME" + else + echo "command: $command" + end + end +end + +macro set_standard_args + # XXX be careful: $ARGS cannot be resolved as "" + # XXX currently, sprintf is the only good way of creating + # an empty variable + if ($?WORKDIR == 0) + echo error in pantasks: WORKDIR not set + stop + end + if ($?DBNAME == 0) + echo error in pantasks: DBNAME not set + stop + end + if ("$DBNAME" != "DEFAULT") + $ARGS = --workdir $WORKDIR --dbname $DBNAME + else + $ARGS = --workdir $WORKDIR + end +end + +macro process_exit + if ($0 != 4) + echo "USAGE: process_exit (bookName) (pageName) (exitCode)" + break + end + + $bookName = $1 + $pageName = $2 + $exitCode = $3 + + if ($VERBOSE > 4) + echo "*** stdout ***" + queueprint stdout + echo "*** stderr ***" + queueprint stderr + echo "JOB_STATUS: $JOB_STATUS" + end + + # success + if ($exitCode == $EXIT_SUCCESS) + # the handler scripts update DB the tables; here we just update the page + book setword $bookName $pageName state DONE + return + end + + # failure related to the data files + # jobs which result in DATAERR must have db state updated + if ($exitCode == $EXIT_DATA_ERR) + showcommand failure + book setword $bookName $pageName state DATA_ERR + return + end + + if ($VERBOSE) + echo "*** stdout ***" + queueprint stdout + echo "*** stderr ***" + queueprint stderr + end + + # failure related to the data files + if ($exitCode == $EXIT_SYS_ERR) + # stop + showcommand "system failure" + book setword $bookName $pageName state SYS_ERR + return + end + + # failure related to the data files + if ($exitCode == $EXIT_CONFIG_ERR) + # stop + showcommand "config error" + book setword $bookName $pageName state CONFIG_ERR + return + end + + # failure related to the data files + if ($exitCode == $EXIT_PROG_ERR) + # stop + showcommand "programming error" + book setword $bookName $pageName state PROG_ERR + return + end + + # any other exit status + showcommand failure + book setword $bookName $pageName state UNKNOWN_ERR +end + +## XXX for the moment, remove all errors +macro process_cleanup + if ($0 != 2) + echo "USAGE: process_cleanup (bookname)" + break + end + + book delpage $1 -key state DONE + book delpage $1 -key state SYS_ERR + book delpage $1 -key state DATA_ERR + book delpage $1 -key state PROG_ERR + book delpage $1 -key state CONFIG_ERR + book delpage $1 -key state UNKNOWN_ERR +end Index: /branches/rel-1-0/ippTasks/phase0.pro =================================================================== --- /branches/rel-1-0/ippTasks/phase0.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/phase0.pro (revision 11645) @@ -0,0 +1,290 @@ +## this file contains the tasks for running the phase 0 stage +## these tasks use the books p0PendingImfile and p0PendingExp + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/phase0 +exec mkdir -p $LOGSUBDIR + +book init p0PendingImfile +book init p0PendingExp + +macro phase0.reset + book init p0PendingImfile + book init p0PendingExp +end + +macro phase0.status + book listbook p0PendingImfile + book listbook p0PendingExp +end + +macro phase0.on + task phase0.imfile.load + active true + end + task phase0.imfile.run + active true + end + task phase0.exp.load + active true + end + task phase0.exp.run + active true + end +end + +macro phase0.off + task phase0.imfile.load + active false + end + task phase0.imfile.run + active false + end + task phase0.exp.load + active false + end + task phase0.exp.run + active false + end +end + +# these variables will cycle through the known database names +$p0PendingImfile_DB = 0 +$p0PendingExp_DB = 0 + +# select images ready for phase0 analysis +# new entries are added to p0PendingImfile +# compare the new list with the ones already selected +task phase0.imfile.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + # silently drop stdout + stdout NULL + stderr $LOGSUBDIR/phase0.imfile.log + + # select entries from the current DB; cycle to the next DB, if it exists + # iff the DB list is not set, use the value defined in .ipprc + task.exec + if ($DB:n == 0) + option DEFAULT + command p0tool -pendingimfile -limit 20 + else + # save the DB name for the exit tasks + option $DB:$p0PendingImfile_DB + command p0tool -pendingimfile -limit 20 -dbname $DB:$p0PendingImfile_DB + $p0PendingImfile_DB ++ + if ($p0PendingImfile_DB >= $DB:n) set p0PendingImfile_DB = 0 + end + end + + # success + task.exit $EXIT_SUCCESS + # convert 'stdout' to book format + ipptool2book stdout p0PendingImfile -key exp_tag -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook p0PendingImfile + end + + # delete existing entries in the appropriate states + process_cleanup p0PendingImfile + end + + # locked list + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# run the phase0_imfile.pl script on pending images +task phase0.imfile.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages p0PendingImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in p0PendingImfile + book getpage p0PendingImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword p0PendingImfile $pageName state RUN + + book getword p0PendingImfile $pageName exp_tag -var EXP_TAG + book getword p0PendingImfile $pageName class -var CLASS + book getword p0PendingImfile $pageName class_id -var CLASS_ID + book getword p0PendingImfile $pageName uri -var URI + book getword p0PendingImfile $pageName workdir -var WORKDIR + book getword p0PendingImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host + if ($parallel) + host anyhost + else + host local + end + + ## generate output log based on filerule XXX put this in a function? + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + # $logfile = `ipp_filename.pl PHASE0.LOG $outroot $CLASS_ID` + $logfile = $outroot/$EXP_TAG.$CLASS_ID.p0.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create the command line + if ($VERBOSE > 1) + echo command phase0_imfile.pl --exp_tag $EXP_TAG --class_id $CLASS_ID --uri $URI $ARGS + end + command phase0_imfile.pl --exp_tag $EXP_TAG --class_id $CLASS_ID --uri $URI $ARGS + end + + # default exit status + task.exit default + process_exit p0PendingImfile $options:0 $JOB_STATUS + end + + # operation timed out? + task.exit timeout + showcommand timeout + book setword p0PendingImfile $options:0 state TIMEOUT + end +end + +# select exposures ready for phase0_exp.pl +task phase0.exp.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + + stdout NULL + stderr $LOGSUBDIR/phase0.exp.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command p0tool -pendingexp -limit 20 + else + # save the DB name for the exit tasks + option $DB:$p0PendingExp_DB + command p0tool -pendingexp -limit 20 -dbname $DB:$p0PendingExp_DB + $p0PendingExp_DB ++ + if ($p0PendingExp_DB >= $DB:n) set p0PendingExp_DB = 0 + end + end + + # success + task.exit $EXIT_SUCCESS + # convert 'stdout' to book format + ipptool2book stdout p0PendingExp -key exp_tag -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook p0PendingExp + end + + # delete existing entries in the appropriate states + process_cleanup p0PendingExp + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# run the phase0_exp.pl script on pending exposures +task phase0.exp.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 30 + + task.exec + book npages p0PendingExp -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in p0PendingExp (state == NULL) + book getpage p0PendingExp 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword p0PendingExp $pageName state RUN + book getword p0PendingExp $pageName exp_tag -var EXP_TAG + book getword p0PendingExp $pageName workdir -var WORKDIR + book getword p0PendingExp $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host:(need to choose based on chips) + if ($parallel) + host anyhost + else + host local + end + + ## generate output log based on filerule + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + $logfile = $outroot/$EXP_TAG.p0.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create the command line + if ($VERBOSE > 1) + echo command phase0_exp.pl --exp_tag $EXP_TAG $ARGS + end + command phase0_exp.pl --exp_tag $EXP_TAG $ARGS + end + + # success + task.exit default + process_exit p0PendingExp $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword p0PendingExp $options:0 state TIMEOUT + end +end + +## XXX add a global path to output files Index: /branches/rel-1-0/ippTasks/phase2.pro =================================================================== --- /branches/rel-1-0/ippTasks/phase2.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/phase2.pro (revision 11645) @@ -0,0 +1,165 @@ +## this file contains the tasks for running the phase 0 stage +## these tasks use the book p2PendingImfile + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/phase2 +exec mkdir -p $LOGSUBDIR + +book init p2PendingImfile + +macro phase2.status + book listbook p2PendingImfile +end + +macro phase2.reset + book init p2PendingImfile +end + +macro phase2.on + task phase2.imfile.load + active true + end + task phase2.imfile.run + active true + end +end + +macro phase2.off + task phase2.imfile.load + active false + end + task phase2.imfile.run + active false + end +end + +# this variable will cycle through the known database names +$phase2_DB = 0 + +# select images ready for phase2 analysis +# new entries are added to p2PendingImfile +# skip already-present entries +task phase2.imfile.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout NULL + stderr $LOGSUBDIR/phase2.imfile.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command p2tool -pendingimfile -limit 20 + else + # save the DB name for the exit tasks + option $DB:$phase2_DB + command p2tool -pendingimfile -limit 20 -dbname $DB:$phase2_DB + $phase2_DB ++ + if ($phase2_DB >= $DB:n) set phase2_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout p2PendingImfile -key exp_tag:class_id -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook p2PendingImfile + end + + # delete existing entries in the appropriate states + process_cleanup p2PendingImfile + end + + # locked list + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# run the phase2.pl script on pending images +task phase2.imfile.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 60 + + task.exec + book npages p2PendingImfile -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in p2PendingImfile (state == NULL) + book getpage p2PendingImfile 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword p2PendingImfile $pageName state RUN + book getword p2PendingImfile $pageName camera -var CAMERA + book getword p2PendingImfile $pageName exp_tag -var EXP_TAG + book getword p2PendingImfile $pageName workdir -var WORKDIR + book getword p2PendingImfile $pageName class_id -var CLASS_ID + book getword p2PendingImfile $pageName uri -var URI + book getword p2PendingImfile $pageName workdir -var WORKDIR + book getword p2PendingImfile $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host + # XXX need to choose based on chips + if ($parallel) + host anyhost + else + host local + end + + ## generate output log based on filerule + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + # $logfile = `ipp_filename.pl PHASE2.LOG $outroot $CLASS_ID` + $logfile = $outroot/$EXP_TAG.$CLASS_ID.p2.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create the command line + if ($VERBOSE > 1) + echo command phase2.pl --exp_tag $EXP_TAG --class_id $CLASS_ID --uri $URI --camera $CAMERA $ARGS + end + command phase2.pl --exp_tag $EXP_TAG --class_id $CLASS_ID --uri $URI --camera $CAMERA $ARGS + end + + # default exit status + task.exit default + process_exit p2PendingImfile $options:0 $JOB_STATUS + end + + # operation timed out? + task.exit timeout + showcommand timeout + book setword p2PendingImfile $options:0 state TIMEOUT + end +end Index: /branches/rel-1-0/ippTasks/phase3.pro =================================================================== --- /branches/rel-1-0/ippTasks/phase3.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/phase3.pro (revision 11645) @@ -0,0 +1,141 @@ +## this file contains the tasks for running the phase 3 stage +## these tasks use the book p3PendingExp + +if ($?network == 0) + $network = 1 +end +if ($?parallel == 0) + $parallel = 0 +end +if ($?VERBOSE == 0) + echo "VERBOSE not defined: load pantasks.pro first" + break +end +if ($?LOGDIR == 0) + echo "LOGDIR not defined: load pantasks.pro first" + break +end + +$LOGSUBDIR = $LOGDIR/phase3 +exec mkdir -p $LOGSUBDIR + +book init p3PendingExp + +macro phase3.status + book listbook p3PendingExp +end + +# this variable will cycle through the known database names +$phase3_DB = 0 + +# select images ready for phase3 analysis +# new entries are added to p3PendingExp +# skip already-present entries +task phase3.exp.load + host local + + periods -poll $loadpoll + periods -exec $loadexec + periods -timeout 30 + npending 1 + + stdout $LOGSUBDIR/phase3.exp.log + stderr $LOGSUBDIR/phase3.exp.log + + task.exec + if ($DB:n == 0) + option DEFAULT + command p3tool -pendingexp -limit 20 + else + # save the DB name for the exit tasks + option $DB:$phase3_DB + command p3tool -pendingexp -limit 20 -dbname $DB:$phase3_DB + $phase3_DB ++ + if ($phase3_DB >= $DB:n) set phase3_DB = 0 + end + end + + # success + task.exit 0 + # convert 'stdout' to book format + ipptool2book stdout p3PendingExp -key exp_tag -uniq -setword dbname $options:0 + if ($VERBOSE > 2) + book listbook p3PendingExp + end + + # delete existing entries in the appropriate states + process_cleanup p3PendingExp + end + + # default exit status + task.exit default + showcommand failure + end + + # operation times out? + task.exit timeout + showcommand timeout + end +end + +# run the phase3exposure script on pending images +task phase3.exp.run + periods -poll $runpoll + periods -exec $runexec + periods -timeout 10 + + task.exec + book npages p3PendingExp -var N + if ($N == 0) break + if ($network == 0) break + + # look for new images in p3PendingExp (state == NULL) + book getpage p3PendingExp 0 -var pageName -key state NULL + if ($pageName == NULL) break + + book setword p3PendingExp $pageName state RUN + book getword p3PendingExp $pageName camera -var CAMERA + book getword p3PendingExp $pageName exp_tag -var EXP_TAG + book getword p3PendingExp $pageName workdir -var WORKDIR + book getword p3PendingExp $pageName dbname -var DBNAME + set_standard_args + + # specify choice of remote host:(need to choose based on chips) + if ($parallel) + host anyhost + else + host local + end + + # output log from filerule + $outroot = `ipp_datapath.pl $WORKDIR` + $outroot = $outroot/$EXP_TAG + # $logfile = `ipp_filename.pl PHASE3.LOG $outroot $CLASS_ID` + $logfile = $outroot/$EXP_TAG.$CLASS_ID.p3.log + stdout $logfile + stderr $logfile + exec mkdir -p $outroot + + # save the pageName for future reference below + options $pageName + + # create the command line + if ($VERBOSE > 1) + echo command phase3.pl --exp_tag $EXP_TAG $ARGS + end + command phase3.pl --exp_tag $EXP_TAG --camera $CAMERA $ARGS + end + + # success + task.exit default + process_exit p3PendingExp $options:0 $JOB_STATUS + end + + # operation times out? + task.exit timeout + showcommand timeout + book setword p3PendingExp $options:0 state TIMEOUT + end +end + +## XXX add a global path to output files Index: /branches/rel-1-0/ippTasks/summit.copy.pro =================================================================== --- /branches/rel-1-0/ippTasks/summit.copy.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/summit.copy.pro (revision 11645) @@ -0,0 +1,308 @@ +## PanTasks scripts for Summit Copy + +$uri = http://otis/ds/skyprobe/ +$camera = skyprobe +$telescope = ps1 +$workdir = /home/moanui/jhoblitt/workdir + +if ($?parallel == 0) + $parallel = 0 +end + +# queue for summit exps that need to be queried +queueinit pzPendingExp +# queue for summit imfiles that need to be downloaded +queueinit pzPendingImfile +# queue for downloaded files that need to registered with pztool -copydone +queueinit pzCopyDone + +# run pzgetexp periodically to populate pzPendingExp +task pzgetexp + command pzgetexp -uri $uri -inst $camera -telescope $telescope + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 30 + periods -poll 10 + periods -timeout 30 + + task.exit 0 + end + + task.exit default + echo "phaseZ pzgetexp: failure" + end + task.exit timeout + echo "phaseZ pzgetexp: timeout" + end +end + +# build a queue of exps/filesetids that need to be queried +task pztool.pendingexp + command pztool -pendingexp -simple + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 30 + periods -poll 10 + periods -timeout 30 + + # success + task.exit 0 + local i Nstdout + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line +# list word -split $line +# $exp_id = $word:0 +# $camera = $word:1 +# $telescope = $word:2 +# $dateobs = $word:3 +# $exp_type = $word:4 +# $uri = $word:5 +# $imfiles = $word:6 +# queuepush pzPendingExp -uniq -key 1:2:3 "NEW $uri $exp_id $camera $telescope $dateobs $exp_type $uri $imfiles" + queuepush pzPendingExp -uniq -key 1:2:3 "NEW $line" + end + + queuedrop pzPendingExp -key 0 DONE + end + + task.exit default + echo "phaseZ pending exp: failure" + end + task.exit timeout + echo "phaseZ pending exp: timeout" + end +end + +# run pzgetimfiles on pending exps +task pzgetimfile + periods -exec 5 + periods -poll 5 + periods -timeout 15 + + task.exec + queuesize pzPendingExp -var N + if ($N == 0) break + + # get a line out of the queue + queuepop pzPendingExp -var line -key 0 NEW + + # take the state out of the line + strpop line state + + # put the line back into the queue with a stea of RUN + queuepush pzPendingExp -replace -key 1:2:3 "RUN $line" + + # the STATE has already been removed from the line + list word -split $line + $exp_id = $word:0 + $camera = $word:1 + $telescope = $word:2 + $dateobs = $word:3 + $exp_type = $word:4 + $uri = $word:5 + $imfiles = $word:6 + + # specify choice of remote host:(need to choose based on chips) +# if ($parallel) +# host anyhost +# else +# host local +# end + + # store the current line + options "$line" + + echo command pzgetimfiles -uri $uri -filesetid $exp_id -inst $camera -telescope $telescope + command pzgetimfiles -uri $uri -filesetid $exp_id -inst $camera -telescope $telescope + end + + # success + task.exit 0 + queuepush pzPendingExp -replace -key 1:2:3 "DONE $options:0" + end + + task.exit default + queuepush pzPendingExp -replace -key 1:2:3 "FAIL $options:0" + end + + task.exit timeout + queuepush pzPendingExp -replace -key 1:2:3 "TIMEOUT $options:0" + end +end + +# build a queue of imfiles/files that need to be downloaded +task pztool.pendingimfile + command pztool -pendingimfile -simple + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 30 + periods -poll 10 + periods -timeout 30 + + # success + task.exit 0 + local i Nstdout + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line +# list word -split $line +# $exp_id = $word:0 +# $camera = $word:1 +# $telescope = $word:2 +# $file_id = $word:3 +# $bytes = $word:4 +# $md5sum = $word:5 +# $class = $word:6 +# $class_id = $word:7 +# $uri = $word:8 +# $exp_tag = $word:9 + + queuepush pzPendingImfile -uniq -key 1:2:3:7:8 "NEW $line" + end + + # delete existing entries which are DONE + queuedrop pzPendingImfile -key 0 DONE + end + + task.exit default + echo "phaseZ pending exp: failure" + end + task.exit timeout + echo "phaseZ pending exp: timeout" + end +end + +# get dsget to retreive an imfile and then call pztool -copydone +task dsget + periods -exec 5 + periods -poll 5 + periods -timeout 30 + + task.exec + queuesize pzPendingImfile -var N + if ($N == 0) break + + # get a line out of the queue + queuepop pzPendingImfile -var line -key 0 NEW + + # take the state out of the line + strpop line state + + # put the line back into the queue with a stea of RUN + queuepush pzPendingImfile -replace -key 1:2:3:7:8 "RUN $line" + + # the STATE has already been removed from the line + list word -split $line + $exp_id = $word:0 + $camera = $word:1 + $telescope = $word:2 + $file_id = $word:3 + $bytes = $word:4 + $md5sum = $word:5 + $class = $word:6 + $class_id = $word:7 + $uri = $word:8 + $exp_tag = $word:9 + +# # specify choice of remote host:(need to choose based on chips) +# if ($parallel) +# host anyhost +# else +# host local +# end + + # store the current line without the state + options "$line" + + echo "parsing line $line" + echo command dsget --uri $uri --filename $workdir/$exp_id.$class_id.fits --bytes $bytes --md5 $md5sum + command dsget --uri $uri --filename $workdir/$exp_id.$class_id.fits --bytes $bytes --md5 $md5sum + end + + # success + task.exit 0 + # could run copydone from here but how would the exit status be + # handled? + + list word -split $options:0 + $exp_id = $word:0 + $camera = $word:1 + $telescope = $word:2 + $file_id = $word:3 + $bytes = $word:4 + $md5sum = $word:5 + $class = $word:6 + $class_id = $word:7 + $uri = $word:8 + $exp_tag = $word:9 + queuepush pzCopyDone -key 1:2:3:7:8 "NEW $options:0 $workdir/$exp_id.$class_id.fits" + queuepush pzPendingExp -replace -key 1:2:3:7:8 "DONE $options:0" + end + + task.exit default + queuepush pzPendingExp -replace -key 1:2:3:7:8 "FAIL $options:0" + end + + task.exit timeout + queuepush pzPendingExp -replace -key 1:2:3:7:8 "TIMEOUT $options:0" + end +end + +task pztool.copydone + periods -exec 5 + periods -poll 5 + periods -timeout 15 + + host local + + task.exec + queuesize pzCopyDone -var N + if ($N == 0) break + + # get a line out of the queue + queuepop pzCopyDone -var line -key 0 NEW + + # take the state out of the line + strpop line state + + # put the line back into the queue with a stea of RUN + queuepush pzCopyDone -replace -key 1:2:3:7:8 "RUN $line" + + # the STATE has already been removed from the line + list word -split $line + $exp_id = $word:0 + $camera = $word:1 + $telescope = $word:2 + $file_id = $word:3 + $bytes = $word:4 + $md5sum = $word:5 + $class = $word:6 + $class_id = $word:7 + $uri = $word:8 + $exp_tag = $word:9 + $new_uri = $word:10 + + # store the current line without the state + options "$line" + + echo command pztool -copydone -exp_id $exp_id -inst $camera -telescope $telescope -class $class -class_id $class_id -uri $uri + command pztool -copydone -exp_id $exp_id -inst $camera -telescope $telescope -class $class -class_id $class_id -uri $new_uri + end + + # success + task.exit 0 + queuepush pzCopyDone -replace -key 1:2:3:6:7 "DONE $options:0" + end + + task.exit default + queuepush pzCopyDone -replace -key 1:2:3:6:7 "FAIL $options:0" + end + + task.exit timeout + queuepush pzCopyDone -replace -key 1:2:3:6:7 "TIMEOUT $options:0" + end +end Index: /branches/rel-1-0/ippTasks/testscript.pro =================================================================== --- /branches/rel-1-0/ippTasks/testscript.pro (revision 11645) +++ /branches/rel-1-0/ippTasks/testscript.pro (revision 11645) @@ -0,0 +1,80 @@ +# test for pcontrol +macro test.pcontrol.simple + stop + for i 0 100 + job ls -R /etc + end + + status + host add alala + + status + run +end + +# test for pcontrol +macro test.pcontrol + exec p0tool -pendingimfile -simple | head -n 100 > list.dat + + stop + list line -x "cat list.dat" + echo $line:0 + for i 0 $line:n + list word -split $line:$i + job phase0_imfile.pl --exp_tag $word:0 --class_id fpa --uri $word:3 + end + + status + host add sn2 + host add sn3 + host add sn4 + host add sn5 + + status + run +end + +# test for pclient +macro test.pclient + exec p0tool -pendingimfile -simple | head -n 100 > list.dat + + list line -x "cat list.dat" + echo $line:0 + for i 0 $line:n + list word -split $line:$i + job phase0_imfile.pl --exp_tag $word:0 --class_id fpa --uri $word:3 + + for i 0 50 + check + output /dev/null + status + output stdout + if ($JOBSTATUS == EXIT) + $i = 50 + reset + echo "DONE" + else + usleep 200000 + end + end + if ($JOBSTATUS != EXIT) + echo TIMEOUT*** + end + end +end + +macro simple.test + for j 0 100 + job ls + for i 0 50 + check + status + if ($JOBSTATUS == EXIT) + $i = 50 + reset + else + usleep 10000 + end + end + end +end Index: /branches/rel-1-0/ippTools/.cvsignore =================================================================== --- /branches/rel-1-0/ippTools/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippTools/.cvsignore (revision 11645) @@ -0,0 +1,16 @@ +Makefile +Makefile.in +aclocal.m4 +autom4te.cache +compile +config.guess +config.log +config.status +config.sub +configure +depcomp +install-sh +libtool +ltmain.sh +missing +pxtools.pc Index: /branches/rel-1-0/ippTools/COPYING =================================================================== --- /branches/rel-1-0/ippTools/COPYING (revision 11645) +++ /branches/rel-1-0/ippTools/COPYING (revision 11645) @@ -0,0 +1,345 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. Index: /branches/rel-1-0/ippTools/Makefile.am =================================================================== --- /branches/rel-1-0/ippTools/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippTools/Makefile.am (revision 11645) @@ -0,0 +1,12 @@ +SUBDIRS = src + +pkgconfigdir = $(libdir)/pkgconfig +pkgconfig_DATA= pxtools.pc + +EXTRA_DIST = \ + pxtools.pc.in \ + autogen.sh + +CLEANFILES = *~ core core.* + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippTools/TODO =================================================================== --- /branches/rel-1-0/ippTools/TODO (revision 11645) +++ /branches/rel-1-0/ippTools/TODO (revision 11645) @@ -0,0 +1,7 @@ +- change dettol -pretend -define... to return a metadata structure that shows what the created detRun would look like +- rename all -b1* options -> output_root +- remove all -b2* options +- rename dettool -residdetrun -> -todetrunsummary +- add forgein key constraints to the database +- add the abililty to remove dettool detResidExps +- add a mode to p2tool to create new pending Exp's from rawExp Index: /branches/rel-1-0/ippTools/autogen.sh =================================================================== --- /branches/rel-1-0/ippTools/autogen.sh (revision 11645) +++ /branches/rel-1-0/ippTools/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=pxtools +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOlIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +#($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $AUTOHEADER installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" +# DIE=1 +#} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL -I m4 || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ippTools/configure.ac =================================================================== --- /branches/rel-1-0/ippTools/configure.ac (revision 11645) +++ /branches/rel-1-0/ippTools/configure.ac (revision 11645) @@ -0,0 +1,57 @@ +AC_PREREQ(2.59) + +AC_INIT([ipptools], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([autogen.sh]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([src/config.h]) +AM_MAINTAINER_MODE + +IPP_STDCFLAGS + +AC_LANG(C) +AC_GNU_SOURCE +AC_PROG_CC +AC_PROG_INSTALL +AC_PROG_LIBTOOL + +PKG_CHECK_MODULES([PSLIB], [pslib >= 1.0.0]) +PKG_CHECK_MODULES([PSMODULES], [psmodules >= 1.0.0]) +PKG_CHECK_MODULES([IPPDB], [ippdb >= 1.0.0]) + +PXTOOLS_CFLAGS="${PSLIB_CFLAGS=} ${PSMODULES_CFLAGS=} ${IPPDB_CFLAGS=}" +PXTOOLS_LIBS="${PSLIB_LIBS=} ${PSMODULES_LIBS=} ${IPPDB_LIBS=}" +AC_SUBST(PXTOOLS_CFLAGS,[$PXTOOLS_CFLAGS]) +AC_SUBST(PXTOOLS_LIBS,[$PXTOOLS_LIBS]) + +dnl check for PSDB support +TMP_CFLAGS=${CFLAGS} +CFLAGS="${CFLAGS=} ${PSLIB_CFLAGS}" + +AC_COMPILE_IFELSE([AC_LANG_PROGRAM( + [[#if !defined HAVE_PSDB + error: PSDB support is not avaiable + #endif + ]])], + [], + [AC_MSG_ERROR([psLib was built without PSDB support, bailing out.])] +) + +CFLAGS=${TMP_CFLAGS} + +dnl -- check for psParseErrorCodes -------------------------------------- + +AC_PATH_PROG([ERRORCODES], [psParseErrorCodes], [missing]) +if test "$ERRORCODES" = "missing" ; then + AC_MSG_ERROR([psParseErrorCodes is required]) +fi + +IPP_STDOPTS +CFLAGS="${CFLAGS=} -Wall -Werror -std=c99" + +AC_CONFIG_FILES([ + Makefile + src/Makefile + pxtools.pc +]) +AC_OUTPUT Index: /branches/rel-1-0/ippTools/doc/dettest.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/dettest.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/dettest.sh (revision 11645) @@ -0,0 +1,120 @@ +#!/bin/sh + +# +# the default value for -iteration (e.g., you didn't specify -iteration) is 0 +# it may make sense to make it default to the latest iteration number for a +# given det run +# +# so of the sanity checking still needs to be improved/tested +# +# also worth noting is that the validity of the iteration value isn't yet +# checked +# + +# create a new detrun +det_id=`dettool -define -exp_id 1 -exp_id 2 -exp_id 3 -det_type flat` + +# list detruns (active only?) +dettool -runs + +# get detrun status (iteration #) + +# list the input exposures in a detRun +dettool -input -det_id $det_id + +# list raw imfiles +dettool -raw -det_id $det_id + +# register a processed imfile +dettool -addprocessed -det_id $det_id ... + +# processed imfiles should mask out raw imfiles +dettool -raw -det_id $det_id + +# list processed imfiles +dettool -processed -det_id $det_id [-iteration] + +# list processed imfiles for a chip (all or none) +dettool -processed -chip -det_id $det_id [-iteration] + +# list all processed imfiles (even those that have been stacked) +dettool -processed -unmask -det_id $det_id [-iteration] + +# register a stacked imfile +dettool -addstacked -det_id $det_id [-iteration] -class_id ... + +# stacked imfiles should mask out processed imfiles +dettool -processed -det_id $det_id [-iteration] + +# list stacked imfiles +dettool -stacked -det_id $det_id [-iteration] +# list stacked imfiles only for complete frames +dettool -stackedframe -det_id $det_id [-iteration] + +# register a normalized imfile +dettool -addnormalized -det_id $det_id [-iteration] ... + +# normalized imfiles should mask out stacked imfiles +dettool -stacked -det_id $det_id [-iteration] +# does a single normalized imfile mask out the entire frame? +dettool -stackedframe -det_id $det_id [-iteration] + +# list normalized imfiles +dettool -normalized -det_id $det_id[ -iteration] + +# register a residual result +dettool -addresid -det_id $det_id [-iteration] -exp_id ... -class_id ... -stats ... + +# residuals mask out normalized imfiles +dettool -normalized -det_id $det_id [-iteration] + +# list residual imfile analyses +dettool -resid -det_id $det_id [iteration] -exp_id ... -class_id ... -stats ... + +# register a residual exp XXX not sanity checked against all input class_ids having a resid registered +dettool -addresidexp -det_id $det_id [iteration] -stats ... -accept + +# list residual exps +dettool -residexp -det_id $det_id [iteration] + +# register a/the master +dettool -declaremasterframe -det_id $det_id [-iteration ...] + +# list masters +dettool -masterframe -det_id + +# list imfiles in a master frame +dettool -master -det_id $det_id [-iteration ...] + +# start another iteration but use only these input exps (must be a subset of +# the original input exps) +dettool -rerun -det_id $det_id -exp_id 1 -exp_id 2 -exp_id 3 + + +# always print out the det_type in the output for imfiles + +# -addresid -uri +# -addresidexp -uri (big/small jpeg) + +# -updatedresid for -accept +# -masterframe needs 'class' +# -nukemaster + +pxinject + +- class is currently in db at the imfile level, not the exp level. + +dettool + +- list of det_ids without registered masters +- -chip option is a bad name + +for normalize to resid, support script needs to run different commands on basis +of type + + +# need a way to quiery live det runs +# add det run state tracking (alive or dead) +# die at the end of a det run... -rerun makes it live again ?? +# -processed -chip (no -det_id) returns a list of ready det_id/class_id pairs + Index: /branches/rel-1-0/ippTools/doc/dettool.txt =================================================================== --- /branches/rel-1-0/ippTools/doc/dettool.txt (revision 11645) +++ /branches/rel-1-0/ippTools/doc/dettool.txt (revision 11645) @@ -0,0 +1,181 @@ +# returns a list of unassociated rawDetrendExp (rawDetrendExps that don't +# correspond to a exp_id in detInputExp) +dettool -pending -exp_type -inst .. + +# define a new detRun semi-automatically by selecting the component exps with +# the specified search criteria +dettool -definebyquery -det_type ... + +# create a detRun with the specified input exp_ids +dettool -definebyexp -det_type -exp_id 1 -exp_id 2 ... + +-- +# returns a list of rawImfiles that are associated with an detInputExp that do +# not appear in detProcessedImfiles +dettool -raw + +# add an imfile to detProcessedImfiles +dettool -addprocessed + +-- +# the list of class_ids that are ready to be stacked +# returns a list of det_id/iter/class_id/det_typs where class_id is unique per +# det_id (not a list of all exp_ids/class_ids) that do not have an entry in +# detStackedImfile +# results are masked by entries in detResidExpAnalysis +# may need to check detResidExp's even on iter 0 - ??? +dettool -tostack + +# list all of imfiles from different exps with the same class_id that have +# been processed +dettool -processed -wholestack -det_id ... + +# adds a stacked class_id and and optionally marks it for normalization +detool -addstacked [-pleasenormalize] -det_id -iter -class_id ... + +-- +# returns a list of det_ids/iters where the entire set of class_ids has been +# processed +dettool -tonormalize + +# returns the complete list of processed imfiles for the specified det_id +dettool -processed -unmask -det_id -iter + +# add per stacked imfile normalization statistics +# O: this could accept a MDC doc as input from stdin +dettool -addnormstat -det_id -iter -class_idi -norm F32 + +-- +# returns a list of det_ids/iter/class_id/uri from detNormStats. Stops +# returning entries after the class_is is inserted into detNormalizedImfile +dettool -normstat -det_id + +# inserts an entry into detNormalizedImfile +dettool -addnormalized -det_id -ter -class_id -uri + +-- +# returns a list of processed imfiles that have also been normalized with +# processed imfiles being masked out per detResidImfileAnalysis +# also returns stackedimfiles that have -pleasenormalize set to false +dettool -toresid -det_id -iter + +# inserts a per det_id/iter/exp_id/class_id residual +dettool -addresidimfile -det_id -exp_id -class_id -iter -stat F32 -stat_stdev F32 -uri -b1 -b2 + +# returns a list of det_id/iter/exp_ids but only for exp_ids that have resids +# for all of their imfiles +dettool -toresidexp + +# returns a list of residual imfile data +dettool -residimfile -det_id -iter -exp_id + +# inserts into detResidExpAnalysis +# sets the accept accept bool unless -reject is specified +dettool -addresidexp -det_id -iter -exp_id -jpeg1 -jpeg2 -stat F32 -stat_stdev +F32 [-reject] + +-- +# lists det_id/iter for detruns that have completed all residexps for their +# current iteration +dettool -residdetrun + +# lists all residexps for the given det_id/iter +dettool -residexp -det_id -iter + +# updates the given residexp to be accept/reject +dettool -updateresid -det_id -iter [-reject] + +# changes the state of the detrun. -rerun cause a new iteration to be started +# with the accepted with just the exp_ids +dettool -updatedetrun -det_id -iter -stat F32 -stat_stdev [-rerun|-ok|-stop] + +# manual starts a new detrun iteration starting from the specified -iter +dettool -rerun -det_id -iter -accept expid -accept expid -reject expid -reject expid +dettool -rerun -det_id -iter + +# detInputExp needs to track... +det_id iter exp_id use bool accept bool + + + +dettool + + -define | creates a a new detrend Run + input: + exp IDs? + type of det run? + creates a new detRun ID automatically + + output: the new det_id + + -raw | lists raw imfiles needing to be processed + input: + det ID + exp ID + class ID + iteration number? + output: list of unprocessed imfiles + + -addprocessed? | marks a raw imfile as processed + input: + det ID + exp ID + class ID + iteration number? + uri + stat + recipe + output: outout on error only + -processed | lists processed imfiles + input: + det ID + exp ID + class ID + iteration number? + output: list of processed imfiles + -addstacked | adds a stacked imfiles + input: + det ID + class ID + iteration number? + URI + recipe + stats + output: output on error only + -stacked | lists stacked imfiles + input: + det ID + class ID + iteration number? + output: list of stacked imfiles + -stacked | lists stacked imfiles + input: + det ID + class ID + iteration number? + output: list of stacked imfiles + -stackedframe | lists stacked imfiles for COMPLETE frames + input: + det ID // required for simplicity + iteration number? + output: list of stacked imfiles + -addmaster | adds a master imfiles + input: + det ID + class ID + iteration number? + URI + recipe + stats + output: output on error only + -master | lists master imfiles + input: + det ID // required for simplicity + class ID + iteration number? + output: a list of master imfiles + -masterframe | lists master imfiles for COMPLETE frames + input: + det ID // required for simplicity + iteration number? + output: a list of master imfiles Index: /branches/rel-1-0/ippTools/doc/dsfilesetls =================================================================== --- /branches/rel-1-0/ippTools/doc/dsfilesetls (revision 11645) +++ /branches/rel-1-0/ippTools/doc/dsfilesetls (revision 11645) @@ -0,0 +1,6 @@ +#!/usr/bin/env perl + +print "# uri fileid bytes md5sum type \n"; +print "http://example.org/1 foo1.0 123 asadfasdfasdfasfdasdf CELL\n"; +print "http://example.org/2 foo1.1 123 asadfasdfasdfasdfasdf CELL\n"; +print "http://example.org/3 foo1.2 123 asadfasdfasdfasdfasdf CELL\n"; Index: /branches/rel-1-0/ippTools/doc/dsproductls =================================================================== --- /branches/rel-1-0/ippTools/doc/dsproductls (revision 11645) +++ /branches/rel-1-0/ippTools/doc/dsproductls (revision 11645) @@ -0,0 +1,8 @@ +#!/usr/bin/env perl + +print "# uri fileset datetime type\n"; +print "http://example.org/1 foo1 2006-04-17T00:00:12 OBJECT\n"; +print "http://example.org/2 foo2 2006-04-17T00:00:12 OBJECT\n"; +print "http://example.org/3 foo3 2006-04-17T00:00:12 OBJECT\n"; + + Index: /branches/rel-1-0/ippTools/doc/install.pod =================================================================== --- /branches/rel-1-0/ippTools/doc/install.pod (revision 11645) +++ /branches/rel-1-0/ippTools/doc/install.pod (revision 11645) @@ -0,0 +1,251 @@ +=pod + +=head1 SYNOPSIS + +C is a collection of utilties for tracking image processing +tasks. Very little decision making logic is built into these programs. The +intent is for higher level programs and scripts to make the real I +and then use C to encode those choices in a persistent manner. + +=head1 INSTALL + +=head2 Prerequisites + +C depends on C (built with C support), C, and +C and requires a C capable compiler. + +XXX document required library versions + +=head2 Build Procedure + +C is built with the so-called suite of autotools. The build +procedure is pretty typical for software built with these tools. E.g. + + ./configure + make + make install + +If you checked out the sources directly from CVS you need to run the +C script. C will run C for you so that +changes the build procedure to just: + + sh autogen.sh + make + make install + +=head2 jhbuild Build Procedure + +If you are using one of the C modulesets jhbuild can build and install C for you. E.g. + + jhbuild build pxtools + +=head2 Database Setup + +All of the C' persistent storage is ultimately done through C's +C facilties. In turn, C uses C to impliment storage (and +relational querying). C requires that each namespace (aka database) is +configured with basic access controls. These include usernames, passwords, +access filtering, and privilege restrictions. Below is a simple (but somewhat +dangerous) example of configuring a database named C. + + mysql -u root mysql -p + (enter your MySQL root password) + + DROP DATABASE IF EXISTS ipp; + CREATE DATABASE ipp; + GRANT ALL PRIVILEGES ON *.* TO 'ippuser'@'localhost' IDENTIFIED BY 'ipppass'; + FLUSH PRIVILEGES; + quit + +After setting up the database you should verify that you can connect to with the username and password you configured + + mysql -u ipp ipp -pipp + +Next, C needs to be told where to to find MySQL, the name of the +database you created, the username, and the password you created. C +uses the typical C configuration system. The simplest method is to +add the follow lines to your $HOME/.ipprc file. + + ### Database configuration + DBSERVER STR localhost # Database host name (for psDBInit) + DBNAME STR ipp + DBUSER STR ipp # Database user name (for psDBInit) + DBPASSWORD STR ipp # Database password (for psDBInit) + +The C utilities require the a schema is pre-loaded into the database. The C tool can do this for you. E.g. + + pxadmin -create + +You should always verify that the scheme was actually loaded. E.g. + + mysql -u ipp ipp -pipp + Welcome to the MySQL monitor. Commands end with ; or \g. + Your MySQL connection id is 6 to server version: 4.1.20-log + + Type 'help;' or '\h' for help. Type '\c' to clear the buffer. + + mysql> show tables; + +------------------------+ + | Tables_in_ipp | + +------------------------+ + | detInputExp | + | detMasterFrame | + | detMasterImfile | + | detNormalizedImfile | + | detProcessedImfile | + | detResidExpAnalysis | + | detResidImfileAnalysis | + | detRun | + | detStackedImfile | + | newExp | + | newImfile | + | p1PendingExp | + | p2DoneExp | + | p2DoneImfile | + | p2PendingExp | + | p2PendingImfile | + | p3PendingExp | + | pzPendingExp | + | pzPendingImfile | + | rawDetrendExp | + | rawImfile | + | rawScienceExp | + | summitExp | + +------------------------+ + 23 rows in set (0.00 sec) + +=head1 USE + +The workflow is as follows: + +XXX + +=head2 Phase 0 tools + +The primary C utility for handling Phase 0 is C. This tool +has two modes C<-pending> and C<-update>. The pending mode lists all +Cs that are part of an exposure that I yet been classified +as either science or detrend data. + +For example: + + p0search -pending + +Would output something like: + + newImfile MULTI # + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 0 + uri STR file://0 + END + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 1 + uri STR file://1 + END + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 2 + uri STR file://2 + END + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 3 + uri STR file://3 + END + +To declare C as science data, you would invoke C like this: + + p0search -update -exp_id 10 + +Note that by default C<-update> flags the exposure as science data. To declare it to be detrend data the command would be: + + p0search -update -exp_id 10 -detrend + +Also keep in mind that this is a one time declaration. Once it has been +declared as either science or detrend data this decision can not be undone +(without manually editing the database) + +=head2 Phase 1 tools + +These tools are not yet in use. + +=head2 Phase 2 tools + +C handles all of state for Phase 2. It has four basic modes: + +=over 4 + +=item * C<-quick> + +=item * C<-define> + +=item * C<-pending> + + p0search -pending + +=item * C<-done> + +=head2 Administrative tools + +=over 4 + +=item * C + +This tool can be used to load/unload the database schema. It has three modes of operation: + +=over 4 + +=item * C<-create> + +Loads the database schema. + +=item * C<-delete> + +Deletes the database schema. + +=item * C<-recreate> + +Deletes the database schema then reloads it. + +=back + +=item * C + +I data directly into the database. It has one mode parameter for each database table. Note: support for only a few tables has been impliemented. + +=back + +=head2 Bypassing 'Summit Copy' + +The C suite assumes that exposures and image data are initially +registered with the C set of I tools. Sometimes this is +inconvenient as you want to work with local data and bypass this set. This is +what the C utility is for. C directly inserts data into +the database, side stepping any ordering rules or sanity checking. + +The end result of the C utilities is that new exposures are registered in the C table and raw image data is in the C table. You can insert this data yourself, preparing the system to run I. + +Here is an example of inserting an exposure that has 4 component image files. + + #!//bin/sh + + inject="./pxinject" + + $inject -newExp -exp_id 10 -inst gpc -telescope ps1 -exp_type object -imfiles 4 + + `$inject -newImfile -exp_id 10 -class OTA -class_id 0 -uri file://0` + `$inject -newImfile -exp_id 10 -class OTA -class_id 1 -uri file://1` + `$inject -newImfile -exp_id 10 -class OTA -class_id 2 -uri file://2` + `$inject -newImfile -exp_id 10 -class OTA -class_id 3 -uri file://3` + +=cut Index: /branches/rel-1-0/ippTools/doc/p0tools.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/p0tools.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p0tools.sh (revision 11645) @@ -0,0 +1,71 @@ + +## PanTasks scripts for Phase 0 + +# tasks: p0ready p0update p0cleanup +# +# queues: +# p0pending: (expID) (cameraID) (state) +# +# globals: + +# p0pending : search for images to be processed +task p0pending + command p0search -pending + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 5 + periods -poll 1 + periods -timeout 4 + + # success + task.exit 0 + local i Nstdout + # keep only new, unique entries (URL is key) + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line + queuepush p0pending -uniq -key 0 "$line new" + end + end +end + +# p0update : submit an exposure to be updated +task p0update + periods -exec 1 + periods -timeout 10 + + task.exec + local Npending + queuesize p0pending -var Npending + if ($Npending == 0) break + + queuepop p0pending -var line -key 2 new + if ("$line" == "NULL") break + + list tmp -split $line + $expID = $tmp:0 + $camera = $tmp:1 + queuepush p0pending "$expID $camera run" + + host any + command p0search -update $expID $camera + end + + # success + task.exit 0 + queueinit stdout + queuepush p0pending $taskargs:0 $taskargs:1 done + end +end + +# p0update : search for images to be processed +task p0cleanup + command p0search -cleanup + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 30 + periods -poll 1 + periods -timeout 10 +end Index: /branches/rel-1-0/ippTools/doc/p0tools.txt =================================================================== --- /branches/rel-1-0/ippTools/doc/p0tools.txt (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p0tools.txt (revision 11645) @@ -0,0 +1,40 @@ + +Phase 0 pipeline tools: + +p0search -pending : + * examine the new.imfiles,new.exposures tables and select exposures ready for analysis + * output is: (expID) (camera) + +p0search -update (expID) (camera): + * select a the corresponding images from the new.imfiles/new.exposures table + * extract the specified header information + * search the summit metadata db tables + * write an entry to the raw.imfiles and raw.exposure tables + * set the state on the new.imfiles,new.exposure tables + * based on the camera config information; + * add an entry to the p1.pending table (mosaic) + - or + * add an entry to the p2.pending table (single) + +p0search -stats (expID) (camera): + * select a specified image in the new.imfiles/new.exposures table + * extract the specified header information + * search the summit metadata db tables + * report the image stats + [-update without output to MDDB] + +p0search -mkraw (expID) (camera): + * select a specified image in the new.imfiles/new.exposures table + * extract the specified header information + * search the summit metadata db tables + * write an entry to the raw.imfiles and raw.exposure tables + * set the state on the new.imfiles,new.exposure tables + +p0search -cleanup: + * remove completed entries from the new.imfiles,new.exposure tables + +** note : the division of -pending and -update allows separate processes + to be examining the image headers and measuring some stats. these + jobs can be run via pcontrol to reduce the load on the PanTasks + machines + Index: /branches/rel-1-0/ippTools/doc/p1tools.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/p1tools.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p1tools.sh (revision 11645) @@ -0,0 +1,72 @@ + +## PanTasks scripts for Phase 1 + +# tasks: p1pending p1astro p1done +# +# queues: +# p1pending: (expID) (version) (cameraID) (state) +# +# globals: + +# p1pending : search for images to be processed +task p1pending + command p1search -pending + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 5 + periods -poll 1 + periods -timeout 4 + + # success + task.exit 0 + local i Nstdout + # keep only new, unique entries (URL is key) + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line + queuepush p1pending -uniq -key 0 "$line new" + end + end +end + +# p1update : submit an exposure to be updated +task p1astro + periods -exec 1 + periods -timeout 10 + + task.exec + local expID version camera Npending + queuesize p1pending -var Npending + if ($Npending == 0) break + + queuepop p1pending -var line -key 2 new + if ("$line" == "NULL") break + + list tmp -split $line + $expID = $tmp:0 + $version = $tmp:1 + $camera = $tmp:2 + queuepush p1pending "$expID $version $camera run" + + host any + command p1astro $expID $version $camera + end + + # success + task.exit 0 + queueinit stdout + queuepush p1pending $taskargs:0 $taskargs:1 $taskargs:2 done + end +end + +# p1update : search for images to be processed +task p1done + command p1search -done + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 30 + periods -poll 1 + periods -timeout 10 +end Index: /branches/rel-1-0/ippTools/doc/p1tools.txt =================================================================== --- /branches/rel-1-0/ippTools/doc/p1tools.txt (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p1tools.txt (revision 11645) @@ -0,0 +1,16 @@ + +Phase 1 pipeline tools: + +p1search -define [constraints]: + * examine the raw.exposures tables and select exposures matching the given criteria + * add entries which are allowed (mosaic) to the p1.pending table + +p1search -pending : + * examine the p1.pending table and select exposures waiting for p1 + * output: lines consisting of: + (expID) (p1version) (camera) + +p1search -done : + * select completed entries in the p1.pending table + * move to the p1.done table + * add new entry to the p2.pending tables Index: /branches/rel-1-0/ippTools/doc/p2search.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/p2search.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p2search.sh (revision 11645) @@ -0,0 +1,8 @@ +#!/bin/csh -f + +echo "# expID class classID url" +echo "654320o fpa megacam 654320o.fits" +echo "654321o fpa megacam 654321o.fits" +echo "654322o fpa megacam 654322o.fits" +echo "654323o fpa megacam 654323o.fits" +echo "654324o fpa megacam 654324o.fits" Index: /branches/rel-1-0/ippTools/doc/p2tools.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/p2tools.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p2tools.sh (revision 11645) @@ -0,0 +1,137 @@ + +## PanTasks scripts for Phase 2 + +# tasks: p2pending p2update ppImage +# +# queues: +# p2pending: expID class classID url +# +# globals: + +queueinit stdout +queueinit stderr +queueinit p2pending + +macro p2parse + if ($0 != 7) + echo "usage: p2parse (key) (expID) (class) (classID) (url) (state)" + break + end + $expID = $2 + $class = $3 + $classID = $4 + $url = $5 + $key = $expID.$classID +end + +# p2pending : search for images to be processed +task p2pending + command p2search.sh -pending + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 5 + periods -poll 1 + periods -timeout 4 + + # success + task.exit 0 + local i Nstdout + + # pop off the header line + queuepop stdout + + # keep only new, unique entries (URL is key) + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line + p2parse unknown $line new + queuepush p2pending -uniq -key 0 "$key $line new" + end + echo "loaded p2pending, current state:" + queueprint p2pending + end +end + +# ppImage : submit an image for processing by ppImage +task ppImage + periods -exec 1 + periods -timeout 120 + + task.exec + local Npending + queuesize p2pending -var Npending + if ($Npending == 0) break + + # add in system status interruptions later + # if ($network == 0) break + + queuepop p2pending -var line -key 5 new + if ("$line" == "NULL") break + p2parse $line + + echo "new ppImaage: $key $expID $class $classID $url run" + queuepush p2pending "$key $expID $class $classID $url run" + + # need to decide real name for this function + # $host = `chip.host $camera $chip` + # host $host + + host local + echo command ppImage.sh $key $url $classID + command ppImage.sh $key $url $classID + end + + # success + task.exit 0 + # if ppImage updates the DB: + echo "done with $taskarg:1" + queueinit stdout + queuepop p2pending -key 0 $taskarg:1 -var line + echo "got line: $line" + + if ("$line" == "NULL") + echo "missing entry in p2pending queue?" + break + end + + p2parse $line + queuepush p2pending "$key $expID $class $classID $url done" + echo queuepush p2pending "$key $expID $class $classID $url done" + + # if ppImage does NOT update the DB: + end + +end + +# p2update : search for images to be processed +task p2update + command p2search -update + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 100 + periods -poll 1 + periods -timeout 4 + + # task is never allowed to run + task.exec + break + end +end + +# p2done : search for images to be processed +task p2done + command p2search -done + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 100 + periods -poll 1 + periods -timeout 4 + + # task is never allowed to run + task.exec + break + end +end Index: /branches/rel-1-0/ippTools/doc/p2tools.txt =================================================================== --- /branches/rel-1-0/ippTools/doc/p2tools.txt (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p2tools.txt (revision 11645) @@ -0,0 +1,41 @@ + +Phase 2 pipeline tools: + +p2search -quick + * search for images which match in raw.exp,raw.imfiles + * output in format which can be used by ppImage pantasks script + +p2search -define [options] + * input: searches mddb:raw_exposures,raw_images + - compares against p2Pending[Exp|Imfile] + - compares against p2Done[Exp|Imfile] + - XXX should be implemented as a special wrapper function that + implements this comparison as a SQL query + * output: updates mddb:P2_exposures_pending,P2_images_pending + * alternative output: identical to p2pending + +p2search -pending + * input: searches mddb:P2_exposures_pending,P2_images_pending + * output: Nlines consisting of: + (URL) (expID) (class) + * options: ? + +p2search -update + * examine the imfiles and identify any completed exposures + +p2search -done + * get exp_id/class/class_id from the CLI + * add the completed imfile to the p2DoneImfile tables + * remove corresponding entries from the p2PendingImfile table + * check to see if any p2PendingExps have no associated p2PendingImfiles + if so move the p2PendingExp(s) to p2DoneExp +// * send new entry to the pending p3 table + +ppImage file://path/filename file://path/outroot -recipe (recipe) +ppImage neb://nebname neb://outroot -recipe (recipe) + +restriction options: + -time (start) (stop) + -camera (camera) + -region (ra,dec) (ra,dec) + Index: /branches/rel-1-0/ippTools/doc/p3tools.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/p3tools.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p3tools.sh (revision 11645) @@ -0,0 +1,72 @@ + +## PanTasks scripts for Phase 3 + +# tasks: p3pending p3astro p3done +# +# queues: +# p3pending: (expID) (version) (cameraID) (state) +# +# globals: + +# p3pending : search for images to be processed +task p3pending + command p3search -pending + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 5 + periods -poll 1 + periods -timeout 4 + + # success + task.exit 0 + local i Nstdout + # keep only new, unique entries (URL is key) + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line + queuepush p3pending -uniq -key 0 "$line new" + end + end +end + +# p3update : submit an exposure to be updated +task p3astro + periods -exec 1 + periods -timeout 10 + + task.exec + local expID version camera Npending + queuesize p3pending -var Npending + if ($Npending == 0) break + + queuepop p3pending -var line -key 2 new + if ("$line" == "NULL") break + + list tmp -split $line + $expID = $tmp:0 + $version = $tmp:1 + $camera = $tmp:2 + queuepush p3pending "$expID $version $camera run" + + host any + command p3astro $expID $version $camera + end + + # success + task.exit 0 + queueinit stdout + queuepush p3pending $taskargs:0 $taskargs:1 $taskargs:2 done + end +end + +# p3update : search for images to be processed +task p3done + command p3search -done + host local + + # timeout shorter than exec so jobs do not build up + periods -exec 30 + periods -poll 1 + periods -timeout 10 +end Index: /branches/rel-1-0/ippTools/doc/p3tools.txt =================================================================== --- /branches/rel-1-0/ippTools/doc/p3tools.txt (revision 11645) +++ /branches/rel-1-0/ippTools/doc/p3tools.txt (revision 11645) @@ -0,0 +1,22 @@ + +Phase 3 pipeline tools: + +p3search -define : + * examine the raw.exposures tables and select exposures matching the given criteria + * add entries which are allowed (mosaic) to the p3.pending table + +p3search -quick : + * examine the raw.exposures tables and select exposures matching the given criteria + * return list of entries for p3 processing + * output: lines consisting of: + (expID) (p3version) (camera) + +p3search -pending : + * examine the p3.pending exposures table and select exposures waiting for p3 + * return list of entries for p3 processing + * output: lines consisting of: + (expID) (p3version) (camera) + +p3search -done : + * select completed entries in the p3.pending table + * move to the p3.done table Index: /branches/rel-1-0/ippTools/doc/ppImage.sh =================================================================== --- /branches/rel-1-0/ippTools/doc/ppImage.sh (revision 11645) +++ /branches/rel-1-0/ippTools/doc/ppImage.sh (revision 11645) @@ -0,0 +1,7 @@ +#!/bin/csh -f + +sleep 3 +echo $* +echo done + +exit 0 Index: /branches/rel-1-0/ippTools/doc/summit_copy.txt =================================================================== --- /branches/rel-1-0/ippTools/doc/summit_copy.txt (revision 11645) +++ /branches/rel-1-0/ippTools/doc/summit_copy.txt (revision 11645) @@ -0,0 +1,49 @@ +The summit copy phase (phase Z or pz) consists of three basic steps. +Discovering what exposures and available, discovering what image files are +associated with those exposures, and the downloading of image files. All +"external" communication with the DataStore is done with three utilities; +pzgetexp, pzgetimfiles & dsget. While local state information is accessed and +updated via the 'pztool' utility. + +The basic work follow is: + +pzgetexp retrieves a list of all of the exposures (filesets) available from a +DataStore "product". If any previously unknown file exposures are found they +are added to the pzPendingExp tables and the summitExp table. New entries to +the summitExp table have a imfile value of NULL as this information can't yet +be known. If any exposures are already in the summitExp table the most recent +exp_id is used to constrain the result set requested from the DataStore. + + ./pzgetexp -uri http://otis/ds/skyprobe/ -inst skyprobe -telescope ps1 + +The results of pzgetexp and accessible with pztool -pendingexp`. Which will +return a list of new exposures from the pzPendingExp table. + + ./pztool -pendingexp + +The output of which is used to invoke pzgetimfiles. + +pzgetimfiles retrieves a list of all of the imfiles (files) in an exposure +(fileset) from the DataStore. It then updates the imfiles count in the +summitExp table and adds each imfile to the pzPendingImfile table. + +./pzgetimfiles -uri http://otis/ds/skyprobe/sep8_twi_05/ -filesetid sep8_twi_05 -inst skyprobe -telescope ps1 + +pztool -pendingimfile returns the list of imfiles that need to be downloaded. + + ./pztool -pendingimfile + +The results of which are used to invoke dsget and pztool -copydone after dsget +completes it's download. + +dsget from the 'DataStore' package takes the follow parameters. + + dsget --uri --filename [--bytes ] [--md5 ] + +Where "filename" may be either just a filename relative to the CWD or a fully +qualified path. After dsget has copied the file pztool -copydone needs to be +invoked + + ./pztool -copydone -exp_id sep8_twi_03 -inst skyprobe -telescope ps1 -class chip -class_id sep8_twi_03.fits -uri file///data/alala/sep8_twi_03.fits + +And then we're Pau. Index: /branches/rel-1-0/ippTools/m4/ac_prog_perl_modules.m4 =================================================================== --- /branches/rel-1-0/ippTools/m4/ac_prog_perl_modules.m4 (revision 11645) +++ /branches/rel-1-0/ippTools/m4/ac_prog_perl_modules.m4 (revision 11645) @@ -0,0 +1,53 @@ +dnl @synopsis AC_PROG_PERL_MODULES([MODULES], [ACTION-IF-TRUE], [ACTION-IF-FALSE]) +dnl +dnl Checks to see if the the given perl modules are available. If true +dnl the shell commands in ACTION-IF-TRUE are executed. If not the shell +dnl commands in ACTION-IF-FALSE are run. Note if $PERL is not set (for +dnl example by calling AC_CHECK_PROG, or AC_PATH_PROG), +dnl AC_CHECK_PROG(PERL, perl, perl) will be run. +dnl +dnl Example: +dnl +dnl AC_CHECK_PERL_MODULES(Text::Wrap Net::LDAP, , +dnl AC_MSG_WARN(Need some Perl modules) +dnl +dnl @category InstalledPackages +dnl @author Dean Povey +dnl @version 2002-09-25 +dnl @license AllPermissive + +AC_DEFUN([AC_PROG_PERL_MODULES],[ + ac_perl_modules="$1" + # Make sure we have perl + if test -z "$PERL"; then + AC_CHECK_PROG(PERL,perl,perl) + fi + + if test "x$PERL" != x; then + ac_perl_modules_failed=0 + for ac_perl_module in $ac_perl_modules; do + AC_MSG_CHECKING(for perl module $ac_perl_module) + + # Would be nice to log result here, but can't rely on autoconf + # internals + $PERL "-M$ac_perl_module" -e exit > /dev/null 2>&1 + if test $? -ne 0; then + AC_MSG_RESULT(no); + ac_perl_modules_failed=1 + else + AC_MSG_RESULT(ok); + fi + done + + # Run optional shell commands + if test "$ac_perl_modules_failed" = 0; then + : + $2 + else + : + $3 + fi + else + AC_MSG_WARN(could not find perl) + fi +]) Index: /branches/rel-1-0/ippTools/pxtools.pc.in =================================================================== --- /branches/rel-1-0/ippTools/pxtools.pc.in (revision 11645) +++ /branches/rel-1-0/ippTools/pxtools.pc.in (revision 11645) @@ -0,0 +1,11 @@ +prefix=@prefix@ +exec_prefix=@exec_prefix@ +libdir=@libdir@ +includedir=@includedir@/@PACKAGE_NAME@ + +Name: @PACKAGE_NAME@ +Description: Pan-STARRS IPP Tools Library +Version: @VERSION@ +Requires: pslib psmodules ippdb +Libs: -L${libdir} -lpxtools +Cflags: -I${includedir} @PXTOOLS_CFLAGS@ Index: /branches/rel-1-0/ippTools/scripts/.cvsignore =================================================================== --- /branches/rel-1-0/ippTools/scripts/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippTools/scripts/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/ippTools/scripts/camtest.sh =================================================================== --- /branches/rel-1-0/ippTools/scripts/camtest.sh (revision 11645) +++ /branches/rel-1-0/ippTools/scripts/camtest.sh (revision 11645) @@ -0,0 +1,11 @@ +#!//bin/sh + +set -o verbose + +p2test.sh || exit 1 + +p3tool -pendingexp || exit 1 +p3tool -pendingimfile || exit 1 + +p3tool -addprocessedexp -exp_tag t10.1 -uri file://p3p -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -sigma_ra 1 -sigma_dec 2 -nastro 42 -b1_uri file:///foo -b2_uri file://bar -zp_mean 1 -zp_stdev 2 || exit 1 +p3tool -addprocessedexp -exp_tag t11.2 -uri file://p3p -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -sigma_ra 1 -sigma_dec 2 -nastro 42 -b1_uri file:///foo -b2_uri file://bar -zp_mean 1 -zp_stdev 2 || exit 1 Index: /branches/rel-1-0/ippTools/scripts/chiptest.sh =================================================================== --- /branches/rel-1-0/ippTools/scripts/chiptest.sh (revision 11645) +++ /branches/rel-1-0/ippTools/scripts/chiptest.sh (revision 11645) @@ -0,0 +1,17 @@ +#!//bin/sh + +set -o verbose + +p0test.sh || exit 1 + +p2tool -pendingimfile || exit 1 + +for ID in `seq 0 3`; do + p2tool -addprocessedimfile -exp_tag t10.1 -class_id $ID -uri file://p2p-t10.$ID -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -b1_uri file:///foo -b2_uri file://bar || exit 1 +done; + +for ID in `seq 0 3`; do + p2tool -addprocessedimfile -exp_tag t11.2 -class_id $ID -uri file://p2p-t11.$ID -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -b1_uri file:///foo -b2_uri file://bar || exit 1 +done; + +p2tool -pendingimfile || exit 1 Index: /branches/rel-1-0/ippTools/scripts/dettest.sh =================================================================== --- /branches/rel-1-0/ippTools/scripts/dettest.sh (revision 11645) +++ /branches/rel-1-0/ippTools/scripts/dettest.sh (revision 11645) @@ -0,0 +1,61 @@ +#!//bin/sh + +set -o verbose + +det_id=1 + +./p0test.sh -detrend || exit 1 + +det_id=`dettool -definebyquery -det_type flat -exp_type fooflat -filter foo -airmass_min 1 -airmass_max 10 -exp_time_min 10 -exp_time_max 30.0 -workdir file::///some/path -simple | cut -f1 -d" "` || exit 1 + +dettool -raw || exit 1 + +for ID in `seq 0 3` ; do + dettool -addprocessedimfile -det_id $det_id -exp_tag t11.2 -class_id $ID -uri file://proc-$ID -recip myrecip -bg 1 -bg_stdev 1 -bg_mean_stdev 1 || exit 1 +done; + +#for ID in `seq 0 3` ; do +# dettool -addprocessedimfile -det_id $det_id -exp_tag t11 -class_id $ID -uri file://proc-$ID -recip myrecip -bg 1 -bg_stdev 1 -bg_mean_stdev 1 || exit 1 +#done; + + +dettool -tostacked || exit 1 + +for ID in `seq 0 3` ; do + dettool -addstacked -det_id $det_id -uri file://stacked-$ID -class_id $ID -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 || exit 1 +done; + +dettool -tonormalizedstat || exit 1 + +for ID in `seq 0 3` ; do + dettool -addnormalizedstat -det_id $det_id -class_id $ID -norm 0.12345 || exit 1 +done; + +dettool -tonormalize || exit 1 +for ID in `seq 0 3` ; do + dettool -addnormalizedimfile -det_id $det_id -class_id $ID -uri file://normalized-$ID -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -b1_uri banana1 -b2_uri banannnnnaaa2 || exit 1 +done; + +dettool -tonormalizedexp || exit 1 +dettool -addnormalizedexp -det_id $det_id -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -b1_uri file://normalizedexp -b2_uri file://normlalizedexp|| exit 1 + +dettool -toresidimfile || exit 1 + +for ID in `seq 0 3` ; do + dettool -addresidimfile -det_id $det_id -exp_tag t11.2 -class_id $ID -recip myrecip -bg 1 -bg_stdev 1 -bg_mean_stdev 1 -uri file://resid-$ID || exit 1 +done; + +dettool -toresidexp || exit 1 +dettool -addresidexp -det_id $det_id -exp_tag t11.2 -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -b1_uri jpeg1 -b2_uri jpeg2 || exit 1 +#dettool -addresidexp -det_id $det_id -exp_tag t11.2 -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -b1_uri jpeg1 -b2_uri jpeg2 -reject || exit 1 + +dettool -residdetrun || exit 1 +dettool -residexp || exit 1 +dettool -updateresidexp -det_id $det_id -iteration 0 -recip yourrecipe || exit 1 +dettool -updateresidexp -det_id $det_id -iteration 0 -exp_tag t11.2 -reject || exit 1 +dettool -updateresidexp -det_id $det_id -iteration 0 -exp_tag t11.2 || exit 1 + +dettool -adddetrunsummary -det_id $det_id -iteration 0 -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -accept || exit 1 +dettool -residexp || exit 1 + +dettool -updatedetrun -det_id $det_id -state stop || exit 1 Index: /branches/rel-1-0/ippTools/scripts/regtest.sh =================================================================== --- /branches/rel-1-0/ippTools/scripts/regtest.sh (revision 11645) +++ /branches/rel-1-0/ippTools/scripts/regtest.sh (revision 11645) @@ -0,0 +1,40 @@ +#!//bin/sh + +set -o verbose + +inject="pxinject" +p0tool="p0tool" + +echo -e "YES\nipp\n\n" | pxadmin -recreate || exit 1 + +exp_tag1=`$inject -newExp -exp_id t10 -inst gpc -telescope ps1 -imfiles 4 -workdir file::///some/path -simple` || exit 1 + +for ID in `seq 0 3`; do + $inject -newImfile -exp_tag $exp_tag1 -class OTA -class_id $ID -uri file://$ID || exit 1 +done; + +exp_tag2=`$inject -newExp -exp_id t11 -inst gpc -telescope ps1 -imfiles 4 -workdir file::///some/path -simple` || exit 1 + +for ID in `seq 0 3`; do + $inject -newImfile -exp_tag $exp_tag2 -class OTA -class_id $ID -uri file://$ID || exit 1 +done; + +$p0tool -pendingimfile || exit 1 + +for ID in `seq 0 3`; do + $p0tool -addprocessedimfile -exp_tag $exp_tag1 -class OTA -class_id $ID -filter r -airmass 10 -ra 1 -decl 2 -exp_time 0 -bg 1 -bg_stdev 1 -bg_mean_stdev 10 -alt 10 -az 10 -ccd_temp 10 -posang 10 -object dog -dateobs 2006-10-20T10:10:10Z || exit 1 +done; + +$p0tool -pendingexp|| exit 1 + +$p0tool -addprocessedexp -exp_tag $exp_tag1 -filter r -airmass 10 -ra 1 -decl 2 -exp_type bias -exp_time 0 -bg 10 -bg_stdev 1 -bg_mean_stdev 10 -alt 10 -az 10 -ccd_temp 45 -posang 10 -object dog -dateobs "2006-10-20T10:10:10Z" -label foobar $* || exit 1 + +$p0tool -pendingimfile || exit 1 + +for ID in `seq 0 3`; do + $p0tool -addprocessedimfile -exp_tag $exp_tag2 -exp_type object -class OTA -class_id $ID -filter r -airmass 10 -ra 1 -decl 2 -exp_time 0 -bg 1 -bg_stdev 1 -bg_mean_stdev 10 -alt 10 -az 10 -ccd_temp 10 -posang 10 -object dog -dateobs 2006-10-20T10:10:10Z || exit 1 +done; + +$p0tool -pendingexp|| exit 1 + +$p0tool -addprocessedexp -exp_tag $exp_tag2 -filter r -airmass 11 -ra 1 -decl 2 -exp_type bias -exp_time 0 -bg 11 -bg_stdev 1 -bg_mean_stdev 11 -alt 11 -az 11 -ccd_temp 45 -posang 11 -object dog -dateobs 2006-10-20T10:10:10Z -label foobar $* || exit 1 Index: /branches/rel-1-0/ippTools/src/.cvsignore =================================================================== --- /branches/rel-1-0/ippTools/src/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippTools/src/.cvsignore (revision 11645) @@ -0,0 +1,24 @@ +.deps +.gdb_history +.libs +Makefile +Makefile.in +config.h +config.h.in +stamp-h1 +*.la +*.lo +p0tool +p1tool +p2tool +pxadmin +pzgetexp +pzgetimfiles +pztool +dettool +pxinject +detselect +p3tool +p4tool +pxtoolsErrorCodes.c +pxtoolsErrorCodes.h Index: /branches/rel-1-0/ippTools/src/Makefile.am =================================================================== --- /branches/rel-1-0/ippTools/src/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippTools/src/Makefile.am (revision 11645) @@ -0,0 +1,136 @@ + +bin_PROGRAMS = \ + detselect \ + dettool \ + p0tool \ + p2tool\ + p3tool\ + p4tool\ + pxadmin \ + pxinject \ + pzgetexp \ + pzgetimfiles \ + pztool + +include_HEADERS = \ + pxtoolsErrorCodes.h \ + pxtools.h + +noinst_HEADERS = \ + dettool.h \ + detselect.h \ + p0tool.h \ + p1tool.h \ + p2tool.h \ + p3tool.h \ + p4tool.h \ + pxadmin.h \ + pxinject.h \ + pxtag.h \ + pzgetexp.h \ + pzgetimfiles.h \ + pztool.h + +lib_LTLIBRARIES = libpxtools.la +libpxtools_la_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +libpxtools_la_LIBS = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) +libpxtools_la_LDFLAGS = -release $(PACKAGE_VERSION) +libpxtools_la_SOURCES = \ + pxtoolsErrorCodes.c \ + pxerrors.c \ + pxconfig.c \ + pxfault.c \ + pxtables.c \ + pxtag.c + +# for pxtools.h +AM_CPPFLAGS = -I$(top_srcdir)/src$ + +pztool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pztool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pztool_SOURCES = \ + pztool.c \ + pztoolConfig.c + +p0tool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +p0tool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +p0tool_SOURCES = \ + p0tool.c \ + p0toolConfig.c + +p1tool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +p1tool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +p1tool_SOURCES = \ + p1tool.c \ + p1toolConfig.c + +p2tool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +p2tool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +p2tool_SOURCES = \ + p2tool.c \ + p2toolConfig.c + +p3tool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +p3tool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +p3tool_SOURCES = \ + p3tool.c \ + p3toolConfig.c + +p4tool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +p4tool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +p4tool_SOURCES = \ + p4tool.c \ + p4toolConfig.c + +pxadmin_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pxadmin_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pxadmin_SOURCES = \ + pxadmin.c \ + pxadminConfig.c + +pzgetexp_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pzgetexp_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pzgetexp_SOURCES = \ + pzgetexp.c \ + pzgetexpConfig.c + +pzgetimfiles_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pzgetimfiles_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pzgetimfiles_SOURCES = \ + pzgetimfiles.c \ + pzgetimfilesConfig.c + +dettool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +dettool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +dettool_SOURCES = \ + dettool.c \ + dettoolConfig.c + +detselect_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +detselect_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +detselect_SOURCES = \ + detselect.c \ + detselectConfig.c + +pxinject_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pxinject_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pxinject_SOURCES = \ + pxinject.c \ + pxinjectConfig.c + +clean-local: + -rm -f TAGS +# Tags for emacs +tags: + etags `find . -name \*.[ch] -print` + +# Error codes. +BUILT_SOURCES = pxtoolsErrorCodes.h pxtoolsErrorCodes.c +CLEANFILES = pxtoolsErrorCodes.h pxtoolsErrorCodes.c +EXTRA_DIST = pxtoolsErrorCodes.dat pxtoolsErrorCodes.c.in pxtoolsErrorCodes.h.in + +pxtoolsErrorCodes.h : pxtoolsErrorCodes.dat pxtoolsErrorCodes.h.in + $(ERRORCODES) --data=pxtoolsErrorCodes.dat --outdir=. pxtoolsErrorCodes.h + +pxtoolsErrorCodes.c : pxtoolsErrorCodes.dat pxtoolsErrorCodes.c.in pxtoolsErrorCodes.h + $(ERRORCODES) --data=pxtoolsErrorCodes.dat --outdir=. pxtoolsErrorCodes.c Index: /branches/rel-1-0/ippTools/src/camtool.c =================================================================== --- /branches/rel-1-0/ippTools/src/camtool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/camtool.c (revision 11645) @@ -0,0 +1,657 @@ +/* + * p3tool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "p3tool.h" + +static bool pendingexpMode(pxConfig *config); +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedexpMode(pxConfig *config); +static bool processedexpMode(pxConfig *config); +static bool updateprocessedexpMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = p3toolConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(P3TOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(P3TOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(P3TOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(P3TOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(P3TOOL_MODE_UPDATEPROCESSEDEXP,updateprocessedexpMode); + MODECASE(P3TOOL_MODE_BLOCK, blockMode); + MODECASE(P3TOOL_MODE_MASKED, maskedMode); + MODECASE(P3TOOL_MODE_UNBLOCK, unblockMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psFree(config); + pmConfigDone(); + psErrorStackPrint(stderr, "\n"); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + psString query = psStringCopy( + "SELECT" + " p3PendingExp.*," + " rawExp.camera," + " rawExp.workdir" + " FROM p3PendingExp" + " JOIN rawExp" + " USING(exp_tag)" + " LEFT JOIN p3ProcessedExp" + " USING(exp_tag)" + " LEFT JOIN p3Mask" + " ON p3PendingExp.label = p3Mask.label" + " WHERE" + " p3ProcessedExp.exp_tag IS NULL" + " AND p3Mask.label IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p3PendingExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p3PendingExp rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p3PendingExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy( + "SELECT" + " p2ProcessedImfile.*," + " rawExp.camera," + " rawExp.workdir" + " FROM p3PendingExp" + " JOIN p2ProcessedImfile" + " USING(exp_tag, p2_version)" + " JOIN rawExp" + " USING(exp_tag)" + " LEFT JOIN p3ProcessedExp" + " USING(exp_tag)" + " LEFT JOIN p3Mask" + " ON p3PendingExp.label = p3Mask.label" + " WHERE" + " p3ProcessedExp.exp_tag IS NULL" + " AND p3Mask.label IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p2ProcessedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query) + return false; + } + psFree(query) + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no pending p2ProcessedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p2ProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, exp_tag, recip, -bg, -bg_stdev, & -bg_mean_stdev + // are required + bool status = false; + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + psF32 sigma_ra = psMetadataLookupF32(&status, config->args, "-sigma_ra"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -sigma_ra"); + return false; + } + if (isnan(sigma_ra)) { + psError(PS_ERR_UNKNOWN, true, "-sigma_ra is required"); + return false; + } + psF32 sigma_dec = psMetadataLookupF32(&status, config->args, "-sigma_dec"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -sigma_dec"); + return false; + } + if (isnan(sigma_dec)) { + psError(PS_ERR_UNKNOWN, true, "-sigma_dec is required"); + return false; + } + psS32 nastro = psMetadataLookupS32(&status, config->args, "-nastro"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -nastro"); + return false; + } + if (nastro < 0) { + psError(PS_ERR_UNKNOWN, true, "-nastro is required"); + return false; + } + psF32 zp_mean = psMetadataLookupF32(&status, config->args, "-zp_mean"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -zp_mean"); + return false; + } + if (isnan(zp_mean)) { + psError(PS_ERR_UNKNOWN, true, "-zp_mean is required"); + return false; + } + psF32 zp_stdev = psMetadataLookupF32(&status, config->args, "-zp_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -zp_stdev"); + return false; + } + if (isnan(zp_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-zp_stdev is required"); + return false; + } + + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // default + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + psString query = psStringCopy( + "SELECT p3PendingExp.*" + " FROM p3PendingExp" + " LEFT JOIN p3ProcessedExp" + " USING(exp_tag)" + " WHERE" + " p3ProcessedExp.exp_tag IS NULL" + " AND p3PendingExp.exp_tag = '%s'" + ); + + if (!p_psDBRunQuery(config->dbh, query, exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p3PendingExp rows found"); + psFree(output); + return true; + } + + // start a transaction so we don't end up with an exp_tag is both + // p3PendingExp & p3ProcessedExp + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + p3PendingExpRow *pendingRow = p3PendingExpObjectFromMetadata(output->data[0]); + psFree(output); + // create a new p3ProcessedImfile object + p3ProcessedExpRow *row = p3ProcessedExpRowAlloc( + exp_tag, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + sigma_ra, + sigma_dec, + nastro, + b1_uri, + b2_uri, + zp_mean, + zp_stdev, + pendingRow->p2_version, + pendingRow->p3_version, + pendingRow->label, + code + ); + + // insert the new row into the p3ProcessedImfile table + if (!p3ProcessedExpInsertObject(config->dbh, row)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + psFree(pendingRow); + return false; + } + psFree(row); + + // delete the p3PendingExp row from the database + if (!p3PendingExpDeleteObject(config->dbh, pendingRow)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(pendingRow); + return false; + } + + psFree(pendingRow); + + // point of no return for p3PendingExp -> p3ProcessedExp + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy( + "SELECT" + " p3ProcessedExp.*," + " rawExp.camera," + " rawExp.workdir" + " FROM p3ProcessedExp" + " JOIN rawExp" + " USING(exp_tag)" + " WHERE" + " p3ProcessedExp.exp_tag IS NOT NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND p3ProcessedExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND p3ProcessedExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p3ProcessedExp rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p3ProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool updateprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + if (!pxSetFaultCode(config->dbh, "p3ProcessedExp", config->where, code)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + return false; + } + + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString label = psMetadataLookupStr(&status, config->args, "-label"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -label"); + return false; + } + if (!label) { + psError(PS_ERR_UNKNOWN, true, "-label is required"); + return false; + } + + if (!p3MaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy("SELECT * FROM p3Mask"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p3Mask rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p3Mask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString label = psMetadataLookupStr(&status, config->args, "-label"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -label"); + return false; + } + if (!label) { + psError(PS_ERR_UNKNOWN, true, "-label is required"); + return false; + } + + char *query = "DELETE FROM p3Mask WHERE label = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} Index: /branches/rel-1-0/ippTools/src/camtool.h =================================================================== --- /branches/rel-1-0/ippTools/src/camtool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/camtool.h (revision 11645) @@ -0,0 +1,39 @@ +/* + * p3tool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef P3TOOL_H +#define P3TOOL_H 1 + +#include "pxtools.h" + +typedef enum { + P3TOOL_MODE_NONE = 0x0, + P3TOOL_MODE_PENDINGEXP, + P3TOOL_MODE_PENDINGIMFILE, + P3TOOL_MODE_ADDPROCESSEDEXP, + P3TOOL_MODE_PROCESSEDEXP, + P3TOOL_MODE_UPDATEPROCESSEDEXP, + P3TOOL_MODE_BLOCK, + P3TOOL_MODE_MASKED, + P3TOOL_MODE_UNBLOCK +} p3toolMode; + +pxConfig *p3toolConfig(pxConfig *config, int argc, char **argv); + +#endif // P3TOOL_H Index: /branches/rel-1-0/ippTools/src/camtoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/camtoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/camtoolConfig.c (revision 11645) @@ -0,0 +1,272 @@ +/* + * p3toolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "pxtools.h" +#include "p3tool.h" + +// this function can not fail -- exits on error +pxConfig *p3toolConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddS32(pendingexpArgs, PS_LIST_TAIL, "-p2_version", 0, + "search for exposures with this p2 version", -1); + psMetadataAddS32(pendingexpArgs, PS_LIST_TAIL, "-p3_version", 0, + "search for exposures with this p3 version", -1); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + + // -addprocessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exp ID (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-uri", 0, + "define URI (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background (required)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev (required)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev (required)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-sigma_ra", 0, + "define exposure E ra (required)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-sigma_dec", 0, + "define exposure E dec (required)", NAN); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-nastro", 0, + "define number of objects (required)", -1); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-zp_mean", 0, + "define zero point mean (required)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-zp_stdev", 0, + "define zero point stdev (required)", NAN); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addprocessedexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + + // -processedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddS32(processedexpArgs, PS_LIST_TAIL, "-p2_version", 0, + "search for exposures with this p2 version", -1); + psMetadataAddS32(processedexpArgs, PS_LIST_TAIL, "-p3_version", 0, + "search for exposures with this p3 version", -1); + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -updateprocessedexp + psMetadata *updateprocessedexpArgs = psMetadataAlloc(); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddS8(updateprocessedexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, + "name of a label to mask out", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, + "name of a label to unmask", NULL); + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option);\ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + // find which mode we're running under + PXTOOL_MODE("-pendingexp", P3TOOL_MODE_PENDINGEXP, pendingexpArgs); + PXTOOL_MODE("-pendingimfile",P3TOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXTOOL_MODE("-addprocessedexp", P3TOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXTOOL_MODE("-processedexp", P3TOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXTOOL_MODE("-updateprocessedexp", P3TOOL_MODE_UPDATEPROCESSEDEXP,updateprocessedexpArgs); + PXTOOL_MODE("-block", P3TOOL_MODE_BLOCK, blockArgs); + PXTOOL_MODE("-masked", P3TOOL_MODE_MASKED, maskedArgs); + PXTOOL_MODE("-unblock", P3TOOL_MODE_UNBLOCK, unblockArgs); + + bool argErr = false; + if (config->mode == P3TOOL_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS Phase 3 Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(exp_tag); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + goto FAIL; + } + } + } + addWhereStr(telescope); + addWhereStr(exp_type); + { + int imfiles = 0; + bool status = false; + if ((imfiles = psMetadataLookupS32(&status, config->args, "-imfiles"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "imfiles", 0, "==", imfiles)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item imfiles"); + goto FAIL; + } + } + } + addWhereStr(class); + addWhereStr(class_id); + addWhereStr(filter); + + if (psListLength(config->where->list) < 1) { + psFree(config->where); + config->where = NULL; + } + + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/chiptool.c =================================================================== --- /branches/rel-1-0/ippTools/src/chiptool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/chiptool.c (revision 11645) @@ -0,0 +1,899 @@ +/* + * p2tool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "p2tool.h" + +static bool queuerawexpMode(pxConfig *config); +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedimfileMode(pxConfig *config); +static bool processedimfileMode(pxConfig *config); +static bool updateprocessedimfileMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); + +static p2ProcessedImfileRow *p2PendingToProcessedImfile(pxConfig *config, p2PendingImfileRow *imfile); +static p2ProcessedExpRow *p2PendingToProcessedExp(pxConfig *config, p2PendingExpRow *pendingExp); +static p3PendingExpRow *p2PendingToP3PendingExp(pxConfig *config, p2PendingExpRow *pendingExp); +static bool p2ProcessedCompleteExp(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = p2toolConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(P2TOOL_MODE_QUEUERAWEXP, queuerawexpMode); + MODECASE(P2TOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(P2TOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileMode); + MODECASE(P2TOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(P2TOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileMode); + MODECASE(P2TOOL_MODE_BLOCK, blockMode); + MODECASE(P2TOOL_MODE_MASKED, maskedMode); + MODECASE(P2TOOL_MODE_UNBLOCK, unblockMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +#define ADDPARAMSTR(from, to, name) \ +{ \ + bool status = false; \ + psString str = psMetadataLookupStr(&status, from, "-" name); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -" name); \ + return false; \ + } \ + if (str) { \ + if (!psMetadataAddStr(to, PS_LIST_TAIL, name, 0, "==", str)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " name); \ + psFree(to); \ + return false; \ + } \ + } \ +} + + +#define ADDPARAMF(from, to, type, name) \ +{ \ + bool status = false; \ + ps##type var = psMetadataLookup##type(&status, from, "-" #name); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -" name); \ + return false; \ + } \ + if (!isnan(var)) { \ + if (!psMetadataAdd##type(to, PS_LIST_TAIL, #name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(to); \ + return false; \ + } \ + } \ +} + + +static bool queuerawexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + ADDPARAMSTR(config->args, where, "exp_tag"); + ADDPARAMSTR(config->args, where, "exp_id"); + ADDPARAMSTR(config->args, where, "inst"); + ADDPARAMSTR(config->args, where, "telescope"); + ADDPARAMSTR(config->args, where, "dateobs"); + ADDPARAMSTR(config->args, where, "exp_type"); + ADDPARAMSTR(config->args, where, "imfiles"); + ADDPARAMSTR(config->args, where, "workdir"); + ADDPARAMSTR(config->args, where, "filter"); + ADDPARAMF(config->args, where, F32, "airmass"); + ADDPARAMF(config->args, where, F32, "ra"); + ADDPARAMF(config->args, where, F32, "decl"); + ADDPARAMF(config->args, where, F32, "exp_time"); + ADDPARAMF(config->args, where, F64, "bg"); + ADDPARAMF(config->args, where, F64, "bg_stdev"); + ADDPARAMF(config->args, where, F64, "bg_mean_stdev"); + ADDPARAMF(config->args, where, F64, "alt"); + ADDPARAMF(config->args, where, F64, "az"); + ADDPARAMF(config->args, where, F32, "ccd_temp"); + ADDPARAMF(config->args, where, F64, "posang"); + ADDPARAMSTR(config->args, where, "object"); + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + bool status = false; + psString label = psMetadataLookupStr(&status, config->args, "-set_label"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_label"); + return false; + } + + // XXX this does not work!!! + psString query = psStringCopy( + "INSERT INTO p2PendingExp\n" + " SElECT\n" + " exp_tag,\n" + " 'my recipe',\n" // recipe + " 255\n" // p1_version + " 255\n" // p2_version + " %s\n" + " FROM rawExp" + " LEFT JOIN p2ProcessedExp" + " USING(exp_tag)" + " WHERE" + " rawExp.fault = 0" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query, label ? label : "NULL")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p2PendingImfile rows found"); + psFree(output); + return false; + } + + return true; +} + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // XXX does this need to be constrained so that it won't return any results + // if a match p2PendingExp hasn't been registered? + psString query = psStringCopy( + "SELECT" + " p2PendingImfile.*," + " rawExp.camera," + " rawExp.workdir" + " FROM p2PendingImfile" + " JOIN p2PendingExp" + " USING(exp_tag)" + " JOIN rawExp" + " USING(exp_tag)" + " LEFT JOIN p2Mask" + " ON p2PendingExp.label = p2Mask.label" + " WHERE" + " p2Mask.label IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p2PendingImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p2PendingImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p2PendingImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // select * from p2ProcessedImfiles + // where + // exp_tag & class_id are not in p2PendingImfile + + // add the completed imfile to + // the p2ProcessedeImfile tables + // remove corresponding entries from the + // p2PendingImfile table + // check to see if any p2PendingExps have no + // associated p2PendingImfiles + // if so move the p2PendingExp(s) to p2ProcessedExp + + psString query = psStringCopy( + "SELECT DISTINCT" + " p2PendingImfile.*" + " FROM p2PendingImfile " + " LEFT JOIN p2ProcessedImfile" + " USING(exp_tag, class_id)" + " WHERE" + " p2ProcessedImfile.exp_tag IS NULL" + " AND p2ProcessedImfile.class_id IS NULL" + ); + { + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + psFree(query); + return false; + } + if (exp_tag) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", 0, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + psFree(query); + return false; + } + } + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + psFree(query); + return false; + } + if (class_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "class_id", 0, "==", class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(where); + psFree(query); + return false; + } + } + + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p2PendingImfile"); + psFree(where) + if (whereClause) { + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p2PendingImfile rows found"); + psFree(output); + return false; + } + + // start a transaction so we don't end up with an incremented iteration + // count but no detInputExps + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // insert into p2ProcessedImfile + // remove p2PendingImfile entry + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a p2PendingImfile object + p2PendingImfileRow *object = p2PendingImfileObjectFromMetadata(row); + // convert p2PendingImfile object into a p2ProcessedImfile object + p2ProcessedImfileRow *imfile = p2PendingToProcessedImfile(config, object); + if (!imfile) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert p2PendingImfile to p2ProcessedImfile"); + psFree(object); + psFree(output); + return false; + } + // insert p2ProccessedImfile object into the database + if (!p2ProcessedImfileInsertObject(config->dbh, imfile)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(imfile); + psFree(object); + psFree(output); + return false; + } + psFree(imfile); + // delete the p2PendingImfile object from the database + if (!p2PendingImfileDeleteObject(config->dbh, object)) { + // there must be atleast 1 Imfile to get this far + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(object); + psFree(output); + return false; + } + psFree(object); + } + + psFree(output); + + // XXX I've decided to make the transaction cover the Exp migration as + // well. Otherwise, if the last imfile in an exp is moved and the exp + // migration fails then the data base is left in a satuation where the exp + // migration can't happen. + + if (!p2ProcessedCompleteExp(config)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // point of no return for p2PendingImfile -> p2ProcessedImfile + // point of no return for p2PendingExp -> p2ProcessedExp + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + // XXX does this need to be constrained so that it won't return any results + // if a match p2PendingExp hasn't been registered? + psString query = psStringCopy( + "SELECT" + " p2ProcessedImfile.*," + " rawExp.camera," + " rawExp.workdir" + " FROM p2ProcessedImfile" + " JOIN rawExp" + " USING(exp_tag)" + " WHERE " + " p2ProcessedImfile.exp_tag is NOT NULL" // bogus test -- just here so there there is a 'WHERE' stmt to append conditionals too + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p2ProcessedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND p2ProcessedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND p2ProcessedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p2ProcessedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p2ProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool updateprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + if (!pxSetFaultCode(config->dbh, "p2ProcessedImfile", config->where, code)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + return false; + } + + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString label = psMetadataLookupStr(&status, config->args, "-label"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -label"); + return false; + } + if (!label) { + psError(PS_ERR_UNKNOWN, true, "-label is required"); + return false; + } + + if (!p2MaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy("SELECT * FROM p2Mask"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p2Mask rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p2Mask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString label = psMetadataLookupStr(&status, config->args, "-label"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -label"); + return false; + } + if (!label) { + psError(PS_ERR_UNKNOWN, true, "-label is required"); + return false; + } + + char *query = "DELETE FROM p2Mask WHERE label = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} + + +static bool p2ProcessedCompleteExp(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // look for completed p2PendingExp + // migrate them to p2ProccessedExp & p3PendingExp + + // select * from p2PendingExp + // where exp_tag is not in p2ProcessedExp + // where exp_tag is not in p2PendingImfile + // where the number of entries in p2ProccessedImfile matches the .imfiles + // entry in rawExp + psString query = psStringCopy( + "SELECT DISTINCT" + " p2PendingExp.*," + " rawExp.imfiles," + " p2ProcessedImfile.class_id" + " FROM p2PendingExp" + " JOIN rawExp" + " ON p2PendingExp.exp_tag = rawExp.exp_tag" + " LEFT JOIN p2ProcessedExp" + " ON p2PendingExp.exp_tag = p2ProcessedExp.exp_tag" + " LEFT JOIN p2PendingImfile" + " ON p2PendingExp.exp_tag = p2PendingImfile.exp_tag" + " LEFT JOIN p2ProcessedImfile" + " ON p2PendingExp.exp_tag = p2ProcessedImfile.exp_tag" + " WHERE" + " p2ProcessedExp.exp_tag IS NULL" + " AND p2PendingImfile.exp_tag IS NULL" + " AND p2ProcessedImfile.exp_tag IS NOT NULL" + " GROUP BY" + " p2PendingExp.exp_tag" + " HAVING rawExp.imfiles = COUNT(p2ProcessedImfile.class_id)" + ); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no p2PendingExp rows found"); + psFree(output); + return true; + } + + // insert into p2ProcessedExp + // insert into p3PendingExp + // remove p2PendingExp entry + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a p2PendingExp object + p2PendingExpRow *object = p2PendingExpObjectFromMetadata(row); + // do both *Exp type conversion first so we don't insert one and then + // have to rollback the change as the 2nd failed + // convert p2PendingExp object into a p2ProcesseExp object + p2ProcessedExpRow *processedExp = p2PendingToProcessedExp(config, object); + if (!processedExp) { + psError(PS_ERR_UNKNOWN, false, "failed to convert p2PendingExp to p2ProcessedExp"); + psFree(object); + psFree(output); + return false; + } + // convert p2PendingExp object into a p3PendingExp object + p3PendingExpRow *pendingExp = p2PendingToP3PendingExp(config, object); + if (!processedExp) { + psError(PS_ERR_UNKNOWN, false, "failed to convert p2PendingExp to p3PendingExp"); + psFree(processedExp); + psFree(object); + psFree(output); + return false; + } + // insert p2ProccessedExp object into the database + if (!p2ProcessedExpInsertObject(config->dbh, processedExp)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(processedExp); + psFree(pendingExp); + psFree(object); + psFree(output); + return false; + } + psFree(processedExp); + // insert p3PendingExp object into the database + if (!p3PendingExpInsertObject(config->dbh, pendingExp)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(pendingExp); + psFree(object); + psFree(output); + return false; + } + psFree(pendingExp); + // delete the p2PendingExp object from the database + if (!p2PendingExpDeleteObject(config->dbh, object)) { + // there must be atleast 1 Imfile to get this far + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(object); + psFree(output); + return false; + } + psFree(object); + } + + psFree(output); + + return true; +} + + +static p2ProcessedImfileRow *p2PendingToProcessedImfile(pxConfig *config, p2PendingImfileRow *imfile) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(imfile, NULL); + + bool status = false; + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + return p2ProcessedImfileRowAlloc( + imfile->exp_tag, + imfile->class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + imfile->p1_version, + imfile->p2_version, + code + ); +} + + +static p2ProcessedExpRow *p2PendingToProcessedExp(pxConfig *config, p2PendingExpRow *pendingExp) +{ + PS_ASSERT_PTR_NON_NULL(pendingExp, NULL); + + return p2ProcessedExpRowAlloc( + pendingExp->exp_tag, + pendingExp->p1_version, + pendingExp->p2_version, + pendingExp->label + ); +} + + +static p3PendingExpRow *p2PendingToP3PendingExp(pxConfig *config, p2PendingExpRow *pendingExp) +{ + PS_ASSERT_PTR_NON_NULL(pendingExp, NULL); + + return p3PendingExpRowAlloc( + pendingExp->exp_tag, + pendingExp->p1_version, + pendingExp->p2_version, + pendingExp->label + ); +} Index: /branches/rel-1-0/ippTools/src/chiptool.h =================================================================== --- /branches/rel-1-0/ippTools/src/chiptool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/chiptool.h (revision 11645) @@ -0,0 +1,39 @@ +/* + * p2tool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef P2TOOL_H +#define P2TOOL_H 1 + +#include "pxtools.h" + +typedef enum { + P2TOOL_MODE_NONE = 0x0, + P2TOOL_MODE_QUEUERAWEXP, + P2TOOL_MODE_PENDINGIMFILE, + P2TOOL_MODE_ADDPROCESSEDIMFILE, + P2TOOL_MODE_PROCESSEDIMFILE, + P2TOOL_MODE_UPDATEPROCESSEDIMFILE, + P2TOOL_MODE_BLOCK, + P2TOOL_MODE_MASKED, + P2TOOL_MODE_UNBLOCK +} p2toolMode; + +pxConfig *p2toolConfig(pxConfig *config, int argc, char **argv); + +#endif // P2TOOL_H Index: /branches/rel-1-0/ippTools/src/chiptoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/chiptoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/chiptoolConfig.c (revision 11645) @@ -0,0 +1,307 @@ +/* + * p2toolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "p2tool.h" + +// this function can not fail -- exits on error +pxConfig *p2toolConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + // -queuerawexp + psMetadata *queuerawexpArgs = psMetadataAlloc(); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exp_tag", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-exp_id", 0, + "search by exp_id", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-inst", 0, + "search by camera", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-telescope", 0, + "search by telescope", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-dateobs", 0, + "search by observation time", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-exp_type", 0, + "search by exposure type", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-imfiles", 0, + "search by number of imfiles", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-workdir", 0, + "search by workdir", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-filter", 0, + "search by filter ", NULL); + psMetadataAddF32(queuerawexpArgs, PS_LIST_TAIL, "-airmass", 0, + "search by airmass", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-ra", 0, + "search by RA", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-decl", 0, + "search by DEC", NAN); + psMetadataAddF32(queuerawexpArgs, PS_LIST_TAIL, "-exp_time", 0, + "search by exposure time", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-bg", 0, + "search by exposue background", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "search by exposue background stdev", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "search by exposue background mean stdev", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-alt", 0, + "search by altitute", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-az", 0, + "search by azimuth", NAN); + psMetadataAddF32(queuerawexpArgs, PS_LIST_TAIL, "-ccd_temp", 0, + "search by ccd tempature", NAN); + psMetadataAddF64(queuerawexpArgs, PS_LIST_TAIL, "-posang", 0, + "search by rotator position angle", NAN); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-object", 0, + "search by exposure object", NULL); + psMetadataAddStr(queuerawexpArgs, PS_LIST_TAIL, "-set_label", 0, + "define label for phase 2 processing (non-detrend data only)", NULL); + + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class", 0, + "define class", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-inst", 0, + "define camera of interest", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-filter", 0, + "define filter of interest", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URL", NULL); + psMetadataAddU64(pendingimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exp_tag (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URL", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addprocessedimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class", 0, + "define class", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-inst", 0, + "define camera of interest", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-filter", 0, + "define filter of interest", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URL", NULL); + psMetadataAddU64(processedimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -updateprocessedimfile + psMetadata *updateprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddS8(updateprocessedimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, + "name of a label to mask out", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, + "name of a label to unmask", NULL); + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {; \ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option); \ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + PXTOOL_MODE("-queuerawexp", P2TOOL_MODE_QUEUERAWEXP, queuerawexpArgs); + PXTOOL_MODE("-pendingimfile", P2TOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXTOOL_MODE("-addprocessedimfile",P2TOOL_MODE_ADDPROCESSEDIMFILE,addprocessedimfileArgs); + PXTOOL_MODE("-processedimfile",P2TOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXTOOL_MODE("-updateprocessedimfile",P2TOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileArgs); + PXTOOL_MODE("-block", P2TOOL_MODE_BLOCK, blockArgs); + PXTOOL_MODE("-masked", P2TOOL_MODE_MASKED, maskedArgs); + PXTOOL_MODE("-unblock", P2TOOL_MODE_UNBLOCK, unblockArgs); + + bool argErr = false; + if (config->mode == P2TOOL_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS Phase 2 Search Tool\n\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(exp_tag); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + goto FAIL; + } + } + } + + + addWhereStr(filter); + addWhereStr(class); + addWhereStr(class_id); + + /* + // psMetadataConfig does not support times yet + if (!psTimeIsZero(config->start) && !psTimeIsZero(config->stop)) { + psMetadataAddTime (where, PS_LIST_TAIL, "TIME_START", 0, "<", config->stop); + psMetadataAddTime (where, PS_LIST_TAIL, "TIME_STOP", 0, ">", config->start); + } + */ + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/detselect.c =================================================================== --- /branches/rel-1-0/ippTools/src/detselect.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/detselect.c (revision 11645) @@ -0,0 +1,302 @@ +/* + * detselect.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "detselect.h" + +static bool searchMode(pxConfig *config); +static bool selectMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = detselectConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(DETSELECT_MODE_SEARCH, searchMode); + MODECASE(DETSELECT_MODE_SELECT, selectMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psFree(config); + pmConfigDone(); + psErrorStackPrint(stderr, "\n"); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool searchMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + // airmass_min < airmass < airmass_max + { + bool status = false; + psF32 airmass = psMetadataLookupF32(&status, config->args, "-airmass"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass"); + psFree(where); + return false; + } + + if (!isnan(airmass)) { + psMetadataAddF32(where, PS_LIST_TAIL, "airmass_min", 0, "<=", airmass); + psMetadataAddF32(where, PS_LIST_TAIL, "airmass_max", 0, ">=", airmass); + } + } + + // exp_time_min < exp_time < exp_time_max + { + bool status = false; + psF32 exp_time = psMetadataLookupF32(&status, config->args, "-exp_time"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time"); + psFree(where); + return false; + } + + if (!isnan(exp_time)) { + psMetadataAddF32(where, PS_LIST_TAIL, "exp_time_min", 0, "<=", exp_time); + psMetadataAddF32(where, PS_LIST_TAIL, "exp_time_max", 0, ">=", exp_time); + } + } + + // ccd_temp_min < ccd_temp < ccd_temp_max + { + bool status = false; + psF32 ccd_temp = psMetadataLookupF32(&status, config->args, "-ccd_temp"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp"); + psFree(where); + return false; + } + + if (!isnan(ccd_temp)) { + psMetadataAddF32(where, PS_LIST_TAIL, "ccd_temp_min", 0, "<=", ccd_temp); + psMetadataAddF32(where, PS_LIST_TAIL, "ccd_temp_max", 0, ">=", ccd_temp); + } + } + + { + bool status = false; + psF64 posang = psMetadataLookupF64(&status, config->args, "-posang"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang"); + psFree(where); + return false; + } + + if (!isnan(posang)) { + psMetadataAddF32(where, PS_LIST_TAIL, "posang_min", 0, "<=", posang); + psMetadataAddF32(where, PS_LIST_TAIL, "posang_max", 0, ">=", posang); + } + } + + // time_begin < time < time_end + { + bool status = false; + psString timeStr = psMetadataLookupStr(&status, config->args, "-time"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time"); + psFree(where); + return false; + } + + if (timeStr) { + psTime *time = psTimeFromISO(timeStr, PS_TIME_UTC); + if (!time) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", timeStr); + psFree(where); + return false; + } + + // the == NULL tests invokes some psDB magic to make an OR + // conditional query + psMetadataAddTime(where, PS_LIST_TAIL, "time_begin", 0, "<=", time); + psMetadataAddTime(where, PS_LIST_TAIL, "time_begin", PS_META_DUPLICATE_OK, "==", NULL); + psMetadataAddTime(where, PS_LIST_TAIL, "time_end", 0, ">=", time); + psMetadataAddTime(where, PS_LIST_TAIL, "time_end", PS_META_DUPLICATE_OK, "==", NULL); + psFree(time); + } + } + + psString query = psStringCopy( + "SELECT DISTINCT" + " detRunSummary.det_id," + " detRunSummary.iteration" + " FROM detRun" + " JOIN detRunSummary" + " USING(det_id)" + " WHERE" + " detRun.state = 'stop'" + " AND detRun.mode = 'master'" + " AND detRunSummary.accept = 1" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detRun"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detRun"); + psFree(where); + if (whereClause) { + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + } + + // XXX this needs to be more controlled: we should specifically choose the + // single detrend image which matches all criteria and has the latest + // insertion date + psStringAppend(&query, " ORDER BY detRun.registered DESC LIMIT 1"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detrend exposures found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool selectMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy( + "SELECT DISTINCT" + " detNormalizedImfile.*" + " FROM detNormalizedImfile" + " JOIN detRun" + " USING(det_id)" + " JOIN detRunSummary" + " ON detNormalizedImfile.det_id = detRunSummary.det_id" + " AND detNormalizedImfile.iteration = detRunSummary.iteration" + " WHERE" + " detRun.state = 'stop'" + " AND detRun.mode = 'master'" + " AND detRunSummary.accept = 1" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detNormalizedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detNormalizedImfile rows found"); + psFree(output) + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} Index: /branches/rel-1-0/ippTools/src/detselect.h =================================================================== --- /branches/rel-1-0/ippTools/src/detselect.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/detselect.h (revision 11645) @@ -0,0 +1,33 @@ +/* + * detselect.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef DETSELECT_H +#define DETSELECT_H 1 + +#include "pxtools.h" + +typedef enum { + DETSELECT_MODE_NONE = 0x0, + DETSELECT_MODE_SEARCH, + DETSELECT_MODE_SELECT, +} detselectMode; + +pxConfig *detselectConfig(pxConfig *config, int argc, char **argv); + +#endif // DETSELECT_H Index: /branches/rel-1-0/ippTools/src/detselectConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/detselectConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/detselectConfig.c (revision 11645) @@ -0,0 +1,265 @@ +/* + * detselectConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "pxtools.h" +#include "detselect.h" + +// this function can not fail -- exits on error +pxConfig *detselectConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + // -search + psMetadata *searchArgs = psMetadataAlloc(); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-inst", 0, + "search by camera", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-telescope", 0, + "search by telescope", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-det_type", 0, + "search by detrend type", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-filter", 0, + "search by filter", NULL); + psMetadataAddF32(searchArgs, PS_LIST_TAIL, "-airmass", 0, + "define airmass", NAN); + psMetadataAddF32(searchArgs, PS_LIST_TAIL, "-exp_time", 0, + "search by exposure time", NAN); + psMetadataAddF32(searchArgs, PS_LIST_TAIL, "-ccd_temp", 0, + "search by ccd tempature", NAN); + psMetadataAddF64(searchArgs, PS_LIST_TAIL, "-posang", 0, + "search by rotator position angle", NAN); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-time", 0, + "define time for desired detrend data", NULL); + psMetadataAddBool(searchArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -select + psMetadata *selectArgs = psMetadataAlloc(); + psMetadataAddStr(selectArgs, PS_LIST_TAIL, "-det_id", 0, + "search by detrend ID", NULL); + psMetadataAddS32(selectArgs, PS_LIST_TAIL, "-iteration", 0, + "search by iteration number", 0); + psMetadataAddStr(selectArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddBool(selectArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option);\ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + // find which mode we're running under + PXTOOL_MODE("-search", DETSELECT_MODE_SEARCH, searchArgs); + PXTOOL_MODE("-select", DETSELECT_MODE_SELECT, selectArgs); + + bool argErr = false; + if (config->mode == DETSELECT_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS detrend select Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + +#define addWhereS32(name) \ +{ \ + psS32 s32 = 0; \ + bool status = false; \ + if ((s32= psMetadataLookupS32(&status, config->args, "-" #name))) { \ + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, #name, 0, "==", s32)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + +#define addWhereF32(name) \ +{ \ + psF32 var = 0; \ + bool status = false; \ + if ((var = psMetadataLookupF32(&status, config->args, "-" #name))) { \ + if (!isnan(var)) { \ + if (!psMetadataAddF32(config->where, PS_LIST_TAIL, #name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ + } \ +} + +#define addWhereF64(name) \ +{ \ + psF64 var = 0; \ + bool status = false; \ + if ((var = psMetadataLookupF64(&status, config->args, "-" #name))) { \ + if (!isnan(var)) { \ + if (!psMetadataAddF64(config->where, PS_LIST_TAIL, #name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ + } \ +} + +#define addWhereTimeStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + psTime *time = psTimeFromISO(str, PS_TIME_UTC); \ + if (!time) { \ + psError(PS_ERR_UNKNOWN, false, "failed to convert " #name " into a psTime object"); \ + goto FAIL; \ + } \ + psMetadataItem *item = psMetadataLookup(config->args, "-" #name); \ + if (item) { \ + str = item->comment; \ + } else { \ + str = NULL; \ + } \ + if (!psMetadataAddTime(config->where, PS_LIST_TAIL, #name, 0, str, time)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + psFree(time); \ + } \ +} + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(det_id); + addWhereS32(iteration); + addWhereStr(exp_tag); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + goto FAIL; + } + } + } + addWhereStr(telescope); + addWhereStr(det_type); + addWhereStr(exp_type); + addWhereS32(imfiles); + addWhereStr(class); + addWhereStr(class_id); + addWhereStr(filter); +// addWhereF32(exp_time); +// addWhereF64(ccd_temp); +// addWhereF32(airmass); +// addWhereTimeStr(use_begin); +// addWhereTimeStr(use_end); + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/dettool.c =================================================================== --- /branches/rel-1-0/ippTools/src/dettool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/dettool.c (revision 11645) @@ -0,0 +1,6518 @@ +/* + * dettool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" +#include "dettool.h" + +static bool pendingMode(pxConfig *config); +static bool definebytagMode(pxConfig *config); +static bool definebyqueryMode(pxConfig *config); +static bool definebydetrunMode(pxConfig *config); +static bool runsMode(pxConfig *config); +static bool inputMode(pxConfig *config); +static bool rawMode(pxConfig *config); +static bool toprocessedimfileMode(pxConfig *config); +static bool addprocessedimfileMode(pxConfig *config); +static bool toprocessedexpMode(pxConfig *config); +static bool addprocessedexpMode(pxConfig *config); +static bool processedexpMode(pxConfig *config); +static bool tostackedMode(pxConfig *config); +static bool processedimfileMode(pxConfig *config); +static bool addstackedMode(pxConfig *config); +static bool stackedMode(pxConfig *config); +static bool tonormalizeMode(pxConfig *config); +static bool addnormalizedstatMode(pxConfig *config); +static bool tonormalizedstatMode(pxConfig *config); +static bool addnormalizedimfileMode(pxConfig *config); +static bool tonormalizedexpMode(pxConfig *config); +static bool addnormalizedexpMode(pxConfig *config); +static bool normalizedexpMode(pxConfig *config); +static bool toresidimfileMode(pxConfig *config); +static bool addresidimfileMode(pxConfig *config); +static bool normalizedimfileMode(pxConfig *config); +static bool toresidexpMode(pxConfig *config); +static bool residimfileMode(pxConfig *config); +static bool addresidexpMode(pxConfig *config); +static bool residexpMode(pxConfig *config); +static bool residdetrunMode(pxConfig *config); +static bool updateresidexpMode(pxConfig *config); +static bool adddetrunsummaryMode(pxConfig *config); +static bool detrunsummaryMode(pxConfig *config); +static bool updatedetrunMode(pxConfig *config); +static bool rerunMode(pxConfig *config); +static bool register_detrendMode(pxConfig *config); +static bool register_detrend_imfileMode(pxConfig *config); + +static bool convertDetIdToStr(psArray *mds); +static detNormalizedStatImfileRow *detStackedToDetNormalizedStatImfile(pxConfig *config, detStackedImfileRow *stackedImfile); +static detNormalizedImfileRow *detNormalizedStatToDetNormalizedmfile(pxConfig *config, detNormalizedStatImfileRow *statImfile); +static detResidImfileRow *detNormalizedToDetResidImfile(pxConfig *config, detNormalizedImfileRow *normalizedImfile); +static detResidExpRow *mdToDetResidExp(pxConfig *config, psMetadata *row); +static detRunSummaryRow *mdToDetRunSummary(pxConfig *config, psMetadata *row); +//static psArray *validDetInputClassIds(pxConfig *config, const char *det_id); +//static psArray *searchInputImfiles(pxConfig *config, const char *det_id); +static detInputExpRow *rawDetrenTodetInputExpRow(rawExpRow *rawExp, psS32 det_id, psS32 iteration); +static psArray *searchRawImfiles(pxConfig *config, psMetadata *where); +static psS32 incrementIteration(pxConfig *config, const char *det_id); +static bool setDetRunState(pxConfig *config, const char *det_id, const char *state); +static bool isValidMode(pxConfig *config, const char *mode); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = dettoolConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(DETTOOL_MODE_PENDING, pendingMode); + MODECASE(DETTOOL_MODE_DEFINEBYTAG, definebytagMode); + MODECASE(DETTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(DETTOOL_MODE_DEFINEBYDETRUN, definebydetrunMode); + MODECASE(DETTOOL_MODE_RUNS, runsMode); + MODECASE(DETTOOL_MODE_INPUT, inputMode); + MODECASE(DETTOOL_MODE_RAW, rawMode); + MODECASE(DETTOOL_MODE_TOPROCESSEDIMFILE,toprocessedimfileMode); + MODECASE(DETTOOL_MODE_ADDPROCESSEDIMFILE,addprocessedimfileMode); + MODECASE(DETTOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(DETTOOL_MODE_TOPROCESSEDEXP, toprocessedexpMode); + MODECASE(DETTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(DETTOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(DETTOOL_MODE_TOSTACKED, tostackedMode); + MODECASE(DETTOOL_MODE_ADDSTACKED, addstackedMode); + MODECASE(DETTOOL_MODE_STACKED, stackedMode); + MODECASE(DETTOOL_MODE_TONORMALIZE, tonormalizeMode); + MODECASE(DETTOOL_MODE_ADDNORMALIZEDSTAT,addnormalizedstatMode); + MODECASE(DETTOOL_MODE_TONORMALIZEDSTAT, tonormalizedstatMode); + MODECASE(DETTOOL_MODE_ADDNORMALIZEDIMFILE,addnormalizedimfileMode); + MODECASE(DETTOOL_MODE_NORMALIZEDIMFILE, normalizedimfileMode); + MODECASE(DETTOOL_MODE_TONORMALIZEDEXP, tonormalizedexpMode); + MODECASE(DETTOOL_MODE_ADDNORMALIZEDEXP, addnormalizedexpMode); + MODECASE(DETTOOL_MODE_NORMALIZEDEXP, normalizedexpMode); + MODECASE(DETTOOL_MODE_TORESIDIMFILE, toresidimfileMode); + MODECASE(DETTOOL_MODE_ADDRESIDIMFILE, addresidimfileMode); + MODECASE(DETTOOL_MODE_TORESIDEXP, toresidexpMode); + MODECASE(DETTOOL_MODE_RESIDIMFILE, residimfileMode); + MODECASE(DETTOOL_MODE_ADDRESIDEXP, addresidexpMode); + MODECASE(DETTOOL_MODE_RESIDEXP, residexpMode); + MODECASE(DETTOOL_MODE_RESIDDETRUN, residdetrunMode); + MODECASE(DETTOOL_MODE_UPDATERESIDEXP, updateresidexpMode); + MODECASE(DETTOOL_MODE_ADDDETRUNSUMMARY, adddetrunsummaryMode); + MODECASE(DETTOOL_MODE_DETRUNSUMMARY, detrunsummaryMode); + MODECASE(DETTOOL_MODE_UPDATEDETRUN, updatedetrunMode); + MODECASE(DETTOOL_MODE_RERUN, rerunMode); + MODECASE(DETTOOL_MODE_REGISTER_DETREND, register_detrendMode); + MODECASE(DETTOOL_MODE_REGISTER_DETREND_IMFILE, register_detrend_imfileMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool pendingMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + "SELECT" + " rawExp.*" + " FROM rawExp" + " LEFT JOIN detInputExp" + " ON rawExp.exp_tag = detInputExp.exp_tag" + " WHERE" + " detInputExp.exp_tag IS NULL" + " AND rawExp.obstype != 'object'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending rawExp rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool definebytagMode(pxConfig *config) +{ + bool status = false; + + PS_ASSERT_PTR_NON_NULL(config, false); + + // what type of detRun is this? + // det_type is required + psString det_type = psMetadataLookupStr(&status, config->args, "-det_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_type"); + return false; + } + if (!det_type) { + psError(PS_ERR_UNKNOWN, true, "-det_type is required"); + return false; + } + + psString workdir = psMetadataLookupStr(&status, config->args, "-workdir"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -workdir"); + return false; + } + if (!workdir) { + psError(PS_ERR_UNKNOWN, true, "-workdir is required"); + return false; + } + + // optional + psString mode = psMetadataLookupStr(&status, config->args, "-mode"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -mode"); + return false; + } + // check mode + if (mode && !isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalud mode"); + return false; + } + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return false; + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return false; + } + + psString exp_type = psMetadataLookupStr(&status, config->args, "-exp_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_type"); + return false; + } + + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + + psF32 airmass_min = psMetadataLookupF32(&status, config->args, "-airmass_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass_min"); + return false; + } + + psF32 airmass_max = psMetadataLookupF32(&status, config->args, "-airmass_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass_max"); + return false; + } + + psF32 exp_time_min = psMetadataLookupF32(&status, config->args, "-exp_time_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time_min"); + return false; + } + + psF32 exp_time_max = psMetadataLookupF32(&status, config->args, "-exp_time_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time_max"); + return false; + } + + psF32 ccd_temp_min = psMetadataLookupF32(&status, config->args, "-ccd_temp_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp_min"); + return false; + } + + psF32 ccd_temp_max = psMetadataLookupF32(&status, config->args, "-ccd_temp_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp_max"); + return false; + } + + psF64 posang_min = psMetadataLookupF32(&status, config->args, "-posang_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang_min"); + return false; + } + + psF64 posang_max = psMetadataLookupF32(&status, config->args, "-posang_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang_max"); + return false; + } + + psTime *registered = NULL; + { + psString registeredStr = psMetadataLookupStr(&status, config->args, "-registered"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -registered"); + return false; + } + // pass through NULL as this is an optional field + if (registeredStr) { + registered = psTimeFromISO(registeredStr, PS_TIME_UTC); + if (!registered) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", registeredStr); + return false; + } + } else { + registered = NULL; + } + } + + psTime *time_begin = NULL; + { + psString time_beginStr = psMetadataLookupStr(&status, config->args, "-time_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time_begin"); + return false; + } + // pass through NULL as this is an optional field + if (time_beginStr) { + time_begin = psTimeFromISO(time_beginStr, PS_TIME_UTC); + if (!time_begin) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", time_beginStr); + return false; + } + } else { + time_begin = NULL; + } + } + + psTime *time_end = NULL; + { + psString time_endStr = psMetadataLookupStr(&status, config->args, "-time_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time_end"); + return false; + } + // pass through NULL as this is an optional field + if (time_endStr) { + time_end = psTimeFromISO(time_endStr, PS_TIME_UTC); + if (!time_end) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", time_endStr); + return false; + } + } else { + time_end = NULL; + } + } + + psTime *use_begin = NULL; + { + psString use_beginStr = psMetadataLookupStr(&status, config->args, "-use_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -use_begin"); + return false; + } + // pass through NULL as this is an optional field + if (use_beginStr) { + use_begin = psTimeFromISO(use_beginStr, PS_TIME_UTC); + if (!use_begin) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", use_beginStr); + return false; + } + } else { + use_begin = NULL; + } + } + + psTime *use_end = NULL; + { + psString use_endStr = psMetadataLookupStr(&status, config->args, "-use_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -use_end"); + return false; + } + // pass through NULL as this is an optional field + if (use_endStr) { + use_end = psTimeFromISO(use_endStr, PS_TIME_UTC); + if (!use_end) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", use_endStr); + return false; + } + } else { + use_end = NULL; + } + } + + // we have to support multipe exp_tags + psMetadataItem *item = psMetadataLookup(config->args, "-exp_tag"); + if (!item) { + // this shouldn't actually happen when using psArgs + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + psMetadata *where = psMetadataAlloc(); + + // make sure that -exp_tag was parsed correctly + // XXX this can be removed someday + if (item->type == PS_DATA_METADATA_MULTI) { + psListIterator *iter = psListIteratorAlloc(item->data.list, 0, false); + psMetadataItem *mItem = NULL; + while ((mItem = psListGetAndIncrement(iter))) { + psString exp_tag = mItem->data.V; + // if exp_tag is NULL then it means that -exp_tag has not been + // specified + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, + "at least one -exp_tag is required"); + psFree(where); + return false; + } + + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", + PS_META_DUPLICATE_OK, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(iter); + psFree(where); + return false; + } + } + psFree(iter); + } else { + psAbort(config->argv[0], + "-exp_tag was not parsed correctly (this should not happen"); + } + + if (psListLength(where->list) < 1) { + psFree(where); + where = NULL; + } + + // check that the specified exp_tags actually exist + psArray *detrendExps = rawExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!detrendExps) { + psError(PS_ERR_UNKNOWN, false, "no rawExp rows found"); + return false; + } + + // we should have one rawExp row per exp_tag specified + if (psListLength(item->data.list) != psArrayLength(detrendExps)) { + psAbort(config->argv[0], + "an -exp_tag matched more then one rawExp (this should not happen"); + + } + + // start a transaction so we don't end up with orphaned det_ids + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detrendExps); + return false; + } + + // the first iteration is always 0 + // XXX the camera name is set from the first inputExp + // XXX det_id + detRunInsert(config->dbh, + 0, + 0, + det_type, + mode, + "run", + workdir, + camera, + telescope, + exp_type, + filter, + airmass_min, + airmass_max, + exp_time_min, + exp_time_max, + ccd_temp_min, + ccd_temp_max, + posang_min, + posang_max, + registered, + time_begin, + time_end, + use_begin, + use_end + ); + psFree(registered); + psFree(time_begin); + psFree(time_end); + psFree(use_begin); + psFree(use_end); + long det_id = psDBLastInsertID(config->dbh); + + // create new detInputExp row(s) from the rawExp row(s) + psArray *inputExps = psArrayAllocEmpty(psArrayLength(detrendExps)); + for (long i = 0; i < psArrayLength(detrendExps); i++) { + detInputExpRow *inputExp = rawDetrenTodetInputExpRow( + detrendExps->data[i], + det_id, + 0 // the first iteration is explicitly 0 + ); + psArrayAdd(inputExps, 0, inputExp); + psFree(inputExp); + } + + psFree(detrendExps); + + // insert detInputExp objects into the database + if (!detInputExpInsertObjects(config->dbh, inputExps)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(inputExps); + return false; + } + psFree(inputExps); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // print the new det_id + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", det_id); + detRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check results + if (psArrayLength(detRuns) != 1) { + psAbort(config->argv[0], "found more then one detRun matching det_id %ld(this should not happen)", det_id); + return false; + } + + // convert det_id to a str externally + if (!convertDetIdToStr(detRuns)) { + psError(PS_ERR_UNKNOWN, false, "failed to convert det_id to a string"); + psFree(detRuns); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, detRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detRuns); + return false; + } + psFree(detRuns); + + return true; +} + +// This function is used to convert the det_id from an int, as it is used +// internally, to be a string for external use. The rational being that we may +// want to change how det_id is generated in the future and don't want to +// external programs to become depending on this value being an int. +static bool convertDetIdToStr(psArray *mds) +{ + PS_ASSERT_PTR_NON_NULL(mds, false); + + for (long i = 0; i < psArrayLength(mds); i++) { + psMetadata *md = mds->data[i]; + bool status = false; + psS32 det_id = psMetadataLookupS32(&status, md, "det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for det_id"); + return false; + } + psMetadataRemoveKey(md, "det_id"); + psString det_idStr = psDBIntToString((psU64)det_id); + psMetadataAddStr(mds->data[i], PS_LIST_HEAD, "det_id", 0, NULL, det_idStr); + psFree(det_idStr); + } + + return true; +} + +static bool definebyqueryMode(pxConfig *config) +{ + bool status = false; + + PS_ASSERT_PTR_NON_NULL(config, false); + + // what type of detRun is this? + // det_type is required + psString det_type = psMetadataLookupStr(&status, config->args, "-det_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_type"); + return false; + } + if (!det_type) { + psError(PS_ERR_UNKNOWN, true, "-det_type is required"); + return false; + } + + psString workdir = psMetadataLookupStr(&status, config->args, "-workdir"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -workdir"); + return false; + } + if (!workdir) { + psError(PS_ERR_UNKNOWN, true, "-workdir is required"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + { + bool status = false; + psString exp_type = psMetadataLookupStr(&status, config->args, "-select_exp_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_exp_type"); + return false; + } + if (exp_type) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_type", 0, "==", exp_type)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_type"); + psFree(where); + return false; + } + } + + // map -inst -> camera + psString camera = psMetadataLookupStr(&status, config->args, "-select_inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_inst"); + return false; + } + if (camera) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "camera", 0, "==", camera)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item inst"); + psFree(where); + return false; + } + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-select_telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_telescope"); + return false; + } + if (telescope) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "telescope", 0, "==", telescope)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(where); + return false; + } + } + + psString filter = psMetadataLookupStr(&status, config->args, "-select_filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_filter"); + return false; + } + if (filter) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "filter", 0, "==", filter)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item filter"); + psFree(where); + return false; + } + } + + psString uri = psMetadataLookupStr(&status, config->args, "-select_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_uri"); + return false; + } + if (uri) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "uri", 0, "==", uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item uri"); + psFree(where); + return false; + } + } + + psString dateobs_begin = psMetadataLookupStr(&status, config->args, "-select_dateobs_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_dateobs_begin"); + psFree(where); + return false; + } + if (dateobs_begin) { + psTime *time = psTimeFromISO(dateobs_begin, PS_TIME_UTC); + if (!time) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", dateobs_begin); + psFree(time); + psFree(where); + return false; + } + if (!psMetadataAddTime(where, PS_LIST_TAIL, "dateobs", PS_META_DUPLICATE_OK, ">=", time)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item dateobs"); + psFree(time); + psFree(where); + return false; + } + psFree(time); + } + + psString dateobs_end = psMetadataLookupStr(&status, config->args, "-select_dateobs_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_dateobs_end"); + psFree(where); + return false; + } + if (dateobs_end) { + psTime *time = psTimeFromISO(dateobs_end, PS_TIME_UTC); + if (!time) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", dateobs_end); + psFree(time); + psFree(where); + return false; + } + if (!psMetadataAddTime(where, PS_LIST_TAIL, "dateobs", PS_META_DUPLICATE_OK, "<", time)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item dateobs"); + psFree(time); + psFree(where); + return false; + } + psFree(time); + } + + /** selection based on airmass range **/ + psF32 select_airmass_min = psMetadataLookupF32(&status, config->args, "-select_airmass_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_airmass_min"); + psFree(where); + return false; + } + if (isfinite(select_airmass_min)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "airmass", PS_META_DUPLICATE_OK, ">=", select_airmass_min)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item airmass"); + psFree(where); + return false; + } + } + psF32 select_airmass_max = psMetadataLookupF32(&status, config->args, "-select_airmass_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_airmass_max"); + psFree(where); + return false; + } + if (isfinite(select_airmass_max)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "airmass", PS_META_DUPLICATE_OK, "<=", select_airmass_max)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item airmass"); + psFree(where); + return false; + } + } + + /** selection based on exp_time range **/ + psF32 select_exp_time_min = psMetadataLookupF32(&status, config->args, "-select_exp_time_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_exp_time_min"); + psFree(where); + return false; + } + if (isfinite(select_exp_time_min)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "exp_time", PS_META_DUPLICATE_OK, ">=", select_exp_time_min)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_time"); + psFree(where); + return false; + } + } + psF32 select_exp_time_max = psMetadataLookupF32(&status, config->args, "-select_exp_time_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_exp_time_max"); + psFree(where); + return false; + } + if (isfinite(select_exp_time_max)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "exp_time", PS_META_DUPLICATE_OK, "<=", select_exp_time_max)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_time"); + psFree(where); + return false; + } + } + + /** selection based on ccd_temp range **/ + psF32 select_ccd_temp_min = psMetadataLookupF32(&status, config->args, "-select_ccd_temp_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_ccd_temp_min"); + psFree(where); + return false; + } + if (isfinite(select_ccd_temp_min)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "ccd_temp", PS_META_DUPLICATE_OK, "<=", select_ccd_temp_min)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item ccd_temp"); + psFree(where); + return false; + } + } + psF32 select_ccd_temp_max = psMetadataLookupF32(&status, config->args, "-select_ccd_temp_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_ccd_temp_max"); + psFree(where); + return false; + } + if (isfinite(select_ccd_temp_max)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "ccd_temp", PS_META_DUPLICATE_OK, ">=", select_ccd_temp_max)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item ccd_temp"); + psFree(where); + return false; + } + } + + /** selection based on posang **/ + psF32 select_posang_min = psMetadataLookupF32(&status, config->args, "-select_posang_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_posang_min"); + psFree(where); + return false; + } + if (isfinite(select_posang_min)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "posang", PS_META_DUPLICATE_OK, "<=", select_posang_min)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item posang"); + psFree(where); + return false; + } + } + psF32 select_posang_max = psMetadataLookupF32(&status, config->args, "-select_posang_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -select_posang_max"); + psFree(where); + return false; + } + if (isfinite(select_posang_max)) { + if (!psMetadataAddF32(where, PS_LIST_TAIL, "posang", PS_META_DUPLICATE_OK, ">=", select_posang_max)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item posang"); + psFree(where); + return false; + } + } + + } + + if (!psListLength(where->list)) { + psFree(where); + where = NULL; + } + + // there is some namespace overlap between the names of the fields we'd + // like to search by to setup a detrun and the names of the fields we'd + // like to assign values to so I've seperated them but prepending set- to + // the assigned values + + // optional + psString mode = psMetadataLookupStr(&status, config->args, "-mode"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -mode"); + return false; + } + // check mode + if (mode && !isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalud mode"); + return false; + } + + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return false; + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return false; + } + + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + + psF32 airmass_min = psMetadataLookupF32(&status, config->args, "-airmass_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass_min"); + return false; + } + + psF32 airmass_max = psMetadataLookupF32(&status, config->args, "-airmass_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass_max"); + return false; + } + + psF32 exp_time_min = psMetadataLookupF32(&status, config->args, "-exp_time_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time_min"); + return false; + } + + psF32 exp_time_max = psMetadataLookupF32(&status, config->args, "-exp_time_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time_max"); + return false; + } + + psF32 ccd_temp_min = psMetadataLookupF32(&status, config->args, "-ccd_temp_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp_min"); + return false; + } + + psF32 ccd_temp_max = psMetadataLookupF32(&status, config->args, "-ccd_temp_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp_max"); + return false; + } + + psF64 posang_min = psMetadataLookupF32(&status, config->args, "-posang_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang_min"); + return false; + } + + psF64 posang_max = psMetadataLookupF32(&status, config->args, "-posang_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang_max"); + return false; + } + + psTime *registered = NULL; + { + psString registeredStr = psMetadataLookupStr(&status, config->args, "-registered"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -registered"); + return false; + } + // pass through NULL as this is an optional field + if (registeredStr) { + registered = psTimeFromISO(registeredStr, PS_TIME_UTC); + if (!registered) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", registeredStr); + return false; + } + } else { + registered = NULL; + } + } + + psTime *time_begin = NULL; + { + psString time_beginStr = psMetadataLookupStr(&status, config->args, "-time_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time_begin"); + return false; + } + // pass through NULL as this is an optional field + if (time_beginStr) { + time_begin = psTimeFromISO(time_beginStr, PS_TIME_UTC); + if (!time_begin) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", time_beginStr); + return false; + } + } else { + time_begin = NULL; + } + } + + psTime *time_end = NULL; + { + psString time_endStr = psMetadataLookupStr(&status, config->args, "-time_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time_end"); + return false; + } + // pass through NULL as this is an optional field + if (time_endStr) { + time_end = psTimeFromISO(time_endStr, PS_TIME_UTC); + if (!time_end) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", time_endStr); + return false; + } + } else { + time_end = NULL; + } + } + + psTime *use_begin = NULL; + { + psString use_beginStr = psMetadataLookupStr(&status, config->args, "-use_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -use_begin"); + return false; + } + // pass through NULL as this is an optional field + if (use_beginStr) { + use_begin = psTimeFromISO(use_beginStr, PS_TIME_UTC); + if (!use_begin) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", use_beginStr); + return false; + } + } else { + use_begin = NULL; + } + } + + psTime *use_end = NULL; + { + psString use_endStr = psMetadataLookupStr(&status, config->args, "-use_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -use_end"); + return false; + } + // pass through NULL as this is an optional field + if (use_endStr) { + use_end = psTimeFromISO(use_endStr, PS_TIME_UTC); + if (!use_end) { + psError(PS_ERR_UNKNOWN, false, "error in time format %s", use_endStr); + return false; + } + } else { + use_end = NULL; + } + } + + bool simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return false; + } + + bool pretend = psMetadataLookupBool(&status, config->args, "-pretend"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -pretend"); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return false; + } + + // search for rawExps with the specified options + psArray *detrendExps = rawExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + // make sure that we found at least one rawExp + if (!detrendExps) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return false; + } + if (!psArrayLength(detrendExps)) { + psError(PS_ERR_UNKNOWN, true, "no rawExp rows found"); + psFree(detrendExps); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!rawExpPrintObjects(stdout, detrendExps, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detrendExps); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return false; + } + psFree(detrendExps); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return true; + } + + // start a transaction so we don't end up with orphaned det_ids + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detrendExps); + psFree(registered); + psFree(use_begin); + psFree(use_end); + return false; + } + + // the first iteration is always 0 + // XXX det_id + detRunInsert(config->dbh, + 0, // det_id + 0, // iteration + det_type, + mode, + "run", // state + workdir, + camera, + telescope, + "NA", + filter, + airmass_min, + airmass_max, + exp_time_min, + exp_time_max, + ccd_temp_min, + ccd_temp_max, + posang_min, + posang_max, + registered, + time_begin, + time_end, + use_begin, + use_end + ); + psFree(registered); + psFree(time_begin); + psFree(time_end); + psFree(use_end); + psFree(use_begin); + psFree(use_end); + long det_id = psDBLastInsertID(config->dbh); + + // create new detInputExp row(s) from the rawExp row(s) + psArray *inputExps = psArrayAllocEmpty(psArrayLength(detrendExps)); + for (long i = 0; i < psArrayLength(detrendExps); i++) { + detInputExpRow *inputExp = rawDetrenTodetInputExpRow( + detrendExps->data[i], + det_id, + 0 // the first iteration is explicitly 0 + ); + psArrayAdd(inputExps, 0, inputExp); + psFree(inputExp); + } + + psFree(detrendExps); + + // insert detInputExp objects into the database + if (!detInputExpInsertObjects(config->dbh, inputExps)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(inputExps); + return false; + } + psFree(inputExps); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + // print the new det_id + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", det_id); + detRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check results + if (psArrayLength(detRuns) != 1) { + psAbort(config->argv[0], "found more then one detRun matching det_id %ld(this should not happen)", det_id); + return false; + } + + // convert det_id to a string externaly + if (!convertDetIdToStr(detRuns)) { + psError(PS_ERR_UNKNOWN, false, "failed to convert det_id to a string"); + psFree(detRuns); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, detRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detRuns); + return false; + } + psFree(detRuns); + + return true; +} + +static bool definebydetrunMode(pxConfig *config) +{ + bool status = false; + + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id is the only required input + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + + // lookup the detRun that we will be basing this one on + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", (psS32)atoi(det_id)); + detRuns = detRunSelectRowObjects(config->dbh, where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + // sanity check the result... we should have only found one det_id + if (psArrayLength(detRuns) != 1) { + psAbort(config->argv[0], "found more then one detRun matching det_id %d(this should not happen)", (psS32)atoi(det_id)); + return false; // unreachable + } + + // pull the detRun object out the result array + detRunRow *detRun = psMemIncrRefCounter(detRuns->data[0]); + + // discard the resultarray + psFree(detRuns) + + // set the det_id to 0/NULL so the database can assign it + detRun->det_id = 0; + + // reset the iteration to 0 + detRun->iteration = 0; + + // reset the state to "run" + psFree(detRun->state); + detRun->state = psStringCopy("run"); + + // walk through the optional values and update the detRun as required + psString det_type = psMetadataLookupStr(&status, config->args, "-set_det_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_det_type"); + return false; + } + if (det_type) { + psFree(detRun->det_type); + detRun->det_type = psStringCopy(det_type); + } + + psString mode = psMetadataLookupStr(&status, config->args, "-set_mode"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_mode"); + return false; + } + // check mode + if (mode && !isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalud mode"); + return false; + } + if (mode) { + psFree(detRun->mode); + detRun->mode = psStringCopy(mode); + } + + psString camera = psMetadataLookupStr(&status, config->args, "-set_inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_inst"); + return false; + } + if (camera) { + psFree(detRun->camera); + detRun->camera = psStringCopy(camera); + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-set_telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_telescope"); + return false; + } + if (telescope) { + psFree(detRun->telescope); + detRun->telescope = psStringCopy(telescope); + } + + psString exp_type = psMetadataLookupStr(&status, config->args, "-set_exp_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_exp_type"); + return false; + } + if (exp_type) { + psFree(detRun->exp_type); + detRun->exp_type = psStringCopy(exp_type); + } + + psString filter = psMetadataLookupStr(&status, config->args, "-set_filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_filter"); + return false; + } + if (filter) { + psFree(detRun->filter); + detRun->filter = psStringCopy(filter); + } + + psF32 airmass_min = psMetadataLookupF32(&status, config->args, "-set_airmass_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_airmass_min"); + return false; + } + if (!isnan(airmass_min)) { + detRun->airmass_min = airmass_min; + } + + psF32 airmass_max = psMetadataLookupF32(&status, config->args, "-set_airmass_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_airmass_max"); + return false; + } + if (!isnan(airmass_max)) { + detRun->airmass_max = airmass_max; + } + + psF32 exp_time_min = psMetadataLookupF32(&status, config->args, "-set_exp_time_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_exp_time_min"); + return false; + } + if (!isnan(exp_time_min)) { + detRun->exp_time_min = exp_time_min; + } + + psF32 exp_time_max = psMetadataLookupF32(&status, config->args, "-set_exp_time_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_exp_time_max"); + return false; + } + if (!isnan(exp_time_max)) { + detRun->exp_time_max = exp_time_max; + } + + psF32 ccd_temp_min = psMetadataLookupF32(&status, config->args, "-set_ccd_temp_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_ccd_temp_min"); + return false; + } + if (!isnan(ccd_temp_min)) { + detRun->ccd_temp_min = ccd_temp_min; + } + + psF32 ccd_temp_max = psMetadataLookupF32(&status, config->args, "-set_ccd_temp_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_ccd_temp_max"); + return false; + } + if (!isnan(ccd_temp_max)) { + detRun->ccd_temp_max = ccd_temp_max; + } + + psF64 posang_min = psMetadataLookupF32(&status, config->args, "-set_posang_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_posang_min"); + return false; + } + if (!isnan(posang_min)) { + detRun->posang_min = posang_min; + } + + psF64 posang_max = psMetadataLookupF32(&status, config->args, "-set_posang_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_posang_max"); + return false; + } + if (!isnan(posang_max)) { + detRun->posang_max = posang_max; + } + + psTime *registered = NULL; + { + psString registeredStr = psMetadataLookupStr(&status, config->args, "-set_registered"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_registered"); + return false; + } + // pass through NULL as this is an optional field + if (registeredStr) { + psFree(detRun->registered); + registered = psTimeFromISO(registeredStr, PS_TIME_UTC); + detRun->registered = psMemIncrRefCounter(registered); + psFree(registered); + } + } + + psTime *time_begin = NULL; + { + psString time_beginStr = psMetadataLookupStr(&status, config->args, "-set_time_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_time_begin"); + return false; + } + // pass through NULL as this is an optional field + if (time_beginStr) { + psFree(detRun->time_begin); + time_begin = psTimeFromISO(time_beginStr, PS_TIME_UTC); + detRun->time_begin = psMemIncrRefCounter(time_begin); + psFree(time_begin); + } + } + + psTime *time_end = NULL; + { + psString time_endStr = psMetadataLookupStr(&status, config->args, "-set_time_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_time_end"); + return false; + } + // pass through NULL as this is an optional field + if (time_endStr) { + psFree(detRun->time_end); + time_end = psTimeFromISO(time_endStr, PS_TIME_UTC); + detRun->time_end = psMemIncrRefCounter(time_end); + psFree(time_end); + } + } + + psTime *use_begin = NULL; + { + psString use_beginStr = psMetadataLookupStr(&status, config->args, "-set_use_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_use_begin"); + return false; + } + // pass through NULL as this is an optional field + if (use_beginStr) { + psFree(detRun->use_begin); + use_begin = psTimeFromISO(use_beginStr, PS_TIME_UTC); + detRun->use_begin = psMemIncrRefCounter(use_begin); + psFree(use_begin); + } + } + + psTime *use_end = NULL; + { + psString use_endStr = psMetadataLookupStr(&status, config->args, "-set_use_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -set_use_end"); + return false; + } + // pass through NULL as this is an optional field + if (use_endStr) { + psFree(detRun->use_end); + use_end = psTimeFromISO(use_endStr, PS_TIME_UTC); + detRun->use_end = psMemIncrRefCounter(use_end); + psFree(use_end); + } + } + + // create a metadata to restrict detInputExp's be in in the specified range + + psMetadata *time_filter = psMetadataAlloc(); + + { + psString timeStr = psMetadataLookupStr(&status, config->args, "-filter_input_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter_input_begin"); + return false; + } + // pass through NULL as this is an optional field + if (timeStr) { + psTime *time = psTimeFromISO(timeStr, PS_TIME_UTC); + if (!psMetadataAddTime(time_filter, PS_LIST_TAIL, "dateobs", PS_META_DUPLICATE_OK, ">=", time)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item dateobs"); + psFree(time); + psFree(time_filter); + return false; + } + psFree(time); + } + } + + { + psString timeStr = psMetadataLookupStr(&status, config->args, "-filter_input_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter_input_end"); + return false; + } + // pass through NULL as this is an optional field + if (timeStr) { + psTime *time = psTimeFromISO(timeStr, PS_TIME_UTC); + if (!psMetadataAddTime(time_filter, PS_LIST_TAIL, "dateobs", PS_META_DUPLICATE_OK, "<", time)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item dateobs"); + psFree(time); + psFree(time_filter); + return false; + } + psFree(time); + } + } + + + // start a transaction so we don't end up with orphaned det_ids + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(time_filter); + psFree(detRun); + return false; + } + + if (!detRunInsertObject(config->dbh, detRun)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(time_filter); + psFree(detRun); + return false; + } + psFree(detRun); + + // get the det_id + long newDet_id = psDBLastInsertID(config->dbh); + + psString query = psStringCopy( + "INSERT INTO detInputExp" + " SELECT" + " %d," + " 0," + " detResidExp.exp_tag," + " detResidExp.accept" + " FROM detResidExp" + " JOIN rawExp" + " USING(exp_tag)" + " WHERE det_id = %d" + ); + + if (time_filter->list->n) { + psString whereClause = psDBGenerateWhereConditionSQL(time_filter, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(time_filter); + + if (!p_psDBRunQuery(config->dbh, query, (psS32)newDet_id, (psS32)atoi(det_id))) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // print the new det_id + psArray *newDetRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", newDet_id); + newDetRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!newDetRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check the result... we should have only found one det_id + if (psArrayLength(newDetRuns) != 1) { + psAbort(config->argv[0], "found more then one detRun matching det_id %ld(this should not happen)", newDet_id); + return false; // unreachable + } + + // convert det_id to a string externaly + if (!convertDetIdToStr(newDetRuns)) { + psError(PS_ERR_UNKNOWN, false, "failed to convert det_id to a string"); + psFree(newDetRuns); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, newDetRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(newDetRuns); + return false; + } + psFree(newDetRuns); + + return true; +} + +static bool runsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // XXX fix the hard coding of the table name + psArray *runs = psDBSelectRows(config->dbh, "detRun", config->where, 0); + if (!runs) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(runs)) { + psError(PS_ERR_UNKNOWN, false, "no detRun rows found"); + psFree(runs); + return true; + } + + // convert det_id to a string externaly + if (!convertDetIdToStr(runs)) { + psError(PS_ERR_UNKNOWN, false, "failed to convert det_id to a string"); + psFree(runs); + return false; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, runs, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(runs); + return false; + } + + psFree(runs); + + return true; +} + +static detInputExpRow *rawDetrenTodetInputExpRow(rawExpRow *rawExp, psS32 det_id, psS32 iteration) +{ + PS_ASSERT_PTR_NON_NULL(rawExp, NULL); + + return detInputExpRowAlloc( + det_id, + iteration, + rawExp->exp_tag, + true // use + ); +} + +static bool inputMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // select detInputExp.* + // select rawExp.* + // by: + // exp_tag + + psString query = psStringCopy( + "SELECT DISTINCT *" + " FROM detInputExp" + " JOIN rawExp" + " USING(exp_tag)" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereSQL(config->where, "detInputExp"); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no pending rawExp rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detInputExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool rawMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy( + "SELECT" + " detInputExp.det_id," + " detRun.det_type," + " rawImfile.*," + " rawExp.camera" + " FROM rawImfile" + " JOIN detInputExp" + " USING(exp_tag) " + " JOIN rawExp" + " USING(exp_tag) " + " JOIN detRun" + " ON detInputExp.det_id = detRun.det_id" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool toprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // select det_id, det_type & rawImfile * + // get det_id from detInputExp.det_id + // get det_type from detRun.det_id + // + // det_id is in detInputExp + // exp_tag is in detInputExp + // det_id is not in detProccessedImfile + // exp_tag is not in detProccessedImfile + // class_is is not in detProccessedImfile + psString query = psStringCopy( + "SELECT" + " detRun.det_id," + " detRun.det_type," + " detRun.workdir," + " rawImfile.*," + " rawExp.camera" + " FROM detRun" + " JOIN detInputExp" + " USING(det_id, iteration)" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN rawImfile" + " ON detInputExp.exp_tag = rawImfile.exp_tag" + " LEFT JOIN detProcessedImfile" + " ON detInputExp.det_id = detProcessedImfile.det_id" + " AND rawImfile.exp_tag = detProcessedImfile.exp_tag" + " AND rawImfile.class_id = detProcessedImfile.class_id" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + " AND detProcessedImfile.det_id IS NULL" + " AND detProcessedImfile.exp_tag IS NULL" + " AND detProcessedImfile.class_id IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static psArray *searchRawImfiles(pxConfig *config, psMetadata *where) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // use the default where if "where" is NULL + if (!where) { + where = config->where; + } + + // select exp_tags from detInputExp matching det_idp + // where query should be pre-generated + psArray *detInputExp = + detInputExpSelectRowObjects(config->dbh, where, 0); + if (!detInputExp) { + psError(PS_ERR_UNKNOWN, false, "no rawExp rows found"); + return NULL; + } + + // generate where query with just the exp_tags + psMetadata *where_exp_tags = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(detInputExp); i++) { + detInputExpRow *row = detInputExp->data[i]; + if (!psMetadataAddStr(where_exp_tags, PS_LIST_TAIL, "exp_tag", + PS_META_DUPLICATE_OK, "==", row->exp_tag) + ) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(detInputExp); + psFree(where_exp_tags); + return NULL; + } + } + psFree(detInputExp); + + // select rawImfiles with matching exp_tags + psArray *rawImfiles = + rawImfileSelectRowObjects(config->dbh, where_exp_tags, 0); + psFree(where_exp_tags); + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + return NULL; + } + + return rawImfiles; +} + +static bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, exp_tag, class_id, uri, recipe, -bg, -bg_stdev, & -bg_mean_stdev + // are required + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // find the matching rawImfile by exp_tag/class_id + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", 0, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + return false; + } + if (!psMetadataAddStr(where, PS_LIST_TAIL, "class_id", 0, "==", class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(where); + return false; + } + + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + psArray *rawImfiles = rawImfileSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found "); + return false; + } + + + // create a new detProcessedImfile object + detProcessedImfileRow *detRow = detProcessedImfileRowAlloc( + (psS32)atol(det_id), + exp_tag, + class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + code + ); + psFree(rawImfiles); + + // insert the new row into the detProcessedImfile table + if (!detProcessedImfileInsertObject(config->dbh, detRow)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detRow); + return false; + } + + psFree(detRow); + + return true; +} + +static bool toprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // select detProcessedImfile.det_id + // select detRun.iteration + // select detRun.det_type + // select detProcessedImfile.exp_tag + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // find all rawImfiles in the current exp_tags + // compare to detProcessedImfiles by det_id/exp_tag + // found how many imfile there are in each class_id + // and: + // det_id is not in detProcessedExp; + // iteration is not in detProcessedExp; + + psString query = psStringCopy( + " SELECT DISTINCT" + " det_id," + " iteration," + " det_type," + " exp_tag," + " camera," + " workdir" + " FROM" + " (SELECT DISTINCT" + " detRun.det_id," + " detRun.iteration," + " detRun.det_type," + " detRun.workdir," + " detProcessedImfile.exp_tag," + " rawExp.camera," + " detProcessedImfile.class_id," + " rawExp.imfiles" + " FROM detRun" + " JOIN detInputExp" + " USING(det_id, iteration)" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN rawImfile" + " ON rawExp.exp_tag = rawImfile.exp_tag" + " LEFT JOIN detProcessedImfile" + " ON detRun.det_id = detProcessedImfile.det_id" + " AND detInputExp.exp_tag = detProcessedImfile.exp_tag" + " AND rawImfile.class_id = detProcessedImfile.class_id" + " LEFT JOIN detProcessedExp" + " ON detProcessedImfile.det_id = detProcessedExp.det_id" + " AND detProcessedImfile.exp_tag = detProcessedExp.exp_tag" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + " AND detProcessedImfile.fault = 0" + " AND detProcessedExp.det_id IS NULL" + " AND detProcessedExp.exp_tag IS NULL" + " AND detInputExp.include = 1" + " GROUP BY" + " rawExp.exp_tag," + " detRun.det_id" + " HAVING" + " COUNT(detProcessedImfile.class_id) = rawExp.imfiles" + " ) AS detProcessedExp" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no pending rawExp rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, exp_tag, recip, -bg, -bg_stdev, & -bg_mean_stdev + // are required + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + psString query = psStringCopy( + " SELECT DISTINCT" + " detProcessedImfile.det_id," + " detRun.iteration," + " detRun.det_type," + " detProcessedImfile.exp_tag" + " FROM detRun" + " JOIN detInputExp" + " ON detRun.det_id = detInputExp.det_id" + " AND detRun.iteration = detInputExp.iteration" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN detProcessedImfile" + " ON detInputExp.det_id = detProcessedImfile.det_id" + " AND detInputExp.exp_tag = detProcessedImfile.exp_tag" + " LEFT JOIN detProcessedExp" + " ON detInputExp.det_id = detProcessedExp.det_id" + " AND detProcessedImfile.exp_tag= detProcessedExp.exp_tag" + " LEFT JOIN rawImfile" + " ON detInputExp.exp_tag = rawImfile.exp_tag" + " AND detProcessedImfile.class_id = rawImfile.class_id" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + " AND detProcessedExp.det_id IS NULL" + " AND detProcessedExp.exp_tag IS NULL" + " AND detInputExp.include = 1" + " AND detRun.det_id = %s" + " AND detProcessedImfile.exp_tag = '%s'" + " GROUP BY" + " detProcessedImfile.class_id," + " rawImfile.class_id," + " detRun.det_id" + " HAVING" + " COUNT(detProcessedImfile.class_id) = COUNT(rawImfile.class_id)" + ); + + if (!p_psDBRunQuery(config->dbh, query, det_id, exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no pending rawExp rows found"); + return true; + } + psFree(output); + + // create a new detProcessedImfile object + detProcessedExpRow *detRow = detProcessedExpRowAlloc( + (psS32)atol(det_id), + exp_tag, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + code + ); + + // insert the new row into the detProcessedImfile table + if (!detProcessedExpInsertObject(config->dbh, detRow)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detRow); + return false; + } + + psFree(detRow); + + return true; +} + +static bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy("SELECT * FROM detProcessedExp"); + + if (config->where) { + psString whereClause = psDBGenerateWhereSQL(config->where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detProcessedExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detProcessedExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detProcessedExp rows found"); + psFree(output) + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool tostackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // select detRun.iteration + // select detProcessedImfile.det_id + // select detProcessedImfile.class_id + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // find all rawImfiles in the current exp_tags + // compare to detProcessedImfiles by det_id/exp_tag + // found how many imfile there are in each class_id + // and: + // det_id is not in detStackedImfile; + // iteration is not in detStackedImfile; + // class_id is not in detStackedImfile; + + psString query = psStringCopy( + " SELECT\n" + " detProcessedImfile.det_id,\n" + " detRun.iteration,\n" + " detRun.det_type,\n" + " detRun.workdir,\n" + " detProcessedImfile.class_id,\n" + " rawExp.camera\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " ON detRun.det_id = detInputExp.det_id\n" + " AND detRun.iteration = detInputExp.iteration\n" + " JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " JOIN rawImfile\n" + " ON detInputExp.exp_tag = rawImfile.exp_tag\n" + " LEFT JOIN detProcessedImfile\n" + " ON detInputExp.det_id = detProcessedImfile.det_id\n" + " AND detInputExp.exp_tag = detProcessedImfile.exp_tag\n" + " AND rawImfile.class_id = detProcessedImfile.class_id\n" + " LEFT JOIN detStackedImfile\n" + " ON detInputExp.det_id = detStackedImfile.det_id\n" + " AND detInputExp.iteration = detStackedImfile.iteration\n" + " AND rawImfile.class_id = detStackedImfile.class_id\n" + " WHERE\n" + " detRun.state = 'run'\n" + " AND detRun.mode = 'master'\n" + " AND (detProcessedImfile.fault = 0 or detProcessedImfile.fault IS NULL)\n" + " AND detStackedImfile.det_id IS NULL\n" + " AND detStackedImfile.iteration IS NULL\n" + " AND detStackedImfile.class_id IS NULL\n" + " AND detInputExp.include = 1\n" + " GROUP BY\n" + " rawImfile.class_id,\n" + " detRun.det_id\n" + " HAVING\n" + " COUNT(detProcessedImfile.class_id) = COUNT(rawImfile.class_id)\n" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no stackable detProcessedImfiles found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingStackedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy( + " SELECT DISTINCT" + " detRun.det_type," + " detProcessedImfile.*" + " FROM detProcessedImfile" + " JOIN detRun" + " USING(det_id)" + " JOIN detInputExp" + " ON detRun.det_id = detInputExp.det_id" + " AND detRun.iteration = detInputExp.iteration" + " AND detProcessedImfile.exp_tag = detInputExp.exp_tag" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detProcessedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + { + bool status = false; + bool included = psMetadataLookupBool(&status, config->args, "-included"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + // restrict search to included imfiles + if (included) { + psStringAppend(&query, " AND detInputExp.include = 1"); + } + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detProcessedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detProcessedImfile.fault = 0"); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detProcessedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addstackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, class_id, uri, & recipe are required + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + + // default values + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); + return false; + } + + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + // correlate the class_id against the input exposure(s) + + // we have to generate our own where clause as we want to search only by the + // det_id + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", + (psS32)atoi(det_id))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + return false; + } + + psArray *rawImfiles = searchRawImfiles(config, where); + psFree(where); + + bool valid_class_id = false; + if (rawImfiles) { + for (long i = 0; i < psArrayLength(rawImfiles); i++) { + if (strcmp(class_id, ((rawImfileRow *)rawImfiles->data[i])->class_id) == 0) { + valid_class_id = true; + break; + } + } + psFree(rawImfiles); + } + + if (!valid_class_id) { + psError(PS_ERR_UNKNOWN, true, + "class_id can not be correlated with the input exposures"); + return false; + } + + // create a new detStackedImfile object + detStackedImfileRow *stackedImfile = detStackedImfileRowAlloc( + (psS32)atol(det_id), + iteration, + class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + code + ); + + // insert the new row into the detProcessedImfile table + if (!detStackedImfileInsertObject(config->dbh, stackedImfile)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(stackedImfile); + return false; + } + + psFree(stackedImfile); + + return true; +} + +static bool stackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + // select detStackedImfile.* + // by: + // where det_id, iteration, class_id is not in detNormalizedImfile + + psString query = psStringCopy( + "SELECT" + " detStackedImfile.*" + " FROM detStackedImfile" + " JOIN detRun" + " USING(det_id, iteration)" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detStackedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detStackedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detStackedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detStackedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(output); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool tonormalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // select detRun.det_id (det_id) + // select detRun.iteration + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // sort to detInputExp.imfiles to find the largest value per det_id/iter + // compare imfiles to the number of detStackedImfiles by class_id + // and: + // ??? + // det_id is not in detStackedImfile; + // iteration is not in detStackedImfile; + // class_id is not in detStackedImfile; + + psString query = psStringCopy( + "SELECT" + " det_id," + " det_type," + " iteration," + " camera," + " workdir" + " FROM" + "(SELECT DISTINCT" + " detRun.det_id," + " detRun.det_type," + " detRun.iteration," + " detRun.workdir," + " rawExp.camera," + " rawExp.imfiles," + " detStackedImfile.class_id" + " FROM detRun" + " JOIN detInputExp" + " ON detRun.det_id = detInputExp.det_id" + " AND detRun.iteration = detInputExp.iteration" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN detStackedImfile" + " ON detInputExp.det_id = detStackedImfile.det_id" + " AND detInputExp.iteration = detStackedImfile.iteration" + " LEFT JOIN detNormalizedStatImfile" + " ON detStackedImfile.det_id = detNormalizedStatImfile.det_id" + " AND detStackedImfile.iteration = detNormalizedStatImfile.iteration" + " AND detStackedImfile.class_id = detNormalizedStatImfile.class_id" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + " AND detStackedImfile.fault = 0" + " AND detNormalizedStatImfile.det_id IS NULL" + " AND detNormalizedStatImfile.iteration IS NULL" + " AND detNormalizedStatImfile.class_id IS NULL" + " GROUP BY" + " rawExp.exp_tag," + " detRun.iteration," + " detRun.det_id" + " HAVING MAX(rawExp.imfiles) = COUNT(detStackedImfile.class_id)" + ") as tonormalizedstat" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detNormalizedStatImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingNormStatImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addnormalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // select * from detStackedImfile + // by det_id, iteration, class_id + // where det_id, iteration, class_id is not in detNormalizedStatImfile + psString query = psStringCopy( + "SELECT DISTINCT" + " detStackedImfile.*" + " FROM detStackedImfile" + " LEFT JOIN detNormalizedStatImfile" + " USING(det_id, iteration, class_id)" + " WHERE" + " detNormalizedStatImfile.det_id IS NULL" + " AND detNormalizedStatImfile.iteration IS NULL" + " AND detNormalizedStatImfile.class_id IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detStackedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detStackedImfile rows found"); + psFree(output); + return true; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a detStackedImfile object + detStackedImfileRow *stackedImfile = detStackedImfileObjectFromMetadata(row); + // convert detStackedImfile object into a detNormalizedStat object + detNormalizedStatImfileRow *stat = detStackedToDetNormalizedStatImfile(config, stackedImfile); + psFree(stackedImfile); + if (!stat) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert detStackedImfile to detNormalizedStatImfile"); + psFree(output); + return false; + } + // insert detNormlized Stat object into the database + if (!detNormalizedStatImfileInsertObject(config->dbh, stat)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(stat); + psFree(output); + } + psFree(stat); + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static detNormalizedStatImfileRow *detStackedToDetNormalizedStatImfile(pxConfig *config, detStackedImfileRow *stackedImfile) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(stackedImfile, NULL); + + bool status = false; + psF32 norm = psMetadataLookupF32(&status, config->args, "-norm"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -norm"); + return false; + } + if (isnan(norm)) { + psError(PS_ERR_UNKNOWN, true, "-norm is required"); + return false; + } + + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + return detNormalizedStatImfileRowAlloc( + stackedImfile->det_id, + stackedImfile->iteration, + stackedImfile->class_id, + norm, + code + ); +} + +static bool tonormalizeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // select detNormalizedStatImfile.* + // by: + // where det_id, iteration, class_id is not in detNormalizedImfile + + psString query = psStringCopy( + "SELECT DISTINCT" + " detRun.det_type," + " detRun.workdir," + " rawExp.camera," + " detStackedImfile.uri," + " detNormalizedStatImfile.*" + " FROM detRun" + " JOIN detStackedImfile" + " USING(det_id, iteration)" + " JOIN detInputExp" + " USING(det_id, iteration)" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN detNormalizedStatImfile" + " ON detStackedImfile.det_id = detNormalizedStatImfile.det_id" + " AND detStackedImfile.iteration = detNormalizedStatImfile.iteration" + " AND detStackedImfile.class_id = detNormalizedStatImfile.class_id" + " LEFT JOIN detNormalizedImfile" + " ON detNormalizedStatImfile.det_id = detNormalizedImfile.det_id" + " AND detNormalizedStatImfile.iteration = detNormalizedImfile.iteration" + " AND detNormalizedStatImfile.class_id = detNormalizedImfile.class_id" + " WHERE" + " detNormalizedImfile.det_id IS NULL" + " AND detNormalizedImfile.iteration IS NULL" + " AND detNormalizedImfile.class_id IS NULL" + " AND detNormalizedStatImfile.fault = 0" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no normalizable detNormalizedStatImfiles found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(output); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingNormImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +#if 0 +// XXX this function was left in commented as this method may be useful in the +// future +static psArray *validDetInputClassIds(pxConfig *config, const char *det_id) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + // det_id is input as a string because the fact that it is an integer + // is just a database impliementation detail. + PS_ASSERT_PTR_NON_NULL(det_id, NULL); + + psArray *rawImfiles = searchInputImfiles(config, det_id); + if (!rawImfiles) { + return NULL; + } + + psArray *valid_class_ids = NULL; + { + // All this jumping through hoops is so that we end up with unique + // values. PP thinks that making multiple passes through this array and + // deleting matched elements would end up being more expensive then a + // double sort and stagger scheme. JH thinks it would be cheaper to + // just do a unqiue sort and delete. So this is really just a cheap + // hack to avoid implimenting a unique sort function but at least it's + // stable. + psHash *hash = psHashAlloc(psArrayLength(rawImfiles)); + for (long i = 0; i < psArrayLength(rawImfiles); i++) { + psHashAdd(hash, + ((rawImfileRow *)rawImfiles->data[i])->class_id, + ((rawImfileRow *)rawImfiles->data[i])->class_id + ); + } + valid_class_ids = psHashToArray(hash); + psFree(hash); + } + psFree(rawImfiles); + + return valid_class_ids; +} + +static psArray *searchInputImfiles(pxConfig *config, const char *det_id) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + // det_id is input as a string because the fact that it is an integer + // is just a database impliementation detail. + PS_ASSERT_PTR_NON_NULL(det_id, NULL); + + psArray *inputExps = NULL; + { + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", + (psS32)atoi(det_id))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return NULL; + } + inputExps = detInputExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + } + if (!inputExps) { + psError(PS_ERR_UNKNOWN, false, "no detInputExp rows found"); + return NULL; + } + + // find rawImfiles associated with detInputExps + psArray *rawImfiles = NULL; + { + psMetadata *where = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(inputExps); i++) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", + PS_META_DUPLICATE_OK, "==", + ((detInputExpRow *)inputExps->data[i])->exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(inputExps); + psFree(where); + return NULL; + } + } + psFree(inputExps); + rawImfiles = rawImfileSelectRowObjects(config->dbh, where, 0); + // XXX this really should be sorted for uniqueness + psFree(where); + } + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + return NULL; + } + + return rawImfiles; +} +#endif + +static bool addnormalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // make sure that there is a respondoing entry in detNormalizedStatImfile + // select * from detNormalizedStatImfile + // by det_id, iteration, class_id + // where det_id, iteration, class_id is not in detNormalizedImfile + psString query = psStringCopy( + "SELECT" + " detNormalizedStatImfile.*" + " FROM detNormalizedStatImfile" + " LEFT JOIN detNormalizedImfile" + " USING(det_id, iteration, class_id)" + " WHERE" + " detNormalizedImfile.det_id IS NULL" + " AND detNormalizedImfile.iteration IS NULL" + " AND detNormalizedImfile.class_id IS NULL" + ); + + { + // build a query to search by det_id, iteration, class_id + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + psFree(where); + psFree(query); + return false; + } + if (det_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "det_id", 0, "==", det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + psFree(query); + return false; + } + } + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); + psFree(where); + psFree(query); + return false; + } + // always set iteration + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", iteration)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + psFree(query); + return false; + } + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + psFree(where); + psFree(query); + return false; + } + if (class_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "class_id", 0, "==", class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(where); + psFree(query); + return false; + } + } + + // there's not + psString whereClause = psDBGenerateWhereConditionSQL(where, "detNormalizedStatImfile"); + psFree(where); + if (whereClause) { + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no pending detNormalizedStatImfile rows found"); + psFree(output); + return true; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a detNormalizedStatImfile object + detNormalizedStatImfileRow *statImfile = detNormalizedStatImfileObjectFromMetadata(row); + // convert detNormalizedStatImfile object into a detNormalizedImfile + detNormalizedImfileRow *normalizedImfile = detNormalizedStatToDetNormalizedmfile(config, statImfile); + psFree(statImfile); + if (!normalizedImfile) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert detStackedImfile to detNormalizedStatImfile"); + psFree(output); + return false; + } + // insert detNormlized Stat object into the database + if (!detNormalizedImfileInsertObject(config->dbh, normalizedImfile)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(normalizedImfile); + psFree(output); + } + psFree(normalizedImfile); + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool tonormalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + psString query = psStringCopy( + " SELECT DISTINCT" + " detRun.det_id as det_id," + " detRun.iteration," + " detRun.det_type," + " detRun.workdir," + " rawExp.camera," + " rawExp.telescope," + " rawExp.exp_type," + " rawExp.imfiles" + " FROM detRun" + " JOIN detInputExp" + " ON detRun.det_id = detInputExp.det_id" + " AND detRun.iteration = detInputExp.iteration" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN detNormalizedImfile" + " ON detInputExp.det_id = detNormalizedImfile.det_id" + " AND detInputExp.iteration = detNormalizedImfile.iteration" + " LEFT JOIN rawImfile" + " ON detInputExp.exp_tag = rawImfile.exp_tag" + " AND detNormalizedImfile.class_id = rawImfile.class_id" + " LEFT JOIN detNormalizedExp" + " ON detInputExp.det_id = detNormalizedExp.det_id" + " AND detInputExp.iteration = detNormalizedExp.iteration" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + " AND detNormalizedImfile.fault = 0" + " AND detNormalizedExp.det_id IS NULL" + " AND detNormalizedExp.iteration IS NULL" + " AND detInputExp.include = 1" + " GROUP BY" + " detNormalizedImfile.iteration," + " detRun.det_id" + " HAVING" + " COUNT(detNormalizedImfile.class_id) = COUNT(rawImfile.class_id)" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingNormExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addnormalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, recip, -bg, -bg_stdev, & -bg_mean_stdev + // are required + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + // iteration has a default value + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); return false; + } + + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + psString query = psStringCopy( + " SELECT DISTINCT" + " detRun.det_id as det_id," + " detRun.iteration," + " detRun.det_type," + " rawExp.camera," + " rawExp.telescope," + " rawExp.exp_type," + " rawExp.imfiles" + " FROM detRun" + " JOIN detInputExp" + " ON detRun.det_id = detInputExp.det_id" + " AND detRun.iteration = detInputExp.iteration" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN detNormalizedImfile" + " ON detInputExp.det_id = detNormalizedImfile.det_id" + " AND detInputExp.iteration = detNormalizedImfile.iteration" + " LEFT JOIN rawImfile" + " ON detInputExp.exp_tag = rawImfile.exp_tag" + " AND detNormalizedImfile.class_id = rawImfile.class_id" + " LEFT JOIN detNormalizedExp" + " ON detInputExp.det_id = detNormalizedExp.det_id" + " AND detInputExp.iteration = detNormalizedExp.iteration" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + " AND detNormalizedExp.det_id IS NULL" + " AND detNormalizedExp.iteration IS NULL" + " AND detInputExp.include = 1" + " AND detRun.det_id = %s" + " AND detNormalizedImfile.iteration = %d" + " GROUP BY" + " detNormalizedImfile.iteration," + " detRun.det_id" + " HAVING" + " COUNT(detNormalizedImfile.class_id) = COUNT(rawImfile.class_id)" + ); + + if (!p_psDBRunQuery(config->dbh, query, det_id, iteration)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no complete detNormalizedImfile exp found"); + psFree(output); + return true; + } + psFree(output); + + // create a new detProcessedImfile object + detNormalizedExpRow *detRow = detNormalizedExpRowAlloc( + (psS32)atol(det_id), + iteration, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + code + ); + + // insert the new row into the detProcessedImfile table + if (!detNormalizedExpInsertObject(config->dbh, detRow)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detRow); + return false; + } + + psFree(detRow); + + return true; +} + +static bool normalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy( + "SELECT" + " detNormalizedExp.*" + " FROM detNormalizedExp" + " JOIN detRun" + " USING(det_id, iteration)" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereSQL(config->where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detNormalizedExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detNormalizedExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detNormalizedExp rows found"); + psFree(output) + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static detNormalizedImfileRow *detNormalizedStatToDetNormalizedmfile(pxConfig *config, detNormalizedStatImfileRow *statImfile) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(statImfile, NULL); + + bool status = false; + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + return detNormalizedImfileRowAlloc( + statImfile->det_id, + statImfile->iteration, + statImfile->class_id, + uri, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + code + ); +} + +static bool toresidimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + /* +which returns a list of processed imfiles (with corresponding detrend +id, iteration, class id, uri, type) for which the appropriate stacked +imfile has been normalised, and which have not been masked out by +detResidImfileAnalysis; +*/ + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + psString query = psStringCopy( + "SELECT DISTINCT\n" + " detRun.det_id,\n" + " detRun.iteration,\n" + " detRun.det_type,\n" + " detRun.mode,\n" + " detRun.workdir,\n" + " detProcessedImfile.exp_tag,\n" + " detProcessedImfile.class_id,\n" + " detProcessedImfile.uri,\n" + " detNormalizedImfile.uri AS det_uri,\n" + " rawExp.camera\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " USING(det_id, iteration)\n" + " JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " JOIN detProcessedImfile\n" + " ON detRun.det_id = detProcessedImfile.det_id\n" + " AND detInputExp.exp_tag = detProcessedImfile.exp_tag\n" + " JOIN detNormalizedImfile\n" + " ON detRun.det_id = detNormalizedImfile.det_id\n" + " AND detRun.iteration = detNormalizedImfile.iteration\n" + " AND detProcessedImfile.class_id = detNormalizedImfile.class_id\n" + " LEFT JOIN detResidImfile\n" + " ON detRun.det_id = detResidImfile.det_id\n" + " AND detRun.iteration = detResidImfile.iteration\n" + " AND detProcessedImfile.exp_tag = detResidImfile.exp_tag\n" + " AND detProcessedImfile.class_id = detResidImfile.class_id\n" + " WHERE\n" + " detRun.state = 'run'\n" + " AND detRun.mode = 'master'\n" + " AND detNormalizedImfile.fault = 0\n" + " AND detResidImfile.det_id IS NULL\n" + " AND detResidImfile.iteration IS NULL\n" + " AND detResidImfile.exp_tag IS NULL\n" + " AND detResidImfile.class_id IS NULL\n" + " UNION" + " SELECT\n" + " detRun.det_id,\n" + " detRun.iteration,\n" + " detRun.det_type,\n" + " detRun.mode,\n" + " detRun.workdir,\n" + " rawImfile.exp_tag,\n" + " rawImfile.class_id,\n" + " rawImfile.uri,\n" + " 'NULL' AS det_uri,\n" + " rawExp.camera\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " USING(det_id, iteration)\n" + " JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " JOIN rawImfile\n" + " ON detInputExp.exp_tag = rawImfile.exp_tag\n" + " LEFT JOIN detResidImfile\n" + " ON detRun.det_id = detResidImfile.det_id\n" + " AND detRun.iteration = detResidImfile.iteration\n" + " AND rawImfile.exp_tag = detResidImfile.exp_tag\n" + " AND rawImfile.class_id = detResidImfile.class_id\n" + " WHERE\n" + " detRun.state = 'run'\n" + " AND detRun.mode = 'verify'\n" + " AND detResidImfile.det_id IS NULL\n" + " AND detResidImfile.iteration IS NULL\n" + " AND detResidImfile.exp_tag IS NULL\n" + " AND detResidImfile.class_id IS NULL\n" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no residable detNormalizedImfileis found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(output); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingResidImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool normalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy( + "SELECT" + " detNormalizedImfile.*" + " FROM detNormalizedImfile" + " JOIN detRun" + " USING(det_id, iteration)" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + ); + + if (config->where) { + bool status; + int iteration = psMetadataLookupS32 (&status, config->where, "iteration"); + if (status) { + psMetadataRemoveKey (config->where, "iteration"); + psMetadataAddS32 (config->where, PS_LIST_TAIL, "iteration", 0, "==", iteration); + } + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detNormalizedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detNormalizedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detNormalizedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detNormalizedImfile rows found"); + psFree(output) + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addresidimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // make sure that there is a coresponding entry in detNormalizedImfile + // and the exp_tag specified is valid + // select * from detNormalizedImfile + // by det_id, iteration, class_id + // where det_id, iteration, class_id is not in detResidImfile + psString query = psStringCopy( + " SELECT * FROM (" + "SELECT\n" + " detNormalizedImfile.*\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " USING(det_id, iteration)\n" + " JOIN detNormalizedImfile\n" + " USING(det_id, iteration)\n" + " LEFT JOIN detResidImfile\n" + " ON detRun.det_id = detResidImfile.det_id\n" + " AND detRun.iteration = detResidImfile.iteration\n" + " AND detInputExp.exp_tag = detResidImfile.exp_tag\n" + " AND detNormalizedImfile.class_id = detResidImfile.class_id\n" + " WHERE\n" + " detRun.state = 'run'\n" + " AND detRun.mode= 'master'\n" + " AND detResidImfile.det_id IS NULL\n" + " AND detResidImfile.iteration IS NULL\n" + " AND detResidImfile.exp_tag IS NULL\n" + " AND detResidImfile.class_id IS NULL\n" + " UNION\n" + " SELECT\n" + " detRun.det_id,\n" + " detRun.iteration,\n" + " rawImfile.class_id,\n" + " rawImfile.uri,\n" + " 0.0 as bg,\n" + " 0.0 as bg_stdev,\n" + " 0.0 as bg_mean_stdev,\n" + " 'NULL' as b1_uri,\n" + " 'NULL' as b2_uri,\n" + " 0 as fault\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " USING(det_id, iteration)\n" + " JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " JOIN rawImfile\n" + " ON detInputExp.exp_tag = rawImfile.exp_tag\n" + " LEFT JOIN detResidImfile\n" + " ON detRun.det_id = detResidImfile.det_id\n" + " AND detRun.iteration = detResidImfile.iteration\n" + " AND rawImfile.exp_tag = detResidImfile.exp_tag\n" + " AND rawImfile.class_id = detResidImfile.class_id\n" + " WHERE\n" + " detRun.state = 'run'\n" + " AND detRun.mode = 'verify'\n" + " AND detResidImfile.det_id IS NULL\n" + " AND detResidImfile.iteration IS NULL\n" + " AND detResidImfile.exp_tag IS NULL\n" + " AND detResidImfile.class_id IS NULL\n" + " AND detInputExp.exp_tag = '%1$s'\n" + ") AS foo\n" + " %2$s" + ); + + // build a query to search by det_id, iteration, class_id + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + psFree(where); + psFree(query); + return false; + } + if (det_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "det_id", 0, "==", det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + psFree(query); + return false; + } + } + + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); + psFree(where); + psFree(query); + return false; + } + // always set iteration + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", iteration)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + psFree(query); + return false; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + psFree(where); + psFree(query); + return false; + } + if (class_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "class_id", 0, "==", class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(where); + psFree(query); + return false; + } + } + + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psFree(where); + + // exp_tag is manadatory to cross check that this is a exp_tag for this + // detRun + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + psFree(whereClause); + psFree(query); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + psFree(whereClause); + psFree(query); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query, exp_tag, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereClause); + psFree(query); + return false; + } + psFree(whereClause); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending detNormalizedImfile rows found"); + psFree(output); + return false; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a detNormalizedImfile object + detNormalizedImfileRow *normalizedImfile = detNormalizedImfileObjectFromMetadata(row); + // convert detNormalizedImfile object into a detResidImfile + detResidImfileRow *residImfile = detNormalizedToDetResidImfile(config, normalizedImfile); + psFree(normalizedImfile); + if (!residImfile) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert detNormalizedImfile to detResidImfile"); + psFree(output); + return false; + } + // insert detResidImfile object into the database + if (!detResidImfileInsertObject(config->dbh, residImfile)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(residImfile); + psFree(output); + } + psFree(residImfile); + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static detResidImfileRow *detNormalizedToDetResidImfile(pxConfig *config, detNormalizedImfileRow *normalizedImfile) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(normalizedImfile, NULL); + + bool status = false; + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + // create a new detResidImfileRow and insert it + return detResidImfileRowAlloc( + normalizedImfile->det_id, + normalizedImfile->iteration, + exp_tag, + normalizedImfile->class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + code + ); +} + +static bool residimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + // select detResidImfile.* + // by: + // where det_id, iteration, exp_tag is not in detResidExp; + + psString query = psStringCopy( + "SELECT" + " detRun.det_type," + " detRun.mode," + " detResidImfile.*" + " FROM detResidImfile" + " JOIN detRun" + " USING(det_id, iteration)" + " WHERE" + " detRun.state = 'run'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detResidImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detResidImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detResidImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detResidImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(output); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawResidImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool toresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + /* +which returns a list of exposures for which all component class ids +have had residuals created. This list includes the detrend id, +iteration, exposure id, detrend type, and whether the exposure was +included in the stack for this iteration. + */ + + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + // select detRun.det_id + // select detRun.iteration + // select detRun.det_type + // select detInputExp.exp_tag + // select detInputExp.include + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // compare to detInputExp.imfiles to derResidImfile by class_id + // and: + // detResidImfile.{det_id, iteration, exp_tag} is not in detResidExp + + psString query = psStringCopy( + "SELECT DISTINCT\n" + " det_id,\n" + " iteration,\n" + " det_type,\n" + " mode,\n" + " exp_tag,\n" + " include,\n" + " camera,\n" + " workdir\n" + " FROM\n" + " (SELECT DISTINCT\n" + " detRun.det_id AS det_id,\n" + " detRun.iteration,\n" + " detRun.det_type,\n" + " detRun.mode,\n" + " detRun.workdir,\n" + " detInputExp.exp_tag,\n" + " detInputExp.include,\n" + " rawExp.imfiles,\n" + " rawExp.camera,\n" + " detResidImfile.class_id\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " USING(det_id, iteration)\n" + " JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " JOIN detResidImfile\n" + " ON detRun.det_id = detResidImfile.det_id\n" + " AND detRun.iteration = detResidImfile.iteration\n" + " AND detInputExp.exp_tag = detResidImfile.exp_tag\n" + " LEFT JOIN detResidExp\n" + " ON detResidImfile.det_id = detResidExp.det_id\n" + " AND detResidImfile.iteration = detResidExp.iteration\n" + " AND detResidImfile.exp_tag = detResidExp.exp_tag\n" + " WHERE\n" + " detRun.state = 'run'\n" + " AND detResidImfile.fault = 0\n" + " AND detResidExp.det_id IS NULL\n" + " AND detResidExp.iteration IS NULL\n" + " AND detResidExp.exp_tag IS NULL\n" + " GROUP BY\n" + " detInputExp.exp_tag,\n" + " detRun.iteration,\n" + " detRun.det_id\n" + " HAVING\n" + " rawExp.imfiles = COUNT(detResidImfile.class_id)\n" + " ) AS toresidexp\n" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no residExpable detResidImfiles found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingResidExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + /* +which returns a list of exposures for which all component class ids +have had residuals created. This list includes the detrend id, +iteration, exposure id, detrend type, and whether the exposure was +included in the stack for this iteration. + */ + + + // select detRun.det_id + // select detRun.iteration + // select detRun.det_type + // select detInputExp.exp_tag + // select detInputExp.include + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // compare to detInputExp.imfiles to derResidImfile by class_id + // and: + // detResidImfile.{det_id, iteration, exp_tag} is not in detResidExp + + psString query = psStringCopy( + "SELECT DISTINCT" + " det_id," + " iteration," + " exp_tag," + " include" + " FROM" + " (SELECT DISTINCT" + " detRun.det_id AS det_id," + " detRun.iteration," + " detRun.det_type," + " detInputExp.exp_tag," + " detInputExp.include," + " rawExp.imfiles" + " FROM detRun" + " JOIN detInputExp" + " USING(det_id, iteration)" + " JOIN rawExp" + " ON detInputExp.exp_tag = rawExp.exp_tag" + " JOIN detResidImfile" + " ON detRun.det_id = detResidImfile.det_id" + " AND detRun.iteration = detResidImfile.iteration" + " AND detInputExp.exp_tag = detResidImfile.exp_tag" + " LEFT JOIN detResidExp" + " ON detResidImfile.det_id = detResidExp.det_id" + " AND detResidImfile.iteration = detResidExp.iteration" + " AND detResidImfile.exp_tag = detResidExp.exp_tag" + " WHERE" + " detRun.state = 'run'" + " AND detResidExp.det_id IS NULL" + " AND detResidExp.iteration IS NULL" + " AND detResidExp.exp_tag IS NULL" + " GROUP BY" + " detInputExp.exp_tag," + " detRun.iteration," + " detRun.det_id" + " HAVING" + " rawExp.imfiles = COUNT(detResidImfile.class_id)" + " ) AS toresidexp" + ); + + { + // build a query to search by det_id, iteration, exp_tag + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + if (!psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", (psS32)atoi(det_id))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + psFree(query); + return false; + } + + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); + psFree(where); + psFree(query); + return false; + } + // always set iteration + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", iteration)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + psFree(query); + return false; + } + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + psFree(where); + psFree(query); + return false; + } + if (exp_tag) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", 0, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + psFree(query); + return false; + } + } + + // there's not + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + if (whereClause) { + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detResidImfile rows found"); + psFree(output); + return false; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a detResidExp object + detResidExpRow *residExp = mdToDetResidExp(config, row); + if (!residExp) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata to detResidExp"); + psFree(output); + return false; + } + // insert detResidExp object into the database + if (!detResidExpInsertObject(config->dbh, residExp)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(residExp); + psFree(output); + } + psFree(residExp); + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static detResidExpRow *mdToDetResidExp(pxConfig *config, psMetadata *row) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(row, NULL); + + bool status = false; + // values from row + psS32 det_id = psMetadataLookupS32(&status, row, "det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for det_id"); + return false; + } + if (isnan(det_id)) { + psError(PS_ERR_UNKNOWN, true, "det_id is required"); + return false; + } + psS32 iteration = psMetadataLookupS32(&status, row, "iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for iteration"); + return false; + } + psString exp_tag = psMetadataLookupStr(&status, row, "exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "exp_tag is required"); + return false; + } + // values from config + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + return false; + } + if (!recipe) { + psError(PS_ERR_UNKNOWN, true, "-recip is required"); + return false; + } + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return false; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return false; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return false; + } + // optional + bool reject = psMetadataLookupBool(&status, config->args, "-reject"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -reject"); + return false; + } + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + if (!b1_uri) { + psError(PS_ERR_UNKNOWN, true, "-b1_uri is required"); + return false; + } + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + // create a new detResidImfileRow and insert it + return detResidExpRowAlloc( + det_id, + iteration, + exp_tag, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + !reject, + code + ); +} + +static bool residexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy( + "SELECT" + " detRun.mode," + " detResidExp.*," + " detInputExp.include" + " FROM detRun" + " JOIN detInputExp" + " USING(det_id, iteration)" + " JOIN detResidExp" + " USING(det_id, iteration, exp_tag)" + " WHERE" + " detRun.state = 'run'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "detResidExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detResidExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detResidExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detResidExp rows found"); + psFree(output) + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detResidExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool residdetrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + /* which returns a list of detrend runs (with detrend id, iteration and + * detrend type) which have completed all residexps. + */ + + // select detRun.det_id + // select detRun.iteration + // select detRun.det_type + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // find all exp_tags in the current det_id/iteration from detResidExp + // compare the counts of exp_tags + + psString query = psStringCopy( + "SELECT DISTINCT\n" + " det_id,\n" + " iteration,\n" + " det_type,\n" + " mode,\n" + " camera\n" + " FROM\n" + " (SELECT DISTINCT\n" + " detRun.det_id,\n" + " detRun.iteration,\n" + " detRun.det_type,\n" + " detRun.mode,\n" + " detInputExp.exp_tag,\n" + " rawExp.camera\n" + " FROM detRun\n" + " JOIN detInputExp\n" + " USING(det_id, iteration)\n" + " JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " LEFT JOIN detResidExp\n" + " ON detRun.det_id = detResidExp.det_id\n" + " AND detRun.iteration = detResidExp.iteration\n" + " AND detInputExp.exp_tag = detResidExp.exp_tag\n" + " WHERE\n" + " detRun.state = 'run'\n" + " GROUP BY\n" + " detRun.det_id,\n" + " detRun.iteration\n" + " HAVING\n" + " COUNT(detResidExp.exp_tag) = COUNT(detInputExp.exp_tag)\n" + " ) AS residdetrun\n" + ); + + // XXX does it make sense to accept any search params? +#if 0 + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } +#endif + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no completed det runs found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detRejectExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool updateresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // build a query to search by det_id, iteration, exp_tag + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + psFree(where); + return false; + } + if (det_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "det_id", 0, "==", det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return false; + } + } + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); + psFree(where); + return false; + } + // always where iteration + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", iteration)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + return false; + } + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + psFree(where); + return false; + } + if (exp_tag) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", 0, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + return false; + } + } + + // find the values we're going to set + // copy everything but det_id, iteration, & exp_tag from the args and + // remove the '-' prefix + psMetadata *set = psMetadataAlloc(); + psString recipe = psMetadataLookupStr(&status, config->args, "-recip"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -recip"); + psFree(set); + psFree(where); + return false; + } + if (recipe) { + if (!psMetadataAddStr(set, PS_LIST_TAIL, "recipe", 0, "==", recipe)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item recipe"); + psFree(set); + psFree(where); + return false; + } + } + + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + psFree(set); + psFree(where); + return false; + } + if (!isnan(bg)) { + if (!psMetadataAddF64(set, PS_LIST_TAIL, "bg", 0, "==", bg)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bg"); + psFree(set); + psFree(where); + return false; + } + } + + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + psFree(set); + psFree(where); + return false; + } + if (!isnan(bg_stdev)) { + if (!psMetadataAddF64(set, PS_LIST_TAIL, "bg_stdev", 0, "==", bg_stdev)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bg_stdev"); + psFree(set); + psFree(where); + return false; + } + } + + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + psFree(set); + psFree(where); + return false; + } + if (!isnan(bg_mean_stdev)) { + if (!psMetadataAddF64(set, PS_LIST_TAIL, "bg_mean_stdev", 0, "==", bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bg_mean_stdev"); + psFree(set); + psFree(where); + return false; + } + } + + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + psFree(set); + psFree(where); + return false; + } + if (b1_uri) { + if (!psMetadataAddStr(set, PS_LIST_TAIL, "b1_uri", 0, "==", b1_uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item b1_uri"); + psFree(set); + psFree(where); + return false; + } + } + + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + psFree(set); + psFree(where); + return false; + } + if (b2_uri) { + if (!psMetadataAddStr(set, PS_LIST_TAIL, "b2_uri", 0, "==", b2_uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item b2_uri"); + psFree(set); + psFree(where); + return false; + } + } + + bool reject = psMetadataLookupBool(&status, config->args, "-reject"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -reject"); + psFree(set); + psFree(where); + return false; + } + if (!psMetadataAddBool(set, PS_LIST_TAIL, "accept", 0, "==", !reject)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item accept"); + psFree(set); + psFree(where); + return false; + } + + long changed = psDBUpdateRows(config->dbh, "detResidExp", where, set); + psFree(set); + psFree(where); + + if (changed < 0) { + psError(PS_ERR_UNKNOWN, false, "no rows were updated"); + return false; + } + + return true; +} + +static bool adddetrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // select detRun.det_id + // select detRun.iteration + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // find all exp_tags in the current det_id/iteration from detResidExp + // compare the counts of exp_tags + + psString query = psStringCopy( + "SELECT DISTINCT\n" + " det_id,\n" + " iteration\n" + " FROM\n" + " (SELECT DISTINCT\n" + " detRun.det_id,\n" + " detRun.iteration,\n" + " detInputExp.exp_tag\n" + " FROM detRun\n" + " LEFT JOIN detInputExp\n" + " ON detRun.det_id = detInputExp.det_id\n" + " AND detRun.iteration = detInputExp.iteration\n" + " LEFT JOIN rawExp\n" + " ON detInputExp.exp_tag = rawExp.exp_tag\n" + " LEFT JOIN detResidExp\n" + " ON detRun.det_id = detResidExp.det_id\n" + " AND detRun.iteration = detResidExp.iteration\n" + " AND detInputExp.exp_tag = detResidExp.exp_tag\n" + " WHERE\n" + " detRun.state = 'run'\n" + " GROUP BY\n" + " detRun.det_id,\n" + " detRun.iteration\n" + " HAVING\n" + " COUNT(detResidExp.exp_tag) = COUNT(detInputExp.exp_tag)\n" + " ) AS residdetrun\n" + ); + + { + // build a query to search by det_id, iteration, exp_tag + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + psFree(where); + psFree(query); + return false; + } + if (det_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "det_id", 0, "==", det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + psFree(query); + return false; + } + } + psS32 iteration = psMetadataLookupS32(&status, config->args, "-iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -iteration"); + psFree(where); + psFree(query); + return false; + } + // always set iteration + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", iteration)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + psFree(query); + return false; + } + + // there's not + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + if (whereClause) { + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no pending completed detrun iterations found"); + psFree(output); + return true; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a detResidExp object + detRunSummaryRow *runSummary = mdToDetRunSummary(config, row); + if (!runSummary) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata to detResidExp"); + psFree(output); + return false; + } + // insert detResidExp object into the database + if (!detRunSummaryInsertObject(config->dbh, runSummary)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(runSummary); + psFree(output); + return false; + } + psFree(runSummary); + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; +} + +static detRunSummaryRow *mdToDetRunSummary(pxConfig *config, psMetadata *row) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + // from row + psS32 det_id = psMetadataLookupS32(&status, row, "det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for det_id"); + return false; + } + psS32 iteration = psMetadataLookupS32(&status, row, "iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for iteration"); + return false; + } + + // from config->args + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return NULL; + } + if (isnan(bg)) { + psError(PS_ERR_UNKNOWN, true, "-bg is required"); + return NULL; + } + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return NULL; + } + if (isnan(bg_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_stdev is required"); + return NULL; + } + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return NULL; + } + if (isnan(bg_mean_stdev)) { + psError(PS_ERR_UNKNOWN, true, "-bg_mean_stdev is required"); + return NULL; + } + // optional + bool accept = psMetadataLookupBool(&status, config->args, "-accept"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -accept"); + return NULL; + } + // default values + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + return detRunSummaryRowAlloc( + det_id, + iteration, + bg, + bg_stdev, + bg_mean_stdev, + accept, + code + ); +} + +static bool detrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -faulted"); + return false; + } + + psString query = psStringCopy( + "SELECT DISTINCT\n" + " detRunSummary.*,\n" + " detRun.det_type,\n" + " detRun.mode\n" + " FROM detRun\n" + " JOIN detRunSummary\n" + " USING(det_id, iteration)\n" + " WHERE\n" + " detRun.state = 'run'\n" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND p2ProcessedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND p2ProcessedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no detRunSummary rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawDetrendImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool updatedetrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id is required + // XXX this isn't strictly nessicary but not having the det_id complicates + // incrementing the iteration number + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + // either -rerun or -state must be specified + bool again = psMetadataLookupBool(&status, config->args, "-again"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -again"); + return false; + } + psString state = psMetadataLookupStr(&status, config->args, "-state"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -state"); + return false; + } + if (!(again || state)) { + psError(PS_ERR_UNKNOWN, true, "either -again or -state must be specified"); + return false; + } + if (again && state) { + psError(PS_ERR_UNKNOWN, true, "either -again or -state must be specified"); + return false; + } + + if (state) { + // set detRun.state to state + return setDetRunState(config, det_id, state); + } + + // else + // -again + + // select detRun.det_id + // select detRun.iteration + // select detInputExp.exp_tag + // select detResidExp.accept + // by: + // find the current iteration bassed on det_id + // find all exp_tags in the current det_id/iteration from detInputExp + // find all exp_tags in the current det_id/iteration from detResidExp + // compare the counts of exp_tags + + psString query = psStringCopy( + "SELECT DISTINCT" + " detRun.det_id AS det_id," + " detRun.iteration," + " detInputExp.exp_tag," + " detResidExp.accept" + " FROM detRun" + " JOIN detInputExp" + " ON detRun.det_id = detInputExp.det_id" + " AND detRun.iteration = detInputExp.iteration" + " JOIN detResidExp" + " ON detRun.det_id = detResidExp.det_id" + " AND detRun.iteration = detResidExp.iteration" + " AND detInputExp.exp_tag = detResidExp.exp_tag" + " WHERE" + " detRun.state = 'run'" + " AND detRun.mode = 'master'" + ); + + // XXX this query was not restricted by det_id, resulting + // in an inconsistent UPDATE below. I added this AND clause + // though there may be a cleaner method (EAM 2006.10.08) + psStringAppend ( + &query, + " AND detRun.det_id = '%s'", det_id); + + psStringAppend ( + &query, + " GROUP BY" + " detRun.det_id," + " detRun.iteration," + " detInputExp.exp_tag" + " HAVING" + " COUNT(detResidExp.exp_tag) = COUNT(detInputExp.exp_tag)" + ); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no completed detrun iterations found"); + psFree(output); + return true; + } + + // start a transaction so we don't end up with an incremented iteration + // count but no detInputExps + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // up the detRuns iteration count for for the single det_id we are + // operating on + // XXX this will have to changed in order to support multiple det_ids with + // a single invocation of this functions + psS32 newIteration = incrementIteration(config, det_id); + if (!newIteration) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + psString exp_tag = psMetadataLookupStr(&status, row, "exp_tag"); + if (!status) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for exp_tag"); + psFree(output); + return false; + } + bool accept = psMetadataLookupBool(&status, row, "accept"); + if (!status) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for accept"); + psFree(output); + return false; + } + + // detResidExp.include is used to set detInputExp.include + if (!detInputExpInsert( + config->dbh, + (psS32)atol(det_id), + newIteration, + exp_tag, + accept + ) + ) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + } + + psFree(output); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool rerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id is required + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + + // we have to support multipe exp_tags + psMetadataItem *item = psMetadataLookup(config->args, "-exp_tag"); + if (!item) { + // this shouldn't actually happen when using psArgs + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + + psList *exp_tag_list = item->data.list; + psMetadata *where = psMetadataAlloc(); + // make sure that -exp_tag was parsed correctly + // XXX this can be removed someday + if (item->type == PS_DATA_METADATA_MULTI) { + psListIterator *iter = psListIteratorAlloc(item->data.list, 0, false); + psMetadataItem *mItem = NULL; + while ((mItem = psListGetAndIncrement(iter))) { + psString exp_tag = mItem->data.V; + // if exp_tag is NULL then it means that -exp_tag has not been + // specified + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, + "at least one -exp_tag is required"); + psFree(where); + return false; + } + + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", + PS_META_DUPLICATE_OK, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(iter); + psFree(where); + return false; + } + } + psFree(iter); + } else { + psAbort(config->argv[0], + "-exp_tag was not parsed correctly (this should not happen"); + } + + // check that the specified exp_tags actually exist in the iteration zero + // detInputExp set + + // add the det_id & iteration == 0 to the where clause + if (!psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", + (psS32)atol(det_id))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return false; + } + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + return false; + } + + psArray *detrendExps = detInputExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!detrendExps) { + psError(PS_ERR_UNKNOWN, false, "no rawExp rows found"); + psFree(where); + return false; + } + + // build a hash for the valid exp_tags + psHash *valid_exp_tags = psHashAlloc(psArrayLength(detrendExps)); + for (long i = 0; i < psArrayLength(detrendExps); i++) { + psHashAdd(valid_exp_tags, + ((detInputExpRow *)detrendExps->data[i])->exp_tag, + detrendExps->data[i] + ); + } + psFree(detrendExps); + + // start a transaction so we don't end up with an incremented iteration + // count but no detInputExps + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(valid_exp_tags); + return false; + } + + // up the detRuns iteration count + psS32 newIteration = incrementIteration(config, det_id); + if (!newIteration) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(valid_exp_tags); + return false; + } + + // check exp_tags and build up an array of new detInputExp rows at the same + // time + psListIterator *iter = psListIteratorAlloc(exp_tag_list, 0, false); + psMetadataItem *mItem = NULL; + psArray *newInputExps = psArrayAllocEmpty(psListLength(exp_tag_list)); + while ((mItem = psListGetAndIncrement(iter))) { + detInputExpRow *inputExp = psHashLookup(valid_exp_tags, + (char *)mItem->data.V); + if (!inputExp) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + // invalid exp_tag + psError(PS_ERR_UNKNOWN, false, "exp_tag %s is invalid for det_id %s", + (char *)mItem->data.V, det_id); + psFree(iter); + psFree(valid_exp_tags); + return false; + } + detInputExpRow *newInputExp = detInputExpRowAlloc( + (psS32)atol(det_id), + newIteration, + inputExp->exp_tag, + true // use + ); + psArrayAdd(newInputExps, 0, newInputExp); + psFree(newInputExp); + } + psFree(iter); + psFree(valid_exp_tags); + + for (long i = 0; i < psArrayLength(newInputExps); i++) { + if (!detInputExpInsertObject(config->dbh, newInputExps->data[i])) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(newInputExps); + return false; + } + } + psFree(newInputExps); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool register_detrendMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + bool status = false; + psString det_type = psMetadataLookupStr(&status, config->args, "-det_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_type"); + return false; + } + if (!det_type) { + psError(PS_ERR_UNKNOWN, true, "-det_type is required"); + return false; + } + + psString workdir = psMetadataLookupStr(&status, config->args, "-workdir"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -workdir"); + return false; + } + if (!workdir) { + psError(PS_ERR_UNKNOWN, true, "-workdir is required"); + return false; + } + + // everything else is optional + psString mode = psMetadataLookupStr(&status, config->args, "-mode"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -mode"); + return false; + } + // check mode + if (mode && !isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalud mode"); + return false; + } + + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return false; + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return false; + } + + psString exp_type = psMetadataLookupStr(&status, config->args, "-exp_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_type"); + return false; + } + + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + + psF32 airmass_min = psMetadataLookupF32(&status, config->args, "-airmass_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass_min"); + return false; + } + + psF32 airmass_max = psMetadataLookupF32(&status, config->args, "-airmass_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass_max"); + return false; + } + + psF32 exp_time_min = psMetadataLookupF32(&status, config->args, "-exp_time_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time_min"); + return false; + } + + psF32 exp_time_max = psMetadataLookupF32(&status, config->args, "-exp_time_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time_max"); + return false; + } + + psF32 ccd_temp_min = psMetadataLookupF32(&status, config->args, "-ccd_temp_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp_min"); + return false; + } + + psF32 ccd_temp_max = psMetadataLookupF32(&status, config->args, "-ccd_temp_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp_max"); + return false; + } + + psF64 posang_min = psMetadataLookupF32(&status, config->args, "-posang_min"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang_min"); + return false; + } + + psF64 posang_max = psMetadataLookupF32(&status, config->args, "-posang_max"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang_max"); + return false; + } + + psTime *registered = NULL; + { + psString registeredStr = psMetadataLookupStr(&status, config->args, "-registered"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -registered"); + return false; + } + // pass through NULL as this is an optional field + if (registeredStr) { + registered = psTimeFromISO(registeredStr, PS_TIME_UTC); + } else { + registered = NULL; + } + } + + psTime *time_begin = NULL; + { + psString time_beginStr = psMetadataLookupStr(&status, config->args, "-time_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time_begin"); + return false; + } + // pass through NULL as this is an optional field + if (time_beginStr) { + time_begin = psTimeFromISO(time_beginStr, PS_TIME_UTC); + } else { + time_begin = NULL; + } + } + + psTime *time_end = NULL; + { + psString time_endStr = psMetadataLookupStr(&status, config->args, "-time_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -time_end"); + return false; + } + // pass through NULL as this is an optional field + if (time_endStr) { + time_end = psTimeFromISO(time_endStr, PS_TIME_UTC); + } else { + time_end = NULL; + } + } + + psTime *use_begin = NULL; + { + psString use_beginStr = psMetadataLookupStr(&status, config->args, "-use_begin"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -use_begin"); + return false; + } + // pass through NULL as this is an optional field + if (use_beginStr) { + use_begin = psTimeFromISO(use_beginStr, PS_TIME_UTC); + } else { + use_begin = NULL; + } + } + + psTime *use_end = NULL; + { + psString use_endStr = psMetadataLookupStr(&status, config->args, "-use_end"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -use_end"); + return false; + } + // pass through NULL as this is an optional field + if (use_endStr) { + use_end = psTimeFromISO(use_endStr, PS_TIME_UTC); + } else { + use_end = NULL; + } + } + + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!detRunInsert(config->dbh, + 0, // det_id + 0, // the iteration is fixed at 0 + det_type, + mode, + "reg", // state + workdir, + camera, + telescope, + exp_type, + filter, + airmass_min, + airmass_max, + exp_time_min, + exp_time_max, + ccd_temp_min, + ccd_temp_max, + posang_min, + posang_max, + registered, + time_begin, + time_end, + use_begin, + use_end + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(registered); + return false; + } + psFree(registered); + + if (!detRunSummaryInsert(config->dbh, + psDBLastInsertID(config->dbh), + 0, // the iteration is fixed at 0 + bg, + bg_stdev, + bg_mean_stdev, + true, // accept + 0 // fault code + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool register_detrend_imfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + bool status = false; + psString det_id = psMetadataLookupStr(&status, config->args, "-det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -det_id"); + return false; + } + if (!det_id) { + psError(PS_ERR_UNKNOWN, true, "-det_id is required"); + return false; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + // everything else is optional + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // XXX this is going in without checking that the det_id is valid. + // it seems like it be better to be using foreign key constraints rather + // then having to resort to a seperate query (and locking) to check that + // the det_id is valid + if (!detNormalizedImfileInsert(config->dbh, + (psS32)atoi(det_id), + 0, // the iteration is fixed at 0 + class_id, + uri, + bg, + bg_stdev, + bg_mean_stdev, + b1_uri, + b2_uri, + 0 // fault code + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static psS32 incrementIteration(pxConfig *config, const char *det_id) +{ + // this function returns zero on error + PS_ASSERT_PTR_NON_NULL(config, 0); + PS_ASSERT_PTR_NON_NULL(det_id, 0); + + char *query = "UPDATE detRun SET iteration = iteration + 1 WHERE det_id = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, det_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to increment iteration for det_id %s", det_id); + return 0; + } + + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", + (psS32)atol(det_id))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return 0; + } + + psArray *detRuns = detRunSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "no detRun rows found"); + return 0; + } + + // sanity check the database + if (psArrayLength(detRuns) != 1) { + // this should no happen + psAbort(config->argv[0], + "database query return too many rows (this should not happen"); + } + + psS32 newIteration = ((detRunRow *)detRuns->data[0])->iteration; + psFree(detRuns); + + return newIteration; +} + +static bool setDetRunState(pxConfig *config, const char *det_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(det_id, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!( + (strncmp(state, "run", 4) == 0) + || (strncmp(state, "stop", 5) == 0) + || (strncmp(state, "reg", 4) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid detRun state: %s", state); + return false; + } + + char *query = "UPDATE detRun SET state = '%s' WHERE det_id = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, state, det_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for det_id %s", det_id); + return false; + } + + return true; +} + +static bool isValidMode(pxConfig *config, const char *mode) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(mode, false); + + // check that state is a valid string value + if (!( + (strncmp(mode, "master", 7) == 0) + || (strncmp(mode, "verify", 7) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid detRun mode: %s", mode); + return false; + } + + return true; +} Index: /branches/rel-1-0/ippTools/src/dettool.h =================================================================== --- /branches/rel-1-0/ippTools/src/dettool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/dettool.h (revision 11645) @@ -0,0 +1,69 @@ +/* + * dettool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef DETTOOL_H +#define DETTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + DETTOOL_MODE_NONE = 0x0, + DETTOOL_MODE_PENDING, + DETTOOL_MODE_DEFINEBYTAG, + DETTOOL_MODE_DEFINEBYQUERY, + DETTOOL_MODE_DEFINEBYDETRUN, + DETTOOL_MODE_RUNS, + DETTOOL_MODE_INPUT, + DETTOOL_MODE_RAW, + DETTOOL_MODE_TOPROCESSEDIMFILE, + DETTOOL_MODE_ADDPROCESSEDIMFILE, + DETTOOL_MODE_PROCESSEDIMFILE, + DETTOOL_MODE_TOPROCESSEDEXP, + DETTOOL_MODE_ADDPROCESSEDEXP, + DETTOOL_MODE_PROCESSEDEXP, + DETTOOL_MODE_TOSTACKED, + DETTOOL_MODE_ADDSTACKED, + DETTOOL_MODE_STACKED, + DETTOOL_MODE_TONORMALIZE, + DETTOOL_MODE_ADDNORMALIZEDSTAT, + DETTOOL_MODE_TONORMALIZEDSTAT, + DETTOOL_MODE_ADDNORMALIZEDIMFILE, + DETTOOL_MODE_NORMALIZEDIMFILE, + DETTOOL_MODE_TONORMALIZEDEXP, + DETTOOL_MODE_ADDNORMALIZEDEXP, + DETTOOL_MODE_NORMALIZEDEXP, + DETTOOL_MODE_TORESIDIMFILE, + DETTOOL_MODE_ADDRESIDIMFILE, + DETTOOL_MODE_RESIDIMFILE, + DETTOOL_MODE_TORESIDEXP, + DETTOOL_MODE_RESIDEXP, + DETTOOL_MODE_ADDRESIDEXP, + DETTOOL_MODE_RESIDDETRUN, + DETTOOL_MODE_UPDATERESIDEXP, + DETTOOL_MODE_ADDDETRUNSUMMARY, + DETTOOL_MODE_DETRUNSUMMARY, + DETTOOL_MODE_UPDATEDETRUN, + DETTOOL_MODE_RERUN, + DETTOOL_MODE_REGISTER_DETREND, + DETTOOL_MODE_REGISTER_DETREND_IMFILE +} dettoolMode; + +pxConfig *dettoolConfig(pxConfig *config, int argc, char **argv); + +#endif // DETTOOL_H Index: /branches/rel-1-0/ippTools/src/dettoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/dettoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/dettoolConfig.c (revision 11645) @@ -0,0 +1,981 @@ +/* + * dettoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "dettool.h" + +// this function can not fail -- exits on error +pxConfig *dettoolConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + psTime *time = psTimeGetNow(PS_TIME_TAI); + psString now = psTimeToISO(time); + psFree(time); + + // -pending + psMetadata *pendingArgs = psMetadataAlloc(); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-exp_type", 0, + "search by exposure type", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-inst", 0, + "search by camera", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-telescope", 0, + "search by telescope", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-filter", 0, + "search by filter", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-uri", 0, + "search by URL", NULL); + psMetadataAddBool(pendingArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -definebytag + psMetadata *definebytagArgs = psMetadataAlloc(); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-exp_tag", + PS_META_DUPLICATE_OK, + "include this exposure (multiple OK, required)", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-det_type", 0, + "define the type of detrend run (required)", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-mode", 0, + "define the mode of this detrend run", "master"); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-workdir", 0, + "define workdir (required)", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-inst", 0, + "define camera", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-filter", 0, + "define filter ", NULL); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-airmass_min", 0, + "define min airmass", NAN); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-airmass_max", 0, + "define max airmass", NAN); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-exp_time_min", 0, + "define min exposure time", NAN); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-exp_time_max", 0, + "define max exposure time", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, + "define min ccd tempature", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, + "define max ccd tempature", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-posang_min", 0, + "define min rotator position angle", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-posang_max", 0, + "define max rotator position angle", NAN); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-registered", 0, + "time detrend run was registered", now); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-time_begin", 0, + "detrend applyes to exposures taken during this peroid", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-time_end", 0, + "detrend applyes to exposures taken during this peroid", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-use_begin", 0, + "start of detrend run applicable peroid", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-use_end", 0, + "end of detrend run applicable peroid", NULL); + psMetadataAddBool(definebytagArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-det_type", 0, + "define the type of detrend run (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-mode", 0, + "define the mode of this detrend run", "master"); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-workdir", 0, + "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-inst", 0, + "define camera", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-filter", 0, + "define filter ", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-airmass_min", 0, + "define min airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-airmass_max", 0, + "define max airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-exp_time_min", 0, + "define min exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-exp_time_max", 0, + "define max exposure time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, + "define min ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, + "define max ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-posang_min", 0, + "define min rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-posang_max", 0, + "define max rotator position angle", NAN); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-registered", 0, + "time detrend run was registered", now); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-time_begin", 0, + "detrend applyes to exposures taken during this peroid", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-time_end", 0, + "detrend applyes to exposures taken during this peroid", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-use_begin", 0, + "start of detrend run applicable peroid", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-use_end", 0, + "end of detrend run applicable peroid", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_type", 0, + "search for exp_type", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_inst", 0, + "search for camera", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_telescope", 0, + "search for telescope", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_filter", 0, + "search for filter", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_uri", 0, + "search for uri", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_dateobs_begin", 0, + "search for exposures by time (>=)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_dateobs_end", 0, + "search for exposures by time (<)", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_airmass_min", 0, + "define min airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_airmass_max", 0, + "define max airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_time_min", 0, + "define min exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_time_max", 0, + "define max exposure time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_ccd_temp_min", 0, + "define min ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_ccd_temp_max", 0, + "define max ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_posang_min", 0, + "define min rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_posang_max", 0, + "define max rotator position angle", NAN); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, + "print the exposures that would be included in the detrend run and exit", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -definebydetrun + psMetadata *definebydetrunArgs = psMetadataAlloc(); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-det_id", 0, + "det ID to base a new detRun on (required)", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_det_type", 0, + "define the type of detrend run (required)", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_mode", 0, + "define the mode of this detrend run", "master"); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_exp_type", 0, + "define exposure type", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_workdir", 0, + "define workdir", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_filter", 0, + "define filter ", NULL); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_airmass_min", 0, + "define airmass", NAN); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_airmass_max", 0, + "define airmass", NAN); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_exp_time_min", 0, + "define exposure time", NAN); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_exp_time_max", 0, + "define exposure time", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_ccd_temp_min", 0, + "define ccd tempature", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_ccd_temp_max", 0, + "define ccd tempature", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_posang_min", 0, + "define rotator position angle", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_posang_max", 0, + "define rotator position angle", NAN); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_registered", 0, + "time detrend run was registered", now); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_time_begin", 0, + "start of peroid to apply detrend too", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_time_end", 0, + "end of peroid to apply detrend too", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_use_begin", 0, + "start of detrend run applicable peroid", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_use_end", 0, + "end of detrend run applicable peroid", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-filter_input_begin", 0, + "filter input detrun exp to be in this peroid", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-filter_input_end", 0, + "filter input detrun exp to be in this peroid", NULL); + psMetadataAddBool(definebydetrunArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -runs + psMetadata *runsArgs = psMetadataAlloc(); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-det_type", 0, + "search for type of detrend run", NULL); + psMetadataAddBool(runsArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -input + psMetadata *inputArgs = psMetadataAlloc(); + psMetadataAddStr(inputArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID", NULL); + psMetadataAddS32(inputArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(inputArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search for exp ID", NULL); + psMetadataAddBool(inputArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -raw + psMetadata *rawArgs = psMetadataAlloc(); + psMetadataAddBool(rawArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -toprocessedimfile + psMetadata *toprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(toprocessedimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID", NULL); + psMetadataAddStr(toprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search for exp ID", NULL); + psMetadataAddStr(toprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search for class ID", NULL); + psMetadataAddU64(toprocessedimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(toprocessedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exp ID (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URI (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addprocessedimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search for exp ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search for class ID", NULL); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-chip", 0, + "restrict results to completed 'chip' sets", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-exp", 0, + "restrict results to complete 'exposures'", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-included", 0, + "restrict results to exposures 'includeded' in the current iteration", false); + psMetadataAddU64(processedimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -toprocessedexp + psMetadata *toprocessedexpArgs = psMetadataAlloc(); + psMetadataAddU64(toprocessedexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(toprocessedexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addproccessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addprocessedexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -proccessedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define detrend ID (required)", NULL); + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -tostacked + psMetadata *tostackedArgs = psMetadataAlloc(); + psMetadataAddU64(tostackedArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(tostackedArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addstacked + psMetadata *addstackedArgs = psMetadataAlloc(); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(addstackedArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", NULL); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-uri", 0, + "define URI (required)", NULL); + psMetadataAddS8(addstackedArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -stacked + psMetadata *stackedArgs = psMetadataAlloc(); + psMetadataAddStr(stackedArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID", NULL); + psMetadataAddS32(stackedArgs, PS_LIST_TAIL, "-iteration", 0, + "search for iteration number", 0); + psMetadataAddStr(stackedArgs, PS_LIST_TAIL, "-class_id", 0, + "search for class ID", NULL); + psMetadataAddStr(stackedArgs, PS_LIST_TAIL, "-recip", 0, + "search for recipe", NULL); + psMetadataAddU64(stackedArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(stackedArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(stackedArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -tonormalize + psMetadata *tonormalizeArgs = psMetadataAlloc(); + psMetadataAddU64(tonormalizeArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(tonormalizeArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addnormstat + psMetadata *addnormstatArgs = psMetadataAlloc(); + psMetadataAddStr(addnormstatArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(addnormstatArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(addnormstatArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID", NULL); + psMetadataAddF32(addnormstatArgs, PS_LIST_TAIL, "-norm", 0, + "define normal value (required)", NAN); + psMetadataAddS8(addnormstatArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -tonormstat + psMetadata *tonormstatArgs = psMetadataAlloc(); + psMetadataAddU64(tonormstatArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(tonormstatArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addnormalizedimfile + psMetadata *addnormalizedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(addnormalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", NULL); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URI (required)", NULL); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addnormalizedimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -tonormalizedexp + psMetadata *tonormalizedexpArgs = psMetadataAlloc(); + psMetadataAddU64(tonormalizedexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(tonormalizedexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addnormalizedexp + psMetadata *addnormalizedexpArgs = psMetadataAlloc(); + psMetadataAddStr(addnormalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(addnormalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(addnormalizedexpArgs, PS_LIST_TAIL, "-recip", 0, + "search for recipe", NULL); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addnormalizedexpArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addnormalizedexpArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addnormalizedexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -normalizedexp + psMetadata *normalizedexpArgs = psMetadataAlloc(); + psMetadataAddStr(normalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(normalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(normalizedexpArgs, PS_LIST_TAIL, "-recip", 0, + "search for recipe", NULL); + psMetadataAddF64(normalizedexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(normalizedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(normalizedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(normalizedexpArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(normalizedexpArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddU64(normalizedexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(normalizedexpArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(normalizedexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -toresidimfile + psMetadata *toresidimfileArgs = psMetadataAlloc(); + psMetadataAddU64(toresidimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(toresidimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -normalizedimfile + psMetadata *normalizedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(normalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID (required)", NULL); + psMetadataAddS32(normalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, + "search for iteration number", 0); + psMetadataAddStr(normalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search for class ID", NULL); + psMetadataAddStr(normalizedimfileArgs, PS_LIST_TAIL, "-recip", 0, + "search for recipe", NULL); + psMetadataAddU64(normalizedimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(normalizedimfileArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(normalizedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -toresidexp + psMetadata *toresidexpArgs = psMetadataAlloc(); + psMetadataAddU64(toresidexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(toresidexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addresidimfile + psMetadata *addresidimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(addresidimfileArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", NULL); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define resid file URI (required)", NULL); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addresidimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -residimfile + psMetadata *residimfileArgs = psMetadataAlloc(); + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID (required)", NULL); + psMetadataAddS32(residimfileArgs, PS_LIST_TAIL, "-iteration", 0, + "search for iteration number", 0); + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search for class ID", NULL); + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-recip", 0, + "search for recipe", NULL); + psMetadataAddU64(residimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(residimfileArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(residimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addresidexp + psMetadata *addresidexpArgs = psMetadataAlloc(); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(addresidexpArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe (required)", NULL); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddS8(addresidexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + psMetadataAddBool(addresidexpArgs, PS_LIST_TAIL, "-reject", 0, + "exposure is not to be stacked in the next iteration", false); + + // -residexp + psMetadata *residexpArgs = psMetadataAlloc(); + psMetadataAddStr(residexpArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID (required)", NULL); + psMetadataAddS32(residexpArgs, PS_LIST_TAIL, "-iteration", 0, + "search for iteration number", 0); + psMetadataAddStr(residexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search for exp ID", NULL); + psMetadataAddStr(residexpArgs, PS_LIST_TAIL, "-recip", 0, + "search for recipe", NULL); + psMetadataAddU64(residexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(residexpArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(residexpArgs, PS_LIST_TAIL, "-reject", 0, + "search for acceptable residuals", false); + psMetadataAddBool(residexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -residdetrun + psMetadata *residdetrunArgs = psMetadataAlloc(); + psMetadataAddBool(residdetrunArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + psMetadataAddU64(residdetrunArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + + // -updateresidexp + psMetadata *updateresidexpArgs = psMetadataAlloc(); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID", NULL); + psMetadataAddS32(updateresidexpArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exp ID", NULL); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-recip", 0, + "define recipe", NULL); + psMetadataAddF64(updateresidexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(updateresidexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(updateresidexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + psMetadataAddBool(updateresidexpArgs, PS_LIST_TAIL, "-reject", 0, + "exposure is not to be stacked in the next iteration", false); + + // -adddetrunsummary + psMetadata *adddetrunsummaryArgs = psMetadataAlloc(); + psMetadataAddStr(adddetrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddS32(adddetrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, + "define iteration number", 0); + psMetadataAddF64(adddetrunsummaryArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(adddetrunsummaryArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(adddetrunsummaryArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddS8(adddetrunsummaryArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + psMetadataAddBool(adddetrunsummaryArgs, PS_LIST_TAIL, "-accept", 0, + "declare that this detrun iteration is accepted as a master", false); + + // -detrunsummary + psMetadata *detrunsummaryArgs = psMetadataAlloc(); + psMetadataAddStr(detrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend ID", NULL); + psMetadataAddS32(detrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, + "search for iteration number", 0); + psMetadataAddBool(detrunsummaryArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(detrunsummaryArgs, PS_LIST_TAIL, "-reject", 0, + "search for acceptable residuals", false); + psMetadataAddBool(detrunsummaryArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -updatedetrun + psMetadata *updatedetrunArgs = psMetadataAlloc(); + psMetadataAddStr(updatedetrunArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend master for detrend ID (required)", NULL); + psMetadataAddBool(updatedetrunArgs, PS_LIST_TAIL, "-again", 0, + "start a new iteration of this detrend run", false); + psMetadataAddStr(updatedetrunArgs, PS_LIST_TAIL, "-state", 0, + "set the state of this detrend run", false); + + // -rerun + psMetadata *rerunArgs = psMetadataAlloc(); + psMetadataAddStr(rerunArgs, PS_LIST_TAIL, "-det_id", 0, + "search for detrend master for detrend ID (required)", NULL); + psMetadataAddStr(rerunArgs, PS_LIST_TAIL, "-exp_tag", PS_META_DUPLICATE_OK, + "include this exposure (multiple OK, required)", NULL); + + // -register_detrend + psMetadata *register_detrendArgs = psMetadataAlloc(); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-det_type", 0, + "define the type of detrend run (required)", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-mode", 0, + "define the mode of this detrend run", "master"); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-workdir", 0, + "define workdir (required)", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-inst", 0, + "define camera", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-filter", 0, + "define filter ", NULL); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-airmass_min", 0, + "define min airmass", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-airmass_max", 0, + "define max airmass", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-exp_time_min", 0, + "define min exposure time", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-exp_time_max", 0, + "define max exposure time", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, + "define min ccd tempature", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, + "define max ccd tempature", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-posang_min", 0, + "define min rotator position angle", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-posang_max", 0, + "define max rotator position angle", NAN); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-registered", 0, + "time detrend run was registered", now); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-time_begin", 0, + "detrend applyes to exposures taken during this peroid", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-time_end", 0, + "detrend applyes to exposures taken during this peroid", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-use_begin", 0, + "start of detrend run applicable peroid", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-use_end", 0, + "end of detrend run applicable peroid", NULL); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddBool(register_detrendArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -register_detrend_imfile + psMetadata *register_detrend_imfileArgs = psMetadataAlloc(); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-det_id", 0, + "define detrend ID (required)", NULL); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search for class ID (required)", NULL); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-uri", 0, + "define resid file URI (required)", NULL); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + + psFree(now); + + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + psFree(argset); \ + psFree(argSets); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option); \ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + // find which mode we're running under + PXTOOL_MODE("-pending", DETTOOL_MODE_PENDING, pendingArgs); + PXTOOL_MODE("-definebytag", DETTOOL_MODE_DEFINEBYTAG, definebytagArgs); + PXTOOL_MODE("-definebyquery", DETTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXTOOL_MODE("-definebydetrun", DETTOOL_MODE_DEFINEBYDETRUN, definebydetrunArgs); + PXTOOL_MODE("-raw", DETTOOL_MODE_RAW, rawArgs); + PXTOOL_MODE("-runs", DETTOOL_MODE_RUNS, runsArgs); + PXTOOL_MODE("-input", DETTOOL_MODE_INPUT, inputArgs); + PXTOOL_MODE("-toprocessedimfile", DETTOOL_MODE_TOPROCESSEDIMFILE, toprocessedimfileArgs); + PXTOOL_MODE("-addprocessedimfile", DETTOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileArgs); + PXTOOL_MODE("-processedimfile", DETTOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXTOOL_MODE("-toprocessedexp", DETTOOL_MODE_TOPROCESSEDEXP, toprocessedexpArgs); + PXTOOL_MODE("-addprocessedexp", DETTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXTOOL_MODE("-processedexp", DETTOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXTOOL_MODE("-tostacked", DETTOOL_MODE_TOSTACKED, tostackedArgs); + PXTOOL_MODE("-addstacked", DETTOOL_MODE_ADDSTACKED, addstackedArgs); + PXTOOL_MODE("-stacked", DETTOOL_MODE_STACKED, stackedArgs); + PXTOOL_MODE("-tonormalize", DETTOOL_MODE_TONORMALIZE, tonormalizeArgs); + PXTOOL_MODE("-addnormalizedstat", DETTOOL_MODE_ADDNORMALIZEDSTAT, addnormstatArgs); + PXTOOL_MODE("-tonormalizedstat", DETTOOL_MODE_TONORMALIZEDSTAT, tonormstatArgs); + PXTOOL_MODE("-addnormalizedimfile", DETTOOL_MODE_ADDNORMALIZEDIMFILE,addnormalizedimfileArgs); + PXTOOL_MODE("-normalizedimfile",DETTOOL_MODE_NORMALIZEDIMFILE, normalizedimfileArgs); + PXTOOL_MODE("-tonormalizedexp", DETTOOL_MODE_TONORMALIZEDEXP, tonormalizedexpArgs); + PXTOOL_MODE("-addnormalizedexp", DETTOOL_MODE_ADDNORMALIZEDEXP, addnormalizedexpArgs); + PXTOOL_MODE("-normalizedexp", DETTOOL_MODE_NORMALIZEDEXP, normalizedexpArgs); + PXTOOL_MODE("-toresidimfile", DETTOOL_MODE_TORESIDIMFILE, toresidimfileArgs); + PXTOOL_MODE("-addresidimfile", DETTOOL_MODE_ADDRESIDIMFILE, addresidimfileArgs); + PXTOOL_MODE("-residimfile", DETTOOL_MODE_RESIDIMFILE, residimfileArgs); + PXTOOL_MODE("-toresidexp", DETTOOL_MODE_TORESIDEXP, toresidexpArgs); + PXTOOL_MODE("-addresidexp", DETTOOL_MODE_ADDRESIDEXP, addresidexpArgs); + PXTOOL_MODE("-residexp", DETTOOL_MODE_RESIDEXP, residexpArgs); + PXTOOL_MODE("-residdetrun", DETTOOL_MODE_RESIDDETRUN, residdetrunArgs); + PXTOOL_MODE("-updateresidexp", DETTOOL_MODE_UPDATERESIDEXP,updateresidexpArgs); + PXTOOL_MODE("-adddetrunsummary", DETTOOL_MODE_ADDDETRUNSUMMARY,adddetrunsummaryArgs); + PXTOOL_MODE("-detrunsummary", DETTOOL_MODE_DETRUNSUMMARY,detrunsummaryArgs); + PXTOOL_MODE("-updatedetrun", DETTOOL_MODE_UPDATEDETRUN, updatedetrunArgs); + PXTOOL_MODE("-rerun", DETTOOL_MODE_RERUN, rerunArgs); + PXTOOL_MODE("-register_detrend", DETTOOL_MODE_REGISTER_DETREND, register_detrendArgs); + PXTOOL_MODE("-register_detrend_imfile", DETTOOL_MODE_REGISTER_DETREND_IMFILE, register_detrend_imfileArgs); + + bool argErr = false; + if (config->mode == DETTOOL_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else { + if (! psArgumentParse(config->args, &argc, argv)) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + if (argc != 1) { + fprintf (stderr, "extra arguments: "); + for (int i = 1; i < argc; i++) { + fprintf (stderr, "%s ", argv[i]); + } + fprintf (stderr, "\n"); + argErr = true; + } + } + + if (argErr) { + printf("\nPan-STARRS Detrend Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); + } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(det_id); + { + int n = 0; + bool status = false; + if ((n = psMetadataLookupS32(&status, config->args, "-iteration"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "iteration", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + goto FAIL; + } + } + } + addWhereStr(det_type); + addWhereStr(exp_tag); + addWhereStr(class_id); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + goto FAIL; + } + } + } + addWhereStr(telescope); + addWhereStr(exp_type); + { + int n = 0; + bool status = false; + if ((n = psMetadataLookupS32(&status, config->args, "-imfiles"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "imfiles", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item imfiles"); + goto FAIL; + } + } + } + addWhereStr(filter); + addWhereStr(recipe); + { + int n = 0; + bool status = false; + if ((n = psMetadataLookupS32(&status, config->args, "-p1_version"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "p1_version", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item p1_version"); + goto FAIL; + } + } + } + { + bool boolean = false; + bool status = false; + + // map reject -> !accept + if ((boolean = psMetadataLookupBool(&status, config->args, "-reject"))) { + if (!psMetadataAddBool(config->where, PS_LIST_TAIL, "accept", 0, "==", !boolean)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item reject"); + goto FAIL; + } + } + } + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/guidetool.c =================================================================== --- /branches/rel-1-0/ippTools/src/guidetool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/guidetool.c (revision 11645) @@ -0,0 +1,201 @@ +/* + * p1tool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "p1tool.h" + +static bool pendingMode(pxConfig *config); +static bool defineMode(pxConfig *config); +static p1PendingExpRow *rawScienceTop1PendingExp(pxConfig *config, rawScienceExpRow *exp); + +int main(int argc, char **argv) +{ + pxConfig *config = p1toolConfig(NULL, argc, argv); + + switch (config->mode) { + case PX_MODE_PENDING: + if (!pendingMode(config)) { + goto FAIL; + } + break; + case PX_MODE_DEFINE: + if (!defineMode(config)) { + goto FAIL; + } + break; + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + exit(EXIT_SUCCESS); + +FAIL: + psFree(config); + exit(EXIT_FAILURE); +} + +static bool pendingMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // return all p1PendingExp rows unless there CLI search options + psArray *p1PendingExp = p1PendingExpSelectRowObjects( + config->dbh, + config->where ? config->where : NULL, + 0 + ); + if (!p1PendingExp) { + psError(PS_ERR_UNKNOWN, false, "no p1PendingExp found"); + return false; + } + + psMetadata *output = psMetadataAlloc(); + + for (long i = 0; i < psArrayLength(p1PendingExp); i++) { + psMetadata *md = p1PendingExpMetadataFromObject(p1PendingExp->data[i]); + psMetadataAddMetadata(output, PS_LIST_TAIL, "p1PendingExp", + PS_META_DUPLICATE_OK, NULL, md); + psFree(md); + } + + psString str = psMetadataConfigFormat(output); + psFree(output); + + fprintf(stdout, "%s\n", str); + psFree(str); + + return true; +} + +static bool defineMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // return all rawScienceExp rows unless there CLI search options + psArray *rawScienceExps = rawScienceExpSelectRowObjects( + config->dbh, + config->where ? config->where : NULL, + 0 + ); + + if (!rawScienceExps) { + psError(PS_ERR_UNKNOWN, false, "no rawScienceExp rows found"); + return true; + } + + // insert the rawScienceExps into p1PendingExp + // XXX for the being we're going to skip checking for duplicate entries + // XXX this will have to implimented unless duplicate key insertion + // XXX support is implimented in psDB + for (long i = 0; i < psArrayLength(rawScienceExps); i++) { + p1PendingExpRow * new = rawScienceTop1PendingExp( + config, rawScienceExps->data[i]); + if (!p1PendingExpInsertObject(config->dbh, new)) { + psError(PS_ERR_UNKNOWN, false, "dbh access failed"); + return false; + } + psFree(new); + } + + return true; +} + +static p1PendingExpRow *rawScienceTop1PendingExp(pxConfig *config, rawScienceExpRow *exp) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(exp, NULL); + + bool status = false; + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + if (!filter) { + psError(PS_ERR_UNKNOWN, true, "-filter is required"); + return false; + } + psF32 airmass = psMetadataLookupF32(&status, config->args, "-airmass"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass"); + return false; + } + if (!airmass) { + psError(PS_ERR_UNKNOWN, true, "-airmass is required"); + return false; + } + psF64 ra = psMetadataLookupF64(&status, config->args, "-ra"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ra"); + return false; + } + if (!ra) { + psError(PS_ERR_UNKNOWN, true, "-airmass is required"); + return false; + } + psF64 dec = psMetadataLookupF64(&status, config->args, "-dec"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -dec"); + return false; + } + if (!dec) { + psError(PS_ERR_UNKNOWN, true, "-dec is required"); + return false; + } + psF32 exp_time = psMetadataLookupF32(&status, config->args, "-exp_time"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time"); + return false; + } + if (!exp_time) { + psError(PS_ERR_UNKNOWN, true, "-exp_time is required"); + return false; + } + psF64 background = psMetadataLookupF64(&status, config->args, "-background"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -background"); + return false; + } + if (!exp_time) { + psError(PS_ERR_UNKNOWN, true, "-background is required"); + return false; + } + + return p1PendingExpRowAlloc( + exp->exp_tag, + exp->camera, + exp->telescope, + exp->exp_type, + exp->imfiles, + filter, + airmass, + ra, + dec, + exp_time, + background, + "my recipe", + 0xff // XXX calc version number + ); +} Index: /branches/rel-1-0/ippTools/src/guidetool.h =================================================================== --- /branches/rel-1-0/ippTools/src/guidetool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/guidetool.h (revision 11645) @@ -0,0 +1,27 @@ +/* + * p1tool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef P1TOOL_H +#define P1TOOL_H 1 + +#include "pxtools.h" + +pxConfig *p1toolConfig(pxConfig *config, int argc, char **argv); + +#endif // P1TOOL_H Index: /branches/rel-1-0/ippTools/src/guidetoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/guidetoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/guidetoolConfig.c (revision 11645) @@ -0,0 +1,205 @@ +/* + * p1toolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" + +// this function can not fail -- exits on error + +pxConfig *p1toolConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + exit(EXIT_FAILURE); + } + + // CLI options + // modes + psMetadata *args = psMetadataAlloc(); + psMetadataAddStr(args , PS_LIST_TAIL, "-pending", 0, + "examine pending image table, write ppImage output", NULL); + psMetadataAddStr(args , PS_LIST_TAIL, "-define", 0, + "update pending image table", NULL); + + // -pending search + psMetadata *pendingArgs = psMetadataAlloc(); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure ID", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-inst", 0, + "define camera of interest", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope of interest", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddS32(pendingArgs, PS_LIST_TAIL, "-imfiles", 0, + "define number of imfiles", 0); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-filter", 0, + "define filter of interest", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-recipe", 0, + "define recipe of interest", NULL); + psMetadataAddS32(pendingArgs, PS_LIST_TAIL, "-p1_version", 0, + "define p1_version of interest", 0); + + // -define inputs + psMetadata *defineArgs = psMetadataAlloc(); + psMetadataAddStr(defineArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define class", NULL); + psMetadataAddStr(defineArgs, PS_LIST_TAIL, "-inst", 0, + "define camera of interest", NULL); + psMetadataAddStr(defineArgs, PS_LIST_TAIL, "-telescope", 0, + "define camera of interest", NULL); + psMetadataAddStr(defineArgs, PS_LIST_TAIL, "-exp_type", 0, + "define class", NULL); + psMetadataAddS32(defineArgs, PS_LIST_TAIL, "-imfiles", 0, + "define number of imfiles", 0); + psMetadataAddStr(defineArgs, PS_LIST_TAIL, "-filter", 0, + "define filter of interest", NULL); + // these two are inputs NOT search parameters + psMetadataAddStr(defineArgs, PS_LIST_TAIL, "-recipe", 0, + "define recipe of interest", NULL); + psMetadataAddS32(defineArgs, PS_LIST_TAIL, "-p1_version", 0, + "define p1_version of interest", 0); + + // find which mode we're running under + int N = 0; + config->mode = PX_MODE_NONE; + if ((N = psArgumentGet (argc, argv, "-pending"))) { + psArgumentRemove (N, &argc, argv); + if (config->mode) { + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); + } + config->mode = PX_MODE_PENDING; + } + if ((N = psArgumentGet (argc, argv, "-define"))) { + psArgumentRemove (N, &argc, argv); + if (config->mode) { + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); + } + config->mode = PX_MODE_DEFINE; + } + + // parse CLI for these options + switch (config->mode) { + case PX_MODE_PENDING: + config->args = psMemIncrRefCounter(pendingArgs); + break; + case PX_MODE_DEFINE: + config->args = psMemIncrRefCounter(defineArgs); + break; + default: + config->args = psMemIncrRefCounter(args); + } + + bool argErr = false; + if (config->mode == PX_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS Phase 1 Search Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" : -pending | -define\n\n"); + fprintf (stdout, "-pending "); + psArgumentHelp(pendingArgs); + fprintf (stdout, "-define "); + psArgumentHelp(defineArgs); + psFree(args); + psFree(pendingArgs); + psFree(defineArgs); + exit(EXIT_FAILURE); + } + + psFree(args); + psFree(pendingArgs); + psFree(defineArgs); + + // setup search criterion +#define addWhereStr(name) \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + exit(EXIT_FAILURE); \ + } \ + } + + // generate SQL where clause + config->where = psMetadataAlloc(); + +{ + psString str = NULL; + int n = 0; + bool status = false; + + addWhereStr(exp_tag); + // convert '-inst' to 'camera' + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + exit(EXIT_FAILURE); + } + } + addWhereStr(telescope); + addWhereStr(exp_type); + if ((n = psMetadataLookupS32(&status, config->args, "-imfiles"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "imfiles", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item imfiles"); + exit(EXIT_FAILURE); + } + } + addWhereStr(filter); + addWhereStr(stats); + addWhereStr(recipe); + if ((n = psMetadataLookupS32(&status, config->args, "-p1_version"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "p1_version", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item p1_version"); + exit(EXIT_FAILURE); + } + } +} + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + exit(EXIT_FAILURE); + } + + return config; +} Index: /branches/rel-1-0/ippTools/src/pxadmin.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxadmin.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxadmin.c (revision 11645) @@ -0,0 +1,69 @@ +/* + * pxadmin.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxadmin.h" + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pxAdminConfig(NULL, argc, argv); + + switch (config->mode) { + case PXADMIN_MODE_RECREATE: + if (!pxDeleteTables(config)) { + goto FAIL; + } + // fall through + case PXADMIN_MODE_CREATE: + if (!pxCreateTables(config)) { + goto FAIL; + } + break; + case PXADMIN_MODE_DELETE: + if (!pxDeleteTables(config)) { + goto FAIL; + } + break; + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/pxadmin.h =================================================================== --- /branches/rel-1-0/ippTools/src/pxadmin.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxadmin.h (revision 11645) @@ -0,0 +1,34 @@ +/* + * pxadmin.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXADMIN_H +#define PXADMIN_H 1 + +#include "pxtools.h" + +typedef enum { + PXADMIN_MODE_NONE = 0x0, + PXADMIN_MODE_CREATE, + PXADMIN_MODE_DELETE, + PXADMIN_MODE_RECREATE +} pxAdminMode; + +pxConfig *pxAdminConfig (pxConfig *config, int argc, char **argv); + +#endif // PXADMIN_H Index: /branches/rel-1-0/ippTools/src/pxadminConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxadminConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxadminConfig.c (revision 11645) @@ -0,0 +1,126 @@ +/* + * pxadminConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxadmin.h" + +pxConfig *pxAdminConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + // Parse other command-line arguments + psMetadata *arguments = psMetadataAlloc(); // The arguments, with default values + + int N; + config->mode = PXADMIN_MODE_NONE; + if ((N = psArgumentGet(argc, argv, "-create"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort(argv[0], "only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_CREATE; + } + if ((N = psArgumentGet(argc, argv, "-delete"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort(argv[0], "only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_DELETE; + } + if ((N = psArgumentGet(argc, argv, "-recreate"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort(argv[0], "only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_RECREATE; + } + + // paul's argument parsing convention requires: -key value + psMetadataAddStr(arguments, PS_LIST_TAIL, "-create", 0, + "create the P2 tables", ""); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-delete", 0, + "delete the P2 tables", ""); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-recreate", 0, + "delete and recreate the P2 tables", ""); + + if (config->mode == PXADMIN_MODE_NONE) { + fprintf (stderr, "admin mode not specified\n"); + psArgumentHelp(arguments); + psFree(arguments); + goto FAIL; + } + + if ((N = psArgumentGet (argc, argv, "-help"))) { + psArgumentHelp(arguments); + psFree(arguments); + goto FAIL; + } + + if (! psArgumentParse(arguments, &argc, argv) || argc != 1) { + printf("\nPan-STARRS Phase 2 Admin Tool\n\n"); + printf("Usage: %s [mode]\n", argv[0]); + printf(" [mode] : -create | -delete\n\n"); + psArgumentHelp(arguments); + psFree(arguments); + goto FAIL; + } + + psFree(arguments); + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} + +/* + * USAGE: + * p2admin -create : create the P2 tables + * p2admin -delete : delete the P2 tables (asks for confirmation) + */ Index: /branches/rel-1-0/ippTools/src/pxconfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxconfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxconfig.c (revision 11645) @@ -0,0 +1,50 @@ +/* + * pxconfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "pxtools.h" + +static void pxConfigFree(pxConfig *ptr); + +pxConfig *pxConfigAlloc(void) +{ + pxConfig *config; + + config = psAlloc(sizeof(pxConfig)); + psMemSetDeallocator(config, (psFreeFunc)pxConfigFree); + + config->mode = 0; + config->dbh = NULL; + config->modules = NULL; + config->where = NULL; + config->args = NULL; + + return config; +} + +static void pxConfigFree(pxConfig *config) +{ + psDBCleanup(config->dbh); + psFree(config->modules); + psFree(config->where); + psFree(config->args); +} Index: /branches/rel-1-0/ippTools/src/pxerrors.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxerrors.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxerrors.c (revision 11645) @@ -0,0 +1,42 @@ +/* + * pxerrors.c + * + * Copyright (C) 2006 Eugene Magnier + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "pxtools.h" + +psExit pxerrorGetExitStatus () { + + psErrorCode err = psErrorCodeLast (); + switch (err) { + case PXTOOLS_ERR_SYS: + return PS_EXIT_SYS_ERROR; + case PXTOOLS_ERR_CONFIG: + return PS_EXIT_CONFIG_ERROR; + case PXTOOLS_ERR_PROG: + return PS_EXIT_PROG_ERROR; + case PXTOOLS_ERR_DATA: + return PS_EXIT_DATA_ERROR; + default: + return PS_EXIT_UNKNOWN_ERROR; + } + return PS_EXIT_UNKNOWN_ERROR; +} Index: /branches/rel-1-0/ippTools/src/pxfault.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxfault.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxfault.c (revision 11645) @@ -0,0 +1,65 @@ +/* + * pxfault.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" + +bool pxSetFaultCode(psDB *dbh, const char *tableName, psMetadata *where, psS8 code) +{ + PS_ASSERT_PTR_NON_NULL(dbh, false); + PS_ASSERT_PTR_NON_NULL(tableName, false); + PS_ASSERT_PTR_NON_NULL(where, false); + +#if 0 + // map code string to numeric fault code + psU32 code = mapCodeStrToInt(codeStr); + if (code == (psU32)-1) { + psError(PS_ERR_UNKNOWN, false, "error resolving error code"); + return false; + } +#endif + + // update the database + psMetadata *values = psMetadataAlloc(); + if (!psMetadataAddS8(values, PS_LIST_HEAD, "fault", 0, NULL, code)) { + psError(PS_ERR_UNKNOWN, false, "failed to add metadata item fault"); + psFree(values); + return false; + } + + long rowsAffected = psDBUpdateRows(dbh, tableName, where, values); + psFree(values); + if (rowsAffected < 0) { + // database error + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (rowsAffected < 1) { + // we didn't do anything + psError(PS_ERR_UNKNOWN, false, "zero rows were affected - either the search criteria didn't match any rows or the field already had the value being set."); + return false; + } + + return true; +} Index: /branches/rel-1-0/ippTools/src/pxinject.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxinject.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxinject.c (revision 11645) @@ -0,0 +1,223 @@ +/* + * pxinject.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "pxtools.h" +#include "pxtag.h" +#include "pxinject.h" + +static bool newExpMode(pxConfig *config); +static bool newImfileMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pxinjectConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(PXINJECT_MODE_NEWEXP, newExpMode); + MODECASE(PXINJECT_MODE_NEWIMFILE, newImfileMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint (stderr, "failure\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool newExpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString exp_id = psMetadataLookupStr(&status, config->args, "-exp_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_id"); + return false; + } + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, "-exp_id is required"); + return false; + } + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return false; + } + if (!camera) { + psError(PS_ERR_UNKNOWN, true, "-inst is required"); + return false; + } + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return false; + } + if (!telescope) { + psError(PS_ERR_UNKNOWN, true, "-telescope is required"); + return false; + } + psS32 imfiles = psMetadataLookupS32(&status, config->args, "-imfiles"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -imfiles"); + return false; + } + if (!imfiles) { + psError(PS_ERR_UNKNOWN, true, "-imfiles is required"); + return false; + } + + psString workdir = psMetadataLookupStr(&status, config->args, "-workdir"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -workdir"); + return false; + } + if (!workdir) { + psError(PS_ERR_UNKNOWN, true, "-workdir is required"); + return false; + } + + psString exp_tag = pxGenExpTag(config, exp_id); + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!newExpInsert(config->dbh, + exp_tag, + exp_id, + camera, + telescope, + imfiles, + workdir + ) + ) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(exp_tag); + return false; + } + + psMetadata *md = psMetadataAlloc(); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_tag", 0, NULL, exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(md); + psFree(exp_tag); + psFree(md); + } + psFree(exp_tag); + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(md); + return false; + } + } + + // negate simple so the default is true + if (!ippdbPrintMetadata(stdout, md, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(md); + return false; + } + + psFree(md); + + return true; +} + +static bool newImfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + psString class = psMetadataLookupStr(&status, config->args, "-class"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class"); + return false; + } + if (!class) { + psError(PS_ERR_UNKNOWN, true, "-class is required"); + return false; + } + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + // insert with error flag state set to 0 (no errors) + if (!newImfileInsert(config->dbh, exp_tag, class, class_id, uri)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} Index: /branches/rel-1-0/ippTools/src/pxinject.h =================================================================== --- /branches/rel-1-0/ippTools/src/pxinject.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxinject.h (revision 11645) @@ -0,0 +1,33 @@ +/* + * pxinject.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXINJECT_H +#define PXINJECT_H 1 + +#include "pxtools.h" + +typedef enum { + PXINJECT_MODE_NONE = 0x0, + PXINJECT_MODE_NEWEXP, + PXINJECT_MODE_NEWIMFILE +} pxinjectMode; + +pxConfig *pxinjectConfig(pxConfig *config, int argc, char **argv); + +#endif // PXINJECT_H Index: /branches/rel-1-0/ippTools/src/pxinjectConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxinjectConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxinjectConfig.c (revision 11645) @@ -0,0 +1,215 @@ +/* + * pxinjectConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxinject.h" + +// this function can not fail -- exits on error +pxConfig *pxinjectConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + // -newExp + psMetadata *newExpArgs = psMetadataAlloc(); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-exp_id", 0, + "define the exp_id (required)", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-inst", 0, + "define the camera name (required)", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-telescope", 0, + "define the telescope name (required)", NULL); + psMetadataAddS32(newExpArgs, PS_LIST_TAIL, "-imfiles", 0, + "define the number of imfiles in this exp (required)", 0); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-workdir", 0, + "define workdir (required)", 0); + psMetadataAddBool(newExpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -newImfile + psMetadata *newImfileArgs = psMetadataAlloc(); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define the exp_tag (required)", NULL); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-class", 0, + "define the class (required)", NULL); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define the class ID (required)", NULL); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-uri", 0, + "define the URI (required)", NULL); + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ +} + + // find which mode we're running under + PXTOOL_MODE("-newExp", PXINJECT_MODE_NEWEXP, newExpArgs); + PXTOOL_MODE("-newImfile", PXINJECT_MODE_NEWIMFILE, newImfileArgs); + + bool argErr = false; + if (config->mode == PXINJECT_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS Detrend Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" : [-newExp|-newImfile]\n\n"); + fprintf (stdout, "-newExp "); + psArgumentHelp(newExpArgs); + psFree(newExpArgs); + fprintf (stdout, "-newImfile "); + psArgumentHelp(newImfileArgs); + psFree(newImfileArgs); + + goto FAIL; + } + + psFree(newExpArgs); + psFree(newImfileArgs); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(det_id); + { + int n = 0; + bool status = false; + if ((n = psMetadataLookupS32(&status, config->args, "-iteration"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "iteration", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + goto FAIL; + } + } + } + addWhereStr(det_type); + addWhereStr(exp_tag); + addWhereStr(class_id); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + goto FAIL; + } + } + } + addWhereStr(telescope); + { + int n = 0; + bool status = false; + if ((n = psMetadataLookupS32(&status, config->args, "-imfiles"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "imfiles", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item imfiles"); + goto FAIL; + } + } + } + addWhereStr(filter); + addWhereStr(stats); + addWhereStr(recipe); + { + int n = 0; + bool status = false; + if ((n = psMetadataLookupS32(&status, config->args, "-p1_version"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "p1_version", 0, "==", n)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item p1_version"); + goto FAIL; + } + } + } + { + bool boolean = false; + bool status = false; + + if ((boolean = psMetadataLookupBool(&status, config->args, "-accept"))) { + if (!psMetadataAddBool(config->where, PS_LIST_TAIL, "accept", 0, "==", boolean)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item accept"); + goto FAIL; + } + } + } + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/pxtables.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxtables.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtables.c (revision 11645) @@ -0,0 +1,161 @@ +/* + * pxtables.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include "pxtools.h" + +#define CREATE_TABLE(createFunc) \ + if (!createFunc(config->dbh)) { \ + psError(PS_ERR_UNKNOWN, false, "dbh access failed"); \ + status = false; \ + } + +#define DROP_TABLE(dropFunc) \ + if (!dropFunc(config->dbh)) { \ + psError(PS_ERR_UNKNOWN, false, "dbh access failed"); \ + status = false; \ + } + +bool pxCreateTables(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = true; + CREATE_TABLE(expTagCounterCreateTable); + // the counter must be initialized to 0 + if (!p_psDBRunQuery(config->dbh, "INSERT INTO expTagCounter VALUES (0)")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + status = false; + } + CREATE_TABLE(summitExpCreateTable); + CREATE_TABLE(summitImfileCreateTable); + CREATE_TABLE(pzPendingExpCreateTable); + CREATE_TABLE(pzPendingImfileCreateTable); + CREATE_TABLE(pzDoneExpCreateTable); + CREATE_TABLE(pzDoneImfileCreateTable); + CREATE_TABLE(newExpCreateTable); + CREATE_TABLE(newImfileCreateTable); + CREATE_TABLE(rawExpCreateTable); + CREATE_TABLE(rawImfileCreateTable); + CREATE_TABLE(p1PendingExpCreateTable); + CREATE_TABLE(p2PendingExpCreateTable); + CREATE_TABLE(p2PendingImfileCreateTable); + CREATE_TABLE(p2ProcessedExpCreateTable); + CREATE_TABLE(p2ProcessedImfileCreateTable); + CREATE_TABLE(p2MaskCreateTable); + CREATE_TABLE(p3PendingExpCreateTable); + CREATE_TABLE(p3ProcessedExpCreateTable); + CREATE_TABLE(p3MaskCreateTable); + CREATE_TABLE(detRunCreateTable); + CREATE_TABLE(detInputExpCreateTable); + CREATE_TABLE(detProcessedImfileCreateTable); + CREATE_TABLE(detProcessedExpCreateTable); + CREATE_TABLE(detStackedImfileCreateTable); + CREATE_TABLE(detNormalizedStatImfileCreateTable); + CREATE_TABLE(detNormalizedImfileCreateTable); + CREATE_TABLE(detNormalizedExpCreateTable); + CREATE_TABLE(detResidImfileCreateTable); + CREATE_TABLE(detResidExpCreateTable); + CREATE_TABLE(detRunSummaryCreateTable); + CREATE_TABLE(p4RunCreateTable); + CREATE_TABLE(p4InputImfileCreateTable); + CREATE_TABLE(p4WarpedImfileCreateTable); + CREATE_TABLE(p4StackedImfileCreateTable); + CREATE_TABLE(p4DiffImfileCreateTable); + CREATE_TABLE(p4MagicMaskImfileCreateTable); + + return true; +} + +bool pxDeleteTables(pxConfig *config) { + bool status = true; + char line[128], answer[128]; + + PS_ASSERT_PTR_NON_NULL(config, false); + + fprintf (stdout, "*** delete the P2 tables? ***\n"); + fprintf (stdout, "*** to delete the tables, answer YES, and give password ***\n"); + fprintf (stdout, "*** WARNING: this action is permanent ***\n\n"); + + fprintf (stdout, "delete the P2 tables (YES/[n]): "); + fgets (line, 128, stdin); + sscanf (line, "%s", answer); + if (strcmp (answer, "YES")) { + psError(PS_ERR_UNKNOWN, true, "tables NOT deleleted"); + return false; + } + + fprintf (stdout, "enter dbh connection password: "); + fgets (line, 128, stdin); + sscanf (line, "%s", answer); + + { + bool status; + psString dbPassword = psMetadataLookupStr(&status, config->modules->site, "DBPASSWORD"); + if (strcmp (answer, dbPassword)) { + psError(PS_ERR_UNKNOWN, true, "tables NOT deleleted"); + return false; + } + } + + DROP_TABLE(expTagCounterDropTable); + DROP_TABLE(summitExpDropTable); + DROP_TABLE(summitImfileDropTable); + DROP_TABLE(pzPendingExpDropTable); + DROP_TABLE(pzPendingImfileDropTable); + DROP_TABLE(pzDoneExpDropTable); + DROP_TABLE(pzDoneImfileDropTable); + DROP_TABLE(newExpDropTable); + DROP_TABLE(newImfileDropTable); + DROP_TABLE(rawExpDropTable); + DROP_TABLE(rawImfileDropTable); + DROP_TABLE(p1PendingExpDropTable); + DROP_TABLE(p2PendingExpDropTable); + DROP_TABLE(p2PendingImfileDropTable); + DROP_TABLE(p2ProcessedExpDropTable); + DROP_TABLE(p2ProcessedImfileDropTable); + DROP_TABLE(p2MaskDropTable); + DROP_TABLE(p3PendingExpDropTable); + DROP_TABLE(p3ProcessedExpDropTable); + DROP_TABLE(p3MaskDropTable); + DROP_TABLE(detRunDropTable); + DROP_TABLE(detInputExpDropTable); + DROP_TABLE(detProcessedImfileDropTable); + DROP_TABLE(detProcessedExpDropTable); + DROP_TABLE(detStackedImfileDropTable); + DROP_TABLE(detNormalizedStatImfileDropTable); + DROP_TABLE(detNormalizedImfileDropTable); + DROP_TABLE(detNormalizedExpDropTable); + DROP_TABLE(detResidImfileDropTable); + DROP_TABLE(detResidExpDropTable); + DROP_TABLE(detRunSummaryDropTable); + DROP_TABLE(p4RunDropTable); + DROP_TABLE(p4InputImfileDropTable); + DROP_TABLE(p4WarpedImfileDropTable); + DROP_TABLE(p4StackedImfileDropTable); + DROP_TABLE(p4DiffImfileDropTable); + DROP_TABLE(p4MagicMaskImfileDropTable); + + return status; +} Index: /branches/rel-1-0/ippTools/src/pxtag.c =================================================================== --- /branches/rel-1-0/ippTools/src/pxtag.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtag.c (revision 11645) @@ -0,0 +1,102 @@ +/* + * pxtag.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxtag.h" + +psString pxGenExpTag(pxConfig *config, const char *exp_id) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // start a transaction so we don't increment the expTag counter unless we + // can successfully retreive it's value + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + + if (!p_psDBRunQuery(config->dbh, + "UPDATE expTagCounter SET counter = LAST_INSERT_ID(counter + 1)")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + + // psDBLastInsertID() can't be used here as it called mysql_insert_id() + // which doesn't work with this trick. See: + // http://dev.mysql.com/doc/refman/4.1/en/information-functions.html + if (!p_psDBRunQuery(config->dbh, "SELECT LAST_INSERT_ID() as counter")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + // sanity check that we only got one row + if (psArrayLength(output) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have gotten 1 row but %lu rows were returned", psArrayLength(output)); + psFree(output); + return NULL; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return NULL; + } + + psMetadata *row = output->data[0]; + bool status = false; + psU64 counter = psMetadataLookupU64(&status, row, "counter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for counter"); + psFree(output); + return NULL; + } + psString exp_tag = NULL; + psStringAppend(&exp_tag, "%s.%" PRIu64, exp_id, counter); + psFree(output); + + return exp_tag; +} Index: /branches/rel-1-0/ippTools/src/pxtag.h =================================================================== --- /branches/rel-1-0/ippTools/src/pxtag.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtag.h (revision 11645) @@ -0,0 +1,27 @@ +/* + * pxtag.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXTAG_H +#define PXTAG_H 1 + +#include "pxtools.h" + +psString pxGenExpTag(pxConfig *config, const char *exp_id); + +#endif // PXTAG_H Index: /branches/rel-1-0/ippTools/src/pxtools.h =================================================================== --- /branches/rel-1-0/ippTools/src/pxtools.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtools.h (revision 11645) @@ -0,0 +1,55 @@ +/* + * pxtools.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXTOOLS_H +#define PXTOOLS_H 1 + +# include +# include // for strcasecmp +# include // for unlink +# include +# include +# include +# include "pxtoolsErrorCodes.h" + +// load these values from the db in the init stage +# define P2_TYPE_OBJECT 1 +# define P2_STATE_READY 2 +# define RECIPE "PHASE2" +# define MAX_ROWS 10e9 + +typedef struct { + int mode; + pmConfig *modules; + psDB *dbh; + psMetadata *args; + psMetadata *where; + int argc; + char **argv; +} pxConfig; + +pxConfig *pxConfigAlloc(void); +bool pxCreateTables (pxConfig *config); +bool pxDeleteTables (pxConfig *config); + +bool pxSetFaultCode(psDB *dbh, const char *tableName, psMetadata *where, psS8 code); + +psExit pxerrorGetExitStatus (); + +#endif // PXTOOLS_H Index: /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.c.in =================================================================== --- /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.c.in (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.c.in (revision 11645) @@ -0,0 +1,25 @@ +/* + * The line + { PXTOOLS_ERR_$X{ErrorCode}, "$X{ErrorDescription}"}, + * (without the Xs) + * will be replaced by values from errorCodes.dat + */ +#include "pslib.h" +#include "pxtoolsErrorCodes.h" + +void pxtoolsErrorRegister(void) +{ + static psErrorDescription errors[] = { + { PXTOOLS_ERR_BASE, "First value we use; lower values belong to psLib" }, + { PXTOOLS_ERR_${ErrorCode}, "${ErrorDescription}"}, + }; + static int nerror = PXTOOLS_ERR_NERROR - PXTOOLS_ERR_BASE; // number of values in enum + + for (int i = 0; i < nerror; i++) { + psErrorDescription *tmp = psAlloc(sizeof(psErrorDescription)); + *tmp = errors[i]; + psErrorRegister(tmp, 1); + psFree(tmp); /* it's on the internal list */ + } + nerror = 0; // don't register more than once +} Index: /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.dat =================================================================== --- /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.dat (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.dat (revision 11645) @@ -0,0 +1,10 @@ +# +# This file is used to generate pxtoolsErrorCodes.h +# +BASE = 1100 First value we use; lower values belong to psLib +UNKNOWN Unknown PM error code +ARGUMENTS Incorrect arguments +SYS System error +CONFIG Problem in configure files +PROG Programming error +DATA invalid data Index: /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.h.in =================================================================== --- /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.h.in (revision 11645) +++ /branches/rel-1-0/ippTools/src/pxtoolsErrorCodes.h.in (revision 11645) @@ -0,0 +1,18 @@ +#if !defined(PXTOOLS_ERROR_CODES_H) +#define PXTOOLS_ERROR_CODES_H +/* + * The line + * PXTOOLS_ERR_$X{ErrorCode}, + * (without the X) + * + * will be replaced by values from errorCodes.dat + */ +typedef enum { + PXTOOLS_ERR_BASE = 512, + PXTOOLS_ERR_${ErrorCode}, + PXTOOLS_ERR_NERROR +} pxtoolsErrorCode; + +void pxtoolsErrorRegister(void); + +#endif Index: /branches/rel-1-0/ippTools/src/pzgetexp.c =================================================================== --- /branches/rel-1-0/ippTools/src/pzgetexp.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pzgetexp.c (revision 11645) @@ -0,0 +1,367 @@ +/* + * pzgetexp.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pzgetexp.h" + +#define PRODUCT_LS_CMD "dsproductls" + +static bool go (pxConfig *config); +static psArray *parseFileSets(pxConfig *config, const char *str); + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pzgetexpConfig(NULL, argc, argv); + + if (!go(config)) { + goto FAIL; + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool go(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + // find last fileset/exp_id (if we have one) + bool haveLastFileSet = false; + psString lastFileSet = NULL; + { + char *query = "SELECT * from summitExp ORDER BY dateobs DESC LIMIT 1"; + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no summitExp rows found"); + haveLastFileSet = false; + psFree(output); + } else { + haveLastFileSet = true; + bool status = false; + lastFileSet = psStringCopy(psMetadataLookupStr(&status, output->data[0], "exp_id")); + psFree(output); + } + } + + // invoke dsproductls + // dsproductls --uri <> --last_fileset <> + bool status = false; + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + psString cmd = NULL; + if (haveLastFileSet) { + psStringAppend(&cmd, "%s --uri %s --last_fileset %s", + PRODUCT_LS_CMD, uri, lastFileSet); + psFree(lastFileSet); + } else { + psStringAppend(&cmd, "%s --uri %s", PRODUCT_LS_CMD, uri); + } + + psTrace("pzgetexp", PS_LOG_INFO, "cmd is: %s\n", cmd); + + FILE *output = popen(cmd, "r"); + psFree(cmd); + + if (!output) { + psError(PS_ERR_UNKNOWN, true, "popen() failed"); + return false; + } + psString cmdOutput = psSlurpFile(output); + pclose(output); + + psArray *newSummitExps = parseFileSets(config, cmdOutput); + psFree(cmdOutput); + if (!newSummitExps) { + // XXX not nessicarily an error + psError(PS_ERR_UNKNOWN, true, "no new fileSet/exp IDs"); + return false; + } + if (!psArrayLength(newSummitExps)) { + psError(PS_ERR_UNKNOWN, true, "no new fileSet/exp IDs"); + psFree(newSummitExps); + return true; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // create a temporry table + { + char *query = "CREATE TEMPORARY TABLE incoming" + " (exp_id VARCHAR(64), camera VARCHAR(64), telescope VARCHAR(64), dateobs DATETIME, exp_type VARCHAR(64), uri VARCHAR(255), PRIMARY KEY(exp_id, camera, telescope))" + " ENGINE=MEMORY"; + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newSummitExps); + return false; + } + } + + { + char *query = "INSERT INTO incoming (exp_id, camera, telescope, dateobs, exp_type, uri) VALUES (?, ?, ?, ?, ?, ?)"; + + long inserted = p_psDBRunQueryPrepared(config->dbh, newSummitExps, query); + if (inserted < 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newSummitExps); + return false; + } + // sanity check that we actually inserted something + if (inserted == 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error -- we should have inserted at least one row"); + psFree(newSummitExps); + return false; + } + } + + psFree(newSummitExps); + + // add new exps to pzPendingExp -- must be done before the new exps are + // added to summitExp because of the SQL logic + { + char *query = + "INSERT INTO pzPendingExp" + " SElECT" + " incoming.exp_id," + " incoming.camera," + " incoming.telescope" + " FROM incoming" + " LEFT JOIN summitExp" + " USING(exp_id, camera, telescope)" + " WHERE" + " summitExp.exp_id is NULL" + " AND summitExp.camera is NULL" + " AND summitExp.telescope is NULL"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // add new exps to summitExp + { + char *query = + "INSERT INTO summitExp" + " SElECT" + " incoming.*," + " NULL" + " FROM incoming" + " LEFT JOIN summitExp" + " USING(exp_id, camera, telescope)" + " WHERE" + " summitExp.exp_id is NULL" + " AND summitExp.camera is NULL" + " AND summitExp.telescope is NULL"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static psArray *parseFileSets(pxConfig *config, const char *str) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(str, NULL); + + // these are constants for all records parsed -- look them up before we do + // any work + bool status = false; + char *camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup item '-inst'"); + return NULL; + } + char *telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup item '-telescope'"); + return NULL; + } + + // split the string into lines + psList *doc = psStringSplit(str, "\n", false); + + psListIterator *lineCursor = psListIteratorAlloc(doc, 0, false); + + psArray *summitExps = psArrayAllocEmpty(psListLength(doc)); + psString line; + while ((line = psListGetAndIncrement(lineCursor))) { + psTrace("pzgetimfile", PS_LOG_INFO, "parsing line: %s\n", line); + + // split line into tokens + psList *tokens = psStringSplit(line, " ", false); + + // check to see if this line is a comment (or if the first token is + // NULL) + if (!psListGet(tokens, 0) || *((char *)psListGet(tokens, 0)) == '#') { + psFree(tokens); + continue; + } + + // check that we have the right number of tokens + // print "# uri fileset datetime type\n"; + if (psListLength(tokens) != 4) { + // error + return false; + } + + // find the values of interest + psListIterator *tokenCursor = psListIteratorAlloc(tokens, 0, false); + char *uri = psListGetAndIncrement(tokenCursor); + char *exp_id = psListGetAndIncrement(tokenCursor); // fileset + char *dateobsStr= psListGetAndIncrement(tokenCursor); // datetime + char *exp_type = psListGetAndIncrement(tokenCursor); // type + + // create a new metadata to represent this line and it's values + psMetadata *md = psMetadataAlloc(); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_id", 0, NULL, exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "camera", 0, NULL, camera)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "telescope", 0, NULL, telescope)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + psTime *dateobs = psTimeFromISO(dateobsStr, PS_TIME_UTC); + if (!psMetadataAddTime(md, PS_LIST_TAIL, "dateobs", 0, NULL, dateobs)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(dateobs); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + psFree(dateobs); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_type", 0, NULL, exp_type)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_type"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "uri", 0, NULL, uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item uri"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + + // must be freed after the new metadata is built -- holds the strings + psFree(tokenCursor); + psFree(tokens); + + // add the new metadata to the result set + psArrayAdd(summitExps, 0, md); + + // debugging + if (psTraceGetLevel("pzgetexp") == PS_LOG_INFO) { + psString doc = psMetadataConfigFormat(md); + psTrace("pzgetexp", PS_LOG_INFO, "parsed line as:\n %s\n", doc); + psFree(doc); + } + + psFree(md); + + } + + psFree(lineCursor); + psFree(doc); + + return summitExps; +} Index: /branches/rel-1-0/ippTools/src/pzgetexp.h =================================================================== --- /branches/rel-1-0/ippTools/src/pzgetexp.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pzgetexp.h (revision 11645) @@ -0,0 +1,27 @@ +/* + * pzgetexp.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PZGETEXP_H +#define PZGETEXP_H 1 + +#include "pxtools.h" + +pxConfig *pzgetexpConfig(pxConfig *config, int argc, char **argv); + +#endif // PZGETEXP_H Index: /branches/rel-1-0/ippTools/src/pzgetexpConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/pzgetexpConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pzgetexpConfig.c (revision 11645) @@ -0,0 +1,81 @@ +/* + * pzgetexpConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" + +pxConfig *pzgetexpConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + + psMetadata *args = psMetadataAlloc(); + psMetadataAddStr(args , PS_LIST_TAIL, "-uri", 0, + "DataStore product URI", ""); + psMetadataAddStr(args , PS_LIST_TAIL, "-inst", 0, + "camera name", ""); + psMetadataAddStr(args , PS_LIST_TAIL, "-telescope", 0, + "telescope name", ""); + + bool status = false; + if (!psArgumentParse(args, &argc, argv) + || argc != 1 + || strcmp(psMetadataLookupStr(&status, args, "-uri"), "") == 0 + || strcmp(psMetadataLookupStr(&status, args, "-inst"), "") == 0 + || strcmp(psMetadataLookupStr(&status, args, "-telescope"), "") == 0 + ) { + fprintf(stderr, "error parsing arguments\n"); + printf("\nPan-STARRS Phase Z Search Tool\n"); + printf("Usage: %s -uri -inst -telescope \n\n", + argv[0]); + psArgumentHelp(args); + psFree(config); + exit(EXIT_FAILURE); + } + + config->args = args; + // don't free args here as it's silly to increment the ref count then + // "free" it + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if(!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't connect to db\n"); + psFree(config); + exit(EXIT_FAILURE); + } + + return config; +} Index: /branches/rel-1-0/ippTools/src/pzgetimfiles.c =================================================================== --- /branches/rel-1-0/ippTools/src/pzgetimfiles.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pzgetimfiles.c (revision 11645) @@ -0,0 +1,677 @@ +/* + * pzgetimfiles.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtag.h" +#include "pxtools.h" +#include "pzgetimfiles.h" + +#define FILESET_LS_CMD "dsfilesetls" + +static bool go (pxConfig *config); +static psArray *parseFiles(pxConfig *config, const char *str); + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pzgetimfilesConfig(NULL, argc, argv); + + if (!go(config)) { + goto FAIL; + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool go(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + psString filesetid = psMetadataLookupStr(&status, config->args, "-filesetid"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filesetid"); + return false; + } + if (!filesetid) { + psError(PS_ERR_UNKNOWN, true, "-filesetid is required"); + return false; + } + + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return NULL; + } + if (!camera) { + psError(PS_ERR_UNKNOWN, true, "-inst is required"); + return NULL; + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return NULL; + } + if (!telescope) { + psError(PS_ERR_UNKNOWN, true, "-telescope is required"); + return NULL; + } + + // invoke dsfilesetls + psString cmd = NULL; + psStringAppend(&cmd, "%s --uri %s", FILESET_LS_CMD, uri); + + psTrace("pzgetimfiles", PS_LOG_INFO, "cmd is: %s\n", cmd); + + FILE *output = popen(cmd, "r"); + psFree(cmd); + + if (!output) { + psError(PS_ERR_UNKNOWN, true, "popen() failed"); + return false; + } + + psString cmdOutput = psSlurpFile(output); + int exitStatus = pclose(output); + + if (exitStatus != 0) { + psError(PS_ERR_UNKNOWN, true, "%s failed with exit status %d", + FILESET_LS_CMD, exitStatus); + psFree(cmdOutput); + return false; + } + + // prase output of dsfilesetls + psArray *newImfiles = parseFiles(config, cmdOutput); + if (!newImfiles) { + // XXX not nessicarily an error + psError(PS_ERR_UNKNOWN, true, "no new files/imfiles"); + psFree(cmdOutput); + return false; + } + psFree(cmdOutput); + + // save the number of new Imfiles; + long imfiles = psArrayLength(newImfiles); + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // if the fileset was empty (no files) then we can bail out early + if (imfiles == 0) { + psFree(newImfiles); + + char *query = + "UPDATE summitExp" + " SET imfiles = %d" + " WHERE exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, imfiles, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // remove the pzPendingExp entry for this exp (fileset) + { + char *query = + "DELETE FROM pzPendingExp" + " WHERE" + " exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have removed only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; + } + + // try not to insert duplicate pzPendingExp/pzPendingImfile entries + + // create a temp table + { + char *query = + "CREATE TEMPORARY TABLE incoming (exp_id VARCHAR(64), camera VARCHAR(64), telescope VARCHAR(64), file_id VARCHAR(64), bytes INT, md5sum VARCHAR(32), class VARCHAR(64), class_id VARCHAR(64), uri VARCHAR(255), PRIMARY KEY(exp_id, camera, telescope, class, class_id)) ENGINE=MEMORY"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newImfiles); + return false; + } + } + + // load the imfiles (files) into the temp table + { + char *query = "INSERT INTO incoming (exp_id, camera, telescope, file_id, bytes, md5sum, class, class_id, uri) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"; + + long inserted = p_psDBRunQueryPrepared(config->dbh, newImfiles, query); + if (inserted < 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newImfiles); + return false; + } + // sanity check that we actually inserted something + if (inserted == 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error -- we should have inserted at least one row"); + psFree(newImfiles); + return false; + } + } + + psFree(newImfiles); + + // queue the imfiles (files) in pzPendingImfile -- must be done before the + // imfile is insterted into summitImfile (after that happens when don't + // know which imfiles are new anymore (well thats not exactly true but it + // make be someday if we have to beable to resync to modified datastore + // entries) + { + char *query = + "INSERT INTO pzPendingImfile" + " SElECT" + " incoming.exp_id," + " incoming.camera," + " incoming.telescope," + " incoming.class," + " incoming.class_id," + " NULL" // exp_tag, will be assigned later + " FROM incoming" + " JOIN pzPendingExp" + " USING(exp_id, camera, telescope)" + " LEFT JOIN summitImfile" + " USING(exp_id, camera, telescope, class, class_id)" + " WHERE" + " summitImfile.exp_id is NULL" + " AND summitImfile.camera is NULL" + " AND summitImfile.telescope is NULL" + " AND summitImfile.class is NULL" + " AND summitImfile.class_id is NULL"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // copy imfiles (files) from the temp table into summitImfiles + { + char *query = + "INSERT INTO summitImfile" + " SElECT" + " incoming.exp_id," + " incoming.camera," + " incoming.telescope," + " incoming.file_id," + " incoming.bytes," + " incoming.md5sum," + " incoming.class," + " incoming.class_id," + " incoming.uri" + " FROM incoming" + " LEFT JOIN summitImfile" + " USING(exp_id, camera, telescope, class, class_id)" + " WHERE" + " summitImfile.exp_id is NULL" + " AND summitImfile.camera is NULL" + " AND summitImfile.telescope is NULL" + " AND summitImfile.class is NULL" + " AND summitImfile.class_id is NULL"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // update summitExp.imfiles (should have been NULL) + { + char *query = + "UPDATE summitExp" + " SET imfiles = (SELECT COUNT(*) FROM summitImfile" + " WHERE" + " exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s'" + ")" + " WHERE" + " exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, filesetid, camera, telescope, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have updated only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // copy the summitExp row into newExp in order to create a new exp_tag + { + psString exp_tag = pxGenExpTag(config, filesetid); + if (!exp_tag) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + char *query = + "INSERT INTO newExp" + " SElECT" + " '%s'," // exp_tag + " summitExp.exp_id," + " summitExp.camera," + " summitExp.telescope," + " summitExp.dateobs," + " summitExp.exp_type," + " summitExp.imfiles," + " 0" // error flags + " FROM summitExp" + " WHERE" + " summitExp.exp_id = '%s'" + " AND summitExp.camera = '%s'" + " AND summitExp.telescope = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, exp_tag, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(exp_tag); + return false; + } + psFree(exp_tag); + + // sanity check: we should have inserted only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // set the exp_tag for the imfiles (files) in the exposure (fileset) that + // we are investigating + { + char *query = + "UPDATE pzPendingImfile" + " SET exp_tag = (SELECT exp_tag from newExp" + " WHERE exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s')" + " WHERE" + " exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, filesetid, camera, telescope, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // cp the pzPendingExp entry to pzDoneExp + { + char *query = + "INSERT INTO pzDoneExp" + " SELECT" + " pzPendingExp.*" + " FROM pzPendingExp" + " WHERE" + " pzPendingExp.exp_id = '%s'" + " AND pzPendingExp.camera = '%s'" + " AND pzPendingExp.telescope = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have inserted only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // remove the pzPendingExp entry for this exp (fileset) + { + char *query = + "DELETE FROM pzPendingExp" + " WHERE" + " exp_id = '%s'" + " AND camera = '%s'" + " AND telescope = '%s'"; + if (!p_psDBRunQuery(config->dbh, query, filesetid, camera, telescope)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have removed only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static psArray *parseFiles(pxConfig *config, const char *str) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(str, NULL); + + // these are constants for all records parsed -- look them up before we do + // any work + bool status = false; + psString exp_id = psMetadataLookupStr(&status, config->args, "-filesetid"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for '-filesetid'"); + return NULL; + } + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, "-filesetid is required"); + return NULL; + } + + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return NULL; + } + if (!camera) { + psError(PS_ERR_UNKNOWN, true, "-inst is required"); + return NULL; + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return NULL; + } + if (!telescope) { + psError(PS_ERR_UNKNOWN, true, "-telescope is required"); + return NULL; + } + + // split the string into lines + psList *doc = psStringSplit(str, "\n", false); + + psListIterator *lineCursor = psListIteratorAlloc(doc, 0, false); + + psArray *pzPendingImfiles = psArrayAllocEmpty(psListLength(doc)); + psString line; + while ((line = psListGetAndIncrement(lineCursor))) { + psTrace("pzgetimfiles", PS_LOG_INFO, "parsing line: %s\n", line); + + // split line into tokens + psList *tokens = psStringSplit(line, " ", false); + + // check to see if this line is a comment (or if the first token is + // NULL) + if (!psListGet(tokens, 0) || *((char *)psListGet(tokens, 0)) == '#') { + psFree(tokens); + continue; + } + + // check that we have the right number of tokens + // print "# uri fileid bytes md5sum type \n"; + if (!psListLength(tokens) > 5) { + psError(PS_ERR_UNKNOWN, true, "invalid line format: %s", line); + psFree(tokens); + psFree(pzPendingImfiles); + psFree(lineCursor); + psFree(doc); + return false; + } + + // find the values of interest + psListIterator *tokenCursor = psListIteratorAlloc(tokens, 0, false); + char *uri = psListGetAndIncrement(tokenCursor); + char *file_id = psListGetAndIncrement(tokenCursor); // fileid + char *bytes = psListGetAndIncrement(tokenCursor); // bytes + char *md5sum = psListGetAndIncrement(tokenCursor); // md5sum + char *class = psListGetAndIncrement(tokenCursor); // type + char *class_id = psListGetAndIncrement(tokenCursor); // chipname + + // create a new metadata to represent this line and it's values + psMetadata *md = psMetadataAlloc(); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_id", 0, NULL, exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "camera", 0, NULL, camera)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "telescope", 0, NULL, telescope)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "file_id", 0, NULL, file_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bytes"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddS32(md, PS_LIST_TAIL, "bytes", 0, NULL, (psS32)atoi(bytes))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bytes"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "md5sum", 0, NULL, md5sum)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item md5sum"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "class", 0, NULL, class)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "class_id", 0, NULL, class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "uri", 0, NULL, uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item uri"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + + // must be freed after the new metadata is built -- holds the strings + psFree(tokenCursor); + psFree(tokens); + + // debugging + if (psTraceGetLevel("pzgetimfiles") >= PS_LOG_INFO) { + psString doc = psMetadataConfigFormat(md); + psTrace("pzgetimfiles", PS_LOG_INFO, "parsed line as:\n %s\n", doc); + psFree(doc); + } + + psArrayAdd(pzPendingImfiles, 0, md); + + psFree(md); + } + + psFree(lineCursor); + psFree(doc); + + return pzPendingImfiles;; +} Index: /branches/rel-1-0/ippTools/src/pzgetimfiles.h =================================================================== --- /branches/rel-1-0/ippTools/src/pzgetimfiles.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pzgetimfiles.h (revision 11645) @@ -0,0 +1,27 @@ +/* + * pzgetimfiles.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PZGETIMFILES_H +#define PZGETIMFILES 1 + +#include "pxtools.h" + +pxConfig *pzgetimfilesConfig(pxConfig *config, int argc, char **argv); + +#endif // PZGETIMFILES_H Index: /branches/rel-1-0/ippTools/src/pzgetimfilesConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/pzgetimfilesConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pzgetimfilesConfig.c (revision 11645) @@ -0,0 +1,87 @@ +/* + * pzgetimfilesConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" + +pxConfig *pzgetimfilesConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration!\n"); + goto FAIL; + } + + psMetadata *args = psMetadataAlloc(); + psMetadataAddStr(args , PS_LIST_TAIL, "-uri", 0, + "DataStore FileSet URI", ""); + psMetadataAddStr(args , PS_LIST_TAIL, "-filesetid", 0, + "FileSet ID", ""); + psMetadataAddStr(args, PS_LIST_TAIL, "-inst", 0, + "search by camera", NULL); + psMetadataAddStr(args, PS_LIST_TAIL, "-telescope", 0, + "search by telescope", NULL); + + + bool status = false; + if (!psArgumentParse(args, &argc, argv) + || argc != 1 + || strcmp(psMetadataLookupStr(&status, args, "-uri"), "") == 0 + || strcmp(psMetadataLookupStr(&status, args, "-filesetid"), "") == 0 + ) { + fprintf(stderr, "error parsing arguments\n"); + printf("\nPan-STARRS Phase Z Search Tool\n"); + printf("Usage: %s -uri -filesetid \n\n", argv[0]); + psArgumentHelp(args); + psFree(args); + goto FAIL; + } + + config->args = args; + // don't free args here as it's silly to increment the ref count then + // "free" it + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if(!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't connect to db\n"); + goto FAIL; + } + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/pztool.c =================================================================== --- /branches/rel-1-0/ippTools/src/pztool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pztool.c (revision 11645) @@ -0,0 +1,439 @@ +/* + * pztool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" +#include "pztool.h" + +static bool seenMode(pxConfig *config); +static bool pendingExpMode(pxConfig *config); +static bool pendingImfileMode(pxConfig *config); +static bool copydoneMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pztoolConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(PZTOOL_MODE_SEEN, seenMode); + MODECASE(PZTOOL_MODE_PENDINGEXP, pendingExpMode); + MODECASE(PZTOOL_MODE_PENDINGIMFILE, pendingImfileMode); + MODECASE(PZTOOL_MODE_COPYDONE, copydoneMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool seenMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy("SELECT * FROM summitExp"); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "summitExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "summitExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingExpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy( + "SELECT" + " summitExp.*" + " FROM pzPendingExp" + " JOIN summitExp" + " USING(exp_id, camera, telescope)" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "pzPendingExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(output); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "summitExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingImfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = psStringCopy( + "SELECT" + " summitImfile.*," + " pzPendingImfile.exp_tag" + " FROM pzPendingImfile" + " JOIN summitImfile" + " USING(exp_id, camera, telescope, class, class_id)" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "pzPendingImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(output); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pzPendingImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool copydoneMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psString exp_id = psMetadataLookupStr(&status, config->args, "-exp_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_id"); + return false; + } + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, "-exp_id is required"); + return false; + } + + psString camera = psMetadataLookupStr(&status, config->args, "-inst"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -inst"); + return NULL; + } + if (!camera) { + psError(PS_ERR_UNKNOWN, true, "-inst is required"); + return NULL; + } + + psString telescope = psMetadataLookupStr(&status, config->args, "-telescope"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -telescope"); + return NULL; + } + if (!telescope) { + psError(PS_ERR_UNKNOWN, true, "-telescope is required"); + return NULL; + } + + psString class = psMetadataLookupStr(&status, config->args, "-class"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class"); + return NULL; + } + if (!class) { + psError(PS_ERR_UNKNOWN, true, "-class is required"); + return NULL; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return NULL; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return NULL; + } + + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + // need to know exp_id, camera, telescope, class, class_id (to find the + // pzPendingImfile entry and the URI for the newImfile entry. + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // insert new imfile into newImfile + { + char *query = + "INSERT INTO newImfile" + " SElECT" + " pzPendingImfile.exp_tag," + " pzPendingImfile.class," + " pzPendingImfile.class_id," + " '%s'," // uri of downloaded file + " 0" // error flags + " FROM pzPendingImfile" + " WHERE" + " pzPendingImfile.exp_id = '%s'" + " AND pzPendingImfile.camera = '%s'" + " AND pzPendingImfile.telescope = '%s'" + " AND pzPendingImfile.class = '%s'" + " AND pzPendingImfile.class_id = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, uri, exp_id, camera, telescope, class, class_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have inserted only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // cp the imfile into pzDoneImfile + { + char *query = + "INSERT INTO pzDoneImfile" + " SElECT" + " pzPendingImfile.*," + " '%s'" // uri of downloaded file + " FROM pzPendingImfile" + " WHERE" + " pzPendingImfile.exp_id = '%s'" + " AND pzPendingImfile.camera = '%s'" + " AND pzPendingImfile.telescope = '%s'" + " AND pzPendingImfile.class = '%s'" + " AND pzPendingImfile.class_id = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, uri, exp_id, camera, telescope, class, class_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have inserted only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // removve the entry from pzPendingImfile + { + char *query = + "DELETE FROM pzPendingImfile" + " WHERE" + " pzPendingImfile.exp_id = '%s'" + " AND pzPendingImfile.camera = '%s'" + " AND pzPendingImfile.telescope = '%s'" + " AND pzPendingImfile.class = '%s'" + " AND pzPendingImfile.class_id = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, exp_id, camera, telescope, class, class_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have removed only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} Index: /branches/rel-1-0/ippTools/src/pztool.h =================================================================== --- /branches/rel-1-0/ippTools/src/pztool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/pztool.h (revision 11645) @@ -0,0 +1,35 @@ +/* + * pztool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PZTOOL_H +#define PZTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + PZTOOL_MODE_NONE = 0x0, + PZTOOL_MODE_SEEN, + PZTOOL_MODE_PENDINGEXP, + PZTOOL_MODE_PENDINGIMFILE, + PZTOOL_MODE_COPYDONE +} pztoolMode; + +pxConfig *pztoolConfig(pxConfig *config, int argc, char **argv); + +#endif // PZTOOL_H Index: /branches/rel-1-0/ippTools/src/pztoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/pztoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/pztoolConfig.c (revision 11645) @@ -0,0 +1,213 @@ +/* + * pztoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pztool.h" + +// this function can not fail -- exits on error +pxConfig *pztoolConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + + // -seen + psMetadata *seenArgs = psMetadataAlloc(); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure ID", NULL); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-inst", 0, + "define camera ID", NULL); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope ID", NULL); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddBool(seenArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure ID", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-inst", 0, + "define camera ID", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope ID", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-inst", 0, + "define camera ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -copydone + psMetadata *copydoneArgs = psMetadataAlloc(); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-exp_id", 0, + "define exposure ID", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-inst", 0, + "define camera ID", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-telescope", 0, + "define telescope ID", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-class", 0, + "define class", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-class_id", 0, + "define class_id", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-uri", 0, + "define storage uri", NULL); + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option); \ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + // find which mode we're running under + PXTOOL_MODE("-seen", PZTOOL_MODE_SEEN, seenArgs); + PXTOOL_MODE("-pendingexp", PZTOOL_MODE_PENDINGEXP, pendingexpArgs); + PXTOOL_MODE("-pendingimfile", PZTOOL_MODE_PENDINGIMFILE,pendingimfileArgs); + PXTOOL_MODE("-copydone", PZTOOL_MODE_COPYDONE, copydoneArgs); + + bool argErr = false; + if (config->mode == PZTOOL_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS Detrend Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); + } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(exp_tag); + addWhereStr(telescope); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + goto FAIL; + } + } + } + addWhereStr(exp_type); + addWhereStr(class); + addWhereStr(class_id); + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippTools/src/regtool.c =================================================================== --- /branches/rel-1-0/ippTools/src/regtool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/regtool.c (revision 11645) @@ -0,0 +1,1237 @@ +/* + * p0tool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "p0tool.h" + +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedimfileMode(pxConfig *config); +static bool processedimfileMode(pxConfig *config); +static bool updateprocessedimfileMode(pxConfig *config); + +static bool pendingexpMode(pxConfig *config); +static bool addprocessedexpMode(pxConfig *config); +static bool processedexpMode(pxConfig *config); +static bool updateprocessedexpMode(pxConfig *config); + + +// static p1PendingExpRow *newToP1PendingExp(newExpRow *newExp); +static p2PendingExpRow *newToP2PendingExp(pxConfig *config, newExpRow *newExp); +static p2PendingImfileRow *rawImfileToP2PendingImfile(pxConfig *config, rawImfileRow *rawImfile); + +static rawExpRow *newToRawExp(pxConfig *config, newExpRow *exp); +static rawImfileRow *newToRawImfile(pxConfig *config, newImfileRow *exp); +//static psU32 mapCodeStrToInt(const char *codeStr); + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psExit exit_status = PS_EXIT_SUCCESS; + + psLibInit(NULL); + + pxConfig *config = p0toolConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(P0TOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(P0TOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileMode); + MODECASE(P0TOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(P0TOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileMode); + MODECASE(P0TOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(P0TOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(P0TOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(P0TOOL_MODE_UPDATEPROCESSEDEXP, updateprocessedexpMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); + +FAIL: + // use the top-most error to determine the exit status + exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psErrorStackPrint(stderr, "\n"); + psLibFinalize(); + + exit(exit_status); +} + + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -limit"); + return false; + } + + // select newImfiles that: + // exp_tag is in newExp + // don't have their exp_tag in rawExp + // XXX having the same exp_tag in newExp and raw*Exp is probably an error + // that should be checked for + + psString query = psStringCopy( + "SELECT\n" + " newImfile.*,\n" + " newExp.workdir\n" + " FROM newImfile\n" + " LEFT JOIN newExp\n" + " USING(exp_tag)\n" + " LEFT JOIN rawExp\n" + " USING(exp_tag)\n" + " WHERE\n" + " newExp.exp_tag is NOT NULL\n" + " AND rawExp.exp_tag IS NULL\n" + ); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psFree(output); + return true; + } + + bool simple = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -simple"); + return false; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p0PendingImfile", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // XXX search by the whole frame some imfiles without a newExp don't get + // processed -- this may not be the correct thing to do + psString query = psStringCopy( + "SELECT\n" + " *\n" + " FROM\n" + " (SELECT newImfile.* FROM newImfile\n" + " LEFT JOIN newExp USING(exp_tag)\n" + " LEFT JOIN rawExp USING(exp_tag)\n" + " WHERE newExp.exp_tag IS NOT NULL\n" + " AND rawExp.exp_tag IS NULL) as Foo\n" + ); // WHERE class is generated from exp_tag, class, & class_id + + { + // build a query to search by exp_tag, class, class_id + psMetadata *where = psMetadataAlloc(); + bool status = false; + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + psFree(query); + return false; + } + if (exp_tag) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", 0, "==", exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + psFree(query); + return false; + } + } + psString class = psMetadataLookupStr(&status, config->args, "-class"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class"); + psFree(query); + return false; + } + if (class) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "class", 0, "==", class)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class"); + psFree(where); + psFree(query); + return false; + } + } + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + psFree(query); + return false; + } + if (class_id) { + if (!psMetadataAddStr(where, PS_LIST_TAIL, "class_id", 0, "==", class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(where); + psFree(query); + return false; + } + } + + // there's not + psString whereClaus = psDBGenerateWhereSQL(where, NULL); + psFree(where); + if (whereClaus) { + psStringAppend(&query, " %s", whereClaus); + psFree(whereClaus); + } + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending newImfile rows found"); + psFree(output); + return false; + } + + // insert 'newImfile's into rawImfile + if (psArrayLength(output) > 0) { + // start a transaction so we don't end up half of the imfiles we were + // trying to update uninserted + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + // convert newImfile metadata -> newImfile object + newImfileRow *object = newImfileObjectFromMetadata(output->data[i]); + // convert newImfile object -> rawImfile object + rawImfileRow *imfile = newToRawImfile(config, object); + if (!imfile) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to create a new rawImfile row"); + psFree(object); + psFree(output); + return false; + } + // insert the rawImfile object into the database + if (!rawImfileInsertObject(config->dbh, imfile)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to insert row into the database"); + psFree(imfile); + psFree(object); + psFree(output); + return false; + } + psFree(imfile); + // remove the neImfile object from the database + if (!newImfileDeleteObject(config->dbh, object)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to delete row from the database"); + psFree(object); + psFree(output); + return false; + } + psFree(object); + } + + // point of no return for rawImfile + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -faulted"); + return false; + } + + // find all rawImfiles matching the default query + psString query = psStringCopy( + "SELECT\n" + " *\n" + " FROM rawImfile\n" + " WHERE rawImfile.exp_tag is NOT NULL\n" //bogus conditional so there is a where clause to append to + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND rawImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND rawImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending rawImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool updateprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + if (!pxSetFaultCode(config->dbh, "rawImfile", config->where, code)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + return false; + } + + return true; +} + + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -limit"); + return false; + } + + // return only exps that: + // are not in rawExp + // have ALL of their imfiles in rawImfile (by count) + // and have no associated imfiles left in newImfile + + psString query = psStringCopy( + "SELECT" + " newExp.*" + " FROM newExp" + " LEFT JOIN newImfile" + " USING(exp_tag)" + " LEFT JOIN rawExp" + " USING(exp_tag)" + " WHERE" + " newImfile.exp_tag IS NULL" + " AND rawExp.exp_tag IS NULL" + " AND newExp.imfiles =" + " (SELECT COUNT(exp_tag) FROM rawImfile" + " WHERE" + " rawImfile.exp_tag = newExp.exp_tag" + " AND rawImfile.fault = 0)" + ); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psFree(output); + return true; + } + + bool simple = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -simple"); + return false; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p0PendingExp", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // make sure that the exp_tag(s) are ready to be updated based on: + // exp_tag is not in rawExp + // exp_tag is not in newImfile + // that the correct count of imfiles is in rawImfile + + bool status = false; + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + + bool detrend = psMetadataLookupBool(&status, config->args, "-detrend"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -detrend"); + return false; + } + + char *query = + "SELECT" + " newExp.*" + " FROM newExp" + " LEFT JOIN newImfile" + " USING(exp_tag)" + " LEFT JOIN rawExp" + " USING(exp_tag)" + " WHERE" + " newExp.exp_tag IS NOT NULL" + " AND newImfile.exp_tag IS NULL" + " AND rawExp.exp_tag IS NULL" + " AND newExp.imfiles =" + " (SELECT COUNT(exp_tag) FROM rawImfile" + " WHERE rawImfile.exp_tag = newExp.exp_tag)" + " AND newExp.exp_tag = '%s'"; + + if (!p_psDBRunQuery(config->dbh, query, exp_tag)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending newExp rows found"); + psFree(output); + return true; + } + + // start a transaction so we don't end up with an exp in both rawExp & + // newExp + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // insert the exp into rawExp + for (long i = 0; psArrayLength(output) > i; i++) { + psMetadata *row = output->data[i]; + // convert metadata into a newExp object + newExpRow *newExp = newExpObjectFromMetadata(row); + // convert newExp object into a rawExp object + rawExpRow *rawExp = newToRawExp(config, newExp); + if (!rawExp) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert newExp to rawExp"); + psFree(newExp); + psFree(output); + return false; + } + + // insert the rawExp object into the database + if (!rawExpInsertObject(config->dbh, rawExp)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(rawExp); + psFree(newExp); + psFree(output); + return false; + } + psFree(rawExp); + + // delete the newExp object from the database + if (!newExpDeleteObject(config->dbh, newExp)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newExp); + psFree(output); + return false; + } + + // if this is a detrend image don't put it in the p2 queue (and we're + // done) + if (detrend) { + psFree(newExp); + continue; + } + + // insert an entry into the p2PendingExp table + p2PendingExpRow *p2PendingExp = newToP2PendingExp(config, newExp); + if (!p2PendingExp) { + psError(PS_ERR_UNKNOWN, false, "failed to convert newExp to p2PendingExp"); + psFree(newExp); + psFree(output); + return false; + } + + // insert the p2PendingExp object into the database + if (!p2PendingExpInsertObject(config->dbh, p2PendingExp)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(p2PendingExp); + psFree(newExp); + psFree(output); + return false; + } + psFree(p2PendingExp); + + // find all of the rawImfiles associated with the p2PendingExp object + psArray *rawImfiles = NULL; + { + // build a query to search by exp_tag + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_tag", 0, "==", newExp->exp_tag)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_tag"); + psFree(where); + psFree(newExp); + psFree(output); + return false; + } + rawImfiles = rawImfileSelectRowObjects(config->dbh, where, 0); + psFree(where); + } + // sanity check that least results were found + if (!rawImfiles) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newExp); + psFree(output); + return false; + } + // sanity check the number of rawImfiles found + if (psArrayLength(rawImfiles) != newExp->imfiles) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "found %ld rawImfiles but expected %d", + psArrayLength(rawImfiles), newExp->imfiles); + psFree(newExp); + psFree(output); + return false; + } + // convert the rawImfiles into p2PendingImfiles and insert them into + // the database + for (long i = 0; i < psArrayLength(rawImfiles); i++) { + rawImfileRow *rawImfile = rawImfiles->data[i]; + p2PendingImfileRow *pendingImfile = rawImfileToP2PendingImfile(config, rawImfile); + if (!p2PendingImfileInsertObject(config->dbh, pendingImfile)){ + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(pendingImfile); + psFree(rawImfiles); + psFree(newExp); + psFree(output); + } + psFree(pendingImfile); + } + psFree(rawImfiles); + psFree(newExp); + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + psU64 limit = psMetadataLookupU64(&status, config->args, "-limit"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -limit"); + return false; + } + + bool faulted = psMetadataLookupU64(&status, config->args, "-faulted"); + if (!status) { + psError(PXTOOLS_ERR_PROG, false, "failed to lookup value for -faulted"); + return false; + } + + // find all rawImfiles matching the default query + psString query = psStringCopy( + "SELECT\n" + " *\n" + " FROM rawExp\n" + " WHERE\n" + " rawExp.exp_tag IS NOT NULL\n" // bogus where clause + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND rawExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND rawExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending rawExp rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool updateprocessedexpMode(pxConfig *config) +{ + bool status = false; + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + if (!pxSetFaultCode(config->dbh, "rawExp", config->where, code)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + return false; + } + + return true; +} + + +# if 0 +static p1PendingExpRow *newToP1PendingExp(newExpRow *newExp) +{ + return p1PendingExpRowAlloc( + newExp->exp_tag, + newExp->camera, + newExp->telescope, + newExp->exp_type, + newExp->imfiles, + "my filter", + 0.1, // airmass + 0.2, // ra + 0.3, // dec + 0.4, // exp time + 0.5, // background + "my recipe", + 0xff // XXX calc version number + ); +} +#endif + + +static p2PendingExpRow *newToP2PendingExp(pxConfig *config, newExpRow *exp) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(exp, NULL); + + // optional + bool status = false; + psString label = psMetadataLookupStr(&status, config->args, "-label"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -label"); + return false; + } + + p2PendingExpRow *p2Exp = p2PendingExpRowAlloc( + exp->exp_tag, + "my recipe", + 0xff, // XXX calc version number + 0xff, // XXX calc version number + label + ); + + return p2Exp; +} + + +static p2PendingImfileRow *rawImfileToP2PendingImfile(pxConfig *config, rawImfileRow *rawImfile) +{ + return p2PendingImfileRowAlloc( + rawImfile->exp_tag, + rawImfile->class_id, + rawImfile->uri, + "my recipe", + 0xff, // XXX calc version number + 0xff // XXX calc version number + ); +} + + +static rawExpRow *newToRawExp(pxConfig *config, newExpRow *exp) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(exp, NULL); + + bool status = false; + // optional + psString exp_type = psMetadataLookupStr(&status, config->args, "-exp_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_type"); + return false; + } + + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + + psF32 airmass = psMetadataLookupF32(&status, config->args, "-airmass"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass"); + return false; + } + + psF64 ra = psMetadataLookupF64(&status, config->args, "-ra"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ra"); + return false; + } + + psF64 decl = psMetadataLookupF64(&status, config->args, "-decl"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -decl"); + return false; + } + + psF32 exp_time = psMetadataLookupF32(&status, config->args, "-exp_time"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time"); + return false; + } + + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + + psF64 alt = psMetadataLookupF64(&status, config->args, "-alt"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -alt"); + return false; + } + + psF64 az = psMetadataLookupF64(&status, config->args, "-az"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -az"); + return false; + } + + psF32 ccd_temp = psMetadataLookupF32(&status, config->args, "-ccd_temp"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp"); + return false; + } + + psF64 posang = psMetadataLookupF32(&status, config->args, "-posang"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang"); + return false; + } + + psString object = psMetadataLookupStr(&status, config->args, "-object"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -object"); + return false; + } + psTime *dateobs = NULL; + { + psString dateobsStr = psMetadataLookupStr(&status, config->args, "-dateobs"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -dateobs"); + return false; + } + // pass through NULL as this is an optional field + if (dateobsStr) { + dateobs = psTimeFromISO(dateobsStr, PS_TIME_UTC); + } else { + dateobs = NULL; + } + } + + // default + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + rawExpRow *raw = rawExpRowAlloc( + exp->exp_tag, + exp->exp_id, + exp->camera, + exp->telescope, + dateobs, + exp_type, + exp->imfiles, + exp->workdir, + filter, + airmass, + ra, + decl, + exp_time, + bg, + bg_stdev, + bg_mean_stdev, + alt, + az, + ccd_temp, + posang, + object, + code + ); + + if (dateobs) { + psFree(dateobs); + } + + return raw; +} + + +static rawImfileRow *newToRawImfile(pxConfig *config, newImfileRow *imfile) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(exp, NULL); + + bool status = false; + // optional + psString exp_type = psMetadataLookupStr(&status, config->args, "-exp_type"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_type"); + return false; + } + + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + + psF32 airmass = psMetadataLookupF32(&status, config->args, "-airmass"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -airmass"); + return false; + } + + psF64 ra = psMetadataLookupF64(&status, config->args, "-ra"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ra"); + return false; + } + + psF64 decl = psMetadataLookupF64(&status, config->args, "-decl"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -decl"); + return false; + } + + psF32 exp_time = psMetadataLookupF32(&status, config->args, "-exp_time"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_time"); + return false; + } + + psF64 bg = psMetadataLookupF64(&status, config->args, "-bg"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg"); + return false; + } + + psF64 bg_stdev = psMetadataLookupF64(&status, config->args, "-bg_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_stdev"); + return false; + } + + psF64 bg_mean_stdev = psMetadataLookupF64(&status, config->args, "-bg_mean_stdev"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -bg_mean_stdev"); + return false; + } + + psF64 alt = psMetadataLookupF64(&status, config->args, "-alt"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -alt"); + return false; + } + + psF64 az = psMetadataLookupF64(&status, config->args, "-az"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -az"); + return false; + } + + psF32 ccd_temp = psMetadataLookupF32(&status, config->args, "-ccd_temp"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ccd_temp"); + return false; + } + + psF64 posang = psMetadataLookupF32(&status, config->args, "-posang"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -posang"); + return false; + } + + psString object = psMetadataLookupStr(&status, config->args, "-object"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -object"); + return false; + } + + psTime *dateobs = NULL; + { + psString dateobsStr = psMetadataLookupStr(&status, config->args, "-dateobs"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -dateobs"); + return false; + } + // pass through NULL as this is an optional field + if (dateobsStr) { + dateobs = psTimeFromISO(dateobsStr, PS_TIME_UTC); + } else { + dateobs = NULL; + } + } + + psS8 code = psMetadataLookupS8(&status, config->args, "-code"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -code"); + return false; + } + + rawImfileRow *raw = rawImfileRowAlloc( + imfile->exp_tag, + imfile->class, + imfile->class_id, + imfile->uri, + exp_type, + filter, + airmass, + ra, + decl, + exp_time, + bg, + bg_stdev, + bg_mean_stdev, + alt, + az, + ccd_temp, + posang, + object, + dateobs, + code + ); + + psFree(dateobs); + + return raw; +} Index: /branches/rel-1-0/ippTools/src/regtool.h =================================================================== --- /branches/rel-1-0/ippTools/src/regtool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/regtool.h (revision 11645) @@ -0,0 +1,39 @@ +/* + * p0tool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef P0TOOL_H +#define P0TOOL_H 1 + +#include "pxtools.h" + +typedef enum { + P0TOOL_MODE_NONE = 0x0, + P0TOOL_MODE_PENDINGIMFILE, + P0TOOL_MODE_ADDPROCESSEDIMFILE, + P0TOOL_MODE_PROCESSEDIMFILE, + P0TOOL_MODE_UPDATEPROCESSEDIMFILE, + P0TOOL_MODE_PENDINGEXP, + P0TOOL_MODE_ADDPROCESSEDEXP, + P0TOOL_MODE_PROCESSEDEXP, + P0TOOL_MODE_UPDATEPROCESSEDEXP, +} p0toolMode; + +pxConfig *p0toolConfig(pxConfig *config, int argc, char **argv); + +#endif // P0TOOL_H Index: /branches/rel-1-0/ippTools/src/regtoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/regtoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/regtoolConfig.c (revision 11645) @@ -0,0 +1,346 @@ +/* + * p0toolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "pxtools.h" +#include "p0tool.h" + +// this function can not fail -- exits on error +pxConfig *p0toolConfig(pxConfig *config, int argc, char **argv) { + + psExit exit_status = PS_EXIT_SUCCESS; + + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PXTOOLS_ERR_CONFIG, false, "Can't find site configuration"); + goto FAIL; + } + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddU64(pendingimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-filter", 0, + "define filter ", NULL); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-airmass", 0, + "define airmass", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-ra", 0, + "define RA", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-decl", 0, + "define DEC", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_time", 0, + "define exposure time", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-alt", 0, + "define altitute", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-az", 0, + "define azimuth", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-ccd_temp", 0, + "define ccd tempature", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-posang", 0, + "define rotator position angle", NAN); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-object", 0, + "define exposure object", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-dateobs", 0, + "define observation time", NULL); + psMetadataAddS8(addprocessedimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddU64(processedimfileArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -updateprocessedimfile + psMetadata *updateprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class", 0, + "search by class", NULL); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "search by class ID", NULL); + psMetadataAddS8(updateprocessedimfileArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-inst", 0, + "search by camera", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-telescope", 0, + "search by telescope", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_type", 0, + "search by exposure type", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-imfiles", 0, + "search for exps with N imfiles", NULL); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addprocessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "exp_tag to operate on (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-exp_type", 0, + "define exposure type", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-filter", 0, + "define filter ", NULL); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-airmass", 0, + "define airmass", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-ra", 0, + "define RA", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-decl", 0, + "define DEC", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-exp_time", 0, + "define exposure time", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg", 0, + "define exposue background", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, + "define exposue background stdev", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, + "define exposue background mean stdev", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-alt", 0, + "define altitute", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-az", 0, + "define azimuth", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ccd_temp", 0, + "define ccd tempature", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-posang", 0, + "define rotator position angle", NAN); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-object", 0, + "define exposure object", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-dateobs", 0, + "define observation time", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-label", 0, + "define label for phase 2 processing (non-detrend data only)", NULL); + psMetadataAddS8(addprocessedexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + /* + psMetadataAddStr(updateArgs, PS_LIST_TAIL, "-recip", 0, + "define URL", NULL); + psMetadataAddStr(updateArgs, PS_LIST_TAIL, "-mosiac", 0, + "define URL", NULL); + */ + psMetadataAddBool(addprocessedexpArgs, PS_LIST_TAIL, "-detrend", 0, + "declare this as detrend data", false); + + // -processedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, + "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-faulted", 0, + "only return imfiles with a fault status set", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -updatedprocessedexp + psMetadata *updatedprocessedexpArgs = psMetadataAlloc(); + psMetadataAddStr(updatedprocessedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, + "search by exposure ID", NULL); + psMetadataAddS8(updatedprocessedexpArgs, PS_LIST_TAIL, "-code", 0, + "set fault code (required)", 0); + + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PXTOOLS_ERR_CONFIG, true, "only one mode selection is allowed"); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {;\ + psError(PXTOOLS_ERR_PROG, false, "failed to add argset for %s", option);\ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + // find which mode we're running under + PXTOOL_MODE("-pendingimfile", P0TOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXTOOL_MODE("-addprocessedimfile", P0TOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileArgs); + PXTOOL_MODE("-processedimfile", P0TOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXTOOL_MODE("-updateprocessedimfile", P0TOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileArgs); + PXTOOL_MODE("-pendingexp", P0TOOL_MODE_PENDINGEXP,pendingexpArgs); + PXTOOL_MODE("-addprocessedexp", P0TOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXTOOL_MODE("-processedexp", P0TOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXTOOL_MODE("-updateprocessedexp", P0TOOL_MODE_UPDATEPROCESSEDEXP, updatedprocessedexpArgs); + + bool argErr = false; + if (config->mode == P0TOOL_MODE_NONE) { + psError(PXTOOLS_ERR_CONFIG, false, "missing mode argument"); + fprintf (stderr, "mode argument is required\n"); + argErr = true; + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + psError(PXTOOLS_ERR_CONFIG, false, "unknown argument"); + fprintf (stderr, "error parsing arguments\n"); + argErr = true; + } + + if (argErr) { + printf("\nPan-STARRS Phase 0 Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PXTOOLS_ERR_PROG, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereStr(exp_tag); + // convert '-inst' to 'camera' + { + psString str = NULL; + bool status = false; + if ((str = psMetadataLookupStr(&status, config->args, "-inst"))) { + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, "camera", 0, "==", str)) { + psError(PXTOOLS_ERR_PROG, false, "failed to add item camera"); + goto FAIL; + } + } + } + addWhereStr(telescope); + addWhereStr(exp_type); + { + int imfiles = 0; + bool status = false; + if ((imfiles = psMetadataLookupS32(&status, config->args, "-imfiles"))) { + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, "imfiles", 0, "==", imfiles)) { + psError(PXTOOLS_ERR_PROG, false, "failed to add item imfiles"); + goto FAIL; + } + } + } + addWhereStr(class); + addWhereStr(class_id); + addWhereStr(filter); + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + + // define Database handle, if used + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PXTOOLS_ERR_SYS, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + + return config; + +FAIL: + // use the top-most error to determine the exit status + exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(exit_status); +} Index: /branches/rel-1-0/ippTools/src/warptool.c =================================================================== --- /branches/rel-1-0/ippTools/src/warptool.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/warptool.c (revision 11645) @@ -0,0 +1,916 @@ +/* + * p4tool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "p4tool.h" + +static bool definerunMode(pxConfig *config); +static bool addinputimfileMode(pxConfig *config); +static bool towarpedimfileMode(pxConfig *config); +static bool addwarpedimfileMode(pxConfig *config); +static bool warpedimfileMode(pxConfig *config); +static bool tostackedimfileMode(pxConfig *config); +static bool addstackedimfileMode(pxConfig *config); +static bool stackedimfileMode(pxConfig *config); +static bool todiffimfileMode(pxConfig *config); +static bool adddiffimfileMode(pxConfig *config); +static bool diffimfileMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = p4toolConfig(NULL, argc, argv); + + switch (config->mode) { + MODECASE(P4TOOL_MODE_DEFINERUN, definerunMode); + MODECASE(P4TOOL_MODE_ADDINPUTIMFILE, addinputimfileMode); + MODECASE(P4TOOL_MODE_TOWARPEDIMFILE, towarpedimfileMode); + MODECASE(P4TOOL_MODE_ADDWARPEDIMFILE, addwarpedimfileMode); + MODECASE(P4TOOL_MODE_WARPEDIMFILE, warpedimfileMode); + MODECASE(P4TOOL_MODE_TOSTACKEDIMFILE, tostackedimfileMode); + MODECASE(P4TOOL_MODE_ADDSTACKEDIMFILE, addstackedimfileMode); + MODECASE(P4TOOL_MODE_STACKEDIMFILE, stackedimfileMode); + MODECASE(P4TOOL_MODE_TODIFFIMFILE, todiffimfileMode); + MODECASE(P4TOOL_MODE_ADDDIFFIMFILE, adddiffimfileMode); + MODECASE(P4TOOL_MODE_DIFFIMFILE, diffimfileMode); + default: + psAbort(argv[0], "invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_FAILURE); +} + +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // required options + bool status = false; + psString survey_mode = psMetadataLookupStr(&status, config->args, "-survey_mode"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -survey_mode"); + return false; + } + if (!survey_mode) { + psError(PS_ERR_UNKNOWN, true, "-survery_mode is required"); + return false; + } + + psString workdir = psMetadataLookupStr(&status, config->args, "-workdir"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -workdir"); + return false; + } + if (!workdir) { + psError(PS_ERR_UNKNOWN, true, "-workdir is required"); + return false; + } + + psString filter = psMetadataLookupStr(&status, config->args, "-filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -filter"); + return false; + } + if (!filter) { + psError(PS_ERR_UNKNOWN, true, "-survery_mode is required"); + return false; + } + + psString skycell_id = psMetadataLookupStr(&status, config->args, "-skycell_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -skycell_id"); + return false; + } + if (!skycell_id) { + psError(PS_ERR_UNKNOWN, true, "-survery_mode is required"); + return false; + } + + psString tess_id = psMetadataLookupStr(&status, config->args, "-tess_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -tess_id"); + return false; + } + if (!tess_id) { + psError(PS_ERR_UNKNOWN, true, "-survery_mode is required"); + return false; + } + + psS32 imfiles = psMetadataLookupS32(&status, config->args, "-imfiles"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -imfiles"); + return false; + } + if (imfiles < 1) { + psError(PS_ERR_UNKNOWN, true, "-imfiles is required"); + return false; + } + + // optional options + psF64 ra = psMetadataLookupS32(&status, config->args, "-ra"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -ra"); + return false; + } + + psF64 decl = psMetadataLookupS32(&status, config->args, "-decl"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -decl"); + return false; + } + + psString input_ss = psMetadataLookupStr(&status, config->args, "-input_ss"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -input_ss"); + return false; + } + + psString output_ss = psMetadataLookupStr(&status, config->args, "-output_ss"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -output_ss"); + return false; + } + + psTime *registered = NULL; + { + psString registeredStr = psMetadataLookupStr(&status, config->args, "-registered"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -registered"); + return false; + } + // pass through NULL as this is an optional field + if (registeredStr) { + registered = psTimeFromISO(registeredStr, PS_TIME_UTC); + } else { + registered = NULL; + } + } + + p4RunRow *p4Run = p4RunRowAlloc( + 0, + survey_mode, + "run", + workdir, + filter, + skycell_id, + tess_id, + ra, + decl, + input_ss, + output_ss + ); + psFree(registered); + if (!p4Run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc p4Run object"); + return true; + } + if (!p4RunInsertObject(config->dbh, p4Run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(p4Run); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + psFree(p4Run); + return false; + } + } + + if (!p4RunPrintObject(stdout, p4Run, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(p4Run); + return false; + } + + psFree(p4Run); + + return true; +} + +static bool addinputimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psString p4_id = psMetadataLookupStr(&status, config->args, "-p4_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p4_id"); + return false; + } + if (!p4_id) { + psError(PS_ERR_UNKNOWN, true, "-p4_id is required"); + return false; + } + + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + + // defaults to 0 + psS32 p3_version = psMetadataLookupS32(&status, config->args, "-p3_version"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p3_version"); + return false; + } + + // XXX need to validate the p4_id here + if (!p4InputImfileInsert(config->dbh, + (psS32)atoi(p4_id), + exp_tag, + p3_version, + class_id + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool towarpedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + " SELECT DISTINCT" + " p4InputImfile.p4_id," + " p2ProcessedImfile.*," + " p4Run.workdir" + " FROM p4Run" + " JOIN p4InputImfile" + " USING(p4_id)" + " LEFT JOIN p4WarpedImfile" + " USING(p4_id, exp_tag, p3_version, class_id)" + " JOIN p3ProcessedExp" + " ON p4InputImfile.exp_tag = p3ProcessedExp.exp_tag" + " AND p4InputImfile.p3_version = p3ProcessedExp.p3_version" + " JOIN p2ProcessedImfile" + " ON p4InputImfile.exp_tag = p2ProcessedImfile.exp_tag" + " AND p3ProcessedExp.p2_version = p2ProcessedImfile.p2_version" + " AND p4InputImfile.class_id = p2ProcessedImfile.class_id" + " WHERE" + " p4Run.state = 'run'" + " AND p4WarpedImfile.p4_id IS NULL" + " AND p4WarpedImfile.exp_tag IS NULL" + " AND p4WarpedImfile.p3_version IS NULL" + " AND p4WarpedImfile.class_id IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p4InputImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p4InputImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p4InputImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addwarpedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psString p4_id = psMetadataLookupStr(&status, config->args, "-p4_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p4_id"); + return false; + } + if (!p4_id) { + psError(PS_ERR_UNKNOWN, true, "-p4_id is required"); + return false; + } + + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + // defaults to 0 + psS32 p3_version = psMetadataLookupS32(&status, config->args, "-p3_version"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p3_version"); + return false; + } + + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // XXX need to validate that this coresponds to an p4InputImfile + if (!p4WarpedImfileInsert(config->dbh, + (psS32)atoi(p4_id), + exp_tag, + p3_version, + class_id, + uri, + b1_uri, + b2_uri + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool warpedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + " SELECT DISTINCT" + " p4WarpedImfile.*" + " FROM p4Run" + " JOIN p4WarpedImfile" + " USING(p4_id)" + " WHERE" + " p4Run.state = 'run'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p4WarpedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p4WarpedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p4WarpedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool tostackedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + " SELECT DISTINCT" + " p4WarpedImfile.*," + " p4Run.workdir" + " FROM p4Run" + " JOIN p4WarpedImfile" + " USING(p4_id)" + " LEFT JOIN p4StackedImfile" + " USING(p4_id, class_id)" + " WHERE" + " p4Run.state = 'run'" + " AND p4StackedImfile.p4_id IS NULL" + " AND p4Stackedmfile.class_id IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p4WarpedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p4WarpedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p4WarpedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addstackedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psString p4_id = psMetadataLookupStr(&status, config->args, "-p4_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p4_id"); + return false; + } + if (!p4_id) { + psError(PS_ERR_UNKNOWN, true, "-p4_id is required"); + return false; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // XXX need to validate that this coresponds to an p4WarpedImfile + if (!p4StackedImfileInsert(config->dbh, + (psS32)atoi(p4_id), + class_id, + uri, + b1_uri, + b2_uri + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool stackedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + " SELECT DISTINCT" + " p4StackedImfile.*" + " FROM p4Run" + " JOIN p4StackedImfile" + " USING(p4_id)" + " WHERE" + " p4Run.state = 'run'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p4StackedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p4StackedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p4StackedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool todiffimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + " SELECT DISTINCT" + " p4StackedImfile.*," + " p4WarpedImfile.uri as warped_uri," + " p4Run.workdir" + " FROM p4Run" + " JOIN p4StackedImfile" + " USING(p4_id)" + " JOIN p4WarpedImfile" + " USING(p4_id, class_id)" + " LEFT JOIN p4DiffImfile" + " ON p4WarpedImfile.p4_id = p4DiffImfile.p4_id" + " AND p4WarpedImfile.exp_tag = p4DiffImfile.exp_tag" + " AND p4WarpedImfile.p3_version = p4DiffImfile.p3_version" + " AND p4WarpedImfile.class_id = p4DiffImfile.class_id" + " WHERE" + " p4Run.state = 'run'" + " AND p4DiffImfile.p4_id IS NULL" + " AND p4DiffImfile.exp_tag IS NULL" + " AND p4DiffImfile.p3_version IS NULL" + " AND p4DiffImfile.class_id IS NULL" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p4StackedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p4StackedImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p4StackedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool adddiffimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + bool status = false; + psString p4_id = psMetadataLookupStr(&status, config->args, "-p4_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p4_id"); + return false; + } + if (!p4_id) { + psError(PS_ERR_UNKNOWN, true, "-p4_id is required"); + return false; + } + + psString exp_tag = psMetadataLookupStr(&status, config->args, "-exp_tag"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -exp_tag"); + return false; + } + if (!exp_tag) { + psError(PS_ERR_UNKNOWN, true, "-exp_tag is required"); + return false; + } + + psString class_id = psMetadataLookupStr(&status, config->args, "-class_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -class_id"); + return false; + } + if (!class_id) { + psError(PS_ERR_UNKNOWN, true, "-class_id is required"); + return false; + } + + psString uri = psMetadataLookupStr(&status, config->args, "-uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -uri"); + return false; + } + if (!uri) { + psError(PS_ERR_UNKNOWN, true, "-uri is required"); + return false; + } + + // defaults to 0 + psS32 p3_version = psMetadataLookupS32(&status, config->args, "-p3_version"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -p3_version"); + return false; + } + + // optional + psString b1_uri = psMetadataLookupStr(&status, config->args, "-b1_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b1_uri"); + return false; + } + + psString b2_uri = psMetadataLookupStr(&status, config->args, "-b2_uri"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -b2_uri"); + return false; + } + + // XXX need to validate that this coresponds to an p4InputImfile + if (!p4DiffImfileInsert(config->dbh, + (psS32)atoi(p4_id), + exp_tag, + p3_version, + class_id, + uri, + b1_uri, + b2_uri + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool diffimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psString query = psStringCopy( + " SELECT DISTINCT" + " p4DiffImfile.*" + " FROM p4Run" + " JOIN p4DiffImfile" + " USING(p4_id)" + " WHERE" + " p4Run.state = 'run'" + ); + + if (config->where) { + psString whereClause = psDBGenerateWhereConditionSQL(config->where, "p4DiffImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no p4DiffImfile rows found"); + psFree(output); + return true; + } + + bool simple = false; + { + bool status = false; + simple = psMetadataLookupBool(&status, config->args, "-simple"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -simple"); + return false; + } + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "p4DiffImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} Index: /branches/rel-1-0/ippTools/src/warptool.h =================================================================== --- /branches/rel-1-0/ippTools/src/warptool.h (revision 11645) +++ /branches/rel-1-0/ippTools/src/warptool.h (revision 11645) @@ -0,0 +1,42 @@ +/* + * p4tool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef P4TOOL_H +#define P4TOOL_H 1 + +#include "pxtools.h" + +typedef enum { + P4TOOL_MODE_NONE = 0x0, + P4TOOL_MODE_DEFINERUN, + P4TOOL_MODE_ADDINPUTIMFILE, + P4TOOL_MODE_TOWARPEDIMFILE, + P4TOOL_MODE_ADDWARPEDIMFILE, + P4TOOL_MODE_WARPEDIMFILE, + P4TOOL_MODE_TOSTACKEDIMFILE, + P4TOOL_MODE_ADDSTACKEDIMFILE, + P4TOOL_MODE_STACKEDIMFILE, + P4TOOL_MODE_TODIFFIMFILE, + P4TOOL_MODE_ADDDIFFIMFILE, + P4TOOL_MODE_DIFFIMFILE +} p4toolMode; + +pxConfig *p4toolConfig(pxConfig *config, int argc, char **argv); + +#endif // P4TOOL_H Index: /branches/rel-1-0/ippTools/src/warptoolConfig.c =================================================================== --- /branches/rel-1-0/ippTools/src/warptoolConfig.c (revision 11645) +++ /branches/rel-1-0/ippTools/src/warptoolConfig.c (revision 11645) @@ -0,0 +1,301 @@ +/* + * p4toolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "p4tool.h" + +// this function can not fail -- exits on error +pxConfig *p4toolConfig(pxConfig *config, int argc, char **argv) { + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + goto FAIL; + } + + psTime *time = psTimeGetNow(PS_TIME_TAI); + psString now = psTimeToISO(time); + psFree(time); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-survey_mode", 0, + "define the type of suvery (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-workdir", 0, + "define workdir (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-filter", 0, + "define filter (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-skycell_id", 0, + "define skycell ID (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-tess_id", 0, + "define tessellation ID (required)", NULL); + psMetadataAddF64(definerunArgs, PS_LIST_TAIL, "-ra", 0, + "define RA", NAN); + psMetadataAddF64(definerunArgs, PS_LIST_TAIL, "-decl", 0, + "define DEC", NAN); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-input_ss", 0, + "define input static sky ", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-output_ss", 0, + "define output static sky ", NULL); + psMetadataAddF64(definerunArgs, PS_LIST_TAIL, "-imfiles", 0, + "define number of imfiles in the sky cell", 0); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-registered", 0, + "time detrend run was registered", now); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addinputimfile + psMetadata *addinputimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addinputimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "define p4 ID (required)", NULL); + psMetadataAddStr(addinputimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure tag (required)", NULL); + psMetadataAddS32(addinputimfileArgs, PS_LIST_TAIL, "-p3_version", 0, + "define phase 3 version of exposure tag (required)", 0); + psMetadataAddStr(addinputimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", 0); + + // -towarpedimfile + psMetadata *towarpedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(towarpedimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "search by p4 ID", NULL); + psMetadataAddBool(towarpedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addwarpedimfile + psMetadata *addwarpedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addwarpedimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "define p4 ID (required)", NULL); + psMetadataAddStr(addwarpedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure tag (required)", NULL); + psMetadataAddS32(addwarpedimfileArgs, PS_LIST_TAIL, "-p3_version", 0, + "define phase 3 version of exposure tag (required)", 0); + psMetadataAddStr(addwarpedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", 0); + psMetadataAddStr(addwarpedimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URI of file (required)", 0); + psMetadataAddStr(addwarpedimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addwarpedimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + + // -warpedimfile + psMetadata *warpedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(warpedimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "search by p4 ID", NULL); + psMetadataAddBool(warpedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -tostackedimfile + psMetadata *tostackedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(tostackedimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "search by p4 ID", NULL); + psMetadataAddBool(tostackedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -addstackedimfile + psMetadata *addstackedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(addstackedimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "define p4 ID (required)", NULL); + psMetadataAddStr(addstackedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure tag (required)", NULL); + psMetadataAddS32(addstackedimfileArgs, PS_LIST_TAIL, "-p3_version", 0, + "define phase 3 version of exposure tag (required)", 0); + psMetadataAddStr(addstackedimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", 0); + psMetadataAddStr(addstackedimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URI of file (required)", 0); + psMetadataAddStr(addstackedimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(addstackedimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + + // -stackedimfile + psMetadata *stackedimfileArgs = psMetadataAlloc(); + psMetadataAddStr(stackedimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "search by p4 ID", NULL); + psMetadataAddBool(stackedimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -todiffimfile + psMetadata *todiffimfileArgs = psMetadataAlloc(); + psMetadataAddStr(todiffimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "search by p4 ID", NULL); + psMetadataAddBool(todiffimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + // -adddiffimfile + psMetadata *adddiffimfileArgs = psMetadataAlloc(); + psMetadataAddStr(adddiffimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "define p4 ID (required)", NULL); + psMetadataAddStr(adddiffimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, + "define exposure tag (required)", NULL); + psMetadataAddS32(adddiffimfileArgs, PS_LIST_TAIL, "-p3_version", 0, + "define phase 3 version of exposure tag (required)", 0); + psMetadataAddStr(adddiffimfileArgs, PS_LIST_TAIL, "-class_id", 0, + "define class ID (required)", 0); + psMetadataAddStr(adddiffimfileArgs, PS_LIST_TAIL, "-uri", 0, + "define URI of file (required)", 0); + psMetadataAddStr(adddiffimfileArgs, PS_LIST_TAIL, "-b1_uri", 0, + "define banana 1", NULL); + psMetadataAddStr(adddiffimfileArgs, PS_LIST_TAIL, "-b2_uri", 0, + "define banana 2", NULL); + + // -diffimfile + psMetadata *diffimfileArgs = psMetadataAlloc(); + psMetadataAddStr(diffimfileArgs, PS_LIST_TAIL, "-p4_id", 0, + "search by p4 ID", NULL); + psMetadataAddBool(diffimfileArgs, PS_LIST_TAIL, "-simple", 0, + "use the simple output format", false); + + psFree(now); + +#define PXTOOL_MODE(option, modeval, argset) \ +{ \ + int N = 0; \ + if ((N = psArgumentGet (argc, argv, option))) { \ + psArgumentRemove (N, &argc, argv); \ + if (config->mode) { \ + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); \ + goto FAIL; \ + } \ + config->mode = modeval; \ + config->args = psMemIncrRefCounter(argset); \ + } \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, NULL, argset)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option); \ + } \ + psFree(argset); \ +} + + psMetadata *argSets = psMetadataAlloc(); + // find which mode we're running under + PXTOOL_MODE("-definerun", P4TOOL_MODE_DEFINERUN, definerunArgs); + PXTOOL_MODE("-addinputimfile", P4TOOL_MODE_ADDINPUTIMFILE, addinputimfileArgs); + PXTOOL_MODE("-towarpedimfile", P4TOOL_MODE_TOWARPEDIMFILE, towarpedimfileArgs); + PXTOOL_MODE("-addwarpedimfile", P4TOOL_MODE_ADDWARPEDIMFILE, addwarpedimfileArgs); + PXTOOL_MODE("-warpedimfile", P4TOOL_MODE_WARPEDIMFILE, warpedimfileArgs); + PXTOOL_MODE("-tostackedimfile", P4TOOL_MODE_TOSTACKEDIMFILE, tostackedimfileArgs); + PXTOOL_MODE("-addstackedimfile", P4TOOL_MODE_ADDSTACKEDIMFILE, addstackedimfileArgs); + PXTOOL_MODE("-stackedimfile", P4TOOL_MODE_STACKEDIMFILE, stackedimfileArgs); + PXTOOL_MODE("-todiffimfile", P4TOOL_MODE_TODIFFIMFILE, todiffimfileArgs); + PXTOOL_MODE("-adddiffimfile", P4TOOL_MODE_ADDDIFFIMFILE, adddiffimfileArgs); + PXTOOL_MODE("-diffimfile", P4TOOL_MODE_DIFFIMFILE, diffimfileArgs); + + bool argErr = false; + if (config->mode == P4TOOL_MODE_NONE) { + argErr = true; + fprintf (stderr, "mode argument is required\n"); + } else if (! psArgumentParse(config->args, &argc, argv) || argc != 1) { + argErr = true; + fprintf (stderr, "error parsing arguments\n"); + } + + if (argErr) { + printf("\nPan-STARRS Phase 4 Tool\n"); + printf("Usage: %s []\n\n", argv[0]); + printf(" :\n\n"); + + psMetadataIterator *iter = psMetadataIteratorAlloc(argSets, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!item->type == PS_DATA_METADATA) { + psAbort(argv[0], "all options must be specified as a metadata"); + } + + fprintf(stdout, "%s ", item->name); + psArgumentHelp(item->data.md); + } + psFree(iter); + + psFree(argSets); + goto FAIL; + } + + psFree(argSets); + + // setup search criterion +#define addWhereStr(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + +#define addWhereS32(name) \ +{ \ + psS32 s32 = 0; \ + bool status = false; \ + if ((s32= psMetadataLookupS32(&status, config->args, "-" #name))) { \ + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, #name, 0, "==", s32)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + goto FAIL; \ + } \ + } \ +} + + + // generate SQL where clause + config->where = psMetadataAlloc(); + + addWhereS32(p4_id); + + if (config->where->list->n < 1) { + psFree(config->where); + config->where = NULL; + } + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = pmConfigDB(config->modules); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + goto FAIL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; + +FAIL: + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ippconfig/.cvsignore =================================================================== --- /branches/rel-1-0/ippconfig/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippconfig/.cvsignore (revision 11645) @@ -0,0 +1,13 @@ +Makefile +Makefile.in +aclocal.m4 +autom4te.cache +config.log +config.status +configure +dvo.site +install-sh +ippconfig-0.0.1.tar.bz2 +ippconfig-0.0.1.tar.gz +missing +ipprc.config Index: /branches/rel-1-0/ippconfig/Makefile.am =================================================================== --- /branches/rel-1-0/ippconfig/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippconfig/Makefile.am (revision 11645) @@ -0,0 +1,31 @@ +SUBDIRS = \ + dvo.cameras \ + recipes \ + isp \ + megacam \ + simple + +install_files = \ + GSCregions.tbl \ + dvo.photcodes + +built_sources = \ + ipprc.config.in \ + dvo.site.in + +built_files = \ + ipprc.config \ + dvo.site + +installdir = $(datadir)/ippconfig + +install_DATA = $(install_files) $(built_files) + +install-data-hook: + chmod 0755 $(installdir) + +EXTRA_DIST = $(install_files) $(built_sources) + +BUILT_SOURCES = $(built_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippconfig/README =================================================================== --- /branches/rel-1-0/ippconfig/README (revision 11645) +++ /branches/rel-1-0/ippconfig/README (revision 11645) @@ -0,0 +1,18 @@ + +Please read installation section of the IPP User's Guide for more details. + +To build and install the IPP configuration scripts, run the following: + +configure +make +make install + +After installation: + +Copy dvo.site to ~/.ptolemyrc + +Copy ipprc.config to ~/.ipprc + +Edit this file: + - Set the DBUSER and DBPASSWORD to your mysql username and password + Index: /branches/rel-1-0/ippconfig/autogen.sh =================================================================== --- /branches/rel-1-0/ippconfig/autogen.sh (revision 11645) +++ /branches/rel-1-0/ippconfig/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=ippConfig +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOlIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +#($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $AUTOHEADER installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" +# DIE=1 +#} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +#$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +#$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ippconfig/configure.ac =================================================================== --- /branches/rel-1-0/ippconfig/configure.ac (revision 11645) +++ /branches/rel-1-0/ippconfig/configure.ac (revision 11645) @@ -0,0 +1,24 @@ +AC_PREREQ(2.59) + +AC_INIT([ippConfig], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([ipprc.config.in]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_MAINTAINER_MODE + +AC_PROG_INSTALL + +IPP_CONFIG_DIR="${datadir}/ippconfig" +AC_SUBST(IPP_CONFIG_DIR) + +AC_CONFIG_FILES([ + Makefile + dvo.cameras/Makefile + recipes/Makefile + isp/Makefile + megacam/Makefile + simple/Makefile + ipprc.config + dvo.site +]) +AC_OUTPUT Index: /branches/rel-1-0/ippconfig/ctio_mosaic2/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/ctio_mosaic2/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ctio_mosaic2/camera.config (revision 11645) @@ -0,0 +1,126 @@ +# Camera configuration file for MegaCam: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR ctio_mosaic2/format.config +END + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + ccd00 STR LeftAmp RightAmp + ccd01 STR LeftAmp RightAmp + ccd02 STR LeftAmp RightAmp + ccd03 STR LeftAmp RightAmp + ccd04 STR LeftAmp RightAmp + ccd05 STR LeftAmp RightAmp + ccd06 STR LeftAmp RightAmp + ccd07 STR LeftAmp RightAmp +END + + +# move this elsewhere? we need a lookup table to go from filter ID to abstract name +FILTER.ID METADATA + VR.Supermacho.c6027 STR VR +END + +# Recipe options +RECIPES METADATA + # Recipes for ppImage + PPIMAGE STR ctio_mosaic2/ppImage.config # Default: all (normal) options on + PSPHOT STR ctio_mosaic2/psphot.config # Default: all (normal) options on +END + + +# Rejection levels for detrend creation +REJECTION METADATA + TYPE LIMITS FILTER EXPECTED IMFILE.MEAN IMFILE.STDEV EXP.MEAN EXP.STDEV EXP.MEANSTDEV ENSEMBLE.MEAN ENSEMBLE.STDEV ENSEMBLE.MEANSTDEV IMFILE.SN EXP.SN IMFILE.MEANSTDEV + FLAT MULTI + SKYFLAT MULTI + DOMEFLAT MULTI + + BIAS LIMITS * 0 1 4 0.5 4 0 3 4 0 0 0 0.5 + DARK LIMITS * 0 1 4 0.5 4 0 3 4 0 0 0 0.5 + FLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 + SKYFLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 + DOMEFLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS u 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS g 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS r 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS i 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS z 0 0 0 0 0 0 0 0 3 0 0 0 + FRINGE LIMITS * 0 0 0 0 0 0 0 0 0 0 0 0 + +# FILTER is an additional qualifier, and may be "*" (or absent!), in which case it matches everything +# EXPECTED is the expected mean value +# IMFILE.MEAN is the maximum permitted mean value for an imfile, relative to the standard deviation +# IMFILE.STDEV is the maximum permitted standard deviation for an imfile +# EXP.MEAN is the maximum permitted mean value for an exposure, relative to the standard deviation +# EXP.STDEV is the maximum permitted standard deviation for an exposure +# EXP.MEANSTDEV is the maximum permitted mean standard deviation for an exposure relative to the mean +# ENSEMBLE.MEAN is the maximum permitted mean for an ensemble of exposures +# ENSEMBLE.STDEV is the maximum permitted standard deviation for an ensemble of exposures +# ENSEMBLE.MEANSTDEV is the maximum permitted mean standard deviation for an ensemble of exposures +# IMFILE.SN is the minimum permitted signal-to-noise for an imfile +# EXP.SN is the minimum permitted signal-to-noise for an exposure +# These values (all except FILTER) may be zero, in which case no clipping is applied. + +END + + +FILERULES METADATA + ### Redirections + PSASTRO.INPUT STR PSASTRO.INPUT.CMP + PSASTRO.OUTPUT STR PSASTRO.OUTPUT.CMP + PSPHOT.OUTPUT STR PSPHOT.OUTPUT.CMF + + ### input file definitions + TYPE INPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA DATA.LEVEL FILE.TYPE + PPIMAGE.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CELL IMAGE + + ### use these entries to get the detrend images from specific files + #PPIMAGE.MASK INPUT megacam/mask.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP MASK + #PPIMAGE.BIAS INPUT path://PAP_ESSENCE/CTIO_MOSAIC2.bias.1.2.fpa.fits {CHIP.NAME} {CELL.NAME} NONE CELL IMAGE + #PPIMAGE.DARK INPUT file:///something.fits {CHIP.NAME} {CHIP.NAME} NONE CELL IMAGE + #PPIMAGE.FLAT INPUT file:///something.fits {CHIP.NAME} {CHIP.NAME} NONE CELL IMAGE + #PPIMAGE.FRINGE INPUT file:///something.fits {CHIP.NAME} {CHIP.NAME} NONE CELL FRINGE + + ### use these entries to get the detrend images from the database + #PPIMAGE.MASK INPUT @DETDB {CHIP.NAME} fpa NONE CHIP IMAGE + PPIMAGE.BIAS INPUT @DETDB {CHIP.NAME} fpa NONE CHIP IMAGE + PPIMAGE.DARK INPUT @DETDB {CHIP.NAME} fpa NONE CHIP IMAGE + PPIMAGE.FLAT INPUT @DETDB {CHIP.NAME} fpa FILTER CHIP IMAGE + + PSPHOT.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + + PSASTRO.INPUT.CMP INPUT @FILES NONE NONE PHU CHIP CMP + PSASTRO.INPUT.CMF INPUT @FILES NONE SMPDATA PHU CHIP CMF + + ### output file definitions + TYPE OUTPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA FILE.LEVEL DATA.LEVEL FILE.TYPE FILE.SAVE FILE.FORMAT + PPIMAGE.OUTPUT OUTPUT {OUTPUT}.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CELL IMAGE TRUE NONE + PPIMAGE.OUTPUT.MASK OUTPUT {OUTPUT}.mask.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CELL MASK FALSE NONE + PPIMAGE.OUTPUT.WEIGHT OUTPUT {OUTPUT}.wt.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CELL WEIGHT FALSE NONE + PPIMAGE.BIN1 OUTPUT {OUTPUT}.b1.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CHIP IMAGE TRUE NONE + PPIMAGE.BIN2 OUTPUT {OUTPUT}.b2.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CHIP IMAGE TRUE NONE + + PPIMAGE.OUTPUT.CHIP OUTPUT {OUTPUT}.chip.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CHIP IMAGE TRUE NONE + PPIMAGE.OUTPUT.FPA1 OUTPUT {OUTPUT}.fpa1.fits NONE NONE NONE FPA FPA IMAGE TRUE NONE + PPIMAGE.OUTPUT.FPA2 OUTPUT {OUTPUT}.fpa2.fits NONE NONE NONE FPA FPA IMAGE TRUE NONE + + PPIMAGE.JPEG1 OUTPUT {OUTPUT}.b1.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + PPIMAGE.JPEG2 OUTPUT {OUTPUT}.b2.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + + PSPHOT.RESID OUTPUT {OUTPUT}.res.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + PSPHOT.BACKGND OUTPUT {OUTPUT}.bck.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + PSPHOT.BACKSUB OUTPUT {OUTPUT}.sub.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + PSPHOT.BACKMDL OUTPUT {OUTPUT}.mdl.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + + PSPHOT.OUTPUT.RAW OUTPUT {OUTPUT}.{CHIP.NAME} NONE NONE PHU CHIP CHIP RAW TRUE NONE + PSPHOT.OUTPUT.SX OUTPUT {OUTPUT}.sx NONE NONE PHU CHIP CHIP SX TRUE NONE + PSPHOT.OUTPUT.OBJ OUTPUT {OUTPUT}.obj NONE NONE PHU CHIP CHIP OBJ TRUE NONE + PSPHOT.OUTPUT.CMF OUTPUT {OUTPUT}.cmf NONE SMPDATA PHU CHIP CHIP CMF TRUE NONE + PSPHOT.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.cmp NONE NONE PHU CHIP CHIP CMP TRUE NONE + + PSASTRO.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.smp NONE NONE PHU CHIP CHIP CMP TRUE NONE + +END Index: /branches/rel-1-0/ippconfig/ctio_mosaic2/format.config =================================================================== --- /branches/rel-1-0/ippconfig/ctio_mosaic2/format.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ctio_mosaic2/format.config (revision 11645) @@ -0,0 +1,158 @@ +# Format file for CTIO Mosaic2 camera used by ESSENCE + +# How to identify this type +RULE METADATA + TELESCOP STR CTIO 4.0 meter telescope + DETECTOR STR Mosaic2 + EXTEND BOOL T + NEXTEND S32 16 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CELL # The extensions represent cells + FPA.NAME STR FILENAME # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:type + im1 STR ccd00:LeftAmp:left + im2 STR ccd00:RightAmp:right + im3 STR ccd01:LeftAmp:left + im4 STR ccd01:RightAmp:right + im5 STR ccd02:LeftAmp:left + im6 STR ccd02:RightAmp:right + im7 STR ccd03:LeftAmp:left + im8 STR ccd03:RightAmp:right + im9 STR ccd04:LeftAmp:left + im10 STR ccd04:RightAmp:right + im11 STR ccd05:LeftAmp:left + im12 STR ccd05:RightAmp:right + im13 STR ccd06:LeftAmp:left + im14 STR ccd06:RightAmp:right + im15 STR ccd07:LeftAmp:left + im16 STR ccd07:RightAmp:right +END + +# Specify the cell data +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR TRIMSEC + CELL.XPARITY S32 1 + CELL.X0 S32 0 + END + right METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR TRIMSEC + CELL.XPARITY S32 1 + CELL.X0 S32 1024 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR OBJECT + FPA.TIME STR MJD-OBS + CHIP.TEMP STR CCDTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.XBIN STR CCDSUM + CELL.YBIN STR CCDSUM +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.POSANGLE F32 0.0 + FPA.TIMESYS STR UTC # Header says "UTC approximate" + FPA.ALT F32 0.0 # Not present in header + FPA.AZ F32 0.0 # Not present in header + CELL.TIMESYS STR UTC # Header says "UTC approximate" + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 + + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA + ccd00 S32 0 + ccd01 S32 2048 + ccd02 S32 4096 + ccd03 S32 6144 + ccd04 S32 0 + ccd05 S32 2048 + ccd06 S32 4096 + ccd07 S32 6144 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA + ccd00 S32 0 + ccd01 S32 0 + ccd02 S32 0 + ccd03 S32 0 + ccd04 S32 4096 + ccd05 S32 4096 + ccd06 S32 4096 + ccd07 S32 4096 + END + CHIP.XPARITY S32 1 + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA + ccd00 S32 1 + ccd01 S32 1 + ccd02 S32 1 + ccd03 S32 1 + ccd04 S32 1 + ccd05 S32 1 + ccd06 S32 1 + ccd07 S32 1 + END + +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# FPA.BIAS METADATA +# TABLE STR Camera +# COLUMN STR gain +# chipId STR {CHIP.NAME} +# cellId STR {CELL.NAME} +# time STR {CELL.TIME} +# END +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +# CHIP.X0 STR FORTRAN +# CHIP.Y0 STR FORTRAN +# CELL.X0 STR FORTRAN +# CELL.Y0 STR FORTRAN +END Index: /branches/rel-1-0/ippconfig/ctio_mosaic2/ppImage.config =================================================================== --- /branches/rel-1-0/ippconfig/ctio_mosaic2/ppImage.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ctio_mosaic2/ppImage.config (revision 11645) @@ -0,0 +1,110 @@ +### ppImage recipe configuration file for CTIO MOSAIC2 + +# List of tasks to perform +SHUTTER BOOL FALSE # Shutter correction +DARK BOOL FALSE # Dark correction + +BASE.FITS BOOL TRUE # Save base detrended image? +BIN1.FITS BOOL TRUE # Save 1st binned chip image? +BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 3 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# Overscan, bias, dark, shutter, flat-field +PPIMAGE_TEST METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + CHIP.FITS BOOL TRUE # Save chip-mosaic-ed image? + FPA1.FITS BOOL TRUE # Save 1st binned fpa image? + FPA2.FITS BOOL TRUE # Save 2nd binned fpa image? + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + BIN1.JPEG BOOL TRUE # Save 1st binned jpeg? + BIN2.JPEG BOOL TRUE # Save 2nd binned jpeg? +END + +#################################################################################### +# Need the following in order to turn off dark and shutter for the site-level recipe +#################################################################################### + +# Dark subtraction only +PPIMAGE_D METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Shutter correction only +PPIMAGE_S METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark +PPIMAGE_OBD METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark, shutter +PPIMAGE_OBDS METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark, shutter, flat-field +PPIMAGE_OBDSF METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark, shutter, flat-field, fringe +PPIMAGE_OBDSFR METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction +END + Index: /branches/rel-1-0/ippconfig/ctio_mosaic2/psphot.config =================================================================== --- /branches/rel-1-0/ippconfig/ctio_mosaic2/psphot.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ctio_mosaic2/psphot.config (revision 11645) @@ -0,0 +1,116 @@ + +# these options turn on different inputs and/or outputs +SAVE.BACKMDL BOOL FALSE +SAVE.BACKGND BOOL FALSE +SAVE.BACKSUB BOOL FALSE +SAVE.RESID BOOL TRUE +SAVE.PSF BOOL FALSE +LOAD.PSF BOOL FALSE +SAVE.PLOTS BOOL TRUE + +# the zero point is used to set a basic scale for DVO +# XXX it may not currently be read : double check this (EAM) +ZERO_POINT F32 25.000 # zero point used by DVO + +# these parameter govern how the background is measured +BACKGROUND_XBIN S32 128 # size of background superpixels +BACKGROUND_YBIN S32 128 # size of background superpixels +IMSTATS_NPIX S32 2000 # number of pixels to use for sky estimate boxes: + +SKY_BIAS F32 0.0 # offset applied to measured sky (FOR TESTING) +SKY_FIT_ORDER S32 0 +SKY_FIT_LINEAR BOOL FALSE +SKY_STAT STR FITTED_MEAN # statistic used to measure background +# allowed values for SKY_STAT: +# SAMPLE_MEAN, SAMPLE_MEDIAN, CLIPPED_MEAN, ROBUST_MEAN, ROBUST_QUARTILE, FITTED_MEAN + +# masking parameters (XXX EAM : rework this to use psRegion like ANALYSIS_REGION) +XMIN F32 0 # minimum valid x-coord +XMAX F32 0 # maximum valid x-coord +YMIN F32 0 # minimum valid y-coord +YMAX F32 0 # maximum valid y-coord +SATURATION F32 64000 # saturation level on this chip + +# peak finding +PEAKS_SMOOTH_SIGMA F32 1.0 # smoothing kernel sigma in pixels +PEAKS_SMOOTH_NSIGMA F32 2.0 # smoothing kernel width in sigmas +PEAKS_NSIGMA_LIMIT F32 50.0 # peak significance threshold + +# parameters to control the selection of the peak in the Sx,Sy plane +MOMENTS_SCALE F32 0.1 +MOMENTS_SN_MIN F32 10.0 +MOMENTS_SX_MAX F32 3.0 +MOMENTS_SY_MAX F32 3.0 +MOMENTS_AR_MAX F32 1.5 # maximum axial ratio: 1 / AR < (sx / sy) < AR + +# basic object statistics +SKY_INNER_RADIUS F32 15 # square annulus for local sky measurement +SKY_OUTER_RADIUS F32 25 # square annulus for local sky measurement +PSF_MOMENTS_RADIUS F32 3 # calculate initial source moments with this radius +PSF_SN_LIM F32 50 # minimum S/N for stars used for PSF model +PSF_MAX_NSTARS S32 300 # limit number of stars used for PSF model + +# PSF model parameters : choose the PSF model +# list as many PSF_MODEL options as desired +# PSF_MODEL MULTI +PSF_MODEL STR PS_MODEL_GAUSS +# PSF_MODEL STR PS_MODEL_PGAUSS +# PSF_MODEL STR PS_MODEL_QGAUSS +# PSF_MODEL STR PS_MODEL_TGAUSS ## not well tested, not very successful + +# PSF.TREND.MASK must be a 2D polynomial +# the specified values are ignored but define the active components of the polynomial +PSF.TREND.MASK METADATA + NORDER_X S32 2 # number of x orders + NORDER_Y S32 2 # number of y orders + VAL_X00_Y00 F64 1 # polynomial coefficient + VAL_X01_Y00 F64 1 # polynomial coefficient + VAL_X00_Y01 F64 1 # polynomial coefficient + VAL_X02_Y00 F64 1 # polynomial coefficient + VAL_X01_Y01 F64 1 # polynomial coefficient + VAL_X00_Y02 F64 1 # polynomial coefficient + NELEMENTS S32 6 # number of unmasked components +END # folder for 2D polynomial + +PSF_FIT_RADIUS F32 15.0 # fitting radius for test PSF model +PSF_REF_RADIUS F32 25.0 # aperture magnitudes are scaled via + # curve-of-growth to this radius +# PSF-like source model parameters +PSF_FIT_NSIGMA F32 1.0 # significance for pixel included in fit +PSF_FIT_PADDING F32 2.0 # extra annulus to use for fit +PSF_SHAPE_NSIGMA F32 3.0 # max significance for shape variation +PSF_MIN_SN F32 2.0 # reject objects below this significance +PSF_MAX_CHI F32 50.0 # reject objects worse that this +FULL_FIT_SN_LIM F32 10.0 + +# EXTended source model parameters +EXT_MODEL STR PS_MODEL_PGAUSS +EXT_MIN_SN F32 50.0 # fit galaxies above this S/N limit +EXT_FIT_NSIGMA F32 1 # significance for pixel included in fit +EXT_FIT_PADDING F32 5 # extra annulus to use for fit +EXT_MOMENTS_RADIUS F32 9 + +FITMODE STR BLEND +DEBLEND_PEAK_FRACTION F32 0.1 +DEBLEND_SKY_NSIGMA F32 10.0 + +# APTREND STR NONE, CONSTANT, SKYBIAS, SKYSAT, XY_LIN, SKY_XY_LIN, SKYSAT_XY_LIN, ALL +APTREND STR CONSTANT +AP_MIN_SN F32 25.0 + +# test options +# BREAK_POINT may be one of (PEAKS, MOMENTS, PSFMODEL, ENSEMBLE) +# BREAK_POINT STR MOMENTS +# PEAKS_OUTPUT_FILE STR peaks.dat +# MOMENTS_OUTPUT_FILE STR moments.dat +# ANALYSIS_REGION STR [1000:1600,2800:3400] +# POISSON_ERRORS BOOL TRUE + +# optional parameter to limit the actual analysis to a fraction of the image +# do not uncomment this in the master psphot.config file +# ANALYSIS_REGION STR [1000:1600,2800:3400] + +IGNORE_GROWTH BOOL FALSE +CONSTANT_PHOTOMETRIC_WEIGHTS BOOL TRUE # Should the photometric code [currently only ensemblePSF] refuse to weight each pixel by it's significance? +INTERPOLATE_AP BOOL TRUE +POISSON_ERRORS BOOL TRUE Index: /branches/rel-1-0/ippconfig/dvo.cameras/.cvsignore =================================================================== --- /branches/rel-1-0/ippconfig/dvo.cameras/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippconfig/dvo.cameras/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/ippconfig/dvo.cameras/ISP-Apogee.config =================================================================== --- /branches/rel-1-0/ippconfig/dvo.cameras/ISP-Apogee.config (revision 11645) +++ /branches/rel-1-0/ippconfig/dvo.cameras/ISP-Apogee.config (revision 11645) @@ -0,0 +1,44 @@ + +# location of DVO database tables +CATDIR /data/alala/eugene/isp/catdir + +# keywords used by DVO to interpret the headers + +# used by parse_time to find time-related keywords +MJD-KEYWORD MJD-OBS +DATE-KEYWORD NONE +DATE-MODE NONE +UT-KEYWORD NONE +JD-KEYWORD NONE + +# other keyword abstractions +EXPTIME-KEYWORD EXPTIME +AIRMASS-KEYWORD AIRMASS +CCDNUM-KEYWORD EXTNAME +ST-KEYWORD NONE +OBSERVATORY-LATITUDE NONE +OBSERVATORY-LONGITUDE NONE +SUBPIX_DATAFILE NONE + +# instrumental magnitude range for calibration mode +CAL_INSTMAG_MAX 0 +CAL_INSTMAG_MIN 0 + +# exclude overscan region from the dB image boundaries +XOVERSCAN 0 +YOVERSCAN 0 + +# only upload stars within region; a value of 0 means ignore the limit +ADDSTAR_XMIN 0 +ADDSTAR_XMAX 0 +ADDSTAR_YMIN 0 +ADDSTAR_YMAX 0 + +# exclude stars with SN > SNLIMIT (ADDSTAR_SNLIMIT overrides old name MIN_SN_FSTAT) +ADDSTAR_SNLIMIT 0 + +# correlation radius (arcseconds) +ADDSTAR_RADIUS 20 + +# scaled correlation radius +ADDSTAR_NSIGMA 0 Index: /branches/rel-1-0/ippconfig/dvo.cameras/Makefile.am =================================================================== --- /branches/rel-1-0/ippconfig/dvo.cameras/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippconfig/dvo.cameras/Makefile.am (revision 11645) @@ -0,0 +1,14 @@ + +installdir = $(datadir)/ippconfig/dvo.cameras + +install_files = \ + ISP-Apogee.config + +install_DATA = $(install_files) + +install-data-hook: + chmod 0755 $(installdir) + +EXTRA_DIST = $(install_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippconfig/dvo.photcodes =================================================================== --- /branches/rel-1-0/ippconfig/dvo.photcodes (revision 11645) +++ /branches/rel-1-0/ippconfig/dvo.photcodes (revision 11645) @@ -0,0 +1,416 @@ + +# this file contains the list of valid photometry codes. there should +# be one code for each combination of CCD, filter, and telescope (and +# possibly additional ones if filter transmission reduces with time). +# The code on each line is a unique number, while the name is a +# representative name. The type defines how the photometry is +# treated. There can only be one primary photcode. There should only +# be a few secondary photcodes: these are the calibrated values for +# stars. The dependent photcodes are those which are equivalent to a +# primary or secondary photcode, and are equated after photometric +# solutions. Reference photcodes are for anything without a dynamic +# solution: the reference values are provided externally. + +# NOTE: the photcodes defined below supplement CFHT Elixir values. +# Do not use this photcode file with old CFHT DVO databases. + +# camera ranges used in this file: +# megacam: u: 100-139, g: 200-239, r: 300-339, i: 400-439, z: 500-539 +# cfh12k: B: 140-152, V: 240-252, R: 340-352, I: 440-452, Z: 540-552, NB: 600-693 +# PS1 ISP: g: 3200, r: 3300, i: 3400, z: 3500, y: 3600 +# PS1 TC3: g: 3280-3296, r: 3380-3396, i: 3480-3496, z: 3580-3596, y: 3680-3696, +# PS1 GPC1: g: 3201-3264, r: 3301-3364, i: 3401-3464, z: 3501-3564, y: 3601-3664 + +# external references: 1000 - 2999 + +# code name type zero airmass offset c1 c2 slope equivalence + 1 g pri 0.0 - - 1003 1004 0.016 - 1050 + 2 r sec 0.0 - - 1003 1004 0.008 - 1050 + 3 i sec 0.0 - - 1003 1004 0.028 - 1050 + 4 z sec 0.0 - - 1003 1004 0.107 - 1050 + 5 y sec 0.0 - - - - - - 1050 + + 1010 U_L92 ref - - - - - - - - + 1011 B_L92 ref - - - - - - - - + 1012 V_L92 ref - - - - - - - - + 1013 R_L92 ref - - - - - - - - + 1014 I_L92 ref - - - - - - - - + + 1020 U_STET ref - - - - - - - - + 1021 B_STET ref - - - - - - - - + 1022 V_STET ref - - - - - - - - + 1023 R_STET ref - - - - - - - - + 1024 I_STET ref - - - - - - - - + + 1051 u_SDSS ref - - - - - - - - + 1052 g_SDSS ref - - - - - - - - + 1053 r_SDSS ref - - - - - - - - + 1054 i_SDSS ref - - - - - - - - + 1055 z_SDSS ref - - - - - - - - + + 1000 USNO_BLUE ref - - - - - - - - + 1001 USNO_RED ref - - - - - - - - + 1002 USNO_J ref - - - - - - - - + 1003 USNO_F ref - - - - - - - - + 1004 USNO_N ref - - - - - - - - + + 1100 GSC ref - - - - - - - - + + 2020 TYCHO_B ref - - - - - - - - + 2021 TYCHO_V ref - - - - - - - - + + 2011 2MASS_J ref - - - - - - - - + 2012 2MASS_H ref - - - - - - - - + 2013 2MASS_K ref - - - - - - - - + +### megacam photcodes: + 100 MEGACAM.u.00 dep 25.280 -0.35 - - - - - 1 + 101 MEGACAM.u.01 dep 25.280 -0.35 - - - - - 1 + 102 MEGACAM.u.02 dep 25.280 -0.35 - - - - - 1 + 103 MEGACAM.u.03 dep 25.280 -0.35 - - - - - 1 + 104 MEGACAM.u.04 dep 25.280 -0.35 - - - - - 1 + 105 MEGACAM.u.05 dep 25.280 -0.35 - - - - - 1 + 106 MEGACAM.u.06 dep 25.280 -0.35 - - - - - 1 + 107 MEGACAM.u.07 dep 25.280 -0.35 - - - - - 1 + 108 MEGACAM.u.08 dep 25.280 -0.35 - - - - - 1 + 109 MEGACAM.u.09 dep 25.280 -0.35 - - - - - 1 + 110 MEGACAM.u.10 dep 25.280 -0.35 - - - - - 1 + 111 MEGACAM.u.11 dep 25.280 -0.35 - - - - - 1 + 112 MEGACAM.u.12 dep 25.280 -0.35 - - - - - 1 + 113 MEGACAM.u.13 dep 25.280 -0.35 - - - - - 1 + 114 MEGACAM.u.14 dep 25.280 -0.35 - - - - - 1 + 115 MEGACAM.u.15 dep 25.280 -0.35 - - - - - 1 + 116 MEGACAM.u.16 dep 25.280 -0.35 - - - - - 1 + 117 MEGACAM.u.17 dep 25.280 -0.35 - - - - - 1 + 118 MEGACAM.u.18 dep 25.280 -0.35 - - - - - 1 + 119 MEGACAM.u.19 dep 25.280 -0.35 - - - - - 1 + 120 MEGACAM.u.20 dep 25.280 -0.35 - - - - - 1 + 121 MEGACAM.u.21 dep 25.280 -0.35 - - - - - 1 + 122 MEGACAM.u.22 dep 25.280 -0.35 - - - - - 1 + 123 MEGACAM.u.23 dep 25.280 -0.35 - - - - - 1 + 124 MEGACAM.u.24 dep 25.280 -0.35 - - - - - 1 + 125 MEGACAM.u.25 dep 25.280 -0.35 - - - - - 1 + 126 MEGACAM.u.26 dep 25.280 -0.35 - - - - - 1 + 127 MEGACAM.u.27 dep 25.280 -0.35 - - - - - 1 + 128 MEGACAM.u.28 dep 25.280 -0.35 - - - - - 1 + 129 MEGACAM.u.29 dep 25.280 -0.35 - - - - - 1 + 130 MEGACAM.u.30 dep 25.280 -0.35 - - - - - 1 + 131 MEGACAM.u.31 dep 25.280 -0.35 - - - - - 1 + 132 MEGACAM.u.32 dep 25.280 -0.35 - - - - - 1 + 133 MEGACAM.u.33 dep 25.280 -0.35 - - - - - 1 + 134 MEGACAM.u.34 dep 25.280 -0.35 - - - - - 1 + 135 MEGACAM.u.35 dep 25.280 -0.35 - - - - - 1 + 136 MEGACAM.u.36 dep 25.280 -0.35 - - - - - 1 + 137 MEGACAM.u.37 dep 25.280 -0.35 - - - - - 1 + 138 MEGACAM.u.38 dep 25.280 -0.35 - - - - - 1 + 139 MEGACAM.u.39 dep 25.280 -0.35 - - - - - 1 + + 200 MEGACAM.g.00 dep 26.460 -0.15 - - - - - 2 + 201 MEGACAM.g.01 dep 26.460 -0.15 - - - - - 2 + 202 MEGACAM.g.02 dep 26.460 -0.15 - - - - - 2 + 203 MEGACAM.g.03 dep 26.460 -0.15 - - - - - 2 + 204 MEGACAM.g.04 dep 26.460 -0.15 - - - - - 2 + 205 MEGACAM.g.05 dep 26.460 -0.15 - - - - - 2 + 206 MEGACAM.g.06 dep 26.460 -0.15 - - - - - 2 + 207 MEGACAM.g.07 dep 26.460 -0.15 - - - - - 2 + 208 MEGACAM.g.08 dep 26.460 -0.15 - - - - - 2 + 209 MEGACAM.g.09 dep 26.460 -0.15 - - - - - 2 + 210 MEGACAM.g.10 dep 26.460 -0.15 - - - - - 2 + 211 MEGACAM.g.11 dep 26.460 -0.15 - - - - - 2 + 212 MEGACAM.g.12 dep 26.460 -0.15 - - - - - 2 + 213 MEGACAM.g.13 dep 26.460 -0.15 - - - - - 2 + 214 MEGACAM.g.14 dep 26.460 -0.15 - - - - - 2 + 215 MEGACAM.g.15 dep 26.460 -0.15 - - - - - 2 + 216 MEGACAM.g.16 dep 26.460 -0.15 - - - - - 2 + 217 MEGACAM.g.17 dep 26.460 -0.15 - - - - - 2 + 218 MEGACAM.g.18 dep 26.460 -0.15 - - - - - 2 + 219 MEGACAM.g.19 dep 26.460 -0.15 - - - - - 2 + 220 MEGACAM.g.20 dep 26.460 -0.15 - - - - - 2 + 221 MEGACAM.g.21 dep 26.460 -0.15 - - - - - 2 + 222 MEGACAM.g.22 dep 26.460 -0.15 - - - - - 2 + 223 MEGACAM.g.23 dep 26.460 -0.15 - - - - - 2 + 224 MEGACAM.g.24 dep 26.460 -0.15 - - - - - 2 + 225 MEGACAM.g.25 dep 26.460 -0.15 - - - - - 2 + 226 MEGACAM.g.26 dep 26.460 -0.15 - - - - - 2 + 227 MEGACAM.g.27 dep 26.460 -0.15 - - - - - 2 + 228 MEGACAM.g.28 dep 26.460 -0.15 - - - - - 2 + 229 MEGACAM.g.29 dep 26.460 -0.15 - - - - - 2 + 230 MEGACAM.g.30 dep 26.460 -0.15 - - - - - 2 + 231 MEGACAM.g.31 dep 26.460 -0.15 - - - - - 2 + 232 MEGACAM.g.32 dep 26.460 -0.15 - - - - - 2 + 233 MEGACAM.g.33 dep 26.460 -0.15 - - - - - 2 + 234 MEGACAM.g.34 dep 26.460 -0.15 - - - - - 2 + 235 MEGACAM.g.35 dep 26.460 -0.15 - - - - - 2 + 236 MEGACAM.g.36 dep 26.460 -0.15 - - - - - 2 + 237 MEGACAM.g.37 dep 26.460 -0.15 - - - - - 2 + 238 MEGACAM.g.38 dep 26.460 -0.15 - - - - - 2 + 239 MEGACAM.g.39 dep 26.460 -0.15 - - - - - 2 + + 300 MEGACAM.r.00 dep 25.978 -0.10 - - - - - 3 + 301 MEGACAM.r.01 dep 25.978 -0.10 - - - - - 3 + 302 MEGACAM.r.02 dep 25.978 -0.10 - - - - - 3 + 303 MEGACAM.r.03 dep 25.978 -0.10 - - - - - 3 + 304 MEGACAM.r.04 dep 25.978 -0.10 - - - - - 3 + 305 MEGACAM.r.05 dep 25.978 -0.10 - - - - - 3 + 306 MEGACAM.r.06 dep 25.978 -0.10 - - - - - 3 + 307 MEGACAM.r.07 dep 25.978 -0.10 - - - - - 3 + 308 MEGACAM.r.08 dep 25.978 -0.10 - - - - - 3 + 309 MEGACAM.r.09 dep 25.978 -0.10 - - - - - 3 + 310 MEGACAM.r.10 dep 25.978 -0.10 - - - - - 3 + 311 MEGACAM.r.11 dep 25.978 -0.10 - - - - - 3 + 312 MEGACAM.r.12 dep 25.978 -0.10 - - - - - 3 + 313 MEGACAM.r.13 dep 25.978 -0.10 - - - - - 3 + 314 MEGACAM.r.14 dep 25.978 -0.10 - - - - - 3 + 315 MEGACAM.r.15 dep 25.978 -0.10 - - - - - 3 + 316 MEGACAM.r.16 dep 25.978 -0.10 - - - - - 3 + 317 MEGACAM.r.17 dep 25.978 -0.10 - - - - - 3 + 318 MEGACAM.r.18 dep 25.978 -0.10 - - - - - 3 + 319 MEGACAM.r.19 dep 25.978 -0.10 - - - - - 3 + 320 MEGACAM.r.20 dep 25.978 -0.10 - - - - - 3 + 321 MEGACAM.r.21 dep 25.978 -0.10 - - - - - 3 + 322 MEGACAM.r.22 dep 25.978 -0.10 - - - - - 3 + 323 MEGACAM.r.23 dep 25.978 -0.10 - - - - - 3 + 324 MEGACAM.r.24 dep 25.978 -0.10 - - - - - 3 + 325 MEGACAM.r.25 dep 25.978 -0.10 - - - - - 3 + 326 MEGACAM.r.26 dep 25.978 -0.10 - - - - - 3 + 327 MEGACAM.r.27 dep 25.978 -0.10 - - - - - 3 + 328 MEGACAM.r.28 dep 25.978 -0.10 - - - - - 3 + 329 MEGACAM.r.29 dep 25.978 -0.10 - - - - - 3 + 330 MEGACAM.r.30 dep 25.978 -0.10 - - - - - 3 + 331 MEGACAM.r.31 dep 25.978 -0.10 - - - - - 3 + 332 MEGACAM.r.32 dep 25.978 -0.10 - - - - - 3 + 333 MEGACAM.r.33 dep 25.978 -0.10 - - - - - 3 + 334 MEGACAM.r.34 dep 25.978 -0.10 - - - - - 3 + 335 MEGACAM.r.35 dep 25.978 -0.10 - - - - - 3 + 336 MEGACAM.r.36 dep 25.978 -0.10 - - - - - 3 + 337 MEGACAM.r.37 dep 25.978 -0.10 - - - - - 3 + 338 MEGACAM.r.38 dep 25.978 -0.10 - - - - - 3 + 339 MEGACAM.r.39 dep 25.978 -0.10 - - - - - 3 + + 400 MEGACAM.i.00 dep 25.744 -0.04 - - - - - 4 + 401 MEGACAM.i.01 dep 25.744 -0.04 - - - - - 4 + 402 MEGACAM.i.02 dep 25.744 -0.04 - - - - - 4 + 403 MEGACAM.i.03 dep 25.744 -0.04 - - - - - 4 + 404 MEGACAM.i.04 dep 25.744 -0.04 - - - - - 4 + 405 MEGACAM.i.05 dep 25.744 -0.04 - - - - - 4 + 406 MEGACAM.i.06 dep 25.744 -0.04 - - - - - 4 + 407 MEGACAM.i.07 dep 25.744 -0.04 - - - - - 4 + 408 MEGACAM.i.08 dep 25.744 -0.04 - - - - - 4 + 409 MEGACAM.i.09 dep 25.744 -0.04 - - - - - 4 + 410 MEGACAM.i.10 dep 25.744 -0.04 - - - - - 4 + 411 MEGACAM.i.11 dep 25.744 -0.04 - - - - - 4 + 412 MEGACAM.i.12 dep 25.744 -0.04 - - - - - 4 + 413 MEGACAM.i.13 dep 25.744 -0.04 - - - - - 4 + 414 MEGACAM.i.14 dep 25.744 -0.04 - - - - - 4 + 415 MEGACAM.i.15 dep 25.744 -0.04 - - - - - 4 + 416 MEGACAM.i.16 dep 25.744 -0.04 - - - - - 4 + 417 MEGACAM.i.17 dep 25.744 -0.04 - - - - - 4 + 418 MEGACAM.i.18 dep 25.744 -0.04 - - - - - 4 + 419 MEGACAM.i.19 dep 25.744 -0.04 - - - - - 4 + 420 MEGACAM.i.20 dep 25.744 -0.04 - - - - - 4 + 421 MEGACAM.i.21 dep 25.744 -0.04 - - - - - 4 + 422 MEGACAM.i.22 dep 25.744 -0.04 - - - - - 4 + 423 MEGACAM.i.23 dep 25.744 -0.04 - - - - - 4 + 424 MEGACAM.i.24 dep 25.744 -0.04 - - - - - 4 + 425 MEGACAM.i.25 dep 25.744 -0.04 - - - - - 4 + 426 MEGACAM.i.26 dep 25.744 -0.04 - - - - - 4 + 427 MEGACAM.i.27 dep 25.744 -0.04 - - - - - 4 + 428 MEGACAM.i.28 dep 25.744 -0.04 - - - - - 4 + 429 MEGACAM.i.29 dep 25.744 -0.04 - - - - - 4 + 430 MEGACAM.i.30 dep 25.744 -0.04 - - - - - 4 + 431 MEGACAM.i.31 dep 25.744 -0.04 - - - - - 4 + 432 MEGACAM.i.32 dep 25.744 -0.04 - - - - - 4 + 433 MEGACAM.i.33 dep 25.744 -0.04 - - - - - 4 + 434 MEGACAM.i.34 dep 25.744 -0.04 - - - - - 4 + 435 MEGACAM.i.35 dep 25.744 -0.04 - - - - - 4 + 436 MEGACAM.i.36 dep 25.744 -0.04 - - - - - 4 + 437 MEGACAM.i.37 dep 25.744 -0.04 - - - - - 4 + 438 MEGACAM.i.38 dep 25.744 -0.04 - - - - - 4 + 439 MEGACAM.i.39 dep 25.744 -0.04 - - - - - 4 + + 500 MEGACAM.z.00 dep 24.800 -0.03 - - - - - 5 + 501 MEGACAM.z.01 dep 24.800 -0.03 - - - - - 5 + 502 MEGACAM.z.02 dep 24.800 -0.03 - - - - - 5 + 503 MEGACAM.z.03 dep 24.800 -0.03 - - - - - 5 + 504 MEGACAM.z.04 dep 24.800 -0.03 - - - - - 5 + 505 MEGACAM.z.05 dep 24.800 -0.03 - - - - - 5 + 506 MEGACAM.z.06 dep 24.800 -0.03 - - - - - 5 + 507 MEGACAM.z.07 dep 24.800 -0.03 - - - - - 5 + 508 MEGACAM.z.08 dep 24.800 -0.03 - - - - - 5 + 509 MEGACAM.z.09 dep 24.800 -0.03 - - - - - 5 + 510 MEGACAM.z.10 dep 24.800 -0.03 - - - - - 5 + 511 MEGACAM.z.11 dep 24.800 -0.03 - - - - - 5 + 512 MEGACAM.z.12 dep 24.800 -0.03 - - - - - 5 + 513 MEGACAM.z.13 dep 24.800 -0.03 - - - - - 5 + 514 MEGACAM.z.14 dep 24.800 -0.03 - - - - - 5 + 515 MEGACAM.z.15 dep 24.800 -0.03 - - - - - 5 + 516 MEGACAM.z.16 dep 24.800 -0.03 - - - - - 5 + 517 MEGACAM.z.17 dep 24.800 -0.03 - - - - - 5 + 518 MEGACAM.z.18 dep 24.800 -0.03 - - - - - 5 + 519 MEGACAM.z.19 dep 24.800 -0.03 - - - - - 5 + 520 MEGACAM.z.20 dep 24.800 -0.03 - - - - - 5 + 521 MEGACAM.z.21 dep 24.800 -0.03 - - - - - 5 + 522 MEGACAM.z.22 dep 24.800 -0.03 - - - - - 5 + 523 MEGACAM.z.23 dep 24.800 -0.03 - - - - - 5 + 524 MEGACAM.z.24 dep 24.800 -0.03 - - - - - 5 + 525 MEGACAM.z.25 dep 24.800 -0.03 - - - - - 5 + 526 MEGACAM.z.26 dep 24.800 -0.03 - - - - - 5 + 527 MEGACAM.z.27 dep 24.800 -0.03 - - - - - 5 + 528 MEGACAM.z.28 dep 24.800 -0.03 - - - - - 5 + 529 MEGACAM.z.29 dep 24.800 -0.03 - - - - - 5 + 530 MEGACAM.z.30 dep 24.800 -0.03 - - - - - 5 + 531 MEGACAM.z.31 dep 24.800 -0.03 - - - - - 5 + 532 MEGACAM.z.32 dep 24.800 -0.03 - - - - - 5 + 533 MEGACAM.z.33 dep 24.800 -0.03 - - - - - 5 + 534 MEGACAM.z.34 dep 24.800 -0.03 - - - - - 5 + 535 MEGACAM.z.35 dep 24.800 -0.03 - - - - - 5 + 536 MEGACAM.z.36 dep 24.800 -0.03 - - - - - 5 + 537 MEGACAM.z.37 dep 24.800 -0.03 - - - - - 5 + 538 MEGACAM.z.38 dep 24.800 -0.03 - - - - - 5 + 539 MEGACAM.z.39 dep 24.800 -0.03 - - - - - 5 + +# cfh12k photcodes + 140 CFH12K.B.00 dep 26.000 -0.15 - - - - - 1 + 141 CFH12K.B.01 dep 26.000 -0.15 - - - - - 1 + 142 CFH12K.B.02 dep 26.000 -0.15 - - - - - 1 + 143 CFH12K.B.03 dep 26.000 -0.15 - - - - - 1 + 144 CFH12K.B.04 dep 26.000 -0.15 - - - - - 1 + 145 CFH12K.B.05 dep 26.000 -0.15 - - - - - 1 + 146 CFH12K.B.06 dep 26.000 -0.15 - - - - - 1 + 147 CFH12K.B.07 dep 26.000 -0.15 - - - - - 1 + 148 CFH12K.B.08 dep 26.000 -0.15 - - - - - 1 + 149 CFH12K.B.09 dep 26.000 -0.15 - - - - - 1 + 150 CFH12K.B.10 dep 26.000 -0.15 - - - - - 1 + 151 CFH12K.B.11 dep 26.000 -0.15 - - - - - 1 + + 240 CFH12K.V.00 dep 26.150 -0.12 - - - - - 2 + 241 CFH12K.V.01 dep 26.150 -0.12 - - - - - 2 + 242 CFH12K.V.02 dep 26.150 -0.12 - - - - - 2 + 243 CFH12K.V.03 dep 26.150 -0.12 - - - - - 2 + 244 CFH12K.V.04 dep 26.150 -0.12 - - - - - 2 + 245 CFH12K.V.05 dep 26.150 -0.12 - - - - - 2 + 246 CFH12K.V.06 dep 26.150 -0.12 - - - - - 2 + 247 CFH12K.V.07 dep 26.150 -0.12 - - - - - 2 + 248 CFH12K.V.08 dep 26.150 -0.12 - - - - - 2 + 249 CFH12K.V.09 dep 26.150 -0.12 - - - - - 2 + 250 CFH12K.V.10 dep 26.150 -0.12 - - - - - 2 + 251 CFH12K.V.11 dep 26.150 -0.12 - - - - - 2 + + 340 CFH12K.R.00 dep 26.190 -0.09 - - - - - 3 + 341 CFH12K.R.01 dep 26.190 -0.09 - - - - - 3 + 342 CFH12K.R.02 dep 26.190 -0.09 - - - - - 3 + 343 CFH12K.R.03 dep 26.190 -0.09 - - - - - 3 + 344 CFH12K.R.04 dep 26.190 -0.09 - - - - - 3 + 345 CFH12K.R.05 dep 26.190 -0.09 - - - - - 3 + 346 CFH12K.R.06 dep 26.190 -0.09 - - - - - 3 + 347 CFH12K.R.07 dep 26.190 -0.09 - - - - - 3 + 348 CFH12K.R.08 dep 26.190 -0.09 - - - - - 3 + 349 CFH12K.R.09 dep 26.190 -0.09 - - - - - 3 + 350 CFH12K.R.10 dep 26.190 -0.09 - - - - - 3 + 351 CFH12K.R.11 dep 26.190 -0.09 - - - - - 3 + + 440 CFH12K.I.00 dep 26.135 -0.04 - - - - - 4 + 441 CFH12K.I.01 dep 26.135 -0.04 - - - - - 4 + 442 CFH12K.I.02 dep 26.135 -0.04 - - - - - 4 + 443 CFH12K.I.03 dep 26.135 -0.04 - - - - - 4 + 444 CFH12K.I.04 dep 26.135 -0.04 - - - - - 4 + 445 CFH12K.I.05 dep 26.135 -0.04 - - - - - 4 + 446 CFH12K.I.06 dep 26.135 -0.04 - - - - - 4 + 447 CFH12K.I.07 dep 26.135 -0.04 - - - - - 4 + 448 CFH12K.I.08 dep 26.135 -0.04 - - - - - 4 + 449 CFH12K.I.09 dep 26.135 -0.04 - - - - - 4 + 450 CFH12K.I.10 dep 26.135 -0.04 - - - - - 4 + 451 CFH12K.I.11 dep 26.135 -0.04 - - - - - 4 + + 540 CFH12K.Z.00 dep 25.83 -0.03 - - - - - 5 + 541 CFH12K.Z.01 dep 25.83 -0.03 - - - - - 5 + 542 CFH12K.Z.02 dep 25.83 -0.03 - - - - - 5 + 543 CFH12K.Z.03 dep 25.83 -0.03 - - - - - 5 + 544 CFH12K.Z.04 dep 25.83 -0.03 - - - - - 5 + 545 CFH12K.Z.05 dep 25.83 -0.03 - - - - - 5 + 546 CFH12K.Z.06 dep 25.83 -0.03 - - - - - 5 + 547 CFH12K.Z.07 dep 25.83 -0.03 - - - - - 5 + 548 CFH12K.Z.08 dep 25.83 -0.03 - - - - - 5 + 549 CFH12K.Z.09 dep 25.83 -0.03 - - - - - 5 + 550 CFH12K.Z.10 dep 25.83 -0.03 - - - - - 5 + 551 CFH12K.Z.11 dep 25.83 -0.03 - - - - - 5 + + 600 CFH12K.Ha.00 dep 25.000 0.04 - - - - - 5 + 601 CFH12K.Ha.01 dep 25.000 0.04 - - - - - 5 + 602 CFH12K.Ha.02 dep 25.000 0.04 - - - - - 5 + 603 CFH12K.Ha.03 dep 25.000 0.04 - - - - - 5 + 604 CFH12K.Ha.04 dep 25.000 0.04 - - - - - 5 + 605 CFH12K.Ha.05 dep 25.000 0.04 - - - - - 5 + 606 CFH12K.Ha.06 dep 25.000 0.04 - - - - - 5 + 607 CFH12K.Ha.07 dep 25.000 0.04 - - - - - 5 + 608 CFH12K.Ha.08 dep 25.000 0.04 - - - - - 5 + 609 CFH12K.Ha.09 dep 25.000 0.04 - - - - - 5 + 610 CFH12K.Ha.10 dep 25.000 0.04 - - - - - 5 + 611 CFH12K.Ha.11 dep 25.000 0.04 - - - - - 5 + + 612 CFH12K.HaOff.00 dep 25.000 0.04 - - - - - 5 + 613 CFH12K.HaOff.01 dep 25.000 0.04 - - - - - 5 + 614 CFH12K.HaOff.02 dep 25.000 0.04 - - - - - 5 + 615 CFH12K.HaOff.03 dep 25.000 0.04 - - - - - 5 + 616 CFH12K.HaOff.04 dep 25.000 0.04 - - - - - 5 + 617 CFH12K.HaOff.05 dep 25.000 0.04 - - - - - 5 + 618 CFH12K.HaOff.06 dep 25.000 0.04 - - - - - 5 + 619 CFH12K.HaOff.07 dep 25.000 0.04 - - - - - 5 + 620 CFH12K.HaOff.08 dep 25.000 0.04 - - - - - 5 + 621 CFH12K.HaOff.09 dep 25.000 0.04 - - - - - 5 + 622 CFH12K.HaOff.10 dep 25.000 0.04 - - - - - 5 + 623 CFH12K.HaOff.11 dep 25.000 0.04 - - - - - 5 + 624 CFH12K.CN.00 dep 25.000 0.04 - - - - - 5 + 625 CFH12K.CN.01 dep 25.000 0.04 - - - - - 5 + 626 CFH12K.CN.02 dep 25.000 0.04 - - - - - 5 + 627 CFH12K.CN.03 dep 25.000 0.04 - - - - - 5 + 628 CFH12K.CN.04 dep 25.000 0.04 - - - - - 5 + 629 CFH12K.CN.05 dep 25.000 0.04 - - - - - 5 + 630 CFH12K.CN.06 dep 25.000 0.04 - - - - - 5 + 631 CFH12K.CN.07 dep 25.000 0.04 - - - - - 5 + 632 CFH12K.CN.08 dep 25.000 0.04 - - - - - 5 + 633 CFH12K.CN.09 dep 25.000 0.04 - - - - - 5 + 634 CFH12K.CN.10 dep 25.000 0.04 - - - - - 5 + 635 CFH12K.CN.11 dep 25.000 0.04 - - - - - 5 + 636 CFH12K.TiO.00 dep 25.000 0.04 - - - - - 5 + 637 CFH12K.TiO.01 dep 25.000 0.04 - - - - - 5 + 638 CFH12K.TiO.02 dep 25.000 0.04 - - - - - 5 + 639 CFH12K.TiO.03 dep 25.000 0.04 - - - - - 5 + 640 CFH12K.TiO.04 dep 25.000 0.04 - - - - - 5 + 641 CFH12K.TiO.05 dep 25.000 0.04 - - - - - 5 + 642 CFH12K.TiO.06 dep 25.000 0.04 - - - - - 5 + 643 CFH12K.TiO.07 dep 25.000 0.04 - - - - - 5 + 644 CFH12K.TiO.08 dep 25.000 0.04 - - - - - 5 + 645 CFH12K.TiO.09 dep 25.000 0.04 - - - - - 5 + 646 CFH12K.TiO.10 dep 25.000 0.04 - - - - - 5 + 647 CFH12K.TiO.11 dep 25.000 0.04 - - - - - 5 + 648 CFH12K.NB920.00 dep 25.000 0.04 - - - - - 5 + 649 CFH12K.NB920.01 dep 25.000 0.04 - - - - - 5 + 650 CFH12K.NB920.02 dep 25.000 0.04 - - - - - 5 + 651 CFH12K.NB920.03 dep 25.000 0.04 - - - - - 5 + 652 CFH12K.NB920.04 dep 25.000 0.04 - - - - - 5 + 653 CFH12K.NB920.05 dep 25.000 0.04 - - - - - 5 + 654 CFH12K.NB920.06 dep 25.000 0.04 - - - - - 5 + 655 CFH12K.NB920.07 dep 25.000 0.04 - - - - - 5 + 678 CFH12K.NB920.08 dep 25.000 0.04 - - - - - 5 + 679 CFH12K.NB920.09 dep 25.000 0.04 - - - - - 5 + 680 CFH12K.NB920.10 dep 25.000 0.04 - - - - - 5 + 681 CFH12K.NB920.11 dep 25.000 0.04 - - - - - 5 + 682 CFH12K.B2F.00 dep 25.000 0.04 - - - - - 5 + 683 CFH12K.B2F.01 dep 25.000 0.04 - - - - - 5 + 684 CFH12K.B2F.02 dep 25.000 0.04 - - - - - 5 + 685 CFH12K.B2F.03 dep 25.000 0.04 - - - - - 5 + 686 CFH12K.B2F.04 dep 25.000 0.04 - - - - - 5 + 687 CFH12K.B2F.05 dep 25.000 0.04 - - - - - 5 + 688 CFH12K.B2F.06 dep 25.000 0.04 - - - - - 5 + 689 CFH12K.B2F.07 dep 25.000 0.04 - - - - - 5 + 690 CFH12K.B2F.08 dep 25.000 0.04 - - - - - 5 + 691 CFH12K.B2F.09 dep 25.000 0.04 - - - - - 5 + 692 CFH12K.B2F.10 dep 25.000 0.04 - - - - - 5 + 693 CFH12K.B2F.11 dep 25.000 0.04 - - - - - 5 + + 3200 ISP-Apogee-01.g dep 19.25 0.00 - - - - - 1 + 3300 ISP-Apogee-01.r dep 19.15 0.00 - - - - - 2 + 3400 ISP-Apogee-01.i dep 18.50 0.00 - - - - - 3 + 3500 ISP-Apogee-01.z dep 17.25 0.00 - - - - - 4 + 3600 ISP-Apogee-01.y dep 14.15 0.00 - - - - - 5 Index: /branches/rel-1-0/ippconfig/dvo.site.in =================================================================== --- /branches/rel-1-0/ippconfig/dvo.site.in (revision 11645) +++ /branches/rel-1-0/ippconfig/dvo.site.in (revision 11645) @@ -0,0 +1,30 @@ + +CONFDIR @IPP_CONFIG_DIR@ +CAMERA default + +# location of DVO database tables +CATDIR default + +# location of possible data sources +2MASS_DIR_AS NONE +2MASS_DIR_DR2 NONE +GSCDIR NONE +USNO_A_DIR NONE +USNO_B_DIR NONE +TYCHO_DIR NONE + +PHOTCODE_FILE $CONFDIR/dvo.photcodes +GSCFILE $CONFDIR/GSCregions.tbl +CATMODE MEF +CATFORMAT ELIXIR +SKY_DEPTH +# SKY_TABLE this may be used to override GSCFILE +ZERO_PT 25.0 + +# access control for client/server addstar mode +# PASSWORD +# HOSTNAME +# VALID_IP + +# load the camera-specific configuration files +input $CONFDIR/dvo.cameras/$CAMERA.config Index: /branches/rel-1-0/ippconfig/gpc1/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/gpc1/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/gpc1/camera.config (revision 11645) @@ -0,0 +1,82 @@ +# Camera configuration file for GPC1: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR gpc1/format_raw.config +END + + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + 64 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY01 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY02 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY03 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY04 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY05 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY06 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY07 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY10 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY11 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY12 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY13 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY14 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY15 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY16 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY17 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY20 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY21 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY22 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY23 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY24 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY25 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY26 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY27 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY30 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY31 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY32 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY33 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY34 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY35 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY36 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY37 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY40 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY41 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY42 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY43 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY44 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY45 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY46 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY47 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY50 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY51 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY52 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY53 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY54 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY55 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY56 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY57 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY60 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY61 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY62 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY63 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY64 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY65 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY66 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY67 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY70 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY71 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY72 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY73 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY74 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY75 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY76 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +# XY77 STR xy00 xy01 xy02 xy03 xy04 xy05 xy06 xy07 xy10 xy11 xy12 xy13 xy14 xy15 xy16 xy17 xy20 xy21 xy22 xy23 xy24 xy25 xy26 xy27 xy30 xy31 xy32 xy33 xy34 xy35 xy36 xy37 xy40 xy41 xy42 xy43 xy44 xy45 xy46 xy47 xy50 xy51 xy52 xy53 xy54 xy55 xy56 xy57 xy60 xy61 xy62 xy63 xy64 xy65 xy66 xy67 xy70 xy71 xy72 xy73 xy74 xy75 xy76 xy77 +END + + +# Recipe options +RECIPES METADATA + PHASE2 STR phase2.config # Phase 2 recipe details +# PSPHOT STR psphot.config # psphot details +END Index: /branches/rel-1-0/ippconfig/gpc1/format_raw.config =================================================================== --- /branches/rel-1-0/ippconfig/gpc1/format_raw.config (revision 11645) +++ /branches/rel-1-0/ippconfig/gpc1/format_raw.config (revision 11645) @@ -0,0 +1,175 @@ +# The raw GPC data comes off the telescope with each of the chips stored in separate files + +# How to identify this type +RULE METADATA +# TELESCOP STR PS1 +# DETECTOR STR GPC1 + EXTEND BOOL T + NEXTEND S32 64 + NAMPS S32 64 +END + +# How to read this data +FILE METADATA + PHU STR CHIP # The FITS file represents a single chip + EXTENSIONS STR CELL # The extensions represent cells + FPA.NAME STR NAMPS # A PHU keyword for unique identifier within the hierarchy level + CHIP.NAME STR NAMPS # An extension keyword for unique identifier +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, type + xy00 STR xy00:pitch10u + xy01 STR xy01:pitch10u + xy02 STR xy02:pitch10u + xy03 STR xy03:pitch10u + xy04 STR xy04:pitch10u + xy05 STR xy05:pitch10u + xy06 STR xy06:pitch10u + xy07 STR xy07:pitch10u + xy10 STR xy10:pitch10u + xy11 STR xy11:pitch10u + xy12 STR xy12:pitch10u + xy13 STR xy13:pitch10u + xy14 STR xy14:pitch10u + xy15 STR xy15:pitch10u + xy16 STR xy16:pitch10u + xy17 STR xy17:pitch10u + xy20 STR xy20:pitch10u + xy21 STR xy21:pitch10u + xy22 STR xy22:pitch10u + xy23 STR xy23:pitch10u + xy24 STR xy24:pitch10u + xy25 STR xy25:pitch10u + xy26 STR xy26:pitch10u + xy27 STR xy27:pitch10u + xy30 STR xy30:pitch10u + xy31 STR xy31:pitch10u + xy32 STR xy32:pitch10u + xy33 STR xy33:pitch10u + xy34 STR xy34:pitch10u + xy35 STR xy35:pitch10u + xy36 STR xy36:pitch10u + xy37 STR xy37:pitch10u + xy40 STR xy40:pitch10u + xy41 STR xy41:pitch10u + xy42 STR xy42:pitch10u + xy43 STR xy43:pitch10u + xy44 STR xy44:pitch10u + xy45 STR xy45:pitch10u + xy46 STR xy46:pitch10u + xy47 STR xy47:pitch10u + xy50 STR xy50:pitch10u + xy51 STR xy51:pitch10u + xy52 STR xy52:pitch10u + xy53 STR xy53:pitch10u + xy54 STR xy54:pitch10u + xy55 STR xy55:pitch10u + xy56 STR xy56:pitch10u + xy57 STR xy57:pitch10u + xy60 STR xy60:pitch10u + xy61 STR xy61:pitch10u + xy62 STR xy62:pitch10u + xy63 STR xy63:pitch10u + xy64 STR xy64:pitch10u + xy65 STR xy65:pitch10u + xy66 STR xy66:pitch10u + xy67 STR xy67:pitch10u + xy70 STR xy70:pitch10u + xy71 STR xy71:pitch10u + xy72 STR xy72:pitch10u + xy73 STR xy73:pitch10u + xy74 STR xy74:pitch10u + xy75 STR xy75:pitch10u + xy76 STR xy76:pitch10u + xy77 STR xy77:pitch10u +END + +# Specify the cell data +CELLS METADATA + pitch10u METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [575:606,1:594] + CELL.TRIMSEC STR [1:574,1:594] + # CELL.BIASSEC STR BIASSEC + # CELL.TRIMSEC STR DATASEC + END + + # This is just in here for fun + pitch12u METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [1:10,1:512];[523:574,1:512] + CELL.TRIMSEC STR [11:522,1:512] + # CELL.BIASSEC STR BIASSEC + # CELL.TRIMSEC STR TRIMSEC + END +END + + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + CELL.XBIN STR CCDSUM + CELL.YBIN STR CCDSUM + CELL.X0 STR IMNPIX1 + CELL.Y0 STR IMNPIX2 + CELL.XPARITY STR LTM1_1 + CELL.YPARITY STR LTM2_2 + CELL.SATURATION STR SATURATE +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.AIRMASS F32 0.0 + FPA.FILTER STR NONE + FPA.POSANGLE F32 0.0 + FPA.RA STR 0:0:0 + FPA.DEC STR 0:0:0 + FPA.RADECSYS STR ICRS + FPA.NAME S32 12345 + CELL.EXPOSURE F32 0.0 + CELL.DARKTIME F32 0.0 + CELL.GAIN F32 1.0 + CELL.READNOISE F32 0.0 + CELL.READDIR S32 1 + CELL.BAD S32 0 + CELL.TIMESYS STR UTC + CELL.TIME STR 2005-11-23T12:34:56.78 +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP,CELL +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP,CELL + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). + +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.BINNING STR TOGETHER + CELL.X0 STR FORTRAN + CELL.Y0 STR FORTRAN +END + +# Recipe options +RECIPES METADATA + PHASE2 STR phase2.config # Phase 2 recipe details +END + +# How to get the supplementary stuff: mask and weight +SUPPLEMENTARY METADATA + MASK.SOURCE STR FILE # Source type for mask: EXT | FILE + MASK.NAME STR %a_mask.fits # Name for mask extension or filename + WEIGHT.SOURCE STR FILE # Source type for weight: EXT | FILE + WEIGHT.NAME STR %a_weight.fits # Name for weight extension or filename +END Index: /branches/rel-1-0/ippconfig/ipprc.config.in =================================================================== --- /branches/rel-1-0/ippconfig/ipprc.config.in (revision 11645) +++ /branches/rel-1-0/ippconfig/ipprc.config.in (revision 11645) @@ -0,0 +1,52 @@ +### Example .ipprc file + +PATH STR @IPP_CONFIG_DIR@:$HOME/.ipp:. # Default search path for configuration files + +# place your data directories here and refer to as path://PATH/remainder +DATAPATH METADATA + DATA1 STR /data/alala/ + PAP_MC STR /data/alala/price/megacam/ + ESSENCE STR /data/sn7/ipp/ + ISPTEST STR /data/alala/eugene/swtests/pantasks/isp + SIMTEST STR /data/alala/eugene/swtests/pantasks/sim + SAN1 STR /s01/price/ +END + +### Database configuration +DBSERVER STR alala # Database host name (for psDBInit) +DBNAME STR ipp # Database name (for psDBInit) +DBUSER STR XXX # Database user name (for psDBInit) +DBPASSWORD STR XXX # Database password (for psDBInit) + +### Setups for each camera system +CAMERAS METADATA + MEGACAM STR megacam/camera.config + ISP-Apogee STR isp/camera.config +# CTIO_MOSAIC2 STR ctio_mosaic2/camera.config # CTIO MOSAIC2 camera, for ESSENCE +# UCAM STR ucam/camera.config +# GPC1 STR gpc1/camera.config +# LRIS_BLUE STR lris_blue/camera.config +# LRIS_RED STR lris_red/camera.config + SIMPLE STR simple/camera.config +END + +### psLib setup +#TIME STR /home/mithrandir/price/pan-starrs/jhroot/i686-pc-linux-gnu/etc/pslib/psTime.config # Time configuration file +LOGLEVEL S32 9 # Logging level; 3=INFO +LOGFORMAT STR THLNM # Log format +LOGDEST STR STDERR # Log destination +TRACEDEST STR STDERR # Trace destination +TRACE METADATA # Trace levels +# place default trace lines here +# example: +# psLib.db S32 10 +END + +RECIPES METADATA # Site-level recipes + PPIMAGE STR recipes/ppImage.config # Image reduction + PPMERGE STR recipes/ppMerge.config # Image combination + PPSTATS STR recipes/ppStats.config # Image statistics + PSPHOT STR recipes/psphot.config # Photometry + PSASTRO STR recipes/psastro.config # Astrometry + PPSTAC STR recipes/ppStac.config # Combination +END Index: /branches/rel-1-0/ippconfig/isp/.cvsignore =================================================================== --- /branches/rel-1-0/ippconfig/isp/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/ippconfig/isp/Makefile.am =================================================================== --- /branches/rel-1-0/ippconfig/isp/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/Makefile.am (revision 11645) @@ -0,0 +1,22 @@ + +installdir = $(datadir)/ippconfig/isp + +install_files = \ + camera.config \ + format.config \ + cmp.config \ + cmf.config \ + ppImage.config \ + ppMerge.config \ + ppStats.config \ + psastro.config \ + psphot.config + +install_DATA = $(install_files) + +install-data-hook: + chmod 0755 $(installdir) + +EXTRA_DIST = $(install_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippconfig/isp/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/camera.config (revision 11645) @@ -0,0 +1,145 @@ +# Camera configuration file for the Pan-STARRS Imaging Sky Probe + +# File formats that we know about +FORMATS METADATA + ISP STR isp/format.config + CMP STR isp/cmp.config + CMF STR isp/cmf.config +END + + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + Chip STR Cell +END + +# Recipe options +RECIPES METADATA + PSPHOT STR isp/psphot.config # psphot details + PSASTRO STR isp/psastro.config # psastro details + PPIMAGE STR isp/ppImage.config # Recipes for ppImage + PPMERGE STR isp/ppMerge.config # Recipes for ppMerge +END + +FILTER.ID METADATA + g STR g + r STR r + i STR i + z STR z + y STR y +END + + +# Rejection levels for detrend creation +REJECTION METADATA + TYPE LIMITS FILTER EXPECTED IMFILE.MEAN IMFILE.STDEV IMFILE.MEANSTDEV EXP.MEAN EXP.STDEV EXP.MEANSTDEV ENSEMBLE.MEAN ENSEMBLE.STDEV ENSEMBLE.MEANSTDEV IMFILE.SN EXP.SN + + BIAS LIMITS * 0 2 10 0.5 1.0 10 0.5 3 3 0 0 0 + DARK LIMITS * 0 2 10 0.5 1.0 10 0.5 3 3 0 0 0 + FRINGE LIMITS * 0 0 0 0 0 0 0 0 0 0 0 0 + SHUTTER LIMITS * 0 0 0 0 0 0 0 0 0 0 0 20.0 + # need to adjust these to get a decent clipping + + FLAT MULTI + FLAT LIMITS * 0 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS u 0 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS g 0 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS r 0 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS i 0 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS z 0 0 0 0 0 0 0 0 0 3 0 0 + +# FILTER is an additional qualifier, and may be "*" (or absent!), in which case it matches everything +# EXPECTED is the expected mean value +# IMFILE.MEAN is the maximum permitted mean value for an imfile, relative to the standard deviation +# IMFILE.STDEV is the maximum permitted standard deviation for an imfile +# EXP.MEAN is the maximum permitted mean value for an exposure, relative to the standard deviation +# EXP.STDEV is the maximum permitted standard deviation for an exposure +# EXP.MEANSTDEV is the maximum permitted mean standard deviation for an exposure relative to the mean +# ENSEMBLE.MEAN is the maximum permitted mean for an ensemble of exposures +# ENSEMBLE.STDEV is the maximum permitted standard deviation for an ensemble of exposures +# ENSEMBLE.MEANSTDEV is the maximum permitted mean standard deviation for an ensemble of exposures +# IMFILE.SN is the minimum permitted signal-to-noise for an imfile +# EXP.SN is the minimum permitted signal-to-noise for an exposure +# These values (all except FILTER) may be zero, in which case no clipping is applied. + +END + + +FILERULES METADATA + PSASTRO.INPUT STR PSASTRO.INPUT.CMF + PSASTRO.OUTPUT STR PSASTRO.OUTPUT.CMF + PSPHOT.OUTPUT STR PSPHOT.OUTPUT.CMF + + ### input file definitions + TYPE INPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA DATA.LEVEL FILE.TYPE + PPIMAGE.INPUT INPUT none.fits NONE NONE NONE FPA IMAGE + PPIMAGE.BIAS INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.DARK INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.FLAT INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.SHUTTER INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.MASK INPUT mask.fits NONE NONE NONE CHIP IMAGE + # PPIMAGE.BIAS INPUT bias.fits NONE NONE NONE CHIP IMAGE + # PPIMAGE.DARK INPUT dark.fits NONE NONE NONE CHIP IMAGE + # PPIMAGE.FLAT INPUT flat.fits NONE NONE NONE CHIP IMAGE + + PPARITH.INPUT INPUT none.fits NONE NONE NONE FPA IMAGE + + PSPHOT.INPUT INPUT none.fits NONE NONE NONE CHIP IMAGE + PSPHOT.PSF.LOAD INPUT none.psf NONE NONE NONE CHIP PSF + + PSASTRO.INPUT.CMP INPUT none.fits NONE NONE PHU FPA CMP + PSASTRO.INPUT.CMF INPUT none.fits NONE SMPDATA PHU FPA CMF + + ### output file definitions + TYPE OUTPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA FILE.TYPE FILE.LEVEL DATA.LEVEL FILE.SAVE FILE.FORMAT + PPIMAGE.OUTPUT OUTPUT {OUTPUT}.isp.fits NONE NONE NONE IMAGE FPA FPA TRUE NONE + PPIMAGE.OUTPUT.MASK OUTPUT {OUTPUT}.mask.fits NONE NONE NONE MASK FPA FPA FALSE NONE + PPIMAGE.OUTPUT.WEIGHT OUTPUT {OUTPUT}.wt.fits NONE NONE NONE WEIGHT FPA FPA FALSE NONE + PPIMAGE.OUTPUT.CHIP OUTPUT {OUTPUT}.chip.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PPIMAGE.OUTPUT.FPA1 OUTPUT {OUTPUT}.b1.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PPIMAGE.OUTPUT.FPA2 OUTPUT {OUTPUT}.b2.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + + PPIMAGE.BIN1 OUTPUT {OUTPUT}.b1.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PPIMAGE.BIN2 OUTPUT {OUTPUT}.b2.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PPIMAGE.JPEG1 OUTPUT {OUTPUT}.b1.jpg NONE NONE NONE JPEG FPA FPA TRUE ISP + PPIMAGE.JPEG2 OUTPUT {OUTPUT}.b2.jpg NONE NONE NONE JPEG FPA FPA TRUE ISP + + PPARITH.OUTPUT OUTPUT {OUTPUT} NONE NONE NONE IMAGE FPA FPA TRUE NONE + + PSPHOT.RESID OUTPUT {OUTPUT}.res.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PSPHOT.BACKGND OUTPUT {OUTPUT}.bck.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PSPHOT.BACKSUB OUTPUT {OUTPUT}.sub.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PSPHOT.BACKMDL OUTPUT {OUTPUT}.mdl.fits NONE NONE NONE IMAGE FPA FPA TRUE ISP + PSPHOT.OUTPUT.RAW OUTPUT {OUTPUT} NONE NONE NONE RAW FPA FPA TRUE NONE + PSPHOT.OUTPUT.SX OUTPUT {OUTPUT}.sx NONE NONE NONE SX FPA FPA TRUE NONE + PSPHOT.OUTPUT.OBJ OUTPUT {OUTPUT}.obj NONE NONE NONE OBJ FPA FPA TRUE NONE + PSPHOT.OUTPUT.CMP OUTPUT {OUTPUT}.cmp NONE NONE NONE CMP FPA FPA TRUE NONE + PSPHOT.OUTPUT.CMF OUTPUT {OUTPUT}.cmf NONE SMPDATA PHU CMF FPA FPA TRUE NONE + PSPHOT.PSF.SAVE OUTPUT {OUTPUT}.psf NONE NONE NONE PSF FPA FPA TRUE NONE + + SOURCE.PLOT.MOMENTS OUTPUT {OUTPUT}.mnt.png NONE NONE NONE KAPA FPA FPA TRUE NONE + SOURCE.PLOT.PSFMODEL OUTPUT {OUTPUT}.psf.png NONE NONE NONE KAPA FPA FPA TRUE ISP + + PSASTRO.OUTPUT.CMP OUTPUT {OUTPUT}.smp NONE NONE NONE CMP FPA FPA TRUE NONE + PSASTRO.OUTPUT.CMF OUTPUT {OUTPUT}.smf NONE SMPDATA PHU CMF FPA FPA TRUE NONE +END + +# FPA file defines properties of a possible input|output object +# user can set the filename (I|O), filename rules (O), or abstract source (@FILES, @DETDB) (I) +# user can set the extension name, if used +# user can set the file type (IMAGE, JPEG, RAW, SX, OBJ, CMP, CMF) : but these are not variable in most cases! +# user can set the file mode (READ, WRITE), but these must be set by the programmer +# user can set the file depth: only valid for output files +# user can set the data depth: must be >= file depth +# user can set the file format: only valid for newly created FPAs +# user can set the colormap, scaling method, scaling range (JPEG only) +# user can set the extension name for the data and header segments (CMF only) + +# known uses of the filename.xtra, extname.rule, and extname.xtra: +# extname.rule: used to construct output EXTNAME keys (CMF DATA) +# extname.xtra: used to construct output EXTNAME keys (CMF HEAD) + +# extname.rule: used to carry classID to DetDB (internal) +# filename.xtra: used to carry detID to DetDB (internal) + +# filename.xtra: used to carry arglist name (internal) Index: /branches/rel-1-0/ippconfig/isp/cmf.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/cmf.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/cmf.config (revision 11645) @@ -0,0 +1,84 @@ +# Pan-STARRS Imaging Sky Probe + +# How to identify this type +RULE METADATA + SIMPLE BOOL TRUE + NAXIS S32 0 + TELESCOP STR ISP-1 + INSTRUME STR ISP-Apogee + DETECTOR STR ISP-Apogee-01 + ISPCAMER STR Apogee U42 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR SEQID # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS STR Chip:Cell:amplifier + +# Specify the cell data +CELLS METADATA + amplifier METADATA + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC STR TRIMSEC + CELL.BIASSEC STR BIASSEC + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR OBSTYPE + FPA.FILTER STR FILTNAME + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.ALT STR ALT + FPA.AZ STR AZ + FPA.POSANGLE STR ROTANGLE + FPA.AIRMASS STR AIRMASS + FPA.TIME STR MJD-OBS + CHIP.TEMP STR CCDTEMP + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.TIME STR MJD-OBS + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.XBIN STR XBIN + CELL.YBIN STR YBIN +# CELL.SATURATION STR SATURATE ### Currently set to 0 ??? + CELL.BAD STR BADLEVEL +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.TIMESYS STR UTC + CELL.SATURATION F32 65535 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC + CHIP.XPARITY S32 1 + CHIP.YPARITY S32 1 + CHIP.X0 S32 0 + CHIP.Y0 S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.X0 S32 0 + CELL.Y0 S32 0 +END + +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + +# PS Concepts to get from the database +DATABASE METADATA +# None. +END Index: /branches/rel-1-0/ippconfig/isp/cmp.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/cmp.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/cmp.config (revision 11645) @@ -0,0 +1,84 @@ +# Pan-STARRS Imaging Sky Probe + +# How to identify this type +RULE METADATA + SIMPLE BOOL FALSE + NAXIS S32 0 + TELESCOP STR ISP-1 + INSTRUME STR ISP-Apogee + DETECTOR STR ISP-Apogee-01 + ISPCAMER STR Apogee U42 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR SEQID # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS STR Chip:Cell:amplifier + +# Specify the cell data +CELLS METADATA + amplifier METADATA + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC STR TRIMSEC + CELL.BIASSEC STR BIASSEC + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR OBSTYPE + FPA.FILTER STR FILTNAME + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.ALT STR ALT + FPA.AZ STR AZ + FPA.POSANGLE STR ROTANGLE + FPA.AIRMASS STR AIRMASS + FPA.TIME STR MJD-OBS + CHIP.TEMP STR CCDTEMP + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.TIME STR MJD-OBS + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.XBIN STR XBIN + CELL.YBIN STR YBIN +# CELL.SATURATION STR SATURATE ### Currently set to 0 ??? + CELL.BAD STR BADLEVEL +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.TIMESYS STR UTC + CELL.SATURATION F32 65535 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC + CHIP.XPARITY S32 1 + CHIP.YPARITY S32 1 + CHIP.X0 S32 0 + CHIP.Y0 S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.X0 S32 0 + CELL.Y0 S32 0 +END + +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + +# PS Concepts to get from the database +DATABASE METADATA +# None. +END Index: /branches/rel-1-0/ippconfig/isp/format.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/format.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/format.config (revision 11645) @@ -0,0 +1,89 @@ +# Pan-STARRS Imaging Sky Probe + +# How to identify this type +RULE METADATA + SIMPLE BOOL TRUE + NAXIS S32 2 + TELESCOP STR ISP-1 + INSTRUME STR ISP-Apogee + DETECTOR STR ISP-Apogee-01 + ISPCAMER STR Apogee U42 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR SEQID # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS STR Chip:Cell:amplifier + +# Specify the cell data +CELLS METADATA + amplifier METADATA + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC STR TRIMSEC + CELL.BIASSEC STR BIASSEC + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.TELESCOPE STR TELESCOP + FPA.INSTRUMENT STR INSTRUME + FPA.DETECTOR STR DETECTOR + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR OBSTYPE + FPA.FILTERID STR FILTNAME + FPA.FILTER STR FILTNAME + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.ALT STR ALT + FPA.AZ STR AZ + FPA.POSANGLE STR ROTANGLE + FPA.AIRMASS STR AIRMASS + FPA.TIME STR MJD-OBS + FPA.EXPOSURE STR EXPTIME + CHIP.TEMP STR CCDTEMP + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.TIME STR MJD-OBS + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.XBIN STR XBIN + CELL.YBIN STR YBIN +# CELL.SATURATION STR SATURATE ### Currently set to 0 ??? + CELL.BAD STR BADLEVEL +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.TIMESYS STR UTC + CELL.SATURATION F32 65535 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC + CHIP.XPARITY S32 1 + CHIP.YPARITY S32 1 + CHIP.X0 S32 0 + CHIP.Y0 S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.X0 S32 0 + CELL.Y0 S32 0 +END + +FORMATS METADATA + FPA.RA STR DEGREES + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + +# PS Concepts to get from the database +DATABASE METADATA +# None. +END Index: /branches/rel-1-0/ippconfig/isp/ppImage.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/ppImage.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/ppImage.config (revision 11645) @@ -0,0 +1,69 @@ +### ppImage recipe configuration file + +# Inherit everything from the top-level recipe file, except for what's below. + +PHOTCODE.RULE STR {DETECTOR}.{FILTER.ID} + +# apply the following constraints when selecting a detrend image +DETREND.CONSTRAINTS METADATA + BIAS METADATA + END + DARK METADATA + END + SHUTTER METADATA + END + FLAT METADATA + FILTER STR FPA.FILTERID +# VERSION STR RAW + END + FRINGE METADATA + FILTER STR FPA.FILTERID +# AIRMASS STR FPA.AIRMASS +# TWILIGHT STR TWILIGHT:FPA.TIME + END +END + +# Overscan, bias, dark, shutter, flat-field, fringe, photom +PPIMAGE_OBDSFRP METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + PHOTOM BOOL TRUE # Source identification and photometry + ASTROM.CHIP BOOL FALSE # Astrometry per chip? + ASTROM.MOSAIC BOOL FALSE # Astrometry for mosaic? +END + +# Overscan, bias, dark, shutter, flat-field, fringe, photom, astrom +PPIMAGE_OBDSFRA METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + PHOTOM BOOL TRUE # Source identification and photometry + ASTROM.CHIP BOOL TRUE # Astrometry per chip? + ASTROM.MOSAIC BOOL FALSE # Astrometry for mosaic? +END + +# Overscan, bias, dark, shutter, flat-field, fringe +PPIMAGE_OBDSFR METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END Index: /branches/rel-1-0/ippconfig/isp/ppMerge.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/ppMerge.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/ppMerge.config (revision 11645) @@ -0,0 +1,3 @@ +### ppImage recipe configuration file + +# Inherit everything from the top-level recipe file, except for what's below. Index: /branches/rel-1-0/ippconfig/isp/ppStats.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/ppStats.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/ppStats.config (revision 11645) @@ -0,0 +1,12 @@ +### ppStats recipe +SAMPLE F32 0.1 # Fraction of cell to sample +MASKVAL U8 0xff # Mask value to use for statistics + +# Define the outputs as MULTI +HEADER MULTI +CONCEPT MULTI +STAT MULTI + +# Values to return +STAT STR ROBUST_MEDIAN # Background estimator +STAT STR ROBUST_STDEV # Background standard deviation estimator Index: /branches/rel-1-0/ippconfig/isp/psastro.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/psastro.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/psastro.config (revision 11645) @@ -0,0 +1,45 @@ + +# astrometry matching parameters + +# nominal plate scale (microns / pixel) +# PSASTRO.PLATE.SCALE F32 13.5 +PSASTRO.PIXEL.SCALE F32 10.0 + +# pmAstromGridMatch: +PSASTRO.GRID.MIN.ANGLE F32 -2.0 +PSASTRO.GRID.MAX.ANGLE F32 +2.0 +#PSASTRO.GRID.MIN.ANGLE F32 102.0 +#PSASTRO.GRID.MAX.ANGLE F32 104.0 +PSASTRO.GRID.DEL.ANGLE F32 0.25 +PSASTRO.GRID.MIN.SIGMA F32 5.0 + +# pmAstromGridAngle +# max grid offset in FP units (microns) +# use plate-scale to make this in pixels? +# PSASTRO.GRID.OFFSET F32 10000. +# PSASTRO.GRID.SCALE F32 500. +PSASTRO.GRID.OFFSET F32 10000. +PSASTRO.GRID.SCALE F32 500 + +# these tweak are in FP units (pixels, currently) +PSASTRO.TWEAK.SCALE F32 1 +PSASTRO.TWEAK.RANGE F32 75 +PSASTRO.TWEAK.SMOOTH F32 2 +PSASTRO.TWEAK.NSIGMA F32 3 + +# pmAstromRadiusMatch +# use plate-scale to make this in pixels? +PSASTRO.MATCH.RADIUS F32 8 + +# pmAstromMatchFit +#PSASTRO.CHIP.NX S32 3 # fit order +#PSASTRO.CHIP.NY S32 3 # fit order +PSASTRO.CHIP.ORDER S32 3 # fit order +PSASTRO.CHIP.NITER S32 3 # fit clipping iterations +PSASTRO.CHIP.NSIGMA F32 3 # fit clipping sigmas + +PSASTRO.MAX.ERROR F32 10.0 # max error in pixels +PSASTRO.MIN.NSTAR S32 10 # min fitted stars in solution + +MAG_MAX F32 10 +DVO.CATDIR STR /data/alala/ipp/ippRefs/catdir.synth.bright Index: /branches/rel-1-0/ippconfig/isp/psphot.config =================================================================== --- /branches/rel-1-0/ippconfig/isp/psphot.config (revision 11645) +++ /branches/rel-1-0/ippconfig/isp/psphot.config (revision 11645) @@ -0,0 +1,128 @@ + +SAVE.OUTPUT BOOL TRUE +#SAVE.BACKMDL BOOL TRUE +#SAVE.BACKGND BOOL TRUE +#SAVE.BACKSUB BOOL TRUE +SAVE.RESID BOOL TRUE +#SAVE.PSF BOOL TRUE + +#LOAD.PSF BOOL FALSE + +# image noise parameters +RDNOISE STR HD:RDNOISE # read-noise in electrons +GAIN STR HD:GAIN # electrons / DN +ZERO_POINT F32 25.000 # zero point used by DVO + +# background model : +BACKGROUND.XBIN S32 128 +BACKGROUND.YBIN S32 128 + +# masking parameters +XMIN F32 2 # minimum valid x-coord +XMAX F32 -2 # maximum valid x-coord +YMIN F32 2 # minimum valid y-coord +YMAX F32 -2 # maximum valid y-coord +SATURATION F32 50000 # saturation level on this chip +MIN_VALID_PIXEL F32 1 # saturation level on this chip + +# image statistics parameters +IMSTATS_NPIX S32 1000 # number of pixels to use for sky estimate boxes: + +# peak finding +PEAKS_SMOOTH_SIGMA F32 1.0 # smoothing kernel sigma in pixels +PEAKS_SMOOTH_NSIGMA F32 3.0 # smoothing kernel width in sigmas +PEAKS_NSIGMA_LIMIT F32 25.0 # peak significance threshold +PEAKS_NSIGMA_LIMIT_2 F32 5.0 # peak significance threshold + +# basic object statistics +SKY_INNER_RADIUS F32 15 # square annulus for local sky measurement +SKY_OUTER_RADIUS F32 25 # square annulus for local sky measurement +PSF_MOMENTS_RADIUS F32 3 # calculate initial source moments with this radius +PSF_SN_LIM F32 25 # minimum S/N for stars used for PSF model +PSF_CLUMP_NSIGMA F32 3 # size of PSF clump in image (use a larger number for images with large PSF variations) +PSF_MAX_NSTARS S32 300 # limit number of stars used for PSF model + +# PSF model parameters : choose the PSF model +# list as many PSF_MODEL options as desired +# PSF_MODEL MULTI +PSF_MODEL STR PS_MODEL_QGAUSS +# PSF_MODEL STR PS_MODEL_GAUSS +# PSF_MODEL STR PS_MODEL_PGAUSS +# PSF_MODEL STR PS_MODEL_TEST1 + +MOMENTS_SCALE F32 0.1 +MOMENTS_SN_MIN F32 10.0 +MOMENTS_SX_MAX F32 3.0 +MOMENTS_SY_MAX F32 3.0 +MOMENTS_AR_MAX F32 5.0 # maximum axial ratio: 1 / AR < (sx / sy) < AR + +PSF_FIT_RADIUS F32 9.0 # fitting radius for test PSF model +PSF_REF_RADIUS F32 11.0 # aperture magnitudes are scaled via + # curve-of-growth to this radius +# PSF-like source model parameters +PSF_FIT_NSIGMA F32 1.0 # significance for pixel included in fit +PSF_FIT_PADDING F32 2.0 # extra annulus to use for fit +PSF_SHAPE_NSIGMA F32 3.0 # max significance for shape variation +PSF_MIN_SN F32 2.0 # reject objects below this significance +PSF_MAX_CHI F32 50.0 # reject objects worse than this + +# EXTended source model parameters +EXT_MODEL STR PS_MODEL_PGAUSS +EXT_MIN_SN F32 50.0 # fit galaxies above this S/N limit +EXT_FIT_NSIGMA F32 1 # significance for pixel included in fit +EXT_FIT_PADDING F32 10 # extra annulus to use for fit +EXT_MOMENTS_RADIUS F32 9 + +FITMODE STR BLEND +DEBLEND_PEAK_FRACTION F32 0.1 +DEBLEND_SKY_NSIGMA F32 10.0 + +# APTREND STR CONSTANT +APTREND STR ALL +# APTREND STR SKYSAT_XY_LIN +# APTREND STR SKYSAT +# APTREND STR SKYBIAS +APTREND.XORDER S32 2 +APTREND.YORDER S32 2 +FULL_FIT_SN_LIM F32 50.0 +AP_MIN_SN F32 20.0 +MAX_AP_OFFSET F32 0.5 +CONSTANT_PHOTOMETRIC_WEIGHTS BOOL TRUE # Should the photometric code [currently only ensemblePSF] + # refuse to weight each pixel by it's significance? + +# test options + +# BREAK_POINT may be one of (PEAKS, MOMENTS, PSFMODEL, ENSEMBLE) +BREAK_POINT STR ENSEMBLE + +# PEAKS_OUTPUT_FILE STR peaks.dat +# MOMENTS_OUTPUT_FILE STR moments.dat +# ANALYSIS_REGION STR [1000:1600,2800:3400] +# POISSON_ERRORS BOOL TRUE +# SKY_BIAS F32 0.0 + +# PSFTREND must be a 2D polynomial +# the specified values are ignored but define the active components of the polynomial +PSF.TREND.MASK METADATA + NORDER_X S32 2 # number of x orders + NORDER_Y S32 2 # number of y orders + VAL_X00_Y00 F64 1 # polynomial coefficient + VAL_X01_Y00 F64 1 # polynomial coefficient + VAL_X00_Y01 F64 1 # polynomial coefficient + VAL_X02_Y00 F64 1 # polynomial coefficient + VAL_X01_Y01 F64 1 # polynomial coefficient + VAL_X00_Y02 F64 1 # polynomial coefficient + NELEMENTS S32 6 # number of unmasked components +END # folder for 4D polynomial + +# APTREND must be a 4D polynomial +# the specified values are ignored but define the active components of the polynomial +AP.TREND.MASK METADATA + NORDER_X S32 2 # number of x orders + NORDER_Y S32 2 # number of y orders + NORDER_Z S32 1 # number of z orders + NORDER_T S32 1 # number of t orders + VAL_X00_Y00_Z00_T00 F64 1 # polynomial coefficient + VAL_X00_Y00_Z01_T00 F64 0 # polynomial coefficient + NELEMENTS S32 2 # number of unmasked components +END # folder for 4D polynomial Index: /branches/rel-1-0/ippconfig/lris_blue/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/lris_blue/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/lris_blue/camera.config (revision 11645) @@ -0,0 +1,21 @@ +# Camera configuration file for LRIS Blue: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR lris_blue/format_raw.config + MEF STR lris_blue/format_mef.config +END + + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + LeftChip STR LeftAmp RightAmp + RightChip STR LeftAmp RightAmp +END + + +# Recipe options +RECIPES METADATA + PHASE2 STR phase2.config # Phase 2 recipe details +# PSPHOT STR psphot.config # psphot details +END Index: /branches/rel-1-0/ippconfig/lris_blue/format_mef.config =================================================================== --- /branches/rel-1-0/ippconfig/lris_blue/format_mef.config (revision 11645) +++ /branches/rel-1-0/ippconfig/lris_blue/format_mef.config (revision 11645) @@ -0,0 +1,79 @@ +# The Low Resolution Imager and Spectrograph (LRIS) blue side + +# How to identify this type +RULE METADATA + TELESCOP STR Keck I + INSTRUME STR LRISBLUE + FORMAT STR MEF +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CHIP # There are no extensions + FPA.NAME STR OBSNUM # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS METADATA + ccd1 STR LeftChip:LeftAmp:left LeftChip:RightAmp:right + ccd2 STR RightChip:LeftAmp:left RightChip:RightAmp:right +END + +# Specify the cell data +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.GAIN.SOURCE STR HEADER + CELL.BIASSEC STR BSECL1 BSECL2 + CELL.TRIMSEC STR TSECL + CELL.GAIN STR GAINL + CELL.X0 S32 0 + END + + right METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.GAIN.SOURCE STR HEADER + CELL.BIASSEC STR BSECR1 BSECR2 + CELL.TRIMSEC STR TSECR + CELL.GAIN STR GAINR + CELL.X0 S32 1024 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR BLUFILT + FPA.POSANGLE STR ROTPOSN + FPA.RA STR RA + FPA.DEC STR DEC + CELL.EXPOSURE STR EXPOSURE + CELL.DARKTIME STR EXPOSURE # No special darktime header; use exposure time + CELL.TIME STR MJD-OBS + CELL.XBIN STR CCDSUM + CELL.YBIN STR CCDSUM +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.RADECSYS STR ICRS + CELL.TIMESYS STR UTC + CELL.READDIR S32 1 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 + CELL.READNOISE F32 2.5 + CELL.SATURATION F32 65536 + CELL.BAD F32 0 +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.TIME STR MJD +END Index: /branches/rel-1-0/ippconfig/lris_blue/format_raw.config =================================================================== --- /branches/rel-1-0/ippconfig/lris_blue/format_raw.config (revision 11645) +++ /branches/rel-1-0/ippconfig/lris_blue/format_raw.config (revision 11645) @@ -0,0 +1,113 @@ +# The Low Resolution Imager and Spectrograph (LRIS) blue side + +# We have no choice but to hard-code the various regions, because Keck +# only stores them as: +# WINDOW = '1,0,0,2048,4096' +# PREPIX = 51 +# POSTPIX = 80 +# BINNING = '1,1 ' +# AMPPSIZE= '[1:1024,1:4096]' + +# I don't know how we would get the IPP to react to changes in the +# windowing on the fly --- we have no mechanism for setting the region +# sizes on the basis of the above keywords. Therefore, we hard-code +# the regions and assert on our assumptions in the RULE. + + +# How to identify this type +RULE METADATA + TELESCOP STR Keck I + INSTRUME STR LRISBLUE + AMPLIST STR 1,4,0,0 + WINDOW STR 1,0,0,2048,4096 + PREPIX S32 51 + POSTPIX S32 80 + BINNING STR 1,1 + AMPPSIZE STR [1:1024,1:4096] + NAXIS1 S32 4620 + NAXIS2 S32 4096 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR OBSNUM # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS STR LeftChip:LeftAmp:amp1 LeftChip:RightAmp:amp2 RightChip:LeftAmp:amp3 RightChip:RightAmp:amp4 + +# Specify the cell data +CELLS METADATA + amp1 METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [1:51,1:4096];[4301:4380,1:4096] + CELL.TRIMSEC STR [205:1228,1:4096] + CELL.GAIN F32 1.542 + CELL.X0 S32 0 + END + + amp2 METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [52:102,1:4096];[4381:4460,1:4096] + CELL.TRIMSEC STR [1229:2252,1:4096] + CELL.GAIN F32 1.546 + CELL.X0 S32 1024 + END + + amp3 METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [103:153,1:4096];[4461:4540,1:4096] + CELL.TRIMSEC STR [2253:3276,1:4096] + CELL.GAIN F32 1.632 + CELL.X0 S32 0 + END + + amp4 METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [154:204,1:4096];[4541:4620,1:4096] + CELL.TRIMSEC STR [3277:4300,1:4096] + CELL.GAIN F32 1.633 + CELL.X0 S32 1024 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR BLUFILT + FPA.POSANGLE STR ROTPOSN + FPA.RA STR RA + FPA.DEC STR DEC + CELL.EXPOSURE STR EXPOSURE + CELL.DARKTIME STR EXPOSURE # No special darktime header; use exposure time + CELL.TIME STR MJD-OBS + CELL.XBIN STR CCDSUM + CELL.YBIN STR CCDSUM +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.RADECSYS STR ICRS + CELL.TIMESYS STR UTC + CELL.READDIR S32 1 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 + CELL.READNOISE F32 2.5 + CELL.SATURATION F32 65536 + CELL.BAD F32 0 +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.TIME STR MJD +END Index: /branches/rel-1-0/ippconfig/lris_red/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/lris_red/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/lris_red/camera.config (revision 11645) @@ -0,0 +1,19 @@ +# Camera configuration file for LRIS-Red: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR lris_red/format_raw.config + SPEC STR lris_red/format_spec.config +END + + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + RedChip STR LeftSide RightSide +END + + +# Recipes to use when processing this camera +RECIPES METADATA + PPMERGE STR ppMerge.config # Recipe to use for ppMerge +END Index: /branches/rel-1-0/ippconfig/lris_red/format_raw.config =================================================================== --- /branches/rel-1-0/ippconfig/lris_red/format_raw.config (revision 11645) +++ /branches/rel-1-0/ippconfig/lris_red/format_raw.config (revision 11645) @@ -0,0 +1,95 @@ +# The Low Resolution Imager and Spectrograph (LRIS) red side + +# We have no choice but to hard-code the various regions, because Keck +# only stores them as: +# WINDOW = '0,0,0,2048,2048' +# PREPIX = 20 +# POSTPIX = 80 +# BINNING = '1,1 ' +# AMPPSIZE= '[1:1024,1:4096]' + +# I don't know how we would get the IPP to react to changes in the +# windowing on the fly --- we have no mechanism for setting the region +# sizes on the basis of the above keywords. Therefore, we hard-code +# the regions and assert on our assumptions in the RULE. + + +# How to identify this type +RULE METADATA + TELESCOP STR Keck I + INSTRUME STR LRIS + AMPLIST STR 2,1,0,0 + WINDOW STR 0,0,0,2048,2048 + PREPIX S32 20 + POSTPIX S32 80 + BINNING STR 1, 1 + CCDPSIZE STR [1:2048,1:2048] + NAXIS1 S32 2248 + NAXIS2 S32 2048 + IMTYPE STR TWOAMPTOP +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents a single chip + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR OBSNUM # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS STR RedChip:LeftSide:left RedChip:RightSide:right + +# Specify the cell data +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [1:20,1:2048];[2089:2168,1:2048] + CELL.TRIMSEC STR [41:1064,1:2048] + CELL.GAIN F32 1.98 + CELL.READNOISE F32 6.1 + CELL.X0 S32 0 + END + + right METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [21:40,1:2048];[2169:2248,1:2048] + CELL.TRIMSEC STR [1065:2088,1:2048] + CELL.GAIN F32 2.17 + CELL.READNOISE F32 6.3 + CELL.X0 S32 1023 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR POSANG + FPA.RA STR OBJ-RA + FPA.DEC STR OBJ-DEC + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.TIME STR MJD-OBS +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.RADECSYS STR ICRS + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.SATURATION F32 65535 + CELL.BAD F32 0 + CELL.XBIN S32 1 + CELL.YBIN S32 1 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC +END + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.TIME STR MJD +END Index: /branches/rel-1-0/ippconfig/lris_red/format_spec.config =================================================================== --- /branches/rel-1-0/ippconfig/lris_red/format_spec.config (revision 11645) +++ /branches/rel-1-0/ippconfig/lris_red/format_spec.config (revision 11645) @@ -0,0 +1,96 @@ +# The Low Resolution Imager and Spectrograph (LRIS) red side + +# We have no choice but to hard-code the various regions, because Keck +# only stores them as: +# WINDOW = '0,0,550,2048,1000' +# PREPIX = 20 +# POSTPIX = 80 +# BINNING = '1,1 ' +# AMPPSIZE= '[1:1024,1:4096]' + +# I don't know how we would get the IPP to react to changes in the +# windowing on the fly --- we have no mechanism for setting the region +# sizes on the basis of the above keywords. Therefore, we hard-code +# the regions and assert on our assumptions in the RULE. + + +# How to identify this type +RULE METADATA + TELESCOP STR Keck I + INSTRUME STR LRIS + AMPLIST STR 2,1,0,0 + WINDOW STR 0,0,550,2048,1000 # Narrow spectroscopy format + PREPIX S32 20 + POSTPIX S32 80 + BINNING STR 1, 1 + CCDPSIZE STR [1:2048,1:2048] + NAXIS1 S32 2248 + NAXIS2 S32 1000 + IMTYPE STR TWOAMPTOP +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents a single chip + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR OBSNUM # A PHU keyword for unique identifier within the hierarchy level + CHIP.NAME STR INSTRUME # An extension keyword for unique identifier +END + +# What's in the FITS file? +CONTENTS STR RedChip:LeftSide:left RedChip:RightSide:right + +# Specify the cell data +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [1:20,1:1000];[2089:2168,1:1000] + CELL.TRIMSEC STR [41:1064,1:1000] + CELL.GAIN F32 1.98 + CELL.READNOISE F32 6.1 + CELL.X0 S32 0 + END + + right METADATA + CELL.BIASSEC.SOURCE STR VALUE + CELL.TRIMSEC.SOURCE STR VALUE + CELL.BIASSEC STR [21:40,1:1000];[2169:2248,1:1000] + CELL.TRIMSEC STR [1065:2088,1:1000] + CELL.GAIN F32 2.17 + CELL.READNOISE F32 6.3 + CELL.X0 S32 1023 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR POSANG + FPA.RA STR OBJ-RA + FPA.DEC STR OBJ-DEC + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.TIME STR MJD-OBS +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.RADECSYS STR ICRS + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.SATURATION F32 65535 + CELL.BAD F32 0 + CELL.XBIN S32 1 + CELL.YBIN S32 1 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC +END + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.TIME STR MJD +END Index: /branches/rel-1-0/ippconfig/mcshort/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/mcshort/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/mcshort/camera.config (revision 11645) @@ -0,0 +1,188 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# Camera configuration file for MegaCam: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR mcshort/format_raw.config + SPLICED STR mcshort/format_spliced.config + SPLIT STR mcshort/format_split.config +END + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA +# ccd00 STR LeftAmp RightAmp +# ccd01 STR LeftAmp RightAmp +# ccd02 STR LeftAmp RightAmp +# ccd03 STR LeftAmp RightAmp +# ccd04 STR LeftAmp RightAmp +# ccd05 STR LeftAmp RightAmp +# ccd06 STR LeftAmp RightAmp +# ccd07 STR LeftAmp RightAmp +# ccd08 STR LeftAmp RightAmp +# ccd09 STR LeftAmp RightAmp +# ccd10 STR LeftAmp RightAmp +# ccd11 STR LeftAmp RightAmp + ccd12 STR LeftAmp RightAmp + ccd13 STR LeftAmp RightAmp + ccd14 STR LeftAmp RightAmp +# ccd15 STR LeftAmp RightAmp +# ccd16 STR LeftAmp RightAmp +# ccd17 STR LeftAmp RightAmp +# ccd18 STR LeftAmp RightAmp +# ccd19 STR LeftAmp RightAmp +# ccd20 STR LeftAmp RightAmp + ccd21 STR LeftAmp RightAmp + ccd22 STR LeftAmp RightAmp + ccd23 STR LeftAmp RightAmp +# ccd24 STR LeftAmp RightAmp +# ccd25 STR LeftAmp RightAmp +# ccd26 STR LeftAmp RightAmp +# ccd27 STR LeftAmp RightAmp +# ccd28 STR LeftAmp RightAmp +# ccd29 STR LeftAmp RightAmp +# ccd30 STR LeftAmp RightAmp +# ccd31 STR LeftAmp RightAmp +# ccd32 STR LeftAmp RightAmp +# ccd33 STR LeftAmp RightAmp +# ccd34 STR LeftAmp RightAmp +# ccd35 STR LeftAmp RightAmp +END + + +# move this elsewhere? we need a lookup table to go from filter ID to abstract name +FILTER.ID METADATA + u.MP9301 STR u + g.MP9401 STR g + r.MP9601 STR r + i.MP9701 STR i + z.MP9801 STR z + Zp STR z + Zprime STR z + Ha.MP7605 STR Ha + Halpha STR Ha + Haalpha.on STR Ha + HaOFF.MP7604 STR HaOff + + CN.MP780 STR CN + cn.MP7803 STR CN + CN.MP7803 STR CN + + TiO.MP77 STR TiO + tio.MP7701 STR TiO + TiO.MP7701 STR TiO + NB920 STR NB920 + + B2F STR B2F + Bj STR Bj + Vj STR Vj + Rj STR Rj + Ij STR Ij + Hb STR Hb + HbOff STR HbOff +END + + +# Recipe options +RECIPES METADATA + # Other recipes + PSPHOT STR megacam/psphot.config # psphot details + PSASTRO STR megacam/psastro.config # psastro details + PPSTATS STR megacam/ppStats.config # ppStats recipe + PPIMAGE STR megacam/ppImage.config # ppImage recipe +END + + +# Rejection levels for detrend creation +REJECTION METADATA + TYPE LIMITS FILTER EXPECTED IMFILE.MEAN IMFILE.STDEV EXP.MEAN EXP.STDEV EXP.MEANSTDEV ENSEMBLE.MEAN ENSEMBLE.STDEV ENSEMBLE.MEANSTDEV IMFILE.SN EXP.SN IMFILE.MEANSTDEV + FLAT MULTI + SKYFLAT MULTI + DOMEFLAT MULTI + + BIAS LIMITS * 0 1 4 0.5 4 0 3 4 0 0 0 0.5 + DARK LIMITS * 0 1 4 0.5 4 0 3 4 0 0 0 0.5 + FLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 + SKYFLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 + DOMEFLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS u 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS g 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS r 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS i 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS z 0 0 0 0 0 0 0 0 3 0 0 0 + FRINGE LIMITS * 0 0 0 0 0 0 0 0 0 0 0 0 + +# FILTER is an additional qualifier, and may be "*" (or absent!), in which case it matches everything +# EXPECTED is the expected mean value +# IMFILE.MEAN is the maximum permitted mean value for an imfile, relative to the standard deviation +# IMFILE.STDEV is the maximum permitted standard deviation for an imfile +# EXP.MEAN is the maximum permitted mean value for an exposure, relative to the standard deviation +# EXP.STDEV is the maximum permitted standard deviation for an exposure +# EXP.MEANSTDEV is the maximum permitted mean standard deviation for an exposure relative to the mean +# ENSEMBLE.MEAN is the maximum permitted mean for an ensemble of exposures +# ENSEMBLE.STDEV is the maximum permitted standard deviation for an ensemble of exposures +# ENSEMBLE.MEANSTDEV is the maximum permitted mean standard deviation for an ensemble of exposures +# IMFILE.SN is the minimum permitted signal-to-noise for an imfile +# EXP.SN is the minimum permitted signal-to-noise for an exposure +# These values (all except FILTER) may be zero, in which case no clipping is applied. + +END + + +FILERULES METADATA + ### Redirections + PSASTRO.INPUT STR PSASTRO.INPUT.CMP + PSASTRO.OUTPUT STR PSASTRO.OUTPUT.CMP + PSPHOT.OUTPUT STR PSPHOT.OUTPUT.CMF + + ### input file definitions + TYPE INPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA DATA.LEVEL FILE.TYPE + PPIMAGE.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + + ### use these entries to get the detrend images from specific files +# PPIMAGE.MASK INPUT mcshort/mask.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP MASK + PPIMAGE.BIAS INPUT file:///data/alala/price/megacam/bias/MCSHORT.bias.norm.1.2.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.DARK INPUT file:///data/alala/price/megacam/bias/MCSHORT.dark.norm.2.1.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.FLAT INPUT file:///data/alala/price/megacam/flats/MCSHORT.flat.norm.3.1.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.FRINGE INPUT file:///fringe.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP FRINGE + + ### use these entries to get the detrend images from the database + #PPIMAGE.MASK INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.BIAS INPUT @DETDB fpa fpa NONE CHIP IMAGE + #PPIMAGE.DARK INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.FLAT INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + + PSPHOT.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + + PSASTRO.INPUT.CMP INPUT @FILES NONE NONE PHU CHIP CMP + PSASTRO.INPUT.CMF INPUT @FILES NONE SMPDATA PHU CHIP CMF + + ### output file definitions + TYPE OUTPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA FILE.LEVEL DATA.LEVEL FILE.TYPE FILE.SAVE FILE.FORMAT +# PPIMAGE.OUTPUT OUTPUT {OUTPUT}.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE NONE + PPIMAGE.OUTPUT OUTPUT {OUTPUT}.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CHIP IMAGE TRUE SPLICED + PPIMAGE.OUTPUT.MASK OUTPUT {OUTPUT}.{CHIP.NAME}.mask.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP MASK FALSE NONE + PPIMAGE.OUTPUT.WEIGHT OUTPUT {OUTPUT}.{CHIP.NAME}.wt.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP WEIGHT FALSE NONE + PPIMAGE.BIN1 OUTPUT {OUTPUT}.{CHIP.NAME}.b1.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE NONE + PPIMAGE.BIN2 OUTPUT {OUTPUT}.{CHIP.NAME}.b2.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE NONE + + PPIMAGE.OUTPUT.CHIP OUTPUT {OUTPUT}.{CHIP.NAME}.chip.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE NONE + PPIMAGE.OUTPUT.FPA1 OUTPUT {OUTPUT}.b1.fits NONE NONE NONE FPA FPA IMAGE TRUE NONE + PPIMAGE.OUTPUT.FPA2 OUTPUT {OUTPUT}.b2.fits NONE NONE NONE FPA FPA IMAGE TRUE NONE + + PPIMAGE.JPEG1 OUTPUT {OUTPUT}.b1.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + PPIMAGE.JPEG2 OUTPUT {OUTPUT}.b2.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + + PSPHOT.RESID OUTPUT {OUTPUT}.res.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + PSPHOT.BACKGND OUTPUT {OUTPUT}.bck.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + PSPHOT.BACKSUB OUTPUT {OUTPUT}.sub.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + PSPHOT.BACKMDL OUTPUT {OUTPUT}.mdl.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + + PSPHOT.OUTPUT.RAW OUTPUT {OUTPUT}.{CHIP.NAME} NONE NONE PHU CHIP CHIP RAW TRUE NONE + PSPHOT.OUTPUT.SX OUTPUT {OUTPUT}.sx NONE NONE PHU CHIP CHIP SX TRUE NONE + PSPHOT.OUTPUT.OBJ OUTPUT {OUTPUT}.obj NONE NONE PHU CHIP CHIP OBJ TRUE NONE + PSPHOT.OUTPUT.CMF OUTPUT {OUTPUT}.cmf NONE SMPDATA PHU CHIP CHIP CMF TRUE NONE + PSPHOT.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.cmp NONE NONE PHU CHIP CHIP CMP TRUE NONE + + PSASTRO.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.smp NONE NONE PHU CHIP CHIP CMP TRUE NONE + +END Index: /branches/rel-1-0/ippconfig/mcshort/format_raw.config =================================================================== --- /branches/rel-1-0/ippconfig/mcshort/format_raw.config (revision 11645) +++ /branches/rel-1-0/ippconfig/mcshort/format_raw.config (revision 11645) @@ -0,0 +1,336 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. + +# How to identify this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 72 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CELL # The extensions represent cells + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:type +# amp00 STR ccd00:LeftAmp:left +# amp01 STR ccd00:RightAmp:right +# amp02 STR ccd01:LeftAmp:left +# amp03 STR ccd01:RightAmp:right +# amp04 STR ccd02:LeftAmp:left +# amp05 STR ccd02:RightAmp:right +# amp06 STR ccd03:LeftAmp:left +# amp07 STR ccd03:RightAmp:right +# amp08 STR ccd04:LeftAmp:left +# amp09 STR ccd04:RightAmp:right +# amp10 STR ccd05:LeftAmp:left +# amp11 STR ccd05:RightAmp:right +# amp12 STR ccd06:LeftAmp:left +# amp13 STR ccd06:RightAmp:right +# amp14 STR ccd07:LeftAmp:left +# amp15 STR ccd07:RightAmp:right +# amp16 STR ccd08:LeftAmp:left +# amp17 STR ccd08:RightAmp:right +# amp18 STR ccd09:LeftAmp:left +# amp19 STR ccd09:RightAmp:right +# amp20 STR ccd10:LeftAmp:left +# amp21 STR ccd10:RightAmp:right +# amp22 STR ccd11:LeftAmp:left +# amp23 STR ccd11:RightAmp:right + amp24 STR ccd12:LeftAmp:left + amp25 STR ccd12:RightAmp:right + amp26 STR ccd13:LeftAmp:left + amp27 STR ccd13:RightAmp:right + amp28 STR ccd14:LeftAmp:left + amp29 STR ccd14:RightAmp:right +# amp30 STR ccd15:LeftAmp:left +# amp31 STR ccd15:RightAmp:right +# amp32 STR ccd16:LeftAmp:left +# amp33 STR ccd16:RightAmp:right +# amp34 STR ccd17:LeftAmp:left +# amp35 STR ccd17:RightAmp:right +# amp36 STR ccd18:LeftAmp:left +# amp37 STR ccd18:RightAmp:right +# amp38 STR ccd19:LeftAmp:left +# amp39 STR ccd19:RightAmp:right +# amp40 STR ccd20:LeftAmp:left +# amp41 STR ccd20:RightAmp:right + amp42 STR ccd21:LeftAmp:left + amp43 STR ccd21:RightAmp:right + amp44 STR ccd22:LeftAmp:left + amp45 STR ccd22:RightAmp:right + amp46 STR ccd23:LeftAmp:left + amp47 STR ccd23:RightAmp:right +# amp48 STR ccd24:LeftAmp:left +# amp49 STR ccd24:RightAmp:right +# amp50 STR ccd25:LeftAmp:left +# amp51 STR ccd25:RightAmp:right +# amp52 STR ccd26:LeftAmp:left +# amp53 STR ccd26:RightAmp:right +# amp54 STR ccd27:LeftAmp:left +# amp55 STR ccd27:RightAmp:right +# amp56 STR ccd28:LeftAmp:left +# amp57 STR ccd28:RightAmp:right +# amp58 STR ccd29:LeftAmp:left +# amp59 STR ccd29:RightAmp:right +# amp60 STR ccd30:LeftAmp:left +# amp61 STR ccd30:RightAmp:right +# amp62 STR ccd31:LeftAmp:left +# amp63 STR ccd31:RightAmp:right +# amp64 STR ccd32:LeftAmp:left +# amp65 STR ccd32:RightAmp:right +# amp66 STR ccd33:LeftAmp:left +# amp67 STR ccd33:RightAmp:right +# amp68 STR ccd34:LeftAmp:left +# amp69 STR ccd34:RightAmp:right +# amp70 STR ccd35:LeftAmp:left +# amp71 STR ccd35:RightAmp:right +END + +# Specify the cell data +CELLS METADATA + left METADATA # Left amplifier + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.XPARITY S32 1 # We could have specified this as a DEFAULT, but this works + CELL.X0 S32 1 + CELL.Y0 S32 1 + END + right METADATA # Right amplifier + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.XPARITY S32 -1 # This cell is read out in the opposite direction + CELL.X0 S32 2048 + CELL.Y0 S32 1 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR CMMTOBS + FPA.TIME STR MJD-OBS + FPA.TIMESYS STR TIMESYS + FPA.ALT STR TELALT + FPA.AZ STR TELAZ + CHIP.TEMP STR DETTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.TIMESYS STR TIMESYS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.YPARITY S32 1 + + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA +# ccd00 S32 0 +# ccd01 S32 2048 +# ccd02 S32 4096 +# ccd03 S32 6144 +# ccd04 S32 8192 +# ccd05 S32 10240 +# ccd06 S32 12288 +# ccd07 S32 14336 +# ccd08 S32 16384 +# ccd09 S32 0 +# ccd10 S32 2048 +# ccd11 S32 4096 + ccd12 S32 6144 + ccd13 S32 8192 + ccd14 S32 10240 +# ccd15 S32 12288 +# ccd16 S32 14336 +# ccd17 S32 16384 +# ccd18 S32 0 +# ccd19 S32 2048 +# ccd20 S32 4096 + ccd21 S32 6144 + ccd22 S32 8192 + ccd23 S32 10240 +# ccd24 S32 12288 +# ccd25 S32 14336 +# ccd26 S32 16384 +# ccd27 S32 0 +# ccd28 S32 2048 +# ccd29 S32 4096 +# ccd30 S32 6144 +# ccd31 S32 8192 +# ccd32 S32 10240 +# ccd33 S32 12288 +# ccd34 S32 14336 +# ccd35 S32 16384 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA +# ccd00 S32 18447 +# ccd01 S32 18447 +# ccd02 S32 18447 +# ccd03 S32 18447 +# ccd04 S32 18447 +# ccd05 S32 18447 +# ccd06 S32 18447 +# ccd07 S32 18447 +# ccd08 S32 18447 +# ccd09 S32 13835 +# ccd10 S32 13835 +# ccd11 S32 13835 + ccd12 S32 13835 + ccd13 S32 13835 + ccd14 S32 13835 +# ccd15 S32 13835 +# ccd16 S32 13835 +# ccd17 S32 13835 +# ccd18 S32 13835 +# ccd19 S32 4612 +# ccd20 S32 4612 + ccd21 S32 4612 + ccd22 S32 4612 + ccd23 S32 4612 +# ccd24 S32 4612 +# ccd25 S32 4612 +# ccd26 S32 4612 +# ccd27 S32 0 +# ccd28 S32 0 +# ccd29 S32 0 +# ccd30 S32 0 +# ccd31 S32 0 +# ccd32 S32 0 +# ccd33 S32 0 +# ccd34 S32 0 +# ccd35 S32 0 + END + CHIP.XPARITY.DEPEND STR CHIP.NAME + CHIP.XPARITY METADATA +# ccd00 S32 1 +# ccd01 S32 1 +# ccd02 S32 1 +# ccd03 S32 1 +# ccd04 S32 1 +# ccd05 S32 1 +# ccd06 S32 1 +# ccd07 S32 1 +# ccd08 S32 1 +# ccd09 S32 1 +# ccd10 S32 1 +# ccd11 S32 1 + ccd12 S32 1 + ccd13 S32 1 + ccd14 S32 1 +# ccd15 S32 1 +# ccd16 S32 1 +# ccd17 S32 1 +# ccd18 S32 1 +# ccd19 S32 1 +# ccd20 S32 1 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 +# ccd24 S32 1 +# ccd25 S32 1 +# ccd26 S32 1 +# ccd27 S32 1 +# ccd28 S32 1 +# ccd29 S32 1 +# ccd30 S32 1 +# ccd31 S32 1 +# ccd32 S32 1 +# ccd33 S32 1 +# ccd34 S32 1 +# ccd35 S32 1 + END + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA +# ccd00 S32 -1 +# ccd01 S32 -1 +# ccd02 S32 -1 +# ccd03 S32 -1 +# ccd04 S32 -1 +# ccd05 S32 -1 +# ccd06 S32 -1 +# ccd07 S32 -1 +# ccd08 S32 -1 +# ccd09 S32 -1 +# ccd10 S32 -1 +# ccd11 S32 -1 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 +# ccd15 S32 -1 +# ccd16 S32 -1 +# ccd17 S32 -1 +# ccd18 S32 1 +# ccd19 S32 1 +# ccd20 S32 1 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 +# ccd24 S32 1 +# ccd25 S32 1 +# ccd26 S32 1 +# ccd27 S32 1 +# ccd28 S32 1 +# ccd29 S32 1 +# ccd30 S32 1 +# ccd31 S32 1 +# ccd32 S32 1 +# ccd33 S32 1 +# ccd34 S32 1 +# ccd35 S32 1 + END +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# FPA.BIAS METADATA +# TABLE STR Camera +# COLUMN STR gain +# chipId STR {CHIP.NAME} +# cellId STR {CELL.NAME} +# time STR {CELL.TIME} +# END +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD + CELL.X0 STR FORTRAN + CELL.Y0 STR FORTRAN +END Index: /branches/rel-1-0/ippconfig/mcshort/format_spliced.config =================================================================== --- /branches/rel-1-0/ippconfig/mcshort/format_spliced.config (revision 11645) +++ /branches/rel-1-0/ippconfig/mcshort/format_spliced.config (revision 11645) @@ -0,0 +1,298 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# The spliced MecaCam data is stored in single extensions for each chip + +# How to recognise this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 36 +END + +FILE METADATA + # How to read this data + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CHIP # The extensions represent chips + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier + CHIP.NAME STR EXTNAME # An extension keyword for unique identifie +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:component cell +# ccd00 STR ccd00:LeftAmp:left ccd00:RightAmp:right +# ccd01 STR ccd01:LeftAmp:left ccd01:RightAmp:right +# ccd02 STR ccd02:LeftAmp:left ccd02:RightAmp:right +# ccd03 STR ccd03:LeftAmp:left ccd03:RightAmp:right +# ccd04 STR ccd04:LeftAmp:left ccd04:RightAmp:right +# ccd05 STR ccd05:LeftAmp:left ccd05:RightAmp:right +# ccd06 STR ccd06:LeftAmp:left ccd06:RightAmp:right +# ccd07 STR ccd07:LeftAmp:left ccd07:RightAmp:right +# ccd08 STR ccd08:LeftAmp:left ccd08:RightAmp:right +# ccd09 STR ccd09:LeftAmp:left ccd09:RightAmp:right +# ccd10 STR ccd10:LeftAmp:left ccd10:RightAmp:right +# ccd11 STR ccd11:LeftAmp:left ccd11:RightAmp:right + ccd12 STR ccd12:LeftAmp:left ccd12:RightAmp:right + ccd13 STR ccd13:LeftAmp:left ccd13:RightAmp:right + ccd14 STR ccd14:LeftAmp:left ccd14:RightAmp:right +# ccd15 STR ccd15:LeftAmp:left ccd15:RightAmp:right +# ccd16 STR ccd16:LeftAmp:left ccd16:RightAmp:right +# ccd17 STR ccd17:LeftAmp:left ccd17:RightAmp:right +# ccd18 STR ccd18:LeftAmp:left ccd18:RightAmp:right +# ccd19 STR ccd19:LeftAmp:left ccd19:RightAmp:right +# ccd20 STR ccd20:LeftAmp:left ccd20:RightAmp:right + ccd21 STR ccd21:LeftAmp:left ccd21:RightAmp:right + ccd22 STR ccd22:LeftAmp:left ccd22:RightAmp:right + ccd23 STR ccd23:LeftAmp:left ccd23:RightAmp:right +# ccd24 STR ccd24:LeftAmp:left ccd24:RightAmp:right +# ccd25 STR ccd25:LeftAmp:left ccd25:RightAmp:right +# ccd26 STR ccd26:LeftAmp:left ccd26:RightAmp:right +# ccd27 STR ccd27:LeftAmp:left ccd27:RightAmp:right +# ccd28 STR ccd28:LeftAmp:left ccd28:RightAmp:right +# ccd29 STR ccd29:LeftAmp:left ccd29:RightAmp:right +# ccd30 STR ccd30:LeftAmp:left ccd30:RightAmp:right +# ccd31 STR ccd31:LeftAmp:left ccd31:RightAmp:right +# ccd32 STR ccd32:LeftAmp:left ccd32:RightAmp:right +# ccd33 STR ccd33:LeftAmp:left ccd33:RightAmp:right +# ccd34 STR ccd34:LeftAmp:left ccd34:RightAmp:right +# ccd35 STR ccd35:LeftAmp:left ccd35:RightAmp:right +END + +# Specify the cells +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECA + CELL.TRIMSEC STR TSECA + CELL.X0 S32 0 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINA + END + + right METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECB + CELL.TRIMSEC STR TSECB + CELL.X0 S32 1024 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINB + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR CMMTOBS + FPA.TIME STR MJD-OBS + FPA.TIMESYS STR TIMESYS + FPA.ALT STR TELALT + FPA.AZ STR TELAZ + CHIP.TEMP STR DETTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.TIMESYS STR TIMESYS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.TIMESYS STR UTC + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 +# PPMERGE.SCALE F32 1.0 +# PPMERGE.ZERO F32 0.0 + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 0 + ccd13 S32 2048 + ccd14 S32 4096 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 0 + ccd22 S32 2048 + ccd23 S32 4096 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 9223 + ccd13 S32 9223 + ccd14 S32 9223 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 0 + ccd22 S32 0 + ccd23 S32 0 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END + CHIP.XPARITY.DEPEND STR CHIP.NAME + CHIP.XPARITY METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 1 + ccd13 S32 1 + ccd14 S32 1 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END +END + + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + Index: /branches/rel-1-0/ippconfig/mcshort/format_split.config =================================================================== --- /branches/rel-1-0/ippconfig/mcshort/format_split.config (revision 11645) +++ /branches/rel-1-0/ippconfig/mcshort/format_split.config (revision 11645) @@ -0,0 +1,296 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# The spliced MecaCam data is stored in single extensions for each chip + +# How to recognise this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam +END + +FILE METADATA + # How to read this data + PHU STR CHIP # The FITS file represents an entire FPA + EXTENSIONS STR NONE # The extensions represent chips + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier + CHIP.NAME STR EXTNAME # An extension keyword for unique identifie + CONTENT STR {CHIP.NAME} # Key to the CONTENTS menu +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:component cell +# ccd00 STR ccd00:LeftAmp:left ccd00:RightAmp:right +# ccd01 STR ccd01:LeftAmp:left ccd01:RightAmp:right +# ccd02 STR ccd02:LeftAmp:left ccd02:RightAmp:right +# ccd03 STR ccd03:LeftAmp:left ccd03:RightAmp:right +# ccd04 STR ccd04:LeftAmp:left ccd04:RightAmp:right +# ccd05 STR ccd05:LeftAmp:left ccd05:RightAmp:right +# ccd06 STR ccd06:LeftAmp:left ccd06:RightAmp:right +# ccd07 STR ccd07:LeftAmp:left ccd07:RightAmp:right +# ccd08 STR ccd08:LeftAmp:left ccd08:RightAmp:right +# ccd09 STR ccd09:LeftAmp:left ccd09:RightAmp:right +# ccd10 STR ccd10:LeftAmp:left ccd10:RightAmp:right +# ccd11 STR ccd11:LeftAmp:left ccd11:RightAmp:right + ccd12 STR ccd12:LeftAmp:left ccd12:RightAmp:right + ccd13 STR ccd13:LeftAmp:left ccd13:RightAmp:right + ccd14 STR ccd14:LeftAmp:left ccd14:RightAmp:right +# ccd15 STR ccd15:LeftAmp:left ccd15:RightAmp:right +# ccd16 STR ccd16:LeftAmp:left ccd16:RightAmp:right +# ccd17 STR ccd17:LeftAmp:left ccd17:RightAmp:right +# ccd18 STR ccd18:LeftAmp:left ccd18:RightAmp:right +# ccd19 STR ccd19:LeftAmp:left ccd19:RightAmp:right +# ccd20 STR ccd20:LeftAmp:left ccd20:RightAmp:right + ccd21 STR ccd21:LeftAmp:left ccd21:RightAmp:right + ccd22 STR ccd22:LeftAmp:left ccd22:RightAmp:right + ccd23 STR ccd23:LeftAmp:left ccd23:RightAmp:right +# ccd24 STR ccd24:LeftAmp:left ccd24:RightAmp:right +# ccd25 STR ccd25:LeftAmp:left ccd25:RightAmp:right +# ccd26 STR ccd26:LeftAmp:left ccd26:RightAmp:right +# ccd27 STR ccd27:LeftAmp:left ccd27:RightAmp:right +# ccd28 STR ccd28:LeftAmp:left ccd28:RightAmp:right +# ccd29 STR ccd29:LeftAmp:left ccd29:RightAmp:right +# ccd30 STR ccd30:LeftAmp:left ccd30:RightAmp:right +# ccd31 STR ccd31:LeftAmp:left ccd31:RightAmp:right +# ccd32 STR ccd32:LeftAmp:left ccd32:RightAmp:right +# ccd33 STR ccd33:LeftAmp:left ccd33:RightAmp:right +# ccd34 STR ccd34:LeftAmp:left ccd34:RightAmp:right +# ccd35 STR ccd35:LeftAmp:left ccd35:RightAmp:right +END + +# Specify the cells +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECA + CELL.TRIMSEC STR TSECA + CELL.X0 S32 0 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINA + END + + right METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECB + CELL.TRIMSEC STR TSECB + CELL.X0 S32 1024 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINB + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR CMMTOBS + FPA.TIME STR MJD-OBS + FPA.TIMESYS STR TIMESYS + FPA.ALT STR TELALT + FPA.AZ STR TELAZ + CHIP.TEMP STR DETTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.TIMESYS STR TIMESYS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 +# PPMERGE.SCALE F32 1.0 +# PPMERGE.ZERO F32 0.0 + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 0 + ccd13 S32 2048 + ccd14 S32 4096 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 0 + ccd22 S32 2048 + ccd23 S32 4096 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 9223 + ccd13 S32 9223 + ccd14 S32 9223 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 0 + ccd22 S32 0 + ccd23 S32 0 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END + CHIP.XPARITY.DEPEND STR CHIP.NAME + CHIP.XPARITY METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 1 + ccd13 S32 1 + ccd14 S32 1 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA +# ccd00 S32 +# ccd01 S32 +# ccd02 S32 +# ccd03 S32 +# ccd04 S32 +# ccd05 S32 +# ccd06 S32 +# ccd07 S32 +# ccd08 S32 +# ccd09 S32 +# ccd10 S32 +# ccd11 S32 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 +# ccd15 S32 +# ccd16 S32 +# ccd17 S32 +# ccd18 S32 +# ccd19 S32 +# ccd20 S32 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 +# ccd24 S32 +# ccd25 S32 +# ccd26 S32 +# ccd27 S32 +# ccd28 S32 +# ccd29 S32 +# ccd30 S32 +# ccd31 S32 +# ccd32 S32 +# ccd33 S32 +# ccd34 S32 +# ccd35 S32 + END +END + + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + Index: /branches/rel-1-0/ippconfig/megacam/.cvsignore =================================================================== --- /branches/rel-1-0/ippconfig/megacam/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/ippconfig/megacam/Makefile.am =================================================================== --- /branches/rel-1-0/ippconfig/megacam/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/Makefile.am (revision 11645) @@ -0,0 +1,22 @@ + +installdir = $(datadir)/ippconfig/megacam + +install_files = \ + camera.config \ + format_raw.config \ + format_split.config \ + format_spliced.config \ + ppImage.config \ + ppMerge.config \ + ppStats.config \ + psastro.config \ + psphot.config + +install_DATA = $(install_files) + +install-data-hook: + chmod 0755 $(installdir) + +EXTRA_DIST = $(install_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippconfig/megacam/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/camera.config (revision 11645) @@ -0,0 +1,197 @@ +# Camera configuration file for MegaCam: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR megacam/format_raw.config + SPLICED STR megacam/format_spliced.config + SPLIT STR megacam/format_split.config +END + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + ccd00 STR LeftAmp RightAmp + ccd01 STR LeftAmp RightAmp + ccd02 STR LeftAmp RightAmp + ccd03 STR LeftAmp RightAmp + ccd04 STR LeftAmp RightAmp + ccd05 STR LeftAmp RightAmp + ccd06 STR LeftAmp RightAmp + ccd07 STR LeftAmp RightAmp + ccd08 STR LeftAmp RightAmp + ccd09 STR LeftAmp RightAmp + ccd10 STR LeftAmp RightAmp + ccd11 STR LeftAmp RightAmp + ccd12 STR LeftAmp RightAmp + ccd13 STR LeftAmp RightAmp + ccd14 STR LeftAmp RightAmp + ccd15 STR LeftAmp RightAmp + ccd16 STR LeftAmp RightAmp + ccd17 STR LeftAmp RightAmp + ccd18 STR LeftAmp RightAmp + ccd19 STR LeftAmp RightAmp + ccd20 STR LeftAmp RightAmp + ccd21 STR LeftAmp RightAmp + ccd22 STR LeftAmp RightAmp + ccd23 STR LeftAmp RightAmp + ccd24 STR LeftAmp RightAmp + ccd25 STR LeftAmp RightAmp + ccd26 STR LeftAmp RightAmp + ccd27 STR LeftAmp RightAmp + ccd28 STR LeftAmp RightAmp + ccd29 STR LeftAmp RightAmp + ccd30 STR LeftAmp RightAmp + ccd31 STR LeftAmp RightAmp + ccd32 STR LeftAmp RightAmp + ccd33 STR LeftAmp RightAmp + ccd34 STR LeftAmp RightAmp + ccd35 STR LeftAmp RightAmp +END + +# move this elsewhere? we need a lookup table to go from filter ID to abstract name +FILTER.ID METADATA + u.MP9301 STR u + g.MP9401 STR g + r.MP9601 STR r + i.MP9701 STR i + z.MP9801 STR z + Zp STR z + Zprime STR z + Ha.MP7605 STR Ha + Halpha STR Ha + Haalpha.on STR Ha + HaOFF.MP7604 STR HaOff + + CN.MP780 STR CN + cn.MP7803 STR CN + CN.MP7803 STR CN + + TiO.MP77 STR TiO + tio.MP7701 STR TiO + TiO.MP7701 STR TiO + NB920 STR NB920 + + B2F STR B2F + Bj STR Bj + Vj STR Vj + Rj STR Rj + Ij STR Ij + Hb STR Hb + HbOff STR HbOff +END + +# convert supplied FPA.OBSTYPE values to abstract exptype names +OBSTYPE.TABLE METADATA + bias STR BIAS + zero STR BIAS + dark STR DARK + flat STR SKYFLAT + skyflat STR SKYFLAT + domeflat STR DOMEFLAT + object STR OBJECT + science STR OBJECT +END + +# Recipe options +RECIPES METADATA + # Other recipes + PSPHOT STR megacam/psphot.config # psphot details + PSASTRO STR megacam/psastro.config # psastro details + PPSTATS STR megacam/ppStats.config # ppStats recipe + PPIMAGE STR megacam/ppImage.config # ppImage recipe +END + + +# Rejection levels for detrend creation +REJECTION METADATA + TYPE LIMITS FILTER EXPECTED IMFILE.MEAN IMFILE.STDEV EXP.MEAN EXP.STDEV EXP.MEANSTDEV ENSEMBLE.MEAN ENSEMBLE.STDEV ENSEMBLE.MEANSTDEV IMFILE.SN EXP.SN + FLAT MULTI + + BIAS LIMITS * 0 1 5 0.5 3 0.5 3 3 0 0 0 + DARK LIMITS * 0 1 5 0.5 3 0.5 3 3 0 0 0 + FLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS u 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS g 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS r 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS i 0 0 0 0 0 0 0 0 3 0 0 +# FLAT LIMITS z 0 0 0 0 0 0 0 0 3 0 0 + FRINGE LIMITS * 0 0 0 0 0 0 0 0 0 0 0 + +# FILTER is an additional qualifier, and may be "*" (or absent!), in which case it matches everything +# EXPECTED is the expected mean value +# IMFILE.MEAN is the maximum permitted mean value for an imfile, relative to the standard deviation +# IMFILE.STDEV is the maximum permitted standard deviation for an imfile +# EXP.MEAN is the maximum permitted mean value for an exposure, relative to the standard deviation +# EXP.STDEV is the maximum permitted standard deviation for an exposure +# EXP.MEANSTDEV is the maximum permitted mean standard deviation for an exposure relative to the mean +# ENSEMBLE.MEAN is the maximum permitted mean for an ensemble of exposures +# ENSEMBLE.STDEV is the maximum permitted standard deviation for an ensemble of exposures +# ENSEMBLE.MEANSTDEV is the maximum permitted mean standard deviation for an ensemble of exposures +# IMFILE.SN is the minimum permitted signal-to-noise for an imfile +# EXP.SN is the minimum permitted signal-to-noise for an exposure +# These values (all except FILTER) may be zero, in which case no clipping is applied. + +END + + +FILERULES METADATA + ### Redirections + PSASTRO.INPUT STR PSASTRO.INPUT.CMP + PSASTRO.OUTPUT STR PSASTRO.OUTPUT.CMP + PSPHOT.OUTPUT STR PSPHOT.OUTPUT.CMF + PPIMAGE.OUTPUT STR PPIMAGE.OUTPUT.MEF + PPIMAGE.BIN1 STR PPIMAGE.BIN1.MEF + PPIMAGE.BIN2 STR PPIMAGE.BIN2.MEF + + ### input file definitions + TYPE INPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA DATA.LEVEL FILE.TYPE + PPIMAGE.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + + ### use these entries to get the detrend images from specific files + #PPIMAGE.MASK INPUT megacam/mask.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP MASK + #PPIMAGE.BIAS INPUT megacam/bias/MEGACAM.bias.norm.1.0.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.DARK INPUT megacam/MEGACAM.dark.norm.2.0.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.FLAT INPUT megacam/MEGACAM.flat.norm.3.0.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.FRINGE INPUT megacam/MEGACAM.fringe.norm.4.0.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP FRINGE + + ### use these entries to get the detrend images from the database + PPIMAGE.MASK INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.BIAS INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.DARK INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.FLAT INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.FRINGE INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP FRINGE + + PSPHOT.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + PSASTRO.INPUT.CMP INPUT @FILES NONE NONE PHU CHIP CMP + PSASTRO.INPUT.CMF INPUT @FILES NONE SMPDATA PHU CHIP CMF + + ### output file definitions + TYPE OUTPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA FILE.LEVEL DATA.LEVEL FILE.TYPE FILE.SAVE FILE.FORMAT + PPIMAGE.OUTPUT.SPL OUTPUT {OUTPUT}.{CHIP.NAME}.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE NONE + PPIMAGE.OUTPUT.MEF OUTPUT {OUTPUT}.fits {CHIP.NAME} {CHIP.NAME} NONE FPA CHIP IMAGE TRUE NONE + PPIMAGE.OUTPUT.MASK OUTPUT {OUTPUT}.{CHIP.NAME}.mask.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP MASK FALSE NONE + PPIMAGE.OUTPUT.WEIGHT OUTPUT {OUTPUT}.{CHIP.NAME}.wt.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP WEIGHT FALSE NONE + PPIMAGE.BIN1.SPL OUTPUT {OUTPUT}.{CHIP.NAME}.b1.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE SPLIT + PPIMAGE.BIN2.SPL OUTPUT {OUTPUT}.{CHIP.NAME}.b2.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE SPLIT + PPIMAGE.BIN1.MEF OUTPUT {OUTPUT}.b1.fits NONE NONE NONE FPA CHIP IMAGE TRUE NONE + PPIMAGE.BIN2.MEF OUTPUT {OUTPUT}.b2.fits NONE NONE NONE FPA CHIP IMAGE TRUE NONE + + PPIMAGE.OUTPUT.CHIP OUTPUT {OUTPUT}.{CHIP.NAME}.chip.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE SPLIT + PPIMAGE.OUTPUT.FPA1 OUTPUT {OUTPUT}.b1.fits NONE NONE NONE FPA FPA IMAGE TRUE SPLICED + PPIMAGE.OUTPUT.FPA2 OUTPUT {OUTPUT}.b2.fits NONE NONE NONE FPA FPA IMAGE TRUE SPLICED + + PPIMAGE.JPEG1 OUTPUT {OUTPUT}.b1.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + PPIMAGE.JPEG2 OUTPUT {OUTPUT}.b2.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + + PSPHOT.RESID OUTPUT {OUTPUT}.res.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + PSPHOT.BACKGND OUTPUT {OUTPUT}.bck.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + PSPHOT.BACKSUB OUTPUT {OUTPUT}.sub.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + PSPHOT.BACKMDL OUTPUT {OUTPUT}.mdl.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE SPLICED + + PSPHOT.OUTPUT.RAW OUTPUT {OUTPUT}.{CHIP.NAME} NONE NONE PHU CHIP CHIP RAW TRUE NONE + PSPHOT.OUTPUT.SX OUTPUT {OUTPUT}.sx NONE NONE PHU CHIP CHIP SX TRUE NONE + PSPHOT.OUTPUT.OBJ OUTPUT {OUTPUT}.obj NONE NONE PHU CHIP CHIP OBJ TRUE NONE + PSPHOT.OUTPUT.CMF OUTPUT {OUTPUT}.cmf NONE SMPDATA PHU CHIP CHIP CMF TRUE NONE + PSPHOT.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.cmp NONE NONE PHU CHIP CHIP CMP TRUE NONE + + PSASTRO.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.smp NONE NONE PHU CHIP CHIP CMP TRUE NONE +END Index: /branches/rel-1-0/ippconfig/megacam/format_raw.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/format_raw.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/format_raw.config (revision 11645) @@ -0,0 +1,301 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. + +# How to identify this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 72 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CELL # The extensions represent cells + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:type + amp00 STR ccd00:LeftAmp:left + amp01 STR ccd00:RightAmp:right + amp02 STR ccd01:LeftAmp:left + amp03 STR ccd01:RightAmp:right + amp04 STR ccd02:LeftAmp:left + amp05 STR ccd02:RightAmp:right + amp06 STR ccd03:LeftAmp:left + amp07 STR ccd03:RightAmp:right + amp08 STR ccd04:LeftAmp:left + amp09 STR ccd04:RightAmp:right + amp10 STR ccd05:LeftAmp:left + amp11 STR ccd05:RightAmp:right + amp12 STR ccd06:LeftAmp:left + amp13 STR ccd06:RightAmp:right + amp14 STR ccd07:LeftAmp:left + amp15 STR ccd07:RightAmp:right + amp16 STR ccd08:LeftAmp:left + amp17 STR ccd08:RightAmp:right + amp18 STR ccd09:LeftAmp:left + amp19 STR ccd09:RightAmp:right + amp20 STR ccd10:LeftAmp:left + amp21 STR ccd10:RightAmp:right + amp22 STR ccd11:LeftAmp:left + amp23 STR ccd11:RightAmp:right + amp24 STR ccd12:LeftAmp:left + amp25 STR ccd12:RightAmp:right + amp26 STR ccd13:LeftAmp:left + amp27 STR ccd13:RightAmp:right + amp28 STR ccd14:LeftAmp:left + amp29 STR ccd14:RightAmp:right + amp30 STR ccd15:LeftAmp:left + amp31 STR ccd15:RightAmp:right + amp32 STR ccd16:LeftAmp:left + amp33 STR ccd16:RightAmp:right + amp34 STR ccd17:LeftAmp:left + amp35 STR ccd17:RightAmp:right + amp36 STR ccd18:LeftAmp:left + amp37 STR ccd18:RightAmp:right + amp38 STR ccd19:LeftAmp:left + amp39 STR ccd19:RightAmp:right + amp40 STR ccd20:LeftAmp:left + amp41 STR ccd20:RightAmp:right + amp42 STR ccd21:LeftAmp:left + amp43 STR ccd21:RightAmp:right + amp44 STR ccd22:LeftAmp:left + amp45 STR ccd22:RightAmp:right + amp46 STR ccd23:LeftAmp:left + amp47 STR ccd23:RightAmp:right + amp48 STR ccd24:LeftAmp:left + amp49 STR ccd24:RightAmp:right + amp50 STR ccd25:LeftAmp:left + amp51 STR ccd25:RightAmp:right + amp52 STR ccd26:LeftAmp:left + amp53 STR ccd26:RightAmp:right + amp54 STR ccd27:LeftAmp:left + amp55 STR ccd27:RightAmp:right + amp56 STR ccd28:LeftAmp:left + amp57 STR ccd28:RightAmp:right + amp58 STR ccd29:LeftAmp:left + amp59 STR ccd29:RightAmp:right + amp60 STR ccd30:LeftAmp:left + amp61 STR ccd30:RightAmp:right + amp62 STR ccd31:LeftAmp:left + amp63 STR ccd31:RightAmp:right + amp64 STR ccd32:LeftAmp:left + amp65 STR ccd32:RightAmp:right + amp66 STR ccd33:LeftAmp:left + amp67 STR ccd33:RightAmp:right + amp68 STR ccd34:LeftAmp:left + amp69 STR ccd34:RightAmp:right + amp70 STR ccd35:LeftAmp:left + amp71 STR ccd35:RightAmp:right +END + +# Specify the cell data +CELLS METADATA + left METADATA # Left amplifier + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.XPARITY S32 1 # We could have specified this as a DEFAULT, but this works + CELL.X0 S32 1 + CELL.Y0 S32 1 + END + right METADATA # Right amplifier + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.XPARITY S32 -1 # This cell is read out in the opposite direction + CELL.X0 S32 2048 + CELL.Y0 S32 1 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.FOCUS STR TELFOCUS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR CMMTOBS + FPA.TIME STR MJD-OBS + FPA.TIMESYS STR TIMESYS + FPA.ALT STR TELALT + FPA.AZ STR TELAZ + FPA.TEMP STR DETTEM + FPA.EXPOSURE STR EXPTIME + CHIP.TEMP STR DETTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.TIMESYS STR TIMESYS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.YPARITY S32 1 + + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA + ccd00 S32 0 + ccd01 S32 2048 + ccd02 S32 4096 + ccd03 S32 6144 + ccd04 S32 8192 + ccd05 S32 10240 + ccd06 S32 12288 + ccd07 S32 14336 + ccd08 S32 16384 + ccd09 S32 0 + ccd10 S32 2048 + ccd11 S32 4096 + ccd12 S32 6144 + ccd13 S32 8192 + ccd14 S32 10240 + ccd15 S32 12288 + ccd16 S32 14336 + ccd17 S32 16384 + ccd18 S32 0 + ccd19 S32 2048 + ccd20 S32 4096 + ccd21 S32 6144 + ccd22 S32 8192 + ccd23 S32 10240 + ccd24 S32 12288 + ccd25 S32 14336 + ccd26 S32 16384 + ccd27 S32 0 + ccd28 S32 2048 + ccd29 S32 4096 + ccd30 S32 6144 + ccd31 S32 8192 + ccd32 S32 10240 + ccd33 S32 12288 + ccd34 S32 14336 + ccd35 S32 16384 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA + ccd00 S32 18447 + ccd01 S32 18447 + ccd02 S32 18447 + ccd03 S32 18447 + ccd04 S32 18447 + ccd05 S32 18447 + ccd06 S32 18447 + ccd07 S32 18447 + ccd08 S32 18447 + ccd09 S32 13835 + ccd10 S32 13835 + ccd11 S32 13835 + ccd12 S32 13835 + ccd13 S32 13835 + ccd14 S32 13835 + ccd15 S32 13835 + ccd16 S32 13835 + ccd17 S32 13835 + ccd18 S32 13835 + ccd19 S32 4612 + ccd20 S32 4612 + ccd21 S32 4612 + ccd22 S32 4612 + ccd23 S32 4612 + ccd24 S32 4612 + ccd25 S32 4612 + ccd26 S32 4612 + ccd27 S32 0 + ccd28 S32 0 + ccd29 S32 0 + ccd30 S32 0 + ccd31 S32 0 + ccd32 S32 0 + ccd33 S32 0 + ccd34 S32 0 + ccd35 S32 0 + END + CHIP.XPARITY S32 1 + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA + ccd00 S32 -1 + ccd01 S32 -1 + ccd02 S32 -1 + ccd03 S32 -1 + ccd04 S32 -1 + ccd05 S32 -1 + ccd06 S32 -1 + ccd07 S32 -1 + ccd08 S32 -1 + ccd09 S32 -1 + ccd10 S32 -1 + ccd11 S32 -1 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 + ccd15 S32 -1 + ccd16 S32 -1 + ccd17 S32 -1 + ccd18 S32 1 + ccd19 S32 1 + ccd20 S32 1 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 + ccd24 S32 1 + ccd25 S32 1 + ccd26 S32 1 + ccd27 S32 1 + ccd28 S32 1 + ccd29 S32 1 + ccd30 S32 1 + ccd31 S32 1 + ccd32 S32 1 + ccd33 S32 1 + ccd34 S32 1 + ccd35 S32 1 + END +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# FPA.BIAS METADATA +# TABLE STR Camera +# COLUMN STR gain +# chipId STR {CHIP.NAME} +# cellId STR {CELL.NAME} +# time STR {CELL.TIME} +# END +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD + CELL.X0 STR FORTRAN + CELL.Y0 STR FORTRAN +END Index: /branches/rel-1-0/ippconfig/megacam/format_spliced.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/format_spliced.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/format_spliced.config (revision 11645) @@ -0,0 +1,259 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# The spliced MecaCam data is stored in single extensions for each chip + +# How to recognise this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 36 +END + +FILE METADATA + # How to read this data + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CHIP # The extensions represent chips + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier + CHIP.NAME STR EXTNAME # An extension keyword for unique identifie +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:component cell + ccd00 STR ccd00:LeftAmp:left ccd00:RightAmp:right + ccd01 STR ccd01:LeftAmp:left ccd01:RightAmp:right + ccd02 STR ccd02:LeftAmp:left ccd02:RightAmp:right + ccd03 STR ccd03:LeftAmp:left ccd03:RightAmp:right + ccd04 STR ccd04:LeftAmp:left ccd04:RightAmp:right + ccd05 STR ccd05:LeftAmp:left ccd05:RightAmp:right + ccd06 STR ccd06:LeftAmp:left ccd06:RightAmp:right + ccd07 STR ccd07:LeftAmp:left ccd07:RightAmp:right + ccd08 STR ccd08:LeftAmp:left ccd08:RightAmp:right + ccd09 STR ccd09:LeftAmp:left ccd09:RightAmp:right + ccd10 STR ccd10:LeftAmp:left ccd10:RightAmp:right + ccd11 STR ccd11:LeftAmp:left ccd11:RightAmp:right + ccd12 STR ccd12:LeftAmp:left ccd12:RightAmp:right + ccd13 STR ccd13:LeftAmp:left ccd13:RightAmp:right + ccd14 STR ccd14:LeftAmp:left ccd14:RightAmp:right + ccd15 STR ccd15:LeftAmp:left ccd15:RightAmp:right + ccd16 STR ccd16:LeftAmp:left ccd16:RightAmp:right + ccd17 STR ccd17:LeftAmp:left ccd17:RightAmp:right + ccd18 STR ccd18:LeftAmp:left ccd18:RightAmp:right + ccd19 STR ccd19:LeftAmp:left ccd19:RightAmp:right + ccd20 STR ccd20:LeftAmp:left ccd20:RightAmp:right + ccd21 STR ccd21:LeftAmp:left ccd21:RightAmp:right + ccd22 STR ccd22:LeftAmp:left ccd22:RightAmp:right + ccd23 STR ccd23:LeftAmp:left ccd23:RightAmp:right + ccd24 STR ccd24:LeftAmp:left ccd24:RightAmp:right + ccd25 STR ccd25:LeftAmp:left ccd25:RightAmp:right + ccd26 STR ccd26:LeftAmp:left ccd26:RightAmp:right + ccd27 STR ccd27:LeftAmp:left ccd27:RightAmp:right + ccd28 STR ccd28:LeftAmp:left ccd28:RightAmp:right + ccd29 STR ccd29:LeftAmp:left ccd29:RightAmp:right + ccd30 STR ccd30:LeftAmp:left ccd30:RightAmp:right + ccd31 STR ccd31:LeftAmp:left ccd31:RightAmp:right + ccd32 STR ccd32:LeftAmp:left ccd32:RightAmp:right + ccd33 STR ccd33:LeftAmp:left ccd33:RightAmp:right + ccd34 STR ccd34:LeftAmp:left ccd34:RightAmp:right + ccd35 STR ccd35:LeftAmp:left ccd35:RightAmp:right +END + +# Specify the cells +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECA + CELL.TRIMSEC STR TSECA + CELL.X0 S32 0 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINA + END + + right METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECB + CELL.TRIMSEC STR TSECB + CELL.X0 S32 1024 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINB + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR CMMTOBS + FPA.FOCUS STR TELFOCUS + FPA.TIME STR MJD-OBS + FPA.TIMESYS STR TIMESYS + FPA.ALT STR TELALT + FPA.AZ STR TELAZ + FPA.TEMP STR DETTEM + FPA.EXPOSURE STR EXPTIME + CHIP.TEMP STR DETTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.TIMESYS STR TIMESYS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA + ccd00 S32 0 + ccd01 S32 2048 + ccd02 S32 4096 + ccd03 S32 6144 + ccd04 S32 8192 + ccd05 S32 10240 + ccd06 S32 12288 + ccd07 S32 14336 + ccd08 S32 16384 + ccd09 S32 0 + ccd10 S32 2048 + ccd11 S32 4096 + ccd12 S32 6144 + ccd13 S32 8192 + ccd14 S32 10240 + ccd15 S32 12288 + ccd16 S32 14336 + ccd17 S32 16384 + ccd18 S32 0 + ccd19 S32 2048 + ccd20 S32 4096 + ccd21 S32 6144 + ccd22 S32 8192 + ccd23 S32 10240 + ccd24 S32 12288 + ccd25 S32 14336 + ccd26 S32 16384 + ccd27 S32 0 + ccd28 S32 2048 + ccd29 S32 4096 + ccd30 S32 6144 + ccd31 S32 8192 + ccd32 S32 10240 + ccd33 S32 12288 + ccd34 S32 14336 + ccd35 S32 16384 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA + ccd00 S32 18447 + ccd01 S32 18447 + ccd02 S32 18447 + ccd03 S32 18447 + ccd04 S32 18447 + ccd05 S32 18447 + ccd06 S32 18447 + ccd07 S32 18447 + ccd08 S32 18447 + ccd09 S32 13835 + ccd10 S32 13835 + ccd11 S32 13835 + ccd12 S32 13835 + ccd13 S32 13835 + ccd14 S32 13835 + ccd15 S32 13835 + ccd16 S32 13835 + ccd17 S32 13835 + ccd18 S32 4612 + ccd19 S32 4612 + ccd20 S32 4612 + ccd21 S32 4612 + ccd22 S32 4612 + ccd23 S32 4612 + ccd24 S32 4612 + ccd25 S32 4612 + ccd26 S32 4612 + ccd27 S32 0 + ccd28 S32 0 + ccd29 S32 0 + ccd30 S32 0 + ccd31 S32 0 + ccd32 S32 0 + ccd33 S32 0 + ccd34 S32 0 + ccd35 S32 0 + END + CHIP.XPARITY S32 1 + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA + ccd00 S32 -1 + ccd01 S32 -1 + ccd02 S32 -1 + ccd03 S32 -1 + ccd04 S32 -1 + ccd05 S32 -1 + ccd06 S32 -1 + ccd07 S32 -1 + ccd08 S32 -1 + ccd09 S32 -1 + ccd10 S32 -1 + ccd11 S32 -1 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 + ccd15 S32 -1 + ccd16 S32 -1 + ccd17 S32 -1 + ccd18 S32 1 + ccd19 S32 1 + ccd20 S32 1 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 + ccd24 S32 1 + ccd25 S32 1 + ccd26 S32 1 + ccd27 S32 1 + ccd28 S32 1 + ccd29 S32 1 + ccd30 S32 1 + ccd31 S32 1 + ccd32 S32 1 + ccd33 S32 1 + ccd34 S32 1 + ccd35 S32 1 + END +END + + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + Index: /branches/rel-1-0/ippconfig/megacam/format_split.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/format_split.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/format_split.config (revision 11645) @@ -0,0 +1,259 @@ +# "mcshort" is a MegaCam camera with only the central six chips --- it's faster than the entire FPA. +# The spliced MecaCam data is stored in single extensions for each chip + +# How to recognise this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam +END + +FILE METADATA + # How to read this data + PHU STR CHIP # The FITS file represents an entire FPA + EXTENSIONS STR NONE # The extensions represent chips + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier + CHIP.NAME STR EXTNAME # An extension keyword for unique identifie + CONTENT STR {CHIP.NAME} # Key to the CONTENTS menu +END + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:component cell + ccd00 STR ccd00:LeftAmp:left ccd00:RightAmp:right + ccd01 STR ccd01:LeftAmp:left ccd01:RightAmp:right + ccd02 STR ccd02:LeftAmp:left ccd02:RightAmp:right + ccd03 STR ccd03:LeftAmp:left ccd03:RightAmp:right + ccd04 STR ccd04:LeftAmp:left ccd04:RightAmp:right + ccd05 STR ccd05:LeftAmp:left ccd05:RightAmp:right + ccd06 STR ccd06:LeftAmp:left ccd06:RightAmp:right + ccd07 STR ccd07:LeftAmp:left ccd07:RightAmp:right + ccd08 STR ccd08:LeftAmp:left ccd08:RightAmp:right + ccd09 STR ccd09:LeftAmp:left ccd09:RightAmp:right + ccd10 STR ccd10:LeftAmp:left ccd10:RightAmp:right + ccd11 STR ccd11:LeftAmp:left ccd11:RightAmp:right + ccd12 STR ccd12:LeftAmp:left ccd12:RightAmp:right + ccd13 STR ccd13:LeftAmp:left ccd13:RightAmp:right + ccd14 STR ccd14:LeftAmp:left ccd14:RightAmp:right + ccd15 STR ccd15:LeftAmp:left ccd15:RightAmp:right + ccd16 STR ccd16:LeftAmp:left ccd16:RightAmp:right + ccd17 STR ccd17:LeftAmp:left ccd17:RightAmp:right + ccd18 STR ccd18:LeftAmp:left ccd18:RightAmp:right + ccd19 STR ccd19:LeftAmp:left ccd19:RightAmp:right + ccd20 STR ccd20:LeftAmp:left ccd20:RightAmp:right + ccd21 STR ccd21:LeftAmp:left ccd21:RightAmp:right + ccd22 STR ccd22:LeftAmp:left ccd22:RightAmp:right + ccd23 STR ccd23:LeftAmp:left ccd23:RightAmp:right + ccd24 STR ccd24:LeftAmp:left ccd24:RightAmp:right + ccd25 STR ccd25:LeftAmp:left ccd25:RightAmp:right + ccd26 STR ccd26:LeftAmp:left ccd26:RightAmp:right + ccd27 STR ccd27:LeftAmp:left ccd27:RightAmp:right + ccd28 STR ccd28:LeftAmp:left ccd28:RightAmp:right + ccd29 STR ccd29:LeftAmp:left ccd29:RightAmp:right + ccd30 STR ccd30:LeftAmp:left ccd30:RightAmp:right + ccd31 STR ccd31:LeftAmp:left ccd31:RightAmp:right + ccd32 STR ccd32:LeftAmp:left ccd32:RightAmp:right + ccd33 STR ccd33:LeftAmp:left ccd33:RightAmp:right + ccd34 STR ccd34:LeftAmp:left ccd34:RightAmp:right + ccd35 STR ccd35:LeftAmp:left ccd35:RightAmp:right +END + +# Specify the cells +CELLS METADATA + left METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECA + CELL.TRIMSEC STR TSECA + CELL.X0 S32 0 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINA + END + + right METADATA + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BSECB + CELL.TRIMSEC STR TSECB + CELL.X0 S32 1024 + CELL.GAIN.SOURCE STR HEADER + CELL.GAIN STR GAINB + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.OBSTYPE STR OBSTYPE + FPA.OBJECT STR CMMTOBS + FPA.FOCUS STR TELFOCUS + FPA.TIME STR MJD-OBS + FPA.TIMESYS STR TIMESYS + FPA.ALT STR TELALT + FPA.AZ STR TELAZ + FPA.TEMP STR DETTEM + FPA.EXPOSURE STR EXPTIME + CHIP.TEMP STR DETTEM + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.TIMESYS STR TIMESYS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CELL.Y0 S32 0 + CHIP.X0.DEPEND STR CHIP.NAME + CHIP.X0 METADATA + ccd00 S32 0 + ccd01 S32 2048 + ccd02 S32 4096 + ccd03 S32 6144 + ccd04 S32 8192 + ccd05 S32 10240 + ccd06 S32 12288 + ccd07 S32 14336 + ccd08 S32 16384 + ccd09 S32 0 + ccd10 S32 2048 + ccd11 S32 4096 + ccd12 S32 6144 + ccd13 S32 8192 + ccd14 S32 10240 + ccd15 S32 12288 + ccd16 S32 14336 + ccd17 S32 16384 + ccd18 S32 0 + ccd19 S32 2048 + ccd20 S32 4096 + ccd21 S32 6144 + ccd22 S32 8192 + ccd23 S32 10240 + ccd24 S32 12288 + ccd25 S32 14336 + ccd26 S32 16384 + ccd27 S32 0 + ccd28 S32 2048 + ccd29 S32 4096 + ccd30 S32 6144 + ccd31 S32 8192 + ccd32 S32 10240 + ccd33 S32 12288 + ccd34 S32 14336 + ccd35 S32 16384 + END + CHIP.Y0.DEPEND STR CHIP.NAME + CHIP.Y0 METADATA + ccd00 S32 18447 + ccd01 S32 18447 + ccd02 S32 18447 + ccd03 S32 18447 + ccd04 S32 18447 + ccd05 S32 18447 + ccd06 S32 18447 + ccd07 S32 18447 + ccd08 S32 18447 + ccd09 S32 13835 + ccd10 S32 13835 + ccd11 S32 13835 + ccd12 S32 13835 + ccd13 S32 13835 + ccd14 S32 13835 + ccd15 S32 13835 + ccd16 S32 13835 + ccd17 S32 13835 + ccd18 S32 4612 + ccd19 S32 4612 + ccd20 S32 4612 + ccd21 S32 4612 + ccd22 S32 4612 + ccd23 S32 4612 + ccd24 S32 4612 + ccd25 S32 4612 + ccd26 S32 4612 + ccd27 S32 0 + ccd28 S32 0 + ccd29 S32 0 + ccd30 S32 0 + ccd31 S32 0 + ccd32 S32 0 + ccd33 S32 0 + ccd34 S32 0 + ccd35 S32 0 + END + CHIP.XPARITY S32 1 + CHIP.YPARITY.DEPEND STR CHIP.NAME + CHIP.YPARITY METADATA + ccd00 S32 -1 + ccd01 S32 -1 + ccd02 S32 -1 + ccd03 S32 -1 + ccd04 S32 -1 + ccd05 S32 -1 + ccd06 S32 -1 + ccd07 S32 -1 + ccd08 S32 -1 + ccd09 S32 -1 + ccd10 S32 -1 + ccd11 S32 -1 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 + ccd15 S32 -1 + ccd16 S32 -1 + ccd17 S32 -1 + ccd18 S32 1 + ccd19 S32 1 + ccd20 S32 1 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 + ccd24 S32 1 + ccd25 S32 1 + ccd26 S32 1 + ccd27 S32 1 + ccd28 S32 1 + ccd29 S32 1 + ccd30 S32 1 + ccd31 S32 1 + ccd32 S32 1 + ccd33 S32 1 + ccd34 S32 1 + ccd35 S32 1 + END +END + + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR MJD + CELL.TIME STR MJD +END + Index: /branches/rel-1-0/ippconfig/megacam/ppImage.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/ppImage.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/ppImage.config (revision 11645) @@ -0,0 +1,48 @@ +### ppImage recipe configuration file for Megacam + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +OVERSCAN.FIT STR NONE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 0 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# for a test, turn off selected detrend types +OVERSCAN BOOL FALSE # Overscan subtraction +BIAS BOOL FALSE # Bias subtraction +DARK BOOL TRUE # Dark subtraction +FLAT BOOL TRUE # Flat-field normalisation + +# megacam needs fringe subtraction +FRINGE BOOL TRUE # Fringe subtraction + +# this table lists extra constraints which should be applied when +# selecting the detrend images. +# note: camera and time are always applied + +DETREND.CONSTRAINTS METADATA + BIAS METADATA + DETTEMP STR FPA.TEMP + END + DARK METADATA + DETTEMP STR FPA.TEMP + EXPTIME STR FPA.EXPTIME + END + FLAT METADATA + FILTER STR FILTER.ID:FPA.FILTER + VERSION STR RAW + END + FRINGE METADATA + FILTER STR FPA.FILTER + AIRMASS STR FPA.AIRMASS + TWILIGHT STR TWILIGHT:FPA.TIME + END + SHUTTER METADATA + END +END + +# only apply the fringe for these filters +FRINGE.FILTERS MULTI +FRINGE.FILTERS STR r +FRINGE.FILTERS STR i +FRINGE.FILTERS STR z + Index: /branches/rel-1-0/ippconfig/megacam/ppMerge.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/ppMerge.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/ppMerge.config (revision 11645) @@ -0,0 +1,44 @@ + +# Bias combination --- don't want min/max rejection +PPMERGE_BIAS METADATA + REJ F32 3.0 # Rejection threshold (sigma) + ITER S32 2 # Number of rejection iterations + FRACHIGH F32 0.0 # Fraction of high pixels to reject immediately + FRACLOW F32 0.0 # Fraction of low pixels to reject immediately + WEIGHTS BOOL FALSE # Use image weights? + COMBINE STR CLIPPED # Statistic to use for combination: +END + +# Dark combination --- don't want min/max rejection +# More aggressive clipping than bias, so as to remove CRs +PPMERGE_DARK METADATA + REJ F32 2.0 # Rejection threshold (sigma) + ITER S32 4 # Number of rejection iterations + FRACHIGH F32 0.0 # Fraction of high pixels to reject immediately + FRACLOW F32 0.0 # Fraction of low pixels to reject immediately + WEIGHTS BOOL FALSE # Use image weights? + COMBINE STR CLIPPED # Statistic to use for combination: +END + +# Flat combination --- use min/max rejection +PPMERGE_FLAT METADATA + REJ F32 3.0 # Rejection threshold (sigma) + ITER S32 1 # Number of rejection iterations + FRACHIGH F32 0.3 # Fraction of high pixels to reject immediately + FRACLOW F32 0.1 # Fraction of low pixels to reject immediately + NKEEP S32 5 # Minimum number of pixels in stack to keep + MASKVAL S32 0xff # Mask value for input data + WEIGHTS BOOL TRUE # Use image weights? + COMBINE STR MEAN # Statistic to use for combination: +END + + +# Fringe combination --- already included in default, above +PPMERGE_FRINGE METADATA + FRACHIGH F32 0.1 # Fraction of high pixels to reject immediately + WEIGHTS BOOL TRUE # Use image weights? +END + +# Mask generation --- already included in default, above +PPMERGE_MASK METADATA +END Index: /branches/rel-1-0/ippconfig/megacam/ppStats.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/ppStats.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/ppStats.config (revision 11645) @@ -0,0 +1,12 @@ +### ppStats recipe +SAMPLE F32 0.1 # Fraction of cell to sample +MASKVAL U8 0xff # Mask value to use for statistics + +# Define the outputs as MULTI +HEADER MULTI +CONCEPT MULTI +STAT MULTI + +# Values to return +STAT STR ROBUST_MEDIAN # Background estimator +STAT STR ROBUST_STDEV # Background standard deviation estimator Index: /branches/rel-1-0/ippconfig/megacam/psastro.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/psastro.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/psastro.config (revision 11645) @@ -0,0 +1,36 @@ + +# astrometry matching parameters + +# nominal plate scale (microns / pixel) +# PSASTRO.PLATE.SCALE F32 13.5 +PSASTRO.PLATE.SCALE F32 1.0 + +# pmAstromGridMatch: +PSASTRO.GRID.MIN.ANGLE F32 -0.0851 +PSASTRO.GRID.MAX.ANGLE F32 +0.0851 +PSASTRO.GRID.DEL.ANGLE F32 0.0170 + +# pmAstromGridAngle +# max grid offset in FP units (microns) +# use plate-scale to make this in pixels? +# PSASTRO.GRID.OFFSET F32 10000. +# PSASTRO.GRID.SCALE F32 500. +PSASTRO.GRID.OFFSET F32 1000. +PSASTRO.GRID.SCALE F32 50 + +# these tweak are in FP units (pixels, currently) +PSASTRO.TWEAK.SCALE F32 1 +PSASTRO.TWEAK.RANGE F32 75 +PSASTRO.TWEAK.SMOOTH F32 2 +PSASTRO.TWEAK.NSIGMA F32 3 + +# pmAstromRadiusMatch +# use plate-scale to make this in pixels? +PSASTRO.MATCH.RADIUS F32 8 + +# pmAstromMatchFit +PSASTRO.CHIP.NX S32 1 # fit order +PSASTRO.CHIP.NY S32 1 # fit order +PSASTRO.CHIP.NITER S32 3 # fit clipping iterations +PSASTRO.CHIP.NSIGMA F32 3 # fit clipping sigmas + Index: /branches/rel-1-0/ippconfig/megacam/psphot.config =================================================================== --- /branches/rel-1-0/ippconfig/megacam/psphot.config (revision 11645) +++ /branches/rel-1-0/ippconfig/megacam/psphot.config (revision 11645) @@ -0,0 +1,84 @@ + +#SAVE.BACKMDL BOOL TRUE +#SAVE.BACKGND BOOL TRUE +#SAVE.BACKSUB BOOL TRUE +#SAVE.RESID BOOL TRUE +#SAVE.PSF BOOL TRUE +#LOAD.PSF BOOL FALSE + +# image noise parameters +RDNOISE STR HD:RDNOISE # read-noise in electrons +GAIN STR HD:GAIN # electrons / DN +ZERO_POINT F32 25.000 # zero point used by DVO + +# background model : +BACKGROUND.XBIN S32 128 +BACKGROUND.YBIN S32 128 + +# masking parameters +XMIN F32 2 # minimum valid x-coord +XMAX F32 -2 # maximum valid x-coord +YMIN F32 2 # minimum valid y-coord +YMAX F32 -2 # maximum valid y-coord +SATURATION F32 50000 # saturation level on this chip +MIN_VALID_PIXEL F32 1 # saturation level on this chip + +# image statistics parameters +IMSTATS_NPIX S32 1000 # number of pixels to use for sky estimate boxes: + +# peak finding +PEAKS_SMOOTH_SIGMA F32 1.0 # smoothing kernel sigma in pixels +PEAKS_SMOOTH_NSIGMA F32 3.0 # smoothing kernel width in sigmas +# PEAKS_NSIGMA_LIMIT F32 10.0 # peak significance threshold +PEAKS_NSIGMA_LIMIT F32 25.0 # peak significance threshold + +# basic object statistics +SKY_INNER_RADIUS F32 15 # square annulus for local sky measurement +SKY_OUTER_RADIUS F32 25 # square annulus for local sky measurement +PSF_MOMENTS_RADIUS F32 3 # calculate initial source moments with this radius +PSF_SN_LIM F32 100 # minimum S/N for stars used for PSF model +PSF_MAX_NSTARS S32 300 # limit number of stars used for PSF model + +# PSF model parameters : choose the PSF model +# list as many PSF_MODEL options as desired +# PSF_MODEL MULTI +PSF_MODEL STR PS_MODEL_QGAUSS +# PSF_MODEL STR PS_MODEL_GAUSS +# PSF_MODEL STR PS_MODEL_PGAUSS +# PSF_MODEL STR PS_MODEL_TGAUSS ## not well tested, not very successful + +MOMENTS_SCALE F32 0.1 +MOMENTS_SN_MIN F32 30.0 +MOMENTS_SX_MAX F32 3.0 +MOMENTS_SY_MAX F32 3.0 +MOMENTS_AR_MAX F32 1.5 # maximum axial ratio: 1 / AR < (sx / sy) < AR + +PSF_FIT_RADIUS F32 15.0 # fitting radius for test PSF model +PSF_REF_RADIUS F32 25.0 # aperture magnitudes are scaled via + +# PSF-like source model parameters +PSF_FIT_NSIGMA F32 1.0 # significance for pixel included in fit +PSF_FIT_PADDING F32 5.0 # extra annulus to use for fit +PSF_SHAPE_NSIGMA F32 3.0 # max significance for shape variation +PSF_MIN_SN F32 2.0 # reject objects below this significance +PSF_MAX_CHI F32 50.0 # reject objects worse that this + +# EXTended source model parameters +EXT_MODEL STR PS_MODEL_PGAUSS +EXT_MIN_SN F32 50.0 # fit galaxies above this S/N limit +EXT_FIT_NSIGMA F32 1 # significance for pixel included in fit +EXT_FIT_PADDING F32 5 # extra annulus to use for fit +EXT_MOMENTS_RADIUS F32 9 + +FITMODE STR BLEND +DEBLEND_PEAK_FRACTION F32 0.1 +DEBLEND_SKY_NSIGMA F32 10.0 + +APTREND STR SKYSAT +FULL_FIT_SN_LIM F32 50.0 +AP_MIN_SN F32 50.0 + +# BREAK_POINT STR PEAKS +# PEAKS_OUTPUT_FILE STR peaks.dat +# MOMENTS_OUTPUT_FILE STR moments.dat +# ANALYSIS_REGION STR [1000:1600,2800:3400] Index: /branches/rel-1-0/ippconfig/recipes/.cvsignore =================================================================== --- /branches/rel-1-0/ippconfig/recipes/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/ippconfig/recipes/Makefile.am =================================================================== --- /branches/rel-1-0/ippconfig/recipes/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/Makefile.am (revision 11645) @@ -0,0 +1,19 @@ + +installdir = $(datadir)/ippconfig/recipes + +install_files = \ + ppImage.config \ + ppMerge.config \ + ppStac.config \ + ppStats.config \ + psastro.config \ + psphot.config + +install_DATA = $(install_files) + +install-data-hook: + chmod 0755 $(installdir) + +EXTRA_DIST = $(install_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippconfig/recipes/ppImage.config =================================================================== --- /branches/rel-1-0/ippconfig/recipes/ppImage.config (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/ppImage.config (revision 11645) @@ -0,0 +1,328 @@ +### ppImage recipe configuration file +### Here we have turned on our 'best guess' defaults for a simple camera +### Camera-specific options may be turned on by recipes in the camera configuration + +# List of tasks to perform +NONLIN BOOL FALSE # Non-linearity correction; not implemented +OVERSCAN BOOL TRUE # Overscan subtraction +BIAS BOOL TRUE # Bias subtraction +DARK BOOL TRUE # Dark subtraction +SHUTTER BOOL FALSE # Shutter correction +FLAT BOOL TRUE # Flat-field normalisation +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0xff # only mask pixels matching this bitmask +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE # Astrometry per chip? +ASTROM.MOSAIC BOOL FALSE # Astrometry for mosaic? + +# output data formats to save +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL FALSE # Save 1st binned chip image? +BIN2.FITS BOOL FALSE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +NONLIN.DATA STR nonlin.dat # Filename for lookup table + +# examples of other possible non-linearity correction representations +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL TRUE # Reduce overscan to a single value? +OVERSCAN.FIT STR NONE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# Fringe subtraction options +FRINGE.ITER S32 10 # Number of rejection iterations for fringe solution +FRINGE.REJ F32 2.0 # Rejection threshold for fringe solution +FRINGE.KEEP F32 0.5 # Minimum fraction to keep in fringe solution + +# binned output image options +BIN1.XBIN S32 4 +BIN1.YBIN S32 4 +BIN2.XBIN S32 16 +BIN2.YBIN S32 16 + +PPIMAGE.JPEG1 METADATA + COLORMAP STR -greyscale + SCALE.MODE STR RANGE + SCALE.MIN F32 -5.0 + SCALE.MAX F32 20.0 +END + +PPIMAGE.JPEG2 METADATA + COLORMAP STR +greyscale + SCALE.MODE STR FRACTION + SCALE.MIN STR 0.50 + SCALE.MAX STR 2.00 +END + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} + +################################################################################ +# Diffferent processing options, which may be loaded symbolically +################################################################################ + +# No operation except potential normalisation +PPIMAGE_N METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Overscan subtraction only +PPIMAGE_O METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Bias subtraction only +PPIMAGE_B METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Dark subtraction only +PPIMAGE_D METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Shutter correction only +PPIMAGE_S METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Flat-fielding only +PPIMAGE_F METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Fringe correction only +PPIMAGE_R METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Photometry only +PPIMAGE_P METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + PHOTOM BOOL TRUE # Source identification and photometry +END + +# Photometry & Astrometry +PPIMAGE_A METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + PHOTOM BOOL TRUE # Source identification and photometry + ASTROM.CHIP BOOL TRUE # Source identification and photometry +END + +# Overscan, bias +PPIMAGE_OB METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Overscan, bias, dark +PPIMAGE_OBD METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Overscan, bias, dark, shutter +PPIMAGE_OBDS METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Overscan, bias, dark, shutter, flat-field +PPIMAGE_OBDSF METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL TRUE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Overscan, bias, dark, shutter, flat-field, fringe +PPIMAGE_OBDSFR METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL TRUE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +END + +# Overscan, bias, dark, shutter, flat-field, fringe, photom +PPIMAGE_OBDSFRP METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL TRUE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + PHOTOM BOOL TRUE # Source identification and photometry + ASTROM.CHIP BOOL FALSE # Astrometry per chip? + ASTROM.MOSAIC BOOL FALSE # Astrometry for mosaic? +END + +# Overscan, bias, dark, shutter, flat-field, fringe, photom, astrom +PPIMAGE_OBDSFRA METADATA + OVERSCAN BOOL TRUE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL TRUE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL TRUE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction + BIN1.FITS BOOL TRUE # Save 1st binned chip image? + BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + PHOTOM BOOL TRUE # Source identification and photometry + ASTROM.CHIP BOOL TRUE # Astrometry per chip? + ASTROM.MOSAIC BOOL FALSE # Astrometry for mosaic? +END + +# Save JPEG from BIN1 +PPIMAGE_J1 METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BASE.FITS BOOL FALSE # Save base image? + BIN1.FITS BOOL FALSE # Save 1st binned chip image? + BIN2.FITS BOOL FALSE # Save 2nd binned chip image? + BIN1.JPEG BOOL TRUE # Save 1st binned jpeg? + BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + BIN1.XBIN S32 1 # Image is already binned + BIN1.YBIN S32 1 # Image is already binned + BIN2.XBIN S32 1 # Image is already binned + BIN2.YBIN S32 1 # Image is already binned +END + +# Save JPEG from BIN2 +PPIMAGE_J2 METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction + BASE.FITS BOOL FALSE # Save base image? + BIN1.FITS BOOL FALSE # Save 1st binned chip image? + BIN2.FITS BOOL FALSE # Save 2nd binned chip image? + BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? + BIN2.JPEG BOOL TRUE # Save 2nd binned jpeg? + BIN1.XBIN S32 1 # Image is already binned + BIN1.YBIN S32 1 # Image is already binned + BIN2.XBIN S32 1 # Image is already binned + BIN2.YBIN S32 1 # Image is already binned +END Index: /branches/rel-1-0/ippconfig/recipes/ppMerge.config =================================================================== --- /branches/rel-1-0/ippconfig/recipes/ppMerge.config (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/ppMerge.config (revision 11645) @@ -0,0 +1,71 @@ +# Recipe configuration for ppMerge + +ROWS S32 512 # Number of rows to read at once +ELECTRONS F32 100.0 # Minimum number of electrons for useful signal +SAMPLE S32 100 # Sampling factor for measuring the background +REJ F32 3.0 # Rejection threshold (sigma) +ITER S32 1 # Number of rejection iterations +FRACHIGH F32 0.0 # Fraction of high pixels to reject immediately +FRACLOW F32 0.0 # Fraction of low pixels to reject immediately +NKEEP S32 5 # Minimum number of pixels in stack to keep +FRINGE.NUM S32 10000 # Number of fringe regions +FRINGE.SIZE S32 5 # Half-size of fringe regions +FRINGE.XSMOOTH S32 5 # Number of smoothing regions in x +FRINGE.YSMOOTH S32 11 # Number of smoothing regions in y +SHUTTER.SIZE S32 128 # Size for shutter measurement regions +SHUTTER.ITER S32 1 # Number of iterations for shutter measurement +SHUTTER.REJECT F32 2 # Rejection limit for shutter measurement +MASK.SUSPECT F32 5.0 # Threshold for suspect pixels (sigma) +MASK.BAD F32 -4.0 # Threshold for bad pixels (sigma) +MASKVAL S32 0xff # Mask value for input data +COMBINE STR CLIPPED # Statistic to use for combination +MEAN STR ROBUST_MEDIAN # Statistic to use to measure the mean +STDEV STR ROBUST_STDEV # Statistic to use to measure the stdev + + +# Bias combination --- don't want min/max rejection +PPMERGE_BIAS METADATA + REJ F32 3.0 # Rejection threshold (sigma) + ITER S32 2 # Number of rejection iterations + FRACHIGH F32 0.0 # Fraction of high pixels to reject immediately + FRACLOW F32 0.0 # Fraction of low pixels to reject immediately + WEIGHTS BOOL FALSE # Use image weights? + MASKVAL S32 0 # Mask value for input data + COMBINE STR CLIPPED # Statistic to use for combination: +END + + +# Dark combination --- don't want min/max rejection +# More aggressive clipping than bias, so as to remove CRs +PPMERGE_DARK METADATA + REJ F32 2.0 # Rejection threshold (sigma) + ITER S32 4 # Number of rejection iterations + FRACHIGH F32 0.0 # Fraction of high pixels to reject immediately + FRACLOW F32 0.0 # Fraction of low pixels to reject immediately + WEIGHTS BOOL FALSE # Use image weights? + MASKVAL S32 0 # Mask value for input data + COMBINE STR CLIPPED # Statistic to use for combination: +END + +# Flat combination --- use min/max rejection +PPMERGE_FLAT METADATA + REJ F32 3.0 # Rejection threshold (sigma) + ITER S32 1 # Number of rejection iterations + FRACHIGH F32 0.3 # Fraction of high pixels to reject immediately + FRACLOW F32 0.1 # Fraction of low pixels to reject immediately + NKEEP S32 5 # Minimum number of pixels in stack to keep + MASKVAL S32 0xff # Mask value for input data + WEIGHTS BOOL TRUE # Use image weights? + COMBINE STR MEAN # Statistic to use for combination: +END + + +# Fringe combination --- already included in default, above +PPMERGE_FRINGE METADATA + FRACHIGH F32 0.1 # Fraction of high pixels to reject immediately + WEIGHTS BOOL TRUE # Use image weights? +END + +# Mask generation --- already included in default, above +PPMERGE_MASK METADATA +END Index: /branches/rel-1-0/ippconfig/recipes/ppStac.config =================================================================== --- /branches/rel-1-0/ippconfig/recipes/ppStac.config (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/ppStac.config (revision 11645) @@ -0,0 +1,5 @@ +# Recipe configuration for ppMerge +MASKVAL S32 0xff # Mask value for input data +COMBINE STR MEAN # Statistic to use for combination +REJ F32 3.0 # Rejection threshold (sigma) +ITER S32 1 # Number of rejection iterations Index: /branches/rel-1-0/ippconfig/recipes/ppStats.config =================================================================== --- /branches/rel-1-0/ippconfig/recipes/ppStats.config (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/ppStats.config (revision 11645) @@ -0,0 +1,41 @@ +### Default ppStats recipe + +# Options governing statistics +SAMPLE F32 0.1 # Fraction of cell to sample +MASKVAL U8 0xff # Mask value to use for statistics + +# Define the outputs as MULTI +HEADER MULTI +CONCEPT MULTI +STAT MULTI + +# Values to return +STAT STR ROBUST_MEDIAN # Background estimator +STAT STR ROBUST_STDEV # Background standard deviation estimator + +PPSTATS_PHASE0 METADATA + ### ppStats recipe for Phase 0 with MegaCam + + # Options governing statistics + SAMPLE F32 0.1 # Fraction of cell to sample + MASKVAL U8 0xff # Mask value to use for statistics + + # Define the outputs as MULTI + HEADER MULTI + CONCEPT MULTI + STAT MULTI + + # Values to return + CONCEPT STR FPA.OBJECT # Object name + CONCEPT STR FPA.OBSTYPE # Observation type + CONCEPT STR FPA.FILTER # Filter + CONCEPT STR FPA.RA FPA.DEC # Telescope pointing + CONCEPT STR FPA.AIRMASS # Airmass + CONCEPT STR FPA.ALT FPA.AZ # Telescopy alt/az + CONCEPT STR FPA.POSANGLE # Rotator angle + CONCEPT STR FPA.TIME # Time of exposure + CONCEPT STR CHIP.TEMP # Detector temperature + CONCEPT STR CELL.EXPOSURE # Exposure time + STAT STR ROBUST_MEDIAN # Background estimator + STAT STR ROBUST_STDEV # Background standard deviation estimator +END Index: /branches/rel-1-0/ippconfig/recipes/psastro.config =================================================================== --- /branches/rel-1-0/ippconfig/recipes/psastro.config (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/psastro.config (revision 11645) @@ -0,0 +1,58 @@ + +# perform single-chip astrometry? +PSASTRO.CHIP.MODE BOOL TRUE + +# nominal plate scale (microns / pixel) +PSASTRO.PIXEL.SCALE F32 13.5 + +# pmAstromGridMatch: +PSASTRO.GRID.MIN.ANGLE F32 -0.0851 +PSASTRO.GRID.MAX.ANGLE F32 +0.0851 +PSASTRO.GRID.DEL.ANGLE F32 0.0170 + +# pmAstromGridAngle +# max grid offset in FP units (microns) +# use plate-scale to make this in pixels? +# PSASTRO.GRID.OFFSET F32 10000. +# PSASTRO.GRID.SCALE F32 500. +PSASTRO.GRID.OFFSET F32 1000. +PSASTRO.GRID.SCALE F32 50 + +# these tweak are in FP units (pixels, currently) +PSASTRO.TWEAK.SCALE F32 1 +PSASTRO.TWEAK.RANGE F32 75 +PSASTRO.TWEAK.SMOOTH F32 2 +PSASTRO.TWEAK.NSIGMA F32 3 + +# single-chip radius match in pixels +PSASTRO.MATCH.RADIUS F32 8 + +# pmAstromMatchFit +PSASTRO.CHIP.ORDER S32 1 # fit order +PSASTRO.CHIP.NITER S32 3 # fit clipping iterations +PSASTRO.CHIP.NSIGMA F32 3 # fit clipping sigmas + +PSASTRO.MAX.ERROR F32 1.5 # max allow error for valid solution (UNITS?) +PSASTRO.MIN.NSTAR S32 3 # min fitted stars in solution +PSASTRO.MAX.NSTAR S32 300 # max stars accepted for fitting + +MAG_MAX F32 23.0 ## XXX change name to PSASTRO.something? + +# option may be MAX, MIN, or VALUE. for VALUE, look up +# plate scale for each chip in the concepts +PSASTRO.COMMON.SCALE.OPTION STR MAX + +# mosaic-mode radius match in pixels? +PSASTRO.MOSAIC.RADIUS.N0 F32 8 +PSASTRO.MOSAIC.RADIUS.N1 F32 6 +PSASTRO.MOSAIC.RADIUS.N2 F32 4 + +# Mosaic Astrometry options +PSASTRO.MOSAIC.MODE BOOL FALSE +PSASTRO.MOSAIC.CHIP.ORDER S32 3 # fit order +PSASTRO.MOSAIC.ORDER S32 3 # fit order +PSASTRO.MOSAIC.CHIP.NITER S32 3 # fit clipping iterations +PSASTRO.MOSAIC.CHIP.NSIGMA F32 3 # fit clipping sigmas + +PSASTRO.MOSAIC.MAX.ERROR F32 1.5 # max allow error for valid solution (UNITS?) +PSASTRO.MOSAIC.MIN.NSTAR S32 6 # min fitted stars in solution Index: /branches/rel-1-0/ippconfig/recipes/psphot.config =================================================================== --- /branches/rel-1-0/ippconfig/recipes/psphot.config (revision 11645) +++ /branches/rel-1-0/ippconfig/recipes/psphot.config (revision 11645) @@ -0,0 +1,116 @@ + +# these options turn on different inputs and/or outputs +SAVE.BACKMDL BOOL FALSE +SAVE.BACKGND BOOL FALSE +SAVE.BACKSUB BOOL FALSE +SAVE.RESID BOOL FALSE +SAVE.PSF BOOL FALSE +LOAD.PSF BOOL FALSE +SAVE.PLOTS BOOL FALSE + +# the zero point is used to set a basic scale for DVO +# XXX it may not currently be read : double check this (EAM) +ZERO_POINT F32 25.000 # zero point used by DVO + +# these parameter govern how the background is measured +BACKGROUND_XBIN S32 128 # size of background superpixels +BACKGROUND_YBIN S32 128 # size of background superpixels +IMSTATS_NPIX S32 10000 # number of pixels to use for sky estimate boxes: + +SKY_BIAS F32 0.0 # offset applied to measured sky (FOR TESTING) +SKY_FIT_ORDER S32 0 +SKY_FIT_LINEAR BOOL FALSE +SKY_STAT STR FITTED_MEAN_V3 # statistic used to measure background +# allowed values for SKY_STAT: +# SAMPLE_MEAN, SAMPLE_MEDIAN, CLIPPED_MEAN, ROBUST_MEAN, ROBUST_QUARTILE, FITTED_MEAN + +# masking parameters (XXX EAM : rework this to use psRegion like ANALYSIS_REGION) +XMIN F32 0 # minimum valid x-coord +XMAX F32 0 # maximum valid x-coord +YMIN F32 0 # minimum valid y-coord +YMAX F32 0 # maximum valid y-coord + +# peak finding +PEAKS_SMOOTH_SIGMA F32 1.0 # smoothing kernel sigma in pixels +PEAKS_SMOOTH_NSIGMA F32 2.0 # smoothing kernel width in sigmas +PEAKS_NSIGMA_LIMIT F32 25.0 # peak significance threshold +PEAKS_NSIGMA_LIMIT_2 F32 5.0 # peak significance threshold + +# parameters to control the selection of the peak in the Sx,Sy plane +MOMENTS_SCALE F32 0.1 +MOMENTS_SN_MIN F32 100.0 # min S/N to measure moments +MOMENTS_SX_MAX F32 3.0 +MOMENTS_SY_MAX F32 3.0 +MOMENTS_AR_MAX F32 1.5 # maximum axial ratio: 1 / AR < (sx / sy) < AR + +# basic object statistics +SKY_INNER_RADIUS F32 15 # square annulus for local sky measurement +SKY_OUTER_RADIUS F32 25 # square annulus for local sky measurement +PSF_MOMENTS_RADIUS F32 3 # calculate initial source moments with this radius +PSF_SN_LIM F32 50 # minimum S/N for stars used for PSF model +PSF_MAX_NSTARS S32 300 # limit number of stars used for PSF model + +# PSF model parameters : choose the PSF model +# list as many PSF_MODEL options as desired +# PSF_MODEL MULTI +PSF_MODEL STR PS_MODEL_GAUSS +# PSF_MODEL STR PS_MODEL_PGAUSS +# PSF_MODEL STR PS_MODEL_QGAUSS +# PSF_MODEL STR PS_MODEL_TGAUSS ## not well tested, not very successful + +# PSF.TREND.MASK must be a 2D polynomial +# the specified values are ignored but define the active components of the polynomial +PSF.TREND.MASK METADATA + NORDER_X S32 2 # number of x orders + NORDER_Y S32 2 # number of y orders + VAL_X00_Y00 F64 1 # polynomial coefficient + VAL_X01_Y00 F64 1 # polynomial coefficient + VAL_X00_Y01 F64 1 # polynomial coefficient + VAL_X02_Y00 F64 1 # polynomial coefficient + VAL_X01_Y01 F64 1 # polynomial coefficient + VAL_X00_Y02 F64 1 # polynomial coefficient + NELEMENTS S32 6 # number of unmasked components +END # folder for 2D polynomial + +PSF_FIT_RADIUS F32 15.0 # fitting radius for test PSF model +PSF_REF_RADIUS F32 25.0 # aperture magnitudes are scaled via + # curve-of-growth to this radius +# PSF-like source model parameters +PSF_FIT_NSIGMA F32 1.0 # significance for pixel included in fit +PSF_FIT_PADDING F32 2.0 # extra annulus to use for fit +PSF_SHAPE_NSIGMA F32 3.0 # max significance for shape variation +PSF_MIN_SN F32 2.0 # reject objects below this significance +PSF_MAX_CHI F32 50.0 # reject objects worse that this +FULL_FIT_SN_LIM F32 50.0 + +# EXTended source model parameters +EXT_MODEL STR PS_MODEL_PGAUSS +EXT_MIN_SN F32 50.0 # fit galaxies above this S/N limit +EXT_FIT_NSIGMA F32 1 # significance for pixel included in fit +EXT_FIT_PADDING F32 5 # extra annulus to use for fit +EXT_MOMENTS_RADIUS F32 9 + +FITMODE STR BLEND +DEBLEND_PEAK_FRACTION F32 0.1 +DEBLEND_SKY_NSIGMA F32 10.0 + +# APTREND STR NONE, CONSTANT, SKYBIAS, SKYSAT, XY_LIN, SKY_XY_LIN, SKYSAT_XY_LIN, ALL +APTREND STR CONSTANT +AP_MIN_SN F32 25.0 + +# test options +# BREAK_POINT may be one of (PEAKS, MOMENTS, PSFMODEL, ENSEMBLE) +# BREAK_POINT STR ENSEMBLE +# PEAKS_OUTPUT_FILE STR peaks.dat +# MOMENTS_OUTPUT_FILE STR moments.dat +# ANALYSIS_REGION STR [1000:1600,2800:3400] +# POISSON_ERRORS BOOL TRUE + +# optional parameter to limit the actual analysis to a fraction of the image +# do not uncomment this in the master psphot.config file +# ANALYSIS_REGION STR [1000:1600,2800:3400] + +IGNORE_GROWTH BOOL FALSE +CONSTANT_PHOTOMETRIC_WEIGHTS BOOL TRUE # Should the photometric code [currently only ensemblePSF] refuse to weight each pixel by it's significance? +INTERPOLATE_AP BOOL TRUE +POISSON_ERRORS BOOL TRUE Index: /branches/rel-1-0/ippconfig/simple/.cvsignore =================================================================== --- /branches/rel-1-0/ippconfig/simple/.cvsignore (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/.cvsignore (revision 11645) @@ -0,0 +1,2 @@ +Makefile +Makefile.in Index: /branches/rel-1-0/ippconfig/simple/Makefile.am =================================================================== --- /branches/rel-1-0/ippconfig/simple/Makefile.am (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/Makefile.am (revision 11645) @@ -0,0 +1,18 @@ + +installdir = $(datadir)/ippconfig/simple + +install_files = \ + camera.config \ + format.config \ + ppImage.config \ + psastro.config \ + psphot.config + +install_DATA = $(install_files) + +install-data-hook: + chmod 0755 $(installdir) + +EXTRA_DIST = $(install_files) + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/rel-1-0/ippconfig/simple/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/simple/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/camera.config (revision 11645) @@ -0,0 +1,137 @@ +# Camera configuration file for drop-dead boring camera, consisting of a single chip with single cell. + +# File formats that we know about +FORMATS METADATA + SIMPLE STR simple/format.config +END + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + Chip STR Cell +END + +# move this elsewhere? we need a lookup table to go from filter ID to abstract name +FILTER.ID METADATA + B STR B + V STR V + R STR R + I STR I + z STR z +END + +# Recipe options +RECIPES METADATA + PPIMAGE STR simple/ppImage.config # Default: all (normal) options on + PSPHOT STR simple/psphot.config # psphot details +END + +# Rejection levels for detrend creation +REJECTION METADATA + TYPE LIMITS FILTER EXPECTED IMFILE.MEAN IMFILE.STDEV EXP.MEAN EXP.STDEV EXP.MEANSTDEV ENSEMBLE.MEAN ENSEMBLE.STDEV ENSEMBLE.MEANSTDEV IMFILE.SN EXP.SN IMFILE.MEANSTDEV + FLAT MULTI + + BIAS LIMITS * 0 1 6 0.5 6 0 3 6 0 0 0 0.5 + DARK LIMITS * 0 1 4 0.5 4 0 3 4 0 0 0 0.5 + FLAT LIMITS * 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS u 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS g 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS r 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS i 0 0 0 0 0 0 0 0 3 0 0 0 +# FLAT LIMITS z 0 0 0 0 0 0 0 0 3 0 0 0 + FRINGE LIMITS * 0 0 0 0 0 0 0 0 0 0 0 0 + +# FILTER is an additional qualifier, and may be "*" (or absent!), in which case it matches everything +# EXPECTED is the expected mean value +# IMFILE.MEAN is the maximum permitted mean value for an imfile, relative to the standard deviation +# IMFILE.STDEV is the maximum permitted standard deviation for an imfile +# EXP.MEAN is the maximum permitted mean value for an exposure, relative to the standard deviation +# EXP.STDEV is the maximum permitted standard deviation for an exposure +# EXP.MEANSTDEV is the maximum permitted mean standard deviation for an exposure relative to the mean +# ENSEMBLE.MEAN is the maximum permitted mean for an ensemble of exposures +# ENSEMBLE.STDEV is the maximum permitted standard deviation for an ensemble of exposures +# ENSEMBLE.MEANSTDEV is the maximum permitted mean standard deviation for an ensemble of exposures +# IMFILE.SN is the minimum permitted signal-to-noise for an imfile +# EXP.SN is the minimum permitted signal-to-noise for an exposure +# These values (all except FILTER) may be zero, in which case no clipping is applied. + +END + +FILERULES METADATA + PSASTRO.INPUT STR PSASTRO.INPUT.CMP + PSASTRO.OUTPUT STR PSASTRO.OUTPUT.CMP + PSPHOT.OUTPUT STR PSPHOT.OUTPUT.CMF + + ### input file definitions + TYPE INPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA DATA.LEVEL FILE.TYPE + PPIMAGE.INPUT INPUT none.fits NONE NONE NONE FPA IMAGE + INPUT.MASK INPUT none.fits NONE NONE NONE FPA MASK + INPUT.WEIGHT INPUT none.fits NONE NONE NONE FPA WEIGHT + INPUT.PSF INPUT none.fits NONE NONE NONE READOUT PSF + INPUT.SRC INPUT none.fits NONE SMPDATA PHU READOUT CMF + PPIMAGE.BIAS INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.DARK INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.FLAT INPUT @DETDB NONE fpa NONE CHIP IMAGE + PPIMAGE.MASK INPUT @DETDB NONE fpa NONE CHIP IMAGE + PSPHOT.INPUT INPUT none.fits NONE NONE NONE CHIP IMAGE + PSWARP.INPUT INPUT none.fits NONE NONE NONE FPA IMAGE + PSWARP.SKYCELL INPUT none.fits NONE NONE NONE FPA IMAGE + PSASTRO.INPUT.CMP INPUT none.fits NONE NONE PHU CHIP CMP + PSASTRO.INPUT.CMF INPUT none.fits NONE SMPDATA PHU CHIP CMF + PSPHOT.PSF.LOAD INPUT {OUTPUT}.psf NONE NONE NONE CHIP PSF + + ### output file definitions + TYPE OUTPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA FILE.TYPE FILE.LEVEL DATA.LEVEL FILE.SAVE FILE.FORMAT + PPIMAGE.OUTPUT OUTPUT {OUTPUT}.fits NONE NONE NONE IMAGE FPA FPA TRUE NONE + PPIMAGE.OUTPUT.MASK OUTPUT {OUTPUT}.mask.fits NONE NONE NONE MASK FPA FPA FALSE NONE + PPIMAGE.OUTPUT.WEIGHT OUTPUT {OUTPUT}.wt.fits NONE NONE NONE WEIGHT FPA FPA FALSE NONE + PPIMAGE.OUTPUT.CHIP OUTPUT {OUTPUT}.ch.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PPIMAGE.OUTPUT.FPA1 OUTPUT {OUTPUT}.b1.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PPIMAGE.OUTPUT.FPA2 OUTPUT {OUTPUT}.b2.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + + PPIMAGE.JPEG1 OUTPUT {OUTPUT}.b1.jpg NONE NONE NONE JPEG FPA FPA TRUE SIMPLE + PPIMAGE.JPEG2 OUTPUT {OUTPUT}.b2.jpg NONE NONE NONE JPEG FPA FPA TRUE SIMPLE + PPIMAGE.BIN1 OUTPUT {OUTPUT}.b1.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PPIMAGE.BIN2 OUTPUT {OUTPUT}.b2.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + + PSWARP.OUTPUT OUTPUT {OUTPUT}.fits NONE NONE NONE IMAGE FPA FPA TRUE NONE + + PSPHOT.RESID OUTPUT {OUTPUT}.res.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PSPHOT.BACKGND OUTPUT {OUTPUT}.bck.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PSPHOT.BACKSUB OUTPUT {OUTPUT}.sub.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PSPHOT.BACKMDL OUTPUT {OUTPUT}.mdl.fits NONE NONE NONE IMAGE FPA FPA TRUE SIMPLE + PSPHOT.OUTPUT.RAW OUTPUT {OUTPUT} NONE NONE NONE RAW FPA FPA TRUE NONE + PSPHOT.OUTPUT.SX OUTPUT {OUTPUT}.sx NONE NONE NONE SX FPA FPA TRUE NONE + PSPHOT.OUTPUT.OBJ OUTPUT {OUTPUT}.obj NONE NONE NONE OBJ FPA FPA TRUE NONE + PSPHOT.OUTPUT.CMP OUTPUT {OUTPUT}.cmp NONE NONE NONE CMP FPA FPA TRUE NONE + PSPHOT.OUTPUT.CMF OUTPUT {OUTPUT}.cmf NONE SMPDATA PHU CMF FPA FPA TRUE NONE + PSPHOT.PSF.SAVE OUTPUT {OUTPUT}.psf NONE NONE NONE PSF FPA FPA TRUE NONE + + PSPHOT.MOMENT.PLT OUTPUT {OUTPUT}.mnt.png NONE NONE NONE KAPA FPA FPA TRUE SIMPLE + PSPHOT.PSFMODEL.PLT OUTPUT {OUTPUT}.psf.png NONE NONE NONE KAPA FPA FPA TRUE SIMPLE + + PSASTRO.OUTPUT.CMP OUTPUT {OUTPUT}.smp NONE NONE NONE CMP FPA FPA TRUE NONE +END + +# FPA file defines properties of a possible input|output object +# user can set the filename (I|O), filename rules (O), or abstract source (@FILES, @DETDB) (I) +# user can set the extension name, if used +# user can set the file type (IMAGE, JPEG, RAW, SX, OBJ, CMP, CMF) : but these are not variable in most cases! +# user can set the file mode (READ, WRITE), but these must be set by the programmer +# user can set the file depth: only valid for output files +# user can set the data depth: must be >= file depth +# user can set the file format: only valid for newly created FPAs +# user can set the colormap, scaling method, scaling range (JPEG only) +# user can set the extension name for the data and header segments (CMF only) + +# known uses of the filename.xtra, extname.rule, and extname.xtra: +# extname.rule: used to construct output EXTNAME keys (CMF DATA) +# extname.xtra: used to construct output EXTNAME keys (CMF HEAD) + +# extname.rule: used to carry classID to DetDB +# filename.xtra: used to carry detID to DetDB (internal) + +# filename.xtra: used to carry arglist name (internal) + +# filename.xtra: used to define the JPEG colormap +# extname.rule: used to define the JPEG scaling method +# extname.xtra: used to carry arguments of JPEG scaling method Index: /branches/rel-1-0/ippconfig/simple/format.config =================================================================== --- /branches/rel-1-0/ippconfig/simple/format.config (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/format.config (revision 11645) @@ -0,0 +1,75 @@ +# Drop-dead simple camera with single chip with single cell + +# How to identify this type +RULE METADATA + SIMPLE BOOL TRUE + NAXIS S32 2 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR NONE # There are no extensions + FPA.NAME STR NAXIS # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS STR Chip:Cell:amplifier + +# Specify the cell data +CELLS METADATA + amplifier METADATA + CELL.TRIMSEC.SOURCE STR VALUE + CELL.TRIMSEC STR [0:0,0:0] + # alternatives for getting these from the headers + # CELL.TRIMSEC.SOURCE STR HEADER + # CELL.BIASSEC.SOURCE STR VALUE + # CELL.TRIMSEC STR DATASEC + # CELL.BIASSEC STR BIASSEC + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.TELESCOPE STR TELESCOP + FPA.INSTRUMENT STR INSTRUME + FPA.OBSTYPE STR OBSTYPE + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR POSANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.OBJECT STR OBJECT + FPA.TIME STR DATE-OBS UTC-OBS # Date and time + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME +# CELL.TIME STR DATE-OBS TIME-OBS # Date and time + CELL.XBIN STR CCDSUM + CELL.YBIN STR CCDSUM +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.TIMESYS STR UTC + FPA.RADECSYS STR ICRS + CELL.GAIN F32 1.0 + CELL.READNOISE F32 2.0 + CELL.SATURATION F32 46000 + CELL.BAD F32 0 +# CELL.SATURATION F32 65535 +# CELL.BAD F32 -65535 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 + CHIP.XPARITY S32 1 + CHIP.YPARITY S32 1 + CELL.X0 S32 0 + CELL.Y0 S32 0 +END + +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + FPA.TIME STR SEPARATE,YEAR.FIRST +END Index: /branches/rel-1-0/ippconfig/simple/ppImage.config =================================================================== --- /branches/rel-1-0/ippconfig/simple/ppImage.config (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/ppImage.config (revision 11645) @@ -0,0 +1,119 @@ +### ppImage recipe configuration file for drop-dead simple camera + +# List of tasks to perform +SHUTTER BOOL FALSE # Shutter correction +DARK BOOL TRUE # Dark correction + +BASE.FITS BOOL TRUE # Save base detrended image? +BIN1.FITS BOOL TRUE # Save 1st binned chip image? +BIN2.FITS BOOL TRUE # Save 2nd binned chip image? + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 3 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# How to select the appropriate detrend image +DETREND.CONSTRAINTS METADATA + BIAS METADATA + END + DARK METADATA + EXPTIME STR FPA.EXPTIME + END + FLAT METADATA + FILTER STR FPA.FILTER + END + FRINGE METADATA + FILTER STR FPA.FILTER + END + SHUTTER METADATA + END +END + +######################################################################################## +# Need the following in order to turn off overscan and shutter for the site-level recipe +######################################################################################## + +# Overscan subtraction only +PPIMAGE_O METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Dark subtraction only +PPIMAGE_D METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Shutter correction only +PPIMAGE_S METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL FALSE # Bias subtraction + DARK BOOL FALSE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark +PPIMAGE_OBD METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark, shutter +PPIMAGE_OBDS METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL FALSE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark, shutter, flat-field +PPIMAGE_OBDSF METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL FALSE # Fringe subtraction +END + +# Overscan, bias, dark, shutter, flat-field, fringe +PPIMAGE_OBDSFR METADATA + OVERSCAN BOOL FALSE # Overscan subtraction + BIAS BOOL TRUE # Bias subtraction + DARK BOOL TRUE # Dark subtraction + SHUTTER BOOL FALSE # Shutter correction + FLAT BOOL TRUE # Flat-field normalisation + MASK BOOL FALSE # Mask bad pixels + FRINGE BOOL TRUE # Fringe subtraction +END Index: /branches/rel-1-0/ippconfig/simple/psastro.config =================================================================== --- /branches/rel-1-0/ippconfig/simple/psastro.config (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/psastro.config (revision 11645) @@ -0,0 +1,36 @@ + +# astrometry matching parameters + +# nominal plate scale (microns / pixel) +# PSASTRO.PLATE.SCALE F32 13.5 +PSASTRO.PLATE.SCALE F32 1.0 + +# pmAstromGridMatch: +PSASTRO.GRID.MIN.ANGLE F32 -0.0851 +PSASTRO.GRID.MAX.ANGLE F32 +0.0851 +PSASTRO.GRID.DEL.ANGLE F32 0.0170 + +# pmAstromGridAngle +# max grid offset in FP units (microns) +# use plate-scale to make this in pixels? +# PSASTRO.GRID.OFFSET F32 10000. +# PSASTRO.GRID.SCALE F32 500. +PSASTRO.GRID.OFFSET F32 1000. +PSASTRO.GRID.SCALE F32 50 + +# these tweak are in FP units (pixels, currently) +PSASTRO.TWEAK.SCALE F32 1 +PSASTRO.TWEAK.RANGE F32 75 +PSASTRO.TWEAK.SMOOTH F32 2 +PSASTRO.TWEAK.NSIGMA F32 3 + +# pmAstromRadiusMatch +# use plate-scale to make this in pixels? +PSASTRO.MATCH.RADIUS F32 8 + +# pmAstromMatchFit +PSASTRO.CHIP.NX S32 1 # fit order +PSASTRO.CHIP.NY S32 1 # fit order +PSASTRO.CHIP.NITER S32 3 # fit clipping iterations +PSASTRO.CHIP.NSIGMA F32 3 # fit clipping sigmas + Index: /branches/rel-1-0/ippconfig/simple/psphot.config =================================================================== --- /branches/rel-1-0/ippconfig/simple/psphot.config (revision 11645) +++ /branches/rel-1-0/ippconfig/simple/psphot.config (revision 11645) @@ -0,0 +1,24 @@ + +SAVE.BACKMDL BOOL TRUE +SAVE.BACKGND BOOL TRUE +SAVE.BACKSUB BOOL TRUE +SAVE.PLOTS BOOL TRUE +SAVE.RESID BOOL TRUE +SAVE.PSF BOOL TRUE +LOAD.PSF BOOL FALSE + +IMSTATS_NPIX S32 3000 # number of pixels to use for sky estimate boxes: + +PSPHOT_TEST METADATA + SAVE.BACKSUB BOOL FALSE + SAVE.RESID BOOL TRUE +END + +PEAKS_OUTPUT_FILE STR peaks.dat +MOMENTS_OUTPUT_FILE STR moments.dat + +MOMENTS_SN_MIN F32 20.0 # min S/N to measure moments +FULL_FIT_SN_LIM F32 100.0 +EXT_MIN_SN F32 100.0 # fit galaxies above this S/N limit + +PEAKS_NSIGMA_LIMIT F32 10.0 # peak significance threshold Index: /branches/rel-1-0/ippconfig/ucam/camera.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/camera.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/camera.config (revision 11645) @@ -0,0 +1,128 @@ +# Camera configuration file for MegaCam: describes the camera + +# File formats that we know about (these are relative to PATH) +FORMATS METADATA + RAW STR ucam/format_raw.config +END + + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + chip STR xy00 xy10 xy20 xy30 xy40 xy50 xy60 xy70 xy01 xy11 xy21 xy31 xy41 xy51 xy61 xy71 xy02 xy12 xy22 xy32 xy42 xy52 xy62 xy72 xy03 xy13 xy23 xy33 xy43 xy53 xy63 xy73 xy04 xy14 xy24 xy34 xy44 xy54 xy64 xy74 xy05 xy15 xy25 xy35 xy45 xy55 xy65 xy75 xy06 xy16 xy26 xy36 xy46 xy56 xy66 xy76 xy07 xy17 xy27 xy37 xy47 xy57 xy67 xy77 +END + +# move this elsewhere? we need a lookup table to go from filter ID to abstract name +FILTER.ID METADATA + u.MP9301 STR u + g.MP9401 STR g + r.MP9601 STR r + i.MP9701 STR i + z.MP9801 STR z + Zp STR z + Zprime STR z + Ha.MP7605 STR Ha + Halpha STR Ha + Haalpha.on STR Ha + HaOFF.MP7604 STR HaOff + + CN.MP780 STR CN + cn.MP7803 STR CN + CN.MP7803 STR CN + + TiO.MP77 STR TiO + tio.MP7701 STR TiO + TiO.MP7701 STR TiO + NB920 STR NB920 + + B2F STR B2F + Bj STR Bj + Vj STR Vj + Rj STR Rj + Ij STR Ij + Hb STR Hb + HbOff STR HbOff +END + +# Recipe options +RECIPES METADATA + # Recipes for ppImage + PPIMAGE STR ucam/ppImage.config # Default: all (normal) options on + PPIMAGE_O STR ucam/ppImage_o.config # Overscan only + PPIMAGE_OB STR ucam/ppImage_ob.config # Overscan, bias only + PPIMAGE_OBD STR ucam/ppImage_obd.config # Overscan, bias, dark only + PPIMAGE_OBDF STR ucam/ppImage_obdf.config # Overscan, bias, dark, flat only + PPIMAGE_B STR ucam/ppImage_b.config # Bias only + PPIMAGE_D STR ucam/ppImage_d.config # Dark only + PPIMAGE_F STR ucam/ppImage_f.config # Flat only + PPIMAGE_J STR ucam/ppImage_j.config # JPEG only + + # Recipes for ppMerge + PPMERGE_BIAS STR ucam/ppMerge_bias.config + # Other recipes + PPMERGE STR ppMerge_template.config # ppMerge recipe + PSPHOT STR ucam/psphot.config # psphot details + PSASTRO STR ucam/psastro.config # psastro details + PPSTATS STR ucam/ppStats.config # ppStats recipe +END + +FILERULES METADATA + PPIMAGE.OUTPUT STR PPIMAGE.OUTPUT + PPIMAGE.BIN1 STR PPIMAGE.BIN1.OUTPUT + PPIMAGE.BIN2 STR PPIMAGE.BIN2.OUTPUT + PPIMAGE.JPEG1 STR PPIMAGE.JPEG1.OUTPUT + PPIMAGE.JPEG2 STR PPIMAGE.JPEG2.OUTPUT + + PSASTRO.INPUT STR PSASTRO.INPUT.CMP + PSASTRO.OUTPUT STR PSASTRO.OUTPUT.CMP + PSPHOT.OUTPUT STR PSPHOT.OUTPUT.CMF + + ### input file definitions + TYPE INPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA DATA.LEVEL FILE.TYPE + PPIMAGE.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + PPARITH.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + + ### use these entries to get the detrend images from specific files + PPIMAGE.MASK INPUT mask.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.BIAS INPUT bias.fit {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.DARK INPUT dark.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + PPIMAGE.FLAT INPUT flat.fits {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + + ### use these entries to get the detrend images from the database + #PPIMAGE.MASK INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.BIAS INPUT @DETDB fpa fpa NONE CHIP IMAGE + #PPIMAGE.DARK INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + #PPIMAGE.FLAT INPUT @DETDB {CHIP.NAME} {CHIP.NAME} NONE CHIP IMAGE + + PSPHOT.INPUT INPUT @FILES {CHIP.NAME} {CELL.NAME} NONE CHIP IMAGE + + PSASTRO.INPUT.CMP INPUT @FILES NONE NONE PHU CHIP CMP + PSASTRO.INPUT.CMF INPUT @FILES NONE SMPDATA PHU CHIP CMF + + ### output file definitions + TYPE OUTPUT FILENAME.RULE FILENAME.XTRA EXTNAME.RULE EXTNAME.XTRA FILE.LEVEL DATA.LEVEL FILE.TYPE FILE.SAVE FILE.FORMAT + PPIMAGE.OUTPUT OUTPUT {OUTPUT}.fit {CHIP.NAME} {CHIP.NAME} NONE FPA CHIP IMAGE TRUE NONE + PPIMAGE.OUTPUT.CHIP OUTPUT {OUTPUT}.chip.fit {CHIP.NAME} {CHIP.NAME} NONE FPA FPA IMAGE TRUE NONE + PPIMAGE.OUTPUT.FPA1 OUTPUT {OUTPUT}.fpa1.fit NONE NONE NONE FPA FPA IMAGE TRUE NONE + PPIMAGE.OUTPUT.FPA2 OUTPUT {OUTPUT}.fpa2.fit NONE NONE NONE FPA FPA IMAGE TRUE NONE + + PPIMAGE.BIN1.OUTPUT OUTPUT {OUTPUT}.b1.fit {CHIP.NAME} {CHIP.NAME} NONE CHIP FPA IMAGE TRUE NONE + PPIMAGE.BIN2.OUTPUT OUTPUT {OUTPUT}.b2.fit {CHIP.NAME} {CHIP.NAME} NONE CHIP FPA IMAGE TRUE NONE + PPIMAGE.JPEG1.OUTPUT OUTPUT {OUTPUT}.b1.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + PPIMAGE.JPEG2.OUTPUT OUTPUT {OUTPUT}.b2.jpg NONE NONE NONE FPA FPA JPEG TRUE NONE + + PSPHOT.RESID OUTPUT {OUTPUT}.res.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + PSPHOT.BACKGND OUTPUT {OUTPUT}.bck.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + PSPHOT.BACKSUB OUTPUT {OUTPUT}.sub.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + PSPHOT.BACKMDL OUTPUT {OUTPUT}.mdl.fits NONE {CELL.NAME} {CELL.NAME} CHIP CHIP IMAGE TRUE RAW + + PSPHOT.OUTPUT.RAW OUTPUT {OUTPUT}.{CHIP.NAME} NONE NONE PHU CHIP CHIP RAW TRUE NONE + PSPHOT.OUTPUT.SX OUTPUT {OUTPUT}.sx NONE NONE PHU CHIP CHIP SX TRUE NONE + PSPHOT.OUTPUT.OBJ OUTPUT {OUTPUT}.obj NONE NONE PHU CHIP CHIP OBJ TRUE NONE + PSPHOT.OUTPUT.CMF OUTPUT {OUTPUT}.cmf NONE SMPDATA PHU CHIP CHIP CMF TRUE NONE + PSPHOT.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.cmp NONE NONE PHU CHIP CHIP CMP TRUE NONE + + PSASTRO.OUTPUT.CMP OUTPUT {OUTPUT}.{CHIP.NAME}.smp NONE NONE PHU CHIP CHIP CMP TRUE NONE + + PPARITH.OUTPUT OUTPUT {OUTPUT}.{CHIP.NAME}.fit {CHIP.NAME} {CHIP.NAME} NONE CHIP CHIP IMAGE TRUE NONE + +END Index: /branches/rel-1-0/ippconfig/ucam/format_raw.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/format_raw.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/format_raw.config (revision 11645) @@ -0,0 +1,145 @@ +### Raw data from microCam + +# How to identify this type +RULE METADATA + ORIGIN STR PS0 +# INSTRUME STR ucam1 + EXTEND BOOL T + NEXTEND S32 64 +END + +# How to read this data +FILE METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CELL # The extensions represent cells + FPA.NAME STR OBSTYPE # A PHU keyword for unique identifier within the hierarchy level +END + +# What's in the FITS file? +CONTENTS METADATA + xy00 STR chip:xy00:otCell + xy10 STR chip:xy10:otCell + xy20 STR chip:xy20:otCell + xy30 STR chip:xy30:otCell + xy40 STR chip:xy40:otCell + xy50 STR chip:xy50:otCell + xy60 STR chip:xy60:otCell + xy70 STR chip:xy70:otCell + xy01 STR chip:xy01:otCell + xy11 STR chip:xy11:otCell + xy21 STR chip:xy21:otCell + xy31 STR chip:xy31:otCell + xy41 STR chip:xy41:otCell + xy51 STR chip:xy51:otCell + xy61 STR chip:xy61:otCell + xy71 STR chip:xy71:otCell + xy02 STR chip:xy02:otCell + xy12 STR chip:xy12:otCell + xy22 STR chip:xy22:otCell + xy32 STR chip:xy32:otCell + xy42 STR chip:xy42:otCell + xy52 STR chip:xy52:otCell + xy62 STR chip:xy62:otCell + xy72 STR chip:xy72:otCell + xy03 STR chip:xy03:otCell + xy13 STR chip:xy13:otCell + xy23 STR chip:xy23:otCell + xy33 STR chip:xy33:otCell + xy43 STR chip:xy43:otCell + xy53 STR chip:xy53:otCell + xy63 STR chip:xy63:otCell + xy73 STR chip:xy73:otCell + xy04 STR chip:xy04:otCell + xy14 STR chip:xy14:otCell + xy24 STR chip:xy24:otCell + xy34 STR chip:xy34:otCell + xy44 STR chip:xy44:otCell + xy54 STR chip:xy54:otCell + xy64 STR chip:xy64:otCell + xy74 STR chip:xy74:otCell + xy05 STR chip:xy05:otCell + xy15 STR chip:xy15:otCell + xy25 STR chip:xy25:otCell + xy35 STR chip:xy35:otCell + xy45 STR chip:xy45:otCell + xy55 STR chip:xy55:otCell + xy65 STR chip:xy65:otCell + xy75 STR chip:xy75:otCell + xy06 STR chip:xy06:otCell + xy16 STR chip:xy16:otCell + xy26 STR chip:xy26:otCell + xy36 STR chip:xy36:otCell + xy46 STR chip:xy46:otCell + xy56 STR chip:xy56:otCell + xy66 STR chip:xy66:otCell + xy76 STR chip:xy76:otCell + xy07 STR chip:xy07:otCell + xy17 STR chip:xy17:otCell + xy27 STR chip:xy27:otCell + xy37 STR chip:xy37:otCell + xy47 STR chip:xy47:otCell + xy57 STR chip:xy57:otCell + xy67 STR chip:xy67:otCell + xy77 STR chip:xy77:otCell +END + +# Specify the cell data +CELLS METADATA + otCell METADATA # Orthogonal transfer cell + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + CELL.TIME STR DATE-OBS + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 + CELL.XPARITY STR IMTM1_1 + CELL.YPARITY STR IMTM2_2 + CELL.X0 STR IMNPIX1 + CELL.Y0 STR IMNPIX2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.GAIN F32 1 + CELL.READNOISE F32 10.0 + CELL.SATURATION F32 65535 + CELL.BAD F32 0 + CELL.READDIR S32 1 + CELL.TIMESYS STR UTC + CHIP.X0 S32 0 + CHIP.Y0 S32 0 + CHIP.XPARITY S32 1 + CHIP.YPARITY S32 1 +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.TIME STR MJD + CELL.X0 STR FORTRAN + CELL.Y0 STR FORTRAN +END Index: /branches/rel-1-0/ippconfig/ucam/ppImage.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL TRUE # Overscan subtraction +BIAS BOOL TRUE # Bias subtraction +DARK BOOL TRUE # Dark subtraction +FLAT BOOL TRUE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL TRUE # Save chip-mosaic-ed image? +FPA1.FITS BOOL TRUE # Save 1st binned fpa image? +FPA2.FITS BOOL TRUE # Save 2nd binned fpa image? +BIN1.FITS BOOL TRUE # Save 1st binned chip image? +BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +BIN1.JPEG BOOL TRUE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_b.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_b.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_b.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL FALSE # Overscan subtraction +BIAS BOOL TRUE # Bias subtraction +DARK BOOL FALSE # Dark subtraction +FLAT BOOL FALSE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL TRUE # Save 1st binned chip image? +BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_d.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_d.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_d.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL FALSE # Overscan subtraction +BIAS BOOL FALSE # Bias subtraction +DARK BOOL TRUE # Dark subtraction +FLAT BOOL FALSE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL TRUE # Save 1st binned chip image? +BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_f.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_f.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_f.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL FALSE # Overscan subtraction +BIAS BOOL FALSE # Bias subtraction +DARK BOOL FALSE # Dark subtraction +FLAT BOOL TRUE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL TRUE # Save 1st binned chip image? +BIN2.FITS BOOL TRUE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_j.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_j.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_j.config (revision 11645) @@ -0,0 +1,85 @@ +### Phase 2 recipe configuration file + +### A little bit of a cheat to make a mosaicked binned JPEG from the binned FITS file. +### Set binning to 1, since the input will already be binned. + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0xff # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL FALSE # Overscan subtraction +BIAS BOOL FALSE # Bias subtraction +DARK BOOL FALSE # Dark subtraction +FLAT BOOL FALSE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL FALSE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL FALSE # Save 1st binned chip image? +BIN2.FITS BOOL FALSE # Save 2nd binned chip image? +BIN1.JPEG BOOL TRUE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 1 +BIN1.YBIN S32 1 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_o.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_o.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_o.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL TRUE # Overscan subtraction +BIAS BOOL FALSE # Bias subtraction +DARK BOOL FALSE # Dark subtraction +FLAT BOOL FALSE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL FALSE # Save 1st binned chip image? +BIN2.FITS BOOL FALSE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_ob.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_ob.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_ob.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL TRUE # Overscan subtraction +BIAS BOOL TRUE # Bias subtraction +DARK BOOL FALSE # Dark subtraction +FLAT BOOL FALSE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL FALSE # Save 1st binned chip image? +BIN2.FITS BOOL FALSE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_obd.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_obd.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_obd.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL TRUE # Overscan subtraction +BIAS BOOL TRUE # Bias subtraction +DARK BOOL FALSE # Dark subtraction +FLAT BOOL FALSE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL FALSE # Save 1st binned chip image? +BIN2.FITS BOOL FALSE # Save 2nd binned chip image? +BIN1.JPEG BOOL FALSE # Save 1st binned jpeg? +BIN2.JPEG BOOL FALSE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppImage_obdf.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppImage_obdf.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppImage_obdf.config (revision 11645) @@ -0,0 +1,82 @@ +### Phase 2 recipe configuration file + +# List of tasks to perform +MASK BOOL FALSE # Mask bad pixels +MASK.VALUE U8 0x10 # only mask pixels matching this bitmask +NONLIN BOOL FALSE # Non-linearity correction +OVERSCAN BOOL TRUE # Overscan subtraction +BIAS BOOL TRUE # Bias subtraction +DARK BOOL TRUE # Dark subtraction +FLAT BOOL TRUE # Flat-field normalisation +FRINGE BOOL FALSE # Fringe subtraction +PHOTOM BOOL FALSE # Source identification and photometry +ASTROM.CHIP BOOL FALSE +ASTROM.MOSAIC BOOL FALSE # Astrometry + +BASE.FITS BOOL TRUE # Save base detrended image? +CHIP.FITS BOOL FALSE # Save chip-mosaic-ed image? +FPA1.FITS BOOL FALSE # Save 1st binned fpa image? +FPA2.FITS BOOL FALSE # Save 2nd binned fpa image? +BIN1.FITS BOOL FALSE # Save 1st binned chip image? +BIN2.FITS BOOL FALSE # Save 2nd binned chip image? +BIN1.JPEG BOOL TRUE # Save 1st binned jpeg? +BIN2.JPEG BOOL TRUE # Save 2nd binned jpeg? + +# Non-linearity correction +NONLIN.SOURCE STR CHIP.NAME # How to determine the source +#@NONLIN.DATA F32 0.0 1.001 0.001 # A polynomial +NONLIN.DATA STR nonlin.dat # Filename for lookup table +#NONLIN.DATA METADATA # Source of non-linearity data +# ccd00 STR nonlin00.dat # A lookup table +# @ccd01 F32 0.0 1.001 0.001 # A polynomial +# @ccd02 F32 0.0 1.001 0.001 # A polynomial +# @ccd03 F32 1.2345 # A polynomial +# @ccd04 F32 0.0 1.001 0.001 # A polynomial +# @ccd05 F32 0.0 1.001 0.001 # A polynomial +# @ccd06 F32 0.0 1.001 0.001 # A polynomial +# @ccd07 F32 0.0 1.001 0.001 # A polynomial +# @ccd08 F32 0.0 1.001 0.001 # A polynomial +# @ccd09 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd11 F32 0.0 1.001 0.001 # A polynomial +# @ccd12 F32 0.0 1.001 0.001 # A polynomial +# @ccd13 F32 0.0 1.001 0.001 # A polynomial +# @ccd14 F32 0.0 1.001 0.001 # A polynomial +# @ccd15 F32 0.0 1.001 0.001 # A polynomial +# @ccd16 F32 0.0 1.001 0.001 # A polynomial +# @ccd17 F32 0.0 1.001 0.001 # A polynomial +# @ccd18 F32 0.0 1.001 0.001 # A polynomial +# @ccd19 F32 0.0 1.001 0.001 # A polynomial +# @ccd10 F32 0.0 1.001 0.001 # A polynomial +# @ccd21 F32 0.0 1.001 0.001 # A polynomial +# @ccd22 F32 0.0 1.001 0.001 # A polynomial +# @ccd23 F32 0.0 1.001 0.001 # A polynomial +# @ccd24 F32 0.0 1.001 0.001 # A polynomial +# @ccd25 F32 0.0 1.001 0.001 # A polynomial +# @ccd26 F32 0.0 1.001 0.001 # A polynomial +# @ccd27 F32 0.0 1.001 0.001 # A polynomial +# @ccd28 F32 0.0 1.001 0.001 # A polynomial +# @ccd29 F32 0.0 1.001 0.001 # A polynomial +# @ccd30 F32 0.0 1.001 0.001 # A polynomial +# @ccd31 F32 0.0 1.001 0.001 # A polynomial +# @ccd32 F32 0.0 1.001 0.001 # A polynomial +# @ccd33 F32 0.0 1.001 0.001 # A polynomial +# @ccd34 F32 0.0 1.001 0.001 # A polynomial +# @ccd35 F32 0.0 1.001 0.001 # A polynomial +# @ccd36 F32 0.0 1.001 0.001 # A polynomial +#END + +# Overscan subtraction +OVERSCAN.SINGLE BOOL FALSE # Reduce overscan to a single value? +#OVERSCAN.FIT STR SPLINE # NONE | POLYNOMIAL | SPLINE +OVERSCAN.FIT STR POLYNOMIAL # NONE | POLYNOMIAL | SPLINE +OVERSCAN.ORDER S32 5 # Order of polynomial fit +OVERSCAN.STAT STR MEAN # MEAN | MEDIAN + +# binned output image options +BIN1.XBIN S32 8 +BIN1.YBIN S32 8 +BIN2.XBIN S32 64 +BIN2.YBIN S32 64 + +PHOTCODE.RULE STR {CAMERA}.{FILTER.ID}.{CHIP.N} Index: /branches/rel-1-0/ippconfig/ucam/ppMerge_bias.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppMerge_bias.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppMerge_bias.config (revision 11645) @@ -0,0 +1,14 @@ +# Recipe configuration for ppMerge + +ROWS S32 128 # Number of rows to read at once +ELECTRONS F32 100.0 # Minimum number of electrons for useful signal +SAMPLE S32 100 # Sampling factor for measuring the background +REJ F32 3.0 # Rejection threshold (sigma) +ITER S32 1 # Number of rejection iterations +FRACHIGH F32 0.3 # Fraction of high pixels to reject immediately +FRACLOW F32 0.1 # Fraction of low pixels to reject immediately +NKEEP S32 5 # Minimum number of pixels in stack to keep +MASKVAL S32 0xff # Mask value for input data +### Statistics options: MEAN | MEDIAN | ROBUST | FITTED | CLIPPED +COMBINE STR MEAN # Statistic to use for combination: +BACKGROUND STR MEDIAN # Statistic to use to measure the background Index: /branches/rel-1-0/ippconfig/ucam/ppMerge_dark.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppMerge_dark.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppMerge_dark.config (revision 11645) @@ -0,0 +1,14 @@ +# Recipe configuration for ppMerge + +ROWS S32 128 # Number of rows to read at once +ELECTRONS F32 100.0 # Minimum number of electrons for useful signal +SAMPLE S32 100 # Sampling factor for measuring the background +REJ F32 3.0 # Rejection threshold (sigma) +ITER S32 1 # Number of rejection iterations +FRACHIGH F32 0.3 # Fraction of high pixels to reject immediately +FRACLOW F32 0.1 # Fraction of low pixels to reject immediately +NKEEP S32 5 # Minimum number of pixels in stack to keep +MASKVAL S32 0xff # Mask value for input data +### Statistics options: MEAN | MEDIAN | ROBUST | FITTED | CLIPPED +COMBINE STR MEAN # Statistic to use for combination: +BACKGROUND STR MEDIAN # Statistic to use to measure the background Index: /branches/rel-1-0/ippconfig/ucam/ppMerge_flat.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppMerge_flat.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppMerge_flat.config (revision 11645) @@ -0,0 +1,14 @@ +# Recipe configuration for ppMerge + +ROWS S32 128 # Number of rows to read at once +ELECTRONS F32 100.0 # Minimum number of electrons for useful signal +SAMPLE S32 100 # Sampling factor for measuring the background +REJ F32 3.0 # Rejection threshold (sigma) +ITER S32 1 # Number of rejection iterations +FRACHIGH F32 0.3 # Fraction of high pixels to reject immediately +FRACLOW F32 0.1 # Fraction of low pixels to reject immediately +NKEEP S32 5 # Minimum number of pixels in stack to keep +MASKVAL S32 0xff # Mask value for input data +### Statistics options: MEAN | MEDIAN | ROBUST | FITTED | CLIPPED +COMBINE STR MEAN # Statistic to use for combination: +BACKGROUND STR MEDIAN # Statistic to use to measure the background Index: /branches/rel-1-0/ippconfig/ucam/ppStats.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/ppStats.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/ppStats.config (revision 11645) @@ -0,0 +1,12 @@ +### ppStats recipe +SAMPLE F32 0.1 # Fraction of cell to sample +MASKVAL U8 0xff # Mask value to use for statistics + +# Define the outputs as MULTI +HEADER MULTI +CONCEPT MULTI +STAT MULTI + +# Values to return +STAT STR ROBUST_MEDIAN # Background estimator +STAT STR ROBUST_STDEV # Background standard deviation estimator Index: /branches/rel-1-0/ippconfig/ucam/psastro.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/psastro.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/psastro.config (revision 11645) @@ -0,0 +1,36 @@ + +# astrometry matching parameters + +# nominal plate scale (microns / pixel) +# PSASTRO.PLATE.SCALE F32 13.5 +PSASTRO.PLATE.SCALE F32 1.0 + +# pmAstromGridMatch: +PSASTRO.GRID.MIN.ANGLE F32 -0.0851 +PSASTRO.GRID.MAX.ANGLE F32 +0.0851 +PSASTRO.GRID.DEL.ANGLE F32 0.0170 + +# pmAstromGridAngle +# max grid offset in FP units (microns) +# use plate-scale to make this in pixels? +# PSASTRO.GRID.OFFSET F32 10000. +# PSASTRO.GRID.SCALE F32 500. +PSASTRO.GRID.OFFSET F32 1000. +PSASTRO.GRID.SCALE F32 50 + +# these tweak are in FP units (pixels, currently) +PSASTRO.TWEAK.SCALE F32 1 +PSASTRO.TWEAK.RANGE F32 75 +PSASTRO.TWEAK.SMOOTH F32 2 +PSASTRO.TWEAK.NSIGMA F32 3 + +# pmAstromRadiusMatch +# use plate-scale to make this in pixels? +PSASTRO.MATCH.RADIUS F32 8 + +# pmAstromMatchFit +PSASTRO.CHIP.NX S32 1 # fit order +PSASTRO.CHIP.NY S32 1 # fit order +PSASTRO.CHIP.NITER S32 3 # fit clipping iterations +PSASTRO.CHIP.NSIGMA F32 3 # fit clipping sigmas + Index: /branches/rel-1-0/ippconfig/ucam/psphot.config =================================================================== --- /branches/rel-1-0/ippconfig/ucam/psphot.config (revision 11645) +++ /branches/rel-1-0/ippconfig/ucam/psphot.config (revision 11645) @@ -0,0 +1,84 @@ + +#SAVE.BACKMDL BOOL TRUE +#SAVE.BACKGND BOOL TRUE +#SAVE.BACKSUB BOOL TRUE +#SAVE.RESID BOOL TRUE +#SAVE.PSF BOOL TRUE +#LOAD.PSF BOOL FALSE + +# image noise parameters +RDNOISE STR HD:RDNOISE # read-noise in electrons +GAIN STR HD:GAIN # electrons / DN +ZERO_POINT F32 25.000 # zero point used by DVO + +# background model : +BACKGROUND.XBIN S32 128 +BACKGROUND.YBIN S32 128 + +# masking parameters +XMIN F32 2 # minimum valid x-coord +XMAX F32 -2 # maximum valid x-coord +YMIN F32 2 # minimum valid y-coord +YMAX F32 -2 # maximum valid y-coord +SATURATION F32 50000 # saturation level on this chip +MIN_VALID_PIXEL F32 1 # saturation level on this chip + +# image statistics parameters +IMSTATS_NPIX S32 1000 # number of pixels to use for sky estimate boxes: + +# peak finding +PEAKS_SMOOTH_SIGMA F32 1.0 # smoothing kernel sigma in pixels +PEAKS_SMOOTH_NSIGMA F32 3.0 # smoothing kernel width in sigmas +# PEAKS_NSIGMA_LIMIT F32 10.0 # peak significance threshold +PEAKS_NSIGMA_LIMIT F32 25.0 # peak significance threshold + +# basic object statistics +SKY_INNER_RADIUS F32 15 # square annulus for local sky measurement +SKY_OUTER_RADIUS F32 25 # square annulus for local sky measurement +PSF_MOMENTS_RADIUS F32 3 # calculate initial source moments with this radius +PSF_SN_LIM F32 100 # minimum S/N for stars used for PSF model +PSF_MAX_NSTARS S32 300 # limit number of stars used for PSF model + +# PSF model parameters : choose the PSF model +# list as many PSF_MODEL options as desired +# PSF_MODEL MULTI +PSF_MODEL STR PS_MODEL_QGAUSS +# PSF_MODEL STR PS_MODEL_GAUSS +# PSF_MODEL STR PS_MODEL_PGAUSS +# PSF_MODEL STR PS_MODEL_TGAUSS ## not well tested, not very successful + +MOMENTS_SCALE F32 0.1 +MOMENTS_SN_MIN F32 30.0 +MOMENTS_SX_MAX F32 3.0 +MOMENTS_SY_MAX F32 3.0 +MOMENTS_AR_MAX F32 1.5 # maximum axial ratio: 1 / AR < (sx / sy) < AR + +PSF_FIT_RADIUS F32 15.0 # fitting radius for test PSF model +PSF_REF_RADIUS F32 25.0 # aperture magnitudes are scaled via + +# PSF-like source model parameters +PSF_FIT_NSIGMA F32 1.0 # significance for pixel included in fit +PSF_FIT_PADDING F32 5.0 # extra annulus to use for fit +PSF_SHAPE_NSIGMA F32 3.0 # max significance for shape variation +PSF_MIN_SN F32 2.0 # reject objects below this significance +PSF_MAX_CHI F32 50.0 # reject objects worse that this + +# EXTended source model parameters +EXT_MODEL STR PS_MODEL_PGAUSS +EXT_MIN_SN F32 50.0 # fit galaxies above this S/N limit +EXT_FIT_NSIGMA F32 1 # significance for pixel included in fit +EXT_FIT_PADDING F32 5 # extra annulus to use for fit +EXT_MOMENTS_RADIUS F32 9 + +FITMODE STR BLEND +DEBLEND_PEAK_FRACTION F32 0.1 +DEBLEND_SKY_NSIGMA F32 10.0 + +APTREND STR SKYSAT +FULL_FIT_SN_LIM F32 50.0 +AP_MIN_SN F32 50.0 + +# BREAK_POINT STR PEAKS +# PEAKS_OUTPUT_FILE STR peaks.dat +# MOMENTS_OUTPUT_FILE STR moments.dat +# ANALYSIS_REGION STR [1000:1600,2800:3400] Index: /branches/rel-1-0/ppImage/.cvsignore =================================================================== --- /branches/rel-1-0/ppImage/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppImage/.cvsignore (revision 11645) @@ -0,0 +1,16 @@ +bin +aclocal.m4 +autom4te.cache +compile +config.log +config.status +configure +depcomp +install-sh +Makefile.in +missing +Makefile +config.cache +config.guess +config.sub +ltmain.sh Index: /branches/rel-1-0/ppImage/Makefile.am =================================================================== --- /branches/rel-1-0/ppImage/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppImage/Makefile.am (revision 11645) @@ -0,0 +1,3 @@ +SUBDIRS = src + +CLEANFILES = *~ core core.* Index: /branches/rel-1-0/ppImage/autogen.sh =================================================================== --- /branches/rel-1-0/ppImage/autogen.sh (revision 11645) +++ /branches/rel-1-0/ppImage/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=ppImage +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $LIBTOOlIZE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" + DIE=1 +} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOHEADER installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ppImage/configure.ac =================================================================== --- /branches/rel-1-0/ppImage/configure.ac (revision 11645) +++ /branches/rel-1-0/ppImage/configure.ac (revision 11645) @@ -0,0 +1,32 @@ +AC_PREREQ(2.59) + +AC_INIT([ppImage], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([src]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([src/config.h]) +AM_MAINTAINER_MODE + +IPP_STDCFLAGS + +AC_LANG(C) +AC_GNU_SOURCE +AC_PROG_CC +AC_PROG_INSTALL +dnl AC_PROG_LIBTOOL +AC_SYS_LARGEFILE + +PKG_CHECK_MODULES([PSLIB], [pslib >= 1.0.0]) +PKG_CHECK_MODULES([PSMODULE], [psmodules >= 1.0.0]) +PKG_CHECK_MODULES([PSPHOT], [psphot >= 0.8.0]) +PKG_CHECK_MODULES([PSASTRO], [psastro >= 0.8.0]) +PKG_CHECK_MODULES([PPSTATS], [ppStats >= 1.0.0]) + +IPP_STDOPTS +CFLAGS="${CFLAGS=} -Wall -Werror -std=c99" + +AC_CONFIG_FILES([ + Makefile + src/Makefile +]) +AC_OUTPUT Index: /branches/rel-1-0/ppImage/src/.cvsignore =================================================================== --- /branches/rel-1-0/ppImage/src/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppImage/src/.cvsignore (revision 11645) @@ -0,0 +1,9 @@ +ppImage +ppTest +Makefile +Makefile.in +.deps +ppFocus +config.h +config.h.in +stamp-h1 Index: /branches/rel-1-0/ppImage/src/Makefile.am =================================================================== --- /branches/rel-1-0/ppImage/src/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppImage/src/Makefile.am (revision 11645) @@ -0,0 +1,62 @@ +bin_PROGRAMS = ppImage ppFocus + +noinst_PROGRAMS = ppTest + +noinst_HEADERS = \ + ppImage.h \ + ppImageOptions.h \ + ppImageDetrendFringe.h \ + ppImageVersion.h + +ppImage_CPPFLAGS = $(PSLIB_CFLAGS) $(PSMODULE_CFLAGS) $(PSPHOT_CFLAGS) $(PSASTRO_CFLAGS) $(PPSTATS_CFLAGS) $(ppImage_CFLAGS) +ppImage_LDFLAGS = $(PSLIB_LIBS) $(PSMODULE_LIBS) $(PSPHOT_LIBS) $(PPSTATS_LIBS) $(PSASTRO_LIBS) +ppImage_SOURCES = \ + ppImage.c \ + ppImageArguments.c \ + ppImageParseCamera.c \ + ppImageLoop.c \ + ppImageCleanup.c \ + ppImageOptions.c \ + ppImageDetrendReadout.c \ + ppImageDetrendNonLinear.c \ + ppImageDetrendFringe.c \ + ppImageRebinReadout.c \ + ppImageMosaic.c \ + ppImagePhotom.c \ + ppImageAstrom.c \ + ppImageAddstar.c \ + ppImageVersion.c + +ppFocus_CPPFLAGS = $(PSLIB_CFLAGS) $(PSMODULE_CFLAGS) $(PSPHOT_CFLAGS) $(PSASTRO_CFLAGS) $(PPSTATS_CFLAGS) $(ppImage_CFLAGS) +ppFocus_LDFLAGS = $(PSLIB_LIBS) $(PSMODULE_LIBS) $(PSPHOT_LIBS) $(PPSTATS_LIBS) $(PSASTRO_LIBS) +ppFocus_SOURCES = \ + ppFocus.c \ + ppFocusArguments.c \ + ppFocusParseCamera.c \ + ppFocusGetFWHM.c \ + ppFocusFitFWHM.c \ + ppImageLoop.c \ + ppImageCleanup.c \ + ppImageOptions.c \ + ppImageDetrendReadout.c \ + ppImageDetrendNonLinear.c \ + ppImageDetrendFringe.c \ + ppImageRebinReadout.c \ + ppImageMosaic.c \ + ppImagePhotom.c \ + ppImageAstrom.c \ + ppImageAddstar.c \ + ppImageVersion.c + +ppTest_CPPFLAGS = $(PSLIB_CFLAGS) $(PSMODULE_CFLAGS) $(ppImage_CFLAGS) +ppTest_LDFLAGS = $(PSLIB_LIBS) $(PSMODULE_LIBS) +ppTest_SOURCES = \ + ppTest.c + +CLEANFILES = *~ + +clean-local: + -rm -f TAGS +# Tags for emacs +tags: + etags `find . -name \*.[ch] -print` Index: /branches/rel-1-0/ppImage/src/ppFocus.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppFocus.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppFocus.c (revision 11645) @@ -0,0 +1,74 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +int main(int argc, char **argv) { + + psLibInit(NULL); + + psTimerStart(TIMERNAME); + + // Parse the configuration and arguments + // Open the input image(s) + // Determine camera, format from header if not already defined + // Construct camera in preparation for reading + pmConfig *config = ppFocusArguments(argc, argv); + if (config == NULL) { + psErrorStackPrint(stderr, " "); + exit(1); + } + + // we search the argument data for the named fileset (argname) + psArray *infiles = psMetadataLookupPtr(NULL, config->arguments, "INPUT"); + if (!infiles) { + psTrace("pmFPAfile", 5, "Failed to find INPUT in argument list"); + exit(1); + } + + // allocate vectors for analysis + psVector *focus = psVectorAllocEmpty(infiles->n, PS_TYPE_F32); + psVector *fwhm = psVectorAllocEmpty(infiles->n, PS_TYPE_F32); + + for (int i = 0; i < infiles->n; i++) { + + // define recipe options + // define the active I/O files + ppImageOptions *options = ppFocusParseCamera(config, i); + if (options == NULL) { + psErrorStackPrint(stderr, " "); + exit(1); + } + + // Image Arithmetic Loop + // XXX ppFocus REQUIRES photom: for it to be true? + // + if (!ppImageLoop(config, options)) { + psErrorStackPrint(stderr, " "); + exit(1); + } + + // determine FWHM at reference location in image + // (also removes PPIMAGE.INPUT from config->files) + ppFocusGetFWHM (config, focus, fwhm); + + ppFocusDropCamera (config); + psFree (options); + } + + ppFocusFitFWHM (config, focus, fwhm); + + psLogMsg ("ppFocus", 3, "complete ppFocus run: %f sec\n", psTimerMark (TIMERNAME)); + + // Cleaning up + psFree (focus); + psFree (fwhm); + ppImageCleanup(config, NULL); + return EXIT_SUCCESS; +} + +// ppFocus is a lot like ppImage, but with a few important differences: +// - the input list is a set of independent images (not multiple files for a single image) +// - each pass to ppImageLoop performs the analysis on a different pmFPAfile +// - after each ppImageLoop, grap the input pmFPAfile and extract the FWHM stats Index: /branches/rel-1-0/ppImage/src/ppFocusArguments.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppFocusArguments.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppFocusArguments.c (revision 11645) @@ -0,0 +1,75 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +static void usage (void) { + fprintf (stderr, "USAGE: ppFocus [-file focus.*.fits] [-list INPUT.txt] OUTPUT\n"); + exit (2); +} + +pmConfig *ppFocusArguments(int argc, char **argv) { + + int N; + bool status; + + if (argc == 1) usage (); + + // load the site-wide configuration information + pmConfig *config = pmConfigRead(&argc, argv, RECIPE_NAME); + if (config == NULL) { + psErrorStackPrint(stderr, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + + // save the following additional recipe values based on command-line options + // these options override the PPIMAGE recipe values loaded from recipe files + psMetadata *options = pmConfigRecipeOptions (config, RECIPE_NAME); + + // save these recipe options until we have loaded the options + // psMetadata *options = psMetadataAlloc (); + // psMetadataAddPtr (config->arguments, PS_LIST_TAIL, "PPIMAGE.OPTIONS", PS_DATA_METADATA, "", options); + + // the following options override the PPIMAGE recipe options + + // recipe option: -usemask : override MASK setting in phase2.recipe + if ((N = psArgumentGet(argc, argv, "-usemask"))) { + psArgumentRemove (N, &argc, argv); + psMetadataAddBool (options, PS_LIST_TAIL, "MASK", PS_META_REPLACE, "", true); + psArgumentRemove (N, &argc, argv); + } + + // XXX add other PPIMAGE recipe options here + + // drop the local view on the options (saved on config->arguments) + psFree (options); + + // the input file is a required argument; if not found, we will exit + status = pmConfigFileSetsMD (config->arguments, config, "INPUT", "-file", "-list"); + if (!status) { usage ();} + + // if these command-line options are supplied, load the file name lists into config->arguments + // override any configuration-specified source for these files + pmConfigFileSetsMD (config->arguments, config, "BIAS", "-bias", "-biaslist"); + pmConfigFileSetsMD (config->arguments, config, "DARK", "-dark", "-darklist"); + pmConfigFileSetsMD (config->arguments, config, "FLAT", "-flat", "-flatlist"); + pmConfigFileSetsMD (config->arguments, config, "MASK", "-mask", "-masklist"); + pmConfigFileSetsMD (config->arguments, config, "FRINGE", "-fringe", "-fringelist"); + + // chip selection is used to limit chips to be processed + if ((N = psArgumentGet (argc, argv, "-chip"))) { + psArgumentRemove (N, &argc, argv); + psMetadataAddStr (config->arguments, PS_LIST_TAIL, "CHIP_SELECTIONS", PS_DATA_STRING, "", + argv[N]); + psArgumentRemove (N, &argc, argv); + } + + if (argc != 2) usage (); + + // Add the input and output images (which remain on the command-line) to the arguments list + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "OUTPUT", 0, "Name of the output image", + argv[1]); + + return config; +} Index: /branches/rel-1-0/ppImage/src/ppFocusFitFWHM.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppFocusFitFWHM.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppFocusFitFWHM.c (revision 11645) @@ -0,0 +1,30 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +bool ppFocusFitFWHM (pmConfig *config, psVector *focus, psVector *fwhm) { + + float minFocus; + + psPolynomial1D *poly = psPolynomial1DAlloc (PS_POLYNOMIAL_ORD, 2); + + if (!psVectorFitPolynomial1D (poly, NULL, 0, fwhm, NULL, focus)) { + psError (PS_ERR_UNKNOWN, false, "failed to fit focus/fwhm trend; invalid image data?"); + return false; + } + + if (poly->coeff[2] <= 0.0) { + psLogMsg ("ppFocus", 3, "poor focus fit: zero or negative curvature\n"); + psLogMsg ("ppFocus", 3, "fit coeffs: %f %f %f\n", + poly->coeff[0], poly->coeff[1], poly->coeff[2]); + } + + minFocus = -0.5 * poly->coeff[1] / poly->coeff[2]; + psLogMsg ("ppFocus", 3, "best fit focus: %f\n", minFocus); + psLogMsg ("ppFocus", 3, "fwhm @ min: %f\n", psPolynomial1DEval (poly, minFocus)); + + psFree (poly); + return true; +} Index: /branches/rel-1-0/ppImage/src/ppFocusGetFWHM.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppFocusGetFWHM.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppFocusGetFWHM.c (revision 11645) @@ -0,0 +1,79 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +bool ppFocusGetFWHM (pmConfig *config, psVector *focus, psVector *fwhm) { + + bool status; + pmChip *chip; + pmCell *cell; + pmReadout *readout; + psMetadata *header; + float FOCUS, FWHM, FWHM_X, FWHM_Y, FWHMsum; + int FWHMnum; + + pmFPAfile *input = psMetadataLookupPtr (&status, config->files, "PSPHOT.OUTPUT"); + if (!status) { + psErrorStackPrint(stderr, "Can't find input data!\n"); + exit(EXIT_FAILURE); + } + + pmFPAview *view = pmFPAviewAlloc (0); + + // - find readouts with measured PSFs + // - measure the average central FWHM for each PSF + FWHMsum = 0.0; + FWHMnum = 0; + + while ((chip = pmFPAviewNextChip (view, input->fpa, 1)) != NULL) { + psLogMsg ("ppImageLoop", 4, "Chip %d: %x %x\n", view->chip, chip->file_exists, chip->process); + if (!chip->process || !chip->file_exists) { continue; } + + while ((cell = pmFPAviewNextCell (view, input->fpa, 1)) != NULL) { + psLogMsg ("ppImageLoop", 4, "Cell %d: %x %x\n", view->cell, cell->file_exists, cell->process); + if (!cell->process || !cell->file_exists) { continue; } + + // process each of the readouts + while ((readout = pmFPAviewNextReadout (view, input->fpa, 1)) != NULL) { + if (!readout->data_exists) { continue; } + + // get average FWHM + // psphotReadout writes the FWHM values into the PSPHOT.HEADER table + // the source of this value depends on the psphot options. + // - if breakPoint is set to PEAKS, the value will not be defined + // - if breakPoint is set to MOMENTS, the PSFSTAR moments are used + // - in all other cases, the psf model is used + header = psMetadataLookupPtr (&status, readout->analysis, "PSPHOT.HEADER"); + if (header == NULL) { + psError(PS_ERR_IO, false, "Missing header in ppFocus"); + continue; + } + + FWHM_X = psMetadataLookupF32 (&status, header, "FWHM_X"); + FWHM_Y = psMetadataLookupF32 (&status, header, "FWHM_Y"); + + FWHMsum += 0.5*(FWHM_X + FWHM_Y); + FWHMnum ++; + + psLogMsg ("ppFocus", 4, "focus pt: %f,%f, fwhm sum: %f, fwhm num: %d\n", FWHM_X, FWHM_Y, FWHMsum, FWHMnum); + } + } + } + + FWHM = FWHMsum / FWHMnum; + + FOCUS = psMetadataLookupF32 (&status, input->fpa->concepts, "FPA.FOCUS"); + + fwhm->data.F32[fwhm->n] = FWHM; + focus->data.F32[focus->n] = FOCUS; + + psVectorExtend (fwhm, 10, 1); + psVectorExtend (focus, 10, 1); + + psLogMsg ("ppFocus", 4, "focus: %f, fwhm: %f (%d)\n", FOCUS, FWHM, FWHMnum); + + psFree (view); + return true; +} Index: /branches/rel-1-0/ppImage/src/ppFocusParseCamera.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppFocusParseCamera.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppFocusParseCamera.c (revision 11645) @@ -0,0 +1,137 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +// XXX clean up error checks: return NULL, not psAbort +ppImageOptions *ppFocusParseCamera (pmConfig *config, int entry) { + + bool status = false; + + // the first input image defines the camera, and all recipes and options that follow + // select only the first file from the INPUT array + pmFPAfile *input = pmFPAfileDefineSingleFromArgs (&status, config, "PPIMAGE.INPUT", "INPUT", entry); + if (!status) { + psError(PS_ERR_IO, false, "Failed to build FPA from PPIMAGE.INPUT"); + return NULL; + } + + // parse the options from the metadata format to the ppImageOptions structure + ppImageOptions *options = ppImageOptionsParse (config); + + // the following are defined from the argument list, if given, + // otherwise they revert to the config information or detrend database if specified + // not all input or output images are used in a given recipe + if (options->doBias) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.BIAS", "BIAS"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.BIAS"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.BIAS", input->fpa, PM_DETREND_TYPE_BIAS); + if (!status) psAbort ("ppImageParseDetrend", "can't find a bias image source"); + } + if (options->doDark) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.DARK", "DARK"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.DARK"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.DARK", input->fpa, PM_DETREND_TYPE_DARK); + if (!status) psAbort ("ppImageParseDetrend", "can't find a dark image source"); + } + if (options->doMask) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.MASK", "MASK"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.MASK"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.MASK", input->fpa, PM_DETREND_TYPE_MASK); + if (!status) psAbort ("ppImageParseDetrend", "can't find a mask image source"); + } + if (options->doFlat) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.FLAT", "FLAT"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.FLAT"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.FLAT", input->fpa, PM_DETREND_TYPE_FLAT); + if (!status) psAbort ("ppImageParseDetrend", "can't find a flat image source"); + } + + // the following files are output targets + pmFPAfile *output = pmFPAfileDefineOutput (config, input->fpa, "PPIMAGE.OUTPUT"); + pmFPAfile *byChip = pmFPAfileDefineNewCamera (config, "PPIMAGE.OUTPUT.CHIP"); + pmFPAfile *byFPA1 = pmFPAfileDefineNewCamera (config, "PPIMAGE.OUTPUT.FPA1"); + pmFPAfile *byFPA2 = pmFPAfileDefineNewCamera (config, "PPIMAGE.OUTPUT.FPA2"); + + // save any of these files? + output->save = options->BaseFITS; + byChip->save = options->ChipFITS; + byFPA1->save = options->FPA1FITS; + byFPA2->save = options->FPA2FITS; + + // output is used as a carrier: input to byChip + output->freeLevel = PM_FPA_LEVEL_CHIP; + + // define the binned target files (which may just be carriers for some camera configurations) + pmFPAfile *bin1 = pmFPAfileDefineFromFPA (config, byChip->fpa, options->xBin1, options->yBin1, "PPIMAGE.BIN1"); + pmFPAfile *bin2 = pmFPAfileDefineFromFPA (config, byChip->fpa, options->xBin2, options->yBin2, "PPIMAGE.BIN2"); + + // bin1 and bin2 are used as carriers: input for byFPA1, byFPA2 + bin1->freeLevel = PM_FPA_LEVEL_FPA; + bin2->freeLevel = PM_FPA_LEVEL_FPA; + + pmFPAfile *jpg1 = pmFPAfileDefineOutput (config, byFPA1->fpa, "PPIMAGE.JPEG1"); + pmFPAfile *jpg2 = pmFPAfileDefineOutput (config, byFPA2->fpa, "PPIMAGE.JPEG2"); + + // XXX we could potentially not define these pmFPAfiles if no output is requested... + bin1->save = options->Bin1FITS; + bin2->save = options->Bin2FITS; + jpg1->save = options->Bin1JPEG; + jpg2->save = options->Bin2JPEG; + + // Chip selection: turn on only the chips specified (pass status to suppress missing-key log msg) + char *chipLine = psMetadataLookupStr(&status, config->arguments, "CHIP_SELECTIONS"); + psArray *chips = psStringSplitArray (chipLine, ",", false); + if (chips->n > 0) { + pmFPASelectChip (input->fpa, -1, true); // deselect all chips + for (int i = 0; i < chips->n; i++) { + int chipNum = atoi(chips->data[i]); + if (! pmFPASelectChip(input->fpa, chipNum, false)) { + psError(PS_ERR_IO, false, "Chip number %d doesn't exist in camera.\n", chipNum); + return false; + } + } + } + psFree (chips); + + return (options); +} + +// remove from config all pmFPAfiles which could have been added (ignore missing entries) +void ppFocusDropCamera (pmConfig *config) { + + psMetadataRemoveKey (config->files, "PPIMAGE.INPUT"); + + psMetadataRemoveKey (config->files, "PPIMAGE.BIAS"); + psMetadataRemoveKey (config->files, "PPIMAGE.DARK"); + psMetadataRemoveKey (config->files, "PPIMAGE.MASK"); + psMetadataRemoveKey (config->files, "PPIMAGE.FLAT"); + + psMetadataRemoveKey (config->files, "PPIMAGE.OUTPUT"); + psMetadataRemoveKey (config->files, "PPIMAGE.OUTPUT.CHIP"); + psMetadataRemoveKey (config->files, "PPIMAGE.OUTPUT.FPA1"); + psMetadataRemoveKey (config->files, "PPIMAGE.OUTPUT.FPA2"); + + psMetadataRemoveKey (config->files, "PPIMAGE.BIN1"); + psMetadataRemoveKey (config->files, "PPIMAGE.BIN2"); + + psMetadataRemoveKey (config->files, "PPIMAGE.JPEG1"); + psMetadataRemoveKey (config->files, "PPIMAGE.JPEG2"); + + psMetadataRemoveKey (config->files, "PSPHOT.INPUT"); + psMetadataRemoveKey (config->files, "PSPHOT.OUTPUT"); + psMetadataRemoveKey (config->files, "PSPHOT.RESID"); + psMetadataRemoveKey (config->files, "PSPHOT.BACKMDL"); + psMetadataRemoveKey (config->files, "PSPHOT.BACKGND"); + psMetadataRemoveKey (config->files, "PSPHOT.BACKSUB"); + psMetadataRemoveKey (config->files, "PSPHOT.PSF.LOAD"); + psMetadataRemoveKey (config->files, "PSPHOT.PSF.SAVE"); + + return; +} + Index: /branches/rel-1-0/ppImage/src/ppImage.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImage.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImage.c (revision 11645) @@ -0,0 +1,90 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +int main(int argc, char **argv) { + + psLibInit(NULL); + + psTimerStart(TIMERNAME); + + // Parse the configuration and arguments + // Open the input image(s) + // Determine camera, format from header if not already defined + // Construct camera in preparation for reading + pmConfig *config = ppImageArguments(argc, argv); + if (config == NULL) { + psErrorStackPrint(stderr, "Unable to parse command-line arguments."); + exit(PS_EXIT_CONFIG_ERROR); + } + + // define recipe options + // define the active I/O files + ppImageOptions *options = ppImageParseCamera(config); + if (options == NULL) { + psErrorStackPrint(stderr, "Unable to parse camera."); + exit(PS_EXIT_CONFIG_ERROR); + } + + // Image Arithmetic Loop + if (!ppImageLoop(config, options)) { + psErrorStackPrint(stderr, "Unable to loop over input"); + exit(PS_EXIT_SYS_ERROR); + } + + psLogMsg ("ppImage", 3, "Complete ppImage run: %f sec\n", psTimerMark(TIMERNAME)); + + // Cleaning up + ppImageCleanup(config, options); + + return PS_EXIT_SUCCESS; +} + +void ppImageFileCheck (pmConfig *config) { + + // add the output names to the output-type files + psMetadataItem *item = NULL; + psMetadataIterator *iter = psMetadataIteratorAlloc (config->files, PS_LIST_HEAD, NULL); + while ((item = psMetadataGetAndIncrement (iter)) != NULL) { + pmFPAfile *file = item->data.V; + pmFPA *fpa = file->fpa; + fprintf (stderr, "file %s\n", file->name); + if (!fpa) { + fprintf (stderr, " has no FPA\n"); + continue; + } + if (fpa->hdu) { + if (fpa->hdu->images) fprintf (stderr, " (%d,%d) images\n", -1, -1); + if (fpa->hdu->weights) fprintf (stderr, " (%d,%d) weights\n", -1, -1); + if (fpa->hdu->masks) fprintf (stderr, " (%d,%d) masks\n", -1, -1); + if (fpa->hdu->header) fprintf (stderr, " (%d,%d) header\n", -1, -1); + continue; + } + fprintf (stderr, " has no fpa data (%d,%d)\n", -1, -1); + for (int i = 0; i < fpa->chips->n; i++) { + pmChip *chip = fpa->chips->data[i]; + if (chip->hdu) { + if (chip->hdu->images) fprintf (stderr, " (%d,%d) images\n", i, -1); + if (chip->hdu->weights) fprintf (stderr, " (%d,%d) weights\n", i, -1); + if (chip->hdu->masks) fprintf (stderr, " (%d,%d) masks\n", i, -1); + if (chip->hdu->header) fprintf (stderr, " (%d,%d) header\n", i, -1); + continue; + } + fprintf (stderr, " has no chip data (%d,%d)\n", i, -1); + for (int j = 0; j < chip->cells->n; j++) { + pmCell *cell = chip->cells->data[j]; + if (cell->hdu) { + if (cell->hdu->images) fprintf (stderr, " (%d,%d) images\n", i, j); + if (cell->hdu->weights) fprintf (stderr, " (%d,%d) weights\n", i, j); + if (cell->hdu->masks) fprintf (stderr, " (%d,%d) masks\n", i, j); + if (cell->hdu->header) fprintf (stderr, " (%d,%d) header\n", i, j); + continue; + } + fprintf (stderr, " has no cell data (%d,%d)\n", i, j); + } + } + } + psFree (iter); +} Index: /branches/rel-1-0/ppImage/src/ppImage.h =================================================================== --- /branches/rel-1-0/ppImage/src/ppImage.h (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImage.h (revision 11645) @@ -0,0 +1,60 @@ +#ifndef PP_IMAGE_H +#define PP_IMAGE_H + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include "pslib.h" +#include "psmodules.h" +#include "psphot.h" +#include "psastro.h" +#include "ppImageOptions.h" + +#define RECIPE_NAME "PPIMAGE" // Name of the recipe to use +#define TIMERNAME "ppImage" // Name of timer + +// Get the configuration +pmConfig *ppImageArguments(int argc, char **argv); + +// Determine what type of camera, and initialise +ppImageOptions *ppImageParseCamera(pmConfig *config); + +// Loop over the input +bool ppImageLoop(pmConfig *config, ppImageOptions *options); + +// free memory, check for leaks +void ppImageCleanup (pmConfig *config, ppImageOptions *options); + +// perform the detrend analysis on the current readout +bool ppImageDetrendReadout (pmConfig *config, ppImageOptions *options, pmFPAview *view); + +bool ppImageDetrendBias(pmReadout *inputReadout, pmReadout *bias, pmReadout *dark, ppImageOptions *options); +bool ppImageDetrendMask(pmReadout *input, pmReadout *mask); + +bool ppImageDetrendNonLinear(pmReadout *input, ppImageOptions *options); +bool ppImageDetrendNonLinearLookup(pmReadout *input, psMetadataItem *dataItem); +bool ppImageDetrendNonLinearPolynomial(pmReadout *input, psMetadataItem *dataItem); + +bool ppImageRebinChip (pmConfig *config, pmFPAview *view, char *outName, char *inName); +bool ppImageRebinReadout (pmReadout *output, pmReadout *input, pmFPAfile *outFile); + +bool ppImagePhotom (pmConfig *config, pmFPAview *view); +bool ppImageAstrom (pmConfig *config); +bool ppImageAddstar (pmConfig *config); + +bool ppImageMosaicChip (pmConfig *config, const pmFPAview *view, char *outFile, char *inFile); +bool ppImageMosaicFPA (pmConfig *config, char *outFile, char *inFile); + +void ppImageFileCheck (pmConfig *config); + +// functions used by ppFocus +pmConfig *ppFocusArguments(int argc, char **argv); +ppImageOptions *ppFocusParseCamera (pmConfig *config, int entry); +bool ppFocusGetFWHM (pmConfig *config, psVector *focus, psVector *fwhm); +bool ppFocusFitFWHM (pmConfig *config, psVector *focus, psVector *fwhm); + +void ppFocusDropCamera (pmConfig *config); + +#endif Index: /branches/rel-1-0/ppImage/src/ppImageAddstar.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageAddstar.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageAddstar.c (revision 11645) @@ -0,0 +1,120 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +bool setFilename (pmFPAfile *file, pmFPAview *view); +bool addstarFile (char *addstarCommand, char *filename); + +bool ppImageAddstar (pmConfig *config) { + + bool status; + pmChip *chip; + pmCell *cell; + pmReadout *readout; + + // select recipe options supplied on command line + psMetadata *recipe = psMetadataLookupPtr (&status, config->recipes, RECIPE_NAME); + + // find a pmFPAfile PSASTRO.OUTPUT + pmFPAfile *file = psMetadataLookupPtr (&status, config->files, "PSASTRO.OUTPUT"); + if (!status) { + psError (PS_ERR_IO, true, "no PSASTRO.OUTPUT file defined"); + return false; + } + + // find the addstar command: %s is replaced with name of output file + char *addstarCommand = psMetadataLookupStr (&status, recipe, "ADDSTAR.COMMAND"); + + pmFPAview *view = pmFPAviewAlloc (0); + if (file->fileLevel == PM_FPA_LEVEL_CHIP) { + // call addstar on this file + setFilename (file, view); + addstarFile (addstarCommand, file->filename); + return true; + } + + while ((chip = pmFPAviewNextChip (view, file->fpa, 1)) != NULL) { + psLogMsg ("ppImageAddstar", 4, "Chip %d: %x %x\n", view->chip, chip->file_exists, chip->process); + if (!chip->process || !chip->file_exists) { continue; } + + if (file->fileLevel == PM_FPA_LEVEL_CHIP) { + // call addstar on this file + setFilename (file, view); + addstarFile (addstarCommand, file->filename); + continue; + } + + while ((cell = pmFPAviewNextCell (view, file->fpa, 1)) != NULL) { + psLogMsg ("ppImageAddstar", 4, "Cell %d: %x %x\n", view->cell, cell->file_exists, cell->process); + if (! cell->process || ! cell->file_exists) { continue; } + + if (file->fileLevel == PM_FPA_LEVEL_CELL) { + // call addstar on this file + setFilename (file, view); + addstarFile (addstarCommand, file->filename); + continue; + } + + // process each of the readouts + while ((readout = pmFPAviewNextReadout (view, file->fpa, 1)) != NULL) { + if (! readout->data_exists) { continue; } + + if (file->fileLevel == PM_FPA_LEVEL_READOUT) { + // call addstar on this file + setFilename (file, view); + addstarFile (addstarCommand, file->filename); + continue; + } else { + psError (PS_ERR_IO, true, "inconsistent fileLevel for %s: %d\n", file->name, file->fileLevel); + return false; + } + } + } + } + return true; +} + +bool setFilename (pmFPAfile *file, pmFPAview *view) { + + // determine the file name + // free a name allocated earlier + psFree (file->filename); + file->filename = pmFPAfileNameFromRule (file->filerule, file, view); + if (file->filename == NULL) { + psError(PS_ERR_IO, true, "Filename is NULL"); + return false; + } + + // indirect filenames are not allowed for output + if (!strcasecmp (file->filename, "@FILES")) { + psError(PS_ERR_IO, true, "indirect filenames are not allowed for output : %s\n", file->name); + return false; + } + if (!strcasecmp (file->filename, "@DETDB")) { + psError(PS_ERR_IO, true, "detrend db filenames are not allowed for output : %s\n", file->name); + return false; + } + return true; +} + +bool addstarFile (char *addstarCommand, char *filename) { + + bool status; + char *addstarLine = psStringCopy(addstarCommand); + psStringSubstitute(&addstarCommand, filename, "%s"); + + // catch addstar stderr/stdout and do what? + psIOBuffer *buffer = psIOBufferAlloc (512); + psPipe *pipe = psPipeOpen (addstarLine); + status = psIOBufferReadEmpty (buffer, 100, pipe->fd_stdout); + if (!status) { + psError (PS_ERR_IO, false, "detselect is not responding"); + } + psFree (addstarLine); + psFree (buffer); + psPipeClose (pipe); + psFree (pipe); + return status; +} Index: /branches/rel-1-0/ppImage/src/ppImageArguments.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageArguments.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageArguments.c (revision 11645) @@ -0,0 +1,128 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" +#include "ppImageVersion.h" +#include "ppStatsVersion.h" + +static void usage (void) { + fprintf(stderr, "USAGE: ppImage [-file INPUT.fits] [-list INPUT.txt] OUTPUT\n\n"); + fprintf(stderr, "Optional arguments:\n"); + fprintf(stderr, "\t-stat STAT.mdc: Output statistics into STAT.mdc\n"); + fprintf(stderr, "\t-isfringe: The input image contains fringe data.\n"); + fprintf(stderr, "\t-usemask MASKVAL: Use this mask value (override recipe).\n"); + fprintf(stderr, "\t-chip CHIPNUM: Only process this chip number.\n"); + fprintf(stderr, "\t-norm VALUE: Divide through by this value when done.\n"); + fprintf(stderr, "\n"); + fprintf(stderr, "Input options (single file / file list):\n"); + fprintf(stderr, "\t-bias/-biaslist: Bias image.\n"); + fprintf(stderr, "\t-dark/-darklist: Dark image.\n"); + fprintf(stderr, "\t-shutter/-shutterlist: Shutter image.\n"); + fprintf(stderr, "\t-flat/-flatlist: Flat image.\n"); + fprintf(stderr, "\t-mask/-masklist: Mask image.\n"); + fprintf(stderr, "\t-fringe/-fringelist: Fringe image and data.\n"); + fprintf(stderr, "\n"); + exit (2); +} + +pmConfig *ppImageArguments(int argc, char **argv) +{ + int argnum; // Argument number of interest + + if (argc == 1) { + usage(); + } + + if (psArgumentGet (argc, argv, "-version")) { + psString version; + version = ppImageVersionLong(); fprintf (stdout, "%s\n", version); psFree (version); + version = ppStatsVersionLong(); fprintf (stdout, "%s\n", version); psFree (version); + version = psphotVersionLong(); fprintf (stdout, "%s\n", version); psFree (version); + version = psastroVersionLong(); fprintf (stdout, "%s\n", version); psFree (version); + version = psModulesVersionLong(); fprintf (stdout, "%s\n", version); psFree (version); + version = psLibVersionLong(); fprintf (stdout, "%s\n", version); psFree (version); + exit (0); + } + + // load the site-wide configuration information + pmConfig *config = pmConfigRead(&argc, argv, RECIPE_NAME); + if (config == NULL) { + psErrorStackPrint(stderr, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + + // save the following additional recipe values based on command-line options + // these options override the PPIMAGE recipe values loaded from recipe files + psMetadata *options = pmConfigRecipeOptions (config, RECIPE_NAME); + + // save these recipe options until we have loaded the options + // psMetadata *options = psMetadataAlloc (); + // psMetadataAddPtr (config->arguments, PS_LIST_TAIL, "PPIMAGE.OPTIONS", PS_DATA_METADATA, "", options); + + if ((argnum = psArgumentGet(argc, argv, "-stat"))) { + psArgumentRemove(argnum, &argc, argv); + psMetadataAddStr(options, PS_LIST_TAIL, "STATS", PS_META_REPLACE, + "Filename for summary statistics", argv[argnum]); + psArgumentRemove(argnum, &argc, argv); + } + + if ((argnum = psArgumentGet(argc, argv, "-isfringe"))) { + psArgumentRemove(argnum, &argc, argv); + psMetadataAddBool(options, PS_LIST_TAIL, "INPUT_IS_FRINGE", PS_META_REPLACE, + "Input is fringe image", true); + } + + // the following options override the PPIMAGE recipe options + + // recipe option: -usemask : override MASK setting in phase2.recipe + if ((argnum = psArgumentGet(argc, argv, "-usemask"))) { + psArgumentRemove (argnum, &argc, argv); + psMetadataAddBool (options, PS_LIST_TAIL, "MASK", PS_META_REPLACE, "", true); + psArgumentRemove (argnum, &argc, argv); + } + + // XXX add other PPIMAGE recipe options here + + // drop the local view on the options (saved on config->arguments) + psFree (options); + + // the input file is a required argument; if not found, we will exit + bool status = pmConfigFileSetsMD (config->arguments, config, "INPUT", "-file", "-list"); + if (!status) { + usage (); + } + + // if these command-line options are supplied, load the file name lists into config->arguments + // override any configuration-specified source for these files + pmConfigFileSetsMD (config->arguments, config, "BIAS", "-bias", "-biaslist"); + pmConfigFileSetsMD (config->arguments, config, "DARK", "-dark", "-darklist"); + pmConfigFileSetsMD (config->arguments, config, "SHUTTER", "-shutter", "-shutterlist"); + pmConfigFileSetsMD (config->arguments, config, "FLAT", "-flat", "-flatlist"); + pmConfigFileSetsMD (config->arguments, config, "MASK", "-mask", "-masklist"); + pmConfigFileSetsMD (config->arguments, config, "FRINGE", "-fringe", "-fringelist"); + + // chip selection is used to limit chips to be processed + if ((argnum = psArgumentGet (argc, argv, "-chip"))) { + psArgumentRemove (argnum, &argc, argv); + psMetadataAddStr (config->arguments, PS_LIST_TAIL, "CHIP_SELECTIONS", PS_DATA_STRING, "", + argv[argnum]); + psArgumentRemove (argnum, &argc, argv); + } + + // Optional normalisation factor + if ((argnum = psArgumentGet(argc, argv, "-norm"))) { + psArgumentRemove(argnum, &argc, argv); + float norm = atof(argv[argnum]); + psMetadataAddF32(config->arguments, PS_LIST_TAIL, "NORMALISATION", 0, + "Normalisation to apply", norm); + psArgumentRemove(argnum, &argc, argv); + } + + if (argc != 2) usage (); + + // Add the input and output images (which remain on the command-line) to the arguments list + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "OUTPUT", 0, "Name of the output image", argv[1]); + + return config; +} Index: /branches/rel-1-0/ppImage/src/ppImageAstrom.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageAstrom.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageAstrom.c (revision 11645) @@ -0,0 +1,81 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +// this function is mostly equivalent to the top-level of psastro, with some +// modifications since the data has already been loaded. +bool ppImageAstrom (pmConfig *config) { + + bool status; + + // select recipe options supplied on command line + // XXX move these options to the "PSASTRO" recipe? + psMetadata *recipe = psMetadataLookupPtr (&status, config->recipes, PSASTRO_RECIPE); + + // find or define a pmFPAfile PSPHOT.INPUT + pmFPAfile *input = psMetadataLookupPtr (&status, config->files, "PSASTRO.INPUT"); + if (!status) { + // psphotReadout requires a pmFPAfile supplied with the name PSASTRO.INPUT + // create a pmFPAfile which points at PSPHOT.OUTPUT + // mode is 'REFERENCE' to prevent double frees of the fpa + pmFPAfile *output = psMetadataLookupPtr (&status, config->files, "PSPHOT.OUTPUT"); + input = pmFPAfileDefineInput (config, output->fpa, "PSASTRO.INPUT"); + input->mode = PM_FPA_MODE_REFERENCE; + + pmFPAfileDefineOutput (config, input->fpa, "PSASTRO.OUTPUT"); + } + + // convert the output sources created by psphot into astrometry objects + if (!psastroConvertFPA (input->fpa, recipe)) { + psErrorStackPrint(stderr, "error loading input data\n"); + exit (1); + } + + // interpret the available initial astrometric information + // apply the initial guess + if (!psastroAstromGuess (config)) { + psErrorStackPrint(stderr, "failed to determine initial astrometry guess\n"); + exit (1); + } + + // load the reference stars overlapping the data stars + psArray *refs = psastroLoadRefstars(config); + if (!refs) { + psErrorStackPrint(stderr, "failed to load reference data\n"); + exit (1); + } + + if (!psastroChooseRefstars (config, refs)) { + psErrorStackPrint(stderr, "failed to select reference data for chips\n"); + exit (1); + } + + // XXX does this check the recipe?? + bool chipastro = psMetadataLookupBool (NULL, config->arguments, "PSASTRO.CHIP.MODE"); + bool mosastro = psMetadataLookupBool (NULL, config->arguments, "PSASTRO.MOSAIC.MODE"); + if (!chipastro && !mosastro) { + psLogMsg ("psastro", 3, "no astrometry mode selected, assuming chip mode\n"); + chipastro= true; + } + + if (chipastro) { + if (!psastroChipAstrom (config, refs)) { + psErrorStackPrint(stderr, "failed to perform single chip astrometry\n"); + exit (1); + } + } + if (mosastro) { + if (!psastroMosaicAstrom (config, refs)) { + psErrorStackPrint(stderr, "failed to perform mosaic camera astrometry\n"); + exit (1); + } + } + + // XXX should this be left to the ppImageLoop? + psastroDataSave (config); + + psFree (refs); + return true; +} Index: /branches/rel-1-0/ppImage/src/ppImageCleanup.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageCleanup.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageCleanup.c (revision 11645) @@ -0,0 +1,24 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +void ppImageCleanup (pmConfig *config, ppImageOptions *options) +{ + // Free memory used by ppImage + psFree(options); + psFree(config); + + // Free memory used by psModules + pmConceptsDone(); + pmConfigDone(); + pmModelGroupCleanup(); + + // Free memory used by psLib + psLibFinalize(); + + fprintf (stderr, "Found %d leaks at %s\n", psMemCheckLeaks (0, NULL, stderr, false), "ppImage"); + + return; +} Index: /branches/rel-1-0/ppImage/src/ppImageDetrend.h =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageDetrend.h (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageDetrend.h (revision 11645) @@ -0,0 +1,27 @@ +#ifndef PP_IMAGE_DETREND_H +#define PP_IMAGE_DETREND_H + +#include "pslib.h" +#include "psmodules.h" +#include "ppImageOptions.h" + +// Cells to be used in the detrend +typedef struct { + pmCell *input; // The input cell, to be operated upon + pmCell *mask; // The bad pixel mask + pmCell *bias; // The bias correction + pmCell *dark; // The dark correction + pmCell *flat; // The flat-field correction +} ppImageDetrend; + + +bool ppImageDetrendCell(ppImageDetrend *detrend, ppImageOptions *options, pmConfig *config); +bool ppImageDetrendMask(pmCell *cell, pmReadout *input, pmReadout *mask); +bool ppImageDetrendNonLinear(pmCell *cell, pmReadout *input, ppImageOptions *options); +bool ppImageDetrendNonLinearLookup(pmReadout *input, psMetadataItem *dataItem); +bool ppImageDetrendNonLinearPolynomial(pmReadout *input, psMetadataItem *dataItem); +bool ppImageDetrendBias(pmReadout *inputReadout, pmReadout *bias, pmReadout *dark, ppImageOptions *options); +pmReadout* ppImageDetrendSelectFirst(pmCell *cell, char *name, bool doThis); + + +#endif Index: /branches/rel-1-0/ppImage/src/ppImageDetrendFringe.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageDetrendFringe.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageDetrendFringe.c (revision 11645) @@ -0,0 +1,222 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "ppImageOptions.h" +#include "ppImageDetrendFringe.h" + +bool ppImageDetrendFringeMeasure(pmReadout *readout, pmCell *fringe, const ppImageOptions *options) +{ + PS_ASSERT_PTR_NON_NULL(readout, false); + PS_ASSERT_PTR_NON_NULL(fringe, false); + PS_ASSERT_PTR_NON_NULL(options, false); + + // Reference fringe measurements + psArray *references = psMemIncrRefCounter(psMetadataLookupPtr(NULL, fringe->analysis, + "FRINGE.MEASUREMENTS")); + if (!references) { + references = pmFringesParse(fringe); // Reference fringes + if (!references) { + psError(PS_ERR_IO, false, "Unable to find fringe references.\n"); + return false; + } + psMetadataAdd(fringe->analysis, PS_LIST_TAIL, "FRINGE.MEASUREMENTS", PS_DATA_UNKNOWN, + "Fringe measurements", references); + } + + pmFringeStats *reference = references->data[0]; // Take the first as representative + pmFringeRegions *regions = reference->regions; // Regions to measure + pmFringeStats *measurements = pmFringeStatsMeasure(regions, readout, options->maskValue); // Fringe stats + + // Normalise measurements by the exposure time + bool mdok; // Status of MD lookup + float expTime = psMetadataLookupF32(&mdok, readout->parent->concepts, "CELL.EXPOSURE"); // Exp. time + if (!mdok || !isfinite(expTime)) { + psError(PS_ERR_UNKNOWN, false, "CELL.EXPOSURE is not set for --- can't normalise fringes\n"); + psFree(measurements); + return false; + } + if (expTime == 0) { + psWarning("Exposure time is zero --- are you sure you want fringe subtraction?\n"); + expTime = 1.0; + } + psBinaryOp(measurements->f, measurements->f, "*", psScalarAlloc(1.0 / expTime, PS_TYPE_F32)); + psBinaryOp(measurements->df, measurements->df, "*", psScalarAlloc(1.0 / expTime, PS_TYPE_F32)); + + // Science fringe measurements + pmFringeStats *previous = psMetadataLookupPtr(NULL, readout->parent->analysis, "FRINGE.MEASUREMENTS"); + if (previous) { + // Multiple readouts: concatenate + psArray *concatenate = psArrayAlloc(2); // Array to hold fringes + + // Concatenate science measurements + concatenate->data[0] = previous; + concatenate->data[1] = measurements; + pmFringeStats *new = pmFringeStatsConcatenate(concatenate, NULL, NULL); // New measurements + psFree(measurements); + measurements = new; + + // Concatenate reference measurements (duplication, so the science and reference line up) + for (int i = 0; i < references->n; i++) { + concatenate->data[0] = concatenate->data[1] = references->data[i]; + pmFringeStats *refNew = pmFringeStatsConcatenate(concatenate, NULL, NULL); + psFree(references->data[i]); + references->data[i] = refNew; + } + concatenate->data[0] = concatenate->data[1] = NULL; + psFree(concatenate); + } + + psMetadataAdd(readout->parent->analysis, PS_LIST_TAIL, "FRINGE.MEASUREMENTS", + PS_DATA_UNKNOWN | PS_META_REPLACE, "Fringe measurements", measurements); + psFree(measurements); + + psFree(references); + + return true; +} + + +// Pull the fringes out of the cell analysis FRINGE.MEASUREMENTS for a chip +static psArray *getFringes(const pmChip *chip // Chip of interest + ) +{ + psArray *cells = chip->cells; // Component cells + psArray *fringes = psArrayAlloc(cells->n); // Fringes, to return + for (int i = 0; i < cells->n; i++) { + pmCell *cell = cells->data[i]; // Cell of interest + fringes->data[i] = psMemIncrRefCounter(psMetadataLookupPtr(NULL, cell->analysis, + "FRINGE.MEASUREMENTS")); + } + + return fringes; +} + + +// Solve the fringe system: we have science fringe measurements for each cell, and an array of reference +// fringe measurements for each cell. Need to concatenate these together first, and then solve. +bool ppImageDetrendFringeSolve(pmChip *scienceChip, const pmChip *refChip, const ppImageOptions *options) +{ + PS_ASSERT_PTR_NON_NULL(scienceChip, NULL); + PS_ASSERT_PTR_NON_NULL(refChip, NULL); + PS_ASSERT_PTR_NON_NULL(options, NULL); + + psArray *science = getFringes(scienceChip); // Fringe measurements on science chip + pmFringeStats *scienceCat = pmFringeStatsConcatenate(science, NULL, NULL); // Science fringes + psFree(science); + + // Need to transform the array of cells each with an array of fringes --> array of fringes for the chip as + // a whole + psArray *references = getFringes(refChip); // Fringe measurements on reference chip + int numRefs = ((psArray*)references->data[0])->n; // Number of reference fringes + psArray *referencesCat = psArrayAlloc(numRefs); // Reference fringes + for (int i = 0; i < numRefs; i++) { // Iterate over fringes + psArray *refs = psArrayAlloc(references->n); // Array of fringes for each cell + for (int j = 0; j < references->n; j++) { // Iterate over cells + psArray *ref = references->data[j]; // Array of references for this cell + refs->data[j] = psMemIncrRefCounter(ref->data[i]); + } + referencesCat->data[i] = pmFringeStatsConcatenate(refs, NULL, NULL); + psFree(refs); + } + psFree(references); + + // Now we can solve + psTrace("ppImage", 3, "Solving fringe system...\n"); + pmFringeScale *solution = pmFringeScaleMeasure(scienceCat, referencesCat, options->fringeRej, + options->fringeIter, options->fringeKeep); + + psMetadataAdd(scienceChip->analysis, PS_LIST_TAIL, "FRINGE.SOLUTION", PS_DATA_UNKNOWN, + "Fringe solution", solution); + psFree(solution); + psFree(scienceCat); + psFree(referencesCat); + + return true; +} + + +bool ppImageDetrendFringeGenerate(pmCell *science, pmCell *fringes) +{ + PS_ASSERT_PTR_NON_NULL(science, false); + PS_ASSERT_PTR_NON_NULL(fringes, false); + + pmFringeScale *solution = psMetadataLookupPtr(NULL, science->parent->analysis, "FRINGE.SOLUTION"); + if (!solution) { + psError(PS_ERR_UNEXPECTED_NULL, false, "Unable to find fringe solution.\n"); + return false; + } + assert(fringes->readouts->n == solution->nFringeFrames); + + bool mdok; // Status of MD lookup + float expTime = psMetadataLookupF32(&mdok, science->concepts, "CELL.EXPOSURE"); // Exp. time + if (!mdok || !isfinite(expTime)) { + psError(PS_ERR_UNKNOWN, true, "CELL.EXPOSURE is not set --- can't renormalise fringes\n"); + return false; + } + if (expTime == 0) { + psWarning("Exposure time is zero --- are you sure you want fringe subtraction?\n"); + expTime = 1.0; + } + + pmHDU *hdu = pmHDUFromCell(science);// HDU of interest + + // Construct the fringe image from the scale + psTrace("ppImage", 3, "Generating fringe correction...\n"); + psImage *sumFringe = NULL; // Sum of the fringes + for (int i = 0; i < solution->nFringeFrames; i++) { + pmReadout *fringeRO = fringes->readouts->data[i]; // Fringe readout + psImage *fringe = fringeRO->image; // Fringe image + psTrace("ppImage", 5, "Scale for fringe component %d is %f\n", + i, solution->coeff->data.F32[i + 1]); + psBinaryOp(fringe, fringe, "*", psScalarAlloc(solution->coeff->data.F32[i + 1], PS_TYPE_F32)); + if (!sumFringe) { + sumFringe = psImageCopy(NULL, fringe, PS_TYPE_F32); + } else { + psBinaryOp(sumFringe, sumFringe, "+", fringe); + } + + psVector *md5 = psImageMD5(fringe); // md5 hash + psString md5string = psMD5toString(md5); // String + psFree(md5); + psStringPrepend(&md5string, "Fringe image %d (scale %.3f) MD5: ", + i, solution->coeff->data.F32[i + 1]); + psMetadataAddStr(hdu->header, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, + md5string, ""); + psFree(md5string); + } + if (expTime != 1.0) { + psBinaryOp(sumFringe, sumFringe, "*", psScalarAlloc(expTime, PS_TYPE_F32)); + } + + // Apply the correction to all readouts + psArray *readouts = science->readouts; // Array of science readouts + for (int i = 0; i < readouts->n; i++) { + pmReadout *readout = readouts->data[i]; // Readout of interest + if (!readout || !readout->data_exists) { + continue; + } + + // XXX: Make generic, so subregions may be subtracted as well + if (!psBinaryOp(readout->image, readout->image, "-", sumFringe)) { + psError(PS_ERR_UNEXPECTED_NULL, false, "Unable to subtract fringe.\n"); + return false; + } + } + psFree(sumFringe); + + psTime *time = psTimeGetNow(PS_TIME_TAI); // The time now, used for reporting + psString timeString = psTimeToISO(time); // String with time + psFree(time); + psStringPrepend(&timeString, "Fringe correction completed at "); + psMetadataAddStr(hdu->header, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, + timeString, ""); + psFree(timeString); + + return sumFringe; +} + Index: /branches/rel-1-0/ppImage/src/ppImageDetrendFringe.h =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageDetrendFringe.h (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageDetrendFringe.h (revision 11645) @@ -0,0 +1,26 @@ +#ifndef PP_IMAGE_DETREND_FRINGE_H +#define PP_IMAGE_DETREND_FRINGE_H + +#include +#include +#include "ppImageOptions.h" + +// Measure fringes +bool ppImageDetrendFringeMeasure(pmReadout *readout, // Readout to measure + pmCell *fringe, // Fringe cell (each readout is a different component) + const ppImageOptions *options // Options + ); + +// Solve the fringe system +bool ppImageDetrendFringeSolve(pmChip *scienceChip, // Chip with science + const pmChip *refChip, // Chip with reference fringes + const ppImageOptions *options // Options + ); + +// Generate fringe frame +bool ppImageDetrendFringeGenerate(pmCell *science, // Science cell + pmCell *fringes // Fringe cell, one readout per fringe component + ); + + +#endif Index: /branches/rel-1-0/ppImage/src/ppImageDetrendNonLinear.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageDetrendNonLinear.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageDetrendNonLinear.c (revision 11645) @@ -0,0 +1,114 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +bool ppImageDetrendNonLinearPolynomial(pmReadout *input, psMetadataItem *dataItem) { + + // These are the polynomial coefficients + psVector *coeff = dataItem->data.V; // The coefficient vector + if (coeff->type.type != PS_TYPE_F64) { + psVector *temp = psVectorCopy(NULL, coeff, PS_TYPE_F64); // F64 version + psFree (coeff); + coeff = temp; + } + psPolynomial1D *correction = psPolynomial1DAlloc(coeff->n - 1, PS_POLYNOMIAL_ORD); + psFree(correction->coeff); + correction->coeff = psMemIncrRefCounter(coeff->data.F64); + pmNonLinearityPolynomial(input, correction); + psFree(coeff); + psFree(correction); + return true; +} + +bool ppImageDetrendNonLinearLookup(pmReadout *input, psMetadataItem *dataItem) { + + // This is a filename: lookup table + char *name = dataItem->data.V; // Filename + psLookupTable *table = psLookupTableAlloc(name, "%f %f", 0); + if (psLookupTableRead(table) <= 0) { + psErrorStackPrint(stderr, "Unable to read non-linearity correction file " + "%s --- ignored\n", name); + return false; + } +#ifdef PRODUCTION + pmNonLinearityLookup(input, table); +#else + psVector *influx = table->values->data[0]; + psVector *outflux = table->values->data[1]; + pmNonLinearityLookup(input, influx, outflux); +#endif + psFree(table); + return true; +} + +bool ppImageDetrendNonLinear(pmReadout *input, ppImageOptions *options) { + + psMetadataItem *concept; + pmCell *cell = input->parent; + + switch (options->nonLinearType) { + case PS_DATA_VECTOR: + ppImageDetrendNonLinearPolynomial (input, options->nonLinearData); + return true; + + case PS_DATA_STRING: + ppImageDetrendNonLinearLookup (input, options->nonLinearData); + return true; + + case PS_DATA_METADATA: + // XXX EAM: this is somewhat confusing : let's wrap in a function when i understand it + + // Go looking for the value in the hierarchy + concept = psMetadataLookup(cell->concepts, options->nonLinearSource); + if (! concept) { + pmChip *chip = cell->parent;// Parent chip + concept = psMetadataLookup(chip->concepts, options->nonLinearSource); + if (! concept) { + pmFPA *fpa = chip->parent; // Parent FPA + concept = psMetadataLookup(fpa->concepts, options->nonLinearSource); + if (! concept) { + psLogMsg("phase2", PS_LOG_WARN, "Unable to find value of concept %s " + "for non-linearity correction --- ignored.\n", (char *)options->nonLinearSource); + return false; + } + } + } + + if (concept->type != PS_DATA_STRING) { + psLogMsg("phase2", PS_LOG_WARN, "Type for concept %p isn't STRING, as" + " expected for non-linearity correction --- ignored.\n", + concept); + return false; + } + + // Get the value of the concept + psString conceptValue = concept->data.V; + psMetadata *folder = (psMetadata *)options->nonLinearData->data.V; + psMetadataItem *optionItem = psMetadataLookup(folder, conceptValue); + if (!optionItem) { + psLogMsg("phase2", PS_LOG_WARN, "Unable to find %s in NONLIN.DATA" + " --- ignored.\n", conceptValue); + return false; + } + + switch (optionItem->type) { + case PS_DATA_VECTOR: + ppImageDetrendNonLinearPolynomial (input, optionItem); + return true; + case PS_DATA_STRING: + ppImageDetrendNonLinearLookup (input, optionItem); + return true; + default: + psLogMsg("phase2", PS_LOG_WARN, "Non-linearity correction " + "desired but unable to interpret NONLIN.DATA for %s" + " --- ignored\n", conceptValue); + return false; + } + default: + psAbort("phase2", "Invalid options->nonLinearType"); + } + return true; +} + Index: /branches/rel-1-0/ppImage/src/ppImageDetrendReadout.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageDetrendReadout.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageDetrendReadout.c (revision 11645) @@ -0,0 +1,93 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" +#include "ppImageDetrendFringe.h" + +bool ppImageDetrendReadout(pmConfig *config, ppImageOptions *options, pmFPAview *view) +{ + // construct a view for the detrend images (which have only one readout) + pmFPAview *detview = pmFPAviewAlloc(0); + *detview = *view; + detview->readout = 0; + + // find the currently selected readout + pmReadout *input = pmFPAfileThisReadout(config->files, view, "PPIMAGE.INPUT"); + +# if 0 + // Non-linearity correction + if (options->doNonLin) { + ppImageDetrendNonLinear(detrend->input, input, options); + } +# endif + + // set up the readouts for dark and bias + pmReadout *dark = NULL; + pmReadout *bias = NULL; + if (options->doBias) { + bias = pmFPAfileThisReadout(config->files, detview, "PPIMAGE.BIAS"); + } + if (options->doDark) { + dark = pmFPAfileThisReadout(config->files, detview, "PPIMAGE.DARK"); + } + + // Bias, dark and overscan subtraction are all merged. + if (options->doBias || options->doDark || options->doOverscan) { + if (!pmBiasSubtract(input, options->overscan, bias, dark)) { + psError(PS_ERR_UNKNOWN, false, "Unable to subtract bias.\n"); + return false; + } + } + + if (options->doMask || options->doShutter || options->doFlat || options->doPhotom) { + // create the target mask and weight images + pmReadoutGenerateMaskWeight(input, true); + } + + // apply the externally supplied mask to the input->mask pixels + if (options->doMask) { + pmReadout *mask = pmFPAfileThisReadout(config->files, detview, "PPIMAGE.MASK"); + pmMaskBadPixels(input, mask, options->maskValue); + } + + // Shutter correction + if (options->doShutter) { + pmReadout *shutter = pmFPAfileThisReadout(config->files, detview, "PPIMAGE.SHUTTER"); + if (!pmShutterCorrectionApply(input, shutter)) { + return false; + } + } + + // Flat-field correction (no options used?) + if (options->doFlat) { + pmReadoutGenerateMaskWeight(input, true); + pmReadout *flat = pmFPAfileThisReadout(config->files, detview, "PPIMAGE.FLAT"); + if (!pmFlatField(input, flat)) { + return false; + } + } + + // Normalisation by a (known) constant + bool mdok; // Status of MD lookup + float norm = psMetadataLookupF32(&mdok, config->arguments, "NORMALISATION"); + if (mdok && isfinite(norm) && norm != 1.0) { + pmHDU *hdu = pmHDUFromReadout(input); // HDU of interest + psString comment = NULL; // Comment to add + psStringAppend(&comment, "Normalisation: %f", norm); + psMetadataAddStr(hdu->header, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, comment, ""); + psFree(comment); + + psBinaryOp(input->image, input->image, "*", psScalarAlloc(norm, PS_TYPE_F32)); + } + + if (options->doFringe) { + pmCell *fringe = pmFPAfileThisCell(config->files, detview, "PPIMAGE.FRINGE"); + if (!ppImageDetrendFringeMeasure(input, fringe, options)) { + return false; + } + } + + psFree(detview); + return true; +} Index: /branches/rel-1-0/ppImage/src/ppImageErrorCodes.c.in =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageErrorCodes.c.in (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageErrorCodes.c.in (revision 11645) @@ -0,0 +1,26 @@ +/* + * The line + { PPIMAGE_ERR_$X{ErrorCode}, "$X{ErrorDescription}"}, + * (without the Xs) + * will be replaced by values from errorCodes.dat + */ +#include "pslib.h" +#include "ppImageErrorCodes.h" + +void ppImageErrorRegister(void) +{ + static psErrorDescription errors[] = { + { PPIMAGE_ERR_BASE, "First value we use; lower values belong to psLib" }, + { PPIMAGE_ERR_${ErrorCode}, "${ErrorDescription}"}, + }; + static int nerror = PPIMAGE_ERR_NERROR - PPIMAGE_ERR_BASE; // number of values in enum + + for (int i = 0; i < nerror; i++) { + psErrorDescription *tmp = psAlloc(sizeof(psErrorDescription)); + p_psMemSetPersistent(tmp, true); + *tmp = errors[i]; + psErrorRegister(tmp, 1); + psFree(tmp); /* it's on the internal list */ + } + nerror = 0; // don't register more than once +} Index: /branches/rel-1-0/ppImage/src/ppImageErrorCodes.dat =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageErrorCodes.dat (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageErrorCodes.dat (revision 11645) @@ -0,0 +1,10 @@ +# +# This file is used to generate ppImageErrorClasses.h +# +BASE = 500 First value we use; lower values belong to psLib +# these errors correspond to standard exit conditions +ARGUMENTS Incorrect arguments +SYS System error +CONFIG Problem in configure files +PROG Programming error +DATA invalid data Index: /branches/rel-1-0/ppImage/src/ppImageErrorCodes.h.in =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageErrorCodes.h.in (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageErrorCodes.h.in (revision 11645) @@ -0,0 +1,18 @@ +#if !defined(PPIMAGE_ERROR_CODES_H) +#define PPIMAGE_ERROR_CODES_H +/* + * The line + * PPIMAGE_ERR_$X{ErrorCode}, + * (without the X) + * + * will be replaced by values from errorCodes.dat + */ +typedef enum { + PPIMAGE_ERR_BASE = 512, + PPIMAGE_ERR_${ErrorCode}, + PPIMAGE_ERR_NERROR +} ppImageErrorCode; + +void ppImageErrorRegister(void); + +#endif Index: /branches/rel-1-0/ppImage/src/ppImageLoop.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageLoop.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageLoop.c (revision 11645) @@ -0,0 +1,223 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include "ppImage.h" +#include "ppImageDetrendFringe.h" +#include "ppImageVersion.h" + +bool ppImageLoop (pmConfig *config, ppImageOptions *options) { + + bool status; + pmChip *chip; + pmCell *cell; + pmReadout *readout; + + bool mdok; // Status of MD lookup + psMetadata *recipe = psMetadataLookupMetadata(&mdok, config->recipes, RECIPE_NAME); + const char *statsName = psMetadataLookupStr(&mdok, recipe, "STATS"); // Filename for statistics + psMetadata *stats = NULL; // Container for statistics + FILE *statsFile = NULL; // File stream for statistics + if (mdok && statsName && strlen(statsName) > 0) { + statsFile = fopen(statsName, "w"); + if (!statsFile) { + psError(PS_ERR_IO, true, "Unable to open statistics file %s for writing.\n", statsName); + } else { + stats = psMetadataAlloc(); + } + } + + pmFPAfile *input = psMetadataLookupPtr(&status, config->files, "PPIMAGE.INPUT"); + if (!status) { + psErrorStackPrint(stderr, "Can't find input data!\n"); + exit(EXIT_FAILURE); + } + + pmFPAview *view = pmFPAviewAlloc(0);// View for level of interest + pmHDU *lastHDU = NULL; // Last HDU that was updated + + // files associated with the science image + if (!pmFPAfileIOChecks(config, view, PM_FPA_BEFORE)) { + return false; + } + + while ((chip = pmFPAviewNextChip(view, input->fpa, 1)) != NULL) { + psLogMsg ("ppImageLoop", 4, "Chip %d: %x %x\n", view->chip, chip->file_exists, chip->process); + if (!chip->process || !chip->file_exists) { + continue; + } + + if (!pmFPAfileIOChecks(config, view, PM_FPA_BEFORE)) { + return false; + } + + while ((cell = pmFPAviewNextCell(view, input->fpa, 1)) != NULL) { + psLogMsg ("ppImageLoop", 4, "Cell %d: %x %x\n", view->cell, cell->file_exists, cell->process); + if (!cell->process || !cell->file_exists) { + continue; + } + if (!pmFPAfileIOChecks(config, view, PM_FPA_BEFORE)) { + return false; + } + + // Put version information into the header + pmHDU *hdu = pmHDUFromCell(cell); + if (hdu && hdu != lastHDU) { + ppImageVersionMetadata(hdu->header); + lastHDU = hdu; + } + + + // process each of the readouts + while ((readout = pmFPAviewNextReadout (view, input->fpa, 1)) != NULL) { + if (!pmFPAfileIOChecks(config, view, PM_FPA_BEFORE)) { + return false; + } + if (!readout->data_exists) { + continue; + } + + // perform the detrend analysis + if (!ppImageDetrendReadout(config, options, view)) { + psError(PS_ERR_UNKNOWN, false, "Unable to detrend readout.\n"); + return false; + } + + } + } + + // Solve the fringe system + if (options->doFringe) { + pmChip *fringe = pmFPAfileThisChip(config->files, view, "PPIMAGE.FRINGE"); + if (!ppImageDetrendFringeSolve(chip, fringe, options)) { + return false; + } + } + + // Go back over the cells to apply the fringe and do statistics + view->cell = view->readout = -1; + while ((cell = pmFPAviewNextCell(view, input->fpa, 1)) != NULL) { + if (!cell->process || !cell->file_exists) { + continue; + } + + // Apply the fringe correction + if (options->doFringe) { + psTrace("ppImage", 3, "Applying fringe correction...\n"); + pmCell *fringeCell = pmFPAfileThisCell(config->files, view, "PPIMAGE.FRINGE"); + if (!ppImageDetrendFringeGenerate(cell, fringeCell)) { + return false; + } + } + + // Perform statistics on the detrended cell + if (stats) { + bool mdok; // Status of MD lookup + pmFPAfile *output = psMetadataLookupPtr(&mdok, config->files, "PPIMAGE.OUTPUT"); + if (!mdok || !output) { + psError(PS_ERR_UNEXPECTED_NULL, true, "Unable to find file PPIMAGE.OUTPUT.\n"); + return false; + } + ppStats(stats, output->fpa, view, config); + } + + // Add MD5 information for cell + pmHDU *hdu = pmHDUFromCell(cell); // HDU that owns the cell + while ((readout = pmFPAviewNextReadout(view, input->fpa, 1)) != NULL) { + const char *chipName = psMetadataLookupStr(NULL, chip->concepts, "CHIP.NAME"); + const char *cellName = psMetadataLookupStr(NULL, cell->concepts, "CELL.NAME"); + + psString headerName = NULL; // Header name for MD5 + psStringAppend(&headerName, "MD5_%s_%s_%d", chipName, cellName, view->readout); + + psVector *md5 = psImageMD5(readout->image); // md5 hash + psString md5string = psMD5toString(md5); // String + psFree(md5); + psMetadataAddStr(hdu->header, PS_LIST_TAIL, headerName, PS_META_REPLACE, + "Image MD5", md5string); + psFree(md5string); + psFree(headerName); + } + +#if 0 + if (!pmFPAfileIOChecks(config, view, PM_FPA_AFTER)) { + return false; + } +#endif + + } + + if (!ppImageMosaicChip(config, view, "PPIMAGE.OUTPUT.CHIP", "PPIMAGE.OUTPUT")) { + psError(PS_ERR_UNKNOWN, false, "Unable to mosaic chip.\n"); + return false; + } + if (!ppImageRebinChip(config, view, "PPIMAGE.BIN1", "PPIMAGE.OUTPUT.CHIP")) { + psError(PS_ERR_UNKNOWN, false, "Unable to bin chip.\n"); + return false; + } + if (!ppImageRebinChip(config, view, "PPIMAGE.BIN2", "PPIMAGE.OUTPUT.CHIP")) { + psError(PS_ERR_UNKNOWN, false, "Unable to bin chip.\n"); + return false; + } + + // we perform photometry on the readouts of this chip in the output + if (options->doPhotom) { + ppImagePhotom(config, view); + } + + // Close cells + view->cell = -1; + while ((cell = pmFPAviewNextCell(view, input->fpa, 1)) != NULL) { + if (!cell->process || !cell->file_exists) { + continue; + } + + if (!pmFPAfileIOChecks(config, view, PM_FPA_AFTER)) { + return false; + } + } + + // Close chip + if (!pmFPAfileIOChecks(config, view, PM_FPA_AFTER)) { + return false; + } + } + + // Write out summary statistics + if (stats) { + const char *statsMDC = psMetadataConfigFormat(stats); + if (!statsMDC || strlen(statsMDC) == 0) { + psError(PS_ERR_IO, false, "Unable to get statistics MDC file.\n"); + } else { + fprintf(statsFile, "%s", statsMDC); + } + psFree(statsMDC); + fclose(statsFile); + + psFree(stats); + } + + // XXX an optional full-scale FPA mosaic could happen here + // it would require us to NOT free PPIMAGE.OUTPUT.CHIP until here + // ppImageMosaicFPA (config, "PPIMAGE.OUTPUT.FPA", "PPIMAGE.OUTPUT.CHIP"); + ppImageMosaicFPA(config, "PPIMAGE.OUTPUT.FPA1", "PPIMAGE.BIN1"); + ppImageMosaicFPA(config, "PPIMAGE.OUTPUT.FPA2", "PPIMAGE.BIN2"); + + // we perform astrometry on all chips after sources have been detected + if (options->doAstromChip || options->doAstromMosaic) { + ppImageAstrom(config); + } + // XXX should we keep the addstar command as an option, or just run it externally? + if (options->doAddstar) { + ppImageAddstar(config); + } + + if (!pmFPAfileIOChecks(config, view, PM_FPA_AFTER)) { + return false; + } + + psFree(view); + + return true; +} Index: /branches/rel-1-0/ppImage/src/ppImageMosaic.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageMosaic.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageMosaic.c (revision 11645) @@ -0,0 +1,68 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +bool ppImageMosaicChip(pmConfig *config, const pmFPAview *view, char *outFile, char *inFile) +{ + bool status; // Status of MD lookup + + pmFPAfile *in = psMetadataLookupPtr(&status, config->files, inFile); // Input file + if (!status) { + psErrorStackPrint(stderr, "Can't find required I/O file!\n"); + exit(EXIT_FAILURE); + } + + pmFPAfile *out = psMetadataLookupPtr(&status, config->files, outFile); // Output file + if (!status) { + psErrorStackPrint(stderr, "Can't find required I/O file!\n"); + exit(EXIT_FAILURE); + } + + pmChip *outChip = pmFPAviewThisChip(view, out->fpa); + pmChip *inChip = pmFPAviewThisChip(view, in->fpa); + if (!outChip->hdu && !outChip->parent->hdu) { + pmFPAAddSourceFromView(out->fpa, view, out->format); + } + + psTrace("pmChipMosaic", 5, "mosaic chip %s to %s (xbin,ybin: %d,%d to %d,%d)\n", + in->name, out->name, in->xBin, in->yBin, out->xBin, out->yBin); + return pmChipMosaic(outChip, inChip); +} + +bool ppImageMosaicFPA (pmConfig *config, char *outFile, char *inFile) +{ + bool status; // Status of MD lookup + + pmFPAfile *in = psMetadataLookupPtr(&status, config->files, inFile); // Input file + if (!status) { + psErrorStackPrint(stderr, "Can't find required I/O file!\n"); + exit(EXIT_FAILURE); + } + + pmFPAfile *out = psMetadataLookupPtr(&status, config->files, outFile); // Output file + if (!status) { + psErrorStackPrint(stderr, "Can't find required I/O file!\n"); + exit(EXIT_FAILURE); + } + + // XXX test printing of all concepts + #if 0 + for (int i = 0; i < in->fpa->chips->n; i++) { + pmChip *chip = in->fpa->chips->data[i]; + for (int j = 0; j < chip->cells->n; j++) { + pmCell *cell = chip->cells->data[j]; + psMetadataPrint(stdout, cell->concepts, 2); + } + } + #endif + + pmFPAview *view = pmFPAviewAlloc(0); + pmFPAAddSourceFromView(out->fpa, view, out->format); + psFree(view); + + psTrace ("pmFPAMosaic", 5, "mosaic fpa %s to %s (xbin,ybin: %d,%d to %d,%d)\n", + in->name, out->name, in->xBin, in->yBin, out->xBin, out->yBin); + return pmFPAMosaic(out->fpa, in->fpa); +} Index: /branches/rel-1-0/ppImage/src/ppImageOptions.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageOptions.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageOptions.c (revision 11645) @@ -0,0 +1,229 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +static void imageOptionsFree(ppImageOptions *options) +{ + psFree(options->overscan); + psFree(options->nonLinearData); + psFree(options->nonLinearSource); +} + +ppImageOptions *ppImageOptionsAlloc(void) +{ + ppImageOptions *options = psAlloc(sizeof(ppImageOptions)); + psMemSetDeallocator(options, (psFreeFunc)imageOptionsFree); + + // Initialise options + options->overscan = NULL; + options->nonLinearData = NULL; + options->nonLinearSource = NULL; + + // default flags for various activities + options->doMask = false; // Mask bad pixels + options->maskValue = 0xff; // Default mask value + + options->doNonLin = false; // Non-linearity correction + options->doBias = false; // Bias subtraction + options->doDark = false; // Dark subtraction + options->doOverscan = false; // Overscan subtraction + options->doShutter = false; // Shutter correction + options->doFlat = false; // Flat-field normalisation + options->doFringe = false; // Fringe subtraction + options->doPhotom = false; // Source identification and photometry + options->doAstromChip = false; // Astrometry + options->doAstromMosaic = false; // Astrometry + + options->BaseFITS = false; // create binned image (scale 1) + options->ChipFITS = false; // create binned image (scale 1) + options->FPA1FITS = false; // create binned image (scale 1) + options->FPA2FITS = false; // create binned image (scale 1) + + options->Bin1FITS = false; // create binned image (scale 1) + options->Bin2FITS = false; // create binned image (scale 2) + options->Bin1JPEG = false; // create jpeg of binned image (scale 1) + options->Bin2JPEG = false; // create jpeg of binned image (scale 2) + + // Overscan defaults + options->overscan = NULL; // Overscan options + + // Fringe defaults + options->fringeRej = NAN; + options->fringeIter = 0; + options->fringeKeep = 1.0; + + // Non-linearity default options + options->nonLinearType = 0; // Type of non-linearity data (vector, string or metadata) + options->nonLinearData = NULL; // The non-linearity data + options->nonLinearSource = NULL; // If the non-linearity data is a menu, this provides the key + + return options; +} + +ppImageOptions *ppImageOptionsParse(pmConfig *config) +{ + bool status; + ppImageOptions *options = ppImageOptionsAlloc (); + + // select the recipe for this analysis + bool mdStatus = false; // Result of MD lookup + psMetadata *recipe = psMetadataLookupMetadata(&mdStatus, config->recipes, RECIPE_NAME); + if (! mdStatus || !recipe) { + psLogMsg("ppImage", PS_LOG_ERROR, "Can't find recipe %s in the RECIPES.\n", RECIPE_NAME); + exit(EXIT_FAILURE); + } + + // Non-linearity recipe options + if (psMetadataLookupBool(NULL, recipe, "NONLIN")) { + psMetadataItem *dataItem = psMetadataLookup(recipe, "NONLIN.DATA"); + if (! dataItem) { + psLogMsg(__func__, PS_LOG_ERROR, "Non-linearity correction desired, but unable to " + "find NONLIN.DATA in recipe %s.", RECIPE_NAME); + exit(EXIT_FAILURE); + } + + options->doNonLin = true; + options->nonLinearType = dataItem->type; + options->nonLinearData = dataItem; + + switch (dataItem->type) { + // No immediate action required + case PS_DATA_VECTOR: + case PS_DATA_STRING: + break; + + // This is a menu; we need the key + case PS_DATA_METADATA: + { + bool status; + options->nonLinearSource = psMetadataLookupStr(&status, recipe, "NONLIN.SOURCE"); + if (! status || ! options->nonLinearSource) { + psLogMsg(__func__, PS_LOG_ERROR, "Non-linearity correction desired, but unable to " + "find NONLIN.SOURCE in recipe %s.", RECIPE_NAME); + exit(EXIT_FAILURE); + } + } + break; + default: + psLogMsg(__func__, PS_LOG_ERROR, "Non-linearity correction desired, but " + "NONLIN.DATA is of invalid type in recipe %s.", RECIPE_NAME); + exit(EXIT_FAILURE); + } + } + + // XXX PAP: The overscan stuff needs to be updated following the reworked API + + // Overscan recipe options + // XXX EAM : we should abort on invalid options. default options? + if (psMetadataLookupBool(NULL, recipe, "OVERSCAN")) { + options->doOverscan = true; + + // Do the overscan as a single value? + bool overscanSingle = psMetadataLookupBool(NULL, recipe, "OVERSCAN.SINGLE"); + + // How do we fit it? + pmFit overscanFit = PM_FIT_NONE; // Fit type for overscan + int overscanOrder = 0; // Order for overscan fit + psString fit = psMetadataLookupStr(NULL, recipe, "OVERSCAN.FIT"); + if (! strcasecmp(fit, "POLYNOMIAL")) { + overscanFit = PM_FIT_POLY_ORD; + overscanOrder = psMetadataLookupS32(NULL, recipe, "OVERSCAN.ORDER"); + } else if (! strcasecmp(fit, "CHEBYSHEV")) { + overscanFit = PM_FIT_POLY_CHEBY; + overscanOrder = psMetadataLookupS32(NULL, recipe, "OVERSCAN.ORDER"); + } else if (! strcasecmp(fit, "SPLINE")) { + overscanFit = PM_FIT_SPLINE; + } else if (strcasecmp(fit, "NONE")) { + psLogMsg(__func__, PS_LOG_WARN, + "OVERSCAN.FIT (%s) in recipe %s is not one of NONE, POLYNOMIAL, or SPLINE", + fit, RECIPE_NAME); + exit(EXIT_FAILURE); + } + + // What method do we use to measure the overscan statistics? + psStats *overscanStats = NULL; // Statistics for overscan + psString stat = psMetadataLookupStr(NULL, recipe, "OVERSCAN.STAT"); + if (! strcasecmp(stat, "MEAN")) { + overscanStats = psStatsAlloc(PS_STAT_SAMPLE_MEAN); + } else if (! strcasecmp(stat, "MEDIAN")) { + overscanStats = psStatsAlloc(PS_STAT_SAMPLE_MEDIAN); + } else { + psErrorStackPrint(stderr, "OVERSCAN.STAT (%s) in recipe %s is not one of MEAN or MEDIAN", + stat, RECIPE_NAME); + exit(EXIT_FAILURE); + } + + // Fill in the options + options->overscan = pmOverscanOptionsAlloc(overscanSingle, overscanFit, overscanOrder, overscanStats); + psFree(overscanStats); + } + + // Mask recipe options + options->doMask = psMetadataLookupBool(NULL, recipe, "MASK"); + psMaskType maskValue = psMetadataLookupU8(&status, recipe, "MASK.VALUE"); + if (status) { + options->maskValue = maskValue; + } + + options->doBias = psMetadataLookupBool(NULL, recipe, "BIAS"); + options->doDark = psMetadataLookupBool(NULL, recipe, "DARK"); + options->doFlat = psMetadataLookupBool(NULL, recipe, "FLAT"); + options->doFringe = psMetadataLookupBool(NULL, recipe, "FRINGE"); + options->doShutter = psMetadataLookupBool(NULL, recipe, "SHUTTER"); + + // binned image options + options->xBin1 = psMetadataLookupS32(&status, recipe, "BIN1.XBIN"); + if (!status) { + psWarning("BIN1.XBIN not found in recipe: setting to default value.\n"); + options->xBin1 = 16; + } + options->yBin1 = psMetadataLookupS32(&status, recipe, "BIN1.YBIN"); + if (!status) { + psWarning("BIN1.YBIN not found in recipe: setting to default value.\n"); + options->yBin1 = 16; + } + + options->xBin2 = psMetadataLookupS32(&status, recipe, "BIN2.XBIN"); + if (!status) { + psWarning("BIN2.XBIN not found in recipe: setting to default value.\n"); + options->xBin1 = 16; + } + options->yBin2 = psMetadataLookupS32(&status, recipe, "BIN2.YBIN"); + if (!status) { + psWarning("BIN2.YBIN not found in recipe: setting to default value.\n"); + options->yBin1 = 16; + } + + options->BaseFITS = psMetadataLookupBool(NULL, recipe, "BASE.FITS"); + options->ChipFITS = psMetadataLookupBool(NULL, recipe, "CHIP.FITS"); + options->FPA1FITS = psMetadataLookupBool(NULL, recipe, "FPA1.FITS"); + options->FPA2FITS = psMetadataLookupBool(NULL, recipe, "FPA2.FITS"); + + options->Bin1FITS = psMetadataLookupBool(NULL, recipe, "BIN1.FITS"); + options->Bin1JPEG = psMetadataLookupBool(NULL, recipe, "BIN1.JPEG"); + options->Bin2FITS = psMetadataLookupBool(NULL, recipe, "BIN2.FITS"); + options->Bin2JPEG = psMetadataLookupBool(NULL, recipe, "BIN2.JPEG"); + + options->doPhotom = psMetadataLookupBool(NULL, recipe, "PHOTOM"); + options->doAstromChip = psMetadataLookupBool(NULL, recipe, "ASTROM.CHIP"); + options->doAstromMosaic = psMetadataLookupBool(NULL, recipe, "ASTROM.MOSAIC"); + if ((options->doAstromChip || options->doAstromMosaic) && !options->doPhotom) { + psLogMsg(__func__, PS_LOG_ERROR, "Invalid PPIMAGE options: cannot do ASTROMetry without PHOTOMetry"); + exit(EXIT_FAILURE); + } + + options->doAddstar = psMetadataLookupBool(NULL, recipe, "ADDSTAR"); + if (options->doAddstar && !(options->doAstromChip || options->doAstromMosaic)) { + psLogMsg(__func__, PS_LOG_ERROR, "Invalid PPIMAGE options: cannot Addstar without Astrometry"); + exit(EXIT_FAILURE); + } + + // Fringe options + options->fringeRej = psMetadataLookupF32(NULL, recipe, "FRINGE.REJ"); + options->fringeIter = psMetadataLookupS32(NULL, recipe, "FRINGE.ITER"); + options->fringeKeep = psMetadataLookupF32(NULL, recipe, "FRINGE.KEEP"); + + return options; +} Index: /branches/rel-1-0/ppImage/src/ppImageOptions.h =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageOptions.h (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageOptions.h (revision 11645) @@ -0,0 +1,51 @@ +#ifndef PP_IMAGE_OPTIONS_H +#define PP_IMAGE_OPTIONS_H + +// Options for ppImage processing +typedef struct { + bool doMask; // Mask bad pixels + psMaskType maskValue; // apply this bit-mask to choose masked bits + + bool doBias; // Bias subtraction + bool doDark; // Dark subtraction + bool doShutter; // Shutter correction + bool doFlat; // Flat-field normalisation + bool doFringe; // Fringe subtraction + bool doPhotom; // Source identification and photometry + bool doAstromChip; // per-chip Astrometry + bool doAstromMosaic; // full-mosaic Astrometry + bool doAddstar; // add results to object database? + + bool doOverscan; // Overscan subtraction + pmOverscanOptions *overscan; // Overscan options + + bool doNonLin; // Non-linearity correction + psDataType nonLinearType; + psMetadataItem *nonLinearData; + void *nonLinearSource; + + bool BaseFITS; + bool ChipFITS; + bool FPA1FITS; + bool FPA2FITS; + + bool Bin1FITS; + bool Bin1JPEG; + int xBin1, yBin1; + + bool Bin2FITS; + bool Bin2JPEG; + int xBin2, yBin2; + + float fringeRej; // Fringe rejection limit + int fringeIter; // Fringe iterations + float fringeKeep; // Fringe keep fraction + +} ppImageOptions; + +ppImageOptions *ppImageOptionsAlloc(void); + +// Determine the processing options +ppImageOptions *ppImageOptionsParse(pmConfig *config); + +#endif Index: /branches/rel-1-0/ppImage/src/ppImageParseCamera.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageParseCamera.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageParseCamera.c (revision 11645) @@ -0,0 +1,262 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +// XXX clean up error checks: return NULL, not psAbort +ppImageOptions *ppImageParseCamera (pmConfig *config) { + + bool status = false; + + // the input image defines the camera, and all recipes and options the follow + pmFPAfile *input = pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.INPUT", "INPUT"); + if (!status || !input) { + psError(PS_ERR_IO, false, "Failed to build FPA from PPIMAGE.INPUT"); + return NULL; + } + +#if 0 + // the input image defines the camera, and all recipes and options the follow + pmFPAfile *inputMask = pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.INPUT.MASK", "INPUT.MASK"); + if (!status || !inputMask) { + psError(PS_ERR_IO, false, "Failed to build FPA from PPIMAGE.INPUT.MASK"); + return NULL; + } + + // the input image defines the camera, and all recipes and options the follow + pmFPAfile *inputWeight = pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.INPUT.WEIGHT", "INPUT.WEIGHT"); + if (!status || !inputWeight) { + psError(PS_ERR_IO, false, "Failed to build FPA from PPIMAGE.INPUT.WEIGHT"); + return NULL; + } +#endif + + // add recipe options supplied on command line + psMetadata *recipe = psMetadataLookupPtr (&status, config->recipes, RECIPE_NAME); + + // parse the options from the metadata format to the ppImageOptions structure + ppImageOptions *options = ppImageOptionsParse (config); + + // the following are defined from the argument list, if given, + // otherwise they revert to the config information + // not all input or output images are used in a given recipe + if (options->doBias) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.BIAS", "BIAS"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.BIAS"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.BIAS", input->fpa, PM_DETREND_TYPE_BIAS); + if (!status) { + psError (PS_ERR_IO, false, "can't find a bias image source"); + return NULL; + } + } + if (options->doDark) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.DARK", "DARK"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.DARK"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.DARK", input->fpa, PM_DETREND_TYPE_DARK); + if (!status) { + psError (PS_ERR_IO, false, "can't find a dark image source"); + return NULL; + } + } + if (options->doMask) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.MASK", "MASK"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.MASK"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.MASK", input->fpa, PM_DETREND_TYPE_MASK); + if (!status) { + psError (PS_ERR_IO, false, "can't find a mask image source"); + return NULL; + } + } + if (options->doShutter) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.SHUTTER", "SHUTTER"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.SHUTTER"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.SHUTTER", input->fpa, PM_DETREND_TYPE_SHUTTER); + if (!status) { + psError (PS_ERR_IO, false, "can't find a shutter image source"); + return NULL; + } + } + + if (options->doFlat) { + // if we fail to find the item in Args, we try again in Conf, then DetDB + // we need to handle the errors and catch serious errors, as opposed to + // data-not-found errors + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.FLAT", "FLAT"); + psErrorClear(); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.FLAT"); + psErrorClear(); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.FLAT", input->fpa, PM_DETREND_TYPE_FLAT); + if (!status) { + psError (PS_ERR_IO, false, "can't find a flat image source"); + return NULL; + } + } + + // fringe frame are only applied for a subset of the filters. + // if the filter is one of those identified by a FRINGE.FILTERS metadata entry + // in ppImage.config, apply the fringe frame + if (options->doFringe) { + // determine filter from the concepts + const char *filter = psMetadataLookupStr(&status, input->fpa->concepts, "FPA.FILTER"); + if (!status || !filter || strlen(filter) == 0) { + psError(PS_ERR_UNEXPECTED_NULL, true, "Unable to find FPA.FILTER.\n"); + return NULL; + } + + // select FRINGE.FILTERS from the recipe and test if the filter matches + // is the mdi saved with psList or string? + psMetadataItem *mdi = psMetadataLookup (recipe, "FRINGE.FILTERS"); + if (mdi == NULL) { + // no valid filters for fringe data for this camera + options->doFringe = false; + goto skip_fringe; + } + // place entry on a list regardless of type + psList *filters = NULL; + if (mdi->type == PS_DATA_STRING) { + filters = psListAlloc(NULL); + psListAdd (filters, PS_LIST_HEAD, mdi); + } else if (mdi->type == PS_DATA_METADATA_MULTI) { + filters = psMemIncrRefCounter(mdi->data.list); + } else { + psAbort ("ppImage", "invalid type for FRINGE.FILTERS"); + } + + // search through list to find the current filter + psListIterator *iter = psListIteratorAlloc (filters, PS_LIST_HEAD, FALSE); + options->doFringe = false; + for (int i = 0; !options->doFringe && (i < filters->n); i++) { + psMetadataItem *item = psListGetAndIncrement (iter); + char *validFilter = item->data.V; + if (strcmp (validFilter, filter)) continue; + options->doFringe = true; + } + psFree(iter); + psFree(filters); + } +skip_fringe: + if (options->doFringe) { + bool status = false; + pmFPAfileDefineFromArgs (&status, config, "PPIMAGE.FRINGE", "FRINGE"); + pmFPAfileDefineFromConf (&status, config, "PPIMAGE.FRINGE"); + pmFPAfileDefineFromDetDB (&status, config, "PPIMAGE.FRINGE", input->fpa, PM_DETREND_TYPE_FRINGE_IMAGE); + if (!status) { + psError (PS_ERR_IO, false, "can't find a fringe image source"); + return NULL; + } + } + + // the following files are output targets + pmFPAfile *output = pmFPAfileDefineOutput(config, input->fpa, "PPIMAGE.OUTPUT"); + if (!output) { + psError(PS_ERR_IO, false, _("Unable to generate output file from PPIMAGE.OUTPUT")); + psFree(options); + return NULL; + } + pmFPAfile *outMask = pmFPAfileDefineOutput(config, input->fpa, "PPIMAGE.OUTPUT.MASK"); + if (!outMask) { + psError(PS_ERR_IO, false, _("Unable to generate output file from PPIMAGE.OUTPUT.MASK")); + psFree(options); + return NULL; + } + pmFPAfile *outWeight = pmFPAfileDefineOutput(config, input->fpa, "PPIMAGE.OUTPUT.WEIGHT"); + if (!outWeight) { + psError(PS_ERR_IO, false, _("Unable to generate output file from PPIMAGE.OUTPUT.WEIGHT")); + psFree(options); + return NULL; + } + pmFPAfile *byChip = pmFPAfileDefineChipMosaic(config, input->fpa, "PPIMAGE.OUTPUT.CHIP"); + if (!byChip) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.OUTPUT.CHIP")); + psFree(options); + return NULL; + } + pmFPAfile *byFPA1 = pmFPAfileDefineFPAMosaic(config, input->fpa, "PPIMAGE.OUTPUT.FPA1"); + if (!byFPA1) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.OUTPUT.FPA1")); + psFree(options); + return NULL; + } + pmFPAfile *byFPA2 = pmFPAfileDefineFPAMosaic(config, input->fpa, "PPIMAGE.OUTPUT.FPA2"); + if (!byFPA2) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.OUTPUT.FPA2")); + psFree(options); + return NULL; + } + + // save any of these files? + output->save = options->BaseFITS; + byChip->save = options->ChipFITS; + byFPA1->save = options->FPA1FITS; + byFPA2->save = options->FPA2FITS; + + // output is used as a carrier: input to byChip + output->freeLevel = PM_FPA_LEVEL_CHIP; + + // define the binned target files (which may just be carriers for some camera configurations) + pmFPAfile *bin1 = pmFPAfileDefineFromFPA (config, byChip->fpa, options->xBin1, options->yBin1, "PPIMAGE.BIN1"); + if (!bin1) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.BIN1")); + psFree(options); + return NULL; + } + + pmFPAfile *bin2 = pmFPAfileDefineFromFPA (config, byChip->fpa, options->xBin2, options->yBin2, "PPIMAGE.BIN2"); + if (!bin2) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.BIN2")); + psFree(options); + return NULL; + } + + // bin1 and bin2 are used as carriers: input for byFPA1, byFPA2 + bin1->freeLevel = PM_FPA_LEVEL_FPA; + bin2->freeLevel = PM_FPA_LEVEL_FPA; + + pmFPAfile *jpg1 = pmFPAfileDefineOutput (config, byFPA1->fpa, "PPIMAGE.JPEG1"); + if (!jpg1) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.JPEG1")); + psFree(options); + return NULL; + } + pmFPAfile *jpg2 = pmFPAfileDefineOutput (config, byFPA2->fpa, "PPIMAGE.JPEG2"); + if (!jpg2) { + psError(PS_ERR_IO, false, _("Unable to generate new file from PPIMAGE.JPEG2")); + psFree(options); + return NULL; + } + + // XXX we could potentially not define these pmFPAfiles if no output is requested... + bin1->save = options->Bin1FITS; + bin2->save = options->Bin2FITS; + jpg1->save = options->Bin1JPEG; + jpg2->save = options->Bin2JPEG; + + // Chip selection: turn on only the chips specified (pass status to suppress missing-key log msg) + char *chipLine = psMetadataLookupStr(&status, config->arguments, "CHIP_SELECTIONS"); + psArray *chips = psStringSplitArray (chipLine, ",", false); + if (chips->n > 0) { + pmFPASelectChip (input->fpa, -1, true); // deselect all chips + for (int i = 0; i < chips->n; i++) { + int chipNum = atoi(chips->data[i]); + if (! pmFPASelectChip(input->fpa, chipNum, false)) { + psError(PS_ERR_IO, false, "Chip number %d doesn't exist in camera.\n", chipNum); + return false; + } + } + } + psFree (chips); + + if (psMetadataLookupBool(NULL, recipe, "INPUT_IS_FRINGE")) { + // It's a fringe file, so change the file type + input->type = PM_FPA_FILE_FRINGE; + output->type = PM_FPA_FILE_FRINGE; + } + + return (options); +} Index: /branches/rel-1-0/ppImage/src/ppImagePhotom.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImagePhotom.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImagePhotom.c (revision 11645) @@ -0,0 +1,151 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +# include "ppImage.h" + +// the top portion of this function is equivalent to psphotParseCamera, +// but different in a few important ways. +// XXX move this section to ppImageParseCamera? +bool ppImagePhotom (pmConfig *config, pmFPAview *view) { + + bool status; + pmCell *cell; + pmReadout *readout; + + psphotModelGroupInit (); + + // select recipe options supplied on command line + psMetadata *recipe = psMetadataLookupPtr (&status, config->recipes, PSPHOT_RECIPE); + + // find or define a pmFPAfile PSPHOT.INPUT + pmFPAfile *input = psMetadataLookupPtr (&status, config->files, "PSPHOT.INPUT"); + if (!status) { + // psphotReadout requires a pmFPAfile supplied with the name PSPHOT.INPUT + // create a pmFPAfile which points to PPIMAGE.OUTPUT.CHIP (guaranteed to be mosaiced by chip) + // mode is 'REFERENCE' to prevent double frees of the fpa + pmFPAfile *output = psMetadataLookupPtr (&status, config->files, "PPIMAGE.OUTPUT.CHIP"); + input = pmFPAfileDefineInput (config, output->fpa, "PSPHOT.INPUT"); + input->mode = PM_FPA_MODE_REFERENCE; + + // we set the freeLevel to be FPA so this pmFPAfile may be used as input to psastro + // XXX make this the resonsibility of a ppImageAstrom call? + // XXX test for psastro requests? + // XXX need to make this an INTERNAL file if output is not requested. + bool saveOutput = psMetadataLookupBool (NULL, recipe, "SAVE.OUTPUT"); + if (saveOutput) { + pmFPAfile *outfile = pmFPAfileDefineOutput (config, input->fpa, "PSPHOT.OUTPUT"); + outfile->freeLevel = PM_FPA_LEVEL_FPA; + } + } + + // if we have requested PSPHOT.SRC (externally supplied sources), attempt to resolve it + if (psMetadataLookupPtr(NULL, config->arguments, "PSPHOT.SRC")) { + pmFPAfileDefineFromArgs (&status, config, "PSPHOT.SRC", "PSPHOT.SRC"); + if (!status) { + psError(PSPHOT_ERR_CONFIG, false, "Failed to find/build PSPHOT.SRC"); + return status; + } + } + + // NOTE the difference here from psphotParseCamera: the mask and weight + // images are supplied by ppImage and need not / should not be loaded here. + + // optionally load the PSF Model and/or fixed stars + pmFPAfileBindFromArgs (NULL, input, config, "PSPHOT.PSF", "PSPHOT.PSF"); + + // set default recipe values here + // XXX place this in a psphot library function? + psMetadataAddStr (recipe, PS_LIST_TAIL, "FITMODE", PS_META_NO_REPLACE, "default fitting mode", "NONE"); + psMetadataAddStr (recipe, PS_LIST_TAIL, "BREAK_POINT", PS_META_NO_REPLACE, "default break point", "NONE"); + psMetadataAddF32 (recipe, PS_LIST_TAIL, "ZERO_PT", PS_META_NO_REPLACE, "default zero point", 25.00); + psMetadataAddS32 (recipe, PS_LIST_TAIL, "BACKGROUND.XBIN", PS_META_NO_REPLACE, "default binning", 64); + psMetadataAddS32 (recipe, PS_LIST_TAIL, "BACKGROUND.YBIN", PS_META_NO_REPLACE, "default binning", 64); + + // determine PHOTCODE from fpa & view, overwrite in recipe + char *photcode = pmConceptsPhotcodeForView (config, input, view); + if (photcode) { + psMetadataAddStr (recipe, PS_LIST_TAIL, "PHOTCODE", PS_META_REPLACE, "photcode from FPA concepts", photcode); + psFree (photcode); + } + + // optionally save the residual image + if (psMetadataLookupBool(NULL, recipe, "SAVE.RESID")) { + if (!pmFPAfileDefineOutput (config, input->fpa, "PSPHOT.RESID")) { + psError(PSPHOT_ERR_CONFIG, false, "Cannot find a rule for PSPHOT.BACKMDL"); + return false; + } + } + + int DX = psMetadataLookupS32 (&status, recipe, "BACKGROUND.XBIN"); + int DY = psMetadataLookupS32 (&status, recipe, "BACKGROUND.YBIN"); + + // these calls construct a new fpa for the I/O handle + + // optionally save the background model (small FITS image) + if (psMetadataLookupBool(NULL, recipe, "SAVE.BACKMDL")) { + if (!pmFPAfileDefineFromFPA (config, input->fpa, DX, DY, "PSPHOT.BACKMDL")) { + psError(PSPHOT_ERR_CONFIG, false, "Cannot find a rule for PSPHOT.BACKMDL"); + return false; + } + } + // optionally save the full background image + if (psMetadataLookupBool(NULL, recipe, "SAVE.BACKGND")) { + if (!pmFPAfileDefineFromFPA (config, input->fpa, 1, 1, "PSPHOT.BACKGND")) { + psError(PSPHOT_ERR_CONFIG, false, "Cannot find a rule for PSPHOT.BACKGND"); + return false; + } + } + // optionally save the background-subtracted image + if (psMetadataLookupBool(NULL, recipe, "SAVE.BACKSUB")) { + if (!pmFPAfileDefineFromFPA (config, input->fpa, 1, 1, "PSPHOT.BACKSUB")) { + psError(PSPHOT_ERR_CONFIG, false, "Cannot find a rule for PSPHOT."); + return false; + } + } + // optionally save the PSF Model + if (psMetadataLookupBool(NULL, recipe, "SAVE.PSF")) { + if (!pmFPAfileDefineOutput (config, input->fpa, "PSPHOT.PSF.SAVE")) { + psError(PSPHOT_ERR_CONFIG, false, "Cannot find a rule for PSPHOT.PSF.SAVE"); + return false; + } + } + + // optionally save output plots + if (psMetadataLookupBool(NULL, recipe, "SAVE.PLOTS")) { + if (!pmFPAfileDefineOutput (config, input->fpa, "SOURCE.PLOT.MOMENTS")) { + psTrace ("psphot", 3, "Cannot find a rule for SOURCE.PLOT.MOMENTS"); + } + if (!pmFPAfileDefineOutput (config, input->fpa, "SOURCE.PLOT.PSFMODEL")) { + psTrace ("psphot", 3, "Cannot find a rule for SOURCE.PLOT.PSFMODEL"); + } + } + + // XXX not tested (or defined?) + // pmFPAfileConstruct (config->files, format, config->camera, "PSPHOT.PSF_SAMPLE"); + + // XXX probably need to deactivate all files and activate the psphot ones explicitly + + // In this section, we iterate over the cells and readout for this chip + // and run the psphot analysis routine here + // XXX worry about this not having unique PHOTCODES... + // view->cell = -1; + // iterate over the cells in the current chip + while ((cell = pmFPAviewNextCell (view, input->fpa, 1)) != NULL) { + psLogMsg ("ppImagePhotom", 4, "Cell %d: %x %x\n", view->cell, cell->file_exists, cell->process); + if (! cell->process || ! cell->file_exists) { continue; } + + // process each of the readouts + while ((readout = pmFPAviewNextReadout (view, input->fpa, 1)) != NULL) { + if (! readout->data_exists) { continue; } + + // run the actual photometry analysis + psphotReadout (config, view); + } + } + + // the PSPHOT.INPUT file is a temporary file used to carry PPIMAGE.OUTPUT.CHIP to psphotReadout + pmFPAfileActivate (config->files, false, "PSPHOT.INPUT"); + + return true; +} Index: /branches/rel-1-0/ppImage/src/ppImageRebinReadout.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageRebinReadout.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageRebinReadout.c (revision 11645) @@ -0,0 +1,100 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "ppImage.h" + +bool ppImageRebinChip (pmConfig *config, pmFPAview *view, char *outName, char *inName) { + + pmCell *cell; + pmReadout *inReadout, *outReadout; + + pmFPAfile *inFile = psMetadataLookupPtr (NULL, config->files, inName); + if (inFile == NULL) return false; + + pmFPAfile *outFile = psMetadataLookupPtr (NULL, config->files, outName); + if (outFile == NULL) return false; + + // XXX double check that chip != -1? + + pmChip *inChip = pmFPAviewThisChip (view, inFile->fpa); + pmChip *outChip = pmFPAviewThisChip (view, outFile->fpa); + if (!pmChipCopyStructure (outChip, inChip, outFile->xBin, outFile->yBin)) { + psError(PS_ERR_UNKNOWN, false, "Unable to copy chip structure."); + return false; + } + + while ((cell = pmFPAviewNextCell (view, inFile->fpa, 1)) != NULL) { + psLogMsg ("ppImageRebinChip", 4, "Cell %d: %x %x\n", view->cell, cell->file_exists, cell->process); + if (! cell->process || ! cell->file_exists) { continue; } + + // process each of the readouts + while ((inReadout = pmFPAviewNextReadout (view, inFile->fpa, 1)) != NULL) { + if (! inReadout->data_exists) { continue; } + + outReadout = pmFPAviewThisReadout (view, outFile->fpa); + + // run the rebin code + ppImageRebinReadout (outReadout, inReadout, outFile); + } + + // Get the original values for the binning + bool mdok = true; // Status of MD lookup + int xBin = psMetadataLookupS32(&mdok, cell->concepts, "CELL.XBIN"); // The binning in x + if (!mdok || xBin <= 0) { + psLogMsg(__func__, PS_LOG_WARN, "CELL.XBIN is not set --- assuming 1.\n"); + xBin = 1; + } + int yBin = psMetadataLookupS32(&mdok, cell->concepts, "CELL.YBIN"); // The binning in y + if (!mdok || yBin <= 0) { + psLogMsg(__func__, PS_LOG_WARN, "CELL.YBIN is not set --- assuming 1.\n"); + yBin = 1; + } + + // Update the concepts with the new values for the binning + pmCell *outCell = pmFPAviewThisCell(view, outFile->fpa); // The output cell + psMetadataItem *binItem = psMetadataLookup(outCell->concepts, "CELL.XBIN"); + binItem->data.S32 = xBin * outFile->xBin; + binItem = psMetadataLookup(outCell->concepts, "CELL.YBIN"); + binItem->data.S32 = yBin * outFile->yBin; + } + + return true; +} + +bool ppImageRebinReadout (pmReadout *output, pmReadout *input, pmFPAfile *outFile) +{ + PS_ASSERT_PTR_NON_NULL(output, false); + PS_ASSERT_IMAGE_NON_NULL(output->image, false); + PS_ASSERT_PTR_NON_NULL(input, false); + PS_ASSERT_IMAGE_NON_NULL(input->image, false); + PS_ASSERT_PTR_NON_NULL(outFile, false); + + // the binning process must not change the size of the output image... + // psStats *stats = psStatsAlloc (PS_STATS_SAMPLE_MEAN); + // psImageRebin (output->image, input->image, NULL, 0, scale, stats); + // psFree (stats); + + int dX = outFile->xBin; + int dY = outFile->yBin; + + int nX = input->image->numCols; + int nY = input->image->numRows; + + // do the rebinning by hand, mean only for test + for (int yOut = 0; yOut < output->image->numRows; yOut++) { + for (int xOut = 0; xOut < output->image->numCols; xOut++) { + float value = 0; + int nPix = 0; + for (int yIn = yOut * dY; (yIn < yOut * dY + dY) && (yIn < nY); yIn ++) { + for (int xIn = xOut * dX; (xIn < xOut * dX + dX) && (xIn < nX); xIn ++) { + value += input->image->data.F32[yIn][xIn]; + nPix ++; + } + } + output->image->data.F32[yOut][xOut] = value / nPix; + } + } + + return true; +} Index: /branches/rel-1-0/ppImage/src/ppImageVersion.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageVersion.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageVersion.c (revision 11645) @@ -0,0 +1,68 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include +#include + +#include "ppImageVersion.h" + +static const char *cvsTag = "$Name: not supported by cvs2svn $";// CVS tag name + +psString ppImageVersion(void) +{ + psString version = NULL; // Version, to return + psStringAppend(&version, "%s-%s",PACKAGE_NAME,PACKAGE_VERSION); + return version; +} + +psString ppImageVersionLong(void) +{ + psString version = ppImageVersion(); // Version, to return + psString tag = psStringStripCVS(cvsTag, "Name"); // CVS tag + psStringAppend(&version, " (cvs tag %s) %s, %s", tag, __DATE__, __TIME__); + psFree(tag); + return version; +} + + +void ppImageVersionMetadata(psMetadata *metadata) +{ + PS_ASSERT_METADATA_NON_NULL(metadata,); + + psString pslib = psLibVersionLong();// psLib version + psString psmodules = psModulesVersionLong(); // psModules version + psString psphot = psphotVersionLong(); // psphot version + psString psastro = psastroVersionLong(); // psastro version + psString ppStats = ppStatsVersionLong(); // ppStats version + psString ppImage = ppImageVersionLong(); // ppImage version + + psTime *time = psTimeGetNow(PS_TIME_TAI); // The time now + psString timeString = psTimeToISO(time); // The time in an ISO string + psFree(time); + psString head = NULL; // Head string + psStringAppend(&head, "ppImage processing at %s. Component information:", timeString); + psFree(timeString); + + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, head, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, pslib, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, psmodules, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, psphot, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, psastro, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, ppStats, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, ppImage, ""); + + psFree(head); + psFree(pslib); + psFree(psmodules); + psFree(psphot); + psFree(psastro); + psFree(ppStats); + psFree(ppImage); + + return; +} Index: /branches/rel-1-0/ppImage/src/ppImageVersion.h =================================================================== --- /branches/rel-1-0/ppImage/src/ppImageVersion.h (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppImageVersion.h (revision 11645) @@ -0,0 +1,14 @@ +#ifndef PP_IMAGE_VERSION_H +#define PP_IMAGE_VERSION_H + +/// Return short version information +psString ppImageVersion(void); + +/// Return long version information +psString ppImageVersionLong(void); + +/// Update the metadata with version information for all dependencies +void ppImageVersionMetadata(psMetadata *metadata ///< Metadata to update with version information + ); + +#endif Index: /branches/rel-1-0/ppImage/src/ppTest.c =================================================================== --- /branches/rel-1-0/ppImage/src/ppTest.c (revision 11645) +++ /branches/rel-1-0/ppImage/src/ppTest.c (revision 11645) @@ -0,0 +1,187 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pslib.h" +#include "psmodules.h" + +int main(int argc, char *argv[]) +{ + psLibInit(NULL); + + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// ppImageConfig.c +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + pmConfig *config = pmConfigRead(&argc, argv, "PPIMAGE"); + if (! config) { + psErrorStackPrint(stderr, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + + // Parse other command-line arguments + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-key", 0, "exposure ID", ""); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-bias", 0, "Name of the bias image", ""); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-dark", 0, "Name of the dark image", ""); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-flat", 0, "Name of the flat-field image", ""); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-mask", 0, "Name of the mask image", ""); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-fringe", 0, "Name of the fringe image", ""); + psMetadataAddS32(config->arguments, PS_LIST_TAIL, "-chip", 0, "Chip number to process (if positive)", -1); + + if (! psArgumentParse(config->arguments, &argc, argv) || argc != 3) { + printf("\nPan-STARRS Phase 2 processing\n\n"); + printf("Usage: %s INPUT.fits OUTPUT.fits\n\n", argv[0]); + psArgumentHelp(config->arguments); + psFree(config->arguments); + exit(EXIT_FAILURE); + } + + // Add the input and output images (which remain on the command-line) to the arguments list + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-input", 0, "Name of the input image", argv[1]); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-output", 0, "Name of the output image", argv[2]); + + // Define database handle, if used +#if 0 + config->database = pmConfigDB(config->site); +#endif + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// ppImageParseCamera.c extract with some alterations +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + + const char *inName = psMetadataLookupStr(NULL, config->arguments, "-input"); + psLogMsg("ppImage", PS_LOG_INFO, "Opening input image: %s\n", inName); + psFits *inFile = psFitsOpen(inName, "r"); // File handle for FITS file + if (! inFile) { + // There's no point in continuing if we can't open the input + psErrorStackPrint(stderr, "Can't open input image: %s\n", inName); + exit(EXIT_FAILURE); + } + psMetadata *phu = psFitsReadHeader(NULL, inFile); // FITS primary header + + psMetadata *cameraFormat = pmConfigCameraFormatFromHeader(config, phu); + if (! config->camera) { + cameraFormat = pmConfigCameraFormatFromHeader(config, phu); + if (! config->camera) { + // There's no point in continuing if we can't recognise what we've got + psErrorStackPrint(stderr, "Can't find camera configuration!\n"); + exit(EXIT_FAILURE); + } + } + // Determine the correct recipe to use + if (! config->recipes && !pmConfigReadRecipes(config, PM_RECIPE_SOURCE_CAMERA | PM_RECIPE_SOURCE_CL)) { + // There's no point in continuing if we can't work out what recipes to use + psErrorStackPrint(stderr, "Can't find recipe configuration!\n"); + exit(EXIT_FAILURE); + } + +#if 1 + const char *outName = psMetadataLookupStr(NULL, config->arguments, "-output"); + psLogMsg("ppImage", PS_LOG_INFO, "Opening output image: %s\n", outName); + psFits *outFile = psFitsOpen(outName, "w"); + if (!outFile) { + // There's no point in continuing if we can't open the output + psErrorStackPrint(stderr, "Can't open output image: %s\n", outName); + exit(EXIT_FAILURE); + } +#endif + + // Construct camera in preparation for reading + pmFPA *fpa = pmFPAConstruct(config->camera); + pmFPAview *view = pmFPAAddSourceFromHeader(fpa, phu, cameraFormat); + printf("View chip: %d\n", view->chip); + printf("View cell: %d\n", view->cell); + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// The action happens here +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + + pmFPAPrint(stdout, fpa, false, true); + exit(0); + + +#if 0 + // A mosaic + psMetadata *mosaicCamera = psMetadataConfigRead(NULL, NULL, "/home/mithrandir/price/ipp/config/mcshort_mosaic/camera.config", true); + pmFPA *mosaicFPA = pmFPAConstruct(mosaicCamera); + psMetadata *mosaicFormat = psMetadataConfigRead(NULL, NULL, "/home/mithrandir/price/ipp/config/mcshort_mosaic/format_mosaic.config", true); + pmFPAview *mosaicView = pmFPAviewAlloc(0); + pmFPAAddSourceFromView(mosaicFPA, mosaicView, mosaicFormat); + psFree(mosaicView); +#endif + + // Read the FPA + pmFPARead(fpa, inFile, NULL); + +#if 1 + psArray *chips = fpa->chips; + pmFPAWrite(fpa, outFile, NULL, false, false); + for (int i = 0; i < chips->n; i++) { + printf("Chip %d\n", i); + pmChip *chip = chips->data[i]; +#if 1 + //pmChipRead(chip, inFile, NULL); + pmChipWrite(chip, outFile, NULL, false, false); + psArray *cells = chip->cells; + for (int j = 0; j < cells->n; j++) { + pmCell *cell = cells->data[j]; + pmCellWrite(cell, outFile, NULL, false); +#if 0 // Read bit by bit + pmReadout *readout = pmReadoutAlloc(cell); + for (int z = 0; pmReadoutReadNext(readout, inFile, z, 512); z++) { + do { + printf("Chip %d, Cell %d, Plane %d, row0 = %d\n", i, j, z, readout->row0); + pmReadoutWriteNext(readout, outFile, z); + //pmFPAPrint(stdout, fpa, false, true); + } while (pmReadoutReadNext(readout, inFile, z, 512)); + } + psFree(readout); +#else // Read the whole cell at once + pmCellRead(cell, inFile, NULL); + pmCellWrite(cell, outFile, NULL, true); +#endif + } +#endif + +#if 0 + pmChipMosaic(mosaicFPA->chips->data[i], chip); + pmChipWrite(mosaicFPA->chips->data[i], outFile, NULL, true, true); + pmChipFreeData(chip); + pmChipFreeData(mosaicFPA->chips->data[i]); +#endif + } +#endif + +#if 0 + pmFPAMosaic(mosaicFPA, fpa); + pmFPAWrite(mosaicFPA, outFile, NULL, true, true); + //pmFPAPrint(stdout, mosaicFPA, true, true); + psFree(mosaicCamera); + psFree(mosaicFormat); + psFree(mosaicView); + //psFree(mosaicFPA); +#endif + +#if 1 + pmFPAPrint(stdout, fpa, true, true); +#endif + +#if 1 + psFitsClose(outFile); +#endif + + psFree(view); + psFree(phu); + psFree(fpa); + psFree(config); + psFitsClose(inFile); + psFree(cameraFormat); + + pmConceptsDone(); + pmConfigDone(); + psLibFinalize(); + + // Pau. +} Index: /branches/rel-1-0/ppImage/test/.cvsignore =================================================================== --- /branches/rel-1-0/ppImage/test/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppImage/test/.cvsignore (revision 11645) @@ -0,0 +1,3 @@ +.gdb_history +*.fits +leaks.dat Index: /branches/rel-1-0/ppImage/test/nonlin.dat =================================================================== --- /branches/rel-1-0/ppImage/test/nonlin.dat (revision 11645) +++ /branches/rel-1-0/ppImage/test/nonlin.dat (revision 11645) @@ -0,0 +1,106 @@ +# Non-linearity correction lookup table +# Col 1: Input value +# Col 2: Corrected value +0 0 +100 1 +200 2 +300 3 +400 4 +500 5 +600 6 +700 7 +800 8 +900 9 +1000 10 +1100 11 +1200 12 +1300 13 +1400 14 +1500 15 +1600 16 +1700 17 +1800 18 +1900 19 +2000 20 +2100 21 +2200 22 +2300 23 +2400 24 +2500 25 +2600 26 +2700 27 +2800 28 +2900 29 +3000 30 +3100 31 +3200 32 +3300 33 +3400 34 +3500 35 +3600 36 +3700 37 +3800 38 +3900 39 +4000 40 +4100 41 +4200 42 +4300 43 +4400 44 +4500 45 +4600 46 +4700 47 +4800 48 +4900 49 +5000 50 +5100 51 +5200 52 +5300 53 +5400 54 +5500 55 +5600 56 +5700 57 +5800 58 +5900 59 +6000 60 +6100 61 +6200 62 +6300 63 +6400 64 +6500 65 +6600 66 +6700 67 +6800 68 +6900 69 +7000 70 +7100 71 +7200 72 +7300 73 +7400 74 +7500 75 +7600 76 +7700 77 +7800 78 +7900 79 +8000 80 +8100 81 +8200 82 +8300 83 +8400 84 +8500 85 +8600 86 +8700 87 +8800 88 +8900 89 +9000 90 +9100 91 +9200 92 +9300 93 +9400 94 +9500 95 +9600 96 +9700 97 +9800 98 +9900 99 +10000 100 +10001 100 +1e6 100 Index: /branches/rel-1-0/ppMerge/.cvsignore =================================================================== --- /branches/rel-1-0/ppMerge/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppMerge/.cvsignore (revision 11645) @@ -0,0 +1,12 @@ +Makefile +Makefile.in +aclocal.m4 +autom4te.cache +compile +config.log +config.status +configure +depcomp +install-sh +missing +test Index: /branches/rel-1-0/ppMerge/Makefile.am =================================================================== --- /branches/rel-1-0/ppMerge/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppMerge/Makefile.am (revision 11645) @@ -0,0 +1,3 @@ +SUBDIRS = src + +CLEANFILES = *~ core core.* Index: /branches/rel-1-0/ppMerge/autogen.sh =================================================================== --- /branches/rel-1-0/ppMerge/autogen.sh (revision 11645) +++ /branches/rel-1-0/ppMerge/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=ppMerge +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOlIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOHEADER installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +#$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ppMerge/configure.ac =================================================================== --- /branches/rel-1-0/ppMerge/configure.ac (revision 11645) +++ /branches/rel-1-0/ppMerge/configure.ac (revision 11645) @@ -0,0 +1,30 @@ +AC_PREREQ(2.59) + +AC_INIT([ppMerge], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([src]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([src/config.h]) +AM_MAINTAINER_MODE + +IPP_STDCFLAGS + +AC_LANG(C) +AC_GNU_SOURCE +AC_PROG_CC +AC_PROG_INSTALL +dnl AC_PROG_LIBTOOL +AC_SYS_LARGEFILE + +PKG_CHECK_MODULES([PSLIB], [pslib >= 1.0.0]) +PKG_CHECK_MODULES([PSMODULE], [psmodules >= 1.0.0]) +PKG_CHECK_MODULES([PPSTATS], [ppStats >= 1.0.0]) + +IPP_STDOPTS +CFLAGS="${CFLAGS=} -Wall -Werror -std=c99" + +AC_CONFIG_FILES([ + Makefile + src/Makefile +]) +AC_OUTPUT Index: /branches/rel-1-0/ppMerge/src/.cvsignore =================================================================== --- /branches/rel-1-0/ppMerge/src/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppMerge/src/.cvsignore (revision 11645) @@ -0,0 +1,7 @@ +.deps +Makefile +Makefile.in +ppMerge +config.h +config.h.in +stamp-h1 Index: /branches/rel-1-0/ppMerge/src/Makefile.am =================================================================== --- /branches/rel-1-0/ppMerge/src/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppMerge/src/Makefile.am (revision 11645) @@ -0,0 +1,35 @@ +bin_PROGRAMS = ppMerge + +ppMerge_CFLAGS = $(PPSTATS_CFLAGS) $(PSMODULE_CFLAGS) $(PSLIB_CFLAGS) +ppMerge_LDFLAGS = $(PPSTATS_LIBS) $(PSMODULE_LIBS) $(PSLIB_LIBS) -Wl,-Bdynamic + +ppMerge_SOURCES = \ + ppMerge.c \ + ppMergeCheckInputs.c \ + ppMergeCombine.c \ + ppMergeConfig.c \ + ppMergeData.c \ + ppMergeMask.c \ + ppMergeOptions.c \ + ppMergeScaleZero.c \ + ppMergeVersion.c + + +noinst_HEADERS = \ + ppMerge.h \ + ppMergeCheckInputs.h \ + ppMergeCombine.h \ + ppMergeConfig.h \ + ppMergeData.h \ + ppMergeMask.h \ + ppMergeOptions.h \ + ppMergeScaleZero.h \ + ppMergeVersion.h + +CLEANFILES = *~ + +clean-local: + -rm -f TAGS +# Tags for emacs +tags: + etags `find . -name \*.[ch] -print` Index: /branches/rel-1-0/ppMerge/src/ppMerge.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMerge.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMerge.c (revision 11645) @@ -0,0 +1,87 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeConfig.h" +#include "ppMergeData.h" +#include "ppMergeOptions.h" +#include "ppMergeCheckInputs.h" +#include "ppMergeCombine.h" +#include "ppMergeScaleZero.h" +#include "ppMergeMask.h" + +//#include "ppMem.h" + +// Yet to do: +// +// 1. Mask pixels with less than the minimum number of electrons +// 2. Sampling factor for background measurement +// 3. On/off pairs + +int main(int argc, char **argv) +{ + psLibInit(NULL); + psMemSetThreadSafety(false); // Turn off thread safety, for more + psTimerStart(TIMERNAME); + + // Parse the configuration and arguments + // Open the input image(s) + // Determine camera, format from header if not already defined + // Construct camera in preparation for reading + pmConfig *config = ppMergeConfig(argc, argv); + + // Set various tasks (define optional operations) + ppMergeOptions *options = ppMergeOptionsParse(config); + + // Check the inputs + ppMergeData *data = ppMergeCheckInputs(options, config); + if (!data) { + psError(PS_ERR_IO, false, "Not enough valid input files.\n"); + exit(EXIT_FAILURE); + } + + psImage *scale = NULL; // The scalings + psImage *zero = NULL; // The zeros + + if (options->mask) { + // Generate a mask + ppMergeMask(data, options, config); + } else { + // Measure the background in each image + ppMergeScaleZero(&scale, &zero, data, options, config); + + // Do the combination and write + ppMergeCombine(scale, zero, data, options, config); + } + + // Output the statistics + if (data->statsFile) { + psString statsOut = psMetadataConfigFormat(data->stats); // String to write out + fprintf(data->statsFile, "%s", statsOut); + psFree(statsOut); + } + +#if 0 + pmFPAPrint(stdout, data->out, true, true); +#endif + + // Cleaning up + psFree(data); + psFree(scale); + psFree(zero); + psFree(options); + psFree(config); + + pmConceptsDone(); + pmConfigDone(); + psLibFinalize(); + +// ppMemCheck(); + + return EXIT_SUCCESS; +} Index: /branches/rel-1-0/ppMerge/src/ppMerge.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMerge.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMerge.h (revision 11645) @@ -0,0 +1,7 @@ +#ifndef PP_MERGE_H +#define PP_MERGE_H + +#define TIMERNAME "ppMerge" +#define PPMERGE_RECIPE "PPMERGE" + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeCheckInputs.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeCheckInputs.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeCheckInputs.c (revision 11645) @@ -0,0 +1,144 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeCheckInputs.h" +#include "ppMergeData.h" + +// Check input files to make sure everything's consistent +ppMergeData *ppMergeCheckInputs(ppMergeOptions *options, // Options + pmConfig *config // Configuration + ) +{ + ppMergeData *data = ppMergeDataAlloc(); // The data, to return + + // Output file + psString outName = psMetadataLookupStr(NULL, config->arguments, "OUTPUT"); // The output file name + assert(outName); // It should be there! + data->outFile = psFitsOpen(outName, "w"); // Output FITS file + if (!data->outFile) { + // There's no point in continuing if we can't open the output + psErrorStackPrint(stderr, "Can't open output image: %s\n", outName); + exit(EXIT_FAILURE); + } + + // Statistics file + psString statsName = psMetadataLookupStr(NULL, config->arguments, "-stats"); // Name for statistics file + if (statsName && strlen(statsName) > 0) { + data->statsFile = fopen(statsName, "w"); + if (!data->statsFile) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to open statistics file %s --- ignored.\n", + statsName); + } + } + + psArray *filenames = psMetadataLookupPtr(NULL, config->arguments, "INPUT"); // The input file names + assert(filenames); + if (!data->in) { + data->in = psArrayAlloc(filenames->n); + } + if (!data->files) { + data->files = psArrayAlloc(filenames->n); + } + int numGood = 0; // Number of good files + for (int i = 0; i < filenames->n; i++) { + psString name = filenames->data[i]; // The name of the file + if (!name || strlen(name) == 0) { + continue; + } + psTrace("ppMerge", 1, "Checking input file %s....\n", name); + psFits *inFile = psFitsOpen(filenames->data[i], "r"); // The FITS file to read + if (!inFile) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to open input file %s --- ignored.\n", name); + // Kick it out + psFree(filenames->data[i]); + filenames->data[i] = NULL; + continue; + } + psMetadata *header = psFitsReadHeader(NULL, inFile); // The FITS (primary) header + data->files->data[i] = inFile; + + // The formats must be identical. The chief reason for this is so that we know what output format to + // use. I guess one could specify a different output format on the command line, but how do we + // generate a PHU for that? Perhaps we could revisit this restriction in the future (construct an + // FPAview from the specified camera format configuration, and use pmFPAAddSourceFromView), but for + // now it's less hassle just to limit the output format to be the input format. + if (!options->format) { + options->format = pmConfigCameraFormatFromHeader(config, header); + psFree(header); + if (!options->format) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to identify camera format for input file %s --- " + "ignored.\n", name); + // Kick it out + psFree(header); + data->in->data[i] = NULL; + continue; + } + } else { + bool valid = false; + if (!pmConfigValidateCameraFormat(&valid, options->format, header)) { + psError (PS_ERR_UNKNOWN, false, "Error in config scripts\n"); + exit (PS_EXIT_CONFIG_ERROR); + } + if (!valid) { + psLogMsg(__func__, PS_LOG_WARN, "Input file %s doesn't match camera format --- ignored.\n", name); + // Kick it out + psFree(header); + data->in->data[i] = NULL; + continue; + } + } + + data->in->data[i] = pmFPAConstruct(config->camera); + pmFPAview *view = pmFPAAddSourceFromHeader(data->in->data[i], header, options->format); + psFree(view); + + // Use the first valid input as the basis for the output --- including the header + if (!data->out) { + psTrace("ppMerge", 5, "Constructing output using %s as a template.\n", name); + data->out = pmFPAConstruct(config->camera); + pmFPAview *view = pmFPAAddSourceFromHeader(data->out, header, options->format); + psFree(view); + } + psFree(header); + + psTrace("ppMerge", 3, "%s checks out.\n", name); + numGood++; + } + + // Count the cells + int numCells = 0; // Number of cells in the output FPA + psArray *chips = data->out->chips; // Array of chips in output + for (int i = 0; i < chips->n; i++) { + pmChip *chip = chips->data[i]; // Chip of interest + if (!chip) { + continue; + } + psArray *cells = chip->cells; // Array of cells + for (int j = 0; j < cells->n; j++) { + pmCell *cell = cells->data[j]; + if (cell) { + numCells++; + } + } + } + data->numCells = numCells; + psTrace("ppMerge", 3, "Output has %d cells.\n", numCells); + + psTrace("ppMerge", 3, "We have %d good inputs.\n", numGood); + if (numGood > 1) { + return data; + } + + psFree(data); + return NULL; +} + + Index: /branches/rel-1-0/ppMerge/src/ppMergeCheckInputs.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeCheckInputs.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeCheckInputs.h (revision 11645) @@ -0,0 +1,13 @@ +#ifndef PP_MERGE_CHECK_INPUTS_H +#define PP_MERGE_CHECK_INPUTS_H + +#include +#include "ppMergeOptions.h" +#include "ppMergeData.h" + +// Check input files to make sure everything's consistent +ppMergeData *ppMergeCheckInputs(ppMergeOptions *options, // Options + pmConfig *config // Configuration + ); + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeCombine.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeCombine.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeCombine.c (revision 11645) @@ -0,0 +1,335 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeData.h" +#include "ppMergeCombine.h" +#include "ppMergeVersion.h" + +#if 0 +static psMemId memId = 0; + +static FILE *dumpFile = NULL; + +static psMemId mbAlloc(psMemBlock *mb) +{ + if (!dumpFile) { + dumpFile = fopen("memBlocks.dat", "w"); + } + fprintf(dumpFile, "Alloc: %12lu\t%12zd\t%s:%d\n", mb->id, mb->userMemorySize, + mb->file, mb->lineno); + return 1; +} + +static void dumpDone(void) +{ + fclose(dumpFile); + exit(EXIT_FAILURE); +} + + +static void memDump(void) +{ + psMemBlock **leaks = NULL; + int numLeaks = psMemCheckLeaks(memId, &leaks, NULL, true); + FILE *memFile = fopen("mem.dat", "w"); + fprintf(memFile, "# MemBlock Size Source\n"); + for (int i = 0; i < numLeaks; i++) { + psMemBlock *mb = leaks[i]; + fprintf(memFile, "%12lu\t%12zd\t%s:%d\n", mb->id, mb->userMemorySize, + mb->file, mb->lineno); + } + fclose(memFile); + psFree(leaks); +} + +static void memCheck(void) +{ + return; + if (psTraceGetLevel("ppMerge") > 9) { + psMemBlock **leaks = NULL; + int numLeaks = psMemCheckLeaks(0, &leaks, NULL, true); + size_t largestSize = 0; + psMemId largest = 0; + size_t totalSize = 0; + for (int i = 0; i < numLeaks; i++) { + psMemBlock *mb = leaks[i]; + totalSize += mb->userMemorySize; + if (mb->userMemorySize > largestSize) { + largestSize = mb->userMemorySize; + largest = mb->id; + } + } + psFree(leaks); + psTrace("ppMerge", 0, "Memory in use: %zd\n", totalSize); + psTrace("ppMerge", 0, "Largest block: %ld\n", largest); + psTrace("ppMerge", 0, "sbrk(): %zd\n", sbrk(0)); + } + return; +} +#endif + + +// Combine the inputs +bool ppMergeCombine(psImage *scales, // Scales for each cell of each integration, or NULL + psImage *zeros, // Zeros for each cell of each integration, or NULL + ppMergeData *data, // Data + ppMergeOptions *options, // Options + pmConfig *config // Configuration + ) +{ + psArray *filenames = psMetadataLookupPtr(NULL, config->arguments, "INPUT"); // The input file names + assert(filenames); // It should be here --- it's put here in ppMergeConfig + + // Sanity checks + assert(!options->scale || scales); + assert(!scales || (scales->type.type == PS_TYPE_F32 && + scales->numCols == data->numCells && + scales->numRows == filenames->n)); + assert(!options->zero || zeros); + assert(!zeros || (zeros->type.type == PS_TYPE_F32 && + zeros->numCols == data->numCells && + zeros->numRows == filenames->n)); + + // Iterate over the FPA + pmFPA *fpa = data->out; // Output FPA + pmFPAview *view = pmFPAviewAlloc(0);// View of FPA, for iteration + int cellNum = -1; // Cell number in the whole FPA + pmFPAWrite(data->out, data->outFile, config->database, true, false); // Write header only + pmChip *chip; // Chip of interest + psRandom *rng = NULL; // Random number generator; required for building a mask + pmHDU *lastHDU = NULL; // Last HDU to be updated + if (options->mask) { + rng = psRandomAlloc(PS_RANDOM_TAUS, 0); + } + while ((chip = pmFPAviewNextChip(view, fpa, 1))) { + pmChipWrite(chip, data->outFile, config->database, true, false); // Write header only + pmCell *cell; // Cell of interest + while ((cell = pmFPAviewNextCell(view, fpa, 1))) { + cellNum++; + pmCellWrite(cell, data->outFile, config->database, true); // Write header only + pmReadout *readout = pmReadoutAlloc(cell); // Output readout of interest + psArray *stack = psArrayAlloc(filenames->n); // Stack of readouts to combine + psVector *cellScales = NULL; // Scales for this cell + if (scales) { + cellScales = psImageCol(NULL, scales, cellNum); + } + psVector *cellZeros = NULL; // Zeros for this cell + if (zeros) { + cellZeros = psImageCol(NULL, zeros, cellNum); + } + + // Read bit by bit + int numRead; // Number of inputs read + int numScan = 0; + + // Put version metadata into header + pmHDU *hdu = pmHDUFromCell(cell); + if (hdu && hdu != lastHDU) { + if (!hdu->header) { + hdu->header = psMetadataAlloc(); + } + ppMergeVersionMetadata(hdu->header); + lastHDU = hdu; + } + + do { + numRead = 0; + for (int i = 0; i < filenames->n; i++) { + if (! filenames->data[i] || strlen(filenames->data[i]) == 0) { + continue; + } + psFits *fits = data->files->data[i]; // FITS file handle + if (!fits) { + continue; + } + + if (!stack->data[i]) { + pmFPA *fpaIn = data->in->data[i]; // Input FPA + pmChip *chipIn = fpaIn->chips->data[view->chip]; // Input chip + pmCell *cellIn = chipIn->cells->data[view->cell]; // Input cell + stack->data[i] = pmReadoutAlloc(cellIn); // Input readout + } + + // Only reading and writing the first readout in each cell (plane 0) + if (pmReadoutReadNext(stack->data[i], fits, 0, options->rows)) { + // If the mask value is set, we want to generate a mask + if (options->combine->maskVal) { + pmReadoutSetMask(stack->data[i]); + } + + // If we're combining with weights, we want to generate weights. + if (options->combine->weights) { + + // If it's a bias or dark, set the gain to zero: noise only contributed by read + if ((!options->zero && !options->scale) || options->darktime) { + pmReadoutSetWeight(stack->data[i], false); + } else { + pmReadoutSetWeight(stack->data[i], true); + } + } + + numRead++; + } else { + psTrace("ppMerge", 3, "Unable to read from file %d for chip %d, " + "cell %d, scan %d\n", i, view->chip, view->cell, numScan); + } + + } + + psTrace("ppMerge", 5, "Chip %d, cell %d, scan %d\n", view->chip, view->cell, numScan); + if (numRead > 0) { + + if (options->shutter) { + readout->image = pmShutterCorrectionMeasure(stack, options->shutterSize, + options->mean, options->stdev, + options->shutterIter, + options->shutterRej, + options->combine->maskVal); + } else { + pmReadoutCombine(readout, stack, cellZeros, cellScales, options->combine); + } + } + numScan++; + + } while (numRead > 0); + + psFree(stack); + + // Set the dark time for the output image, since we normalised + if (options->darktime) { + psMetadataItem *darkItem = psMetadataLookup(cell->concepts, "CELL.DARKTIME"); + darkItem->data.F32 = 1.0; + psMetadataItem *expItem = psMetadataLookup(cell->concepts, "CELL.EXPOSURE"); + expItem->data.F32 = 1.0; + } + + // Measure the fringes for this cell + // + // XXX Need to deal with multiple components: we will do this by building up the components + // Read the existing fringe measurements + // Use existing regions to measure fringe statistics + // Add the new fringe measurements to the existing fringe measurements. + // Write the appended fringe measurements. + // Read in the "output" file to get the existing components. + // Put the new readout into the cell after the existing readouts. + if (options->fringe && readout->image) { + pmFringeRegions *regions = pmFringeRegionsAlloc(options->fringeNum, options->fringeSize, + options->fringeSize, options->fringeSmoothX, + options->fringeSmoothY); // Fringe regions + pmFringeStats *fringe = pmFringeStatsMeasure(regions, readout, options->combine->maskVal); + psFree(regions); + if (!fringe) { + psError(PS_ERR_UNKNOWN, false, "Unable to measure fringe statistics.\n"); + psFree(readout); + return false; + } + + psArray *fringes = psArrayAlloc(1); // Array of fringes + fringes->data[0] = fringe; + + pmFringesFormat(cell, NULL, fringes); + psFree(fringes); // Drop reference + } + + // Add MD5 information for cell + if (readout->image) { + pmHDU *hdu = pmHDUFromCell(cell); // HDU that owns the cell + const char *chipName = psMetadataLookupStr(NULL, chip->concepts, "CHIP.NAME"); + const char *cellName = psMetadataLookupStr(NULL, cell->concepts, "CELL.NAME"); + + psString headerName = NULL; // Header name for MD5 + psStringAppend(&headerName, "MD5_%s_%s", chipName, cellName); + + psVector *md5 = psImageMD5(readout->image); // md5 hash + psString md5string = psMD5toString(md5); // String + psFree(md5); + psMetadataAddStr(hdu->header, PS_LIST_TAIL, headerName, PS_META_REPLACE, + "Image MD5", md5string); + psFree(md5string); + psFree(headerName); + } + + psFree(readout); // Drop reference + + // Statistics on the merged cell + if (data->statsFile) { + data->stats = ppStats(data->stats, data->out, view, config); // Statistics for output FPA + } + + // We threw away the bias sections --- record this + psMetadataItem *biassecItem = psMetadataLookup(cell->concepts, "CELL.BIASSEC"); // Item of BIASSEC + psList *biassecList = biassecItem->data.V; // List of BIASSECs + while (psListRemove(biassecList, PS_LIST_TAIL)); // Removing all entries + + // Blow away the cell data + for (int i = 0; i < filenames->n; i++) { + pmFPA *fpaIn = data->in->data[i]; // Input FPA + pmChip *chipIn = fpaIn->chips->data[view->chip]; // Input chip + pmCell *cellIn = chipIn->cells->data[view->cell]; // Input cell + pmCellFreeData(cellIn); + } + + // Write the pixels + if (cell->hdu && !cell->hdu->blankPHU) { + psTrace("ppMerge", 5, "Writing out cell HDU.\n"); + pmCellWrite(cell, data->outFile, config->database, false); + if (options->fringe) { + pmCellWriteTable(data->outFile, cell, "FRINGE"); + } + + pmCellFreeData(cell); + } + } + + // Blow away the chip data + for (int i = 0; i < filenames->n; i++) { + pmFPA *fpaIn = data->in->data[i]; // Input FPA + pmChip *chipIn = fpaIn->chips->data[view->chip]; // Input chip + pmChipFreeData(chipIn); + } + + // Write the pixels + if (chip->hdu && !chip->hdu->blankPHU) { + psTrace("ppMerge", 5, "Writing out chip HDU.\n"); + pmChipWrite(chip, data->outFile, config->database, false, false); + if (options->fringe) { + pmChipWriteTable(data->outFile, chip, "FRINGE"); + } + + pmChipFreeData(chip); + } + } + + // Blow away the FPA data + for (int i = 0; i < filenames->n; i++) { + pmFPA *fpaIn = data->in->data[i]; // Input FPA + pmFPAFreeData(fpaIn); + } + + if (data->out->hdu && !data->out->hdu->blankPHU) { + // Write the pixels + psTrace("ppMerge", 5, "Writing out FPA HDU.\n"); + pmFPAWrite(data->out, data->outFile, config->database, false, false); + if (options->fringe) { + pmFPAWriteTable(data->outFile, fpa, "FRINGE"); + } + } + + pmFPAFreeData(data->out); + + psFree(view); + psFree(rng); + + return true; + +} Index: /branches/rel-1-0/ppMerge/src/ppMergeCombine.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeCombine.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeCombine.h (revision 11645) @@ -0,0 +1,18 @@ +#ifndef PP_MERGE_COMBINE_H +#define PP_MERGE_COMBINE_H + +#include +#include + +#include "ppMergeData.h" +#include "ppMergeOptions.h" + +// Combine readouts +bool ppMergeCombine(psImage *scales, // Scales for each cell of each integration, or NULL + psImage *zeros, // Zeros for each cell of each integration, or NULL + ppMergeData *data, // Data + ppMergeOptions *options, // Options + pmConfig *config // Configuration + ); + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeConfig.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeConfig.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeConfig.c (revision 11645) @@ -0,0 +1,86 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeConfig.h" + +#define DONT_USE_DB + +// Output usage information +static void usage(const char *programName, // Name of the program + pmConfig *config // Configuration (contains the arguments list) + ) +{ + printf("Merge multiple calibration frames into a master frame by stacking.\n\n" + "Usage:\n" + "\t%s OUTPUT.fits INPUT1.fits INPUT2.fits ...\n" + "\n", programName); + psArgumentHelp(config->arguments); + exit(EXIT_FAILURE); +} + +pmConfig *ppMergeConfig(int argc, char **argv) +{ + pmConfig *config = pmConfigRead(&argc, argv, PPMERGE_RECIPE); + // Load the site-wide configuration information + if (! config) { + psErrorStackPrint(stderr, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + + // Command-line arguments + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-type", 0, "Type of calibration frame", ""); + psMetadataAddBool(config->arguments, PS_LIST_TAIL, "-zero", 0, "Subtract background?", false); + psMetadataAddBool(config->arguments, PS_LIST_TAIL, "-scale", 0, "Scale by background?", false); + psMetadataAddBool(config->arguments, PS_LIST_TAIL, "-exptime", 0, "Scale by the exposure time?", false); + psMetadataAddS32(config->arguments, PS_LIST_TAIL, "-onoff", 0, "Number of on/off pairs", 0); + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "-stats", 0, "MDC file to hold statistics ", NULL); + + if (argc == 1) { + usage(argv[0], config); + } + + // Parse the arguments + if (! psArgumentParse(config->arguments, &argc, argv) || argc < 3) { + usage(argv[0], config); + } + + // Add the output image to the arguments list + psMetadataAddStr(config->arguments, PS_LIST_TAIL, "OUTPUT", 0, "Name of the output image", + argv[1]); + + // Everything remaining must be input files + if (argc - 2 <= 1) { + psErrorStackPrint(stderr, "No files to combine.\n"); + exit(EXIT_FAILURE); + } + psArray *files = psArrayAlloc(argc - 2); + for (int i = 2; i < argc; i++) { + files->data[i - 2] = psStringCopy(argv[i]); + } + psMetadataAddPtr(config->arguments, PS_LIST_TAIL, "INPUT", PS_DATA_ARRAY, + "Array of inputs images", files); + psFree(files); // Drop reference + +#ifndef DONT_USE_DB +#ifdef HAVE_PSDB + // Define database handle, if required + config->database = pmConfigDB(config); +#endif +#endif + + // Add concepts for scale and zero + psMetadataItem *scaleItem = psMetadataItemAllocF32("PPMERGE.SCALE", "Scaling for ppMerge", NAN); + psMetadataItem *zeroItem = psMetadataItemAllocF32("PPMERGE.ZERO", "Zero offset for ppMerge", NAN); + pmConceptRegister(scaleItem, NULL, NULL, PM_FPA_LEVEL_CELL); + pmConceptRegister(zeroItem, NULL, NULL, PM_FPA_LEVEL_CELL); + psFree(scaleItem); + psFree(zeroItem); + + return config; +} Index: /branches/rel-1-0/ppMerge/src/ppMergeConfig.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeConfig.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeConfig.h (revision 11645) @@ -0,0 +1,10 @@ +#ifndef PP_MERGE_CONFIG_H +#define PP_MERGE_CONFIG_H + +#include + +// Get the configuration information +pmConfig *ppMergeConfig(int argc, char **argv // The standard command-line parameters (but pointer to number) + ); + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeData.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeData.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeData.c (revision 11645) @@ -0,0 +1,50 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "ppMerge.h" +#include "ppMergeData.h" + +// Free function for ppMergeData +static void mergeDataFree(ppMergeData *data // Data to free + ) +{ + if (data->files) { + for (long i = 0; i < data->files->n; i++) { + psFitsClose(data->files->data[i]); + data->files->data[i] = NULL; + } + psFree(data->files); + } + psFree(data->in); + psFree(data->out); + if (data->outFile) { + psFitsClose(data->outFile); + data->outFile = NULL; + } + if (data->statsFile) { + fclose(data->statsFile); + data->statsFile = NULL; + } + psFree(data->stats); +} + +// Allocator for ppMergeData +ppMergeData *ppMergeDataAlloc(void) +{ + ppMergeData *data = psAlloc(sizeof(ppMergeData)); // The data, to return + psMemSetDeallocator(data, (psFreeFunc)mergeDataFree); + + data->numCells = 0; + data->files = NULL; + data->in = NULL; + data->out = NULL; + data->outFile = NULL; + data->stats = NULL; + data->statsFile = NULL; + + return data; +} Index: /branches/rel-1-0/ppMerge/src/ppMergeData.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeData.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeData.h (revision 11645) @@ -0,0 +1,23 @@ +#ifndef PP_MERGE_DATA_H +#define PP_MERGE_DATA_H + +#include +#include + +// Container for the data +typedef struct { + int numCells; // Number of (valid) cells in the FPA + psArray *files; // Input file pointers + psArray *in; // Input FPA structures + pmFPA *out; // Output FPA structure + psFits *outFile; // FITS file handle for output + psMetadata *stats; // Statistics on the combined image + FILE *statsFile; // File stream for statistics output +} ppMergeData; + + +// Allocator +ppMergeData *ppMergeDataAlloc(void); + + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.c.in =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.c.in (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.c.in (revision 11645) @@ -0,0 +1,26 @@ +/* + * The line + { PPMERGE_ERR_$X{ErrorCode}, "$X{ErrorDescription}"}, + * (without the Xs) + * will be replaced by values from errorCodes.dat + */ +#include "pslib.h" +#include "ppMergeErrorCodes.h" + +void ppMergeErrorRegister(void) +{ + static psErrorDescription errors[] = { + { PPMERGE_ERR_BASE, "First value we use; lower values belong to psLib" }, + { PPMERGE_ERR_${ErrorCode}, "${ErrorDescription}"}, + }; + static int nerror = PPMERGE_ERR_NERROR - PPMERGE_ERR_BASE; // number of values in enum + + for (int i = 0; i < nerror; i++) { + psErrorDescription *tmp = psAlloc(sizeof(psErrorDescription)); + p_psMemSetPersistent(tmp, true); + *tmp = errors[i]; + psErrorRegister(tmp, 1); + psFree(tmp); /* it's on the internal list */ + } + nerror = 0; // don't register more than once +} Index: /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.dat =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.dat (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.dat (revision 11645) @@ -0,0 +1,10 @@ +# +# This file is used to generate ppMergeErrorClasses.h +# +BASE = 700 First value we use; lower values belong to psLib +# these errors correspond to standard exit conditions +ARGUMENTS Incorrect arguments +SYS System error +CONFIG Problem in configure files +PROG Programming error +DATA invalid data Index: /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.h.in =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.h.in (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeErrorCodes.h.in (revision 11645) @@ -0,0 +1,18 @@ +#if !defined(PPMERGE_ERROR_CODES_H) +#define PPMERGE_ERROR_CODES_H +/* + * The line + * PPMERGE_ERR_$X{ErrorCode}, + * (without the X) + * + * will be replaced by values from errorCodes.dat + */ +typedef enum { + PPMERGE_ERR_BASE = 512, + PPMERGE_ERR_${ErrorCode}, + PPMERGE_ERR_NERROR +} ppMergeErrorCode; + +void ppMergeErrorRegister(void); + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeMask.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeMask.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeMask.c (revision 11645) @@ -0,0 +1,126 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeData.h" +#include "ppMergeMask.h" + + +// Generate a mask +bool ppMergeMask(ppMergeData *data, // Data + ppMergeOptions *options, // Options + pmConfig *config // Configuration + ) +{ + psArray *filenames = psMetadataLookupPtr(NULL, config->arguments, "INPUT"); // The input file names + assert(filenames); // It should be here --- it's put here in ppMergeConfig + + psRandom *rng = psRandomAlloc(PS_RANDOM_TAUS, 0); // Random number generator + pmFPA *fpaOut = data->out; // Output FPA + + // Iterate over each file + for (int i = 0; i < filenames->n; i++) { + if (! filenames->data[i] || strlen(filenames->data[i]) == 0) { + continue; + } + psFits *fits = data->files->data[i]; // FITS file handle + if (!fits) { + continue; + } + psTrace("ppMerge", 3, "File %d: %s\n", i, (const char*)filenames->data[i]); + + pmFPA *fpaIn = data->in->data[i]; // Input FPA + pmFPAview *view = pmFPAviewAlloc(0); // View of FPA, for iteration + pmChip *chipIn; // Input chip of interest + while ((chipIn = pmFPAviewNextChip(view, fpaIn, 1))) { + pmCell *cellIn; // Input cell of interest + while ((cellIn = pmFPAviewNextCell(view, fpaIn, 1))) { + if (!pmCellRead(cellIn, fits, config->database)) { + continue; + } + if (cellIn->readouts->n == 0) { + continue; + } + + pmCell *cellOut = pmFPAviewThisCell(view, fpaOut); // Output cell + // Suspect pixels image + psImage *suspect = psMetadataLookupPtr(NULL, cellOut->analysis, "MASK.SUSPECT"); + bool first = suspect ? false : true; + + pmReadout *roIn; // Input readout of interest + while ((roIn = pmFPAviewNextReadout(view, fpaIn, 1))) { + if (!roIn->image) { + continue; + } + psTrace("ppMerge", 4, "Flagging suspect pixels in chip %d, cell %d, ro %d\n", + view->chip, view->cell, view->readout); + float frac = options->sample / (float)(roIn->image->numCols * roIn->image->numRows); + suspect = pmMaskFlagSuspectPixels(suspect, roIn, options->maskSuspect, + options->combine->maskVal, frac, rng); + } + + if (first) { + psMetadataAddImage(cellOut->analysis, PS_LIST_TAIL, "MASK.SUSPECT", 0, + "Suspect pixels", suspect); + psFree(suspect); + } + + pmCellFreeData(cellIn); + } + pmChipFreeData(chipIn); + } + pmFPAFreeData(fpaIn); + psFree(view); + } + psFree(rng); + + pmFPAview *view = pmFPAviewAlloc(0);// View of FPA, for iteration + pmFPAWriteMask(fpaOut, data->outFile, config->database, true, false); // Write header only + pmChip *chipOut; // Output chip of interest + while ((chipOut = pmFPAviewNextChip(view, fpaOut, 1))) { + pmChipWriteMask(chipOut, data->outFile, config->database, true, false); // Write header only + pmCell *cellOut; // Output cell of interest + while ((cellOut = pmFPAviewNextCell(view, fpaOut, 1))) { + pmCellWriteMask(cellOut, data->outFile, config->database, true); // Write header only + + psImage *suspect = psMetadataLookupPtr(NULL, cellOut->analysis, "MASK.SUSPECT"); + if (! suspect) { + continue; + } + + pmReadout *roOut = pmReadoutAlloc(cellOut); // Output readout + roOut->mask = pmMaskIdentifyBadPixels(suspect, options->maskBad, options->combine->maskVal); + psFree(roOut); // Drop reference + + if (cellOut->hdu && !cellOut->hdu->blankPHU) { + psTrace("ppMerge", 5, "Writing out cell HDU.\n"); + pmCellWriteMask(cellOut, data->outFile, config->database, false); + pmCellFreeData(cellOut); + } + } + + if (chipOut->hdu && !chipOut->hdu->blankPHU) { + psTrace("ppMerge", 5, "Writing out chip HDU.\n"); + pmChipWriteMask(chipOut, data->outFile, config->database, false, false); + pmChipFreeData(chipOut); + } + } + + if (fpaOut->hdu && !fpaOut->hdu->blankPHU) { + psTrace("ppMerge", 5, "Writing out FPA HDU.\n"); + pmFPAWriteMask(fpaOut, data->outFile, config->database, false, false); + } + pmFPAFreeData(fpaOut); + + psFree(view); + + return true; +} Index: /branches/rel-1-0/ppMerge/src/ppMergeMask.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeMask.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeMask.h (revision 11645) @@ -0,0 +1,15 @@ +#ifndef PP_MERGE_MASK +#define PP_MERGE_MASK + +#include +#include "ppMergeData.h" +#include "ppMergeOptions.h" + +// Generate a mask +bool ppMergeMask(ppMergeData *data, // Data + ppMergeOptions *options, // Options + pmConfig *config // Configuration + ); + + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeOptions.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeOptions.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeOptions.c (revision 11645) @@ -0,0 +1,249 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeOptions.h" + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// ppMergeOptions +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + +// Free function +static void mergeOptionsFree(ppMergeOptions *options // Options to free + ) +{ + psFree(options->format); + psFree(options->combine); +} + +// Allocator +ppMergeOptions *ppMergeOptionsAlloc(void) +{ + ppMergeOptions *options = psAlloc(sizeof(ppMergeOptions)); // The options, to return + psMemSetDeallocator(options, (psFreeFunc)mergeOptionsFree); + + options->format = NULL; + options->rows = 0; + options->minElectrons = NAN; + options->zero = false; + options->scale = false; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + options->sample = 1; + options->mean = PS_STAT_SAMPLE_MEDIAN; + options->stdev = PS_STAT_SAMPLE_STDEV; + options->fringeNum = 100; + options->fringeSize = 10; + options->fringeSmoothX = 5; + options->fringeSmoothY = 5; + options->shutterSize = 10; + options->shutterIter = 2; + options->shutterRej = 3.0; + options->maskSuspect = 5.0; + options->maskBad = 10.0; + options->onOff = 0; + options->combine = pmCombineParamsAlloc(PS_STAT_SAMPLE_MEAN); + options->combine->rej = 3.0; + options->combine->iter = 1; + options->combine->fracHigh = 0.0; + options->combine->fracLow = 0.0; + options->combine->nKeep = 1; + options->combine->maskVal = 0xff; + options->combine->weights = false; + + return options; +} + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// ppMergeOptionsParse +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + +// Parse a recipe option according to its type +#define OPTION_PARSE(OPTION,MD,NAME,TYPE) \ +{ \ + psMetadataItem *item = psMetadataLookup(MD, NAME); \ + if (item) { \ + OPTION = psMetadataItemParse##TYPE(item); \ + } else { \ + psWarning("Recipe option %s isn't specified; using default.\n", NAME); \ + } \ +} + +// Parse a statistic +static psStatsOptions parseStat(psMetadata *source, // Source of the statistics option + const char *name // Name of the statistics option + ) +{ + bool mdok = true; // Status of MD lookup + const char *stat = psMetadataLookupStr(&mdok, source, name); // The statistic string + if (!mdok || !stat || strlen(stat) == 0) { + return 0; + } + return psStatsOptionFromString(stat); +} + +// Parse the options +ppMergeOptions *ppMergeOptionsParse(pmConfig *config // Configuration + ) +{ + ppMergeOptions *options = ppMergeOptionsAlloc(); // The merge options + + // We need to work out the camera before we can get the recipe. Take the first input and inspect it. + if (!config->camera) { + psArray *filenames = psMetadataLookupPtr(NULL, config->arguments, "INPUT"); // The input file names + psFits *inFile = psFitsOpen(filenames->data[0], "r"); // The FITS file to read + if (!inFile) { + psError(PS_ERR_IO, false, "Unable to open input file %s to determine camera.\n", + (char *)filenames->data[0]); + exit(EXIT_FAILURE); + } + psMetadata *header = psFitsReadHeader(NULL, inFile); // The FITS (primary) header + psFitsClose(inFile); + + options->format = pmConfigCameraFormatFromHeader(config, header); + psFree(header); + if (!options->format) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to identify camera format for input file %s\n", + (char *)filenames->data[0]); + exit(EXIT_FAILURE); + } + } + + // we must have the recipes by this point + assert (config->recipes); + + // Now we can read the recipe + bool mdok = true; // Status of MD lookup + psMetadata *recipe = psMetadataLookupMetadata(&mdok, config->recipes, PPMERGE_RECIPE); // Recipe information + if (!mdok || !recipe) { + psError(PS_ERR_IO, true, "Unable to find recipe %s", PPMERGE_RECIPE); + exit(EXIT_FAILURE); + } + + // First, deal with the recipe. These are parameters that will typically be constant for a camera. + OPTION_PARSE(options->rows, recipe, "ROWS", U16); + OPTION_PARSE(options->minElectrons, recipe, "ELECTRONS", F32); + OPTION_PARSE(options->sample, recipe, "SAMPLE", S32); + OPTION_PARSE(options->combine->rej, recipe, "REJ", F32); + OPTION_PARSE(options->combine->iter, recipe, "ITER", S32); + OPTION_PARSE(options->combine->fracHigh, recipe, "FRACHIGH", F32); + OPTION_PARSE(options->combine->fracLow, recipe, "FRACLOW", F32); + OPTION_PARSE(options->combine->nKeep, recipe, "NKEEP", S32); + OPTION_PARSE(options->combine->maskVal, recipe, "MASKVAL", S32); + OPTION_PARSE(options->combine->weights, recipe, "WEIGHTS", Bool); + OPTION_PARSE(options->fringeNum, recipe, "FRINGE.NUM", S32); + OPTION_PARSE(options->fringeSize, recipe, "FRINGE.SIZE", S32); + OPTION_PARSE(options->fringeSmoothX, recipe, "FRINGE.XSMOOTH", S32); + OPTION_PARSE(options->fringeSmoothY, recipe, "FRINGE.YSMOOTH", S32); + OPTION_PARSE(options->shutterSize, recipe, "SHUTTER.SIZE", S32); + OPTION_PARSE(options->shutterIter, recipe, "SHUTTER.ITER", S32); + OPTION_PARSE(options->shutterRej, recipe, "SHUTTER.REJECT", F32); + OPTION_PARSE(options->maskSuspect, recipe, "MASK.SUSPECT", F32); + OPTION_PARSE(options->maskBad, recipe, "MASK.BAD", F32); + + options->combine->combine = parseStat(recipe, "COMBINE"); + options->mean = parseStat(recipe, "MEAN"); + options->stdev = parseStat(recipe, "STDEV"); + + // Now the command-line options. These are parameters that depend on what type of frame is being combined + + // Set options based on the type of calibration frame + const char *type = psMetadataLookupStr(NULL, config->arguments, "-type"); // The type of calibration frame + if (strlen(type) > 0) { + if (strcasecmp(type, "BIAS") == 0) { + options->zero = false; + options->scale = false; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + } else if (strcasecmp(type, "DARK") == 0) { + options->zero = false; + options->scale = false; + options->darktime = true; + options->fringe = false; + options->shutter = false; + options->mask = false; + } else if (strcasecmp(type, "FLAT") == 0) { + options->zero = false; + options->scale = true; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + } else if (strcasecmp(type, "SKYFLAT") == 0) { + options->zero = false; + options->scale = true; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + } else if (strcasecmp(type, "DOMEFLAT") == 0) { + options->zero = false; + options->scale = true; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + } else if (strcasecmp(type, "FRINGE") == 0) { + options->zero = true; + options->scale = true; + options->darktime = false; + options->fringe = true; + options->shutter = false; + options->mask = false; + } else if (strcasecmp(type, "SHUTTER") == 0) { + options->zero = false; + options->scale = false; + options->darktime = false; + options->fringe = false; + options->shutter = true; + options->mask = false; + options->rows = 0; // Read the whole image at once + } else if (strcasecmp(type, "MASK") == 0) { + options->zero = false; + options->scale = false; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = true; + } else { + psLogMsg(__func__, PS_LOG_WARN, "Unrecognised image type: %s --- assuming BIAS.\n", type); + options->zero = false; + options->scale = false; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + } + } else { + psLogMsg(__func__, PS_LOG_WARN, "No calibration type specified; assuming BIAS.\n"); + options->zero = false; + options->scale = false; + options->darktime = false; + options->fringe = false; + options->shutter = false; + options->mask = false; + } + + +#if 0 + // Or you can set them individually + OPTION_PARSE(options->zero, config->arguments, "-zero", Bool); + OPTION_PARSE(options->scale, config->arguments, "-scale", Bool); + OPTION_PARSE(options->darktime, config->arguments, "-exptime", Bool); +#endif + + // Number of on/off images + OPTION_PARSE(options->onOff, config->arguments, "-onoff", S32); + + return options; +} Index: /branches/rel-1-0/ppMerge/src/ppMergeOptions.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeOptions.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeOptions.h (revision 11645) @@ -0,0 +1,52 @@ +#ifndef PP_MERGE_OPTIONS_H +#define PP_MERGE_OPTIONS_H + +#include +#include + +// Mode of on/off pairs; the value corresponds to how many images are in each set +typedef enum { + PP_ONOFF_ABBA = -1, // On/off pairs in the ABBA mode + PP_ONOFF_NONE = 0, // No on/off pairs + PP_ONOFF_ABAB = 1, // On/off pairs in the ABAB mode (one image each) + PP_ONOFF_AABB = 2, // On/off pairs, two images each + // And so on and so forth... just use a number beyond this +} ppOnOff; + +// Options for ppMerge +typedef struct { + psMetadata *format; // Camera format configuration + unsigned int rows; // Number of rows to read at once + float minElectrons; // Minimum number of electrons for useful signal + bool zero; // Subtract background before combining? + bool scale; // Scale by the background before combining? + bool darktime; // Normalise by the dark time (time since flush)? + bool fringe; // Make fringe measurements? + bool shutter; // Generate shutter correction? + bool mask; // Generate bad pixel mask? + unsigned int sample; // Sampling factor for measuring the background + psStatsOptions mean; // Statistic to use to measure the mean + psStatsOptions stdev; // Statistic to use to measure the stdev + int fringeNum; // Number of fringe regions per cell + int fringeSize; // Size of fringe regions + int fringeSmoothX; // Number of smoothing regions per cell, in x + int fringeSmoothY; // Number of smoothing regions per cell, in y + int shutterSize; // Size for shutter measurement regions + int shutterIter; // Number of iterations for shutter measurement + float shutterRej; // Rejection limit for shutter measurement + float maskSuspect; // Threshold for identifying suspect pixels + float maskBad; // Threshold for identifying bad pixels + ppOnOff onOff; // On/off pairs? + pmCombineParams *combine; // Combination parameters +} ppMergeOptions; + +// Allocator +ppMergeOptions *ppMergeOptionsAlloc(void); + + +// Parse the options for ppMerge +ppMergeOptions *ppMergeOptionsParse(pmConfig *config // Configuration + ); + + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeScaleZero.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeScaleZero.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeScaleZero.c (revision 11645) @@ -0,0 +1,325 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "ppMerge.h" +#include "ppMergeScaleZero.h" + +// Get the scale and zero for each chip of each input +bool ppMergeScaleZero(psImage **scales, // The scales for each integration/cell + psImage **zeros, // The zeroes for each integration/cell + ppMergeData *data,// The data + const ppMergeOptions *options, // The options + const pmConfig *config // The configuration + ) +{ + assert(data); + assert(options); + assert(config); + + if (!options->scale && !options->zero && !options->darktime) { + return true; // We did everything we were asked for + } + + assert(config->camera); // Need the camera configuration + assert(config->arguments); // Need the list of files + + psArray *filenames = psMetadataLookupPtr(NULL, config->arguments, "INPUT"); // The input file names + assert(filenames); // It should be here --- it's put here in ppMergeConfig + + // Sanity checks + assert(!options->scale || scales); + assert(!scales || !*scales || ((*scales)->type.type == PS_TYPE_F32 && + (*scales)->numCols == data->numCells && + (*scales)->numRows == filenames->n)); + assert(!options->zero || zeros); + assert(!zeros || !*zeros || ((*zeros)->type.type == PS_TYPE_F32 && + (*zeros)->numCols == data->numCells && + (*zeros)->numRows == filenames->n)); + + // Allocate the outputs + if (options->scale) { + if (*scales) { + psFree(*scales); + } + *scales = psImageAlloc(data->numCells, filenames->n, PS_TYPE_F32); + } + if (options->zero) { + if (*zeros) { + psFree(*zeros); + } + *zeros = psImageAlloc(data->numCells, filenames->n, PS_TYPE_F32); + } + + bool fromConcepts = false; // Do we get the scale and zero points from the concepts + bool first = true; // Are we on the first cell (that sets the standard for the rest)? + bool done = false; // Are we done going through the list? + bool mdok = true; // Status of MD lookup + for (long i = 0; i < data->in->n && !done; i++) { + pmFPA *fpa = data->in->data[i]; // The FPA + if (!fpa) { + continue; + } + long cellNum = -1; // Number of the cell + psArray *chips = fpa->chips; // The array of chips + for (long j = 0; j < chips->n && !done; j++) { + pmChip *chip = chips->data[j]; // The chip + if (!chip) { + continue; + } + psArray *cells = chip->cells; // The array of cells + for (long k = 0; k < cells->n && !done; k++) { + pmCell *cell = cells->data[k]; // The cell + if (!cell) { + continue; + } + cellNum++; + + if (options->scale) { + float scale = psMetadataLookupF32(&mdok, cell->concepts, "PPMERGE.SCALE"); // The scale + if (mdok && !isnan(scale)) { + if (!first && !fromConcepts) { + psLogMsg(__func__, PS_LOG_WARN, "PPMERGE.SCALE and PPMERGE.ZERO have been set " + "for some, but not all cells --- we will re-measure it for all cells."); + done = true; + continue; + } + fromConcepts = true; + (*scales)->data.F32[i][cellNum] = scale; + psTrace("ppMerge", 9, "Scale for input %ld, chip %ld, cell %ld: %f\n", i, j, k, scale); + } else if (!first && fromConcepts) { + psLogMsg(__func__, PS_LOG_WARN, "PPMERGE.SCALE and PPMERGE.ZERO have been set " + "for some, but not all cells --- we will re-measure it for all cells."); + fromConcepts = false; + done = true; + continue; + } + } + + if (options->zero) { + float zero = psMetadataLookupF32(&mdok, cell->concepts, "PPMERGE.ZERO"); // The zero + if (mdok && !isnan(zero)) { + if (!first && !fromConcepts) { + psLogMsg(__func__, PS_LOG_WARN, "PPMERGE.SCALE and PPMERGE.ZERO have been set " + "for some, but not all cells --- we will re-measure it for all cells."); + done = true; + continue; + } + fromConcepts = true; + (*zeros)->data.F32[i][cellNum] = zero; + psTrace("ppMerge", 9, "Zero for input %ld, chip %ld, cell %ld: %f\n", i, j, k, zero); + } else if (!first && fromConcepts) { + psLogMsg(__func__, PS_LOG_WARN, "PPMERGE.SCALE and PPMERGE.ZERO have been set " + "for some, but not all cells --- we will re-measure it for all cells."); + fromConcepts = false; + done = true; + continue; + } + } + + first = false; + } + } + } + + if (fromConcepts) { + // We've already done everything we need to + return true; + } + + psImage *background = psImageAlloc(data->numCells, filenames->n, PS_TYPE_F32); // Background measurements + psImageInit(background, NAN); + psStats *bgStats = psStatsAlloc(options->mean); // Statistic to measure the background + psVector *gains = NULL; // The gains for each cell + psImage *darktime = NULL; // The dark time for each integration of each cell + if (options->scale) { + gains = psVectorAlloc(data->numCells, PS_TYPE_F32); + } + if (options->darktime) { + darktime = psImageAlloc(data->numCells, filenames->n, PS_TYPE_F32); + } + + bool status = true; // Status of getting the scale and zero --- did everything go right? + for (int i = 0; i < filenames->n; i++) { + psString name = filenames->data[i]; // The name of the file + if (!name || strlen(name) == 0) { + continue; + } + psTrace("ppMerge", 9, "Opening %s to get background...\n", name); + psFits *inFile = data->files->data[i]; // The FITS file to read + pmFPA *fpa = data->in->data[i]; // The FPA for this input + int cellNum = -1; // Number of the cell + psArray *chips = fpa->chips; // Array of chips + for (int j = 0; j < chips->n; j++) { + pmChip *chip = chips->data[j]; // The chip of interest + if (!chip) { + continue; + } + psArray *cells = chip->cells; // Array of cells + for (int k = 0; k < cells->n; k++) { + pmCell *cell = cells->data[k]; // The cell of interest + if (!cell) { + continue; + } + cellNum++; + + if (!pmCellReadHeader(cell, inFile)) { + continue; + } + + // Normalising by the exposure time + if (options->darktime) { + darktime->data.F32[i][cellNum] = psMetadataLookupF32(&mdok, cell->concepts, + "CELL.DARKTIME"); + if (!mdok || isnan(darktime->data.F32[i][cellNum])) { + psLogMsg(__func__, PS_LOG_WARN, "CELL.DARKTIME for file %s chip %d cell %d is not " + "set.\n", name, j, k); + darktime->data.F32[i][cellNum] = NAN; + status = false; + } + } + + // Scaling by the background + if (options->scale || options->zero) { + if (!pmCellRead(cell, inFile, config->database)) { + // Nothing here + pmCellFreeData(cell); + continue; + } + + if (cell->readouts->n > 1) { + psLogMsg(__func__, PS_LOG_WARN, "File %s chip %d cell %d contains more than one " + "readout --- ignoring all but the first.\n", name, j, k); + status = false; + } + + pmReadout *readout = cell->readouts->data[0]; // The readout of interest + psImage *image = readout->image; // The pixels of interest + if (!image) { + pmCellFreeData(cell); + continue; + } + + // Get the gain + if (options->scale) { + bool mdok = true; // Status of MD lookup + gains->data.F32[cellNum] = psMetadataLookupF32(&mdok, cell->concepts, "CELL.GAIN"); + if (!mdok || isnan(gains->data.F32[cellNum])) { + psLogMsg(__func__, PS_LOG_WARN, "CELL.GAIN for file %s chip %d cell %d is not " + "set.\n", name, j, k); + gains->data.F32[cellNum] = NAN; + status = false; + } + } + + // Get the background + int sampleSize = (image->numCols * image->numRows) / options->sample; // Size of sample + psVector *sample = psVectorAlloc(sampleSize, PS_TYPE_F32); // Sample of the image + psVector *sampleMask = NULL; // Mask for sample + if (readout->mask) { + sampleMask = psVectorAlloc(sampleSize, PS_TYPE_U8); + } + psImage *mask = readout->mask; // The mask image + for (long i = 0; i < sampleSize; i++) { + int j = i * options->sample; // Index into image + int x = j % image->numCols; // x index + int y = j / image->numCols; // y index + sample->data.F32[i] = image->data.F32[y][x]; + if (readout->mask) { + sampleMask->data.U8[i] = mask->data.U8[y][x]; + } + } + status = psVectorStats(bgStats, sample, sampleMask, NULL, options->combine->maskVal); + if (!status) { + psTrace("ppMerge", 3, "failed to get stats for for %s, cell %d is %f\n", name, cellNum, + background->data.F32[i][cellNum]); + psErrorClear(); + } + psFree(sample); + psFree(sampleMask); + background->data.F32[i][cellNum] = psStatsGetValue(bgStats, options->mean); + psTrace("ppMerge", 3, "Background for %s, cell %d is %f\n", name, cellNum, + background->data.F32[i][cellNum]); + } + + pmCellFreeData(cell); + } + pmChipFreeData(chip); + } + pmFPAFreeData(fpa); + } + psFree(bgStats); + + if (options->scale) { + // Need to normalize over the focal plane + if (psTraceGetLevel("ppMerge") > 9) { + for (int i = 0; i < gains->n; i++) { + psTrace("ppMerge", 10, "Gain for cell %d is %f\n", i, gains->data.F32[i]); + } + } + psVector *fluxes = NULL; // Solution to fluxes + if (!pmFlatNormalize(&fluxes, &gains, background)) { + psLogMsg(__func__, PS_LOG_WARN, "Normalisation failed to converge --- continuing anyway.\n"); + status = false; + } + psFree(gains); + + psImage *scalesDeref = *scales; // Dereference the pointer + + for (int i = 0; i < scalesDeref->numRows; i++) { + psF32 bg = fluxes->data.F32[i]; + for (int j = 0; j < scalesDeref->numCols; j++) { + scalesDeref->data.F32[i][j] = bg; + } + } + } + + if (options->darktime) { + if (!options->scale) { + // Copy over the exposure times + *scales = psImageCopy(*scales, darktime, PS_TYPE_F32); + } else { + *scales = (psImage*)psBinaryOp(*scales, *scales, "*", darktime); + } + } + + if (options->zero) { + if (!*zeros) { + // This is much faster than copying! + *zeros = psMemIncrRefCounter(background); + } else { + *zeros = psImageCopy(*zeros, background, PS_TYPE_F32); + } + } + + // Diagnostic stuff + if (scales && *scales && psTraceGetLevel("ppMerge") > 9) { + psImage *scalesDeref = *scales; // Dereference the pointer + for (int i = 0; i < scalesDeref->numRows; i++) { + for (int j = 0; j < scalesDeref->numCols; j++) { + psTrace("ppMerge", 9, "Scale for exposure %d, cell %d is: %f\n", i, j, + scalesDeref->data.F32[i][j]); + } + } + } + if (zeros && *zeros && psTraceGetLevel("ppMerge") > 9) { + psImage *zerosDeref = *zeros; // Dereference the pointer + for (int i = 0; i < zerosDeref->numRows; i++) { + for (int j = 0; j < zerosDeref->numCols; j++) { + psTrace("ppMerge", 9, "Zero for exposure %d, cell %d is: %f\n", i, j, + zerosDeref->data.F32[i][j]); + } + } + } + + psFree(background); + + return status; +} + Index: /branches/rel-1-0/ppMerge/src/ppMergeScaleZero.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeScaleZero.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeScaleZero.h (revision 11645) @@ -0,0 +1,18 @@ +#ifndef PP_MERGE_SCALE_ZERO_H +#define PP_MERGE_SCALE_ZERO_H + +#include +#include + +#include "ppMergeData.h" +#include "ppMergeOptions.h" + +// Get the scale and zero for each chip of each input +bool ppMergeScaleZero(psImage **scales, // The scales for each integration/cell + psImage **zeros, // The zeroes for each integration/cell + ppMergeData *data,// The data + const ppMergeOptions *options, // The options + const pmConfig *config // The configuration + ); + +#endif Index: /branches/rel-1-0/ppMerge/src/ppMergeVersion.c =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeVersion.c (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeVersion.c (revision 11645) @@ -0,0 +1,60 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "ppMergeVersion.h" + +static const char *cvsTag = "$Name: not supported by cvs2svn $";// CVS tag name + +psString ppMergeVersion(void) +{ + psString version = NULL; // Version, to return + psStringAppend(&version, "%s-%s",PACKAGE_NAME,PACKAGE_VERSION); + return version; +} + +psString ppMergeVersionLong(void) +{ + psString version = ppMergeVersion(); // Version, to return + psString tag = psStringStripCVS(cvsTag, "Name"); // CVS tag + psStringAppend(&version, " (cvs tag %s) %s, %s", tag, __DATE__, __TIME__); + psFree(tag); + return version; +} + + +void ppMergeVersionMetadata(psMetadata *metadata) +{ + PS_ASSERT_METADATA_NON_NULL(metadata,); + + psString pslib = psLibVersionLong();// psLib version + psString psmodules = psModulesVersionLong(); // psModules version + psString ppStats = ppStatsVersionLong(); // ppStats version + psString ppMerge = ppMergeVersionLong(); // ppMerge version + + psTime *time = psTimeGetNow(PS_TIME_TAI); // The time now + psString timeString = psTimeToISO(time); // The time in an ISO string + psFree(time); + psString head = NULL; // Head string + psStringAppend(&head, "ppMerge processing at %s. Component information:", timeString); + psFree(timeString); + + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, head, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, pslib, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, psmodules, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, ppStats, ""); + psMetadataAddStr(metadata, PS_LIST_TAIL, "HISTORY", PS_META_DUPLICATE_OK, ppMerge, ""); + + psFree(head); + psFree(pslib); + psFree(psmodules); + psFree(ppStats); + psFree(ppMerge); + + return; +} Index: /branches/rel-1-0/ppMerge/src/ppMergeVersion.h =================================================================== --- /branches/rel-1-0/ppMerge/src/ppMergeVersion.h (revision 11645) +++ /branches/rel-1-0/ppMerge/src/ppMergeVersion.h (revision 11645) @@ -0,0 +1,14 @@ +#ifndef PP_MERGE_VERSION_H +#define PP_MERGE_VERSION_H + +/// Return short version information +psString ppMergeVersion(void); + +/// Return long version information +psString ppMergeVersionLong(void); + +/// Update the metadata with version information for all dependencies +void ppMergeVersionMetadata(psMetadata *metadata ///< Metadata to update with version information + ); + +#endif Index: /branches/rel-1-0/ppNorm/.cvsignore =================================================================== --- /branches/rel-1-0/ppNorm/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppNorm/.cvsignore (revision 11645) @@ -0,0 +1,12 @@ +Makefile +Makefile.in +aclocal.m4 +autom4te.cache +compile +config.log +config.status +configure +depcomp +install-sh +missing +test Index: /branches/rel-1-0/ppNorm/Makefile.am =================================================================== --- /branches/rel-1-0/ppNorm/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppNorm/Makefile.am (revision 11645) @@ -0,0 +1,3 @@ +SUBDIRS = src + +CLEANFILES = *~ core core.* Index: /branches/rel-1-0/ppNorm/autogen.sh =================================================================== --- /branches/rel-1-0/ppNorm/autogen.sh (revision 11645) +++ /branches/rel-1-0/ppNorm/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=ppNorm +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOlIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOHEADER installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +#$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ppNorm/configure.ac =================================================================== --- /branches/rel-1-0/ppNorm/configure.ac (revision 11645) +++ /branches/rel-1-0/ppNorm/configure.ac (revision 11645) @@ -0,0 +1,29 @@ +AC_PREREQ(2.59) + +AC_INIT([ppNorm], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([src]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([src/config.h]) +AM_MAINTAINER_MODE + +IPP_STDCFLAGS + +AC_LANG(C) +AC_GNU_SOURCE +AC_PROG_CC +AC_PROG_INSTALL +dnl AC_PROG_LIBTOOL +AC_SYS_LARGEFILE + +PKG_CHECK_MODULES([PSLIB], [pslib >= 1.0.0]) +PKG_CHECK_MODULES([PSMODULE], [psmodules >= 1.0.0]) + +IPP_STDOPTS +CFLAGS="${CFLAGS=} -Wall -Werror -std=c99" + +AC_CONFIG_FILES([ + Makefile + src/Makefile +]) +AC_OUTPUT Index: /branches/rel-1-0/ppNorm/src/.cvsignore =================================================================== --- /branches/rel-1-0/ppNorm/src/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppNorm/src/.cvsignore (revision 11645) @@ -0,0 +1,7 @@ +Makefile +Makefile.in +ppNormCalc +.deps +config.h +config.h.in +stamp-h1 Index: /branches/rel-1-0/ppNorm/src/Makefile.am =================================================================== --- /branches/rel-1-0/ppNorm/src/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppNorm/src/Makefile.am (revision 11645) @@ -0,0 +1,17 @@ +bin_PROGRAMS = ppNormCalc + +ppNormCalc_CFLAGS = $(PSMODULE_CFLAGS) $(PSLIB_CFLAGS) +ppNormCalc_LDFLAGS = $(PSMODULE_LIBS) $(PSLIB_LIBS) -Wl,-Bdynamic + +ppNormCalc_SOURCES = \ + ppNormCalc.c + +noinst_HEADERS = + +CLEANFILES = *~ + +clean-local: + -rm -f TAGS +# Tags for emacs +tags: + etags `find . -name \*.[ch] -print` Index: /branches/rel-1-0/ppNorm/src/ppNormCalc.c =================================================================== --- /branches/rel-1-0/ppNorm/src/ppNormCalc.c (revision 11645) +++ /branches/rel-1-0/ppNorm/src/ppNormCalc.c (revision 11645) @@ -0,0 +1,149 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +void helpAndDie(const char *programName) +{ + printf("Calculate normalisation for flat fields.\n\n" + "Usage: %s [IN.mdc [OUT.mdc]]\n\n" + "where IN.mdc is a metadata config file containing the background\n" + " value for each component of each exposure;\n" + "and OUT.mdc is a metadata config file containing the output\n" + " normalisation factors.\n" + "\n", programName); + exit(EXIT_FAILURE); +} + + +int main(int argc, char *argv[]) +{ + psLibInit(NULL); + pmConfig *config = pmConfigRead(&argc, argv, NULL); + psFree(config); + + if (argc > 3 || + psArgumentGet(argc, argv, "-h") || + psArgumentGet(argc, argv, "-help") || + psArgumentGet(argc, argv, "--help") || + psArgumentGet(argc, argv, "-?")) { + helpAndDie(argv[0]); + } + + FILE *inFile = stdin; // Input file stream + FILE *outFile = stdout; // Output file stream + + if (argc >= 2) { + inFile = fopen(argv[1], "r"); + if (!inFile) { + psError(PS_ERR_IO, true, "Unable to open input file: %s\n\n", argv[1]); + helpAndDie(argv[0]); + } + } + if (argc == 3) { + outFile = fopen(argv[2], "w"); + if (!outFile) { + psError(PS_ERR_IO, true, "Unable to open output file: %s\n\n", argv[2]); + helpAndDie(argv[0]); + } + } + + psString inputMDC = psSlurpFile(inFile); // Input metadata config stuff + if (argc >= 2) { + fclose(inFile); + } + + psU32 badLines = 0; // Number of bad lines + psMetadata *exposures = psMetadataConfigParse(NULL, &badLines, inputMDC, false); // Exposure statistics + if (badLines > 0) { + psWarning("%d bad lines found when reading input\n", badLines); + } + + // Get a list of all the components + psMetadata *components = psMetadataAlloc(); // Components of the exposures + psMetadataIterator *expIter = psMetadataIteratorAlloc(exposures, PS_LIST_HEAD, NULL); // Iterator + psMetadataItem *expItem; // Item from iteration + while ((expItem = psMetadataGetAndIncrement(expIter))) { + if (expItem->type != PS_DATA_METADATA) { + psLogMsg("ppNormCalc", PS_LOG_WARN, + "Metadata item %s is not of type METADATA --- ignored.\n", + expItem->name); + continue; + } + + // Inspect each component for this exposure; add it if we don't know about it + psMetadata *comps = expItem->data.V; // The components + psMetadataIterator *compsIter = psMetadataIteratorAlloc(comps, PS_LIST_HEAD, NULL); // Iterator + psMetadataItem *compsItem; // Item from iteration + while ((compsItem = psMetadataGetAndIncrement(compsIter))) { + if (compsItem->type != PS_TYPE_F32) { + psLogMsg("ppNormCalc", PS_LOG_WARN, + "Component %s within exposure %s is not of type F32 --- ignored.\n", + compsItem->name, expItem->name); + continue; + } + if (!psMetadataLookup(components, compsItem->name)) { + psMetadataAddBool(components, PS_LIST_TAIL, compsItem->name, 0, NULL, true); + } + } + psFree(compsIter); + } + + // Convert to a matrix + int numComps = psListLength(components->list); // Number of components + int numExps = psListLength(exposures->list); // Number of exposures + psImage *matrix = psImageAlloc(numComps, numExps, PS_TYPE_F32); // Matrix of backgrounds + psMetadataIteratorSet(expIter, PS_LIST_HEAD); + for (int expNum = 0; (expItem = psMetadataGetAndIncrement(expIter)); expNum++) { + if (expItem->type != PS_DATA_METADATA) { + continue; + } + psMetadata *comps = expItem->data.V; // The components + psMetadataIterator *compsIter = psMetadataIteratorAlloc(components, PS_LIST_HEAD, NULL); // Iterator + psMetadataItem *compsItem; // Item from iteration + for (int compNum = 0; (compsItem = psMetadataGetAndIncrement(compsIter)); compNum++) { + matrix->data.F32[expNum][compNum] = psMetadataLookupF32(NULL, comps, compsItem->name); + } + psFree(compsIter); + } + psFree(expIter); + + // Do the normalisation + psVector *gains = psVectorAlloc(numComps, PS_TYPE_F32); // Vector of gains for each component + psVectorInit(gains, 100.0); + if (!pmFlatNormalize(NULL, &gains, matrix)) { + psLogMsg("ppNormCalc", PS_LOG_ERROR, "Normalisation didn't converge.\n"); + exit(EXIT_FAILURE); + } + psFree(matrix); + + + // Output a MDC format to stdout + psMetadata *outputMD = psMetadataAlloc(); // Output metadata + psMetadataIterator *compsIter = psMetadataIteratorAlloc(components, PS_LIST_HEAD, NULL); // Iterator + psMetadataItem *compsItem; // Item from iteration + for (int compNum = 0; (compsItem = psMetadataGetAndIncrement(compsIter)); compNum++) { + psMetadataAddF32(outputMD, PS_LIST_TAIL, compsItem->name, 0, NULL, gains->data.F32[compNum]); + } + psFree(compsIter); + psString outputString = psMetadataConfigFormat(outputMD); + fprintf(outFile, "%s", outputString); + psFree(outputString); + psFree(outputMD); + if (argc == 3) { + fclose(outFile); + } + + // Clean up + psFree(gains); + psFree(components); + psFree(exposures); + + pmConfigDone(); + psLibFinalize(); + + return EXIT_SUCCESS; +} Index: /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.c.in =================================================================== --- /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.c.in (revision 11645) +++ /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.c.in (revision 11645) @@ -0,0 +1,26 @@ +/* + * The line + { PPNORM_ERR_$X{ErrorCode}, "$X{ErrorDescription}"}, + * (without the Xs) + * will be replaced by values from errorCodes.dat + */ +#include "pslib.h" +#include "ppNormErrorCodes.h" + +void ppNormErrorRegister(void) +{ + static psErrorDescription errors[] = { + { PPNORM_ERR_BASE, "First value we use; lower values belong to psLib" }, + { PPNORM_ERR_${ErrorCode}, "${ErrorDescription}"}, + }; + static int nerror = PPNORM_ERR_NERROR - PPNORM_ERR_BASE; // number of values in enum + + for (int i = 0; i < nerror; i++) { + psErrorDescription *tmp = psAlloc(sizeof(psErrorDescription)); + p_psMemSetPersistent(tmp, true); + *tmp = errors[i]; + psErrorRegister(tmp, 1); + psFree(tmp); /* it's on the internal list */ + } + nerror = 0; // don't register more than once +} Index: /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.dat =================================================================== --- /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.dat (revision 11645) +++ /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.dat (revision 11645) @@ -0,0 +1,10 @@ +# +# This file is used to generate ppNormErrorClasses.h +# +BASE = 600 First value we use; lower values belong to psLib +# these errors correspond to standard exit conditions +ARGUMENTS Incorrect arguments +SYS System error +CONFIG Problem in configure files +PROG Programming error +DATA invalid data Index: /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.h.in =================================================================== --- /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.h.in (revision 11645) +++ /branches/rel-1-0/ppNorm/src/ppNormErrorCodes.h.in (revision 11645) @@ -0,0 +1,18 @@ +#if !defined(PPNORM_ERROR_CODES_H) +#define PPNORM_ERROR_CODES_H +/* + * The line + * PPNORM_ERR_$X{ErrorCode}, + * (without the X) + * + * will be replaced by values from errorCodes.dat + */ +typedef enum { + PPNORM_ERR_BASE = 512, + PPNORM_ERR_${ErrorCode}, + PPNORM_ERR_NERROR +} ppNormErrorCode; + +void ppNormErrorRegister(void); + +#endif Index: /branches/rel-1-0/ppStats/.cvsignore =================================================================== --- /branches/rel-1-0/ppStats/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppStats/.cvsignore (revision 11645) @@ -0,0 +1,16 @@ +Makefile +Makefile.in +aclocal.m4 +autom4te.cache +compile +config.log +config.status +configure +depcomp +install-sh +missing +config.guess +config.sub +libtool +ltmain.sh +ppStats.pc Index: /branches/rel-1-0/ppStats/Makefile.am =================================================================== --- /branches/rel-1-0/ppStats/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppStats/Makefile.am (revision 11645) @@ -0,0 +1,9 @@ +SUBDIRS = src + +CLEANFILES = *~ core core.* + +pkgconfigdir = $(libdir)/pkgconfig +pkgconfig_DATA= ppStats.pc + +EXTRA_DIST = \ + ppStats.pc.in Index: /branches/rel-1-0/ppStats/autogen.sh =================================================================== --- /branches/rel-1-0/ppStats/autogen.sh (revision 11645) +++ /branches/rel-1-0/ppStats/autogen.sh (revision 11645) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=ppStats +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $LIBTOOlIZE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" + DIE=1 +} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOHEADER installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +$LIBTOOLIZE --copy --force || echo "$LIBTOOlIZE failed" +$ACLOCAL || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/rel-1-0/ppStats/configure.ac =================================================================== --- /branches/rel-1-0/ppStats/configure.ac (revision 11645) +++ /branches/rel-1-0/ppStats/configure.ac (revision 11645) @@ -0,0 +1,30 @@ +AC_PREREQ(2.59) + +AC_INIT([ppStats], [1.0.0], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([src]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([src/config.h]) +AM_MAINTAINER_MODE + +IPP_STDCFLAGS + +AC_LANG(C) +AC_GNU_SOURCE +AC_PROG_CC +AC_PROG_INSTALL +AC_PROG_LIBTOOL +AC_SYS_LARGEFILE + +PKG_CHECK_MODULES([PSLIB], [pslib >= 1.0.0]) +PKG_CHECK_MODULES([PSMODULE], [psmodules >= 1.0.0]) + +IPP_STDOPTS +CFLAGS="${CFLAGS=} -Wall -Werror -std=c99" + +AC_CONFIG_FILES([ + Makefile + src/Makefile + ppStats.pc +]) +AC_OUTPUT Index: /branches/rel-1-0/ppStats/ppStats.pc.in =================================================================== --- /branches/rel-1-0/ppStats/ppStats.pc.in (revision 11645) +++ /branches/rel-1-0/ppStats/ppStats.pc.in (revision 11645) @@ -0,0 +1,11 @@ +prefix=@prefix@ +exec_prefix=@exec_prefix@ +libdir=@libdir@ +includedir=@includedir@ + +Name: libppStats +Description: Pan-STARRS Image Statistics +Version: @VERSION@ +Requires: pslib psmodules +Libs: -L${libdir} -lppStats +Cflags: -I${includedir} Index: /branches/rel-1-0/ppStats/src/.cvsignore =================================================================== --- /branches/rel-1-0/ppStats/src/.cvsignore (revision 11645) +++ /branches/rel-1-0/ppStats/src/.cvsignore (revision 11645) @@ -0,0 +1,10 @@ +.deps +Makefile +Makefile.in +.libs +*.lo +*.la +ppStats +config.h +config.h.in +stamp-h1 Index: /branches/rel-1-0/ppStats/src/Makefile.am =================================================================== --- /branches/rel-1-0/ppStats/src/Makefile.am (revision 11645) +++ /branches/rel-1-0/ppStats/src/Makefile.am (revision 11645) @@ -0,0 +1,41 @@ +lib_LTLIBRARIES = libppStats.la +libppStats_la_CPPFLAGS = $(PSMODULE_CFLAGS) $(PSLIB_CFLAGS) + +bin_PROGRAMS = ppStats + +ppStats_CFLAGS = $(PSMODULE_CFLAGS) $(PSLIB_CFLAGS) +ppStats_LDFLAGS = $(PSMODULE_LIBS) $(PSLIB_LIBS) -Wl,-Bdynamic +ppStats_LDADD = libppStats.la + +libppStats_la_SOURCES = \ + ppStats.c \ + ppStatsData.c \ + ppStatsLoop.c \ + ppStatsSetupFromRecipe.c \ + ppStatsVersion.c + +ppStats_SOURCES = \ + ppStatsData.c \ + ppStatsLoop.c \ + ppStatsSetupFromArgs.c \ + ppStatsStandAlone.c \ + ppStatsVersion.c + +include_HEADERS = \ + ppStats.h \ + ppStatsData.h \ + ppStatsLoop.h \ + ppStatsSetupFromRecipe.h \ + ppStatsVersion.h + +noinst_HEADERS = \ + ppStats.h \ + ppStatsSetupFromArgs.h + +CLEANFILES = *~ + +clean-local: + -rm -f TAGS +# Tags for emacs +tags: + etags `find . -name \*.[ch] -print` Index: /branches/rel-1-0/ppStats/src/ppStats.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStats.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStats.c (revision 11645) @@ -0,0 +1,36 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "ppStats.h" + +psMetadata *ppStats(psMetadata *out, // Output metadata + pmFPA *fpa, // FPA for which to get statistics + pmFPAview *view, // View for analysis + pmConfig *config // Configuration + ) +{ + // Get the options, open the files + ppStatsData *data = ppStatsSetupFromRecipe(NULL, config); + + if (data->fpa) { + psFree(data->fpa); + } + data->fpa = psMemIncrRefCounter(fpa); + + if (data->view) { + psFree(data->view); + } + data->view = psMemIncrRefCounter(view); + + // Go through the FPA and do the hard work + out = ppStatsLoop(out, data, config); + + psFree(data); + + return out; +} Index: /branches/rel-1-0/ppStats/src/ppStats.config =================================================================== --- /branches/rel-1-0/ppStats/src/ppStats.config (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStats.config (revision 11645) @@ -0,0 +1,20 @@ +### Example ppStats recipe + +MULTI HEADERS # May specify multiple HEADERS +HEADERS STR HEADER1,HEADER2 # Read these headers +HEADERS STR HEADER3 # Read these headers too +MULTI STATS # May specify multiple STATS +STATS STR SAMPLE_MEAN,SAMPLE_STDEV # Calculate these statistics +STATS STR ROBUST_MEDIAN # Calculate these statistics too + +SAMPLE F32 0.1 # Fraction of cell to sample + +MULTI CHIPS # May specify multiple CHIPS +CHIPS STR chip00, chip01 # Look at these chips +CHIPS STR chip22 # Look at this chip too + +MULTI CELLS # May specify multiple CELLS +CELLS STR LeftCell,RightCell # Look at these cells +CELLS STR UpsideDownCell # Look at this cell too + +MASKVAL U8 0xff Index: /branches/rel-1-0/ppStats/src/ppStats.h =================================================================== --- /branches/rel-1-0/ppStats/src/ppStats.h (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStats.h (revision 11645) @@ -0,0 +1,20 @@ +#ifndef PP_STATS_H +#define PP_STATS_H + +#define PPSTATS_RECIPE "PPSTATS" + +#include + +#include "ppStatsData.h" +#include "ppStatsSetupFromRecipe.h" +#include "ppStatsLoop.h" +#include "ppStatsVersion.h" + +// Perform the ppStats steps +psMetadata *ppStats(psMetadata *out, // Output metadata + pmFPA *fpa, // FPA for which to get statistics + pmFPAview *view, // View for analysis + pmConfig *config // Configuration + ); + +#endif Index: /branches/rel-1-0/ppStats/src/ppStatsData.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsData.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsData.c (revision 11645) @@ -0,0 +1,51 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "ppStatsData.h" + +static void statsDataFree(ppStatsData *data // Data to free + ) +{ + if (data->fits) { + psFitsClose(data->fits); + data->fits = NULL; + } + psFree(data->fpa); + psFree(data->view); + psFree(data->headers); + psFree(data->concepts); + psFree(data->stats); + + psFree(data->chips); + psFree(data->cells); + + return; +} + + +ppStatsData *ppStatsDataAlloc(void) +{ + ppStatsData *data = psAlloc(sizeof(ppStatsData)); // Newly allocated data + psMemSetDeallocator(data, (psFreeFunc)statsDataFree); + + data->fits = NULL; + data->fpa = NULL; + data->view = NULL; + + data->headers = psListAlloc(NULL); + data->concepts = psListAlloc(NULL); + data->stats = psStatsAlloc(0); + data->doStats = false; + + data->sample = 0; + data->maskVal = 0; + data->chips = psListAlloc(NULL); + data->cells = psListAlloc(NULL); + + return data; +} + Index: /branches/rel-1-0/ppStats/src/ppStatsData.h =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsData.h (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsData.h (revision 11645) @@ -0,0 +1,27 @@ +#ifndef PP_STATS_DATA_H +#define PP_STATS_DATA_H + +#include +#include + +typedef struct { + // Inputs + psFits *fits; // Input file handle + pmFPA *fpa; // FPA to analyse + pmFPAview *view; // View to analyse + // Stuff to output + psStats *stats; // Statistics to calculate + bool doStats; // Do statistics? + psList *headers; // Headers to read + psList *concepts; // Concepts to read + // Options for input data + float sample; // Fraction of cell to sample for statistics + psMaskType maskVal; // Mask value for images + psList *chips; // Chips to look at + psList *cells; // Cells to look at +} ppStatsData; + +// Allocator +ppStatsData *ppStatsDataAlloc(void); + +#endif Index: /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.c.in =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.c.in (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.c.in (revision 11645) @@ -0,0 +1,26 @@ +/* + * The line + { PPSTATS_ERR_$X{ErrorCode}, "$X{ErrorDescription}"}, + * (without the Xs) + * will be replaced by values from errorCodes.dat + */ +#include "pslib.h" +#include "ppStatsErrorCodes.h" + +void ppStatsErrorRegister(void) +{ + static psErrorDescription errors[] = { + { PPSTATS_ERR_BASE, "First value we use; lower values belong to psLib" }, + { PPSTATS_ERR_${ErrorCode}, "${ErrorDescription}"}, + }; + static int nerror = PPSTATS_ERR_NERROR - PPSTATS_ERR_BASE; // number of values in enum + + for (int i = 0; i < nerror; i++) { + psErrorDescription *tmp = psAlloc(sizeof(psErrorDescription)); + p_psMemSetPersistent(tmp, true); + *tmp = errors[i]; + psErrorRegister(tmp, 1); + psFree(tmp); /* it's on the internal list */ + } + nerror = 0; // don't register more than once +} Index: /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.dat =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.dat (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.dat (revision 11645) @@ -0,0 +1,10 @@ +# +# This file is used to generate ppStatsErrorClasses.h +# +BASE = 400 First value we use; lower values belong to psLib +# these errors correspond to standard exit conditions +ARGUMENTS Incorrect arguments +SYS System error +CONFIG Problem in configure files +PROG Programming error +DATA invalid data Index: /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.h.in =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.h.in (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsErrorCodes.h.in (revision 11645) @@ -0,0 +1,18 @@ +#if !defined(PPSTATS_ERROR_CODES_H) +#define PPSTATS_ERROR_CODES_H +/* + * The line + * PPSTATS_ERR_$X{ErrorCode}, + * (without the X) + * + * will be replaced by values from errorCodes.dat + */ +typedef enum { + PPSTATS_ERR_BASE = 512, + PPSTATS_ERR_${ErrorCode}, + PPSTATS_ERR_NERROR +} ppStatsErrorCode; + +void ppStatsErrorRegister(void); + +#endif Index: /branches/rel-1-0/ppStats/src/ppStatsLoop.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsLoop.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsLoop.c (revision 11645) @@ -0,0 +1,339 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "ppStats.h" +#include "ppStatsLoop.h" + + +static void getMetadata(psMetadata *target, // Target for metadata + psMetadata *source, // Source for metadata + psList *list // List containing keywords + ) +{ + psListIterator *iterator = psListIteratorAlloc(list, PS_LIST_HEAD, false); // Iterator + psString name; // Name from iteration + while ((name = psListGetAndIncrement(iterator))) { + psMetadataItem *item = psMetadataLookup(source, name); // Item of interest, or NULL + if (item) { + psMetadataAddItem(target, item, PS_LIST_TAIL, 0); + } + } + psFree(iterator); + return; +} + +static bool doThis(psList *toDoList, // List of things to do + const char *this // The name of "this" + ) +{ + if (psListLength(toDoList) == 0) { + // No list --- do everything + return true; + } + + psListIterator *iterator = psListIteratorAlloc(toDoList, PS_LIST_HEAD, false); // Iterator + psString test; // Test string, from iteration + while ((test = psListGetAndIncrement(iterator))) { + if (strcmp(this, test) == 0) { + // It's in the list --- do it + psFree(iterator); + return true; + } + } + psFree(iterator); + // Couldn't find it --- don't do it + return false; +} + +static void addToHierarchy(psMetadata *source, // Source to add + psMetadata *target, // Target to which to add + const char *name, // Name of source + const char *comment // Comment for source + ) +{ + if (psListLength(source->list) > 0 && !psMetadataLookup(target, name)) { + psMetadataAdd(target, PS_LIST_TAIL, name, PS_DATA_METADATA, + comment, source); + } + psFree(source); // Drop reference + return; +} + + +static void cellStats(psMetadata *chipResults, // Metadata holding the chip results + pmCell *cell, // Cell for which to get statistics + psFits *fits, // FITS file handle + ppStatsData *data,// The data + const pmConfig *config // Configuration + ) +{ + assert(chipResults); + assert(cell); + assert(data); + assert(config); + + const char *cellName = psMetadataLookupStr(NULL, cell->concepts, "CELL.NAME"); // Name of cell + + // Check to see if this is a cell of interest + if (!doThis(data->cells, cellName)) { + return; + } + + // Cell-level results + bool mdok; // Status of MD lookup + psMetadata *cellResults = psMemIncrRefCounter(psMetadataLookupMetadata(&mdok, chipResults, cellName)); + if (!mdok || !cellResults) { + cellResults = psMetadataAlloc(); + } + + if (psListLength(data->headers) > 0 && cell->hdu) { + if (fits && !pmCellReadHeader(cell, fits)) { + goto cellDone; + } + pmHDU *hdu = cell->hdu; // HDU for headers + getMetadata(cellResults, hdu->header, data->headers); + } + if (psListLength(data->concepts) > 0) { + if (fits && !pmCellReadHeader(cell, fits)) { + goto cellDone; + } + pmConceptsReadCell(cell, PM_CONCEPT_SOURCE_ALL, false, config->database); + getMetadata(cellResults, cell->concepts, data->concepts); + } + + if (!data->doStats) { + // Nothing further to do --- don't want to waste our time reading the data + if (psListLength(cellResults->list) > 0) { + psMetadataAdd(chipResults, PS_LIST_TAIL, cellName, PS_DATA_METADATA, + "Results for cell", cellResults); + } + goto cellDone; + } + + pmHDU *hdu = pmHDUFromCell(cell); // HDU for cell + if (!hdu || hdu->blankPHU) { + goto cellDone; + } + + if (fits && !pmCellRead(cell, fits, config->database)) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to read cell %s\n", cellName); + goto cellDone; + } + + psArray *readouts = cell->readouts; // Array of component readouts + if (readouts->n == 0) { + goto cellDone; + } + if (readouts->n > 1) { + psLogMsg(__func__, PS_LOG_WARN, "Multiple readouts (%ld) present in cell %s --- " + "using only the first.\n", readouts->n, cellName); + } + pmReadout *readout = readouts->data[0]; // The readout of interest + if (!readout->image) { + psLogMsg(__func__, PS_LOG_WARN, "No image associated with readout in cell %s --- " + "ignored.\n", cellName); + goto cellDone; + } + + // Do the statistics + if (data->sample <= 0.0) { + if (!psImageStats(data->stats, readout->image, readout->mask, data->maskVal)) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to perform statistics on cell %s --- " + "ignored.\n", cellName); + goto cellDone; + } + } else { + // Apply sampling + psImage *image = readout->image; // The image of interest + psImage *mask = readout->mask; // The mask image + int numSamples = data->sample * image->numCols * image->numRows; // Number of samples + int sampleSpace = 1.0 / data->sample; // Space between samples + psVector *sampleValues = psVectorAlloc(numSamples, PS_TYPE_F32); // Vector of samples + psVector *sampleMask = NULL; // Corresponding mask + if (mask) { + sampleMask = psVectorAlloc(numSamples, PS_TYPE_U8); + } + for (int i = 0; i < numSamples; i++) { + int j = i * sampleSpace; + int y = j / image->numCols; + int x = j % image->numCols; + sampleValues->data.F32[i] = image->data.F32[y][x]; + if (mask) { + sampleMask->data.U8[i] = mask->data.U8[y][x]; + } + } + if (!psVectorStats(data->stats, sampleValues, NULL, sampleMask, data->maskVal)) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to perform statistics on cell %s --- " + "ignored.\n", cellName); + psFree(sampleValues); + psFree(sampleMask); + goto cellDone; + } + psFree(sampleValues); + psFree(sampleMask); + } + +#define WRITE_STAT(SYMBOL, NAME, SOURCE) \ + if (data->stats->options & SYMBOL) { \ + psMetadataAddF32(cellResults, PS_LIST_TAIL, NAME, 0, NULL, data->stats->SOURCE); \ + } + + WRITE_STAT(PS_STAT_SAMPLE_MEAN, "SAMPLE_MEAN", sampleMean); + WRITE_STAT(PS_STAT_SAMPLE_MEDIAN, "SAMPLE_MEDIAN", sampleMedian); + WRITE_STAT(PS_STAT_SAMPLE_STDEV, "SAMPLE_STDEV", sampleStdev); + WRITE_STAT(PS_STAT_SAMPLE_QUARTILE, "SAMPLE_LQ", sampleLQ); + WRITE_STAT(PS_STAT_SAMPLE_QUARTILE, "SAMPLE_UQ", sampleUQ); + WRITE_STAT(PS_STAT_ROBUST_MEDIAN, "ROBUST_MEDIAN", robustMedian); + WRITE_STAT(PS_STAT_ROBUST_STDEV, "ROBUST_STDEV", robustStdev); + WRITE_STAT(PS_STAT_ROBUST_QUARTILE, "ROBUST_LQ", robustLQ); + WRITE_STAT(PS_STAT_ROBUST_QUARTILE, "ROBUST_UQ", robustUQ); + WRITE_STAT(PS_STAT_ROBUST_QUARTILE, "ROBUST_N50", robustN50); + WRITE_STAT(PS_STAT_FITTED_MEAN, "FITTED_MEAN", fittedMean); + WRITE_STAT(PS_STAT_FITTED_STDEV, "FITTED_STDEV", fittedStdev); + WRITE_STAT(PS_STAT_CLIPPED_MEAN, "CLIPPED_MEAN", clippedMean); + WRITE_STAT(PS_STAT_CLIPPED_STDEV, "CLIPPED_STDEV", clippedStdev); + +cellDone: + // Add the cell results to the chip + addToHierarchy(cellResults, chipResults, cellName, "Results for cell"); + if (fits) { + pmCellFreeData(cell); + } + return; +} + +static bool chipStats(psMetadata *fpaResults, // Metadata holding the fpa results + pmChip *chip, // Chip for which to get statistics + psFits *fits, // FITS file handle + pmFPAview *view, // View for analysis + ppStatsData *data,// The data + const pmConfig *config // Configuration + ) +{ + assert(fpaResults); + assert(chip); + assert(view); + assert(data); + assert(config); + + const char *chipName = psMetadataLookupStr(NULL, chip->concepts, "CHIP.NAME"); // Name of chip + + // Check to see if this is a chip of interest + if (!doThis(data->chips, chipName)) { + return true; + } + + psArray *cells = chip->cells; // Array of cells + if (view->cell >= cells->n) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "Desired cell view (%d) doesn't match " + "number of cells (%ld)\n", view->cell, cells->n); + return false; + } + + // Chip-level results + bool mdok; // Status of MD lookup + psMetadata *chipResults = psMemIncrRefCounter(psMetadataLookupMetadata(&mdok, fpaResults, chipName)); + if (!mdok || !chipResults) { + chipResults = psMetadataAlloc(); + } + + if (psListLength(data->headers) > 0 && chip->hdu) { + if (fits && !pmChipReadHeader(chip, fits)) { + goto chipDone; + } + pmHDU *hdu = chip->hdu; // HDU for headers + getMetadata(chipResults, hdu->header, data->headers); + } + if (psListLength(data->concepts) > 0) { + if (fits && !pmChipReadHeader(chip, fits)) { + goto chipDone; + } + pmConceptsReadChip(chip, PM_CONCEPT_SOURCE_ALL, false, false, config->database); + getMetadata(chipResults, chip->concepts, data->concepts); + } + + if (view->cell >= 0) { + pmCell *cell = cells->data[view->cell]; // Cell of interest + cellStats(chipResults, cell, fits, data, config); + goto chipDone; + } + + // Iterate over cells + for (int i = 0; i < cells->n; i++) { + pmCell *cell = cells->data[i]; // Cell of interest + cellStats(chipResults, cell, fits, data, config); + } + +chipDone: + addToHierarchy(chipResults, fpaResults, chipName, "Results for chip"); + if (fits) { + pmChipFreeData(chip); + } + + return true; +} + + +psMetadata *ppStatsLoop(psMetadata *fpaResults, // Metadata to hold the FPA results + ppStatsData *data, // The data + const pmConfig *config // Configuration + ) +{ + PS_ASSERT_PTR_NON_NULL(data, NULL); + pmFPA *fpa = data->fpa; // FPA to analyse + psFits *fits = data->fits; // FITS file handle + PS_ASSERT_PTR_NON_NULL(fpa, NULL); + + pmFPAview *view = psMemIncrRefCounter(data->view); // View for analysis + if (!view) { + view = pmFPAviewAlloc(0); + } + + if (!fpaResults) { + fpaResults = psMetadataAlloc(); + } + + // Iterate through the FPA + if (psListLength(data->headers) > 0 && fpa->hdu) { + if (fits) { + pmFPAReadHeader(fpa, fits); + } + pmHDU *hdu = fpa->hdu; // HDU for headers + getMetadata(fpaResults, hdu->header, data->headers); + } + if (psListLength(data->concepts) > 0) { + if (fits) { + pmFPAReadHeader(fpa, fits); + } + pmConceptsReadFPA(fpa, PM_CONCEPT_SOURCE_ALL, false, config->database); + getMetadata(fpaResults, fpa->concepts, data->concepts); + } + + psArray *chips = fpa->chips; // Array of chips + if (view->chip >= 0) { + pmChip *chip = chips->data[view->chip]; // Chip of interest + chipStats(fpaResults, chip, fits, view, data, config); + goto fpaDone; + } + + // Iterate over cells + for (int i = 0; i < chips->n; i++) { + pmChip *chip = chips->data[i]; // Chip of interest + chipStats(fpaResults, chip, fits, view, data, config); + } + +fpaDone: + if (fits) { + pmFPAFreeData(fpa); + } + psFree(view); + + return fpaResults; +} Index: /branches/rel-1-0/ppStats/src/ppStatsLoop.h =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsLoop.h (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsLoop.h (revision 11645) @@ -0,0 +1,13 @@ +#ifndef PP_STATS_LOOP_H +#define PP_STATS_LOOP_H + +#include +#include "ppStatsData.h" + +// Loop over the input image and do all the hard work +psMetadata *ppStatsLoop(psMetadata *fpaResults, // Metadata to hold the FPA results + ppStatsData *data, // The data + const pmConfig *config // Configuration + ); + +#endif Index: /branches/rel-1-0/ppStats/src/ppStatsSetupFromArgs.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsSetupFromArgs.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsSetupFromArgs.c (revision 11645) @@ -0,0 +1,202 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "ppStats.h" +#include "ppStatsData.h" +#include "ppStatsSetupFromArgs.h" + +// This file is for setting up the required inputs from the command-line + +// Print usage information and die +static void usageAndDie(pmConfig *config // Configuration (contains the arguments list) + ) +{ + printf("Return headers, concepts and/or image statistics.\n\n" + "Usage:\n" + "\t%s INPUT.fits [OUTPUT_NAME]\n" + "\n", config->argv[0]); + psArgumentHelp(config->arguments); + psFree(config); + psLibFinalize(); + pmConceptsDone(); + pmConfigDone(); + exit(EXIT_FAILURE); +} + +// Generate a list from the arguments +static void listFromArguments(psMetadata *arguments, // Arguments to parse + const char *name, // Name of the item, for error message + const char *flag, // The flag for the argument of interest + psList *target // The target list + ) +{ + psString regex = NULL; // Regular expression for the flag + psStringAppend(®ex, "^%s$", flag); + psMetadataIterator *iterator = psMetadataIteratorAlloc(arguments, PS_LIST_HEAD, regex); // Iterator + psFree(regex); + psMetadataItem *item; // Item from iteration + while ((item = psMetadataGetAndIncrement(iterator))) { + if (item->type != PS_DATA_STRING) { + psLogMsg(__func__, PS_LOG_WARN, "%s name is not of type STRING (%x) --- ignored.\n", + name, item->type); + continue; + } + if (item->data.V && strlen(item->data.V) > 0) { + psListAdd(target, PS_LIST_TAIL, item->data.V); + } + } + psFree(iterator); + + return; +} + +// Set the statistics option; for arguments +static inline void statsOptionArguments(psMetadata *arguments, // Arguments to parse + const char *name, // Name for option + ppStatsData *data, // Configuration data + psStatsOptions option // Option to check for + ) +{ + if (psMetadataLookupBool(NULL, arguments, name)) { + data->stats->options |= option; + data->doStats = true; + } + return; +} + +// Print out what we're going to do, from the list +static void checkList(psList *list, // List + const char *name // Name of list + ) +{ + psString value; + psListIterator *iterator = psListIteratorAlloc(list, PS_LIST_HEAD, false); + while ((value = psListGetAndIncrement(iterator))) { + printf("%s: %s\n", name, value); + } + psFree(iterator); + return; +} + + +ppStatsData *ppStatsSetupFromArgs(pmConfig *config // Configuration + ) +{ + // Setup and parse command-line arguments + psMetadata *arguments = config->arguments; // Arguments + psMetadataAddStr(arguments, PS_LIST_TAIL, "-chip", PS_META_DUPLICATE_OK, "Chip to inspect", NULL); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-cell", PS_META_DUPLICATE_OK, "Cell to inspect", NULL); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-header", PS_META_DUPLICATE_OK, "Header to look up", NULL); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-concept", PS_META_DUPLICATE_OK, "Concept to look up", NULL); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-mean", 0, "Calculate sample mean", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-stdev", 0, "Calculate sample standard deviation", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-median", 0, "Calculate sample median", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-quartile", 0, "Calculate sample quartiles", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-robust-median", 0, "Calculate robust median", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-robust-stdev", 0, "Calculate robust standard deviation", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-robust-quartile", 0, "Calculate robust quartile range", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-fitted-mean", 0, "Calculate fitted mean", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-fitted-stdev", 0, "Calculate fitted standard deviation", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-clipped-mean", 0, "Calculate clipped median", false); + psMetadataAddBool(arguments, PS_LIST_TAIL, "-clipped-stdev", 0, "Calculate clipped standard deviation", false); + psMetadataAddS32(arguments, PS_LIST_TAIL, "-iter", 0, "Clipping iterations", 0); + psMetadataAddF32(arguments, PS_LIST_TAIL, "-rej", 0, "Clipping level", 0.0); + psMetadataAddF32(arguments, PS_LIST_TAIL, "-sample", 0, "Sampling fraction", 0.0); + + if (*config->argc == 1) { + // No command-line arguments: print the help + usageAndDie(config); + } + if (!psArgumentParse(arguments, config->argc, config->argv) || + (*config->argc != 2 && *config->argc != 3)) { + printf("Unable to parse command-line arguments.\n\n"); + usageAndDie(config); + } + + // Parse the command-line options + ppStatsData *data = ppStatsDataAlloc(); // The data + const char *inName = config->argv[1]; // Input file name + psArgumentRemove(1, config->argc, config->argv); + + listFromArguments(arguments, "Chip", "-chip", data->chips); + listFromArguments(arguments, "Cell", "-cell", data->cells); + listFromArguments(arguments, "Header", "-header", data->headers); + listFromArguments(arguments, "Concept", "-concept", data->concepts); + + // Set the statistics options + statsOptionArguments(arguments, "-mean", data, PS_STAT_SAMPLE_MEAN); + statsOptionArguments(arguments, "-stdev", data, PS_STAT_SAMPLE_STDEV); + statsOptionArguments(arguments, "-median", data, PS_STAT_SAMPLE_MEDIAN); + statsOptionArguments(arguments, "-quartile", data, PS_STAT_SAMPLE_QUARTILE); + statsOptionArguments(arguments, "-robust-median", data, PS_STAT_ROBUST_MEDIAN); + statsOptionArguments(arguments, "-robust-stdev", data, PS_STAT_ROBUST_STDEV); + statsOptionArguments(arguments, "-robust-quartile", data, PS_STAT_ROBUST_QUARTILE); + statsOptionArguments(arguments, "-fitted-mean", data, PS_STAT_FITTED_MEAN); + statsOptionArguments(arguments, "-fitted-stdev", data, PS_STAT_FITTED_STDEV); + statsOptionArguments(arguments, "-clipped-mean", data, PS_STAT_CLIPPED_MEAN); + statsOptionArguments(arguments, "-clipped-stdev", data, PS_STAT_CLIPPED_STDEV); + data->stats->clipSigma = psMetadataLookupF32(NULL, arguments, "-rej"); + data->stats->clipIter = psMetadataLookupS32(NULL, arguments, "-iter"); + data->sample = psMetadataLookupF32(NULL, arguments, "-sample"); + + + // Open the input file, determine the camera + { + data->fits = psFitsOpen(inName, "r"); + if (!data->fits) { + psError(PS_ERR_IO, false, "Unable to open input file %s\n", inName); + goto die; + } + psMetadata *header = psFitsReadHeader(NULL, data->fits); // The FITS (primary) header + psMetadata *format = pmConfigCameraFormatFromHeader(config, header); + if (!format) { + psError(PS_ERR_UNKNOWN, false, "Unable to determine camera format for %s\n", inName); + psFree(header); + goto die; + } + data->fpa = pmFPAConstruct(config->camera); + if (!data->fpa) { + psError(PS_ERR_UNKNOWN, false, "Unable to construct FPA for %s\n", inName); + psFree(header); + psFree(format); + goto die; + } + pmFPAview *view = pmFPAAddSourceFromHeader(data->fpa, header, format); + psFree(header); + psFree(format); + if (!view) { + psError(PS_ERR_UNKNOWN, false, "Unable to add input file %s to FPA.\n", inName); + goto die; + } + psFree(view); + } + + // Get the rest from the recipe + ppStatsSetupFromRecipe(data, config); + + // Print out what we're going to do + if (psTraceGetLevel("ppStats") > 9) { + checkList(data->chips, "CHIP"); + checkList(data->cells, "CELL"); + checkList(data->headers, "HEADER"); + checkList(data->concepts, "CONCEPT"); + } + + return data; + + // Common path for error conditions: clean up and exit. +die: + psErrorStackPrint (stderr, "failure in %s", __func__); + psFree(config); + psFree(data); + pmConceptsDone(); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_FAILURE); +} Index: /branches/rel-1-0/ppStats/src/ppStatsSetupFromArgs.h =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsSetupFromArgs.h (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsSetupFromArgs.h (revision 11645) @@ -0,0 +1,12 @@ +#ifndef PP_STATS_SETUP_FROM_ARGS_H +#define PP_STATS_SETUP_FROM_ARGS_H + +#include +#include "ppStatsData.h" + +// Set up the options and input/output files +ppStatsData *ppStatsSetupFromArgs(pmConfig *config // Configuration + ); + + +#endif Index: /branches/rel-1-0/ppStats/src/ppStatsSetupFromRecipe.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsSetupFromRecipe.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsSetupFromRecipe.c (revision 11645) @@ -0,0 +1,139 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "ppStats.h" +#include "ppStatsData.h" +#include "ppStatsSetupFromRecipe.h" + +// Strings in a recipe may be defined multiply (with MULTI) or listed on a single line +static void listFromRecipe(psMetadata *recipe, // Recipe to search + const char *name, // Name for item within recipe + psList *target // The target list + ) +{ + // If the list already has entries, don't read anything else + if (psListLength(target) > 0) { + return; + } + + // First check that at least one item of interest exists + psMetadataItem *checkItem = psMetadataLookup(recipe, name); + if (!checkItem) { + // Nothing to see here + return; + } + + psString regex = NULL; // Regular expression for the flag + psStringAppend(®ex, "^%s$", name); + psMetadataIterator *iterator = psMetadataIteratorAlloc(recipe, PS_LIST_HEAD, regex); + psFree(regex); + psMetadataItem *item; + int numItem = 0; // Occurrence of the item in the recipe; to help the user in case of trouble + while ((item = psMetadataGetAndIncrement(iterator))) { + numItem++; + if (item->type != PS_DATA_STRING) { + psLogMsg(__func__, PS_LOG_WARN, "Occurrence %d of %s in the recipe is " + "not of type STRING (%x) --- ignored.\n", numItem, name, item->type); + continue; + } + // Parse into a list of independent values + psList *values = psStringSplit(item->data.V, " ,;", false); + // Copy into the target + psListIterator *valuesIter = psListIteratorAlloc(values, PS_LIST_HEAD, false); + psString valueString; + while ((valueString = psListGetAndIncrement(valuesIter))) { + psListAdd(target, PS_LIST_TAIL, valueString); + } + psFree(valuesIter); + psFree(values); + } + psFree(iterator); + + return; +} + +ppStatsData *ppStatsSetupFromRecipe(ppStatsData *data, // Data for running ppStats + pmConfig *config // Configuration + ) +{ + if (!data) { + data = ppStatsDataAlloc(); + } + + // Determine recipe parameters + bool mdok; // Status of MD lookup + psMetadata *recipe = psMetadataLookupMetadata(&mdok, config->recipes, PPSTATS_RECIPE); + if (!mdok || !recipe) { + psLogMsg(__func__, PS_LOG_WARN, "Unable to find recipe %s.\n", PPSTATS_RECIPE); + return data; + } + + listFromRecipe(recipe, "CHIP", data->chips); + listFromRecipe(recipe, "CELL", data->cells); + listFromRecipe(recipe, "HEADER", data->headers); + listFromRecipe(recipe, "CONCEPT", data->concepts); + + // Parse the statistics options + psList *recipeStats = psListAlloc(NULL); // List of statistics options + listFromRecipe(recipe, "STAT", recipeStats); + if (psListLength(recipeStats) > 0) { + psListIterator *iterator = psListIteratorAlloc(recipeStats, PS_LIST_HEAD, false); + psString statString; // Statistic string, from iteration + while ((statString = psListGetAndIncrement(iterator))) { + psStatsOptions stat = psStatsOptionFromString(statString); + if (stat == 0) { + psLogMsg(__func__, PS_LOG_WARN, "Can't interpret STATS entry in recipe: " + "%s --- ignored.\n", statString); + continue; + } + data->stats->options |= stat; + data->doStats = true; + } + } + psFree(recipeStats); + + // Clipping options + if (data->stats->clipIter == 0 && isnan(data->stats->clipSigma)) { + int iter = psMetadataLookupS32(&mdok, recipe, "ITER"); // Number of clipping iterations + if (mdok && iter > 0) { + data->stats->clipIter = iter; + } else { + psLogMsg(__func__, PS_LOG_WARN, "ITER in recipe is not of type S32 and positive --- " + "retaining default.\n"); + } + float rej = psMetadataLookupF32(&mdok, recipe, "REJ"); // Clipping level + if (mdok && rej > 0) { + data->stats->clipSigma = rej; + } else { + psLogMsg(__func__, PS_LOG_WARN, "REJ in recipe is not of type F32 and positive --- " + "retaining default.\n"); + } + } + + if (data->sample == 0) { + float sample = psMetadataLookupF32(&mdok, recipe, "SAMPLE"); // Sample fraction + if (mdok && sample > 0) { + data->sample = sample; + } else { + psLogMsg(__func__, PS_LOG_WARN, "SAMPLE in recipe is not of type F32 and positive --- " + "retaining default.\n"); + } + } + + if (data->maskVal == 0) { + psMaskType maskVal = psMetadataLookupU8(&mdok, recipe, "MASKVAL"); // Mask value + if (mdok) { + data->maskVal = maskVal; + } else { + psLogMsg(__func__, PS_LOG_WARN, "ITER in recipe is not of type U8 --- retaining default.\n"); + } + } + + return data; +} Index: /branches/rel-1-0/ppStats/src/ppStatsSetupFromRecipe.h =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsSetupFromRecipe.h (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsSetupFromRecipe.h (revision 11645) @@ -0,0 +1,11 @@ +#ifndef PP_STATS_SETUP_FROM_RECIPE_H +#define PP_STATS_SETUP_FROM_RECIPE_H + +#include +#include "ppStatsData.h" + +ppStatsData *ppStatsSetupFromRecipe(ppStatsData *data, // Data for running ppStats + pmConfig *config // Configuration + ); + +#endif Index: /branches/rel-1-0/ppStats/src/ppStatsStandAlone.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsStandAlone.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsStandAlone.c (revision 11645) @@ -0,0 +1,94 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#define TIMERNAME "PPSTATS" + +#include "ppStats.h" +#include "ppStatsSetupFromArgs.h" + +int main(int argc, char *argv[]) +{ + psExit status = PS_EXIT_SUCCESS; + + psLibInit(NULL); + psTimerStart(TIMERNAME); + + // Parse the configuration and arguments + pmConfig *config = pmConfigRead(&argc, argv, PPSTATS_RECIPE); + if (!config) { + psError(PS_ERR_UNKNOWN, false, "Unable to read configuration.\n"); + exit(PS_EXIT_CONFIG_ERROR); + } + + // Get the options, open the files + ppStatsData *data = ppStatsSetupFromArgs(config); + if (!data) { + psError(PS_ERR_UNKNOWN, false, "Unable to parse command-line arguments.\n"); + exit(PS_EXIT_CONFIG_ERROR); + } + + // Output filename is optional + const char *outName = NULL; // Output file name + FILE *outFile = stdout; // Output file + if (*config->argc == 2) { + outName = config->argv[1]; + if (outName && strlen(outName) > 0) { + outFile = fopen(outName, "w"); + if (!outFile) { + psLogMsg("ppStats", PS_LOG_ERROR, "Unable to open output file %s\n", outName); + // XXX this could be a system or config error, but not a data error + status = PS_EXIT_CONFIG_ERROR; + goto die; + } + } else { + psLogMsg("ppStats", PS_LOG_ERROR, "Unable to open output file.\n"); + status = PS_EXIT_CONFIG_ERROR; + goto die; + } + } + + // Go through the FPA and do the hard work + psMetadata *results = ppStatsLoop(NULL, data, config); + if (psListLength(results->list) == 0) { + psLogMsg("ppStats", PS_LOG_ERROR, "No output.\n"); + psFree(results); + status = PS_EXIT_DATA_ERROR; + goto die; + } + + // Format and print the output + psString output = psMetadataConfigFormat(results); + if (!output) { + psLogMsg("ppStats", PS_LOG_ERROR, "Unable to generate configuration file with result.\n"); + psFree(results); + status = PS_EXIT_CONFIG_ERROR; + goto die; + } + fprintf(outFile, "%s", output); + psFree(output); + + // Clean up + psFree(results); + if (outName) { + fclose(outFile); + } + + // Common code for the death. +die: + if (status) { + psErrorStackPrint (stderr, "failure in %s", __func__); + } + psFree(data); + psFree(config); + pmConceptsDone(); + pmConfigDone(); + psLibFinalize(); + + return status; +} Index: /branches/rel-1-0/ppStats/src/ppStatsVersion.c =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsVersion.c (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsVersion.c (revision 11645) @@ -0,0 +1,26 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include "ppStatsVersion.h" + +static const char *cvsTag = "$Name: not supported by cvs2svn $";// CVS tag name + +psString ppStatsVersion(void) +{ + psString version = NULL; // Version, to return + psStringAppend(&version, "%s-%s",PACKAGE_NAME,PACKAGE_VERSION); + return version; +} + +psString ppStatsVersionLong(void) +{ + psString version = ppStatsVersion(); // Version, to return + psString tag = psStringStripCVS(cvsTag, "Name"); // CVS tag + psStringAppend(&version, " (cvs tag %s) %s, %s", tag, __DATE__, __TIME__); + psFree(tag); + return version; +} Index: /branches/rel-1-0/ppStats/src/ppStatsVersion.h =================================================================== --- /branches/rel-1-0/ppStats/src/ppStatsVersion.h (revision 11645) +++ /branches/rel-1-0/ppStats/src/ppStatsVersion.h (revision 11645) @@ -0,0 +1,10 @@ +#ifndef PP_STATS_VERSION_H +#define PP_STATS_VERSION_H + +/// Return short version information +psString ppStatsVersion(void); + +/// Return long version information +psString ppStatsVersionLong(void); + +#endif Index: /branches/rel-1-0/psconfig/INSTALL =================================================================== --- /branches/rel-1-0/psconfig/INSTALL (revision 11645) +++ /branches/rel-1-0/psconfig/INSTALL (revision 11645) @@ -0,0 +1,50 @@ + +The psconfig system allows the user to build and install the IPP +software suite into a location which is flexibly defined by the user. +The tools here also set up the user's environment variables (PATH, +PERL5LIB, LD_LIBRARY_PATH, etc) to make use of the installed +software. With the psconfig tools, it is easy to switch between +different installed versions or to recompile subsets of the IPP tree. + +NOTE: the psconfig.sh script uses the csh / tcsh shell. A bash version +is not currently available. + +--- + +Install Locations + +To use the psconfig system, place the following line in your ~/.cshrc +file: + +alias psconfig "source PATH/psconfig.sh" + +where PATH is the path to the file psconfig.sh in this directory +(ipp/psconfig/psconfig.sh). + +The psconfig system places the installed binary files by default in +directories below ~/psconfig. To use a different location, place the +following line in ~/.psconfigrc (otherwise not needed): + +set PSCONFIG = INSTALL_PATH + +where INSTALL_PATH is the top-level directory for all binary +installations. + +Using psconfig to set / examine your install system: + +psconfig: set or show the current pslib configuration information +USAGE: psconfig (version) : set configuration to specified version + psconfig --bin : return the current path + psconfig --libs : return the current library path + psconfig --list : list currently availabe configuration versions + psconfig --help : this listing + +--- + +Building + +To build the full IPP tree using the psconfig system, run 'psbuild' in +this directory, and provide a release version. (NOTE: if you are +building from a checked-out tarball, the CVS tags are not used, and +you will not be able to use the update. + Index: /branches/rel-1-0/psconfig/TODO =================================================================== --- /branches/rel-1-0/psconfig/TODO (revision 11645) +++ /branches/rel-1-0/psconfig/TODO (revision 11645) @@ -0,0 +1,40 @@ + +2007.02.02 : IPP release 1.0 : Known Outstanding Issues + +This release of the IPP software is missing a number of major features +defined in the IPP System Concept Definition. There are also a number +of missing minor features, bells and whistles which have been +identified. + +Major Missing Features: + +* Phase 4 Tools. This release does not include the Phase 4 programs +pswarp, ppStac, and poisub, and the related infrastructure support in +ippTools and ippScripts. + +* Static Sky tools. This release does not include a number of elements +related to the static sky definition: the definition of the static sky +cells and tools to manipulate them; the static sky surface brightness +modelling tool; and photometry analysis operations for large, extended +objects. + +* relastro. The global astrometry analysis software is unimplemented. + +* flat-field corrections. The DVO flat-field correction analysis is +not integrated with the rest of the IPP detrend creation system. + +Minor Missing Features: + +* full set of columns in output object tables from Phase 2. +* ippMonitor user interface missing a variety of options +* output plots from psphot and pspastro are not exported to ippMonitor +* this list is complete. +* ETC... + +In addition, there are + +* incomplete documentation + +* incomplete testing on: + - Nebulous integration + - DVO integration Index: /branches/rel-1-0/psconfig/mkdist =================================================================== --- /branches/rel-1-0/psconfig/mkdist (revision 11645) +++ /branches/rel-1-0/psconfig/mkdist (revision 11645) @@ -0,0 +1,188 @@ +#!/usr/bin/env perl + +$cvs = 0; +$diff = 0; +$settag = 0; +$deltag = 0; +$setbranch = 0; +$list = 0; +$module = ""; +@tARGV = (); +for (; @ARGV > 0; ) { + if ($ARGV[0] eq "-diff") { + $diff = 1; + shift; next; + } + if ($ARGV[0] eq "-tag") { + $settag = 1; + shift; next; + } + if ($ARGV[0] eq "-branch") { + $setbranch = 1; + shift; next; + } + if ($ARGV[0] eq "-deltag") { + $deltag = 1; + shift; next; + } + if ($ARGV[0] eq "-list") { + $list = 1; + shift; next; + } + if ($ARGV[0] eq "-cvs") { + $cvs = 1; + shift; next; + } + if ($ARGV[0] eq "-module") { + shift; + $module = $ARGV[0]; + shift; next; + } + if ($ARGV[0] eq "-h") { &usage (); } + if ($ARGV[0] eq "-help") { &usage (); } + if ($ARGV[0] eq "--help") { &usage (); } + @tARGV = (@tARGV, $ARGV[0]); + shift; +} +@ARGV = @tARGV; + +$distdir = "tagsets"; + +if (!$list && @ARGV != 1) { &usage (); } +if ( $list && @ARGV != 0) { &usage (); } + +$file = "$distdir/$ARGV[0].dst"; +open (FILE, $file) || die "ERROR: can't open distribution file\n"; +@list = ; +close (FILE); + +$version = $ARGV[0]; + +@name = (); +@tag = (); +@branch = (); + +if ($list) { + printf STDERR "Available Distributions\n"; + printf STDERR "%-16s %7s\n", "Module", "Version"; + printf STDERR "-------------------------\n"; +} + +foreach $line (@list) { + chop $line; + if ($line =~ m|^\s*$|) { next; } + if ($line =~ m|^\s*\#|) { next; } + + ($name, $branch, $branchver, $mode) = split (":", $line); + + if ($name eq "") { die "missing module name\n"; } + + if ($branch eq "") { + $branch = "HEAD"; + } + if ($branch eq "HEAD") { + $tag = "HEAD"; + } + if ($branch eq "NONE") { + $tag = "NONE"; + } + if (($tag ne "HEAD") && ($branchver eq "")) { + $branchver = "0"; + } + if (($branch ne "HEAD") && ($branch ne NONE)) { + $tag = "$branch-$branchver"; + } + # print "module: $name, tag: $tag, branch: $branch\n"; + + push @name, $name; + push @branch, $branch; + push @tag, $tag; + push @mode, $mode; +} + +if ($list) { exit 0; } + +if ($diff) { &difflist (); } + +if ($settag) { &settags (); } + +mkdir $version; +chdir $version; +for ($i = 0; $i < @name; $i++) { + if (($module ne "") && ($module ne $name[$i])) { next; } + if ($mode[$i] eq "SKIP") { next; } + if ($tag[$i] eq "HEAD") { + &vsystem ("cvs co $name[$i]"); + if ($status) { die "error running cvs"; } + } else { + &vsystem ("cvs co -r $tag[$i] $name[$i]"); + if ($status) { die "error running cvs"; } + } +} +chdir ".."; +if ($cvs) { exit 0; } + +&vsystem ("rm -r `find $version -name CVS`"); +if (! -e tarballs) { mkdir "tarballs"; } +&vsystem ("tar cvzf tarballs/$version.tgz $version"); +if ($status) { die "error creating tarball\n"; } +exit 0; + +sub vsystem { + print STDERR "@_\n"; + $status = system ("@_"); + $status; +} + +sub usage { + print STDERR "USAGE: mkdist -list\n"; + print STDERR " : list valid distributions \n\n"; + print STDERR "USAGE: mkdist (group) (version) [-cvs]\n"; + print STDERR " : make a distribution tarball\n"; + print STDERR " : -cvs : only check out the cvs tree, don't make a tarball\n\n"; + print STDERR "USAGE: mkdist -diff (group) (version)\n"; + print STDERR " : show the difference between the distribution and current tree\n\n"; + exit 2; +} + +sub difflist { + + for ($i = 0; $i < @name; $i++) { + # the base component cannot be rdiffed + print STDERR "--- $name[$i] ---\n"; + if ($name[$i] eq "base") { next; } + if ($tag[$i] eq "") { + &vsystem ("cvs -q rdiff -s -r HEAD $name[$i]"); + } else { + &vsystem ("cvs -q rdiff -s -r $tag[$i] $name[$i]"); + } + print STDERR "\n\n"; + } + exit 0; +} + +sub settags { + print STDERR "setting tags\n"; + for ($i = 0; $i < @name; $i++) { + # can the base component be tagged? + if ($name[$i] eq "base") { + # make this automatic by checking it out and cd'ing there + print STDERR "remember to set the base tag by hand\n"; + next; + } + if ($tag[$i] eq "") { next; } + if ($tag[$i] eq "HEAD") { next; } + if ($tag[$i] eq "NONE") { next; } + if ($mode[$i] eq "EXTERN") { + print STDERR "remember to set tag for $name[$i]\n"; + next; + } + $flag = ""; + if ($setbranch) { + &vsystem ("cvs -q rtag -b $branch[$i] $name[$i]"); + } else { + &vsystem ("cvs -q rtag -r $branch[$i] $tag[$i] $name[$i]"); + } + } + exit 0; +} Index: /branches/rel-1-0/psconfig/psbuild =================================================================== --- /branches/rel-1-0/psconfig/psbuild (revision 11645) +++ /branches/rel-1-0/psconfig/psbuild (revision 11645) @@ -0,0 +1,205 @@ +#!/bin/csh + +set args="" +set clean=0 +set rebuild=0 +set update=0 +set optimize=0 +set version="" +set start="" +set stop="" +while ($#argv) + switch ($1) + case -version: + shift + set version=$1; + breaksw; + case -clean: + set clean=1; + breaksw; + case -rebuild: + set rebuild=1; + set clean=1; + breaksw; + case -update: + set update=1; + breaksw; + case -optimize: + set optimize=1; + breaksw; + case -only: + shift + set start=$1; + set stop=$1; + breaksw; + case -start: + shift + set start=$1; + breaksw; + case -stop: + shift + set stop=$1; + breaksw; + default: + set args=($args $1); + breaksw; + endsw + shift +end + +if ($#args != 1) goto usage +if ("$args" == "") goto usage + +set group = $args[1] +set tagsets = tagsets/$group + +if (! -e $tagsets) then + echo "target $group not found in tagsets" + exit 2 +endif + +# set the version based on psconfig or -version (if set) +if ("$version" != "") then + source psconfig.sh $version +else + if (! $?PSVERSION) then + source psconfig.sh default + else + source psconfig.sh $PSVERSION + endif +endif + +set psopts = "" +if ($optimize) set psopts = "$psopts --enable-optimize" + +foreach target (`cat $tagsets | grep -v \#`) + set name = `echo $target | tr ':' ' '` + + set dir = $name[1] + set tag = $name[2] + + set mode = "NORMAL" + set branch = 0 + + if ($#name >= 3) then + set branch = $name[3] + endif + if ($#name >= 4) then + set mode = $name[4] + endif + + if ("$start" != "") then + if ("$start" != "$dir") then + echo "skipping $dir" + continue + endif + set start = "" + endif + + if ("$mode" == "SKIP") then + echo "skipping $dir" + continue + endif + + set current = `pwd` + cd ../$dir + + echo "" + echo "***** building $name ($target) *****" + pwd + + if ($update) then + switch ($tag) + case HEAD: + cvs -q up -d -A + if ($status) goto failure; + breaksw; + case NONE: + breaksw; + default: + cvs -q up -d -r $tag + if ($status) goto failure; + breaksw; + endsw + endif + + # how do we build this component? + # - autogen.sh : configure : make : make install + # - configure : make : make install + # - make : make install + # - perl Build.PL : ./Build : ./Build install + + if (-e Build.PL) then + echo $PERL5LIB + alias psperlbuild + psperlbuild + # perl Build.PL --install_path script=$bindir --install_path lib=$libdir --install_path bindoc={$PSCONFDIR}/man/man1 --install_path libdoc={$PSCONFDIR}/man/man3 + if ($status) goto failure; + + ./Build + if ($status) goto failure; + + ./Build install + if ($status) goto failure; + + goto success; + endif + + if ($rebuild && $clean) then + if (-e configure) rm -f Makefile + if (-e configure.ac && -e autogen.sh) rm -f configure + endif + + if (! -e Makefile) set rebuild = 1 + + # fix this: this runs autogen, which runs configure, then re-runs configure! + if ($rebuild && ! -e configure && -e autogen.sh) then + echo psautogen $psopts + psautogen $psopts + if ($status) goto failure; + endif + + if ($rebuild && -e configure) then + echo psconfigure $psopts + psconfigure $psopts + if ($status) goto failure; + endif + + if (! -e Makefile) goto failure; + + if ($clean) then + echo make clean + make clean + if ($status) goto failure; + endif + + make + if ($status) goto failure; + + make install + if ($status) goto failure; + + success: + cd $current + + if ("$stop" != "") then + if ("$stop" == "$dir") then + echo "stopping at $dir" + exit 0 + endif + endif + +end + +exit 0 + +usage: + echo "USAGE: psbuild (group) [-version version] [-clean] [-rebuild] [-update] [-optimize]" + exit 2 + +failure: + echo "psbuild failed" + echo "target: $target" + echo "dir: $dir" + echo "tag: $tag" + exit 2 Index: /branches/rel-1-0/psconfig/psconfig.sh =================================================================== --- /branches/rel-1-0/psconfig/psconfig.sh (revision 11645) +++ /branches/rel-1-0/psconfig/psconfig.sh (revision 11645) @@ -0,0 +1,246 @@ +#/bin/csh -f +# this file is sourced using the command 'psconfig' + +# list the current settings +if ($#argv == 0) then + if (! $?PSCONFDIR) echo "PSCONFDIR is not set : run psconfig with an argument to setup" + if (! $?PSVERSION) echo "PSVERSION is not set : run psconfig with an argument to setup" + if (! $?ARCH) echo "ARCH is not set : run psconfig with an argument to setup" + echo $PSCONFDIR : $PSVERSION : $ARCH + echo "psconfig --help for additional info" + exit 2 +endif + +# show usage information +if ("$argv[1]" == "--help") then + echo "psconfig: set or show the current pslib configuration information" + echo "USAGE: psconfig (version) : set configuration to specified version" + echo " psconfig --bin : return the current path" + echo " psconfig --libs : return the current library path" + echo " psconfig --list : list currently availabe configuration versions" + echo " psconfig --help : this listing" + echo " use ~/.psconfigrc to set PSCONFDIR as desired" + exit 0 +endif + +# list the defined psconfig versions +if ("$argv[1]" == "--list") then + /bin/ls $PSCONFDIR | grep -v "\" | grep -v "\" | awk -v dir=$PSCONFDIR -F. '{printf "%s : %-10s : %s\n", dir, $1, $2}' + exit 0 +endif + +# list the defined psconfig versions +if ("$argv[1]" == "--libs") then + echo $PSCONFDIR/$PSVERSION.$ARCH/lib + exit 0 +endif + +# list the defined psconfig versions +if ("$argv[1]" == "--bin") then + echo $PSCONFDIR/$PSVERSION.$ARCH/bin + exit 0 +endif + +# make this configurable by the user +setenv PSCONFDIR {$HOME}/psconfig +setenv PSVERSION default + +if (-e $HOME/.psconfigrc) then + source $HOME/.psconfigrc +endif + +setenv PSVERSION $argv[1] + +if ($?LD_LIBRARY_PATH == 0) setenv LD_LIBRARY_PATH +if ($?PKG_CONFIG_PATH == 0) setenv PKG_CONFIG_PATH +if ($?PERL5LIB == 0) setenv PERL5LIB + +# identify system architecture +set sys=`uname -s` +if ("$sys" == "SunOS") then + set ver = `uname -r | awk '{print substr($1,1,1)}'`; + if ($ver > 4) then + set sys = "Solaris" + endif +endif +if ("$sys" == "Linux") then + grep "Sidious" /etc/issue > /dev/null + if ($status == 0) then + set sys = Sidious + endif + grep "Red Hat Enterprise" /etc/issue > /dev/null + if ($status == 0) then + set sys = RedHat + endif +endif + +# determine architecture-dependent paths & variables for all shells +switch ($sys) + case IRIX64: + setenv ARCH irix; + breaksw; + + case Solaris: + setenv ARCH sol + breaksw; + + case SunOs: + setenv ARCH sun4 + breaksw; + + case Linux: + setenv ARCH linux; + set mach=`uname -m` + if ("$mach" == "x86_64") setenv ARCH lin64 + breaksw; + + case Sidious: + setenv ARCH sid; + breaksw; + + case RedHat: + setenv ARCH linrh; + breaksw; + + case HP-UX: + setenv ARCH hp; + breaksw; + + default: + echo "unknown architecture"; + setenv ARCH unknown; + breaksw; +endsw + +setenv PSCONFIG $PSVERSION.$ARCH + +set newpath = "" +set pathlist = `echo $PATH | tr ':' '\n'` +foreach name ($pathlist) + echo $name | grep $PSCONFDIR > /dev/null + if ($status == 0) continue + if ($newpath == "") then + set newpath = {$name}: + else + set newpath = {$newpath}{$name}: + endif +end +set bindir = {$PSCONFDIR}/{$PSCONFIG}/bin +if ("$PSCONFIG" == "none") then + setenv PATH {$newpath} +else + setenv PATH {$bindir}:{$newpath} +endif + +set mandir = {$PSCONFDIR}/man +set newpath = "" +set pathlist = `echo $MANPATH | tr ':' '\n'` +foreach name ($pathlist) + echo $name | grep $PSCONFDIR > /dev/null + if ($status == 0) continue + if ($newpath == "") then + set newpath = {$name}: + else + set newpath = {$newpath}{$name}: + endif +end +if ("$PSCONFIG" == "none") then + if ("$newpath" == "") then + unsetenv MANPATH + else + setenv MANPATH {$newpath} + endif +else + if ("$newpath" == "") then + setenv MANPATH {$mandir}: + else + setenv MANPATH {$mandir}:{$newpath} + endif +endif + +set libdir = {$PSCONFDIR}/{$PSCONFIG}/lib +set newpath = "" +set pathlist = `echo $LD_LIBRARY_PATH | tr ':' '\n'` +foreach name ($pathlist) + echo $name | grep $PSCONFDIR > /dev/null + if ($status == 0) continue + if ($newpath == "") then + set newpath = {$name}: + else + set newpath = {$newpath}{$name}: + endif +end +if ("$PSCONFIG" == "none") then + if ("$newpath" == "") then + unsetenv LD_LIBRARY_PATH + else + setenv LD_LIBRARY_PATH {$newpath} + endif +else + if ("$newpath" == "") then + setenv LD_LIBRARY_PATH {$libdir}: + else + setenv LD_LIBRARY_PATH {$libdir}:{$newpath} + endif +endif + +set plibdir = {$PSCONFDIR}/{$PSCONFIG}/lib +set newpath = "" +set pathlist = `echo $PERL5LIB | tr ':' '\n'` +foreach name ($pathlist) + echo $name | grep $PSCONFDIR > /dev/null + if ($status == 0) continue + if ($newpath == "") then + set newpath = {$name}: + else + set newpath = {$newpath}{$name}: + endif +end +if ("$PSCONFIG" == "none") then + if ("$newpath" == "") then + unsetenv PERL5LIB + else + setenv PERL5LIB {$newpath} + endif +else + if ("$newpath" == "") then + setenv PERL5LIB {$plibdir}: + else + setenv PERL5LIB {$plibdir}:{$newpath} + endif +endif + +set newpath = "" +set pathlist = `echo $PKG_CONFIG_PATH | tr ':' '\n'` +foreach name ($pathlist) + echo $name | grep $PSCONFDIR > /dev/null + if ($status == 0) continue + if ($newpath == "") then + set newpath = {$name}: + else + set newpath = {$newpath}{$name}: + endif +end +if ("$PSCONFIG" == "none") then + if ("$newpath" == "") then + unsetenv PKG_CONFIG_PATH + else + setenv PKG_CONFIG_PATH {$newpath} + endif +else + if ("$newpath" == "") then + setenv PKG_CONFIG_PATH {$PSCONFDIR}/{$PSCONFIG}/lib/pkgconfig: + else + setenv PKG_CONFIG_PATH {$PSCONFDIR}/{$PSCONFIG}/lib/pkgconfig:{$newpath} + endif +endif + +if ("$PSCONFIG" == "none") then + alias psconfigure configure + alias psautogen autogen.sh + alias psperlbuild perl Build.PL +else + alias psconfigure configure --prefix={$PSCONFDIR}/{$PSCONFIG} --bindir=$bindir --libdir=$libdir --includedir={$PSCONFDIR}/{$PSCONFIG}/include --sysconfdir={$PSCONFDIR}/{$PSCONFIG}/etc --datadir={$PSCONFDIR}/share --mandir={$PSCONFDIR}/man + alias psautogen autogen.sh --bindir=$bindir --libdir=$libdir --mandir=$mandir --includedir={$PSCONFDIR}/{$PSCONFIG}/include --sysconfdir={$PSCONFDIR}/{$PSCONFIG}/etc --datadir={$PSCONFDIR}/share + alias psperlbuild perl Build.PL --prefix {$PSCONFDIR}/{$PSCONFIG} --install_path script=$bindir --install_path arch=$bindir --install_path bin=$bindir --install_path lib=$libdir --install_path bindoc={$PSCONFDIR}/man/man1 --install_path libdoc={$PSCONFDIR}/man/man3 +endif Index: /branches/rel-1-0/psconfig/pstag =================================================================== --- /branches/rel-1-0/psconfig/pstag (revision 11645) +++ /branches/rel-1-0/psconfig/pstag (revision 11645) @@ -0,0 +1,66 @@ +#!/bin/tcsh + +set args="" +while ($#argv) + switch ($1) + default: + set args=($args $1); + breaksw; + endsw + shift +end + +if ($#args != 1) goto usage +if ("$args" == "") goto usage + +set group = $args[1] +set tagsets = tagsets/$group + +if (! -e $tagsets) then + echo "target $group not found in tagsets" + exit 2 +endif + +foreach target (`cat $tagsets | grep -v \#`) + set name = `echo $target | tr ':' ' '` + set dir = $name[1] + set tag = $name[2] + + set current = `pwd` + cd ../$dir + + echo "" + echo "***** updating $name ($target) *****" + pwd + + switch ($tag) + case HEAD: + echo "HEAD is not a valid tag" + goto failure; + breaksw; + case NONE: + breaksw; + default: + cvs tag $tag $dir + if ($status) goto failure; + breaksw; + endsw + + success: + cd $current +end + +exit 0 + +usage: + echo "USAGE: pstag (group)" + echo " applies the tags defined in the group file" + echo " tags may not be HEAD" + exit 2 + +failure: + echo "pstag failed" + echo "target: $target" + echo "dir: $dir" + echo "tag: $tag" + exit 2 Index: /branches/rel-1-0/psconfig/psupdate =================================================================== --- /branches/rel-1-0/psconfig/psupdate (revision 11645) +++ /branches/rel-1-0/psconfig/psupdate (revision 11645) @@ -0,0 +1,64 @@ +#!/bin/tcsh + +set args="" +while ($#argv) + switch ($1) + default: + set args=($args $1); + breaksw; + endsw + shift +end + +if ($#args != 1) goto usage +if ("$args" == "") goto usage + +set group = $args[1] +set tagsets = tagsets/$group + +if (! -e $tagsets) then + echo "target $group not found in tagsets" + exit 2 +endif + +foreach target (`cat $tagsets | grep -v \#`) + set name = `echo $target | tr ':' ' '` + set dir = $name[1] + set tag = $name[2] + + set current = `pwd` + cd ../$dir + + echo "" + echo "***** updating $name ($target) *****" + pwd + + switch ($tag) + case HEAD: + cvs -q up -d -A + if ($status) goto failure; + breaksw; + case NONE: + breaksw; + default: + cvs -q up -d -r $tag + if ($status) goto failure; + breaksw; + endsw + + success: + cd $current +end + +exit 0 + +usage: + echo "USAGE: psupdate (group)" + exit 2 + +failure: + echo "psupdate failed" + echo "target: $target" + echo "dir: $dir" + echo "tag: $tag" + exit 2 Index: /branches/rel-1-0/psconfig/tagsets/ipp-1.0.build =================================================================== --- /branches/rel-1-0/psconfig/tagsets/ipp-1.0.build (revision 11645) +++ /branches/rel-1-0/psconfig/tagsets/ipp-1.0.build (revision 11645) @@ -0,0 +1,41 @@ + +# mode EXTERN means package cannot be tagged / updated in this tree +# mode SKIP means exclude from the build process +# model + +Ohana:HEAD::EXTERN + +# psLib:rel-0-0:0:EXTERN +# psModules:rel-0-0:0:EXTERN + +psLib:HEAD::EXTERN +psModules:HEAD::EXTERN + +psphot:rel-0-8:0 +psastro:rel-0-8:0 +ppStats:rel-1-0:0 +ppImage:rel-1-0:0 +ppNorm:rel-1-0:0 +ppMerge:rel-1-0:0 + +pois:rel-0-1:0:SKIP +pswarp:rel-0-1:0:SKIP +ppStac:rel-0-1:0:SKIP + +# these are the required perl modules from IPP +# there are externally required perl modules (see README?) +PS-IPP-Metadata-Config:rel-1-0:0 +PS-IPP-Config:rel-1-0:0 +ippScripts:rel-1-0:0 + +glueforge:rel-1:0 +dbconfig:rel-1-0:0 + +ippdb.src:NONE + +ippTools:rel-1-0:0 +ippTasks:rel-1-0:0 +simtest:rel-0-8:0 + +psconfig:rel-1-0:0:SKIP +ippMonitor:rel-1-0:0:SKIP Index: /branches/rel-1-0/psconfig/tagsets/ipp-1.0.dst =================================================================== --- /branches/rel-1-0/psconfig/tagsets/ipp-1.0.dst (revision 11645) +++ /branches/rel-1-0/psconfig/tagsets/ipp-1.0.dst (revision 11645) @@ -0,0 +1,65 @@ + +# mode EXTERN means package cannot be tagged / updated in this tree +# mode SKIP means exclude from the build process +# model + +ohana.base:base-1-5:0:EXTERN + +libohana:libohana-1-9:0:EXTERN +libfits:libfits-1-7:0:EXTERN +libautocode:libautocode-1-6:0:EXTERN +libdvo:libdvo-1-4:0:EXTERN +libkapa:libkapa-1-3:0:EXTERN + +elixir:elixir-1-6:0:EXTERN +gcompare:gcompare-1-1:0:EXTERN + +addstar:addstar-1-8:0:EXTERN +delstar:delstar-1-7:0:EXTERN +getstar:getstar-1-3:0:EXTERN +kapa:kapa-1-7:0:EXTERN +kii:kii-1-7:0:EXTERN +relphot:relphot-1-5:0:EXTERN +uniphot:uniphot-1-5:0:EXTERN + +opihi.base:opihi-2-9:0:EXTERN +mana:mana-1-7:0:EXTERN +dvo:dvo-1-0:0:EXTERN +pantasks:pantasks-1-0:0:EXTERN +pcontrol:pcontrol-1-0:0:EXTERN +pclient:pclient-1-0:0:EXTERN + +# psLib:rel-0-0:0:EXTERN +# psModules:rel-0-0:0:EXTERN + +psLib:HEAD::EXTERN +psModules:HEAD::EXTERN + +psphot:rel-0-8:0 +psastro:rel-0-8:0 +ppStats:rel-1-0:0 +ppImage:rel-1-0:0 +ppNorm:rel-1-0:0 +ppMerge:rel-1-0:0 + +pois:rel-0-1:0:SKIP +pswarp:rel-0-1:0:SKIP +ppStac:rel-0-1:0:SKIP + +# these are the required perl modules from IPP +# there are externally required perl modules (see README?) +PS-IPP-Metadata-Config:rel-1-0:0 +PS-IPP-Config:rel-1-0:0 +ippScripts:rel-1-0:0 + +glueforge:rel-1:0 +dbconfig:rel-1-0:0 + +ippdb.src:NONE + +ippTools:rel-1-0:0 +ippTasks:rel-1-0:0 +simtest:rel-0-8:0 + +psconfig:rel-1-0:0:SKIP +ippMonitor:rel-1-0:0:SKIP Index: /branches/rel-1-0/psconfig/tagsets/ippHead =================================================================== --- /branches/rel-1-0/psconfig/tagsets/ippHead (revision 11645) +++ /branches/rel-1-0/psconfig/tagsets/ippHead (revision 11645) @@ -0,0 +1,21 @@ +../mhpcc/psLib:HEAD +../mhpcc/psModule:HEAD + +psphot:HEAD +psastro:HEAD +ppStats:HEAD +ppImage:HEAD +ppNorm:HEAD +ppMerge:HEAD + +# these are the required perl modules from IPP +# there are externally required perl modules (see README?) +PS-IPP-Metadata-Config:HEAD +PS-IPP-Metadata:HEAD +ippScripts:HEAD + +glueforge:HEAD +dbconfig:HEAD +ippdb.src:NONE +ippTools:HEAD +ippTasks:HEAD