Index: /branches/ccl_branches/ipponly-20191108/ippTools/COPYING =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/COPYING (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/COPYING (revision 41104) @@ -0,0 +1,345 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. Index: /branches/ccl_branches/ipponly-20191108/ippTools/Makefile.am =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/Makefile.am (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/Makefile.am (revision 41104) @@ -0,0 +1,12 @@ +SUBDIRS = src share scripts + +pkgconfigdir = $(libdir)/pkgconfig +pkgconfig_DATA= pxtools.pc + +EXTRA_DIST = \ + pxtools.pc.in \ + autogen.sh + +CLEANFILES = *~ core core.* + +ACLOCAL_AMFLAGS = -I m4 Index: /branches/ccl_branches/ipponly-20191108/ippTools/TODO =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/TODO (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/TODO (revision 41104) @@ -0,0 +1,14 @@ +- change dettol -pretend -define... to return a metadata structure that shows what the created detRun would look like +- rename dettool -residdetrun -> -todetrunsummary +- add foreign key constraints to the database +- add the ability to remove dettool detResidExps +- change p4tool to accept input from/with magic masks +- change p5tool to accept input from warp * stack +- difftool/stacktool error handling +- move all large SQL statements out into separate files +- add the option of not automatically flowing through from reg -> chip -> cam + tools +- add workdir inheritance rooted in rawExp, eg. + select chipProcessedExp.workdir, rawExp.workdir, IFNULL(chipProcessedExp.workdir, rawExp.workdir) from chipProcessedExp join rawExp using(exp_tag); +- dettool/detRun support for filelevel +- combind dettool -definebytag & -definebyquery Index: /branches/ccl_branches/ipponly-20191108/ippTools/autogen.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/autogen.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/autogen.sh (revision 41104) @@ -0,0 +1,103 @@ +#!/bin/sh +# Run this to generate all the initial makefiles, etc. + +srcdir=`dirname $0` +test -z "$srcdir" && srcdir=. + +ORIGDIR=`pwd` +cd $srcdir + +PROJECT=pxtools +TEST_TYPE=-f +# change this to be a unique filename in the top level dir +FILE=autogen.sh + +DIE=0 + +LIBTOOLIZE=libtoolize +ACLOCAL="aclocal $ACLOCAL_FLAGS" +AUTOHEADER=autoheader +AUTOMAKE=automake +AUTOCONF=autoconf + +#($LIBTOOLIZE --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $LIBTOOLIZE installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/libtool/" +# DIE=1 +#} + +($ACLOCAL --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $ACLOCAL installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +#($AUTOHEADER --version) < /dev/null > /dev/null 2>&1 || { +# echo +# echo "You must have $AUTOHEADER installed to compile $PROJECT." +# echo "Download the appropriate package for your distribution," +# echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" +# DIE=1 +#} + +($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOMAKE installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/automake/" + DIE=1 +} + +($AUTOCONF --version) < /dev/null > /dev/null 2>&1 || { + echo + echo "You must have $AUTOCONF installed to compile $PROJECT." + echo "Download the appropriate package for your distribution," + echo "or get the source tarball at http://ftp.gnu.org/gnu/autoconf/" + DIE=1 +} + +if test "$DIE" -eq 1; then + exit 1 +fi + +test $TEST_TYPE $FILE || { + echo "You must run this script in the top-level $PROJECT directory" + exit 1 +} + +if test -z "$*"; then + echo "I am going to run ./configure with no arguments - if you wish " + echo "to pass any to it, please specify them on the $0 command line." +fi + +$LIBTOOLIZE --copy --force || echo "$LIBTOOLIZE failed" +$ACLOCAL -I m4 || echo "$ACLOCAL failed" +$AUTOHEADER || echo "$AUTOHEADER failed" +$AUTOMAKE --add-missing --force-missing --copy || echo "$AUTOMAKE failed" +$AUTOCONF || echo "$AUTOCONF failed" + +cd $ORIGDIR + +run_configure=true +for arg in $*; do + case $arg in + --no-configure) + run_configure=false + ;; + *) + ;; + esac +done + +if $run_configure; then + $srcdir/configure --enable-maintainer-mode "$@" + echo + echo "Now type 'make' to compile $PROJECT." +else + echo + echo "Now run 'configure' and 'make' to compile $PROJECT." +fi Index: /branches/ccl_branches/ipponly-20191108/ippTools/configure.ac =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/configure.ac (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/configure.ac (revision 41104) @@ -0,0 +1,71 @@ +AC_PREREQ(2.61) + +AC_INIT([ipptools], [1.1.77], [ipp-support@ifa.hawaii.edu]) +AC_CONFIG_SRCDIR([autogen.sh]) + +AM_INIT_AUTOMAKE([1.6 foreign dist-bzip2]) +AM_CONFIG_HEADER([src/config.h]) +AM_MAINTAINER_MODE + +IPP_STDLDFLAGS + +AC_LANG(C) +AC_GNU_SOURCE +AC_PROG_CC_C99 +AC_PROG_INSTALL +AC_PROG_LIBTOOL + +PKG_CHECK_MODULES([PSLIB], [pslib >= 1.1.0]) +PKG_CHECK_MODULES([PSMODULES], [psmodules >= 1.1.0]) +PKG_CHECK_MODULES([IPPDB], [ippdb >= 1.1.77]) +PKG_CHECK_MODULES([PPSTAMP], [ppstamp >= 0.1.1]) + +PXTOOLS_CFLAGS="${PSLIB_CFLAGS=} ${PSMODULES_CFLAGS=} ${PPSTAMP_CFLAGS=} ${IPPDB_CFLAGS=}" +PXTOOLS_LIBS="${PSLIB_LIBS=} ${PSMODULES_LIBS=} ${IPPDB_LIBS=}" +AC_SUBST(PXTOOLS_CFLAGS,[$PXTOOLS_CFLAGS]) +AC_SUBST(PXTOOLS_LIBS,[$PXTOOLS_LIBS]) + +dnl check for PSDB support +TMP_CFLAGS=${CFLAGS} +CFLAGS="${CFLAGS=} ${PSLIB_CFLAGS}" + +AC_COMPILE_IFELSE([AC_LANG_PROGRAM( + [[#if !defined HAVE_PSDB + error: PSDB support is not avaiable + #endif + ]])], + [], + [AC_MSG_ERROR([psLib was built without PSDB support, bailing out.])] +) + +CFLAGS=${TMP_CFLAGS} + +dnl -- check for psParseErrorCodes -------------------------------------- + +AC_PATH_PROG([ERRORCODES], [psParseErrorCodes], [missing]) +if test "$ERRORCODES" = "missing" ; then + AC_MSG_ERROR([psParseErrorCodes is required]) +fi + +dnl check for perl +AC_PATH_PROG([PERL], [perl], [missing]) +if test "$PERL" = "missing" ; then + AC_MSG_ERROR([perl is required]) +fi + +AC_PROG_SED + +IPP_STDOPTS +IPP_STDCFLAGS + +IPPDB_VERSION=`$PKG_CONFIG --modversion ippdb` +AC_DEFINE_UNQUOTED(IPPDB_VERSION, ["$IPPDB_VERSION"], [Version of ippdb]) + +AC_CONFIG_FILES([ + Makefile + src/Makefile + share/Makefile + scripts/Makefile + pxtools.pc +]) +AC_OUTPUT Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/addstar_flow.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/addstar_flow.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/addstar_flow.txt (revision 41104) @@ -0,0 +1,34 @@ +### Set up the addRun row during the previous stage +#>> camtool.c @ addprocessedexpMode(pxConfig *config) +camtool -cam_id $cam_id -addprocessedexp ... + +### Catch that things now need to be added. This should check for magic_status as well, probably. +#>> add.pro / addtool.c / addtoolConfig.c +addtool -pendingsmf +#addMe METADATA +# add_id S64 5 +#END + +### Run the addstar script. +#>> addstar_run.pl +addstar_run.pl --add_id $add_id --camera $camera --camroot=$camroot --dvodb $DVODB + addtool -pendingsmf -add_id $add_id +#addMe METADATA +# add_id S64 5 +# workdir STR /path/to/workdir +# +#END + + ppConfigDump -camera $camera -recipe ADDSTAR $recipt_addstar -dump-recipe ADDSTAR - + addstar -D CAMERA $camdir -update -D CATDIR $dvodbReal $realFile + addtool -add_id $add_id -addprocessedsmf ... + addtool -add_id $add_id -updaterun -state full + +### Reverting +#>> addtool.c +addtool -revertprocessedsmf ... + + + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/dettool.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/dettool.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/dettool.txt (revision 41104) @@ -0,0 +1,181 @@ +# returns a list of unassociated rawDetrendExp (rawDetrendExps that don't +# correspond to a exp_id in detInputExp) +dettool -pending -exp_type -inst .. + +# define a new detRun semi-automatically by selecting the component exps with +# the specified search criteria +dettool -definebyquery -det_type ... + +# create a detRun with the specified input exp_ids +dettool -definebyexp -det_type -exp_id 1 -exp_id 2 ... + +-- +# returns a list of rawImfiles that are associated with an detInputExp that do +# not appear in detProcessedImfiles +dettool -raw + +# add an imfile to detProcessedImfiles +dettool -addprocessed + +-- +# the list of class_ids that are ready to be stacked +# returns a list of det_id/iter/class_id/det_typs where class_id is unique per +# det_id (not a list of all exp_ids/class_ids) that do not have an entry in +# detStackedImfile +# results are masked by entries in detResidExpAnalysis +# may need to check detResidExp's even on iter 0 - ??? +dettool -tostack + +# list all of imfiles from different exps with the same class_id that have +# been processed +dettool -processed -wholestack -det_id ... + +# adds a stacked class_id and and optionally marks it for normalization +detool -addstacked [-pleasenormalize] -det_id -iter -class_id ... + +-- +# returns a list of det_ids/iters where the entire set of class_ids has been +# processed +dettool -tonormalize + +# returns the complete list of processed imfiles for the specified det_id +dettool -processed -unmask -det_id -iter + +# add per stacked imfile normalization statistics +# O: this could accept a MDC doc as input from stdin +dettool -addnormstat -det_id -iter -class_idi -norm F32 + +-- +# returns a list of det_ids/iter/class_id/uri from detNormStats. Stops +# returning entries after the class_is is inserted into detNormalizedImfile +dettool -normstat -det_id + +# inserts an entry into detNormalizedImfile +dettool -addnormalized -det_id -ter -class_id -uri + +-- +# returns a list of processed imfiles that have also been normalized with +# processed imfiles being masked out per detResidImfileAnalysis +# also returns stackedimfiles that have -pleasenormalize set to false +dettool -toresid -det_id -iter + +# inserts a per det_id/iter/exp_id/class_id residual +dettool -addresidimfile -det_id -exp_id -class_id -iter -stat F32 -stat_stdev F32 -uri -b1 -b2 + +# returns a list of det_id/iter/exp_ids but only for exp_ids that have resids +# for all of their imfiles +dettool -toresidexp + +# returns a list of residual imfile data +dettool -residimfile -det_id -iter -exp_id + +# inserts into detResidExpAnalysis +# sets the accept accept bool unless -reject is specified +dettool -addresidexp -det_id -iter -exp_id -jpeg1 -jpeg2 -stat F32 -stat_stdev +F32 [-reject] + +-- +# lists det_id/iter for detruns that have completed all residexps for their +# current iteration +dettool -residdetrun + +# lists all residexps for the given det_id/iter +dettool -residexp -det_id -iter + +# updates the given residexp to be accept/reject +dettool -updateresid -det_id -iter [-reject] + +# changes the state of the detrun. -rerun cause a new iteration to be started +# with the accepted with just the exp_ids +dettool -updatedetrun -det_id -iter -stat F32 -stat_stdev [-rerun|-ok|-stop] + +# manual starts a new detrun iteration starting from the specified -iter +dettool -rerun -det_id -iter -accept expid -accept expid -reject expid -reject expid +dettool -rerun -det_id -iter + +# detInputExp needs to track... +det_id iter exp_id use bool accept bool + + + +dettool + + -define | creates a a new detrend Run + input: + exp IDs? + type of det run? + creates a new detRun ID automatically + + output: the new det_id + + -raw | lists raw imfiles needing to be processed + input: + det ID + exp ID + class ID + iteration number? + output: list of unprocessed imfiles + + -addprocessed? | marks a raw imfile as processed + input: + det ID + exp ID + class ID + iteration number? + uri + stat + recipe + output: outout on error only + -processed | lists processed imfiles + input: + det ID + exp ID + class ID + iteration number? + output: list of processed imfiles + -addstacked | adds a stacked imfiles + input: + det ID + class ID + iteration number? + URI + recipe + stats + output: output on error only + -stacked | lists stacked imfiles + input: + det ID + class ID + iteration number? + output: list of stacked imfiles + -stacked | lists stacked imfiles + input: + det ID + class ID + iteration number? + output: list of stacked imfiles + -stackedframe | lists stacked imfiles for COMPLETE frames + input: + det ID // required for simplicity + iteration number? + output: list of stacked imfiles + -addmaster | adds a master imfiles + input: + det ID + class ID + iteration number? + URI + recipe + stats + output: output on error only + -master | lists master imfiles + input: + det ID // required for simplicity + class ID + iteration number? + output: a list of master imfiles + -masterframe | lists master imfiles for COMPLETE frames + input: + det ID // required for simplicity + iteration number? + output: a list of master imfiles Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/dsfilesetls =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/dsfilesetls (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/dsfilesetls (revision 41104) @@ -0,0 +1,6 @@ +#!/usr/bin/env perl + +print "# uri fileid bytes md5sum type \n"; +print "http://example.org/1 foo1.0 123 asadfasdfasdfasfdasdf CELL\n"; +print "http://example.org/2 foo1.1 123 asadfasdfasdfasdfasdf CELL\n"; +print "http://example.org/3 foo1.2 123 asadfasdfasdfasdfasdf CELL\n"; Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/dsproductls =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/dsproductls (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/dsproductls (revision 41104) @@ -0,0 +1,8 @@ +#!/usr/bin/env perl + +print "# uri fileset datetime type\n"; +print "http://example.org/1 foo1 2006-04-17T00:00:12 OBJECT\n"; +print "http://example.org/2 foo2 2006-04-17T00:00:12 OBJECT\n"; +print "http://example.org/3 foo3 2006-04-17T00:00:12 OBJECT\n"; + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/install.pod =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/install.pod (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/install.pod (revision 41104) @@ -0,0 +1,251 @@ +=pod + +=head1 SYNOPSIS + +C is a collection of utilties for tracking image processing +tasks. Very little decision making logic is built into these programs. The +intent is for higher level programs and scripts to make the real I +and then use C to encode those choices in a persistent manner. + +=head1 INSTALL + +=head2 Prerequisites + +C depends on C (built with C support), C, and +C and requires a C capable compiler. + +XXX document required library versions + +=head2 Build Procedure + +C is built with the so-called suite of autotools. The build +procedure is pretty typical for software built with these tools. E.g. + + ./configure + make + make install + +If you checked out the sources directly from CVS you need to run the +C script. C will run C for you so that +changes the build procedure to just: + + sh autogen.sh + make + make install + +=head2 jhbuild Build Procedure + +If you are using one of the C modulesets jhbuild can build and install C for you. E.g. + + jhbuild build pxtools + +=head2 Database Setup + +All of the C' persistent storage is ultimately done through C's +C facilties. In turn, C uses C to impliment storage (and +relational querying). C requires that each namespace (aka database) is +configured with basic access controls. These include usernames, passwords, +access filtering, and privilege restrictions. Below is a simple (but somewhat +dangerous) example of configuring a database named C. + + mysql -u root mysql -p + (enter your MySQL root password) + + DROP DATABASE IF EXISTS ipp; + CREATE DATABASE ipp; + GRANT ALL PRIVILEGES ON *.* TO 'ippuser'@'localhost' IDENTIFIED BY 'ipppass'; + FLUSH PRIVILEGES; + quit + +After setting up the database you should verify that you can connect to with the username and password you configured + + mysql -u ipp ipp -pipp + +Next, C needs to be told where to to find MySQL, the name of the +database you created, the username, and the password you created. C +uses the typical C configuration system. The simplest method is to +add the follow lines to your $HOME/.ipprc file. + + ### Database configuration + DBSERVER STR localhost # Database host name (for psDBInit) + DBNAME STR ipp + DBUSER STR ipp # Database user name (for psDBInit) + DBPASSWORD STR ipp # Database password (for psDBInit) + +The C utilities require the a schema is pre-loaded into the database. The C tool can do this for you. E.g. + + pxadmin -create + +You should always verify that the scheme was actually loaded. E.g. + + mysql -u ipp ipp -pipp + Welcome to the MySQL monitor. Commands end with ; or \g. + Your MySQL connection id is 6 to server version: 4.1.20-log + + Type 'help;' or '\h' for help. Type '\c' to clear the buffer. + + mysql> show tables; + +------------------------+ + | Tables_in_ipp | + +------------------------+ + | detInputExp | + | detMasterFrame | + | detMasterImfile | + | detNormalizedImfile | + | detProcessedImfile | + | detResidExpAnalysis | + | detResidImfileAnalysis | + | detRun | + | detStackedImfile | + | newExp | + | newImfile | + | p1PendingExp | + | p2DoneExp | + | p2DoneImfile | + | p2PendingExp | + | p2PendingImfile | + | p3PendingExp | + | pzPendingExp | + | pzPendingImfile | + | rawDetrendExp | + | rawImfile | + | rawScienceExp | + | summitExp | + +------------------------+ + 23 rows in set (0.00 sec) + +=head1 USE + +The workflow is as follows: + +XXX + +=head2 Phase 0 tools + +The primary C utility for handling Phase 0 is C. This tool +has two modes C<-pending> and C<-update>. The pending mode lists all +Cs that are part of an exposure that I yet been classified +as either science or detrend data. + +For example: + + p0search -pending + +Would output something like: + + newImfile MULTI # + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 0 + uri STR file://0 + END + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 1 + uri STR file://1 + END + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 2 + uri STR file://2 + END + + newImfile METADATA + exp_id STR 10 + class STR OTA + class_id STR 3 + uri STR file://3 + END + +To declare C as science data, you would invoke C like this: + + p0search -update -exp_id 10 + +Note that by default C<-update> flags the exposure as science data. To declare it to be detrend data the command would be: + + p0search -update -exp_id 10 -detrend + +Also keep in mind that this is a one time declaration. Once it has been +declared as either science or detrend data this decision can not be undone +(without manually editing the database) + +=head2 Phase 1 tools + +These tools are not yet in use. + +=head2 Phase 2 tools + +C handles all of state for Phase 2. It has four basic modes: + +=over 4 + +=item * C<-quick> + +=item * C<-define> + +=item * C<-pending> + + p0search -pending + +=item * C<-done> + +=head2 Administrative tools + +=over 4 + +=item * C + +This tool can be used to load/unload the database schema. It has three modes of operation: + +=over 4 + +=item * C<-create> + +Loads the database schema. + +=item * C<-delete> + +Deletes the database schema. + +=item * C<-recreate> + +Deletes the database schema then reloads it. + +=back + +=item * C + +I data directly into the database. It has one mode parameter for each database table. Note: support for only a few tables has been impliemented. + +=back + +=head2 Bypassing 'Summit Copy' + +The C suite assumes that exposures and image data are initially +registered with the C set of I tools. Sometimes this is +inconvenient as you want to work with local data and bypass this set. This is +what the C utility is for. C directly inserts data into +the database, side stepping any ordering rules or sanity checking. + +The end result of the C utilities is that new exposures are registered in the C table and raw image data is in the C table. You can insert this data yourself, preparing the system to run I. + +Here is an example of inserting an exposure that has 4 component image files. + + #!//bin/sh + + inject="./pxinject" + + $inject -newExp -exp_id 10 -inst gpc -telescope ps1 -exp_type object -imfiles 4 + + `$inject -newImfile -exp_id 10 -class OTA -class_id 0 -uri file://0` + `$inject -newImfile -exp_id 10 -class OTA -class_id 1 -uri file://1` + `$inject -newImfile -exp_id 10 -class OTA -class_id 2 -uri file://2` + `$inject -newImfile -exp_id 10 -class OTA -class_id 3 -uri file://3` + +=cut Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/magic_flow.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/magic_flow.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/magic_flow.txt (revision 41104) @@ -0,0 +1,65 @@ +### Set up a run: +magictool -definerun -workdir /path/to/workdir +magictool -addinputskyfile -magic_id 6 -diff_id 1 -node 5.1 +magictool -addinputskyfile -magic_id 6 -diff_id 2 -node 5.2 +magictool -addinputskyfile -magic_id 6 -diff_id 3 -node 5.3 +magictool -addinputskyfile -magic_id 6 -diff_id 4 -node 5.4 +magictool -updaterun -magic_id 6 -state run + +### Set up a run in a single call: +magictool -queue -select_XXX + +### Generate a processing tree +magictool -totree +#magic_tree.pl --magic_id X [camera, tess_id, ra0, dec0] +* magictool -inputskyfile -magic_id 6 +* magictool -inputtree -magic_id 6 -dep_file magic_dep.mdc + +### Process the tree +magictool -toprocess +#magicMe METADATA +# magic_id S64 5 +# node STR 5.1 +# dep STR NULL +# done S64 0 +# uri STR NULL +#END +#magic_process.pl --magic_id 6 --node 5.1 +magictool -inputs -magic_id 6 -node 5.1 + +magictool -addresult -magic_id 6 -node 5.1 -uri foo +magictool -addresult -magic_id 6 -node 5.2 -uri bar +magictool -addresult -magic_id 6 -node 5.3 -uri baz +magictool -addresult -magic_id 6 -node 5.4 -uri qix + +### Merging the tree +magictool -toprocess +#magicMe METADATA +# magic_id S64 5 +# node STR 5A +# uri MULTI +# uri STR foo +# uri STR bar +#END +#magic_process.pl --magic_id 6 --node 5A +magictool -inputs -magic_id 6 -node 5A + +magictool -addresult -magic_id 6 -node 5A -uri foobar +magictool -addresult -magic_id 6 -node 5B -uri bazqix + +### Final merge +magictool -toprocess +#magicMe METADATA +# magic_id S64 5 +# node STR root +#END +#magic_process.pl --magic_id 6 --node root +magictool -inputs -magic_id 6 -node root + +magictool -addresult -magic_id 6 -node root -uri foobarbazqix + +### Generate the mask description +magictool -tomask +magic_mask.pl --magic_id X +magictool -inputs -magic_id 6 -node root +magictool -addmask -magic_id 6 -uri foobarbazqixqit Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/p0tools.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/p0tools.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/p0tools.txt (revision 41104) @@ -0,0 +1,40 @@ + +Phase 0 pipeline tools: + +p0search -pending : + * examine the new.imfiles,new.exposures tables and select exposures ready for analysis + * output is: (expID) (camera) + +p0search -update (expID) (camera): + * select a the corresponding images from the new.imfiles/new.exposures table + * extract the specified header information + * search the summit metadata db tables + * write an entry to the raw.imfiles and raw.exposure tables + * set the state on the new.imfiles,new.exposure tables + * based on the camera config information; + * add an entry to the p1.pending table (mosaic) + - or + * add an entry to the p2.pending table (single) + +p0search -stats (expID) (camera): + * select a specified image in the new.imfiles/new.exposures table + * extract the specified header information + * search the summit metadata db tables + * report the image stats + [-update without output to MDDB] + +p0search -mkraw (expID) (camera): + * select a specified image in the new.imfiles/new.exposures table + * extract the specified header information + * search the summit metadata db tables + * write an entry to the raw.imfiles and raw.exposure tables + * set the state on the new.imfiles,new.exposure tables + +p0search -cleanup: + * remove completed entries from the new.imfiles,new.exposure tables + +** note : the division of -pending and -update allows separate processes + to be examining the image headers and measuring some stats. these + jobs can be run via pcontrol to reduce the load on the PanTasks + machines + Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/p1tools.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/p1tools.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/p1tools.txt (revision 41104) @@ -0,0 +1,16 @@ + +Phase 1 pipeline tools: + +p1search -define [constraints]: + * examine the raw.exposures tables and select exposures matching the given criteria + * add entries which are allowed (mosaic) to the p1.pending table + +p1search -pending : + * examine the p1.pending table and select exposures waiting for p1 + * output: lines consisting of: + (expID) (p1version) (camera) + +p1search -done : + * select completed entries in the p1.pending table + * move to the p1.done table + * add new entry to the p2.pending tables Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/p2search.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/p2search.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/p2search.sh (revision 41104) @@ -0,0 +1,8 @@ +#!/bin/csh -f + +echo "# expID class classID url" +echo "654320o fpa megacam 654320o.fits" +echo "654321o fpa megacam 654321o.fits" +echo "654322o fpa megacam 654322o.fits" +echo "654323o fpa megacam 654323o.fits" +echo "654324o fpa megacam 654324o.fits" Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/p2tools.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/p2tools.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/p2tools.txt (revision 41104) @@ -0,0 +1,41 @@ + +Phase 2 pipeline tools: + +p2search -quick + * search for images which match in raw.exp,raw.imfiles + * output in format which can be used by ppImage pantasks script + +p2search -define [options] + * input: searches mddb:raw_exposures,raw_images + - compares against p2Pending[Exp|Imfile] + - compares against p2Done[Exp|Imfile] + - XXX should be implemented as a special wrapper function that + implements this comparison as a SQL query + * output: updates mddb:P2_exposures_pending,P2_images_pending + * alternative output: identical to p2pending + +p2search -pending + * input: searches mddb:P2_exposures_pending,P2_images_pending + * output: Nlines consisting of: + (URL) (expID) (class) + * options: ? + +p2search -update + * examine the imfiles and identify any completed exposures + +p2search -done + * get exp_id/class/class_id from the CLI + * add the completed imfile to the p2DoneImfile tables + * remove corresponding entries from the p2PendingImfile table + * check to see if any p2PendingExps have no associated p2PendingImfiles + if so move the p2PendingExp(s) to p2DoneExp +// * send new entry to the pending p3 table + +ppImage file://path/filename file://path/outroot -recipe (recipe) +ppImage neb://nebname neb://outroot -recipe (recipe) + +restriction options: + -time (start) (stop) + -camera (camera) + -region (ra,dec) (ra,dec) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/p3tools.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/p3tools.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/p3tools.txt (revision 41104) @@ -0,0 +1,22 @@ + +Phase 3 pipeline tools: + +p3search -define : + * examine the raw.exposures tables and select exposures matching the given criteria + * add entries which are allowed (mosaic) to the p3.pending table + +p3search -quick : + * examine the raw.exposures tables and select exposures matching the given criteria + * return list of entries for p3 processing + * output: lines consisting of: + (expID) (p3version) (camera) + +p3search -pending : + * examine the p3.pending exposures table and select exposures waiting for p3 + * return list of entries for p3 processing + * output: lines consisting of: + (expID) (p3version) (camera) + +p3search -done : + * select completed entries in the p3.pending table + * move to the p3.done table Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/publish_flow.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/publish_flow.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/publish_flow.txt (revision 41104) @@ -0,0 +1,14 @@ +### Setup +pubtool -defineclient -stage diff -comment Comment -product IPP-MOPS -workdir /path/to/somewhere + +### Regularly run by pantasks to generate publishing runs based on data that has been processed +pubtool -definerun + +### Regularly run by pantasks to get list of files to publish +pubtool -pending +--> publish_file.pl --pub_id 12345 --workdir /path/to/somewhere --product IPP-MOPS --stage diff --stage_id 67890 +----> pubtool -add -pub_id 12345 -path_base /path/to/somewherenice -fault 0 + +### Run as required +pubtool -revert + Index: /branches/ccl_branches/ipponly-20191108/ippTools/doc/summit_copy.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/doc/summit_copy.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/doc/summit_copy.txt (revision 41104) @@ -0,0 +1,49 @@ +The summit copy phase (phase Z or pz) consists of three basic steps. +Discovering what exposures and available, discovering what image files are +associated with those exposures, and the downloading of image files. All +"external" communication with the DataStore is done with three utilities; +pzgetexp, pzgetimfiles & dsget. While local state information is accessed and +updated via the 'pztool' utility. + +The basic work follow is: + +pzgetexp retrieves a list of all of the exposures (filesets) available from a +DataStore "product". If any previously unknown file exposures are found they +are added to the pzPendingExp tables and the summitExp table. New entries to +the summitExp table have a imfile value of NULL as this information can't yet +be known. If any exposures are already in the summitExp table the most recent +exp_id is used to constrain the result set requested from the DataStore. + + ./pzgetexp -uri http://otis/ds/skyprobe/ -inst skyprobe -telescope ps1 + +The results of pzgetexp and accessible with pztool -pendingexp`. Which will +return a list of new exposures from the pzPendingExp table. + + ./pztool -pendingexp + +The output of which is used to invoke pzgetimfiles. + +pzgetimfiles retrieves a list of all of the imfiles (files) in an exposure +(fileset) from the DataStore. It then updates the imfiles count in the +summitExp table and adds each imfile to the pzPendingImfile table. + +./pzgetimfiles -uri http://otis/ds/skyprobe/sep8_twi_05/ -filesetid sep8_twi_05 -inst skyprobe -telescope ps1 + +pztool -pendingimfile returns the list of imfiles that need to be downloaded. + + ./pztool -pendingimfile + +The results of which are used to invoke dsget and pztool -copydone after dsget +completes it's download. + +dsget from the 'DataStore' package takes the follow parameters. + + dsget --uri --filename [--bytes ] [--md5 ] + +Where "filename" may be either just a filename relative to the CWD or a fully +qualified path. After dsget has copied the file pztool -copydone needs to be +invoked + + ./pztool -copydone -exp_id sep8_twi_03 -inst skyprobe -telescope ps1 -class chip -class_id sep8_twi_03.fits -uri file///data/alala/sep8_twi_03.fits + +And then we're Pau. Index: /branches/ccl_branches/ipponly-20191108/ippTools/m4/ac_prog_perl_modules.m4 =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/m4/ac_prog_perl_modules.m4 (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/m4/ac_prog_perl_modules.m4 (revision 41104) @@ -0,0 +1,53 @@ +dnl @synopsis AC_PROG_PERL_MODULES([MODULES], [ACTION-IF-TRUE], [ACTION-IF-FALSE]) +dnl +dnl Checks to see if the the given perl modules are available. If true +dnl the shell commands in ACTION-IF-TRUE are executed. If not the shell +dnl commands in ACTION-IF-FALSE are run. Note if $PERL is not set (for +dnl example by calling AC_CHECK_PROG, or AC_PATH_PROG), +dnl AC_CHECK_PROG(PERL, perl, perl) will be run. +dnl +dnl Example: +dnl +dnl AC_CHECK_PERL_MODULES(Text::Wrap Net::LDAP, , +dnl AC_MSG_WARN(Need some Perl modules) +dnl +dnl @category InstalledPackages +dnl @author Dean Povey +dnl @version 2002-09-25 +dnl @license AllPermissive + +AC_DEFUN([AC_PROG_PERL_MODULES],[ + ac_perl_modules="$1" + # Make sure we have perl + if test -z "$PERL"; then + AC_CHECK_PROG(PERL,perl,perl) + fi + + if test "x$PERL" != x; then + ac_perl_modules_failed=0 + for ac_perl_module in $ac_perl_modules; do + AC_MSG_CHECKING(for perl module $ac_perl_module) + + # Would be nice to log result here, but can't rely on autoconf + # internals + $PERL "-M$ac_perl_module" -e exit > /dev/null 2>&1 + if test $? -ne 0; then + AC_MSG_RESULT(no); + ac_perl_modules_failed=1 + else + AC_MSG_RESULT(ok); + fi + done + + # Run optional shell commands + if test "$ac_perl_modules_failed" = 0; then + : + $2 + else + : + $3 + fi + else + AC_MSG_WARN(could not find perl) + fi +]) Index: /branches/ccl_branches/ipponly-20191108/ippTools/notes.txt =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/notes.txt (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/notes.txt (revision 41104) @@ -0,0 +1,75 @@ + +2010.05.24 EAM + + staticskytool: + + definebyquery : choose input stacks that meet a list of + restrictions. Require a specific list of filters? + Require a minimum number of inputs? + + eg: sst definebyquery -filterlist g,r,i,z,y [-label md04%] [-ra_min x] [-dec_min x] ... + + can we do the filter selects in the database? + +2008.07.19 EAM + + * need to add cleanup modes to all stages. here is the minimum list + of stages that need the cleanups: + + * chiptool - DONE + * camtool + * faketool + * warptool + * difftool + * stacktool + * dettool processedimfile + * dettool processedexp + * dettool stackedimfile + * dettool normalizedstat + * dettool normalizedimfile + * dettool normalizedexp + * dettool residimfile + * dettool residexp + + * I am modifiying the tools to use the new states: new, full, goto_cleaned, etc. + I am NOT changing: regtool, pztool, magictool, flatcorr, pxregister, pstamptool + + * for dettool, I have added a new field, 'data_state' to the + detRunSummary. My plan here is that a complete detRunIteration + is cleaned at once. I've added functions to all of the dettool + major modes to list pending entries to be cleaned and a function + to update the data_state when cleaning is done. + +2008.07.11 EAM + + * break dettool.c into files for each detrend stage + * require -iteration in more places? (esp -add...) + * allow %s, %d arguments in the *.sql files or not? + * adddetrunsummary has the ability to set values for multiple detRunSummary entries. why? + +2008.07.09 EAM + + In order to implement the 'cleanup' and 'update' strategies, we have + changed the state names and defined addtional states as follows: + + * new : entry is unprocessed (was 'run') + * full : entry is processed and has all output data products (was 'done') + * goto_cleaned : entry should be processed by the 'cleanup' system + * cleaned : entry has been processed by the 'cleanup' system -- + some output data products are now missing. + + * goto_full : entry should be re-processed by the 'update' system + * full : entry has been re-processed by the 'update' system + + * goto_purged : entry should be processed by the 'cleanup' ssytem + to purge all output data + * purged : entry has been purged + + + +2008.05.16 EAM + +caltool + * add the active field, -active flag to -dbs + * add the -region + * change 'catdir' to 'dvo_id' Index: /branches/ccl_branches/ipponly-20191108/ippTools/pxtools.pc.in =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/pxtools.pc.in (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/pxtools.pc.in (revision 41104) @@ -0,0 +1,11 @@ +prefix=@prefix@ +exec_prefix=@exec_prefix@ +libdir=@libdir@ +includedir=@includedir@/@PACKAGE_NAME@ + +Name: @PACKAGE_NAME@ +Description: Pan-STARRS IPP Tools Library +Version: @VERSION@ +Requires: pslib psmodules ippdb +Libs: -L${libdir} -lpxtools +Cflags: -I${includedir} @PXTOOLS_CFLAGS@ Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/Makefile.am =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/Makefile.am (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/Makefile.am (revision 41104) @@ -0,0 +1,3 @@ + +bin_SCRIPTS = \ + ippadmin Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/camtest.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/camtest.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/camtest.sh (revision 41104) @@ -0,0 +1,11 @@ +#!//bin/sh + +set -o verbose + +chiptest.sh || exit 1 + +camtool -pendingexp || exit 1 +camtool -pendingimfile || exit 1 + +camtool -addprocessedexp -cam_id 1 -uri file:///cam -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -sigma_ra 1 -sigma_dec 2 -zp_mean 10 -zp_stdev 2 -n_stars 2 -n_extended 0 -n_astrom 42 -n_cr 10000000 -path_base file:///foo || exit 1 +camtool -addprocessedexp -cam_id 2 -uri file:///cam -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -sigma_ra 1 -sigma_dec 2 -zp_mean 10 -zp_stdev 2 -n_stars 2 -n_extended 0 -n_astrom 42 -n_cr 10000000 -path_base file:///foo || exit 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/chiptest.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/chiptest.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/chiptest.sh (revision 41104) @@ -0,0 +1,17 @@ +#!//bin/sh + +set -o verbose + +regtest.sh || exit 1 + +chiptool -pendingimfile || exit 1 + +for ID in `seq 0 3`; do + chiptool -addprocessedimfile -chip_id 1 -exp_id 1 -class_id $ID -uri file://chipp-t10.$ID -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -path_base file:///foo || exit 1 +done; + +for ID in `seq 0 3`; do + chiptool -addprocessedimfile -chip_id 2 -exp_id 2 -class_id $ID -uri file://chipp-t11.$ID -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -path_base file:///foo || exit 1 +done; + +chiptool -pendingimfile || exit 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/dettest.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/dettest.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/dettest.sh (revision 41104) @@ -0,0 +1,88 @@ +#!//bin/sh + +set -o verbose + +det_id=1 + +./regtest.sh -end_stage reg || exit 1 + +det_id=`dettool -definebyquery -det_type bias -inst gpc -filelevel fpa -select_exp_type bias -airmass_min 1 -airmass_max 10 -exp_time_min 10 -exp_time_max 30.0 -workdir file::///some/path -simple | cut -f1 -d" "` || exit 1 + +dettool -raw || exit 1 + +dettool -toprocessedimfile || exit 1 + +for ID in `seq 0 3` ; do + dettool -addprocessedimfile -det_id 1 -exp_id 1 -class_id $ID -uri file://proc-$ID -recip myrecip -bg 2 -bg_stdev 3 -bg_mean_stdev 4 || exit 1 +done; + +for ID in `seq 0 3` ; do + dettool -addprocessedimfile -det_id 1 -exp_id 2 -class_id $ID -uri file://proc-$ID -recip myrecip -bg 2 -bg_stdev 3 -bg_mean_stdev 4 || exit 1 +done; + + +dettool -tostacked || exit 1 + +for ID in `seq 0 3` ; do + dettool -addstacked -det_id $det_id -uri file://stacked-$ID -class_id $ID -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 || exit 1 +done; + +dettool -tonormalizedstat || exit 1 + +for ID in `seq 0 3` ; do + dettool -addnormalizedstat -det_id $det_id -class_id $ID -norm 0.12345 || exit 1 +done; + +dettool -tonormalize || exit 1 +for ID in `seq 0 3` ; do + dettool -addnormalizedimfile -det_id $det_id -class_id $ID -uri file://normalized-$ID -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -path_base banana1 || exit 1 +done; + +dettool -tonormalizedexp || exit 1 +dettool -addnormalizedexp -det_id $det_id -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -path_base file://normalizedexp || exit 1 + +dettool -toresidimfile || exit 1 + +for ID in `seq 0 3` ; do + dettool -addresidimfile -det_id $det_id -exp_id 1 -class_id $ID -recip myrecip -bg 1 -bg_stdev 1 -bg_mean_stdev 1 -uri file://resid-$ID || exit 1 +done; + +for ID in `seq 0 3` ; do + dettool -addresidimfile -det_id $det_id -exp_id 2 -class_id $ID -recip myrecip -bg 1 -bg_stdev 1 -bg_mean_stdev 1 -uri file://resid-$ID || exit 1 +done; + +dettool -toresidexp || exit 1 +dettool -addresidexp -det_id $det_id -exp_id 1 -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -path_base jpeg1 || exit 1 +dettool -addresidexp -det_id $det_id -exp_id 2 -recip myrecipe -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -path_base jpeg1 -reject || exit 1 + +dettool -todetrunsummary || exit 1 +dettool -residexp || exit 1 +dettool -updateresidexp -det_id $det_id -iteration 0 -recip yourrecipe || exit 1 +dettool -updateresidexp -det_id $det_id -iteration 0 -exp_id 2 -reject || exit 1 +dettool -updateresidexp -det_id $det_id -iteration 0 -exp_id 2 || exit 1 + +dettool -adddetrunsummary -det_id $det_id -iteration 0 -bg 1 -bg_stdev 2 -bg_mean_stdev 3 -accept || exit 1 + +dettool -todetrunsummary || exit 1 + +dettool -residexp || exit 1 + +dettool -updatedetrun -det_id $det_id -state stop || exit 1 + +dettool -register_detrend -det_type bias -mode master -filelevel fpa -workdir file:/// -label foo || exit 1 +dettool -register_detrend_imfile -det_id 2 -class_id 1 -uri file:///foo || exit 1 +dettool -register_detrend_imfile -det_id 2 -class_id 2 -uri file:///foo || exit 1 +dettool -register_detrend_imfile -det_id 2 -class_id 3 -uri file:///foo || exit 1 +dettool -register_detrend_imfile -det_id 2 -class_id 4 -uri file:///foo || exit 1 +dettool -updatedetrun -det_id 2 -state stop || exit 1 + +# correct test +dettool -makecorrection -det_id 1 || exit 1 +dettool -tocorrectexp || exit 1 +dettool -tocorrectimfile -det_id 3 || exit 1 +dettool -addcorrectimfile -det_id 3 -class_id 0 -uri file:///correct/0 || exit 1 +dettool -addcorrectimfile -det_id 3 -class_id 1 -uri file:///correct/1 || exit 1 +dettool -addcorrectimfile -det_id 3 -class_id 2 -uri file:///correct/2 || exit 1 +dettool -addcorrectimfile -det_id 3 -class_id 3 -uri file:///correct/3 || exit 1 +# detRun 3 should be automatically set to stop by this point +dettool -tocorrectexp || exit 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/difftest.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/difftest.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/difftest.sh (revision 41104) @@ -0,0 +1,29 @@ +#!//bin/sh + +set -o verbose + +warptest.sh || exit 1 + + +difftool -definerun -workdir file:///tmp/diff -skycell_id foo1 -tess_id bar || exit 1 +difftool -addinputskyfile -diff_id 1 -warp_id 1 -kind warped -template || exit 1 +difftool -addinputskyfile -diff_id 1 -warp_id 1 -kind warped || exit 1 +#difftool -updaterun -state run -diff_id 1 || exit 1 +difftool -todiffskyfile || exit 1 +difftool -inputskyfile || exit 1 +difftool -adddiffskyfile -diff_id 1 -uri file:///tmp/diff/skyfile -path_base file://lalaland -bg 1 -bg_stdev 2 || exit 1 +difftool -diffskyfile -diff_id 1 || exit 1 +difftool -updaterun -state stop -diff_id 1 || exit 1 + +# +# diff_id 2 +# +difftool -definerun -workdir file:///tmp/diff -skycell_id foo1 -tess_id bar || exit 1 +difftool -addinputskyfile -diff_id 2 -warp_id 1 -kind warped -template || exit 1 +difftool -addinputskyfile -diff_id 2 -warp_id 1 -kind warped || exit 1 +#difftool -updaterun -state run -diff_id 1 || exit 1 +difftool -todiffskyfile || exit 1 +difftool -inputskyfile || exit 1 +difftool -adddiffskyfile -diff_id 2 -uri file:///tmp/diff/skyfile -path_base file://lalaland -bg 1 -bg_stdev 2 || exit 1 +difftool -diffskyfile -diff_id 2 || exit 1 +difftool -updaterun -state stop -diff_id 2 || exit 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/faketest.sh =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/faketest.sh (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/faketest.sh (revision 41104) @@ -0,0 +1,19 @@ +#!//bin/sh + +set -o verbose + +camtest.sh || exit 1 + +#faketool -pendingexp || exit 1 +faketool -pendingimfile || exit 1 + +for ID in `seq 0 3`; do + faketool -addprocessedimfile -fake_id 1 -exp_id 1 -class_id $ID -uri file://chipp-t10.$ID -path_base file:///foo || exit 1 +done; + +for ID in `seq 0 3`; do + faketool -addprocessedimfile -fake_id 2 -exp_id 2 -class_id $ID -uri file://chipp-t11.$ID -path_base file:///foo || exit 1 +done; + +faketool -pendingimfile || exit 1 +faketool -processedimfile || exit 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/scripts/ippadmin =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/scripts/ippadmin (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/scripts/ippadmin (revision 41104) @@ -0,0 +1,106 @@ +#!/bin/csh -f +## this script creates the top-level tables needed to manage the IPP databases +set IPPADMIN = "ippadmin" + +if ($#argv < 3) goto usage; + +if ("$1" == "init") then + if ($#argv != 4) goto usage; + set dbserver = $2 + set dbuser = $3 + set dbpass = $4 + + mysql -h $dbserver -u root -p < pxadmin_create_mirror_tables.sql + +dist_pkgdata_DATA = \ + addtool_checkminidvodbaddrun.sql \ + addtool_donecleanup.sql \ + addtool_find_cam_id_dvo.sql \ + addtool_find_cam_id.sql \ + addtool_find_diff_id.sql \ + addtool_find_diff_id_dvo.sql \ + addtool_find_ff_id.sql \ + addtool_find_ff_id_dvo.sql \ + addtool_find_ffsummary_id.sql \ + addtool_find_ffsummary_id_dvo.sql \ + addtool_find_skycal_id_dvo.sql \ + addtool_find_skycal_id.sql \ + addtool_find_sky_id_multi_dvo.sql \ + addtool_find_sky_id_multi.sql \ + addtool_find_stack_id_dvo.sql \ + addtool_find_stack_id.sql \ + addtool_find_pendingexp.sql \ + addtool_find_pendingexp_cam.sql \ + addtool_find_pendingexp_diff.sql \ + addtool_find_pendingexp_ff.sql \ + addtool_find_pendingexp_ffsummary.sql \ + addtool_find_pendingexp_skycal.sql \ + addtool_find_pendingexp_stack.sql \ + addtool_find_pendingexp_staticsky_multi.sql \ + addtool_find_pendingmergeprocess.sql \ + addtool_find_processedexp_cam.sql \ + addtool_find_processedexp_diff.sql \ + addtool_find_processedexp_ff.sql \ + addtool_find_processedexp_ffsummary.sql \ + addtool_find_processedexp_skycal.sql \ + addtool_find_processedexp_stack.sql \ + addtool_find_processedexp_staticsky_multi.sql \ + addtool_find_minidvodbprocessed.sql \ + addtool_find_minidvodbrun.sql \ + addtool_pendingcleanupexp.sql \ + addtool_pendingcleanuprun.sql \ + addtool_queue_cam_id.sql \ + addtool_queue_diff_id.sql \ + addtool_queue_ff_id.sql \ + addtool_queue_ffsummary.sql \ + addtool_queue_stack_id.sql \ + addtool_queue_sky_id_multi.sql \ + addtool_queue_skycal_id.sql \ + addtool_queue_minidvodbrun.sql \ + addtool_revertminidvodbprocessed.sql \ + addtool_revertprocessedexp_cam.sql \ + addtool_revertprocessedexp_diff.sql \ + addtool_revertprocessedexp_ff.sql \ + addtool_revertprocessedexp_ffsummary.sql \ + addtool_revertprocessedexp_skycal.sql \ + addtool_revertprocessedexp_stack.sql \ + addtool_revertprocessedexp_staticsky.sql \ + addtool_revertprocessedexp_staticsky_multi.sql \ + bgtool_advancechip.sql \ + bgtool_advancewarp.sql \ + bgtool_change_chiprun_state.sql \ + bgtool_change_imfile_data_state.sql \ + bgtool_change_warprun_state.sql \ + bgtool_change_skyfile_data_state.sql \ + bgtool_chipinputs.sql \ + bgtool_chip.sql \ + bgtool_cleanedchip.sql \ + bgtool_cleanedwarp.sql \ + bgtool_definechip.sql \ + bgtool_definewarp.sql \ + bgtool_listchip.sql \ + bgtool_listwarp.sql \ + bgtool_pendingcleanupchiprun.sql \ + bgtool_pendingcleanupchipimfile.sql \ + bgtool_pendingcleanupwarprun.sql \ + bgtool_pendingcleanupwarpskyfile.sql \ + bgtool_revertchip.sql \ + bgtool_revertwarp.sql \ + bgtool_tochip.sql \ + bgtool_tocleanchip.sql \ + bgtool_tocleanwarp.sql \ + bgtool_towarp.sql \ + bgtool_warpinputs.sql \ + bgtool_warp.sql \ + camtool_addprocessedexp.sql \ + camtool_donecleanup.sql \ + camtool_find_chip_id.sql \ + camtool_pendingexp.sql \ + camtool_find_pendingimfile.sql \ + camtool_find_processedexp.sql \ + camtool_pendingcleanupexp.sql \ + camtool_pendingcleanuprun.sql \ + camtool_queue_chip_id.sql \ + camtool_revertprocessedexp.sql \ + camtool_revertupdatedexp.sql \ + camtool_export_run.sql \ + camtool_export_processed_exp.sql \ + chiptool_change_exp_state.sql \ + chiptool_change_imfile_data_state.sql \ + chiptool_completely_processed_exp.sql \ + chiptool_coalesce_run.sql \ + chiptool_definecopy.sql \ + chiptool_donecleanup.sql \ + chiptool_dropprocessedimfile_update.sql \ + chiptool_dropprocessedimfile_delete.sql \ + chiptool_find_rawexp.sql \ + chiptool_listrun.sql \ + chiptool_pendingcleanupimfile.sql \ + chiptool_pendingcleanupimfile_all.sql \ + chiptool_pendingcleanuprun.sql \ + chiptool_pendingimfile.sql \ + chiptool_processedimfile.sql \ + chiptool_revertcleanup.sql \ + chiptool_revertprocessedimfile.sql \ + chiptool_revertupdatedimfile.sql \ + chiptool_run.sql \ + chiptool_runstate.sql \ + chiptool_setimfiletoupdate.sql \ + chiptool_export_run.sql \ + chiptool_export_imfile.sql \ + chiptool_export_processed_imfile.sql \ + chiptool_unmasked.sql \ + detselect_search.sql \ + detselect_select.sql \ + detselect_show.sql \ + dettool_addprocessedexp.sql \ + dettool_childlessrun.sql \ + dettool_definebydetrun.sql \ + dettool_detrunsummary.sql \ + dettool_find_completed_runs.sql \ + dettool_input.sql \ + dettool_normalizedexp.sql \ + dettool_normalizedimfile.sql \ + dettool_normalizedstat.sql \ + dettool_pending.sql \ + dettool_pendingcleanup_detrunsummary.sql \ + dettool_pendingcleanup_normalizedexp.sql \ + dettool_pendingcleanup_normalizedimfile.sql \ + dettool_pendingcleanup_normalizedstat.sql \ + dettool_pendingcleanup_processedexp.sql \ + dettool_pendingcleanup_processedimfile.sql \ + dettool_pendingcleanup_residexp.sql \ + dettool_pendingcleanup_residimfile.sql \ + dettool_pendingcleanup_stacked.sql \ + dettool_processedimfile.sql \ + dettool_raw.sql \ + dettool_residexp.sql \ + dettool_residimfile.sql \ + dettool_revertdetrunsummary.sql \ + dettool_revertnormalizedexp.sql \ + dettool_revertnormalizedimfile.sql \ + dettool_revertnormalizedstat.sql \ + dettool_revertprocessedexp.sql \ + dettool_revertprocessedimfile.sql \ + dettool_revertresidexp.sql \ + dettool_revertresidimfile.sql \ + dettool_revertstacked.sql \ + dettool_runs.sql \ + dettool_stacked.sql \ + dettool_start_new_iteration.sql \ + dettool_stop_completed_correct_runs.sql \ + dettool_tocorrectexp.sql \ + dettool_tocorrectimfile.sql \ + dettool_todetrunsummary.sql \ + dettool_tonormalize.sql \ + dettool_tonormalizedexp.sql \ + dettool_tonormalizedstat.sql \ + dettool_toprocessedexp.sql \ + dettool_toprocessedimfile.sql \ + dettool_toresidexp.sql \ + dettool_toresidimfile.sql \ + dettool_tostacked.sql \ + difftool_change_skyfile_data_state.sql \ + difftool_change_run_state.sql \ + difftool_completed_runs.sql \ + difftool_coalesce_run.sql \ + difftool_definewarpstack.sql \ + difftool_definewarpstack_old.sql \ + difftool_definewarpstack_part1.sql \ + difftool_definewarpstack_part2.sql \ + difftool_definewarpstack_temp_create.sql \ + difftool_definewarpwarp_temp_create.sql \ + difftool_definewarpwarp_temp_insert.sql \ + difftool_definewarpwarp_select.sql \ + difftool_definewarpwarp_insert.sql \ + difftool_definestackstack_part0.sql \ + difftool_definestackstack_part1.sql \ + difftool_donecleanup.sql \ + difftool_export_input_skyfile.sql \ + difftool_export_run.sql \ + difftool_export_skyfile.sql \ + difftool_inputskyfile.sql \ + difftool_listrun.sql \ + difftool_listssrun.sql \ + difftool_pendingcleanuprun.sql \ + difftool_pendingcleanupskyfile.sql \ + difftool_pendingcleanupskyfile_all.sql \ + difftool_revertdiffskyfile_delete.sql \ + difftool_revertdiffskyfile_updated.sql \ + difftool_revertcleanup.sql \ + difftool_setskyfiletoupdate.sql \ + difftool_skyfile.sql \ + difftool_todiffskyfile.sql \ + difftool_tosummary.sql \ + difftool_addsummary.sql \ + disttool_definebyquery_camera.sql \ + disttool_definebyquery_chip.sql \ + disttool_definebyquery_chip_bg.sql \ + disttool_definebyquery_diff.sql \ + disttool_definebyquery_fake.sql \ + disttool_definebyquery_ff.sql \ + disttool_definebyquery_raw.sql \ + disttool_definebyquery_raw_no_magic.sql \ + disttool_definebyquery_sky.sql \ + disttool_definebyquery_sky_singlefilter.sql \ + disttool_definebyquery_skycal.sql \ + disttool_definebyquery_stack.sql \ + disttool_definebyquery_warp.sql \ + disttool_definebyquery_warp_bg.sql \ + disttool_definebyquery_SSdiff.sql \ + disttool_defineinterest.sql \ + disttool_listfilesets.sql \ + disttool_listinterests.sql \ + disttool_pending_camera.sql \ + disttool_pending_chip.sql \ + disttool_pending_chip_bg.sql \ + disttool_pending_diff.sql \ + disttool_pending_fake.sql \ + disttool_pending_ff.sql \ + disttool_pending_raw.sql \ + disttool_pending_sky.sql \ + disttool_pending_skycal.sql \ + disttool_pending_stack.sql \ + disttool_pending_warp.sql \ + disttool_pending_warp_bg.sql \ + disttool_pending_SSdiff.sql \ + disttool_pendingfileset.sql \ + disttool_pendingdest.sql \ + disttool_pendingcleanup.sql \ + disttool_processedcomponent.sql \ + disttool_queuercrun.sql \ + disttool_rerun_select.sql \ + disttool_revertrcrun.sql \ + disttool_revertrun.sql \ + disttool_revertcomponent.sql \ + disttool_revertfileset.sql \ + disttool_toadvance.sql \ + disttool_updateinterest.sql \ + disttool_updatercrun.sql \ + dqstatstool_createbundle.sql \ + dqstatstool_definebyquery.sql \ + dqstatstool_get_contents.sql \ + dqstatstool_get_run.sql \ + faketool_change_exp_state.sql \ + faketool_change_imfile_data_state.sql \ + faketool_completely_processed_exp.sql \ + faketool_donecleanup.sql \ + faketool_export_processed_imfile.sql \ + faketool_export_run.sql \ + faketool_find_camrun.sql \ + faketool_find_pendingexp.sql \ + faketool_pendingcleanupimfile.sql \ + faketool_pendingcleanuprun.sql \ + faketool_pendingimfile.sql \ + faketool_processedimfile.sql \ + faketool_queue_cam_id.sql \ + faketool_revertprocessedimfile.sql \ + faketool_unmasked.sql \ + flatcorr_chiprundone.sql \ + flatcorr_camerarundone.sql \ + flatcorr_pendingprocess.sql \ + flatcorr_inputimfile.sql \ + flatcorr_dropchip.sql \ + flatcorr_dropcamera.sql \ + magictool_deletemask.sql \ + magictool_restore_camera.sql \ + magictool_restore_chip.sql \ + magictool_restore_diff.sql \ + magictool_restore_raw.sql \ + magictool_restore_warp.sql \ + magictool_create_tmp_warpcomplete.sql \ + magictool_definebyquery_insert.sql \ + magictool_definebyquery_select.sql \ + magictool_definebyquery_select_multidiff.sql \ + magictool_inputs.sql \ + magictool_inputskyfile.sql \ + magictool_mask.sql \ + magictool_tomask.sql \ + magictool_toprocess_inputs.sql \ + magictool_toprocess_runs.sql \ + magictool_toprocess_tree.sql \ + magictool_toskyfilemask.sql \ + magictool_totree.sql \ + magictool_diffskyfile.sql \ + magictool_warpskyfile.sql \ + magictool_chipprocessedimfile.sql \ + magictool_rawimfile.sql \ + magictool_revertnode.sql \ + magictool_exposure.sql \ + magictool_setfull.sql \ + magictool_setgotocleaned.sql \ + magictool_tocleanup.sql \ + magicdstool_clearstatefaults.sql \ + magicdstool_change_file_data_state.sql \ + magicdstool_change_run_state.sql \ + magicdstool_completed_runs.sql \ + magicdstool_completedrevert.sql \ + magicdstool_definebyquery_raw.sql \ + magicdstool_definebyquery_chip.sql \ + magicdstool_definebyquery_chip_bg.sql \ + magicdstool_definebyquery_camera.sql \ + magicdstool_definebyquery_warp.sql \ + magicdstool_definebyquery_warp_bg.sql \ + magicdstool_definebyquery_diff.sql \ + magicdstool_definecopy_camera.sql \ + magicdstool_definecopy_chip.sql \ + magicdstool_definecopy_warp.sql \ + magicdstool_destreakedfile.sql \ + magicdstool_getrunids.sql \ + magicdstool_getskycells.sql \ + magicdstool_listrun.sql \ + magicdstool_revertdestreakedfile.sql \ + magicdstool_revertupdated.sql \ + magicdstool_setfiletoupdate.sql \ + magicdstool_tocleanup.sql \ + magicdstool_todestreak_camera.sql \ + magicdstool_todestreak_chip.sql \ + magicdstool_todestreak_diff.sql \ + magicdstool_todestreak_raw.sql \ + magicdstool_todestreak_warp.sql \ + magicdstool_toremove.sql \ + magicdstool_torevert_raw.sql \ + magicdstool_torevert_chip.sql \ + magicdstool_torevert_camera.sql \ + magicdstool_torevert_warp.sql \ + magicdstool_torevert_diff.sql \ + minidvodbtool_find_minidvodb_id_dvo.sql \ + minidvodbtool_find_minidvodbcopy.sql \ + minidvodbtool_queue_minidvodb_id.sql \ + minidvodbtool_revertminidvodbcopy.sql \ + mergetool_find_minidvodb_id.sql \ + mergetool_find_mergedvodbcopy.sql \ + mergetool_find_merge_id_dvo.sql \ + mergetool_find_pendingmergeprocess.sql \ + mergetool_find_pending_merge.sql \ + mergetool_find_processed.sql \ + mergetool_queue_mergecopy_id.sql \ + mergetool_queue_merge_id.sql \ + mergetool_revertmergedvodbcopy.sql \ + mergetool_revertmergedvodbprocessed.sql \ + pstamptool_completedreq.sql \ + pstamptool_datastore.sql \ + pstamptool_getdependent.sql \ + pstamptool_listjob.sql \ + pstamptool_listfile.sql \ + pstamptool_listuser.sql \ + pstamptool_pendingcleanup.sql \ + pstamptool_pendingdependent.sql \ + pstamptool_pendingjob.sql \ + pstamptool_pendingreq.sql \ + pstamptool_project.sql \ + pstamptool_revertdependent.sql \ + pstamptool_revertjob.sql \ + pstamptool_revertreq.sql \ + pstamptool_revertreq_deletejobs.sql \ + pstamptool_stopdependentjob.sql \ + pstamptool_updatejob.sql \ + pstamptool_addaccesslevel.sql \ + pxadmin_create_tables.sql \ + pxadmin_create_mirror_tables.sql \ + pxadmin_drop_tables.sql \ + pxadmin_update_version.sql \ + pubtool_definerun.sql \ + pubtool_pending.sql \ + pubtool_revert.sql \ + pztool_find_completed_exp.sql \ + pztool_pendingimfile.sql \ + pztool_revert_downloadimfile_faults.sql \ + pztool_revert_fileset_faults.sql \ + pztool_revertcopied.sql \ + receivetool_list.sql \ + receivetool_addfileset.sql \ + receivetool_pendingfileset.sql \ + receivetool_pendingfile.sql \ + receivetool_revert.sql \ + receivetool_toadvance.sql \ + regtool_checkburntoolimfile.sql \ + regtool_create_dup_table.sql \ + regtool_export_exp.sql \ + regtool_export_imfile.sql \ + regtool_pendingexp.sql \ + regtool_pendingimfile.sql \ + regtool_populate_dup_table.sql \ + regtool_processedexp.sql \ + regtool_processedimfile.sql \ + regtool_revertprocessedexp.sql \ + regtool_revertprocessedimfile.sql \ + regtool_updateprocessedimfile.sql \ + regtool_pendingcompressimfile.sql \ + regtool_finishcompressexp.sql \ + regtool_updatebyquery.sql \ + regtool_updatebyqueryimfile.sql \ + regtool_pendingburntoolimfile.sql \ + regtool_checkstatus.sql \ + stacktool_associationdefine_select.sql \ + stacktool_definebyquery_insert.sql \ + stacktool_definebyquery_insert_random_part1.sql \ + stacktool_definebyquery_insert_random_part2.sql \ + stacktool_definebyquery_select.sql \ + stacktool_definebyquery_test.sql \ + stacktool_donecleanup.sql \ + stacktool_export_input_skyfile.sql \ + stacktool_export_run.sql \ + stacktool_export_sum_skyfile.sql \ + stacktool_find_complete_warps.sql \ + stacktool_inputskyfile.sql \ + stacktool_pendingcleanuprun.sql \ + stacktool_pendingcleanupskyfile.sql \ + stacktool_revertsumskyfile_delete.sql \ + stacktool_sumskyfile.sql \ + stacktool_sassskyfile.sql \ + stacktool_tosum.sql \ + stacktool_tobkg.sql \ + stacktool_tosummary.sql \ + stacktool_addsummary.sql \ + stacktool_revertsummary.sql \ + stacktool_summary.sql \ + staticskytool_definebyquery_select.sql \ + staticskytool_definebyquery_select_by_dg.sql \ + staticskytool_definebyquery_inputs.sql \ + staticskytool_defineskycalrun.sql \ + staticskytool_pendingskycalrun.sql \ + staticskytool_revertskycal.sql \ + staticskytool_skycalresult.sql \ + staticskytool_export_run.sql \ + staticskytool_export_input.sql \ + staticskytool_export_result.sql \ + staticskytool_export_skycalrun.sql \ + staticskytool_export_skycalresult.sql \ + staticskytool_inputs.sql \ + staticskytool_todo.sql \ + staticskytool_result.sql \ + staticskytool_revert.sql \ + staticskytool_revert_update.sql \ + warptool_change_skyfile_data_state.sql \ + warptool_change_run_state.sql \ + warptool_definebyquery.sql \ + warptool_donecleanup.sql \ + warptool_exp.sql \ + warptool_export_imfile.sql \ + warptool_export_run.sql \ + warptool_export_skyfile.sql \ + warptool_export_skycell_map.sql \ + warptool_finished_run_select.sql \ + warptool_finish_run.sql \ + warptool_coalesce_run.sql \ + warptool_imfile.sql \ + warptool_listrun.sql \ + warptool_pendingcleanuprun.sql \ + warptool_pendingcleanupskyfile.sql \ + warptool_pendingcleanupskyfile_all.sql \ + warptool_revertoverlap.sql \ + warptool_revertwarped_delete.sql \ + warptool_revertwarped_updated.sql \ + warptool_revertcleanup.sql \ + warptool_runstate.sql \ + warptool_scmap.sql \ + warptool_setskyfiletoupdate.sql \ + warptool_tooverlap.sql \ + warptool_towarped.sql \ + warptool_towarped_labels.sql \ + warptool_updateskyfile.sql \ + warptool_warped.sql \ + warptool_tosummary.sql \ + warptool_addsummary.sql \ + diffphottool_definerun.sql \ + diffphottool_input.sql \ + diffphottool_pending.sql \ + diffphottool_advance.sql \ + diffphottool_revert.sql \ + diffphottool_data.sql \ + laptool_definerun.sql \ + laptool_definerunbyrelease.sql \ + laptool_exposures.sql \ + laptool_inactiveexp.sql \ + laptool_listsequence.sql \ + laptool_pendingchipexp.sql \ + laptool_pendingexp.sql \ + laptool_pendingrun.sql \ + laptool_WSdiff_check.sql \ + laptool_stacks.sql \ + laptool_pendinggroup.sql \ + laptool_revertgroup.sql \ + laptool_filtersforgroup.sql \ + laptool_revertsass.sql \ + vptool_find_rawexp.sql \ + vptool_pendingimfile.sql \ + vptool_pendingrun.sql \ + vptool_processedcell.sql \ + vptool_revertrun.sql \ + sctool_list.sql \ + releasetool_listsurvey.sql \ + releasetool_tocalibexp.sql \ + releasetool_listrelease.sql \ + releasetool_listrelexp.sql \ + releasetool_definerelexp.sql \ + releasetool_definerelstack.sql \ + releasetool_definerelstack_with_skycal.sql \ + releasetool_deleterelexp.sql \ + releasetool_deleterelstack.sql \ + releasetool_listrelstack.sql \ + releasetool_setrelstackcalibratedfromskycal.sql \ + releasetool_summary.sql \ + releasetool_definerelgroup_select_lap.sql \ + releasetool_definerelgroup_select_data_group.sql \ + releasetool_definerelgroup_select_exp_data_group.sql \ + releasetool_definerelgroup_select_exp_lap.sql \ + releasetool_stacksummary.sql \ + releasetool_pendingrelgroup.sql \ + fftool_definebyquery.sql \ + fftool_definebyquery_select_warps.sql \ + fftool_result.sql \ + fftool_revert.sql \ + fftool_revertsummary.sql \ + fftool_summary.sql \ + fftool_todo.sql \ + fftool_toadvance.sql \ + fftool_export_run.sql \ + fftool_export_input.sql \ + fftool_export_result.sql \ + fftool_export_summary.sql \ + remotetool_definebyquery_chip.sql \ + remotetool_definebyquery_cam.sql \ + remotetool_definebyquery_warp.sql \ + remotetool_definebyquery_stack.sql \ + remotetool_definebyquery_staticsky.sql \ + remotetool_definebyquery_diff.sql \ + remotetool_definebyquery_ff.sql \ + remotetool_dropcomponent.sql \ + remotetool_listcomponent.sql \ + remotetool_doneprep.sql \ + remotetool_updatepoll.sql Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_checkminidvodbaddrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_checkminidvodbaddrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_checkminidvodbaddrun.sql (revision 41104) @@ -0,0 +1,20 @@ +-- this selects the minidvodbs that have complete addRuns +SELECT minidvodb_id, minidvodb_name, minidvodb_path, state, minidbg +AS minidvodb_group, (cnt - cnt2) as not_full, cnt as addRun_count +FROM minidvodbRun +JOIN + (SELECT minidvodb_name, count(state) as cnt2, cnt, minidbg, minidbn + FROM addRun LEFT JOIN + -- this select grabs (and counts) all the addRuns with any state + -- and groups them by the name + (SELECT minidvodb_group as minidbg, minidvodb_name as minidbn, count(state) + AS cnt + FROM addRun where minidvodb_group = '%s' + GROUP BY minidvodb_name ) + AS foo1 ON minidvodb_name = minidbn + -- the second select grabs and counts all the addRuns with state + -- of 'full' + WHERE addRun.state = 'full' and minidvodb_group = '%s' + GROUP BY minidvodb_name ) +AS foo2 +USING(minidvodb_name) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + addRun.* +FROM addRun +WHERE + addRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_cam_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_cam_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_cam_id.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + camRun.* +FROM camRun +JOIN camProcessedExp + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +LEFT JOIN (SELECT exp_id AS added_exp_id, + addRun.dvodb AS previous_dvodb + FROM addRun + JOIN camRun on cam_id=stage_id + JOIN chipRun USING(chip_id) + ) as foo + ON exp_id = added_exp_id + AND stage = 'cam' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + camRun.state = 'full' + AND camProcessedExp.quality = 0 + AND added_exp_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_cam_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_cam_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_cam_id_dvo.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT camRun.* FROM camRun +JOIN camProcessedExp USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +WHERE camRun.state = 'full' and camProcessedExp.quality = 0 + AND exp_id NOT IN (SELECT exp_id + FROM addRun + JOIN camRun on cam_id=stage_id + JOIN chipRun USING(chip_id) + WHERE stage = 'cam' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_diff_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_diff_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_diff_id.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + diffRun.*, diffInputSkyfile.diff_skyfile_id +FROM + diffInputSkyfile +JOIN diffSkyfile using (diff_id, skycell_id) +JOIN diffRun using (diff_id) + +LEFT JOIN (SELECT diff_id AS added_diff_id, + addRun.dvodb AS previous_dvodb + FROM addRun +JOIN diffRun on (diff_id = stage_id and addRun.stage = 'diff') + ) as foo + ON diff_id = added_diff_id + -- AND stage = 'diff' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + diffRun.state = 'full' + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 + AND added_diff_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_diff_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_diff_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_diff_id_dvo.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT diffRun.*, diffInputSkyfile.diff_skyfile_id +FROM + diffInputSkyfile +JOIN diffSkyfile using (diff_id, skycell_id) +JOIN diffRun using (diff_id,tess_id) +JOIN skycell using (skycell_id, tess_id) + JOIN warpRun on (warp1 = warp_id + AND diffInputSkyfile.skycell_id = skycell_id) + JOIN fakeRun using (fake_id) + JOIN camRun using (cam_id) + JOIN chipRun using (chip_id) + JOIN rawExp using (exp_id) +WHERE diffRun.state = 'full' and diffSkyfile.quality = 0 + AND diffInputSkyfile.diff_skyfile_id NOT IN (SELECT diffInputSkyfile.diff_skyfile_id + FROM + diffInputSkyfile + JOIN diffSkyfile USING (diff_id, skycell_id) + JOIN diffRun using(diff_id, tess_id) + JOIN skycell using (skycell_id, tess_id) + left join addRun ON (diff_id = stage_id and diff_skyfile_id = stage_extra1 and stage = "diff") + WHERE addRun.stage = 'diff' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ff_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ff_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ff_id.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT + fullForceRun.*, fullForceResult.warp_id +FROM + fullForceRun +JOIN fullForceResult using (ff_id) + +LEFT JOIN (SELECT ff_id AS added_ff_id, + addRun.dvodb AS previous_dvodb + FROM addRun +JOIN fullForceRun on (ff_id = stage_id and addRun.stage = 'fullforce') + ) as foo + ON ff_id = added_ff_id + --AND stage = 'skycal' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + fullForceRun.state = 'full' + AND fullForceResult.quality = 0 + AND fullForceResult.fault = 0 + AND added_ff_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ff_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ff_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ff_id_dvo.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT fullForceRun.*, fullForceResult.warp_id FROM fullForceRun JOIN fullForceResult using (ff_id) +JOIN skycalRun using (skycal_id) join stackRun using (stack_id) +JOIN skycell using (skycell_id, tess_id) + +WHERE fullForceRun.state = 'full' and fullForceResult.quality = 0 + AND fullForceRun.ff_id NOT IN (SELECT fullForceRun.ff_id + FROM addRun + JOIN fullForceRun on (fullForceRun.ff_id = addRun.stage_id and addRun.stage = 'fullforce') JOIN fullForceResult on (fullForceRun.ff_id = fullForceResult.ff_id) + WHERE addRun.stage = 'fullforce' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ffsummary_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ffsummary_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ffsummary_id.sql (revision 41104) @@ -0,0 +1,18 @@ +SELECT fullForceRun.* FROM fullForceRun +JOIN fullForceSummary USING (ff_id) +LEFT JOIN ( + SELECT ff_id AS added_ff_id, + addRun.dvodb AS previous_dvodb + FROM addRun + JOIN fullForceRun ON (ff_id = stage_id and addRun.stage = 'fullforce_summary') + ) as foo +ON ff_id = added_ff_id + -- AND stage = 'skycal' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + fullForceRun.state = 'full' + AND fullForceSummary.quality = 0 + AND fullForceSummary.fault = 0 + AND added_ff_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ffsummary_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ffsummary_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_ffsummary_id_dvo.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT fullForceRun.* FROM fullForceRun +JOIN fullForceSummary USING (ff_id) +JOIN skycalRun USING (skycal_id) +JOIN stackRun USING (stack_id) +JOIN skycell USING (skycell_id, tess_id) + +WHERE fullForceRun.state = 'full' +AND fullForceSummary.quality = 0 +AND fullForceSummary.fault = 0 +AND fullForceRun.ff_id NOT IN ( + SELECT fullForceRun.ff_id FROM addRun + JOIN fullForceRun ON (fullForceRun.ff_id = addRun.stage_id and addRun.stage = 'fullforce_summary') + JOIN fullForceSummary ON (fullForceRun.ff_id = fullForceSummary.ff_id) + WHERE addRun.stage = 'fullforce_summary' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbprocessed.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbprocessed.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbprocessed.sql (revision 41104) @@ -0,0 +1,8 @@ +SELECT + minidvodbProcessed.*, + minidvodbRun.minidvodb_name, + minidvodbRun.minidvodb_group +FROM minidvodbProcessed +JOIN minidvodbRun + USING(minidvodb_id) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbrun.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + minidvodbRun.* +FROM minidvodbRun + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbrun2.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbrun2.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_minidvodbrun2.sql (revision 41104) @@ -0,0 +1,23 @@ +-- this selects the minidvodbs that have complete addRuns +SELECT minidvodb_name, minidvodb_path, state, minidbg +AS minidvodb_group +FROM minidvodbRun +JOIN + (SELECT minidvodb_name, count(state) as cnt2, cnt, minidbg, minidbn + FROM addRun LEFT JOIN + -- this select grabs (and counts) all the addRuns with any state + -- and groups them by the name + (SELECT minidvodb_group as minidbg, minidvodb_name as minidbn, count(state) + AS cnt + FROM addRun + GROUP BY minidvodb_name ) + AS foo1 ON minidvodb_name = minidbn + -- the second select grabs and counts all the addRuns with state + -- of 'full' + WHERE addRun.state = 'full' + GROUP BY minidvodb_name ) +AS foo2 +USING(minidvodb_name) +WHERE cnt2 = cnt + -- the 3rd select only grabs the columns where the counts are the same + -- (same number of 'full' and any state) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT + addRun.* +FROM addRun +LEFT JOIN addProcessedExp + USING(add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE +((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_cam.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_cam.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_cam.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT + addRun.*, + camProcessedExp.path_base as stageroot, + rawExp.exp_tag, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel +FROM addRun +JOIN camRun + on cam_id = stage_id +JOIN camProcessedExp + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +LEFT JOIN addProcessedExp + USING(add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + camRun.state = 'full' + AND addRun.stage = 'cam' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_diff.sql (revision 41104) @@ -0,0 +1,21 @@ + SELECT + addRun.*, + diffSkyfile.path_base as stageroot, + 'GPC1' as camera, + 'PS1' as telescope +FROM addRun +JOIN diffInputSkyfile + ON (diff_id = stage_id and diff_skyfile_id = stage_extra1 and stage = "diff") +JOIN diffSkyfile + USING (diff_id, skycell_id) +JOIN diffRun + using(diff_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + diffRun.state = 'full' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_ff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_ff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_ff.sql (revision 41104) @@ -0,0 +1,19 @@ + SELECT + addRun.*, + fullForceResult.path_base as stageroot, + 'GPC1' as camera, + 'PS1' as telescope +FROM addRun +JOIN fullForceResult + ON (ff_id = stage_id and warp_id = stage_extra1 and stage = "fullforce") +JOIN fullForceRun + using(ff_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + fullForceRun.state = 'full' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_ffsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_ffsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_ffsummary.sql (revision 41104) @@ -0,0 +1,19 @@ + SELECT + addRun.*, + fullForceSummary.path_base as stageroot, + 'GPC1' as camera, + 'PS1' as telescope +FROM addRun +JOIN fullForceSummary + ON (ff_id = stage_id and stage = "fullforce_summary") +JOIN fullForceRun + using(ff_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + fullForceRun.state = 'full' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_skycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_skycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_skycal.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT + addRun.*, + skycalResult.path_base as stageroot, + rawExp.camera, + rawExp.telescope +FROM addRun +JOIN skycalRun + ON skycal_id = stage_id +JOIN skycalResult + USING (skycal_id) +JOIN stackRun + using(stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpRun using(warp_id) +JOIN fakeRun using(fake_id) +JOIN camRun using(cam_id) +JOIN chipRun using(chip_id) +JOIN rawExp using (exp_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + skycalRun.state = 'full' + AND stage = 'skycal' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_stack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_stack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_stack.sql (revision 41104) @@ -0,0 +1,28 @@ +SELECT + addRun.*, + stackSumSkyfile.path_base as stageroot, + rawExp.camera, + rawExp.telescope +FROM addRun +JOIN stackRun + ON stack_id = stage_id +JOIN stackSumSkyfile + USING(stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpRun using(warp_id) +JOIN fakeRun using(fake_id) +JOIN camRun using(cam_id) +JOIN chipRun using(chip_id) +JOIN rawExp using (exp_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + stackRun.state = 'full' + AND stage = 'stack' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_staticsky.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_staticsky.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_staticsky.sql (revision 41104) @@ -0,0 +1,32 @@ +SELECT + addRun.*, + staticskyResult.path_base as stageroot, + rawExp.camera, + rawExp.telescope +FROM addRun +JOIN staticskyRun + ON sky_id = stage_id +JOIN staticskyResult + USING (sky_id) +JOIN staticskyInput + USING (sky_id) +JOIN stackRun + USING (stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpRun using(warp_id) +JOIN fakeRun using(fake_id) +JOIN camRun using(cam_id) +JOIN chipRun using(chip_id) +JOIN rawExp using (exp_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + staticskyRun.state = 'full' + AND stage = 'staticsky' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_staticsky_multi.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_staticsky_multi.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingexp_staticsky_multi.sql (revision 41104) @@ -0,0 +1,32 @@ +SELECT + addRun.*, + staticskyResult.path_base as stageroot, + rawExp.camera, + rawExp.telescope +FROM addRun +JOIN staticskyRun + ON sky_id = stage_id +JOIN staticskyResult + USING (sky_id) +JOIN staticskyInput + USING (sky_id) +JOIN stackRun + USING (stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpRun using(warp_id) +JOIN fakeRun using(fake_id) +JOIN camRun using(cam_id) +JOIN chipRun using(chip_id) +JOIN rawExp using (exp_id) +LEFT JOIN addProcessedExp using (add_id) +LEFT JOIN addMask + ON addRun.label = addMask.label +WHERE + staticskyRun.state = 'full' + AND stage = 'staticsky' + AND ((addRun.state = 'new' AND addProcessedExp.add_id IS NULL) OR addRun.state = 'update') + AND addRun.dvodb IS NOT NULL + AND addRun.workdir IS NOT NULL + AND addMask.label IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingmergeprocess.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingmergeprocess.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_pendingmergeprocess.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT minidvodbRun.* +FROM minidvodbRun +LEFT JOIN minidvodbProcessed +USING(minidvodb_id) +WHERE (minidvodbRun.state = 'to_be_merged' AND minidvodbProcessed.minidvodb_id IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_cam.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_cam.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_cam.sql (revision 41104) @@ -0,0 +1,18 @@ +SELECT + addProcessedExp.*, + addRun.label, + addRun.workdir, + camRun.cam_id, + camRun.label as cam_label, + camRun.data_group as cam_data_group, + rawExp.exp_id, + exp_name +FROM addProcessedExp +JOIN addRun + USING(add_id) +JOIN camRun + on cam_id = stage_id +JOIN chipRun + USING(chip_id) +JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_diff.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_ff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_ff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_ff.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_ffsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_ffsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_ffsummary.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_skycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_skycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_skycal.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_stack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_stack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_stack.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_staticsky.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_staticsky.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_staticsky.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_staticsky_multi.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_staticsky_multi.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_processedexp_staticsky_multi.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + addProcessedExp.*, + addRun.workdir +FROM addProcessedExp +JOIN addRun + USING(add_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + staticskyRun.* +FROM + staticskyRun +join staticskyResult using (sky_id) +join staticskyInput using(sky_id) +JOIN stackRun USING(stack_id) + +LEFT JOIN (SELECT sky_id AS added_sky_id, + addRun.dvodb AS previous_dvodb + FROM addRun +JOIN staticskyRun on sky_id = stage_id + ) as foo + ON sky_id = added_sky_id + AND stage = 'staticsky' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + staticskyRun.state = 'full' + AND staticskyResult.quality = 0 + AND added_sky_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_dvo.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT staticskyRun.* FROM staticskyRun +JOIN staticskyResult USING(sky_id) +join staticskyInput using(sky_id) +JOIN stackRun USING(stack_id) + +WHERE staticskyRun.state = 'full' and staticskyResult.quality = 0 + AND stack_id NOT IN (SELECT stack_id + FROM addRun + JOIN staticskyResult on staticskyResult.sky_id = addRun.stage_id + JOIN staticskyRun USING(sky_id) + JOIN staticskyInput USING(sky_id) + WHERE addRun.stage = 'staticsky' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_multi.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_multi.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_multi.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + distinct(sky_id) as stage_id, + staticskyRun.*, + staticskyResult.num_inputs, stack_id +FROM + staticskyRun +join staticskyResult using (sky_id) +join staticskyInput using(sky_id) +JOIN stackRun USING(stack_id) + +LEFT JOIN (SELECT stack_id AS added_stack_id, + addRun.dvodb AS previous_dvodb + FROM addRun +JOIN staticskyRun on sky_id = stage_id + ) as foo + ON stack_id = added_stack_id + AND stage = 'staticsky' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + staticskyRun.state = 'full' + AND staticskyResult.quality = 0 + AND added_stack_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_multi_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_multi_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_sky_id_multi_dvo.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT distinct(sky_id) as stage_id, staticskyRun.*, staticskyResult.num_inputs, stack_id FROM staticskyResult +JOIN staticskyRun USING(sky_id) +join staticskyInput using(sky_id) +JOIN stackRun USING(stack_id) + +WHERE staticskyRun.state = 'full' and staticskyResult.quality = 0 + AND stack_id NOT IN (SELECT stack_id + FROM addRun + JOIN staticskyResult on staticskyResult.sky_id = addRun.stage_id + JOIN staticskyRun USING(sky_id) + JOIN staticskyInput USING(sky_id) + WHERE addRun.stage = 'staticsky' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_skycal_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_skycal_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_skycal_id.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + skycalRun.* +FROM + skycalRun +JOIN skycalResult using (skycal_id) +JOIN staticskyInput using (sky_id, stack_id) +JOIN staticskyRun using (sky_id) + +LEFT JOIN (SELECT skycal_id AS added_skycal_id, + addRun.dvodb AS previous_dvodb + FROM addRun +JOIN skycalRun on skycal_id = stage_id + ) as foo + ON skycal_id = added_skycal_id + AND stage = 'skycal' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + skycalRun.state = 'full' + AND skycalResult.quality = 0 + AND added_skycal_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_skycal_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_skycal_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_skycal_id_dvo.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT skycalRun.* FROM skycalRun +JOIN skycalResult USING (skycal_id) +JOIN staticskyInput USING (sky_id,stack_id) +JOIN stackRun USING(stack_id) + +WHERE skycalRun.state = 'full' and skycalResult.quality = 0 + AND stack_id NOT IN (SELECT stack_id + FROM addRun + JOIN skycalResult on skycalResult.skycal_id = addRun.stage_id + JOIN skycalRun USING(skycal_id) + WHERE addRun.stage = 'skycal' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_stack_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_stack_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_stack_id.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT + stackRun.* +FROM stackSumSkyfile +JOIN stackRun USING(stack_id) + +LEFT JOIN (SELECT stack_id AS added_stack_id, + addRun.dvodb AS previous_dvodb + FROM addRun +JOIN stackRun on stack_id = stage_id + ) as foo + ON stack_id = added_stack_id + AND stage = 'stack' + -- hook for qualifying the join on the previous_dvodb + AND %s +WHERE + stackRun.state = 'full' + AND stackSumSkyfile.quality = 0 + AND added_exp_id IS NULL + -- addtool adds checks on exposure being added to the dvodb previously Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_stack_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_stack_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_find_stack_id_dvo.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT stackRun.* FROM stackSumSkyfile +JOIN stackRun USING(stack_id) +WHERE stackRun.state = 'full' and stackSumSkyfile.quality = 0 + AND stack_id NOT IN (SELECT stack_id + FROM addRun + JOIN stackInputSkyfile on stackInputSkyfile.stack_id = addRun.stage_id + JOIN stackRun USING(stack_id) + WHERE addRun.stage = 'stack' AND %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_pendingcleanupexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_pendingcleanupexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_pendingcleanupexp.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + addProcessedExp.*, + addRun.state, + addRun.workdir, + addRun.label, + addRun.dvodb, +FROM addRun +JOIN addProcessedExp + USING(add_id) +WHERE + (addRun.state = 'goto_cleaned' OR addRun.state = 'goto_purged' OR addRun.state = 'goto_scrubbed') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT + addRun.add_id, + rawExp.camera, + addRun.state +FROM addRun +JOIN camRun +USING (cam_id) +JOIN chipRun +USING (chip_id) +JOIN rawExp +USING (exp_id) +WHERE + (addRun.state = 'goto_cleaned' OR addRun.state = 'goto_purged' OR addRun.state = 'goto_scrubbed') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_cam_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_cam_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_cam_id.sql (revision 41104) @@ -0,0 +1,22 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'cam', -- stage + cam_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM camRun + WHERE + camRun.state = 'full' + AND camRun.cam_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_diff_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_diff_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_diff_id.sql (revision 41104) @@ -0,0 +1,23 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'diff', -- stage + diff_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM diffRun + WHERE + diffRun.state = 'full' + AND diffRun.diff_id = %lld + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_ff_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_ff_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_ff_id.sql (revision 41104) @@ -0,0 +1,22 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'fullforce', -- stage + ff_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM fullForceRun + WHERE + fullForceRun.state = 'full' + AND fullForceRun.ff_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_ffsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_ffsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_ffsummary.sql (revision 41104) @@ -0,0 +1,23 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'fullforce_summary', -- stage + ff_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM fullForceRun + WHERE + fullForceRun.state = 'full' + AND fullForceRun.ff_id = %lld + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_minidvodbrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_minidvodbrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_minidvodbrun.sql (revision 41104) @@ -0,0 +1,11 @@ +-- create rcRun's for all destinations that have filesets that they are interested in available +INSERT INTO minidvodbRun +SELECT -- rows in this select must match rcRun + minidvodb_name, + minidvodb_path, + 'new' + +FROM addRun +LEFT JOIN minidvodbRun USING(minidvodb_name) +-- WHERE minidvodbRun.minidvodb_name IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_sky_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_sky_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_sky_id.sql (revision 41104) @@ -0,0 +1,22 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'staticsky', -- stage + sky_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM staticskyRun + WHERE + staticskyRun.state = 'full' + AND staticskyRun.sky_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_sky_id_multi.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_sky_id_multi.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_sky_id_multi.sql (revision 41104) @@ -0,0 +1,22 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'staticsky', -- stage + sky_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM staticskyRun + WHERE + staticskyRun.state = 'full' + AND staticskyRun.sky_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_skycal_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_skycal_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_skycal_id.sql (revision 41104) @@ -0,0 +1,22 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'skycal', -- stage + skycal_id, -- stage_id + %d, -- stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM skycalRun + WHERE + skycalRun.state = 'full' + AND skycalRun.skycal_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_stack_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_stack_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_queue_stack_id.sql (revision 41104) @@ -0,0 +1,22 @@ +INSERT INTO addRun + SELECT + 0, -- add_id + 'stack', -- stage + stack_id, -- stage_id + %d, --stage_extra1 + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- reduction + '%s', -- label + '%s', -- data_group + '%s', -- dvodb + '%s', -- note + %d, -- image_only + %d, -- minidvodb + '%s', -- minidvodb_group + '%s' -- minidvodb_name + FROM stackRun + WHERE + stackRun.state = 'full' + AND stackRun.stack_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertminidvodbprocessed.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertminidvodbprocessed.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertminidvodbprocessed.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM minidvodbProcessed +USING minidvodbProcessed, minidvodbRun, addRun +WHERE + minidvodbProcessed.minidvodb_id = minidvodbRun.minidvodb_id + AND addRun.minidvodb_name = minidvodbRun.minidvodb_name + AND minidvodbProcessed.fault != 0 + AND minidvodbRun.state = 'merged' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp.sql (revision 41104) @@ -0,0 +1,9 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun, camRun, chipRun, rawExp +WHERE + addRun.add_id = addProcessedExp.add_id + AND addRun.cam_id = camRun.cam_id + AND camRun.chip_id = chipRun.chip_id + AND chipRun.exp_id = rawExp.exp_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_cam.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_cam.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_cam.sql (revision 41104) @@ -0,0 +1,10 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun, camRun, chipRun, rawExp +WHERE + addRun.add_id = addProcessedExp.add_id + AND addRun.stage_id = camRun.cam_id + AND camRun.chip_id = chipRun.chip_id + AND chipRun.exp_id = rawExp.exp_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'cam' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_diff.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'diff' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_ff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_ff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_ff.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'fullforce' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_ffsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_ffsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_ffsummary.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'fullforce_summary' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_skycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_skycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_skycal.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'skycal' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_stack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_stack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_stack.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'stack' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_staticsky.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_staticsky.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_staticsky.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'staticsky' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_staticsky_multi.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_staticsky_multi.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/addtool_revertprocessedexp_staticsky_multi.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM addProcessedExp +USING addProcessedExp, addRun +WHERE + addRun.add_id = addProcessedExp.add_id + AND addProcessedExp.fault != 0 + AND addRun.state = 'new' + AND addRun.stage = 'staticsky' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_advancechip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_advancechip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_advancechip.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT DISTINCT + chipBackgroundRun.chip_bg_id, + chipBackgroundImfile.magicked +FROM chipBackgroundRun +JOIN camRun using(cam_id) +JOIN chipProcessedImfile ON chipBackgroundRun.chip_id = chipProcessedImfile.chip_id +JOIN chipRun AS bgsub_chipRun on camRun.chip_id = bgsub_chipRun.chip_id +JOIN chipProcessedImfile AS bgsub_chipProcessedImfile + ON (bgsub_chipRun.chip_id = bgsub_chipProcessedImfile.chip_id + AND bgsub_chipProcessedImfile.class_id = chipProcessedImfile.class_id) +LEFT JOIN chipBackgroundImfile + ON (chipBackgroundRun.chip_bg_id = chipBackgroundImfile.chip_bg_id + AND chipBackgroundImfile.class_id = chipProcessedImfile.class_id) +WHERE chipBackgroundRun.state = 'new' + AND chipProcessedImfile.quality = 0 + AND chipProcessedImfile.fault = 0 + AND bgsub_chipProcessedImfile.quality = 0 +-- WHERE hook %s +GROUP BY chipBackgroundRun.chip_bg_id +HAVING + COUNT(chipBackgroundImfile.class_id) = COUNT(chipProcessedImfile.class_id) + AND SUM(IF(chipBackgroundImfile.fault > 0, 1, 0)) = 0 + AND SUM(IF(chipBackgroundImfile.quality > 0, 1, 0)) != COUNT(*) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_advancewarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_advancewarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_advancewarp.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT DISTINCT + warpBackgroundRun.warp_bg_id, + warpBackgroundSkyfile.magicked +FROM warpBackgroundRun +JOIN warpSkyfile USING(warp_id) +LEFT JOIN warpBackgroundSkyfile USING(warp_bg_id, skycell_id) +WHERE warpBackgroundRun.state = 'new' + AND warpSkyfile.quality = 0 + AND warpSkyfile.fault = 0 +-- WHERE hook %s +GROUP BY warp_bg_id +HAVING + COUNT(warpBackgroundSkyfile.skycell_id) = COUNT(warpSkyfile.skycell_id) + AND SUM(IF(warpBackgroundSkyfile.fault > 0, 1, 0)) = 0 + AND SUM(IF(warpBackgroundSkyfile.quality > 0, 1, 0)) != COUNT(*) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_chiprun_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_chiprun_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_chiprun_state.sql (revision 41104) @@ -0,0 +1,10 @@ +UPDATE chipBackgroundRun +SET chipBackgroundRun.state = '%s', + chipBackgroundRun.magicked = IF(chipBackgroundRun.magicked != 0, -1, 0) +WHERE chip_bg_id = %lld +AND (SELECT COUNT(class_id) + FROM chipBackgroundImfile + WHERE + chipBackgroundImfile.chip_bg_id = chipBackgroundRun.chip_bg_id + AND data_state != '%s' + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_imfile_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_imfile_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_imfile_data_state.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE chipBackgroundImfile + JOIN chipBackgroundRun USING(chip_bg_id) +SET data_state = '%s', + chipBackgroundImfile.magicked = IF(chipBackgroundImfile.magicked != 0, -1, 0) +WHERE chip_bg_id = %lld AND class_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_skyfile_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_skyfile_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_skyfile_data_state.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE warpBackgroundSkyfile + JOIN warpBackgroundRun USING(warp_bg_id) +SET data_state = '%s', + warpBackgroundSkyfile.magicked = IF(warpBackgroundSkyfile.magicked != 0, -1, 0) +WHERE warp_bg_id = %lld AND skycell_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_warprun_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_warprun_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_change_warprun_state.sql (revision 41104) @@ -0,0 +1,10 @@ +UPDATE warpBackgroundRun +SET state = '%s', + warpBackgroundRun.magicked = IF(warpBackgroundRun.magicked != 0, -1, 0) +WHERE warp_bg_id = %lld +AND (SELECT COUNT(skycell_id) + FROM warpBackgroundSkyfile + WHERE + warpBackgroundSkyfile.warp_bg_id = warpBackgroundRun.warp_bg_id + AND data_state != '%s' + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_chip.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + chipBackgroundImfile.*, + chipBackgroundRun.state, + chipBackgroundRun.workdir, + chipBackgroundRun.label, + chipBackgroundRun.data_group, + chipBackgroundRun.chip_id, + chipBackgroundRun.cam_id, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.filter, + rawExp.dateobs, + rawExp.ra, + rawExp.decl, + rawExp.exp_time +FROM chipBackgroundRun +JOIN chipBackgroundImfile USING(chip_bg_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_chipinputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_chipinputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_chipinputs.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + chipProcessedImfile.*, + camProcessedExp.path_base AS cam_path_base +FROM chipBackgroundRun +JOIN chipRun USING(chip_id) +JOIN chipProcessedImfile USING(chip_id) +LEFT JOIN camProcessedExp ON chipBackgroundRun.cam_id = camProcessedExp.cam_id +WHERE chipRun.state = 'full' + AND chipProcessedImfile.fault = 0 + AND chipProcessedImfile.quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_cleanedchip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_cleanedchip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_cleanedchip.sql (revision 41104) @@ -0,0 +1,3 @@ +UPDATE chipBackgroundRun +SET state = '%s' +WHERE state IN ('goto_cleaned', 'goto_scrubbed', 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_cleanedwarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_cleanedwarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_cleanedwarp.sql (revision 41104) @@ -0,0 +1,3 @@ +UPDATE warpBackgroundRun +SET state = '%s' +WHERE state IN ('goto_cleaned', 'goto_scrubbed', 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_definechip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_definechip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_definechip.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + chipRun.*, + camRun.cam_id, + chip_bg_id, + CURRENT_TIMESTAMP AS registered +FROM chipRun +JOIN rawExp USING(exp_id) +JOIN chipRun as altChipRun USING(exp_id) +JOIN camRun ON altChipRun.chip_id = camRun.chip_id +JOIN camProcessedExp USING(cam_id) +LEFT JOIN chipBackgroundRun ON chipBackgroundRun.chip_id = chipRun.chip_id -- labelHook %s +WHERE chipRun.state = 'full' AND camRun.state ='full' AND camProcessedExp.quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_definewarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_definewarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_definewarp.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT + warpRun.warp_id, + chipBackgroundRun.* + -- the following items are selected for aid in debugging + , + warpsChipRun.chip_id as warpschip_id, + chipRun.chip_id AS thischip_id, + warpBackgroundRun.warp_bg_id +FROM warpRun +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun as warpsChipRun ON camRun.chip_id = warpsChipRun.chip_id +JOIN rawExp USING(exp_id) +JOIN chipRun USING(exp_id) +JOIN chipBackgroundRun ON chipRun.chip_id = chipBackgroundRun.chip_id + AND (chipBackgroundRun.cam_id = camRun.cam_id OR chipBackgroundRun.cam_id = 0) +LEFT JOIN warpBackgroundRun ON chipBackgroundRun.chip_bg_id = warpBackgroundRun.chip_bg_id -- label hook %s +WHERE chipBackgroundRun.state = 'full' + AND warpRun.state IN ('full', 'cleaned', 'goto_cleaned') -- need warp to have completed so warpSkyCellMap is populated Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_listchip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_listchip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_listchip.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT chip_bg_id, + class_id, + chipProcessedImfile.path_base +FROM chipBackgroundRun + JOIN chipBackgroundImfile USING(chip_bg_id) + JOIN chipProcessedImfile USING(chip_id, class_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_listwarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_listwarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_listwarp.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT warp_bg_id, + skycell_id, + warpSkyfile.path_base +FROM warpBackgroundRun + JOIN warpBackgroundSkyfile USING(warp_bg_id) + JOIN warpSkyfile USING(warp_id, skycell_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupchipimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupchipimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupchipimfile.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + chipBackgroundImfile.*, + chipBackgroundRun.state, + chipBackgroundRun.workdir, + chipBackgroundRun.label +FROM chipBackgroundRun +JOIN chipBackgroundImfile + USING(chip_bg_id) +WHERE + ((chipBackgroundRun.state = 'goto_cleaned' AND (chipBackgroundImfile.data_state = 'full' + OR chipBackgroundImfile.data_state = 'update')) +OR + (chipBackgroundRun.state = 'goto_scrubbed' AND chipBackgroundImfile.data_state != 'scrubbed') +OR + (chipBackgroundRun.state = 'goto_purged' AND chipBackgroundImfile.data_state != 'purged')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupchiprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupchiprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupchiprun.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + chipBackgroundRun.*, + CONCAT_WS('.', exp_name, exp_id) AS exp_tag, + rawExp.camera +FROM chipBackgroundRun + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) +WHERE (chipBackgroundRun.state = 'goto_cleaned' + OR chipBackgroundRun.state = 'goto_purged' + OR chipBackgroundRun.state = 'goto_scrubbed' + ) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupwarprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupwarprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupwarprun.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT warpBackgroundRun.*, + CONCAT_WS('.', exp_name, exp_id) AS exp_tag, + rawExp.camera +FROM warpBackgroundRun + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) +WHERE (warpBackgroundRun.state = 'goto_cleaned' + OR warpBackgroundRun.state = 'goto_purged' + OR warpBackgroundRun.state = 'goto_scrubbed') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupwarpskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupwarpskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_pendingcleanupwarpskyfile.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + warpBackgroundSkyfile.*, + warpBackgroundRun.state, + warpBackgroundRun.workdir, + warpBackgroundRun.label +FROM warpBackgroundRun +JOIN warpBackgroundSkyfile + USING(warp_bg_id) +WHERE + ((warpBackgroundRun.state = 'goto_cleaned' AND (warpBackgroundSkyfile.data_state = 'full' + OR warpBackgroundSkyfile.data_state = 'update')) +OR + (warpBackgroundRun.state = 'goto_scrubbed' AND warpBackgroundSkyfile.data_state != 'scrubbed') +OR + (warpBackgroundRun.state = 'goto_purged' AND warpBackgroundSkyfile.data_state != 'purged')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_revertchip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_revertchip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_revertchip.sql (revision 41104) @@ -0,0 +1,4 @@ +DELETE chipBackgroundImfile +FROM chipBackgroundRun +JOIN chipBackgroundImfile USING(chip_bg_id) +WHERE chipBackgroundImfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_revertwarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_revertwarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_revertwarp.sql (revision 41104) @@ -0,0 +1,4 @@ +DELETE warpBackgroundSkyfile +FROM warpBackgroundRun +JOIN warpBackgroundSkyfile USING(warp_bg_id) +WHERE warpBackgroundSkyfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tochip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tochip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tochip.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT + chipBackgroundRun.*, + chipProcessedImfile.class_id, + rawExp.exp_tag, + rawExp.camera +FROM chipBackgroundRun +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN chipProcessedImfile USING(chip_id) +JOIN camRun using(cam_id) +JOIN chipRun AS bgsub_chipRun on camRun.chip_id = bgsub_chipRun.chip_id +JOIN chipProcessedImfile AS bgsub_chipProcessedImfile + ON (bgsub_chipRun.chip_id = bgsub_chipProcessedImfile.chip_id + AND bgsub_chipProcessedImfile.class_id = chipProcessedImfile.class_id) +LEFT JOIN chipBackgroundImfile + ON (chipBackgroundRun.chip_bg_id = chipBackgroundImfile.chip_bg_id + AND chipBackgroundImfile.class_id = chipProcessedImfile.class_id) +LEFT JOIN Label ON chipBackgroundRun.label = Label.label +WHERE chipBackgroundImfile.chip_bg_id IS NULL + AND chipRun.state = 'full' + AND chipProcessedImfile.fault = 0 + AND chipProcessedImfile.quality = 0 + AND bgsub_chipProcessedImfile.quality = 0 + AND (Label.active OR Label.active IS NULL) +-- WHERE hook %s +ORDER BY priority DESC, chip_bg_id, class_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tocleanchip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tocleanchip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tocleanchip.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + chipBackgroundRun.chip_bg_id, + chipBackgroundRun.state, + rawExp.camera +FROM chipBackgroundRun +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +WHERE chipBackgroundRun.state IN ('goto_cleaned', 'goto_scrubbed', 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tocleanwarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tocleanwarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_tocleanwarp.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + warpBackgroundRun.warp_bg_id, + warpBackgroundRun.state, + rawExp.camera +FROM warpBackgroundRun +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +WHERE warpBackgroundRun.state IN ('goto_cleaned', 'goto_scrubbed', 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_towarp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_towarp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_towarp.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT DISTINCT + warpBackgroundRun.*, + warpRun.tess_id, + skycell_id, + rawExp.exp_tag, + rawExp.camera +FROM warpBackgroundRun +JOIN warpRun USING(warp_id) +JOIN warpSkyCellMap USING(warp_id) +JOIN warpSkyfile USING(warp_id, skycell_id) +JOIN chipBackgroundRun USING(chip_bg_id) +JOIN chipBackgroundImfile USING(chip_bg_id, class_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN warpBackgroundSkyfile USING(warp_bg_id, skycell_id) +LEFT JOIN Label ON Label.label = warpBackgroundRun.label +WHERE warpBackgroundSkyfile.warp_bg_id IS NULL + AND chipBackgroundRun.state = 'full' + AND chipBackgroundImfile.fault = 0 + AND chipBackgroundImfile.quality = 0 + AND warpRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + AND (Label.active OR Label.active IS NULL) +-- WHERE hook %s +ORDER BY priority DESC, warp_bg_id, skycell_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_warp.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + warpBackgroundSkyfile.*, + warpBackgroundRun.state, + warpBackgroundRun.warp_id, + camRun.cam_id, + warpBackgroundRun.chip_bg_id, + warpBackgroundRun.workdir, + warpBackgroundRun.label, + warpRun.label as warp_label, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.filter, + rawExp.dateobs, + rawExp.ra, + rawExp.decl, + rawExp.exp_time +FROM warpBackgroundRun +JOIN warpBackgroundSkyfile USING(warp_bg_id) +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_warpinputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_warpinputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/bgtool_warpinputs.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT + chipBackgroundImfile.path_base AS chip_path_base, + chipBackgroundImfile.class_id, + chipBackgroundImfile.magicked, + camProcessedExp.path_base AS cam_path_base +FROM warpBackgroundRun +JOIN warpRun USING(warp_id) +JOIN warpSkyCellMap USING(warp_id) +JOIN warpSkyfile USING(warp_id, skycell_id) +JOIN chipBackgroundRun USING(chip_bg_id) +JOIN chipBackgroundImfile USING(chip_bg_id, class_id) +JOIN fakeRun USING(fake_id) +JOIN camProcessedExp ON camProcessedExp.cam_id = fakeRun.cam_id +WHERE warpRun.state IN ('full', 'cleaned', 'goto_cleaned') -- only need it to have been completed + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + AND chipBackgroundRun.state = 'full' + AND chipBackgroundImfile.fault = 0 + AND chipBackgroundImfile.quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_addprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_addprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_addprocessedexp.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT + camRun.*, + rawExp.exp_tag, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel, + chipRun.magicked AS chip_magicked, + camProcessedExp.path_base +FROM camRun +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +LEFT JOIN camProcessedExp + USING(cam_id) +LEFT JOIN camMask + ON camRun.label = camMask.label +WHERE + chipRun.state = 'full' + AND ((camRun.state = 'new' AND camProcessedExp.cam_id IS NULL) OR camRun.state = 'update') + AND camMask.label IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + camRun.* +FROM camRun +WHERE + camRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_export_processed_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_export_processed_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_export_processed_exp.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * from camProcessedExp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_export_run.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT camRun.* from camRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_chip_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_chip_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_chip_id.sql (revision 41104) @@ -0,0 +1,28 @@ +SELECT + chipRun.*, + rawExp.camera, + rawExp.telescope, + rawExp.dateobs, + rawExp.exp_tag, + rawExp.exp_type, + rawExp.filelevel, + rawExp.filter, + rawExp.airmass, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + rawExp.sat_pixel_frac, + rawExp.bg, + rawExp.bg_stdev, + rawExp.bg_mean_stdev, + rawExp.alt, + rawExp.az, + rawExp.ccd_temp, + rawExp.posang, + rawExp.object, + rawExp.sun_angle +FROM chipRun +JOIN rawExp + using(exp_id) +WHERE + chipRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_pendingimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_pendingimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_pendingimfile.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT + camRun.cam_id, + chipProcessedImfile.*, + (IF(rawExp.exp_time IS NOT NULL AND deteff_magref IS NOT NULL, + (2.5 * LOG10(rawExp.exp_time) + chipProcessedImfile.deteff_magref), + NULL)) + AS deteff_inst, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel +FROM camRun +JOIN chipRun + USING (chip_id) +JOIN chipProcessedImfile + USING (chip_id, exp_id) +JOIN rawExp + USING(exp_id) +JOIN rawImfile + USING(exp_id, class_id) +LEFT JOIN camProcessedExp + USING(cam_id) +LEFT JOIN camMask + ON camRun.label = camMask.label +WHERE + camMask.label IS NULL + AND rawImfile.ignored = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_processedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_processedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_find_processedexp.sql (revision 41104) @@ -0,0 +1,29 @@ +SELECT + camProcessedExp.*, + HEX(IFNULL(camProcessedExp.astrom_chips, 0)) AS good_astrom_chips, + camRun.workdir, + camRun.label, + camRun.data_group, + camRun.state, + camRun.magicked, + chipRun.chip_id, + chipRun.state as chip_state, + chipRun.magicked as chip_magicked, + rawExp.exp_tag, + rawExp.exp_name, + rawExp.exp_time, + rawExp.exp_id, + rawExp.camera, + rawExp.telescope, + rawExp.dateobs, + rawExp.filter, + rawExp.filelevel, + rawExp.magicked as raw_magicked, + rawExp.comment +FROM camRun +JOIN camProcessedExp + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingcleanupexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingcleanupexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingcleanupexp.sql (revision 41104) @@ -0,0 +1,16 @@ +SELECT + camProcessedExp.*, + camRun.state, + camRun.workdir, + camRun.label, + camRun.reduction, + camRun.expgroup, + camRun.dvodb, + camRun.tess_id, + camRun.end_stage +FROM camRun +JOIN camProcessedExp + USING(cam_id) +WHERE + (camRun.state = 'goto_cleaned' OR camRun.state = 'goto_purged' OR camRun.state = 'goto_scrubbed') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT + camRun.cam_id, + rawExp.camera, + rawExp.exp_tag, + camRun.workdir, + camRun.state +FROM camRun +JOIN chipRun +USING (chip_id) +JOIN rawExp +USING (exp_id) +WHERE + (camRun.state = 'goto_cleaned' OR camRun.state = 'goto_purged' OR camRun.state = 'goto_scrubbed') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_pendingexp.sql (revision 41104) @@ -0,0 +1,28 @@ +SELECT + camRun.*, + rawExp.exp_tag, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel, + chipRun.magicked AS chip_magicked, + camProcessedExp.path_base, + IFNULL(Label.priority, 10000) AS priority +FROM camRun +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +LEFT JOIN camProcessedExp + USING(cam_id) +LEFT JOIN camMask + ON camRun.label = camMask.label +LEFT JOIN Label + ON camRun.label = Label.label +WHERE + chipRun.state = 'full' + AND ((camRun.state = 'new' AND camProcessedExp.cam_id IS NULL) OR + (camRun.state = 'update' and camProcessedExp.fault = 0 and camProcessedExp.quality = 0)) + AND camMask.label IS NULL + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_queue_chip_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_queue_chip_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_queue_chip_id.sql (revision 41104) @@ -0,0 +1,34 @@ +-- camtool only operates on exposures so we can safely queue more then one at a +-- time without worrying about losing the track of the generated cam_id +INSERT INTO camRun + SElECT + 0, -- cam_id + chip_id, -- chip_id + '%s', -- state + '%s', -- workdir + '%s', -- workdir_state + '%s', -- label + '%s', -- data_group + '%s', -- dist_group + '%s', -- reduction + '%s', -- expgroup + '%s', -- dvodb + '%s', -- tess_id + '%s', -- end_stage + %lld, -- magicked + '%s', -- software ver + '%s', -- maskfrac_ref_npix + '%s', -- _ref_static + '%s', -- _ref_dynamic + '%s', -- _ref_magic + '%s', -- _ref_advisory + '%s', -- maskfrac_max_npix + '%s', -- _max_static + '%s', -- _max_dynamic + '%s', -- _max_magic + '%s', -- _max_advisory + '%s' -- note + FROM chipRun + WHERE + chipRun.state = 'full' + AND chipRun.chip_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_revertprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_revertprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_revertprocessedexp.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM camProcessedExp +USING camProcessedExp, camRun, chipRun, rawExp +WHERE + camRun.cam_id = camProcessedExp.cam_id + AND camRun.chip_id = chipRun.chip_id + AND chipRun.exp_id = rawExp.exp_id + AND camProcessedExp.fault != 0 + AND camRun.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_revertupdatedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_revertupdatedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/camtool_revertupdatedexp.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE camProcessedExp + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) +SET camProcessedExp.fault = 0 +WHERE + camRun.state = 'update' + AND camProcessedExp.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_change_exp_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_change_exp_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_change_exp_state.sql (revision 41104) @@ -0,0 +1,18 @@ +-- change state of chipRun from goto_cleaned to cleaned or goto_purged to purged +-- when all of the constituant imfiles are in the right state +-- arguments are new state (cleaned or purged) chip_id and new state again for +-- the chipProcessedImfile sub query +UPDATE chipRun +JOIN rawExp using(exp_id) + SET chipRun.state = '%s', chipRun.update_mode = 0 + -- set magicked hook %s + WHERE + chipRun.chip_id = %lld + AND (SELECT + COUNT(chip_id) + FROM chipProcessedImfile + WHERE + chipProcessedImfile.chip_id = chipRun.chip_id + AND chipProcessedImfile.quality = 0 + AND data_state != '%s' + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_change_imfile_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_change_imfile_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_change_imfile_data_state.sql (revision 41104) @@ -0,0 +1,12 @@ +-- handle changes in data_state. Used for the modes tocleanedimfile and topurgedimfile +-- args are new data_state, a possibly empty string for updating the magicked state, +-- chip_id and class_id +UPDATE chipProcessedImfile +JOIN chipRun USING(chip_id, exp_id) +JOIN rawImfile USING(exp_id, class_id) + SET + chipProcessedImfile.data_state = '%s' + -- set magicked hook %s +WHERE + chip_id = %lld + AND class_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_coalesce_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_coalesce_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_coalesce_run.sql (revision 41104) @@ -0,0 +1,1 @@ +select software_ver,maskfrac_npix,maskfrac_static,maskfrac_dynamic,maskfrac_magic,maskfrac_advisory from chipProcessedImfile where data_state = 'full' AND fault = 0 AND chip_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_completely_processed_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_completely_processed_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_completely_processed_exp.sql (revision 41104) @@ -0,0 +1,48 @@ +-- the output of this query must match the format of chipRun row +SELECT DISTINCT + chip_id, + exp_id, + state, + workdir, + workdir_state, + label, + data_group, + dist_group, + reduction, + expgroup, + dvodb, + tess_id, + end_stage, + imfile_magicked as magicked, + 0 as update_mode, + note +FROM + (SELECT + chipRun.*, + rawImfile.class_id as rawimfile_class_id, + chipProcessedImfile.class_id, + -- XXX using chipProcessedImfile assumes that all imfile's have + -- the same magicked value if that isn't right then more than one + -- row will be returned. In practice this is the case but bugs could + -- cause it to not be true. + -- We could use rawExp.magicked but that would make it possible for + -- the chipRun to have a different magicked value than the imfiles + chipProcessedImfile.magicked AS imfile_magicked + FROM chipRun + JOIN rawImfile + ON chipRun.exp_id = rawImfile.exp_id + AND rawImfile.ignored = 0 + LEFT JOIN chipProcessedImfile + ON chipRun.chip_id = chipProcessedImfile.chip_id + AND rawImfile.exp_id = chipProcessedImfile.exp_id + AND rawImfile.class_id = chipProcessedImfile.class_id + WHERE + chipRun.state = 'new' + GROUP BY + chipRun.chip_id, + chipRun.exp_id + HAVING + COUNT(rawImfile.class_id) = COUNT(chipProcessedImfile.class_id) + AND SUM(chipProcessedImfile.fault) = 0 + AND SUM(chipProcessedImfile.quality > 0) != COUNT(*) + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_definecopy.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_definecopy.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_definecopy.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + rawExp.*, + newExp.label, + chipOld.workdir AS old_workdir, + chipOld.data_group AS old_data_group, + chipOld.state AS old_state +FROM chipRun AS chipOld +JOIN rawExp USING(exp_id) +JOIN newExp using (exp_id) +LEFT JOIN chipRun AS chipNew + ON chipNew.exp_id = chipOld.exp_id + AND chipNew.label = '%s' +WHERE chipNew.chip_id IS NULL + AND rawExp.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + chipRun.* +FROM chipRun +WHERE + chipRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_dropprocessedimfile_delete.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_dropprocessedimfile_delete.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_dropprocessedimfile_delete.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM magicDSFile + USING magicDSRun, magicDSFile, chipRun, chipProcessedImfile +WHERE chipRun.chip_id = chipProcessedImfile.chip_id + AND magicDSRun.stage ='chip' + AND magicDSRun.stage_id = chipRun.chip_id + AND magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = chipProcessedImfile.class_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_dropprocessedimfile_update.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_dropprocessedimfile_update.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_dropprocessedimfile_update.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE chipProcessedImfile LEFT JOIN camRun USING(chip_id) + SET chipProcessedImfile.quality = %d, + chipProcessedImfile.data_state = 'full', + chipProcessedImfile.magicked = 0, + chipProcessedImfile.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + chipImfile.* +FROM chipImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_processed_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_processed_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_processed_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + chipProcessedImfile.* +FROM chipProcessedImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_export_run.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + chipRun.* +FROM chipRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_find_rawexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_find_rawexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_find_rawexp.sql (revision 41104) @@ -0,0 +1,10 @@ +-- this query is used to find potental rawExps to be queued for chiptool +-- processeing +SELECT DISTINCT + rawExp.*, + newExp.label +FROM rawExp +JOIN newExp using (exp_id) +LEFT JOIN chipRun ON chipRun.exp_id = rawExp.exp_id -- JOIN hook %s +WHERE + rawExp.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_find_unprocessed_imfile.pl =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_find_unprocessed_imfile.pl (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_find_unprocessed_imfile.pl (revision 41104) @@ -0,0 +1,8 @@ +SELECT DISTINCT + chipInputImfile.* +FROM chipInputImfile +LEFT JOIN chipProcessedImfile + USING(chip_id, class_id) +WHERE + chipProcessedImfile.chip_id IS NULL + AND chipProcessedImfile.class_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_listrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_listrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_listrun.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT DISTINCT + chipRun.chip_id, + chipRun.state, + chipRun.workdir, + chipRun.label, + chipRun.data_group, + chipRun.dist_group, + chipRun.note, + chipRun.magicked, + chipRun.reduction, + camRun.cam_id, + camRun.state AS camRun_state, + camProcessedExp.path_base AS cam_path_base, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.filter, + rawExp.dateobs, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + rawExp.magicked AS raw_magicked, + magicDSRun.state AS dsRun_state, + IFNULL(magicDSRun.magic_ds_id, 0) AS magic_ds_id +FROM chipRun +JOIN rawExp + USING(exp_id) +LEFT JOIN camRun USING(chip_id) +LEFT JOIN camProcessedExp USING(cam_id) +LEFT JOIN magicDSRun + ON stage_id = chip_id AND stage = 'chip' AND magicDSRun.re_place AND magicDSRun.state != 'drop' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanupimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanupimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanupimfile.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT + chipProcessedImfile.*, + chipRun.state, + chipRun.workdir, + chipRun.label, + chipRun.reduction, + chipRun.expgroup, + chipRun.dvodb, + chipRun.tess_id, + chipRun.end_stage +FROM chipRun +JOIN chipProcessedImfile + USING(chip_id) +WHERE + ((chipRun.state = 'goto_cleaned' + AND (chipProcessedImfile.data_state = 'full' + OR chipProcessedImfile.data_state = 'update' + OR chipProcessedImfile.data_state = 'error_cleaned') + AND chipProcessedImfile.quality = 0) +OR + (chipRun.state = 'goto_scrubbed' AND chipProcessedImfile.data_state != 'scrubbed') +OR + (chipRun.state = 'goto_purged' AND chipProcessedImfile.data_state != 'purged')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanupimfile_all.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanupimfile_all.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanupimfile_all.sql (revision 41104) @@ -0,0 +1,16 @@ +-- list of chipProcessedImfiles for runs in a goto_cleaned like state without regard to +-- data_state or quality. Used for confirming that a run is really in the cleaned state +SELECT + chipProcessedImfile.*, + chipRun.state, + chipRun.workdir, + chipRun.label, + chipRun.reduction, + chipRun.expgroup, + chipRun.dvodb, + chipRun.tess_id, + chipRun.end_stage +FROM chipRun +JOIN chipProcessedImfile + USING(chip_id) +WHERE ( chipRun.state = 'goto_cleaned' OR chipRun.state = 'goto_scrubbed' OR chipRun.state = 'goto_purged' ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT DISTINCT + chipRun.chip_id, + rawExp.camera, + rawExp.exp_tag, + chipRun.state, + chipRun.workdir, + chipRun.label, + IFNULL(Label.priority, 10000) AS priority +FROM chipRun +JOIN rawExp +USING (exp_id) +LEFT JOIN Label ON chipRun.label = Label.label +WHERE + (chipRun.state = 'goto_cleaned' OR chipRun.state = 'goto_scrubbed' OR chipRun.state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_pendingimfile.sql (revision 41104) @@ -0,0 +1,42 @@ +SELECT + chipRun.*, + chipImfile.chip_imfile_id, + rawImfile.class_id, + rawImfile.uri, + rawImfile.magicked as raw_magicked, + (rawImfile.user_1 is not NULL and rawImfile.user_1 > 0.5) as deburned, + rawImfile.burntool_state as burntool_state, + rawImfile.video_cells as video_cells, + rawExp.exp_tag, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel, + IFNULL(Label.priority, 10000) AS priority, + chipProcessedImfile.path_base +FROM chipRun +JOIN rawExp + USING(exp_id) +JOIN rawImfile + ON rawExp.exp_id = rawImfile.exp_id + AND rawImfile.ignored = 0 +JOIN chipImfile + USING(chip_id, class_id) +LEFT JOIN chipProcessedImfile + ON chipRun.chip_id = chipProcessedImfile.chip_id + AND rawImfile.exp_id = chipProcessedImfile.exp_id + AND rawImfile.class_id = chipProcessedImfile.class_id +LEFT JOIN chipMask + ON chipRun.label = chipMask.label +LEFT JOIN Label ON chipRun.label = Label.label +WHERE + ((chipRun.state = 'new' + AND chipProcessedImfile.chip_id IS NULL + AND chipProcessedImfile.exp_id IS NULL + AND chipProcessedImfile.class_id IS NULL + AND chipMask.label IS NULL) + OR + (chipRun.state = 'update' + AND chipProcessedImfile.data_state = 'update' + AND chipProcessedImfile.fault = 0)) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_processedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_processedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_processedimfile.sql (revision 41104) @@ -0,0 +1,47 @@ +SELECT DISTINCT + chipProcessedImfile.chip_id, + chipImfile.chip_imfile_id, + chipProcessedImfile.class_id, + chipProcessedImfile.uri, + chipProcessedImfile.bg, + chipProcessedImfile.bg_stdev, + chipProcessedImfile.bg_mean_stdev, + chipProcessedImfile.path_base, + chipProcessedImfile.magicked, + chipProcessedImfile.data_state, + chipProcessedImfile.fault, + chipProcessedImfile.quality, + chipRun.state, + chipRun.workdir, + chipRun.label, + chipRun.data_group, + rawExp.exp_id, + rawExp.exp_tag, + rawExp.exp_name, + rawExp.filter, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel, + rawExp.dateobs, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + rawImfile.magicked AS raw_magicked, + rawImfile.burntool_state, + IFNULL(magicDSRun.magic_ds_id, 0) AS magic_ds_id, + IFNULL(magicDSRun.state,0) AS dsRun_state, + IFNULL(magicDSFile.fault,0) AS dsFile_fault, + IFNULL(magicDSFile.data_state,0) AS dsFile_data_state +FROM chipRun +JOIN chipImfile + USING(chip_id) +JOIN chipProcessedImfile + USING(chip_id, class_id) +JOIN rawExp + ON chipProcessedImfile.exp_id = rawExp.exp_id +JOIN rawImfile + ON rawExp.exp_id = rawImfile.exp_id + AND chipProcessedImfile.class_id = rawImfile.class_id +LEFT JOIN magicDSRun + ON stage_id = chip_id AND stage = 'chip' AND magicDSRun.re_place AND magicDSRun.state != 'drop' +LEFT JOIN magicDSFile ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id AND chipProcessedImfile.class_id = magicDSFile.component Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertcleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertcleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertcleanup.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE chipProcessedImfile + JOIN chipRun using(chip_id,exp_id) +SET chipProcessedImfile.data_state = 'full', + chipRun.state = '%s' +WHERE + chipRun.state = '%s' + AND chipProcessedImfile.data_state = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertprocessedimfile.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM chipProcessedImfile +USING chipProcessedImfile, chipRun, rawExp +WHERE + chipRun.chip_id = chipProcessedImfile.chip_id + AND rawExp.exp_id = chipProcessedImfile.exp_id + AND chipRun.state = 'new' + AND chipProcessedImfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertupdatedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertupdatedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_revertupdatedimfile.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE chipProcessedImfile + JOIN chipRun using(chip_id,exp_id) + JOIN rawExp using(exp_id) +SET chipProcessedImfile.fault = 0 +WHERE + chipRun.state = 'update' + AND chipProcessedImfile.data_state = 'update' + AND chipProcessedImfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_run.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + chipRun.* +FROM chipRun +JOIN rawExp + USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_runstate.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_runstate.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_runstate.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + chip_id, + chipRun.state, + exp_id, + magic_ds_id, + magicDSRun.state as magic_ds_state +FROM chipRun + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun on stage = 'chip' and stage_id = chip_id and re_place + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_setimfiletoupdate.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_setimfiletoupdate.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_setimfiletoupdate.sql (revision 41104) @@ -0,0 +1,18 @@ +UPDATE chipRun + JOIN chipProcessedImfile USING(chip_id) + -- LEFT JOIN magicDSRun ON (stage_id = chip_id AND stage = 'chip') + -- LEFT JOIN magicDSFile ON (magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + -- AND component = class_id) +SET chipRun.state = 'update', + chipProcessedImfile.data_state = 'update', + chipProcessedImfile.fault = 0 + -- set hook %s +WHERE chip_id = %lld + AND (chipRun.state = 'cleaned' OR chipRun.state = 'update') + AND (chipProcessedImfile.data_state = 'cleaned') + AND (chipProcessedImfile.quality = 0) + -- don't queue update if the associated magicDSFile exists and isn't cleaned + -- AND (chipRun.magicked = 0 + -- OR ((magicDSRun.state = 'cleaned' OR magicDSRun.state = 'update') + -- AND (magicDSFile.data_state = 'cleaned' OR magicDSFile.data_state = 'update')) + -- ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_unmasked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_unmasked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/chiptool_unmasked.sql (revision 41104) @@ -0,0 +1,43 @@ +-- SELECT +-- * +-- FROM +-- (SELECT +-- chipMask.label as label, +-- count(chipRun.chip_id) as n_chipruns +-- FROM +-- chipMask +-- LEFT JOIN +-- chipRun +-- USING +-- (label) +-- WHERE chipRun.chip_id IS NOT NULL +-- GROUP BY label) as chipMask + +-- SELECT +-- * +-- FROM +-- SELECT +-- chipRun.*, +-- rawExp.*, +-- chipRun.chip_id as n_chipruns +-- FROM +-- chipRun +-- JOIN rawExp ON chipRun.exp_id = rawExp.exp_id +-- LEFT JOIN chipMask +-- USING (label) +-- WHERE chipMask.label IS NULL +-- AND filter = 'r' +-- AND chipRun.exp_id = 61 +-- GROUP BY chipRun.label) as chipUnmask + +SELECT + * +FROM + (SELECT + chipRun.label, + count(chipRun.chip_id) as n_chipruns + FROM chipRun + LEFT JOIN chipMask + USING (label) + WHERE chipMask.label IS NULL + GROUP BY chipRun.label) as chipUnmask Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_search.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_search.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_search.sql (revision 41104) @@ -0,0 +1,29 @@ +-- this query needs to use the same fields in both of the tables in +-- the union statement, but we need to report the +-- detRunSummary.iteration in the first case, and this is missing in +-- the second case. + +SELECT DISTINCT + det_id, + good_iteration as iteration, + filelevel +FROM + (SELECT DISTINCT + detRun.*, + detRunSummary.iteration as good_iteration + FROM detRun + JOIN detRunSummary + USING(det_id) + WHERE + detRun.mode = 'master' + AND detRunSummary.accept = 1 + UNION + SELECT DISTINCT + detRun.*, + detRun.iteration as good_iteration + FROM detRun + WHERE + (detRun.mode = 'register' OR detRun.mode = 'correction') + ) as Foo +WHERE + (state = 'stop') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_select.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT DISTINCT + * +FROM + (SELECT DISTINCT + detRun.state, + detNormalizedImfile.* + FROM detNormalizedImfile + JOIN detRun + USING(det_id) + JOIN detRunSummary + ON detNormalizedImfile.det_id = detRunSummary.det_id + AND detNormalizedImfile.iteration = detRunSummary.iteration + WHERE + detRun.mode = 'master' + AND detRunSummary.accept = 1 + UNION + SELECT DISTINCT + detRun.state, + detRegisteredImfile.* + FROM detRegisteredImfile + JOIN detRun + USING(det_id) + WHERE + (detRun.mode = 'register' OR detRun.mode = 'correction') + ) as Foo +WHERE + (state = 'stop' OR state = 'register') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_show.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_show.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/detselect_show.sql (revision 41104) @@ -0,0 +1,35 @@ +-- this query needs to use the same fields in both of the tables in +-- the union statement, but we need to report the +-- detRunSummary.iteration in the first case, and this is missing in +-- the second case. + +SELECT DISTINCT + det_id, + good_iteration as iteration, + filelevel, + det_type, + mode, + registered, + filter, + time_begin, + time_end +FROM + (SELECT DISTINCT + detRun.*, + detRunSummary.iteration as good_iteration + FROM detRun + JOIN detRunSummary + USING(det_id) + WHERE + detRun.mode = 'master' + AND detRunSummary.accept = 1 + UNION + SELECT DISTINCT + detRun.*, + detRun.iteration as good_iteration + FROM detRun + WHERE + (detRun.mode = 'register' OR detRun.mode = 'correction') + ) as Foo +WHERE + (state = 'stop') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_addprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_addprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_addprocessedexp.sql (revision 41104) @@ -0,0 +1,34 @@ +SELECT DISTINCT + detProcessedImfile.det_id, + detRun.iteration, + detRun.det_type, + detProcessedImfile.exp_id +FROM detRun +JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration +JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id +JOIN detProcessedImfile + ON detInputExp.det_id = detProcessedImfile.det_id + AND detInputExp.exp_id = detProcessedImfile.exp_id +LEFT JOIN detProcessedExp + ON detInputExp.det_id = detProcessedExp.det_id + AND detProcessedImfile.exp_id= detProcessedExp.exp_id +LEFT JOIN rawImfile + ON detInputExp.exp_id = rawImfile.exp_id + AND detProcessedImfile.class_id = rawImfile.class_id +WHERE + detRun.state = 'run' + AND (detRun.mode = 'master' or detRun.mode = 'verify') + AND detProcessedExp.det_id IS NULL + AND detProcessedExp.exp_id IS NULL + AND detInputExp.include = 1 + AND detRun.det_id = %lld + AND detProcessedImfile.exp_id = %lld +GROUP BY + detProcessedImfile.class_id, + rawImfile.class_id, + detRun.det_id +HAVING + COUNT(detProcessedImfile.class_id) = COUNT(rawImfile.class_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_childlessrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_childlessrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_childlessrun.sql (revision 41104) @@ -0,0 +1,12 @@ +-- dettool -childlessrun + +SELECT DISTINCT + detRun.* + FROM detRun + LEFT JOIN detRun as foo + ON foo.ref_det_id = detRun.det_id +-- XXX do we need to restrict to foo.ref_iter = detRun.iteration ? + WHERE + detRun.state = 'stop' + AND detRun.mode = 'master' + AND foo.det_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_definebydetrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_definebydetrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_definebydetrun.sql (revision 41104) @@ -0,0 +1,13 @@ +-- dettool -definebydetrun + +INSERT INTO detInputExp + SELECT + %lld, + 0, + detResidExp.exp_id, + detResidExp.accept + FROM detResidExp + JOIN rawExp + USING(exp_id) + WHERE det_id = %lld + AND iteration = %d Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_detrunsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_detrunsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_detrunsummary.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT DISTINCT + detRunSummary.*, + detRun.det_type, + detRun.mode + FROM detRun + JOIN detRunSummary + USING(det_id, iteration) + WHERE + detRun.state = 'run' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedexp.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + detNormalizedExp.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detNormalizedExp + USING(det_id,iteration) +WHERE + (detNormalizedExp.data_state = 'cleaned' + OR detNormalizedExp.data_state = 'scrubbed' + OR detNormalizedExp.data_state = 'purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedimfile.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + detNormalizedImfile.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detNormalizedImfile + USING(det_id,iteration) +WHERE + (detNormalizedImfile.data_state = 'cleaned' + OR detNormalizedImfile.data_state = 'scrubbed' + OR detNormalizedImfile.data_state = 'purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedstat.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedstat.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_normalizedstat.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + detNormalizedStatImfile.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detNormalizedStatImfile + USING(det_id,iteration) +WHERE + (detNormalizedStatImfile.data_state = 'cleaned' + OR detNormalizedStatImfile.data_state = 'scrubbed' + OR detNormalizedStatImfile.data_state = 'purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_processedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_processedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_processedexp.sql (revision 41104) @@ -0,0 +1,11 @@ +-- need to restrict to a single detRunSummary (require all to say 'cleaned'?) +SELECT + detProcessedExp.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detProcessedExp + USING(det_id) +WHERE + (detProcessedExp.data_state = 'cleaned' + OR detProcessedExp.data_state = 'scrubbed' + OR detProcessedExp.data_state = 'purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_processedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_processedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_processedimfile.sql (revision 41104) @@ -0,0 +1,11 @@ +-- need to restrict to a single detRunSummary (require all to say 'cleaned'?) +SELECT + detProcessedImfile.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detProcessedImfile + USING(det_id) +WHERE + (detProcessedImfile.data_state = 'cleaned' + OR detProcessedImfile.data_state = 'scrubbed' + OR detProcessedImfile.data_state = 'purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_residexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_residexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_residexp.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + detResidExp.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detResidExp + USING(det_id,iteration) +WHERE + (detResidExp.data_state = 'cleaned' + OR detResidExp.data_state = 'scrubbed' + OR detResidExp.data_state = 'purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_residimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_residimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_residimfile.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + detResidImfile.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detResidImfile + USING(det_id,iteration) +WHERE + (detResidImfile.data_state = 'cleaned' + OR detResidImfile.data_state = 'scrubbed' + OR detResidImfile.data_state = 'purged') + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_stacked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_stacked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_donecleanup_stacked.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + detStackedImfile.*, + detRunSummary.data_state +FROM detRunSummary +JOIN detStackedImfile + USING(det_id,iteration) +WHERE + (detStackedImfile.data_state = 'cleaned' + OR detStackedImfile.data_state = 'scrubbed' + OR detStackedImfile.data_state = 'purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_input_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_input_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_input_exp.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detInputExp.* +FROM detInputExp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_exp.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detNormalizedExp.* +FROM detNormalizedExp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detNormalizedImfile.* +FROM detNormalizedImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_stat_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_stat_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_normalized_stat_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detNormalizedStatImfile.* +FROM detNormalizedStatImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_processed_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_processed_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_processed_exp.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detProcessedExp.* +FROM detProcessedExp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_processed_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_processed_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_processed_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detProcessedImfile.* +FROM detProcessedImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_registered_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_registered_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_registered_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detRegisteredImfile.* +FROM detRegisteredImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_resid_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_resid_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_resid_exp.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detResidExp.* +FROM detResidExp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_resid_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_resid_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_resid_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detResidImfile.* +FROM detResidImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_run.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detRun.* +FROM detRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_run_summary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_run_summary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_run_summary.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detRunSummary.* +FROM detRunSummary Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_stacked_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_stacked_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_export_stacked_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + detStackedImfile.* +FROM detStackedImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_find_completed_runs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_find_completed_runs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_find_completed_runs.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT DISTINCT + det_id, + iteration +FROM + (SELECT DISTINCT + detRun.det_id, + detRun.iteration, + detInputExp.exp_id + FROM detRun + LEFT JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration + LEFT JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + LEFT JOIN detResidExp + ON detRun.det_id = detResidExp.det_id + AND detRun.iteration = detResidExp.iteration + AND detInputExp.exp_id = detResidExp.exp_id + WHERE + detRun.state = 'run' + GROUP BY + detRun.det_id, + detRun.iteration + HAVING + COUNT(detResidExp.exp_id) = COUNT(detInputExp.exp_id) + ) AS residdetrun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_input.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_input.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_input.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT DISTINCT * + FROM detInputExp + JOIN rawExp + USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedexp.sql (revision 41104) @@ -0,0 +1,8 @@ +SELECT + detNormalizedExp.* + FROM detNormalizedExp + JOIN detRun + USING(det_id, iteration) + WHERE + detRun.state = 'run' + AND detRun.mode = 'master' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedimfile.sql (revision 41104) @@ -0,0 +1,8 @@ +SELECT + detNormalizedImfile.* + FROM detNormalizedImfile + JOIN detRun + USING(det_id, iteration) + WHERE + detRun.state = 'run' + AND detRun.mode = 'master' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedstat.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedstat.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_normalizedstat.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + * +FROM + (SELECT + detNormalizedStatImfile.* + FROM detNormalizedStatImfile + JOIN detRun + USING(det_id, iteration) + WHERE + detRun.state = 'run' + AND detRun.mode = 'master' + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pending.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pending.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pending.sql (revision 41104) @@ -0,0 +1,10 @@ +-- dettool -pending : not sure this is still used (EAM : 2008.07.10) + +SELECT + rawExp.* + FROM rawExp + LEFT JOIN detInputExp + ON rawExp.exp_id = detInputExp.exp_id + WHERE + detInputExp.exp_id IS NULL + AND rawExp.object != 'object' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_detrunsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_detrunsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_detrunsummary.sql (revision 41104) @@ -0,0 +1,42 @@ +SELECT DISTINCT detRunSummary.* +FROM detRunSummary + JOIN detProcessedImfile USING (det_id) + JOIN detProcessedExp USING (det_id) + JOIN detStackedImfile USING (det_id,iteration) + JOIN detResidExp USING (det_id,iteration) + JOIN detResidImfile USING (det_id,iteration) + JOIN detNormalizedStatImfile USING (det_id,iteration) + JOIN detNormalizedImfile USING (det_id,iteration) + JOIN detNormalizedExp USING (det_id,iteration) +WHERE (!( + ( + detRunSummary.data_state = 'full' OR + detRunSummary.data_state = 'cleaned' + ) OR + ( + !((detProcessedImfile.data_state = 'cleaned' OR + detProcessedImfile.data_state = 'scrubbed' OR + detProcessedImfile.data_state = 'purged') AND + (detProcessedExp.data_state = 'cleaned' OR + detProcessedExp.data_state = 'scrubbed' OR + detProcessedExp.data_state = 'purged') AND + (detStackedImfile.data_state = 'cleaned' OR + detStackedImfile.data_state = 'scrubbed' OR + detStackedImfile.data_state = 'purged') AND + (detResidExp.data_state = 'cleaned' OR + detResidExp.data_state = 'scrubbed' OR + detResidExp.data_state = 'purged') AND + (detResidImfile.data_state = 'cleaned' OR + detResidImfile.data_state = 'scrubbed' OR + detResidImfile.data_state = 'purged') AND + (detNormalizedStatImfile.data_state = 'cleaned' OR + detNormalizedStatImfile.data_state = 'scrubbed' OR + detNormalizedStatImfile.data_state = 'purged') AND + (detNormalizedImfile.data_state = 'cleaned' OR + detNormalizedImfile.data_state = 'scrubbed' OR + detNormalizedImfile.data_state = 'purged') AND + (detNormalizedExp.data_state = 'cleaned' OR + detNormalizedExp.data_state = 'scrubbed' OR + detNormalizedExp.data_state = 'purged') + ) + ))) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedexp.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + detNormalizedExp.*, + rawExp.camera +FROM detNormalizedExp +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detNormalizedExp.data_state = 'goto_cleaned' + OR detNormalizedExp.data_state = 'goto_scrubbed' + OR detNormalizedExp.data_state = 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedimfile.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + detNormalizedImfile.*, + rawExp.camera +FROM detNormalizedImfile +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detNormalizedImfile.data_state = 'goto_cleaned' + OR detNormalizedImfile.data_state = 'goto_scrubbed' + OR detNormalizedImfile.data_state = 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedstat.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedstat.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_normalizedstat.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + detNormalizedStatImfile.*, + rawExp.camera +FROM detNormalizedStatImfile +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detNormalizedStatImfile.data_state = 'goto_cleaned' + OR detNormalizedStatImfile.data_state = 'goto_scrubbed' + OR detNormalizedStatImfile.data_state = 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_processedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_processedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_processedexp.sql (revision 41104) @@ -0,0 +1,13 @@ +-- need to restrict to a single detRunSummary (require all to say 'cleaned'?) +SELECT DISTINCT + detProcessedExp.*, + rawExp.camera +FROM detProcessedExp +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detProcessedExp.data_state = 'goto_cleaned' + OR detProcessedExp.data_state = 'goto_scrubbed' + OR detProcessedExp.data_state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_processedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_processedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_processedimfile.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + detProcessedImfile.*, + rawExp.camera + FROM detProcessedImfile +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detProcessedImfile.data_state = 'goto_cleaned' + OR detProcessedImfile.data_state = 'goto_scrubbed' + OR detProcessedImfile.data_state = 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_residexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_residexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_residexp.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT DISTINCT + detResidExp.*, + rawExp.camera +FROM detResidExp +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detResidExp.data_state = 'goto_cleaned' + OR detResidExp.data_state = 'goto_scrubbed' + OR detResidExp.data_state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_residimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_residimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_residimfile.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT DISTINCT + detResidImfile.*, + rawExp.camera +FROM detResidImfile +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detResidImfile.data_state = 'goto_cleaned' + OR detResidImfile.data_state = 'goto_scrubbed' + OR detResidImfile.data_state = 'goto_purged') + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_stacked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_stacked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_pendingcleanup_stacked.sql (revision 41104) @@ -0,0 +1,12 @@ +ySELECT DISTINCT + detStackedImfile.*, + rawExp.camera +FROM detStackedImfile +JOIN detInputExp + USING(det_id,exp_id) +JOIN rawExp + USING(exp_id) +WHERE + (detStackedImfile.data_state = 'goto_cleaned' + OR detStackedImfile.data_state = 'goto_scrubbed' + OR detStackedImfile.data_state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_processedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_processedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_processedimfile.sql (revision 41104) @@ -0,0 +1,14 @@ +-- is this DISTINCT needed? +SELECT DISTINCT + detRun.det_type, + rawExp.exp_time, + detProcessedImfile.* +FROM detProcessedImfile +JOIN detRun + USING(det_id) +JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration + AND detProcessedImfile.exp_id = detInputExp.exp_id +JOIN rawExp + ON rawExp.exp_id = detProcessedImfile.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_raw.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + detInputExp.det_id, + detRun.det_type, + rawImfile.* +FROM detRun +JOIN detInputExp + USING(det_id) +JOIN rawImfile + ON detInputExp.exp_id = rawImfile.exp_id +WHERE + detRun.state = 'run' + AND detRun.mode = 'master' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_residexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_residexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_residexp.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + detRun.mode, + detResidExp.*, + detInputExp.include + FROM detRun + JOIN detInputExp + USING(det_id, iteration) + JOIN detResidExp + USING(det_id, iteration, exp_id) + WHERE + detRun.state = 'run' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_residimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_residimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_residimfile.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + detRun.det_type, + detRun.mode, + rawExp.exp_time, + detResidImfile.* +FROM detResidImfile +JOIN detRun + USING(det_id, iteration) +JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration + AND detResidImfile.exp_id = detInputExp.exp_id +JOIN rawExp + ON rawExp.exp_id = detResidImfile.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertdetrunsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertdetrunsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertdetrunsummary.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detRunSummary +USING detRunSummary, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detRunSummary.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedexp.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detNormalizedExp +USING detNormalizedExp, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detNormalizedExp.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedimfile.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detNormalizedImfile +USING detNormalizedImfile, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detNormalizedImfile.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedstat.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedstat.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertnormalizedstat.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detNormalizedStatImfile +USING detNormalizedStatImfile, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detNormalizedStatImfile.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertprocessedexp.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detProcessedExp +USING detProcessedExp, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detProcessedExp.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertprocessedimfile.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detProcessedImfile +USING detProcessedImfile, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detProcessedImfile.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertresidexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertresidexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertresidexp.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detResidExp +USING detResidExp, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detResidExp.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertresidimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertresidimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertresidimfile.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detResidImfile +USING detResidImfile, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detResidImfile.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertstacked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertstacked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_revertstacked.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM detStackedImfile +USING detStackedImfile, detRun +WHERE + fault != 0 +AND + state = 'run' +AND + detStackedImfile.det_id = detRun.det_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_runs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_runs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_runs.sql (revision 41104) @@ -0,0 +1,2 @@ + +SELECT * from detRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_stacked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_stacked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_stacked.sql (revision 41104) @@ -0,0 +1,12 @@ +-- select detStackedImfile.* +-- by: +-- where det_id, iteration, class_id is not in detNormalizedImfile + +SELECT + detStackedImfile.* + FROM detStackedImfile + JOIN detRun + USING(det_id, iteration) + WHERE + detRun.state = 'run' + AND detRun.mode = 'master' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_start_new_iteration.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_start_new_iteration.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_start_new_iteration.sql (revision 41104) @@ -0,0 +1,33 @@ +-- select detRun.iteration +-- select detInputExp.exp_id +-- select detResidExp.accept +-- by: +-- find the current iteration bassed on det_id +-- find all exp_ids in the current det_id/iteration from detInputExp +-- find all exp_ids in the current det_id/iteration from detResidExp +-- compare the counts of exp_ids + +SELECT * FROM + (SELECT DISTINCT + detRun.det_id AS det_id, + detRun.iteration, + detInputExp.exp_id, + detResidExp.accept + FROM detRun + JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration + JOIN detResidExp + ON detRun.det_id = detResidExp.det_id + AND detRun.iteration = detResidExp.iteration + AND detInputExp.exp_id = detResidExp.exp_id + WHERE + detRun.state = 'run' + AND detRun.mode = 'master' + GROUP BY + detRun.det_id, + detRun.iteration, + detInputExp.exp_id + HAVING + COUNT(detResidExp.exp_id) = COUNT(detInputExp.exp_id) + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_stop_completed_correct_runs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_stop_completed_correct_runs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_stop_completed_correct_runs.sql (revision 41104) @@ -0,0 +1,30 @@ +-- det1 is the 'correction' detRun, det2 is the 'parent' detRun +UPDATE detRun +SET state = 'stop' +WHERE det_id = (SELECT det_id FROM + (SELECT + det1.det_id, + imfile.class_id, + detRegisteredImfile.class_id as reg_class_id + FROM detRun AS det1 + INNER JOIN detRun as det2 + ON det1.ref_det_id = det2.det_id + -- do we need to restrict by iteration? not clear this logic below is right or tested + JOIN detNormalizedImfile as imfile + ON det2.det_id = imfile.det_id + AND det2.iteration = imfile.iteration + LEFT JOIN detRegisteredImfile + ON det1.det_id = detRegisteredImfile.det_id + AND det1.iteration = detRegisteredImfile.iteration + AND imfile.class_id = detRegisteredImfile.class_id + WHERE + det1.state = 'run' + AND det1.mode = 'correction' + AND det2.state = 'stop' + GROUP BY + det1.det_id + HAVING + COUNT(imfile.class_id) = COUNT(reg_class_id) + AND SUM(imfile.fault) = 0 + ) as Foo +) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tocorrectexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tocorrectexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tocorrectexp.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT * FROM detRun +WHERE + detRun.mode = 'correction' + AND detRun.state = 'run' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tocorrectimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tocorrectimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tocorrectimfile.sql (revision 41104) @@ -0,0 +1,32 @@ +-- select imfiles from detRun 1 (the reference detRun) that match the desired correction in detRun 2 +SELECT + det1.*, + imfile.class_id, + imfile.uri, + imfile.bg, + imfile.bg_stdev, + imfile.bg_mean_stdev, + imfile.user_1, + imfile.user_2, + imfile.user_3, + imfile.user_4, + imfile.user_5, + imfile.path_base, + imfile.fault +FROM detRun AS det1 +JOIN detRun as det2 + ON det1.ref_det_id = det2.det_id +JOIN detNormalizedImfile as imfile + ON det1.ref_det_id = imfile.det_id + AND det1.ref_iter = imfile.iteration +LEFT JOIN detRegisteredImfile + ON det1.det_id = detRegisteredImfile.det_id + AND det1.iteration = detRegisteredImfile.iteration + AND imfile.class_id = detRegisteredImfile.class_id +WHERE + det1.state = 'run' + AND det1.mode = 'correction' + AND det2.state = 'stop' + AND detRegisteredImfile.det_id IS NULL + AND detRegisteredImfile.iteration IS NULL + AND detRegisteredImfile.class_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_todetrunsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_todetrunsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_todetrunsummary.sql (revision 41104) @@ -0,0 +1,48 @@ +-- select detRun.det_id +-- select detRun.iteration +-- select detRun.det_type +-- by: +-- find the current iteration bassed on det_id +-- find all exp_ids in the current det_id/iteration from detInputExp +-- find all exp_ids in the current det_id/iteration from detResidExp +-- compare the counts of exp_ids + +SELECT DISTINCT + det_id, + iteration, + det_type, + mode, + workdir, + camera +FROM + (SELECT DISTINCT + detRun.det_id, + detRun.iteration, + detRun.det_type, + detRun.mode, + detRun.workdir, + detInputExp.exp_id, + rawExp.camera + FROM detRun + JOIN detInputExp + USING(det_id, iteration) + JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + LEFT JOIN detResidExp + ON detRun.det_id = detResidExp.det_id + AND detRun.iteration = detResidExp.iteration + AND detInputExp.exp_id = detResidExp.exp_id + LEFT JOIN detRunSummary + ON detRun.det_id = detRunSummary.det_id + AND detRun.iteration = detRunSummary.iteration + WHERE + detRun.state = 'run' + AND detRunSummary.det_id IS NULL + AND detRunSummary.iteration IS NULL + AND detResidExp.fault = 0 + GROUP BY + detRun.det_id, + detRun.iteration + HAVING + COUNT(detResidExp.exp_id) = COUNT(detInputExp.exp_id) + ) AS Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalize.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalize.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalize.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT DISTINCT + detRun.det_type, + detRun.workdir, + rawExp.camera, + detStackedImfile.uri, + detNormalizedStatImfile.* + FROM detRun + JOIN detStackedImfile + USING(det_id, iteration) + JOIN detInputExp + USING(det_id, iteration) + JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + JOIN detNormalizedStatImfile + ON detStackedImfile.det_id = detNormalizedStatImfile.det_id + AND detStackedImfile.iteration = detNormalizedStatImfile.iteration + AND detStackedImfile.class_id = detNormalizedStatImfile.class_id + LEFT JOIN detNormalizedImfile + ON detNormalizedStatImfile.det_id = detNormalizedImfile.det_id + AND detNormalizedStatImfile.iteration = detNormalizedImfile.iteration + AND detNormalizedStatImfile.class_id = detNormalizedImfile.class_id + WHERE + detNormalizedImfile.det_id IS NULL + AND detNormalizedImfile.iteration IS NULL + AND detNormalizedImfile.class_id IS NULL + AND detNormalizedStatImfile.fault = 0 + AND detRun.state = 'run' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalizedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalizedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalizedexp.sql (revision 41104) @@ -0,0 +1,48 @@ +-- this is used by both 'tonormalizedexp' and 'addnormalizedexp' + +SELECT + * +FROM (SELECT + * + FROM ( + SELECT DISTINCT + detRun.det_id, + detRun.iteration, + detRun.det_type, + detRun.workdir, + rawExp.camera, + rawExp.telescope, + rawExp.exp_type, + detNormalizedImfile.class_id as norm_class_id, + detNormalizedImfile.fault, + detStackedImfile.class_id as stack_class_id + FROM detRun + JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration + JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + JOIN detStackedImfile + ON detStackedImfile.det_id = detRun.det_id + AND detStackedImfile.iteration = detRun.iteration + LEFT JOIN detNormalizedImfile + ON detStackedImfile.det_id = detNormalizedImfile.det_id + AND detStackedImfile.iteration = detNormalizedImfile.iteration + AND detStackedImfile.class_id = detNormalizedImfile.class_id + LEFT JOIN detNormalizedExp + ON detInputExp.det_id = detNormalizedExp.det_id + AND detInputExp.iteration = detNormalizedExp.iteration + WHERE + detRun.state = 'run' AND + detRun.mode = 'master' AND + detNormalizedExp.det_id IS NULL AND + detNormalizedExp.iteration IS NULL AND + detInputExp.include = 1 + ) as Foo + GROUP BY + det_id, + iteration + HAVING + COUNT(norm_class_id) = COUNT(stack_class_id) + AND SUM(fault) = 0 +) as Bar Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalizedstat.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalizedstat.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tonormalizedstat.sql (revision 41104) @@ -0,0 +1,37 @@ +-- a det_run + iteration is ready for normstat when: all detResidImfile entries for that iteration corresponding to all detProcessedImfile entries are available +SELECT + detRun.det_id, + detRun.det_type, + detRun.iteration, + detRun.camera, + detRun.workdir, + detProcessedImfile.class_id, + COUNT(detProcessedImfile.class_id), + COUNT(detResidImfile.class_id) +FROM detRun +JOIN detInputExp + USING (det_id, iteration) +JOIN detProcessedImfile + ON detProcessedImfile.det_id = detRun.det_id + AND detProcessedImfile.exp_id = detInputExp.exp_id +LEFT JOIN detResidImfile + ON detResidImfile.det_id = detProcessedImfile.det_id + AND detResidImfile.exp_id = detProcessedImfile.exp_id + AND detResidImfile.class_id = detProcessedImfile.class_id + AND detResidImfile.iteration = detRun.iteration +LEFT JOIN detNormalizedStatImfile + ON detNormalizedStatImfile.det_id = detRun.det_id + AND detNormalizedStatImfile.iteration = detRun.iteration + AND detNormalizedStatImfile.class_id = detProcessedImfile.class_id +WHERE + detRun.state = 'run' + AND detRun.mode = 'master' + AND detNormalizedStatImfile.det_id IS NULL + AND detNormalizedStatImfile.iteration IS NULL + AND detNormalizedStatImfile.class_id IS NULL +GROUP BY + detRun.iteration, + detRun.det_id +HAVING + COUNT(detProcessedImfile.class_id) = COUNT(detResidImfile.class_id) + AND SUM(detResidImfile.fault) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toprocessedexp.sql (revision 41104) @@ -0,0 +1,61 @@ +-- select detProcessedImfile.det_id +-- select detRun.iteration +-- select detRun.det_type +-- select detProcessedImfile.exp_id +-- by: +-- find the current iteration bassed on det_id +-- find all exp_ids in the current det_id/iteration from detInputExp +-- find all rawImfiles in the current exp_ids +-- compare to detProcessedImfiles by det_id/exp_id +-- found how many imfile there are in each class_id +-- and: +-- det_id is not in detProcessedExp; +-- iteration is not in detProcessedExp; + +SELECT DISTINCT + det_id, + iteration, + det_type, + exp_id, + camera, + workdir, + exp_tag +FROM + (SELECT DISTINCT + detRun.det_id, + detRun.iteration, + detRun.det_type, + detRun.workdir, + detProcessedImfile.exp_id, + detProcessedImfile.class_id, + rawImfile.class_id as rawimfile_class_id, + rawExp.camera, + rawExp.exp_tag + FROM detRun + JOIN detInputExp + USING(det_id, iteration) + JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + JOIN rawImfile + ON rawExp.exp_id = rawImfile.exp_id + LEFT JOIN detProcessedImfile + ON detRun.det_id = detProcessedImfile.det_id + AND detInputExp.exp_id = detProcessedImfile.exp_id + AND rawImfile.class_id = detProcessedImfile.class_id + LEFT JOIN detProcessedExp + ON detProcessedImfile.det_id = detProcessedExp.det_id + AND detProcessedImfile.exp_id = detProcessedExp.exp_id + WHERE + detRun.state = 'run' + AND (detRun.mode = 'master' or detRun.mode = 'verify') + AND detProcessedExp.det_id IS NULL + AND detProcessedExp.exp_id IS NULL + AND detInputExp.include = 1 + GROUP BY + rawExp.exp_id, + detRun.det_id + HAVING + COUNT(detProcessedImfile.class_id) = COUNT(rawImfile.class_id) + AND SUM(detProcessedImfile.fault) = 0 + ) AS detProcessedExp + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toprocessedimfile.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + detRun.det_id, + detRun.det_type, + detRun.workdir, + detRun.reduction, + rawImfile.*, + rawExp.exp_tag +FROM detRun +JOIN detInputExp + USING(det_id, iteration) +JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id +JOIN rawImfile + ON detInputExp.exp_id = rawImfile.exp_id +LEFT JOIN detProcessedImfile + ON detInputExp.det_id = detProcessedImfile.det_id + AND rawImfile.exp_id = detProcessedImfile.exp_id + AND rawImfile.class_id = detProcessedImfile.class_id +WHERE + detRun.state = 'run' + AND (detRun.mode = 'master' or detRun.mode = 'verify') + AND detProcessedImfile.det_id IS NULL + AND detProcessedImfile.exp_id IS NULL + AND detProcessedImfile.class_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toresidexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toresidexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toresidexp.sql (revision 41104) @@ -0,0 +1,114 @@ +-- this is used by both 'toresidexp' and 'addresidexp' + +-- which returns a list of exposures for which all component class ids have had +-- residuals created. This list includes the detrend id, iteration, exposure +-- id, detrend type, and whether the exposure was included in the stack for +-- this iteration. + +-- require the corresponding detNormalizedExp to complete before starting + +-- select detRun.det_id +-- select detRun.iteration +-- select detRun.det_type +-- select detInputExp.exp_id +-- select detInputExp.include +-- by: +-- find the current iteration bassed on det_id +-- find all exp_ids in the current det_id/iteration from detInputExp +-- compare to detInputExp.imfiles to derResidImfile by class_id +-- and: +-- detResidImfile.{det_id, iteration, exp_id} is not in detResidExp + +SELECT + det_id, + iteration, + det_type, + mode, + exp_id, + include, + camera, + workdir, + exp_tag +FROM +( SELECT + detRun.det_id AS det_id, + detRun.iteration, + detRun.det_type, + detRun.mode, + detInputExp.exp_id, + detInputExp.include, + rawExp.camera, + detRun.workdir, + rawExp.exp_tag + FROM detRun + JOIN detNormalizedExp + USING(det_id, iteration) + JOIN detInputExp + USING(det_id, iteration) + JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + JOIN rawImfile + on rawExp.exp_id = rawImfile.exp_id + LEFT JOIN detResidImfile + ON detRun.det_id = detResidImfile.det_id + AND detRun.iteration = detResidImfile.iteration + AND detInputExp.exp_id = detResidImfile.exp_id + AND rawImfile.class_id = detResidImfile.class_id + LEFT JOIN detResidExp + ON detResidImfile.det_id = detResidExp.det_id + AND detResidImfile.iteration = detResidExp.iteration + AND detResidImfile.exp_id = detResidExp.exp_id + WHERE + detRun.state = 'run' + AND detRun.mode = 'master' + AND detResidExp.det_id IS NULL + AND detResidExp.iteration IS NULL + AND detResidExp.exp_id IS NULL + GROUP BY + detInputExp.exp_id, + detRun.iteration, + detRun.det_id + HAVING + COUNT(rawImfile.class_id) = COUNT(detResidImfile.class_id) + AND SUM(detResidImfile.fault) = 0 + UNION + SELECT + detRun.det_id AS det_id, + detRun.iteration, + detRun.det_type, + detRun.mode, + detInputExp.exp_id, + detInputExp.include, + rawExp.camera, + detRun.workdir, + rawExp.exp_tag + FROM detRun + JOIN detInputExp + USING(det_id, iteration) + JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id + JOIN rawImfile + on rawExp.exp_id = rawImfile.exp_id + LEFT JOIN detResidImfile + ON detRun.det_id = detResidImfile.det_id + AND detRun.iteration = detResidImfile.iteration + AND detInputExp.exp_id = detResidImfile.exp_id + AND rawImfile.class_id = detResidImfile.class_id + LEFT JOIN detResidExp + ON detResidImfile.det_id = detResidExp.det_id + AND detResidImfile.iteration = detResidExp.iteration + AND detResidImfile.exp_id = detResidExp.exp_id + WHERE + detRun.state = 'run' + AND detRun.mode = 'verify' + AND detResidExp.det_id IS NULL + AND detResidExp.iteration IS NULL + AND detResidExp.exp_id IS NULL + GROUP BY + detInputExp.exp_id, + detRun.iteration, + detRun.det_id + HAVING + COUNT(rawImfile.class_id) = COUNT(detResidImfile.class_id) + AND SUM(detResidImfile.fault) = 0 +) as temp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toresidimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toresidimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_toresidimfile.sql (revision 41104) @@ -0,0 +1,154 @@ +-- which returns a list of processed imfiles (with corresponding detrend id, +-- iteration, class id, uri, type) for which the appropriate stacked imfile has +-- been normalised, and which have not been masked out by detResidImfileAnalysis + +SELECT DISTINCT + detRun.det_id, + detRun.iteration, + detRun.det_type, + detRun.mode, + detRun.workdir, + detRun.reduction, + detProcessedImfile.exp_id, + detProcessedImfile.class_id, + detProcessedImfile.uri, +-- detNormalizedImfile.uri AS det_uri, + detStackedImfile.uri AS det_uri, + detRun.det_id AS ref_det_id, + detRun.iteration AS ref_iter, + rawExp.camera, + rawExp.exp_tag +FROM detRun +JOIN detInputExp +-- USING(det_id, iteration) + ON detInputExp.det_id = detRun.det_id + AND detInputExp.iteration = detRun.iteration +-- AND detInputExp.iteration = 0 +JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id +JOIN detProcessedImfile + ON detRun.det_id = detProcessedImfile.det_id + AND detInputExp.exp_id = detProcessedImfile.exp_id +JOIN detStackedImfile + ON detRun.det_id = detStackedImfile.det_id + AND detRun.iteration = detStackedImfile.iteration + AND detProcessedImfile.class_id = detStackedImfile.class_id +-- EAM : replacing detNormalizedImfile with detStackedImfile to change the sequencing +-- JOIN detNormalizedImfile +-- ON detRun.det_id = detNormalizedImfile.det_id +-- AND detRun.iteration = detNormalizedImfile.iteration +-- AND detProcessedImfile.class_id = detNormalizedImfile.class_id +-- EAM : we there is no reason to wait for all stacks to complete before continuing +-- JOIN detNormalizedExp +-- ON detRun.det_id = detNormalizedExp.det_id +-- AND detRun.iteration = detNormalizedExp.iteration +LEFT JOIN detResidImfile + ON detRun.det_id = detResidImfile.det_id + AND detRun.iteration = detResidImfile.iteration + AND detProcessedImfile.exp_id = detResidImfile.exp_id + AND detProcessedImfile.class_id = detResidImfile.class_id +WHERE + detRun.state = 'run' + AND detRun.mode = 'master' + AND detStackedImfile.fault = 0 +-- AND detNormalizedImfile.fault = 0 +-- AND detNormalizedExp.fault = 0 + AND detResidImfile.det_id IS NULL + AND detResidImfile.iteration IS NULL + AND detResidImfile.exp_id IS NULL + AND detResidImfile.class_id IS NULL +UNION +SELECT + detRun.det_id, + detRun.iteration, + detRun.det_type, + detRun.mode, + detRun.workdir, + detRun.reduction, + detProcessedImfile.exp_id, + detProcessedImfile.class_id, + detProcessedImfile.uri, + detNormalizedImfile.uri AS det_uri, + detRun.ref_det_id, + detRun.ref_iter, + rawExp.camera, + rawExp.exp_tag +FROM detRun +JOIN detRun AS detRunRef + ON detRunRef.det_id = detRun.ref_det_id +JOIN detInputExp + ON detInputExp.det_id = detRun.det_id + AND detInputExp.iteration = detRun.iteration +-- AND detInputExp.iteration = 0 +JOIN rawExp + ON rawExp.exp_id = detInputExp.exp_id +JOIN detProcessedImfile + ON detProcessedImfile.det_id = detRun.det_id + AND detProcessedImfile.exp_id = detInputExp.exp_id +JOIN detNormalizedImfile + ON detNormalizedImfile.det_id = detRun.ref_det_id + AND detNormalizedImfile.iteration = detRun.ref_iter + AND detNormalizedImfile.class_id = detProcessedImfile.class_id +LEFT JOIN detResidImfile + ON detResidImfile.det_id = detRun.det_id + AND detResidImfile.iteration = detRun.iteration + AND detResidImfile.exp_id = detProcessedImfile.exp_id + AND detResidImfile.class_id = detProcessedImfile.class_id +WHERE + detRun.state = 'run' + AND detRun.mode = 'verify' + AND detRunRef.state = 'stop' + AND detNormalizedImfile.fault = 0 + AND detProcessedImfile.fault = 0 + AND detResidImfile.det_id IS NULL + AND detResidImfile.iteration IS NULL + AND detResidImfile.exp_id IS NULL + AND detResidImfile.class_id IS NULL +UNION +SELECT + detRun.det_id, + detRun.iteration, + detRun.det_type, + detRun.mode, + detRun.workdir, + detRun.reduction, + detProcessedImfile.exp_id, + detProcessedImfile.class_id, + detProcessedImfile.uri, + detRegisteredImfile.uri AS det_uri, + detRun.ref_det_id, + detRun.ref_iter, + rawExp.camera, + rawExp.exp_tag +FROM detRun +JOIN detRun AS detRunRef + ON detRunRef.det_id = detRun.ref_det_id +JOIN detInputExp + ON detInputExp.det_id = detRun.det_id + AND detInputExp.iteration = detRun.iteration +-- AND detInputExp.iteration = 0 +JOIN rawExp + ON rawExp.exp_id = detInputExp.exp_id +JOIN detProcessedImfile + ON detProcessedImfile.det_id = detRun.det_id + AND detProcessedImfile.exp_id = detInputExp.exp_id +JOIN detRegisteredImfile + ON detRegisteredImfile.det_id = detRun.ref_det_id + AND detRegisteredImfile.iteration = detRun.ref_iter + AND detRegisteredImfile.class_id = detProcessedImfile.class_id +LEFT JOIN detResidImfile + ON detResidImfile.det_id = detRun.det_id + AND detResidImfile.iteration = detRun.iteration + AND detResidImfile.exp_id = detProcessedImfile.exp_id + AND detResidImfile.class_id = detProcessedImfile.class_id +WHERE + detRun.state = 'run' + AND detRun.mode = 'verify' + AND detRunRef.state = 'stop' + AND detRegisteredImfile.fault = 0 + AND detProcessedImfile.fault = 0 + AND detResidImfile.det_id IS NULL + AND detResidImfile.iteration IS NULL + AND detResidImfile.exp_id IS NULL + AND detResidImfile.class_id IS NULL +ORDER BY exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tostacked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tostacked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dettool_tostacked.sql (revision 41104) @@ -0,0 +1,52 @@ +-- select detRun.iteration +-- select detProcessedImfile.det_id +-- select detProcessedImfile.class_id +-- by: +-- find the current iteration bassed on det_id +-- find all exp_ids in the current det_id/iteration from detInputExp +-- find all rawImfiles in the current exp_ids +-- compare to detProcessedImfiles by det_id/exp_id +-- found how many imfile there are in each class_id +-- and: +-- det_id is not in detStackedImfile; +-- iteration is not in detStackedImfile; +-- class_id is not in detStackedImfile; + +SELECT + detProcessedImfile.det_id, + detRun.iteration, + detRun.det_type, + detRun.workdir, + detRun.reduction, + detProcessedImfile.class_id, + rawExp.camera, + rawExp.exp_tag +FROM detRun +JOIN detInputExp + ON detRun.det_id = detInputExp.det_id + AND detRun.iteration = detInputExp.iteration +JOIN rawExp + ON detInputExp.exp_id = rawExp.exp_id +JOIN rawImfile + ON detInputExp.exp_id = rawImfile.exp_id +LEFT JOIN detProcessedImfile + ON detInputExp.det_id = detProcessedImfile.det_id + AND detInputExp.exp_id = detProcessedImfile.exp_id + AND rawImfile.class_id = detProcessedImfile.class_id +LEFT JOIN detStackedImfile + ON detInputExp.det_id = detStackedImfile.det_id + AND detInputExp.iteration = detStackedImfile.iteration + AND rawImfile.class_id = detStackedImfile.class_id +WHERE + detRun.state = 'run' + AND detRun.mode = 'master' + AND detStackedImfile.det_id IS NULL + AND detStackedImfile.iteration IS NULL + AND detStackedImfile.class_id IS NULL + AND detInputExp.include = 1 +GROUP BY + rawImfile.class_id, + detRun.det_id +HAVING + COUNT(detProcessedImfile.class_id) = COUNT(rawImfile.class_id) + AND SUM(detProcessedImfile.fault) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_advance.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_advance.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_advance.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT + diff_phot_id, + diffPhotSkyfile.magicked +FROM diffPhotRun +JOIN diffSkyfile USING(diff_id) +LEFT JOIN diffPhotSkyfile USING(diff_phot_id, skycell_id) +WHERE diffPhotRun.state = 'new' + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 +-- WHERE hook %s +GROUP BY diff_phot_id +HAVING COUNT(diffPhotSkyfile.skycell_id) = COUNT(diffSkyfile.skycell_id) + AND SUM(IF(diffPhotSkyfile.fault > 0, 1, 0)) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_data.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_data.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_data.sql (revision 41104) @@ -0,0 +1,63 @@ +SELECT DISTINCT + diffPhotSkyfile.*, + diffRun.tess_id, + diffPhotRun.state, + diffPhotRun.workdir, + diffPhotRun.label, + diffPhotRun.diff_id, + diffRun.bothways, + diffRun.diff_mode, + -- The following are only valid for warps + -- XXX This needs to be more clever to handle diffs between stacks + -- Zero points are appropriate for both forward and backward diffs + camProcessedInput.zpt_obs, + camProcessedInput.zpt_stdev, + camProcessedInput.zpt_lq, + camProcessedInput.zpt_uq, + rawInput.comment, + rawInput.exp_time, + rawInput.camera, + rawInput.exp_name AS exp_name_1, + rawInput.exp_id AS exp_id_1, + chipInput.chip_id AS chip_id_1, + camInput.cam_id AS cam_id_1, + fakeInput.fake_id AS fake_id_1, + warp1, + camProcessedInput.sigma_ra AS sigma_ra_1, + camProcessedInput.sigma_dec AS sigma_dec_1, + rawTemplate.exp_name AS exp_name_2, + rawTemplate.exp_id AS exp_id_2, + chipTemplate.chip_id AS chip_id_2, + camTemplate.cam_id AS cam_id_2, + fakeTemplate.fake_id AS fake_id_2, + warp2, + camProcessedTemplate.sigma_ra AS sigma_ra_2, + camProcessedTemplate.sigma_dec AS sigma_dec_2 +FROM diffPhotRun +JOIN diffPhotSkyfile USING(diff_phot_id) +JOIN diffRun USING(diff_id) +JOIN diffInputSkyfile USING(diff_id, skycell_id) +LEFT JOIN warpRun AS warpInput + ON warpInput.warp_id = diffInputSkyfile.warp1 +LEFT JOIN fakeRun AS fakeInput + ON fakeInput.fake_id = warpInput.fake_id +LEFT JOIN camRun AS camInput + ON camInput.cam_id = fakeInput.cam_id +LEFT JOIN camProcessedExp AS camProcessedInput + ON camProcessedInput.cam_id = camInput.cam_id +LEFT JOIN chipRun AS chipInput + ON chipInput.chip_id = camInput.chip_id +LEFT JOIN rawExp AS rawInput + ON rawInput.exp_id = chipInput.exp_id +LEFT JOIN warpRun AS warpTemplate + ON warpTemplate.warp_id = diffInputSkyfile.warp2 +LEFT JOIN fakeRun AS fakeTemplate + ON fakeTemplate.fake_id = warpTemplate.fake_id +LEFT JOIN camRun AS camTemplate + ON camTemplate.cam_id = fakeTemplate.cam_id +LEFT JOIN camProcessedExp AS camProcessedTemplate + ON camProcessedTemplate.cam_id = camTemplate.cam_id +LEFT JOIN chipRun AS chipTemplate + ON chipTemplate.chip_id = camTemplate.chip_id +LEFT JOIN rawExp AS rawTemplate + ON rawTemplate.exp_id = chipTemplate.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_definerun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_definerun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_definerun.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT DISTINCT + diffRun.* +FROM diffRun +JOIN diffInputSkyfile USING(diff_id) +JOIN warpRun ON warpRun.warp_id = diffInputSkyfile.warp1 -- only JOINing inputs, not templates! +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +WHERE diffRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_input.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_input.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_input.sql (revision 41104) @@ -0,0 +1,16 @@ +SELECT + diff_phot_id, + skycell_id, + diffSkyfile.path_base, + diffSkyfile.magicked, + diffRun.bothways, + rawExp.camera +FROM diffPhotRun +JOIN diffRun USING(diff_id) +JOIN diffSkyfile USING(diff_id) +JOIN diffInputSkyfile USING(diff_id, skycell_id) +JOIN warpRun ON warp1 = warp_id -- only JOINing on input warps! +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_pending.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_pending.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_pending.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + diffPhotRun.*, + diffRun.tess_id, + diffSkyfile.skycell_id +FROM diffPhotRun +JOIN diffRun USING(diff_id) +JOIN diffSkyfile USING(diff_id) +LEFT JOIN diffPhotSkyfile USING(diff_phot_id, skycell_id) +WHERE diffPhotSkyfile.skycell_id IS NULL + AND diffRun.state = 'full' + AND diffSkyfile.magicked >= 0 + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_revert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_revert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/diffphottool_revert.sql (revision 41104) @@ -0,0 +1,4 @@ +DELETE diffPhotSkyfile +FROM diffPhotRun +JOIN diffPhotSkyfile USING(diff_phot_id) +WHERE fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_addsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_addsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_addsummary.sql (revision 41104) @@ -0,0 +1,5 @@ +INSERT INTO diffSummary (diff_id,projection_cell,path_base) VALUES ( + %lld, -- warp_id + '%s', -- projection_cell + '%s' -- path_base + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_change_run_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_change_run_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_change_run_state.sql (revision 41104) @@ -0,0 +1,17 @@ +-- change state of ddiffRun from goto_cleaned to cleaned or goto_purged to purged +-- when all of the consituant skyfiles are in the end state +-- arguments are new state (cleaned or purged) diff_id and new state again for +-- the chipProcessedImfile sub query +UPDATE diffRun + SET state = '%s' + -- set magicked hook %s + WHERE + diffRun.diff_id = %lld + AND (SELECT + COUNT(diff_id) + FROM diffSkyfile + WHERE + diffSkyfile.diff_id = diffRun.diff_id + AND data_state != '%s' + AND diffSkyfile.quality = 0 + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_change_skyfile_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_change_skyfile_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_change_skyfile_data_state.sql (revision 41104) @@ -0,0 +1,10 @@ +-- handle changes in diffSkyfile.data_state. +-- Used for the modes tocleanedskyfile and topurgedskyfile +-- args are new data_state, diff_id, skycell_id and current expected state for diffRun +UPDATE diffSkyfile + SET + data_state = '%s' + -- magicked hook %s +WHERE + diff_id = %lld + AND skycell_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_coalesce_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_coalesce_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_coalesce_run.sql (revision 41104) @@ -0,0 +1,1 @@ +select software_ver,maskfrac_npix,maskfrac_static,maskfrac_dynamic,maskfrac_magic,maskfrac_advisory from diffSkyfile where diff_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_completed_runs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_completed_runs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_completed_runs.sql (revision 41104) @@ -0,0 +1,16 @@ +SELECT + COUNT(diffInputSkyfile.skycell_id), + COUNT(diffSkyfile.skycell_id), + diffSkyfile.*, + SUM(!diffSkyfile.magicked) = 0 as all_magicked +FROM diffRun +JOIN diffInputSkyfile USING(diff_id) +LEFT JOIN diffSkyfile USING(diff_id, skycell_id) +WHERE + diffRun.state = 'new' +-- WHERE hook %s +GROUP BY + diffInputSkyfile.diff_id +HAVING + COUNT(diffInputSkyfile.skycell_id) = COUNT(diffSkyfile.skycell_id) + AND SUM(diffSkyfile.fault) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definestackstack_part0.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definestackstack_part0.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definestackstack_part0.sql (revision 41104) @@ -0,0 +1,46 @@ +SELECT DISTINCT + stackRun.data_group AS INPUT_data_group, + stackRun.dist_group AS INPUT_dist_group, + stackRun.filter AS INPUT_filter, + stackRun.label AS INPUT_label, + stackRun.tess_id AS INPUT_tess_id +-- , +-- stackRun.stack_id AS INPUT_stack_id, +-- template.max_stack_id, +-- diffExp.diff_id +FROM stackRun JOIN stackSumSkyfile USING(stack_id) +JOIN ( + SELECT + MAX(stackRun.stack_id) as max_stack_id, + stackRun.filter, + stackRun.label as template_label, + stackRun.tess_id, + stackRun.skycell_id, + stackSumSkyfile.good_frac + FROM stackRun JOIN stackSumSkyfile USING(stack_id) + WHERE + stackRun.state = 'full' AND + stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0 + @STACK2_QUERY@ -- template condition + GROUP BY + skycell_id, + filter +) AS template ON stackRun.filter = template.filter + AND stackRun.tess_id = template.tess_id + AND stackRun.skycell_id = template.skycell_id +LEFT JOIN ( + SELECT + diffRun.diff_id, + diffInputSkyfile.stack1, + diffInputSkyfile.stack2 + FROM diffRun JOIN diffInputSkyfile USING(diff_id) + JOIN stackRun ON stackRun.stack_id = diffInputSkyfile.stack1 + JOIN stackSumSkyfile USING(stack_id) + WHERE 1 + @STACK1_QUERY@ -- input condition +) AS diffExp ON diffExp.stack1 = stackRun.stack_id +WHERE + stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0 + @DIFF0_QUERY@ -- diff constraint + @STACK1_QUERY@ -- input constraint +ORDER BY stackRun.data_group,stackRun.filter Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definestackstack_part1.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definestackstack_part1.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definestackstack_part1.sql (revision 41104) @@ -0,0 +1,56 @@ +SELECT distinct + stackRun.stack_id, + stackRun.data_group, + stackRun.filter, + stackRun.label, + stackRun.tess_id, + stackRun.skycell_id, + stackSumSkyfile.good_frac, + template.max_stack_id, + template.template_label, + count(diffExp.diff_id) as n_diff +FROM stackRun JOIN stackSumSkyfile USING(stack_id) +JOIN ( + SELECT + MAX(stackRun.stack_id) as max_stack_id, + stackRun.filter, + stackRun.label as template_label, + stackRun.tess_id, + stackRun.skycell_id, + stackSumSkyfile.good_frac + FROM stackRun JOIN stackSumSkyfile USING(stack_id) + WHERE + stackRun.state = 'full' AND + stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0 + @STACK2_QUERY@ -- template constraint + GROUP BY + skycell_id, + filter +) AS template ON stackRun.filter = template.filter + AND stackRun.tess_id = template.tess_id + AND stackRun.skycell_id = template.skycell_id +LEFT JOIN ( + SELECT + diffRun.diff_id, + diffInputSkyfile.stack1, + diffInputSkyfile.stack2 + FROM diffRun JOIN diffInputSkyfile USING(diff_id) + JOIN stackRun ON stackRun.stack_id = diffInputSkyfile.stack1 + JOIN stackSumSkyfile USING(stack_id) + WHERE 1 + @STACK1_QUERY@ -- input constraint +) AS diffExp ON diffExp.stack1 = stackRun.stack_id +WHERE + stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0 + @STACK1_QUERY@ -- input constraint +group by stackRun.stack_id, + stackRun.data_group, + stackRun.filter, + stackRun.label, + stackRun.tess_id, + stackRun.skycell_id, + stackSumSkyfile.good_frac, + template.max_stack_id, + template.template_label + @DIFF1_QUERY@ -- diff constraint +ORDER BY stackRun.data_group,stackRun.filter,stackRun.skycell_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack.sql (revision 41104) @@ -0,0 +1,73 @@ +-- Get list of warp-stack pairs that can be diffed +-- and check results against existing diffs +SELECT + exp_id, + warp_id, + filter, + warpLabel, + warpDataGroup, + Inputs.tess_id, + Inputs.skycell_id, + stack_id, + stackLabel, + stackDataGroup, + diff_id, + diffLabel +FROM +( +SELECT + exp_id, + warp_id, + rawExp.filter, + warpRun.label AS warpLabel, + warpRun.data_group AS warpDataGroup, + warpRun.tess_id, + warpSkyfile.skycell_id, + MAX(stack_id) AS stack_id, + stackRun.label AS stackLabel, + stackRun.data_group AS stackDataGroup +FROM warpRun + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + JOIN warpSkyfile USING(warp_id) + JOIN stackRun ON + (stackRun.skycell_id = warpSkyfile.skycell_id AND + stackRun.filter = rawExp.filter AND + stackRun.tess_id = warpRun.tess_id) + JOIN stackSumSkyfile USING(stack_id) +WHERE + warpRun.state = 'full' + AND stackRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + AND exp_id IS NOT NULL +-- %s +GROUP BY exp_id,warp_id,skycell_id +) AS Inputs +LEFT JOIN +( +SELECT +diff_id, +diffRun.label AS diffLabel, +warp1, +stack2, +skycell_id, +diffInputSkyfile.tess_id +FROM +diffRun JOIN diffInputSkyfile USING(diff_id,tess_id) +WHERE +diff_mode = 2 +-- %s +) AS Diffs ON + (warp_id = warp1 AND + stack_id = stack2 AND + Diffs.skycell_id = Inputs.skycell_id AND + Diffs.tess_id = Inputs.tess_id) +WHERE +1 +-- %s + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_old.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_old.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_old.sql (revision 41104) @@ -0,0 +1,42 @@ +-- Get list of warp-stack pairs that can be diffed +-- and check results against existing diffs +SELECT + exp_id, + warp_id, + rawExp.filter, + warpRun.label AS warpLabel, + warpRun.data_group AS warpDataGroup, + warpRun.tess_id, + warpSkyfile.skycell_id, + MAX(stack_id) AS stack_id, + stackRun.label AS stackLabel, + stackRun.data_group AS stackDataGroup, + diff_id, + diffRun.label AS diffLabel +FROM warpRun + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + JOIN warpSkyfile USING(warp_id) + JOIN stackRun ON + (stackRun.skycell_id = warpSkyfile.skycell_id AND + stackRun.filter = rawExp.filter AND + stackRun.tess_id = warpRun.tess_id) + JOIN stackSumSkyfile USING(stack_id) + LEFT JOIN diffInputSkyfile ON + (warp_id = diffInputSkyfile.warp1 AND + stack_id = diffInputSkyfile.stack2 AND + diffInputSkyfile.skycell_id = stackRun.skycell_id AND + diffInputSkyfile.tess_id = stackRun.tess_id) + LEFT JOIN diffRun USING(diff_id) +WHERE + warpRun.state = 'full' + AND stackRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + AND exp_id IS NOT NULL +-- %s +GROUP BY exp_id,warp_id,skycell_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_part1.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_part1.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_part1.sql (revision 41104) @@ -0,0 +1,38 @@ +SELECT + exp_id, + warp_id, + rawExp.filter, + warpRun.label, + warpRun.tess_id, + warpRun.data_group, + COUNT(skycell_id) as skycell_count +FROM warpRun +JOIN warpSkyfile USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN ( + SELECT DISTINCT + diffRun.*, + warp_id, + exp_id + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun + ON warpRun.warp_id = diffInputSkyfile.warp1 + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + WHERE warp1 IS NOT NULL + AND warpRun.state = 'full' + -- warp where hook %s +) AS diffExp USING(exp_id, warp_id) +WHERE + warpRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + -- warp where hook %s + -- exp where hook %s + -- diff where hook %s +GROUP BY exp_id, warp_id DESC Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_part2.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_part2.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_part2.sql (revision 41104) @@ -0,0 +1,37 @@ +-- insert skycells to diff +INSERT INTO skycellsToDiff +SELECT + 0, -- diff_id + warpSkyfile.skycell_id, + warpSkyfile.warp_id, -- warp1 + NULL, -- stack1 + NULL, -- warp2 + max_stack_id, -- stack2 + warpSkyfile.tess_id, + 0 +FROM warpSkyfile +JOIN warpRun USING(warp_id) +LEFT JOIN ( + SELECT + MAX(stack_id) AS max_stack_id, -- most recent stack, by virtue of auto-increment + stackRun.skycell_id, + stackRun.tess_id, + filter + FROM stackRun + JOIN stackSumSkyfile USING(stack_id) + WHERE stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + AND stackRun.state = 'full' + -- stacks where hook %s + GROUP BY + skycell_id, + filter + ) as bestStacks USING(skycell_id) +WHERE + warpSkyfile.warp_id = %lld + AND warpRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + AND filter = '%s' + AND warpSkyfile.tess_id = bestStacks.tess_id +-- warp where hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_temp_create.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_temp_create.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpstack_temp_create.sql (revision 41104) @@ -0,0 +1,10 @@ +CREATE TEMPORARY TABLE skycellsToDiff ( +diff_id BIGINT, +skycell_id VARCHAR(64), +warp1 BIGINT, +stack1 BIGINT, +warp2 BIGINT, +stack2 BIGINT, +tess_id VARCHAR(64), +diff_skycell_id BIGINT +) ENGINE=MEMORY; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_insert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_insert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_insert.sql (revision 41104) @@ -0,0 +1,16 @@ +INSERT INTO diffInputSkyfile +SELECT + %s, -- diff_id + skycell_id, + %s, -- input warp_id + NULL, -- input stack_id + %s, -- template warp_id + NULL, -- template stack_id + tess_id, + 0 -- diff_skyfile_id +FROM warpSkyfile AS inputWarpSkyfile +JOIN warpSkyfile AS templateWarpSkyfile USING(skycell_id, tess_id) +WHERE inputWarpSkyfile.quality = 0 + AND templateWarpSkyfile.quality = 0 + AND inputWarpSkyfile.warp_id = %s + AND templateWarpSkyfile.warp_id = %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_select.sql (revision 41104) @@ -0,0 +1,33 @@ +SELECT + inputWarpRun.warp_id AS input_warp_id, + inputWarpRun.tess_id AS tess_id, + inputRawExp.exp_id AS input_exp_id, + inputWarpRun.data_group AS input_data_group, + -- The following trick pulls out the warp_id that has the smallest distance + SUBSTRING_INDEX(GROUP_CONCAT(templateWarpRun.warp_id ORDER BY ABS(ASIN(SQRT(POW(SIN(0.5*(inputRawExp.decl - templateRawExp.decl)),2) + COS(inputRawExp.decl) * COS(templateRawExp.decl) * POW(SIN(0.5*(inputRawExp.ra - templateRawExp.ra)),2))))), ',', 1) AS template_warp_id +FROM warpRun AS inputWarpRun +JOIN fakeRun AS inputFakeRun USING(fake_id) +JOIN camRun AS inputCamRun USING(cam_id) +JOIN chipRun AS inputChipRun USING(chip_id) +JOIN rawExp AS inputRawExp USING(exp_id) +-- To find exposures that haven't been diffed, insert newline here:%s LEFT JOIN diffs USING(exp_id) +JOIN warpRun AS templateWarpRun + ON templateWarpRun.warp_id != inputWarpRun.warp_id -- Don't use self as template! + AND templateWarpRun.tess_id = inputWarpRun.tess_id -- Ensure using same tessellation +JOIN fakeRun AS templateFakeRun + ON templateFakeRun.fake_id = templateWarpRun.fake_id + AND templateFakeRun.fake_id != inputFakeRun.fake_id -- Don't use self as template! +JOIN camRun AS templateCamRun + ON templateCamRun.cam_id = templateFakeRun.cam_id + AND templateCamRun.cam_id != inputCamRun.cam_id -- Don't use self as template! +JOIN chipRun AS templateChipRun + ON templateChipRun.chip_id = templateCamRun.chip_id + AND templateChipRun.chip_id != inputChipRun.chip_id -- Don't use self as template! +JOIN rawExp AS templateRawExp + ON templateRawExp.exp_id = templateChipRun.exp_id + AND templateRawExp.filter = inputRawExp.filter + AND templateRawExp.exp_id != inputRawExp.exp_id -- Don't use self as template! +WHERE inputWarpRun.state = 'full' + AND templateWarpRun.state = 'full' +-- WHERE hook %s +GROUP BY input_warp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_temp_create.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_temp_create.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_temp_create.sql (revision 41104) @@ -0,0 +1,8 @@ +-- Exposures that have diffs +-- Temporary table has useful indices that make future query faster! +CREATE TEMPORARY TABLE diffs ( + diff_id BIGINT, + exp_id BIGINT, + PRIMARY KEY(diff_id, exp_id), + KEY(exp_id) +) ENGINE=MEMORY; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_temp_insert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_temp_insert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_temp_insert.sql (revision 41104) @@ -0,0 +1,32 @@ +-- Get list of exposures that have diffs +-- Only interested in whole exposures (diffRun.exposure = 1) +INSERT INTO diffs +SELECT DISTINCT + diffWarps.diff_id, + exp_id +FROM ( + -- Forward diffs + SELECT + diffRun.diff_id, + warp1 AS warp_id + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + WHERE diffInputSkyfile.warp1 IS NOT NULL + AND diffRun.exposure = 1 + -- WHERE hook %s + UNION + -- Backward diffs + SELECT + diffRun.diff_id, + warp2 AS warp_id + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + WHERE diffInputSkyfile.warp2 IS NOT NULL + AND diffRun.exposure = 1 + AND diffRun.bothways = 1 + -- WHERE hook %s + ) AS diffWarps +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_test.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_test.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_definewarpwarp_test.sql (revision 41104) @@ -0,0 +1,82 @@ +DROP TABLE IF EXISTS diffs; + +-- Create a temporary table with useful indices +CREATE TEMPORARY TABLE diffs ( + diff_id BIGINT, + exp_id BIGINT, + PRIMARY KEY(diff_id, exp_id), + KEY(exp_id) +) ENGINE=MEMORY; + + +-- Get list of exposures that have diffs +INSERT INTO diffs +SELECT DISTINCT + diffWarps.diff_id, + exp_id +FROM ( + -- Forward diffs + SELECT + diffRun.diff_id, + warp1 AS warp_id + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + WHERE warp1 IS NOT NULL + -- WHERE hook %s + AND diffRun.label = 'ThreePi.Run2.r.v0' + UNION + -- Backward diffs + SELECT + diffRun.diff_id, + warp2 AS warp_id + FROM diffRun + JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffRun.diff_id + AND diffRun.bothways = 1 + WHERE warp2 IS NOT NULL + -- WHERE hook %s + AND diffRun.label = 'ThreePi.Run2.r.v0' + ) AS diffWarps +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +; + + + +-- EXPLAIN +SELECT + inputWarpRun.warp_id AS input_warp_id, + inputWarpRun.tess_id AS tess_id, + inputRawExp.exp_id AS input_exp_id, + SUBSTRING_INDEX(GROUP_CONCAT(templateWarpRun.warp_id ORDER BY ABS(ASIN(SQRT(POW(SIN(0.5*(inputRawExp.decl - templateRawExp.decl)),2) + COS(inputRawExp.decl) * COS(templateRawExp.decl) * POW(SIN(0.5*(inputRawExp.ra - templateRawExp.ra)),2))))), ',', 1) AS template_warp_id +FROM warpRun AS inputWarpRun +JOIN fakeRun AS inputFakeRun USING(fake_id) +JOIN camRun AS inputCamRun USING(cam_id) +JOIN chipRun AS inputChipRun USING(chip_id) +JOIN rawExp AS inputRawExp USING(exp_id) +-- To find exposures that haven't been diffed, insert newline here:%s + LEFT JOIN diffs USING(exp_id) +JOIN warpRun AS templateWarpRun + ON templateWarpRun.warp_id != inputWarpRun.warp_id -- Don't use self as template! + AND templateWarpRun.tess_id = inputWarpRun.tess_id -- Ensure using same tessellation +JOIN fakeRun AS templateFakeRun + ON templateFakeRun.fake_id = templateWarpRun.fake_id +JOIN camRun AS templateCamRun + ON templateCamRun.cam_id = templateFakeRun.cam_id +JOIN chipRun AS templateChipRun + ON templateChipRun.chip_id = templateCamRun.chip_id +JOIN rawExp AS templateRawExp + ON templateRawExp.exp_id = templateChipRun.exp_id + AND templateRawExp.filter = inputRawExp.filter +-- WHERE hook %s +WHERE (inputWarpRun.label = 'ThreePi.Run2.r.v0') + AND ((DEGREES(2*ASIN(SQRT(POW(SIN(inputRawExp.decl - templateRawExp.decl),2) + COS(inputRawExp.decl)*COS(templateRawExp.decl)*POW(SIN(inputRawExp.ra - templateRawExp.ra),2)))) < 0.10000000 + 0.00000119)) + AND ((TIME_TO_SEC(TIMEDIFF(inputRawExp.dateobs, templateRawExp.dateobs)) > 0.00000000 - 0.00000119)) + AND (templateWarpRun.label = 'ThreePi.Run2.r.v0') +-- AND (diffs.diff_id IS NULL) + AND inputWarpRun.state = 'full' +-- WHERE done +GROUP BY input_warp_id +; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + diffRun.* +FROM diffRun +WHERE + diffRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_input_skyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_input_skyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_input_skyfile.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT diffInputSkyfile.* FROM diffInputSkyfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_run.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT diffRun.* from diffRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_skyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_skyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_export_skyfile.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT diffSkyfile.* FROM diffSkyfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_inputskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_inputskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_inputskyfile.sql (revision 41104) @@ -0,0 +1,126 @@ +SELECT * FROM + (SELECT + -- warp input + diffRun.diff_id, + diffInputSkyfile.skycell_id, + diffInputSkyfile.tess_id, + 0 as stack_id, + warpSkyfile.warp_id, + warpSkyfile.uri, + warpSkyfile.path_base, + warpSkyfile.magicked as magicked, + 0 as template, + rawExp.camera + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) + JOIN warpSkyfile + ON diffInputSkyfile.warp1 = warpSkyfile.warp_id + AND diffInputSkyfile.skycell_id = warpSkyfile.skycell_id + AND diffInputSkyfile.tess_id = warpSkyfile.tess_id + JOIN warpRun + ON diffInputSkyfile.warp1 = warpRun.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + -- where hook %s + UNION + SELECT + -- warp template + diffRun.diff_id, + diffInputSkyfile.skycell_id, + diffInputSkyfile.tess_id, + 0 as stack_id, + warpSkyfile.warp_id, + warpSkyfile.uri, + warpSkyfile.path_base, + warpSkyfile.magicked as magicked, + 1 as template, + rawExp.camera + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) + JOIN warpSkyfile + ON diffInputSkyfile.warp2 = warpSkyfile.warp_id + AND diffInputSkyfile.skycell_id = warpSkyfile.skycell_id + AND diffInputSkyfile.tess_id = warpSkyfile.tess_id + JOIN warpRun + ON diffInputSkyfile.warp2 = warpRun.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + -- where hook %s + UNION + SELECT + -- stack input + diffRun.diff_id, + diffInputSkyfile.skycell_id, + diffInputSkyfile.tess_id, + stackSumSkyfile.stack_id, + 0 as warp_id, + stackSumSkyfile.uri, + stackSumSkyfile.path_base, + 0 as magicked, + 0 as template, + rawExp.camera + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) + JOIN stackSumSkyfile + ON diffInputSkyfile.stack1 = stackSumSkyfile.stack_id + JOIN stackInputSkyfile + ON diffInputSkyfile.stack1 = stackInputSkyfile.stack_id + JOIN warpRun + ON stackInputSkyfile.warp_id = warpRun.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + -- where hook %s + UNION + SELECT + -- stack template + diffRun.diff_id, + diffInputSkyfile.skycell_id, + diffInputSkyfile.tess_id, + stackSumSkyfile.stack_id, + 0 as warp_id, + stackSumSkyfile.uri, + stackSumSkyfile.path_base, + 0 as magicked, + 1 as template, + rawExp.camera + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) + JOIN stackSumSkyfile + ON diffInputSkyfile.stack2 = stackSumSkyfile.stack_id + JOIN stackInputSkyfile + ON diffInputSkyfile.stack2 = stackInputSkyfile.stack_id + JOIN warpRun + ON stackInputSkyfile.warp_id = warpRun.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + -- where hook %s + ) as Foo +-- template where hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_listrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_listrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_listrun.sql (revision 41104) @@ -0,0 +1,74 @@ +SELECT DISTINCT + diffRun.diff_id, + diffRun.state, + diffRun.workdir, + diffRun.label, + diffRun.data_group, + diffRun.dist_group, + diffRun.reduction, + diffRun.note, + diffRun.magicked, + diffRun.diff_mode, + diffRun.tess_id, + diffRun.bothways, + warp1, + warp2, + -- The following are only valid for warps + -- XXX This needs to be more clever to handle diffs between stacks + -- Zero points are appropriate for both forward and backward diffs + camProcessedInput.zpt_obs, + camProcessedInput.zpt_stdev, + camProcessedInput.zpt_lq, + camProcessedInput.zpt_uq, + camProcessedInput.fwhm_major, + rawInput.comment, + rawInput.exp_time, + rawInput.camera, + rawInput.exp_name AS exp_name_1, + rawInput.exp_id AS exp_id_1, + rawInput.filter AS filter_1, + rawInput.comment AS comment_1, + rawInput.dateobs AS dateobs_1, + chipInput.chip_id AS chip_id_1, + camInput.cam_id AS cam_id_1, + fakeInput.fake_id AS fake_id_1, + camProcessedInput.sigma_ra AS sigma_ra_1, + camProcessedInput.sigma_dec AS sigma_dec_1, + rawTemplate.exp_name AS exp_name_2, + rawTemplate.exp_id AS exp_id_2, + rawInput.filter AS filter_2, + rawTemplate.comment AS comment_2, + rawTemplate.dateobs AS dateobs_2, + chipTemplate.chip_id AS chip_id_2, + camTemplate.cam_id AS cam_id_2, + fakeTemplate.fake_id AS fake_id_2, + camProcessedTemplate.sigma_ra AS sigma_ra_2, + camProcessedTemplate.sigma_dec AS sigma_dec_2 + -- ,stack1, + -- stack2 +FROM diffRun +JOIN diffInputSkyfile USING(diff_id) +LEFT JOIN warpRun AS warpInput + ON warpInput.warp_id = diffInputSkyfile.warp1 +LEFT JOIN fakeRun AS fakeInput + ON fakeInput.fake_id = warpInput.fake_id +LEFT JOIN camRun AS camInput + ON camInput.cam_id = fakeInput.cam_id +LEFT JOIN camProcessedExp AS camProcessedInput + ON camProcessedInput.cam_id = camInput.cam_id +LEFT JOIN chipRun AS chipInput + ON chipInput.chip_id = camInput.chip_id +LEFT JOIN rawExp AS rawInput + ON rawInput.exp_id = chipInput.exp_id +LEFT JOIN warpRun AS warpTemplate + ON warpTemplate.warp_id = diffInputSkyfile.warp2 +LEFT JOIN fakeRun AS fakeTemplate + ON fakeTemplate.fake_id = warpTemplate.fake_id +LEFT JOIN camRun AS camTemplate + ON camTemplate.cam_id = fakeTemplate.cam_id +LEFT JOIN camProcessedExp AS camProcessedTemplate + ON camProcessedTemplate.cam_id = camTemplate.cam_id +LEFT JOIN chipRun AS chipTemplate + ON chipTemplate.chip_id = camTemplate.chip_id +LEFT JOIN rawExp AS rawTemplate + ON rawTemplate.exp_id = chipTemplate.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_listssrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_listssrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_listssrun.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT DISTINCT + diffRun.diff_id, + diffRun.state, + diffRun.workdir, + diffRun.label, + diffRun.data_group, + diffRun.dist_group, + diffRun.reduction, + diffRun.note, + diffRun.tess_id +FROM diffRun +JOIN diffInputSkyfile USING(diff_id) +JOIN stackRun AS stackInputRun + ON stackInputRun.stack_id = diffInputSkyfile.stack1 +JOIN stackSumSkyfile AS stackInput + ON stackInputRun.stack_id = stackInput.stack_id +JOIN stackRun AS stackTemplateRun + ON stackTemplateRun.stack_id = diffInputSkyfile.stack2 +JOIN stackSumSkyfile AS stackTemplate + ON stackTemplateRun.stack_id = stackTemplate.stack_id +WHERE diff_mode = 4 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,128 @@ +-- does this result in too many entries (one for each diffInputSkyfile?) +-- all of this is just to get the camera used for the diff run +SELECT DISTINCT * FROM + (SELECT DISTINCT + diffRun.diff_id, + rawExp.camera, + diffRun.state, + diffRun.workdir, + diffRun.tess_id, + diffRun.label + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) + JOIN warpSkyfile + ON diffInputSkyfile.warp1 = warpSkyfile.warp_id + AND diffInputSkyfile.skycell_id = warpSkyfile.skycell_id + AND diffInputSkyfile.tess_id = warpSkyfile.tess_id + JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + WHERE + @INNERCONSTRAINT@ + (diffRun.state = 'goto_cleaned' OR + diffRun.state = 'goto_scrubbed' OR + diffRun.state = 'goto_purged') + @INNERLIMITS@ + UNION + SELECT DISTINCT + diffRun.diff_id, + rawExp.camera, + diffRun.state, + diffRun.workdir, + diffRun.tess_id, + diffRun.label + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) + JOIN warpSkyfile + ON diffInputSkyfile.warp2 = warpSkyfile.warp_id + AND diffInputSkyfile.skycell_id = warpSkyfile.skycell_id + AND diffInputSkyfile.tess_id = warpSkyfile.tess_id + JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + WHERE + @INNERCONSTRAINT@ + (diffRun.state = 'goto_cleaned' OR + diffRun.state = 'goto_scrubbed' OR + diffRun.state = 'goto_purged') + @INNERLIMITS@ + UNION + SELECT DISTINCT + diffRun.diff_id, + rawExp.camera, + diffRun.state, + diffRun.workdir, + diffRun.tess_id, + diffRun.label + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) +-- JOIN stackSumSkyfile +-- ON diffInputSkyfile.stack1 = stackSumSkyfile.stack_id + JOIN stackInputSkyfile + ON diffInputSkyfile.stack1 = stackInputSkyfile.stack_id + JOIN warpRun + ON warpRun.warp_id = stackInputSkyfile.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + WHERE + @INNERCONSTRAINT@ + (diffRun.state = 'goto_cleaned' OR + diffRun.state = 'goto_scrubbed' OR + diffRun.state = 'goto_purged') + @INNERLIMITS@ + UNION + SELECT DISTINCT + diffRun.diff_id, + rawExp.camera, + diffRun.state, + diffRun.workdir, + diffRun.tess_id, + diffRun.label + FROM diffRun + JOIN diffInputSkyfile + USING(diff_id) +-- JOIN stackSumSkyfile +-- ON diffInputSkyfile.stack2 = stackSumSkyfile.stack_id + JOIN stackInputSkyfile + ON diffInputSkyfile.stack2 = stackInputSkyfile.stack_id + JOIN warpRun + ON warpRun.warp_id = stackInputSkyfile.warp_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + WHERE + @INNERCONSTRAINT@ + (diffRun.state = 'goto_cleaned' OR + diffRun.state = 'goto_scrubbed' OR + diffRun.state = 'goto_purged') + @INNERLIMITS@ + ) as Foo + WHERE 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanupskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanupskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanupskyfile.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + diffSkyfile.*, + diffRun.state, + diffRun.workdir, + diffRun.dvodb, + diffRun.tess_id +FROM diffRun +JOIN diffSkyfile + USING(diff_id) +WHERE + ((diffRun.state = 'goto_cleaned' AND (diffSkyfile.data_state = 'full' OR diffSkyfile.data_state = 'update') AND diffSkyfile.quality = 0) + OR + (diffRun.state = 'goto_scrubbed' AND diffSkyfile.data_state != 'scrubbed') + OR + (diffRun.state = 'goto_purged' AND diffSkyfile.data_state != 'purged')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanupskyfile_all.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanupskyfile_all.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_pendingcleanupskyfile_all.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + diffSkyfile.*, + diffRun.state, + diffRun.workdir, + diffRun.dvodb, + diffRun.tess_id +FROM diffRun +JOIN diffSkyfile + USING(diff_id) +WHERE ( diffRun.state = 'goto_cleaned' OR diffRun.state = 'goto_scrubbed' OR diffRun.state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertcleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertcleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertcleanup.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE diffSkyfile + JOIN diffRun using(diff_id) +SET diffSkyfile.data_state = 'full', + diffRun.state = '%s' +WHERE + diffRun.state = '%s' + AND diffSkyfile.data_state = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_delete.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_delete.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_delete.sql (revision 41104) @@ -0,0 +1,6 @@ +DELETE FROM diffSkyfile +USING diffSkyfile, diffRun +WHERE + diffRun.diff_id = diffSkyfile.diff_id + AND diffRun.state = 'new' + AND fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_update.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_update.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_update.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE diffRun +JOIN diffSkyfile USING(diff_id) +SET diffRun.state = 'new' +WHERE + diffSkyfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_updated.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_updated.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_revertdiffskyfile_updated.sql (revision 41104) @@ -0,0 +1,6 @@ +UPDATE diffSkyfile + JOIN diffRun USING(diff_id) +SET diffSkyfile.fault = 0 +WHERE diffRun.state = 'update' + AND diffSkyfile.data_state = 'update' + AND diffSkyfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_setskyfiletoupdate.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_setskyfiletoupdate.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_setskyfiletoupdate.sql (revision 41104) @@ -0,0 +1,10 @@ +UPDATE diffRun + JOIN diffSkyfile USING(diff_id) +SET diffRun.state = 'update', + diffSkyfile.data_state = 'update', + diffSkyfile.fault = 0 + -- set hook %s +WHERE diff_id = %lld + AND (diffRun.state = 'cleaned' OR diffRun.state = 'update') + AND (diffSkyfile.data_state = 'cleaned' OR diffSkyfile.data_state = 'update') + AND (diffSkyfile.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_skyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_skyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_skyfile.sql (revision 41104) @@ -0,0 +1,66 @@ +SELECT + diffSkyfile.*, + diffInputSkyfile.diff_skyfile_id, + diffRun.tess_id, + diffRun.state, + diffRun.workdir, + diffRun.label, + diffRun.bothways, + diffRun.diff_mode, + warp1, + stack1, + warp2, + stack2, + -- The following are only valid for warps + -- XXX This needs to be more clever to handle diffs between stacks + -- Zero points are appropriate for both forward and backward diffs + camProcessedInput.zpt_obs, + camProcessedInput.zpt_stdev, + camProcessedInput.zpt_lq, + camProcessedInput.zpt_uq, + rawInput.comment, + rawInput.exp_time, + rawInput.camera, + rawInput.exp_name AS exp_name_1, + rawInput.exp_id AS exp_id_1, + rawInput.filter AS filter_1, + chipInput.chip_id AS chip_id_1, + camInput.cam_id AS cam_id_1, + fakeInput.fake_id AS fake_id_1, + camProcessedInput.sigma_ra AS sigma_ra_1, + camProcessedInput.sigma_dec AS sigma_dec_1, + rawTemplate.exp_name AS exp_name_2, + rawTemplate.exp_id AS exp_id_2, + rawTemplate.filter AS filter_2, + chipTemplate.chip_id AS chip_id_2, + camTemplate.cam_id AS cam_id_2, + fakeTemplate.fake_id AS fake_id_2, + camProcessedTemplate.sigma_ra AS sigma_ra_2, + camProcessedTemplate.sigma_dec AS sigma_dec_2 +FROM diffRun +JOIN diffSkyfile USING(diff_id) +JOIN diffInputSkyfile USING(diff_id, skycell_id) +LEFT JOIN warpRun AS warpInput + ON warpInput.warp_id = diffInputSkyfile.warp1 +LEFT JOIN fakeRun AS fakeInput + ON fakeInput.fake_id = warpInput.fake_id +LEFT JOIN camRun AS camInput + ON camInput.cam_id = fakeInput.cam_id +LEFT JOIN camProcessedExp AS camProcessedInput + ON camProcessedInput.cam_id = camInput.cam_id +LEFT JOIN chipRun AS chipInput + ON chipInput.chip_id = camInput.chip_id +LEFT JOIN rawExp AS rawInput + ON rawInput.exp_id = chipInput.exp_id +LEFT JOIN warpRun AS warpTemplate + ON warpTemplate.warp_id = diffInputSkyfile.warp2 +LEFT JOIN fakeRun AS fakeTemplate + ON fakeTemplate.fake_id = warpTemplate.fake_id +LEFT JOIN camRun AS camTemplate + ON camTemplate.cam_id = fakeTemplate.cam_id +LEFT JOIN camProcessedExp AS camProcessedTemplate + ON camProcessedTemplate.cam_id = camTemplate.cam_id +LEFT JOIN chipRun AS chipTemplate + ON chipTemplate.chip_id = camTemplate.chip_id +LEFT JOIN rawExp AS rawTemplate + ON rawTemplate.exp_id = chipTemplate.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_todiffskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_todiffskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_todiffskyfile.sql (revision 41104) @@ -0,0 +1,97 @@ +-- This query is a little complicated, since it contains multiple self-joins. +-- This is because of the options involved: both templates and inputs can be either warps or stacks. +-- All of the templates and inputs need to exist, so we query against all of them. +SELECT DISTINCT + diffRun.diff_id, + diffInputSkyfile.diff_skyfile_id, + diffRun.workdir, + diffInputSkyfile.skycell_id, + diffRun.tess_id, + diffRun.label, + diffRun.state, + diffRun.reduction, + diffRun.bothways, + diffRun.diff_mode, + diffSkyfile.path_base, + IFNULL(priority, 10000) AS priority +FROM diffRun +JOIN diffInputSkyfile USING(diff_id) + +-- Get list of templates for each diffRun +-- JOIN diffInputSkyfile AS diffTemplateSkyfile +-- ON diffRun.diff_id = diffTemplateSkyfile.diff_id +-- AND diffRun.skycell_id = diffTemplateSkyfile.skycell_id +-- AND diffTemplateSkyfile.template = 1 +-- Get list of inputs for each diffRun +-- JOIN diffInputSkyfile +-- ON diffRun.diff_id = diffInputSkyfile.diff_id +-- AND diffRun.skycell_id = diffInputSkyfile.skycell_id +-- AND diffInputSkyfile.template = 0 + +-- Get warp templates +LEFT JOIN warpRun AS warpTemplateRun + ON warpTemplateRun.warp_id = diffInputSkyfile.warp2 + AND diffInputSkyfile.warp2 IS NOT NULL +LEFT JOIN warpSkyfile AS warpTemplateSkyfile + ON warpTemplateSkyfile.warp_id = warpTemplateRun.warp_id + AND warpTemplateSkyfile.skycell_id = diffInputSkyfile.skycell_id + +-- Get warp inputs +LEFT JOIN warpRun + ON warpRun.warp_id = diffInputSkyfile.warp1 + AND diffInputSkyfile.warp1 IS NOT NULL +LEFT JOIN warpSkyfile + ON warpSkyfile.warp_id = warpRun.warp_id + AND warpSkyfile.skycell_id = diffInputSkyfile.skycell_id + +-- Get stack templates +LEFT JOIN stackRun AS stackTemplateRun + ON stackTemplateRun.stack_id = diffInputSkyfile.stack2 + AND diffInputSkyfile.stack2 IS NOT NULL +LEFT JOIN stackSumSkyfile AS stackTemplateSkyfile + ON stackTemplateSkyfile.stack_id = stackTemplateRun.stack_id + +-- Get stack inputs +LEFT JOIN stackRun + ON stackRun.stack_id = diffInputSkyfile.stack1 + AND diffInputSkyfile.stack1 IS NOT NULL +LEFT JOIN stackSumSkyfile + ON stackSumSkyfile.stack_id = stackRun.stack_id + +-- Get what's already been processed +LEFT JOIN diffSkyfile + ON diffInputSkyfile.diff_id = diffSkyfile.diff_id + AND diffInputSkyfile.skycell_id = diffSkyfile.skycell_id +LEFT JOIN Label ON Label.label = diffRun.label +WHERE +-- Ready to be processed + ((diffRun.state = 'new' + AND diffSkyfile.diff_id IS NULL) + OR (diffRun.state = 'update' + AND diffSkyfile.fault = 0 + AND diffSkyfile.data_state = 'update') + ) + AND (Label.active OR Label.active IS NULL) +-- Ensure input warps are available + AND (diffInputSkyfile.warp1 IS NULL + OR (warpSkyfile.data_state = 'full' + AND warpSkyfile.fault = 0 +-- AND warpSkyfile.magicked >= 0 + AND warpSkyfile.quality = 0)) +-- Ensure input stacks are available + AND (diffInputSkyfile.stack1 IS NULL + OR (stackRun.state = 'full' + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0)) +-- Ensure template warps are available + AND (diffInputSkyfile.warp2 IS NULL + OR (warpTemplateSkyfile.data_state = 'full' + AND warpTemplateSkyfile.fault = 0 +-- AND warpSkyfile.magicked >= 0 + AND warpTemplateSkyfile.quality = 0)) +-- Ensure template stacks are available + AND (diffInputSkyfile.stack2 IS NULL + OR (stackTemplateRun.state = 'full' + AND stackTemplateSkyfile.fault = 0 + AND stackTemplateSkyfile.quality = 0)) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_tosummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_tosummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/difftool_tosummary.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT DISTINCT diff_id,"GPC1" as camera,diffRun.workdir,diffRun.tess_id,diffRun.diff_mode,diffRun.state + FROM diffRun + LEFT JOIN diffSummary USING(diff_id) +WHERE diffRun.state = 'full' AND + diffSummary.projection_cell IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_SSdiff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_SSdiff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_SSdiff.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT DISTINCT + 'SSdiff' as stage, + diffRun.diff_id AS stage_id, + diffRun.magicked, + CONCAT_WS('.','SSdiff',convert(diffRun.diff_id,CHAR)) as run_tag, + diffRun.label, + diffRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM diffRun +JOIN diffInputSkyfile using(diff_id) +JOIN stackRun ON diffInputSkyfile.stack1 = stackRun.stack_id +JOIN distTarget ON distTarget.stage = 'SSdiff' + AND distTarget.filter = stackRun.filter + AND distTarget.dist_group = diffRun.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = diff_id) + AND distTarget.target_id = distRun.target_id + -- JOIN hook %s +WHERE diffInputSkyfile.stack1 IS NOT NULL AND diffInputSkyfile.stack2 IS NOT NULL + AND distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((diffRun.state = 'full') OR (distTarget.clean AND diffRun.state = 'cleaned')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_camera.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT DISTINCT + 'camera' as stage, + camRun.cam_id as stage_id, + rawExp.exp_name as run_tag, + camRun.magicked, + camRun.label, + camRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM camRun +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN distTarget ON distTarget.stage = 'camera' + AND rawExp.filter = distTarget.filter + AND camRun.dist_group = distTarget.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON camRun.cam_id = distRun.stage_id + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((camRun.state = 'full') OR (distTarget.clean AND camRun.state = 'cleaned')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_chip.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT DISTINCT + 'chip' as stage, + chipRun.chip_id as stage_id, + chipRun.magicked, + rawExp.exp_name as run_tag, + chipRun.label, + chipRun.data_group, + chipRun.dist_group, + distTarget.target_id, + distTarget.clean +FROM chipRun +JOIN rawExp USING(exp_id) +JOIN distTarget ON distTarget.stage = 'chip' + AND chipRun.dist_group = distTarget.dist_group + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON distRun.stage_id = chipRun.chip_id + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((chipRun.state = 'full') OR (distTarget.clean AND chipRun.state = 'cleaned')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_chip_bg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_chip_bg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_chip_bg.sql (revision 41104) @@ -0,0 +1,25 @@ +SELECT DISTINCT + 'chip_bg' as stage, + chipBackgroundRun.chip_bg_id as stage_id, + chipBackgroundRun.magicked, + rawExp.exp_name as run_tag, + chipBackgroundRun.label, + chipBackgroundRun.data_group, + chipBackgroundRun.dist_group, + distTarget.target_id, + distTarget.clean +FROM chipBackgroundRun +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN distTarget + ON distTarget.stage = 'chip_bg' + AND chipBackgroundRun.dist_group = distTarget.dist_group + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun + ON distRun.stage_id = chipBackgroundRun.chip_bg_id + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((chipBackgroundRun.state = 'full') OR (distTarget.clean AND chipBackgroundRun.state = 'cleaned')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_diff.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT DISTINCT + 'diff' as stage, + diffRun.diff_id AS stage_id, + diffRun.magicked, + rawExp.exp_name as run_tag, + diffRun.label, + diffRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM diffRun +JOIN diffInputSkyfile using(diff_id) +JOIN warpRun on diffInputSkyfile.warp1 = warpRun.warp_id +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN distTarget ON distTarget.stage = 'diff' + AND distTarget.filter = rawExp.filter + AND distTarget.dist_group = diffRun.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = diff_id) + AND distTarget.target_id = distRun.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((diffRun.state = 'full') OR (distTarget.clean AND diffRun.state = 'cleaned')) + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_fake.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_fake.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_fake.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT DISTINCT + 'fake' as stage, + fakeRun.fake_id AS stage_id, + CAST(0 AS SIGNED) AS magicked, + rawExp.exp_name as run_tag, + fakeRun.label, + fakeRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM fakeRun +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN distTarget ON distTarget.stage = 'fake' AND fakeRun.dist_group = distTarget.dist_group + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = fake_id) + AND distRun.target_id = distTarget.target_id +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((fakeRun.state = 'full') OR (distTarget.clean AND fakeRun.state = 'cleaned')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_ff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_ff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_ff.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT DISTINCT + 'ff' as stage, + fullForceRun.ff_id AS stage_id, + CAST(0 AS SIGNED) AS magicked, + -- run tag in the form 'ff.$skycell_id.$ff_id' + CONCAT_WS('.', 'ff', stackRun.skycell_id, convert(fullForceRun.ff_id, CHAR)) as run_tag, + fullForceRun.label, + fullForceRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM fullForceRun +JOIN skycalRun USING(skycal_id) +JOIN stackRun using(stack_id) +JOIN distTarget ON distTarget.stage = 'ff' + AND fullForceRun.dist_group = distTarget.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = ff_id) + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND fullForceRun.state = 'full' + AND stackRun.filter = distTarget.filter + -- we shouldn't need to check fault. If faulted it shouldn't be full + -- AND (skycalResult.fault = 0 AND skycalResult.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_raw.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + 'raw' AS stage, + rawExp.exp_id AS stage_id, + rawExp.exp_name AS run_tag, + rawExp.magicked, + camRun.label, + camRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM rawExp +JOIN magicDSRun ON stage = 'raw' AND magicDSRun.stage_id = exp_id +JOIN camRun USING(cam_id) +JOIN distTarget ON distTarget.dist_group = camRun.dist_group AND distTarget.stage = 'raw' + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON distRun.stage_id = exp_id AND distRun.target_id = distTarget.target_id +WHERE distTarget.state = 'enabled' -- target and intrest are enabled + AND rcInterest.state = 'enabled' + AND magicDSRun.state = 'full' -- destreaked files are available Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_raw_no_magic.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_raw_no_magic.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_raw_no_magic.sql (revision 41104) @@ -0,0 +1,17 @@ +SELECT + 'raw' AS stage, + rawExp.exp_id AS stage_id, + rawExp.exp_name AS run_tag, + rawExp.magicked, + CAST(NULL AS CHAR(255)) AS label, + CAST(NULL AS CHAR(255)) AS data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM rawExp +JOIN distTarget ON distTarget.stage = 'raw' + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON distRun.stage_id = exp_id AND distRun.target_id = distTarget.target_id +WHERE distTarget.state = 'enabled' -- target and intrest are enabled + AND rcInterest.state = 'enabled' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_sky.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_sky.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_sky.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT DISTINCT + 'sky' as stage, + staticskyRun.sky_id AS stage_id, + CAST(0 AS SIGNED) AS magicked, + -- run tag in the form 'sky.$skycell_id.$sky_id' + CONCAT_WS('.', 'sky', stackRun.skycell_id, convert(staticskyRun.sky_id, CHAR)) as run_tag, + staticskyRun.label, + staticskyRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM staticskyRun +JOIN staticskyResult USING(sky_id) +JOIN staticskyInput USING(sky_id) +JOIN stackRun using(stack_id) +JOIN distTarget ON distTarget.stage = 'sky' + AND staticskyRun.dist_group = distTarget.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = sky_id) + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND distTarget.filter = 'multi' + AND ((staticskyRun.state = 'full') OR (distTarget.clean AND staticskyRun.state = 'cleaned')) + -- we shouldn't need to check fault. If faulted it shouldn't be full + AND (staticskyResult.fault = 0 AND staticskyResult.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_sky_singlefilter.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_sky_singlefilter.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_sky_singlefilter.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT + 'sky' as stage, + staticskyRun.sky_id AS stage_id, + CAST(0 AS SIGNED) AS magicked, + -- run tag in the form 'sky.$skycell_id.$stack_id' + CONCAT_WS('.', 'sky', stackRun.skycell_id, convert(staticskyRun.sky_id, CHAR)) as run_tag, + staticskyRun.label, + staticskyRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM staticskyRun +JOIN staticskyResult USING(sky_id) +JOIN staticskyInput USING(sky_id) +JOIN stackRun using(stack_id) +JOIN distTarget ON distTarget.stage = 'sky' + AND staticskyRun.dist_group = distTarget.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = sky_id) + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((staticskyRun.state = 'full') OR (distTarget.clean AND staticskyRun.state = 'cleaned')) + AND stackRun.filter = distTarget.filter + -- we shouldn't need to check fault. If faulted it shouldn't be full + AND (staticskyResult.fault = 0 AND staticskyResult.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_skycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_skycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_skycal.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT DISTINCT + 'skycal' as stage, + skycalRun.skycal_id AS stage_id, + CAST(0 AS SIGNED) AS magicked, + -- run tag in the form 'skycal.$skycell_id.$skycal_id' + CONCAT_WS('.', 'skycal', stackRun.skycell_id, convert(skycalRun.skycal_id, CHAR)) as run_tag, + skycalRun.label, + skycalRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM skycalRun +JOIN skycalResult USING(skycal_id) +JOIN stackRun using(stack_id) +JOIN distTarget ON distTarget.stage = 'skycal' + AND skycalRun.dist_group = distTarget.dist_group +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = skycal_id) + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((skycalRun.state = 'full') OR (distTarget.clean AND skycalRun.state = 'cleaned')) + AND stackRun.filter = distTarget.filter + -- we shouldn't need to check fault. If faulted it shouldn't be full + AND (skycalResult.fault = 0 AND skycalResult.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_stack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_stack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_stack.sql (revision 41104) @@ -0,0 +1,25 @@ +SELECT DISTINCT + 'stack' as stage, + stackRun.stack_id AS stage_id, + CAST(0 AS SIGNED) AS magicked, + -- run tag in the form 'stack.$skycell_id.$stack_id' + CONCAT_WS('.', 'stack', stackRun.skycell_id, convert(stackRun.stack_id, CHAR)) as run_tag, + stackRun.label, + stackRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM stackRun +JOIN stackSumSkyfile USING(stack_id) +JOIN distTarget ON distTarget.stage = 'stack' + AND stackRun.dist_group = distTarget.dist_group + AND stackRun.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = stack_id) + AND distRun.target_id = distTarget.target_id + -- JOIN hook %s +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((stackRun.state = 'full') OR (distTarget.clean AND stackRun.state = 'cleaned')) + -- we shouldn't need to check fault. If faulted it shouldn't be full + AND (stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_warp.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT DISTINCT + 'warp' as stage, + warpRun.warp_id AS stage_id, + warpRun.magicked, + rawExp.exp_name as run_tag, + warpRun.label, + warpRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM warpRun +JOIN fakeRun USING(fake_id) +JOIN camRun ON fakeRun.cam_id = camRun.cam_id +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN distTarget ON distTarget.stage = 'warp' + AND warpRun.dist_group = distTarget.dist_group + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = warp_id) + AND distRun.target_id = distTarget.target_id +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((warpRun.state = 'full') OR (distTarget.clean AND warpRun.state = 'cleaned')) + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_warp_bg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_warp_bg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_definebyquery_warp_bg.sql (revision 41104) @@ -0,0 +1,29 @@ +SELECT DISTINCT + 'warp_bg' as stage, + warpBackgroundRun.warp_bg_id AS stage_id, + warpBackgroundRun.magicked, + rawExp.exp_name as run_tag, + warpBackgroundRun.label, + warpBackgroundRun.data_group, + distTarget.dist_group, + distTarget.target_id, + distTarget.clean +FROM warpBackgroundRun +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun ON fakeRun.cam_id = camRun.cam_id +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +JOIN distTarget + ON distTarget.stage = 'warp_bg' + AND warpBackgroundRun.dist_group = distTarget.dist_group + AND rawExp.filter = distTarget.filter +JOIN rcInterest USING(target_id) +LEFT JOIN distRun ON (distRun.stage_id = warp_bg_id) + AND distRun.target_id = distTarget.target_id +WHERE distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND ((warpBackgroundRun.state = 'full') OR (distTarget.clean AND warpBackgroundRun.state = 'cleaned')) + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_defineinterest.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_defineinterest.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_defineinterest.sql (revision 41104) @@ -0,0 +1,6 @@ +INSERT INTO rcInterest + SELECT 0, rcDestination.dest_id, distTarget.target_id, '@STATE@' + FROM distTarget + JOIN rcDestination + LEFT JOIN rcInterest USING(dest_id, target_id) + WHERE (int_id IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_listfilesets.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_listfilesets.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_listfilesets.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + fs_id, + dist_id, + distRun.label, + rcDSFileset.state AS fileset_state, + rcDSFileset.name AS fileset, + rcDestination.name AS product, + rcDestination.dbname, + rcDestination.dbhost +FROM rcDSFileset +JOIN distRun USING(dist_id) +JOIN distTarget USING(target_id, stage, clean) +JOIN rcInterest USING(target_id, dest_id) +JOIN rcDestination USING(dest_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_listinterests.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_listinterests.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_listinterests.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + int_id, + rcInterest.state AS state, + dest_id, + name AS dest_name, + rcDestination.state as dest_state, + dist_group, + filter, + stage, + clean, + target_id, + distTarget.state as target_state +FROM rcInterest + JOIN distTarget USING(target_id) + JOIN rcDestination USING(dest_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_SSdiff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_SSdiff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_SSdiff.sql (revision 41104) @@ -0,0 +1,53 @@ +SELECT DISTINCT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + diffSkyfile.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + diffSkyfile.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + diffRun.state, + -- data_state doesn't exist yet + -- diffSkyfile.data_state, + 'full' AS data_state, + diffSkyfile.quality, + distRun.no_magic, + 0 AS magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN diffRun ON stage_id = diff_id +JOIN diffSkyfile using(diff_id) +JOIN diffInputSkyfile USING(diff_id, skycell_id) +JOIN stackRun + ON stackRun.stack_id = diffInputSkyfile.stack1 +JOIN stackSumSkyfile + USING(stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpSkyfile + ON stackInputSkyfile.warp_id = warpSkyfile.warp_id + AND stackRun.skycell_id = warpSkyfile.skycell_id + AND stackRun.tess_id = warpSkyfile.tess_id +JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND diffSkyfile.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'SSdiff' + AND distComponent.dist_id IS NULL + AND ((diffRun.magicked > 0) OR distRun.no_magic) + AND (diffRun.state = 'full' OR (distRun.clean AND diffRun.state = 'cleaned')) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_camera.sql (revision 41104) @@ -0,0 +1,36 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + 'exposure' AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + camProcessedExp.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + camRun.state, + camRun.state AS data_state, + camProcessedExp.quality, + distRun.no_magic, + chipRun.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN camRun ON camRun.cam_id = distRun.stage_id +JOIN camProcessedExp USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp using(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id +LEFT JOIN Label + ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'camera' + AND distComponent.dist_id IS NULL + AND (((clean OR (chipRun.magicked != 0)) AND (camRun.magicked > 0)) OR distRun.no_magic) + AND (camRun.state = 'full' OR (distRun.clean AND camRun.state = 'cleaned')) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_chip.sql (revision 41104) @@ -0,0 +1,36 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + chipProcessedImfile.class_id AS component, + exp_type, + distRun.clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + chipProcessedImfile.path_base, + chipProcessedImfile.path_base as chip_path_base, + chipRun.state, + chipProcessedImfile.data_state, + chipProcessedImfile.quality, + distRun.no_magic, + chipProcessedImfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN chipRun ON chipRun.chip_id = distRun.stage_id +JOIN rawExp using(exp_id) +JOIN chipProcessedImfile ON chipProcessedImfile.chip_id = stage_id +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND chipProcessedImfile.class_id = distComponent.component +LEFT JOIN Label + ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'chip' + AND distComponent.dist_id IS NULL + AND ((chipRun.magicked > 0) OR distRun.no_magic) + AND (chipRun.state = 'full' OR (distRun.clean AND chipRun.state = 'cleaned')) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_chip_bg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_chip_bg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_chip_bg.sql (revision 41104) @@ -0,0 +1,36 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + chipBackgroundImfile.class_id AS component, + exp_type, + distRun.clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + chipBackgroundImfile.path_base, + chipBackgroundImfile.path_base as chip_path_base, + chipBackgroundRun.state, + chipBackgroundRun.state AS data_state, + chipBackgroundImfile.quality AS quality, + distRun.no_magic, + chipBackgroundImfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN chipBackgroundRun ON chipBackgroundRun.chip_bg_id = distRun.stage_id +JOIN chipRun using(chip_id) +JOIN rawExp using(exp_id) +JOIN chipBackgroundImfile ON chipBackgroundImfile.chip_bg_id = stage_id +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND chipBackgroundImfile.class_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'chip_bg' + AND distComponent.dist_id IS NULL + AND ((chipBackgroundRun.magicked > 0) OR distRun.no_magic) + AND (chipBackgroundRun.state = 'full') + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_diff.sql (revision 41104) @@ -0,0 +1,43 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + diffSkyfile.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + diffSkyfile.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + diffRun.state, + -- data_state doesn't exist yet + -- diffSkyfile.data_state, + 'full' AS data_state, + diffSkyfile.quality, + distRun.no_magic, + diffSkyfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN diffRun ON stage_id = diff_id +JOIN diffSkyfile using(diff_id) +JOIN diffInputSkyfile USING(diff_id, skycell_id) +JOIN warpRun + ON warpRun.warp_id = diffInputSkyfile.warp1 +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND diffSkyfile.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'diff' + AND distComponent.dist_id IS NULL + AND ((diffRun.magicked > 0) OR distRun.no_magic) + AND (diffRun.state = 'full' OR (distRun.clean AND diffRun.state = 'cleaned')) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_fake.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_fake.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_fake.sql (revision 41104) @@ -0,0 +1,35 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + fakeProcessedImfile.class_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + fakeProcessedImfile.path_base, + CAST(NULL AS CHAR(255)) AS chip_path_base, + fakeRun.state, + fakeRun.state AS data_state, + 0 as quality, + distRun.no_magic, + 0 as magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN fakeRun ON fakeRun.fake_id = distRun.stage_id +JOIN fakeProcessedImfile USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id, exp_id) +JOIN rawExp using(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND fakeProcessedImfile.class_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'fake' + AND distComponent.dist_id IS NULL + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_ff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_ff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_ff.sql (revision 41104) @@ -0,0 +1,86 @@ +-- look for fullForceResult components and a summary components +SELECT * FROM +( +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + CONCAT_WS('.', warp_id, stackRun.skycell_id) AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + fullForceResult.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + fullForceRun.state, + 'full' AS data_state, + fullForceResult.quality, + distRun.no_magic, + warpRun.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN fullForceRun ON stage_id = ff_id +JOIN skycalRun USING(skycal_id) +JOIN stackRun USING(stack_id) +JOIN fullForceResult USING(ff_id) +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun ON camRun.chip_id = chipRun.chip_id +JOIN rawExp using(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND CONCAT_WS('.', warp_id, stackRun.skycell_id) = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'ff' + AND distComponent.dist_id IS NULL + AND (Label.active OR Label.active IS NULL) +UNION +SELECT DISTINCT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + 'summary' AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + fullForceSummary.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + fullForceRun.state, + 'full' AS data_state, + fullForceSummary.quality, + distRun.no_magic, + 0 AS magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN fullForceRun ON stage_id = ff_id +JOIN fullForceSummary USING(ff_id) +JOIN skycalRun USING(skycal_id) +JOIN stackRun USING(stack_id) +JOIN stackInputSkyfile USING(stack_id) +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun ON camRun.chip_id = chipRun.chip_id +JOIN rawExp using(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND 'summary' = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'ff' + AND distComponent.dist_id IS NULL + AND (Label.active OR Label.active IS NULL) +) as distRun +-- where arguments are appended here +WHERE 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_raw.sql (revision 41104) @@ -0,0 +1,172 @@ +SELECT * FROM ( +-- rawExp magicked with re_place +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + 'raw' AS stage, + rawExp.exp_id AS stage_id, + rawImfile.class_id AS component, + rawExp.exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', distRun.outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + -- XXX: replace this with rawImfile.path_base once it exists + TRIM(TRAILING '.fits' FROM rawImfile.uri) AS path_base, + CAST(NULL AS CHAR(255)) AS alt_path_base, + -- pass camera stage path base since we want the camera mask file. The script knows what to do + camProcessedExp.path_base as chip_path_base, + CAST(NULL AS CHAR(255)) AS state, + CAST(NULL AS CHAR(255)) AS data_state, + 0 as quality, + distRun.no_magic, + rawImfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN rawExp ON rawExp.exp_id = stage_id AND distTarget.stage = 'raw' AND distRun.alternate = 0 +JOIN rawImfile USING(exp_id) +JOIN chipRun ON chipRun.exp_id = rawExp.exp_id AND chipRun.magicked +JOIN chipProcessedImfile + USING(chip_id, class_id) +JOIN camRun USING(chip_id) +JOIN camProcessedExp using(cam_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND rawImfile.class_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.clean = 0 + AND distRun.stage = 'raw' + AND distComponent.dist_id IS NULL + -- AND (rawExp.magicked OR distRun.no_magic) + AND rawExp.magicked + AND rawImfile.ignored = 0 + -- need to have magicked the chip image which makes the camera mask + AND chipProcessedImfile.magicked != 0 + AND camRun.magicked > 0 + AND (Label.active OR Label.active IS NULL) +UNION +SELECT + -- raw images no_magic required + distRun.dist_id, + distRun.label, + distTarget.dist_group, + 'raw' AS stage, + rawExp.exp_id AS stage_id, + rawImfile.class_id AS component, + rawExp.exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', distRun.outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + -- XXX: replace this with rawImfile.path_base once it exists + TRIM(TRAILING '.fits' FROM rawImfile.uri) AS path_base, + CAST(NULL AS CHAR(255)) AS alt_path_base, + -- pass camera stage path base since we want the camera mask file. The script knows what to do + CAST(NULL AS CHAR(255)) AS chip_path_base, + CAST(NULL AS CHAR(255)) AS state, + CAST(NULL AS CHAR(255)) AS data_state, + 0 as quality, + distRun.no_magic, + rawImfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN rawExp ON rawExp.exp_id = stage_id + AND distTarget.stage = 'raw' AND distRun.alternate = 0 +JOIN rawImfile USING(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND rawImfile.class_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.clean = 0 + AND distRun.stage = 'raw' + AND distComponent.dist_id IS NULL + AND (distRun.no_magic) + AND rawImfile.ignored = 0 + AND (Label.active OR Label.active IS NULL) +UNION + -- raw stage alternate inputs +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + 'raw' AS stage, + rawExp.exp_id AS stage_id, + rawImfile.class_id AS component, + rawExp.exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', distRun.outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + -- XXX: replace this with rawImfile.path_base once it exists + TRIM(TRAILING '.fits' FROM rawImfile.uri) AS path_base, + magicDSFile.backup_path_base AS alt_path_base, + -- pass camera stage path base since we want the camera mask file. The script knows what to do + camProcessedExp.path_base AS chip_path_base, + CAST('full' AS CHAR(255)) AS state, + CAST('full' AS CHAR(255)) AS data_state, + 0 as quality, + distRun.no_magic, + rawImfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN rawExp ON rawExp.exp_id = stage_id AND distTarget.stage = 'raw' AND distRun.alternate = 1 +JOIN rawImfile USING(exp_id) +JOIN magicDSRun ON magicDSRun.stage = 'raw' AND magicDSRun.stage_id = rawExp.exp_id +JOIN magicDSFile ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND rawImfile.class_id = magicDSFile.component +JOIN camRun USING(cam_id) +JOIN camProcessedExp USING(cam_id) +JOIN chipProcessedImfile USING(chip_id, class_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND rawImfile.class_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.clean = 0 + AND distComponent.dist_id IS NULL + AND chipProcessedImfile.magicked != 0 + AND camRun.magicked > 0 + AND rawImfile.ignored = 0 + AND (Label.active OR Label.active IS NULL) +UNION + -- raw stage clean (dbinfo only) +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + 'raw' AS stage, + rawExp.exp_id AS stage_id, + 'exposure' AS component, + rawExp.exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', distRun.outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + CAST(NULL AS CHAR(255)) AS path_base, + CAST(NULL AS CHAR(255)) AS alt_path_base, + CAST(NULL AS CHAR(255)) AS chip_path_base, + CAST(NULL AS CHAR(255)) AS state, + CAST(NULL AS CHAR(255)) AS data_state, + 0 as quality, + distRun.no_magic, + rawExp.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN rawExp ON rawExp.exp_id = stage_id +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'raw' + AND distRun.clean + AND distComponent.dist_id IS NULL + AND (Label.active OR Label.active IS NULL) +) AS distRun +WHERE 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_sky.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_sky.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_sky.sql (revision 41104) @@ -0,0 +1,54 @@ +SELECT DISTINCT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + stackRun.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + staticskyResult.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + staticskyRun.state, + staticskyRun.state AS data_state, + staticskyResult.quality, + 1 AS no_magic, + 0 AS magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN staticskyRun + ON stage_id = sky_id +JOIN staticskyResult + USING(sky_id) +JOIN staticskyInput + USING(sky_id) +JOIN stackRun USING(stack_id) +JOIN stackInputSkyfile USING(stack_id) +JOIN warpSkyfile + ON stackInputSkyfile.warp_id = warpSkyfile.warp_id + AND stackRun.skycell_id = warpSkyfile.skycell_id + AND stackRun.tess_id = warpSkyfile.tess_id +JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + ON camRun.chip_id = chipRun.chip_id +JOIN rawExp + USING (exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND stackRun.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'sky' + AND distComponent.dist_id IS NULL + AND (staticskyRun.state = 'full' OR (distRun.clean AND staticskyRun.state = 'cleaned')) + AND (staticskyResult.fault = 0 AND staticskyResult.quality = 0) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_skycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_skycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_skycal.sql (revision 41104) @@ -0,0 +1,52 @@ +SELECT DISTINCT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + stackRun.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + skycalResult.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + skycalRun.state, + skycalRun.state AS data_state, + skycalResult.quality, + 1 AS no_magic, + 0 AS magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN skycalRun + ON stage_id = skycal_id +JOIN skycalResult + USING(skycal_id) +JOIN stackRun USING(stack_id) +JOIN stackInputSkyfile USING(stack_id) +JOIN warpSkyfile + ON stackInputSkyfile.warp_id = warpSkyfile.warp_id + AND stackRun.skycell_id = warpSkyfile.skycell_id + AND stackRun.tess_id = warpSkyfile.tess_id +JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING (exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND stackRun.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'skycal' + AND distComponent.dist_id IS NULL + AND (skycalRun.state = 'full' OR (distRun.clean AND skycalRun.state = 'cleaned')) + AND (skycalResult.fault = 0 AND skycalResult.quality = 0) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_stack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_stack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_stack.sql (revision 41104) @@ -0,0 +1,54 @@ +SELECT DISTINCT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + stackRun.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + stackSumSkyfile.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + stackRun.state, + -- data_state doesn't exist yet + -- stackSumSkyfile.data_state, + 'full' AS data_state, + stackSumSkyfile.quality, + 1 AS no_magic, + 0 AS magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN stackRun + ON stage_id = stack_id +JOIN stackSumSkyfile + USING(stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpSkyfile + ON stackInputSkyfile.warp_id = warpSkyfile.warp_id + AND stackRun.skycell_id = warpSkyfile.skycell_id + AND stackRun.tess_id = warpSkyfile.tess_id +JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + ON camRun.chip_id = chipRun.chip_id +JOIN rawExp + USING (exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND stackRun.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'stack' + AND distComponent.dist_id IS NULL + AND (stackRun.state = 'full' OR (distRun.clean AND stackRun.state = 'cleaned')) + AND (stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_warp.sql (revision 41104) @@ -0,0 +1,38 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + warpSkyfile.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + warpSkyfile.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + warpRun.state, + warpSkyfile.data_state, + warpSkyfile.quality, + distRun.no_magic, + warpSkyfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN warpRun ON stage_id = warp_id +JOIN warpSkyfile using(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun ON camRun.chip_id = chipRun.chip_id +JOIN rawExp using(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND warpSkyfile.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'warp' + AND distComponent.dist_id IS NULL + AND ((warpRun.magicked > 0) OR distRun.no_magic) + AND (warpRun.state = 'full' OR (distRun.clean AND warpRun.state = 'cleaned')) + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_warp_bg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_warp_bg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pending_warp_bg.sql (revision 41104) @@ -0,0 +1,39 @@ +SELECT + distRun.dist_id, + distRun.label, + distTarget.dist_group, + stage, + stage_id, + warpBackgroundSkyfile.skycell_id AS component, + exp_type, + clean, + rawExp.camera, + CONCAT_WS('.', outroot, CONVERT(distRun.dist_id, CHAR)) as outdir, + warpBackgroundSkyfile.path_base, + CAST(NULL AS CHAR(255)) as chip_path_base, + warpBackgroundRun.state, + warpBackgroundRun.state as data_state, + 0 as quality, + distRun.no_magic, + warpBackgroundSkyfile.magicked, + IFNULL(Label.priority, 10000) AS priority +FROM distRun +JOIN distTarget USING(target_id, stage, clean) +JOIN warpBackgroundRun ON stage_id = warp_bg_id +JOIN warpBackgroundSkyfile using(warp_bg_id) +JOIN warpRun using(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun ON camRun.chip_id = chipRun.chip_id +JOIN rawExp using(exp_id) +LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND warpBackgroundSkyfile.skycell_id = distComponent.component +LEFT JOIN Label ON distRun.label = Label.label +WHERE + distRun.state = 'new' + AND distRun.stage = 'warp_bg' + AND distComponent.dist_id IS NULL + AND ((warpBackgroundRun.magicked > 0) OR distRun.no_magic) + AND (warpBackgroundRun.state = 'full') + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingcleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingcleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingcleanup.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + dist_id, + stage, + stage_id, + label, + outdir +FROM distRun +WHERE state = 'goto_cleaned' + AND distRun.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingdest.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingdest.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingdest.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT DISTINCT + rcDestination.*, + count(fs_id) as pending_fs +FROM rcDestination +JOIN rcRun using(dest_id) +WHERE rcRun.state = 'new' + AND rcDestination.state = 'enabled' + AND status_uri IS NOT NULL + -- where hook %s +GROUP BY dest_id +HAVING count(fs_id) > 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingfileset.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingfileset.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_pendingfileset.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + dist_id, + target_id, + distRun.stage, + distRun.outdir as dist_dir, + stage_id, + distRun.data_group, + distTarget.filter, + rcDestination.name AS product_name, + rcDestination.dest_id, + rcDestination.dbname AS ds_dbname, + rcDestination.dbhost AS ds_dbhost +FROM rcDestination +JOIN rcInterest USING(dest_id) +JOIN distTarget USING(target_id) +JOIN distRun USING(target_id) +LEFT JOIN rcDSFileset USING(dest_id, dist_id) +WHERE distRun.state = 'full' + AND rcDestination.state = 'enabled' + AND distTarget.state = 'enabled' + AND rcInterest.state = 'enabled' + AND rcDSFileset.fs_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_processedcomponent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_processedcomponent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_processedcomponent.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + dist_id, + target_id, + stage, + stage_id, + component, + distComponent.outdir, + bytes, + md5sum, + name, + distComponent.state, + no_magic, + distComponent.fault +FROM distRun +JOIN distComponent USING(dist_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_queuercrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_queuercrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_queuercrun.sql (revision 41104) @@ -0,0 +1,20 @@ +-- create rcRun's for all destinations that have filesets that they are interested in available +INSERT INTO rcRun +SELECT -- rows in this select must match rcRun + 0, -- rc_id (auto-increment) + fs_id, + dest_id, + 'new', + NULL, -- status_fs + NULL, -- registered (database sets this to CURRENT_TIMESTAMP) + 0 -- fault + +FROM rcDestination +JOIN rcInterest USING(dest_id) +JOIN distTarget USING(target_id) +JOIN distRun USING(target_id, stage) +JOIN rcDSFileset USING(dest_id, dist_id) +LEFT JOIN rcRun using(fs_id, dest_id) +WHERE rcRun.rc_id IS NULL + AND rcDSFileset.state = 'full' + AND rcDSFileset.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_rerun_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_rerun_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_rerun_select.sql (revision 41104) @@ -0,0 +1,2 @@ +SELECT * FROM distRun +WHERE state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertcomponent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertcomponent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertcomponent.sql (revision 41104) @@ -0,0 +1,5 @@ +DELETE FROM distComponent +USING distComponent, distRun +WHERE distComponent.dist_id = distRun.dist_id + AND distRun.state = 'new' + AND distComponent.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertfileset.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertfileset.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertfileset.sql (revision 41104) @@ -0,0 +1,4 @@ +DELETE FROM rcDSFileset +USING rcDSFileset, distRun +WHERE distRun.dist_id = rcDSFileset.dist_id + AND rcDSFileset.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertrcrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertrcrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertrcrun.sql (revision 41104) @@ -0,0 +1,3 @@ +UPDATE rcRun SET fault = 0 +WHERE fault != 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_revertrun.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE distRun +SET distRun.fault = 0 +WHERE (distRun.state = 'new' OR distRun.state = 'goto_cleaned') + AND distRun.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_toadvance.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_toadvance.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_toadvance.sql (revision 41104) @@ -0,0 +1,340 @@ +SELECT DISTINCT + dist_id, + stage, + stage_id, + CONCAT_WS('.', outroot, CONVERT(dist_id, CHAR)) as outdir, + label, + clean +FROM + ( +-- raw stage not clean +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN rawImfile ON stage_id = rawImfile.exp_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND distComponent.component = rawImfile.class_id + WHERE + distRun.state = 'new' + AND distRun.clean = 0 + AND distRun.fault = 0 + AND distRun.stage = 'raw' + AND rawImfile.ignored = 0 + GROUP BY + distRun.dist_id, + rawImfile.exp_id + HAVING + COUNT(rawImfile.class_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- clean distribution of raw files (dbinfo only) only 1 component +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + WHERE + distRun.state = 'new' + AND distRun.clean + AND distRun.fault = 0 + AND distRun.stage = 'raw' + AND distComponent.component IS NOT NULL + AND distComponent.fault = 0 +UNION +-- chip stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN chipProcessedImfile ON stage_id = chipProcessedImfile.chip_id + LEFT JOIN distComponent + ON distComponent.dist_id = distRun.dist_id + AND distComponent.component = chipProcessedImfile.class_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'chip' + GROUP BY + dist_id, + chipProcessedImfile.chip_id + HAVING + COUNT(chipProcessedImfile.class_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- chip_bg stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN chipBackgroundImfile ON stage_id = chipBackgroundImfile.chip_bg_id + LEFT JOIN distComponent + ON distComponent.dist_id = distRun.dist_id + AND distComponent.component = chipBackgroundImfile.class_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'chip_bg' + GROUP BY + dist_id, + chipBackgroundImfile.chip_bg_id + HAVING + COUNT(chipBackgroundImfile.class_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- camera stage +SELECT distRun.dist_id, + stage, + stage_id, + outroot, + distRun.label, + clean + FROM distRun + JOIN camRun ON stage_id = cam_id + JOIN chipRun USING(chip_id) + LEFT JOIN distComponent USING(dist_id) + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distComponent.fault = 0 + AND distRun.stage = 'camera' +-- AND ((chipRun.magicked AND camRun.state = 'full') OR distRun.no_magic) +-- AND (camRun.state = 'full' OR (distRun.clean and camRun.state = 'cleaned')) +UNION +-- fake stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN fakeProcessedImfile ON stage_id = fakeProcessedImfile.fake_id + LEFT JOIN distComponent + ON distComponent.dist_id = distRun.dist_id + AND distComponent.component = fakeProcessedImfile.class_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'fake' + GROUP BY + dist_id, + fakeProcessedImfile.fake_id + HAVING + COUNT(fakeProcessedImfile.class_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- warp stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN warpSkyfile on stage_id = warp_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND distComponent.component = warpSkyfile.skycell_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'warp' +-- AND warpSkyfile.fault = 0 +-- AND warpSkyfile.quality = 0 + GROUP BY + distRun.dist_id, + warp_id + HAVING + COUNT(warpSkyfile.skycell_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- warp_bg stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN warpBackgroundSkyfile on stage_id = warp_bg_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND distComponent.component = warpBackgroundSkyfile.skycell_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'warp_bg' +-- AND warpSkyfile.fault = 0 +-- AND warpSkyfile.quality = 0 + GROUP BY + distRun.dist_id, + warp_bg_id + HAVING + COUNT(warpBackgroundSkyfile.skycell_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- diff stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + distRun.label, + clean + FROM distRun + JOIN diffSkyfile + ON stage_id = diffSkyfile.diff_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND distComponent.component = diffSkyfile.skycell_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'diff' +-- AND diffSkyfile.fault = 0 + GROUP BY + distRun.dist_id, + diff_id + HAVING + COUNT(diffSkyfile.skycell_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- stack stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN stackSumSkyfile on stage_id = stack_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'stack' + AND distComponent.component IS NOT NULL + AND distComponent.fault = 0 +UNION +-- staticsky stage +-- NOTE this assumes that there is only one component per staticskyRun +-- (one skycell) +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN staticskyResult on stage_id = sky_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'sky' + AND distComponent.component IS NOT NULL + AND distComponent.fault = 0 +UNION +-- skycal stage +-- NOTE this assumes that there is only one component per skycalRun +-- (one skycell) +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + label, + clean + FROM distRun + JOIN skycalResult on stage_id = skycal_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'skycal' + AND distComponent.component IS NOT NULL + AND distComponent.fault = 0 +UNION +-- SSdiff stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + distRun.label, + clean + FROM distRun + JOIN diffSkyfile + ON stage_id = diffSkyfile.diff_id + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND distComponent.component = diffSkyfile.skycell_id + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'SSdiff' +-- AND diffSkyfile.fault = 0 + GROUP BY + distRun.dist_id, + diff_id + HAVING + COUNT(diffSkyfile.skycell_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +UNION +-- ff stage +SELECT + distRun.dist_id, + stage, + stage_id, + outroot, + distRun.label, + clean + -- ,COUNT(warp_id) + -- ,COUNT(ff_id) + FROM distRun + JOIN fullForceRun ON stage_id = ff_id + JOIN fullForceResult USING(ff_id) + JOIN skycalRun USING(skycal_id) + JOIN stackRun USING(stack_id) + LEFT JOIN distComponent + ON distRun.dist_id = distComponent.dist_id + AND distComponent.component = CONCAT_WS('.', warp_id, stackRun.skycell_id) + WHERE + distRun.state = 'new' + AND distRun.fault = 0 + AND distRun.stage = 'ff' + GROUP BY + distRun.dist_id, + ff_id + HAVING + -- number of dist components is the number of warps plus 1 (for the summary component) + COUNT(fullForceResult.ff_id) = COUNT(distComponent.component) + AND SUM(distComponent.fault) = 0 +) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_updateinterest.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_updateinterest.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_updateinterest.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE rcInterest +JOIN distTarget USING(target_id) +JOIN rcDestination USING(dest_id) +SET rcInterest.state = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_updatercrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_updatercrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/disttool_updatercrun.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE rcRun +JOIN rcDestination USING(dest_id) +JOIN rcDSFileset using(dest_id, fs_id) +SET + -- set hook %s +WHERE + rcRun.state = 'new' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_createbundle.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_createbundle.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_createbundle.sql (revision 41104) @@ -0,0 +1,7 @@ +-- Select the information we want from the various tables +SELECT @COLLIST@ FROM rawExp + JOIN chipRun USING(exp_id) + JOIN camRun USING(chip_id) + JOIN camProcessedExp USING(cam_id) + LEFT OUTER JOIN fakeRun USING(cam_id) + LEFT OUTER JOIN warpRun USING(fake_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_definebyquery.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_definebyquery.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_definebyquery.sql (revision 41104) @@ -0,0 +1,10 @@ +-- query to identify unique data to bundle + +SELECT DISTINCT exp_id,chip_id,cam_id,warpRun.warp_id,invalid from rawExp + JOIN chipRun USING(exp_id) + JOIN camRun USING(chip_id) + JOIN camProcessedExp USING(cam_id) + LEFT OUTER JOIN fakeRun USING(cam_id) + LEFT OUTER JOIN warpRun USING(fake_id) + LEFT OUTER JOIN dqstatsContent USING (exp_id,chip_id,cam_id) +WHERE camRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_get_contents.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_get_contents.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_get_contents.sql (revision 41104) @@ -0,0 +1,2 @@ +SELECT dqstats_id,exp_id,chip_id,cam_id,warp_id from dqstatsContent + WHERE invalid = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_get_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_get_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/dqstatstool_get_run.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT DISTINCT dqstatsRun.*,rawExp.camera from dqstatsRun + JOIN dqstatsContent USING(dqstats_id) + JOIN rawExp USING(exp_id) + WHERE dqstatsRun.state = 'new' AND (dqstatsRun.fault = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_change_exp_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_change_exp_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_change_exp_state.sql (revision 41104) @@ -0,0 +1,15 @@ +-- change state of fakeRun from goto_cleaned to cleaned or goto_purged to purged +-- when all of the constituant imfiles are in the right state +-- arguments are new state (cleaned or purged) fake_id and new state again for +-- the fakeProcessedImfile sub query +UPDATE fakeRun + SET state = '%s' + WHERE + fakeRun.fake_id = %lld + AND (SELECT + COUNT(fake_id) + FROM fakeProcessedImfile + WHERE + fakeProcessedImfile.fake_id = fakeRun.fake_id + AND data_state != '%s' + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_change_imfile_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_change_imfile_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_change_imfile_data_state.sql (revision 41104) @@ -0,0 +1,14 @@ +-- handle changes in data_state. Used for the modes tocleanedimfile and topurgedimfile +-- args are new data_state, fake_id, class_id, and current expected state for fakeRun +UPDATE fakeProcessedImfile + SET + data_state = '%s' +WHERE + fake_id = %lld + AND class_id = '%s' + -- only update if fakeRun.state has the expected value + AND ( + SELECT state from fakeRun where fakeRun.fake_id = fakeProcessedImfile.fake_id + ) = '%s' + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_completely_processed_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_completely_processed_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_completely_processed_exp.sql (revision 41104) @@ -0,0 +1,44 @@ +-- the output of this query must match the format of fakeRun row +SELECT DISTINCT + fake_id, + cam_id, + state, + workdir, + label, + data_group, + dist_group, + reduction, + expgroup, + dvodb, + tess_id, + end_stage, + epoch, + note +FROM + (SELECT + fakeRun.*, + rawImfile.class_id as rawimfile_class_id, + fakeProcessedImfile.class_id + FROM fakeRun + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + JOIN rawImfile + ON rawImfile.exp_id = rawExp.exp_id + AND rawImfile.ignored = 0 + LEFT JOIN fakeProcessedImfile + ON fakeRun.fake_id = fakeProcessedImfile.fake_id + AND rawImfile.exp_id = fakeProcessedImfile.exp_id + AND rawImfile.class_id = fakeProcessedImfile.class_id + WHERE + fakeRun.state = 'new' + GROUP BY + fakeRun.fake_id, + rawExp.exp_id + HAVING + COUNT(rawImfile.class_id) = COUNT(fakeProcessedImfile.class_id) + AND SUM(fakeProcessedImfile.fault) = 0 + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + fakeRun.* +FROM fakeRun +WHERE + fakeRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_export_processed_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_export_processed_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_export_processed_imfile.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT fakeProcessedImfile.* FROM fakeProcessedImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_export_run.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT fakeRun.* FROM fakeRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_find_camrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_find_camrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_find_camrun.sql (revision 41104) @@ -0,0 +1,37 @@ +SELECT + * +FROM + (SELECT DISTINCT + camRun.cam_id, + chipRun.*, + rawExp.camera, + rawExp.telescope, + rawExp.dateobs, + rawExp.exp_tag, + rawExp.exp_type, + rawExp.filelevel, + rawExp.filter, + rawExp.airmass, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + rawExp.sat_pixel_frac, + rawExp.bg, + rawExp.bg_stdev, + rawExp.bg_mean_stdev, + rawExp.alt, + rawExp.az, + rawExp.ccd_temp, + rawExp.posang, + rawExp.object, + rawExp.sun_angle, + rawExp.comment + FROM camRun + JOIN chipRun + using(chip_id) + JOIN rawExp + using(exp_id) + WHERE + camRun.state = 'full' + AND chipRun.state = 'full' +) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_find_pendingexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_find_pendingexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_find_pendingexp.sql (revision 41104) @@ -0,0 +1,34 @@ +-- this query is used by both camtool -pendingexp & camtool -addprocessedexp it +-- does a little more work then is necessary for -addprocessed but it seems +-- "cleaner" to use the same query for both cases +SELECT * FROM + (SELECT + fakeRun.*, + rawExp.exp_tag, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel + FROM fakeRun + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN chipProcessedImfile + USING(chip_id) + JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id + LEFT JOIN fakeProcessedImfile + USING(fake_id) + LEFT JOIN fakeMask + ON fakeRun.label = fakeMask.label + WHERE + fakeRun.state = 'new' + AND camRun.state = 'full' + AND chipRun.state = 'full' + AND fakeMask.label IS NULL + AND fakeProcessedImfile.fake_id IS NOT NULL + GROUP BY + fakeRun.fake_id + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingcleanupimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingcleanupimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingcleanupimfile.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT + fakeProcessedImfile.*, + fakeRun.state, + fakeRun.workdir, + fakeRun.label, + fakeRun.reduction, + fakeRun.expgroup, + fakeRun.dvodb, + fakeRun.tess_id, + fakeRun.end_stage +FROM fakeRun +JOIN fakeProcessedImfile + USING(fake_id) +WHERE + ((fakeRun.state = 'goto_cleaned' AND fakeProcessedImfile.data_state = 'full') +OR + (fakeRun.state = 'goto_scrubbed' AND fakeProcessedImfile.data_state = 'full') +OR + (fakeRun.state = 'goto_purged' AND fakeProcessedImfile.data_state != 'purged')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + fakeRun.fake_id, + rawExp.camera, + fakeRun.state +FROM fakeRun +JOIN camRun +USING (cam_id) +JOIN chipRun +USING (chip_id) +JOIN rawExp +USING (exp_id) +WHERE + (fakeRun.state = 'goto_cleaned' OR fakeRun.state = 'goto_scrubbed' OR fakeRun.state = 'goto_purged') + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_pendingimfile.sql (revision 41104) @@ -0,0 +1,46 @@ +-- the subselect is so where criteria can be specified without knowing +-- which table the field came from +SELECT DISTINCT + fakeRun.*, + chipProcessedImfile.exp_id, + chipProcessedImfile.class_id, + chipProcessedImfile.uri, + chipProcessedImfile.bg, + chipProcessedImfile.bg_stdev, + chipProcessedImfile.bg_mean_stdev, + chipProcessedImfile.fringe_0, + chipProcessedImfile.fringe_1, + chipProcessedImfile.fringe_2, + chipProcessedImfile.ap_resid, + chipProcessedImfile.ap_resid_stdev, + chipProcessedImfile.n_stars, + chipProcessedImfile.n_extended, + chipProcessedImfile.n_cr, + chipProcessedImfile.path_base as chip_path_base, + camProcessedExp.path_base as cam_path_base, + rawExp.exp_name, + rawExp.exp_tag, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel +FROM fakeRun +JOIN camRun USING(cam_id) +JOIN camProcessedExp USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN chipProcessedImfile USING(chip_id,exp_id) +JOIN rawExp USING(exp_id) +-- ON chipProcessedImfile.exp_id = rawExp.exp_id +LEFT JOIN fakeProcessedImfile + ON fakeRun.fake_id = fakeProcessedImfile.fake_id + AND chipProcessedImfile.class_id = fakeProcessedImfile.class_id +LEFT JOIN fakeMask + ON fakeRun.label = fakeMask.label +WHERE + ((fakeRun.state = 'new' + AND fakeProcessedImfile.fake_id IS NULL + AND fakeProcessedImfile.class_id IS NULL + ) + OR (fakeRun.state = 'update' + AND fakeProcessedImfile.data_state = 'cleaned') + ) + AND fakeMask.label IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_processedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_processedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_processedimfile.sql (revision 41104) @@ -0,0 +1,18 @@ +SELECT DISTINCT + fakeProcessedImfile.*, + fakeRun.workdir, + rawExp.exp_tag, + rawExp.exp_name, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel +FROM fakeRun +JOIN fakeProcessedImfile + USING(fake_id) +JOIN rawExp + ON fakeProcessedImfile.exp_id = rawExp.exp_id +WHERE +-- bogus test; just here so there there is a 'WHERE' stmt to append +-- conditionals too + fakeProcessedImfile.exp_id is NOT NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_queue_cam_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_queue_cam_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_queue_cam_id.sql (revision 41104) @@ -0,0 +1,20 @@ +INSERT INTO fakeRun + SElECT + 0, -- fake_id + cam_id, -- cam_id + '%s', -- state + '%s', -- workdir + '%s', -- label + '%s', -- data_group + '%s', -- dist_group + '%s', -- reduction + '%s', -- expgroup + '%s', -- dvodb + '%s', -- tess_id + '%s', -- end_stage + NULL, -- epoch + '%s' -- note + FROM camRun + WHERE + camRun.state = 'full' + AND camRun.cam_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_revertprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_revertprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_revertprocessedimfile.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM fakeProcessedImfile +USING fakeProcessedImfile, fakeRun, camRun, chipRun, rawExp +WHERE + fakeProcessedImfile.fake_id = fakeRun.fake_id + AND fakeRun.cam_id = camRun.cam_id + AND camRun.chip_id = chipRun.chip_id + AND chipRun.exp_id = rawExp.exp_id + AND fakeProcessedImfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_unmasked.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_unmasked.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/faketool_unmasked.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + * +FROM + (SELECT + fakeRun.label, + count(fakeRun.fake_id) as n_fakeruns + FROM fakeRun + LEFT JOIN fakeMask + USING (label) + WHERE fakeMask.label IS NULL + GROUP BY fakeRun.label) as fakeUnmask Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_definebyquery.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_definebyquery.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_definebyquery.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + skycalRun.skycal_id, + skycalResult.path_base, + skycalRun.data_group, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter +FROM skycalRun + JOIN skycalResult USING(skycal_id) + JOIN stackRun USING(stack_id) + JOIN skycell USING(tess_id, skycell_id) + -- join hook %s +WHERE + skycalRun.state = 'full' AND skycalResult.quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_definebyquery_select_warps.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_definebyquery_select_warps.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_definebyquery_select_warps.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT warp_id +FROM warpRun join warpSkyfile using(warp_id, tess_id) + JOIN fakeRun using(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + JOIN skycell ON skycell.tess_id = warpSkyfile.tess_id AND skycell.skycell_id = warpSkyfile.skycell_id +WHERE warpRun.tess_id = '%s' AND warpSkyfile.skycell_id = '%s' AND rawExp.filter = '%s' + AND warpRun.state = 'full' AND warpSkyfile.quality = 0 and warpSkyfile.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_input.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_input.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_input.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + fullForceInput.* +FROM fullForceInput Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_result.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_result.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_result.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + fullForceResult.* +FROM fullForceResult Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_run.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + fullForceRun.* +FROM fullForceRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_summary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_summary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_export_summary.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + fullForceSummary.* +FROM fullForceSummary Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_result.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_result.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_result.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT + fullForceResult.*, + fullForceRun.state, + fullForceRun.label, + fullForceRun.data_group, + fullForceRun.skycal_id, + fullForceRun.workdir, + fullForceRun.sources_path_base, + CONCAT_WS('.', warp_id, skycell_id) AS dist_component, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + warpImfile.warp_skyfile_id, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera +FROM fullForceRun + JOIN fullForceResult USING(ff_id) + JOIN skycalRun using(skycal_id) + JOIN stackRun using(stack_id) + JOIN warpRun using(warp_id, tess_id) + JOIN warpSkyfile USING(warp_id, tess_id, skycell_id) + JOIN warpImfile USING(warp_id, skycell_id) + JOIN fakeRun using(fake_id) + JOIN camRun using(cam_id) + JOIN chipRun using(chip_id) + JOIN rawExp using(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_revert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_revert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_revert.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM fullForceResult +USING fullForceResult, fullForceRun, skycalRun, stackRun, skycell +WHERE fullForceRun.ff_id = fullForceResult.ff_id + AND fullForceResult.fault != 0 + AND fullForceRun.skycal_id = skycalRun.skycal_id + AND skycalRun.stack_id = stackRun.stack_id + AND stackRun.tess_id = skycell.tess_id AND stackRun.skycell_id = skycell.skycell_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_revertsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_revertsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_revertsummary.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM fullForceSummary +USING fullForceSummary, fullForceRun, skycalRun, stackRun, skycell +WHERE fullForceRun.ff_id = fullForceSummary.ff_id + AND fullForceSummary.fault != 0 + AND fullForceRun.skycal_id = skycalRun.skycal_id + AND skycalRun.stack_id = stackRun.stack_id + AND stackRun.tess_id = skycell.tess_id AND stackRun.skycell_id = skycell.skycell_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_summary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_summary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_summary.sql (revision 41104) @@ -0,0 +1,25 @@ +SELECT DISTINCT + fullForceSummary.*, + fullForceRun.state, + fullForceRun.label, + fullForceRun.data_group, + fullForceRun.skycal_id, + fullForceRun.workdir, + fullForceRun.sources_path_base, + 'summary' AS dist_component, + stackRun.stack_id, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + rawExp.camera +FROM fullForceRun + JOIN fullForceSummary USING(ff_id) + JOIN fullForceInput USING(ff_id) + JOIN skycalRun using(skycal_id) + JOIN stackRun using(stack_id) + JOIN warpRun using(warp_id, tess_id) + JOIN warpSkyfile USING(warp_id, tess_id, skycell_id) + JOIN fakeRun using(fake_id) + JOIN camRun using(cam_id) + JOIN chipRun using(chip_id) + JOIN rawExp using(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_summaryinputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_summaryinputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_summaryinputs.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT + fullForceResult.* +FROM fullForceRun + JOIN fullForceResult USING(ff_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_toadvance.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_toadvance.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_toadvance.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + fullForceRun.*, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + rawExp.camera +FROM fullForceRun + JOIN fullForceInput USING(ff_id) + JOIN skycalRun USING(skycal_id) + JOIN stackRun USING(stack_id) + JOIN warpRun USING(warp_id, tess_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + LEFT JOIN fullForceResult USING(ff_id, warp_id) + LEFT JOIN fullForceSummary USING(ff_id) +WHERE fullForceRun.state = 'new' AND fullForceSummary.ff_id IS NULL + -- WHERE hook %s + GROUP BY ff_id + HAVING COUNT(fullForceInput.warp_id) = COUNT(fullForceResult.warp_id) + AND SUM(fullForceResult.fault) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_todo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_todo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/fftool_todo.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + fullForceRun.*, + fullForceInput.warp_id, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + rawExp.camera, + warpSkyfile.path_base AS warp_path_base +FROM fullForceRun + JOIN fullForceInput USING(ff_id) + JOIN skycalRun USING(skycal_id) + JOIN stackRun USING(stack_id) + JOIN warpRun USING(warp_id, tess_id) + JOIN warpSkyfile USING(warp_id, tess_id, skycell_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + LEFT JOIN fullForceResult USING(ff_id, warp_id) +WHERE fullForceRun.state = 'new' + AND fullForceResult.ff_id IS NULL + AND warpSkyfile.data_state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_camerarundone.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_camerarundone.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_camerarundone.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + corr_id, + camRun.* +FROM flatcorrRun +JOIN flatcorrCamLink + USING(corr_id) +JOIN camRun + USING(cam_id) +LEFT JOIN flatcorrAddstarLink + using(corr_id, cam_id) +WHERE + flatcorrRun.state = 'new' + AND flatcorrCamLink.include = 1 + AND camRun.state = 'full' + AND flatcorrAddstarLink.corr_id is NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_chiprundone.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_chiprundone.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_chiprundone.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + corr_id, + chipRun.* +FROM flatcorrRun +JOIN flatcorrChipLink + USING(corr_id) +JOIN chipRun + USING(chip_id) +LEFT JOIN flatcorrCamLink + using(corr_id, chip_id) +WHERE + flatcorrRun.state = 'new' + AND flatcorrChipLink.include = 1 + AND chipRun.state = 'full' + AND flatcorrCamLink.corr_id is NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_completely_processed_chiprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_completely_processed_chiprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_completely_processed_chiprun.sql (revision 41104) @@ -0,0 +1,37 @@ +INSERT INTO chipRunDone +SELECT + chip_id, + exp_id +FROM (SELECT DISTINCT + chip_id, + exp_id, + state, + workdir, + workdir_state, + label, + reduction, + expgroup, + dvodb + FROM + (SELECT + chipRun.*, + rawImfile.class_id as rawimfile_class_id, + chipProcessedImfile.class_id + FROM chipRun + JOIN rawImfile + USING(exp_id) + LEFT JOIN chipProcessedImfile + ON chipRun.chip_id = chipProcessedImfile.chip_id + AND rawImfile.exp_id = chipProcessedImfile.exp_id + AND rawImfile.class_id = chipProcessedImfile.class_id + WHERE + chipRun.state = 'full' + GROUP BY + chipRun.chip_id, + chipRun.exp_id + HAVING + COUNT(rawImfile.class_id) = COUNT(chipProcessedImfile.class_id) + AND SUM(chipProcessedImfile.fault) = 0 + AND SUM(chipProcessedImfile.quality > 0) != COUNT(*) + ) as Foo + ) as Bar Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_create_tmp_chiprundone.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_create_tmp_chiprundone.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_create_tmp_chiprundone.sql (revision 41104) @@ -0,0 +1,3 @@ +CREATE TEMPORARY TABLE chipRunDone +(chip_id BIGINT, exp_id VARCHAR(64), PRIMARY KEY(chip_id)) ENGINE=MEMORY + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_dropcamera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_dropcamera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_dropcamera.sql (revision 41104) @@ -0,0 +1,5 @@ + +UPDATE flatcorrCamLink + SET include = 0 +WHERE corr_id = %lld + AND cam_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_dropchip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_dropchip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_dropchip.sql (revision 41104) @@ -0,0 +1,5 @@ + +UPDATE flatcorrChipLink + SET include = 0 +WHERE corr_id = %lld + AND chip_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_find_completed_floatcorruns.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_find_completed_floatcorruns.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_find_completed_floatcorruns.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + flatcorrRun.* +FROM flatcorrRun +JOIN flatcorrExp + USING(corr_id) +JOIN chipRunDone + USING(chip_id) +WHERE + flatcorrRun.state != "stop" +GROUP BY + corr_id, chip_id +HAVING + COUNT(flatcorrExp.chip_id) = COUNT(chipRunDone.chip_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_find_processedimfiles.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_find_processedimfiles.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_find_processedimfiles.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + corr_id, + chipProcessedImfile.* +FROM flatcorrRun +JOIN flatcorrExp + USING(corr_id) +JOIN chipRun + USING(chip_id) +JOIN chipProcessedImfile + USING(chip_id) +WHERE + flatcorrRun.state = 'run' + AND chipRun.state = 'new' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_inputimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_inputimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_inputimfile.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + chipProcessedImfile.*, + rawExp.filelevel , + rawExp.camera, + rawExp.telescope, + rawExp.filter +FROM chipProcessedImfile +JOIN rawExp +USING (exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_pending.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_pending.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_pending.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + * +FROM +flatcorrRun +JOIN flatcorrExp + USING(corr_id) +JOIN chipRun + USING(chip_id) +JOIN rawImfile + USING(exp_id) +LEFT JOIN chipProcessedImfile + ON chipRun.chip_id = chipProcessedImfile.chip_id + AND rawImfile.exp_id = chipProcessedImfile.exp_id + AND rawImfile.class_id = chipProcessedImfile.class_id +WHERE + flatcorrRun.state = 'run' + AND chipRun.state = 'full' +GROUP BY + chipRun.chip_id, + chipRun.exp_id +HAVING + COUNT(rawImfile.class_id) = COUNT(chipProcessedImfile.class_id) + AND SUM(chipProcessedImfile.fault > 0) = 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_pendingprocess.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_pendingprocess.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/flatcorr_pendingprocess.sql (revision 41104) @@ -0,0 +1,39 @@ +SELECT + corr_id, + det_type, + dvodb, + camera, + filter, + state, + make_corr, + workdir, + label, + reduction, + region, + cam_count, + add_count +FROM + (SELECT + flatcorrRun.*, + flatcorrCamLink.corr_id as cam_corr_id, + count(flatcorrCamLink.cam_id) as cam_count + FROM flatcorrRun + JOIN flatcorrCamLink + USING (corr_id) + WHERE flatcorrCamLink.include = 1 + GROUP BY + flatcorrCamLink.corr_id) AS t1 +LEFT JOIN + (SELECT + flatcorrAddstarLink.corr_id as add_corr_id, + count(flatcorrAddstarLink.add_id) as add_count + FROM flatcorrAddstarLink + JOIN addRun + USING (add_id) + WHERE addRun.state = 'full' + AND flatcorrAddstarLink.include = 1 + GROUP BY + flatcorrAddstarLink.corr_id) AS t2 +ON t1.cam_corr_id = t2.add_corr_id +WHERE cam_count = add_count +AND state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_WSdiff_check.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_WSdiff_check.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_WSdiff_check.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT warp_id,warp_state,W.skycell_id,stack_id,stack_state,S.seq_id + FROM +( SELECT warp_id,skycell_id,seq_id,warpRun.state AS warp_state + FROM warpSkyfile + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + JOIN lapExp USING(exp_id,chip_id) + JOIN lapRun USING(lap_id) + WHERE 1 + @WHERE@ +) AS W +LEFT JOIN +( SELECT stack_id,skycell_id,seq_id,stackRun.state AS stack_state + FROM stackRun + JOIN stackAssociationMap USING(stack_id) + JOIN lapRun ON (stackAssociationMap.sass_id = lapRun.final_sass_id) + ) AS S +ON ( (W.seq_id = S.seq_id) + AND (W.skycell_id = S.skycell_id) ) + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_definerun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_definerun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_definerun.sql (revision 41104) @@ -0,0 +1,35 @@ +SELECT DISTINCT + want.exp_id, + have.chip_id, + COALESCE(have.private,false) as private, + true as active, + false as pairwise + FROM + (SELECT exp_id FROM rawExp + WHERE rawExp.exp_type= 'OBJECT' AND + rawExp.dateobs >= '2009-04-01T00:00:00.000000' AND +-- Position restriction goes here. + @WHERE@ + ) AS want + LEFT JOIN + (SELECT + exp_id, + MAX(chip_id) AS chip_id, + chipRun.state, + lapExp.private + FROM lapExp JOIN chipRun USING(exp_id,chip_id) + JOIN lapRun USING(lap_id) + where chip_id IS NOT NULL + AND +-- This is the restriction to only draw from the current lapSequence + @CHIPWHERE@ +-- ( + -- (active = TRUE) OR + -- (chipRun.state = 'full') OR + -- (chipRun.state = 'new') + -- when we can do updates, put that here. +-- ) + GROUP BY exp_id +) AS have USING(exp_id) + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_definerunbyrelease.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_definerunbyrelease.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_definerunbyrelease.sql (revision 41104) @@ -0,0 +1,23 @@ +select + rel.exp_id AS exp_id, + rel.chip_id AS chip_id, + false AS private, + false AS active, + false AS pairwise +FROM + (select exp_id,chip_id FROM ippRelease + JOIN relExp USING(rel_id) + JOIN rawExp USING(exp_id) + WHERE + @RELWHERE@ + ) AS rel +LEFT JOIN + (select MIN(lap_id) AS lap_id, + exp_id,chip_id FROM lapRun + JOIN lapExp USING(lap_id) + WHERE + @LAPWHERE@ + GROUP BY exp_id,chip_id + ) AS lap +USING(exp_id) +WHERE lap_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_exposures.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_exposures.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_exposures.sql (revision 41104) @@ -0,0 +1,49 @@ +-- Join the diff information +-- select DISTINCT V3.*, +-- diffRun.diff_id,diffRun.state as diffRun_state, +-- coalesce(CONVERT(sum(diffSkyfile.quality != 0),SIGNED),0) AS diff_bad_quality, +-- coalesce(CONVERT(count(diffSkyfile.diff_id),SIGNED),0) AS diff_component_count +-- FROM +select V3.*,-1 as diff_id ,"full" as diffRun_state,0 as diff_bad_quality,0 AS diff_component_count FROM +-- Join the warp information +( select V2.*, + warpRun.warp_id,warpRun.state as warpRun_state, + coalesce(CONVERT(sum(warpSkyfile.quality != 0),SIGNED),0) AS warp_bad_quality, + coalesce(CONVERT(count(warpSkyfile.warp_id),SIGNED),0) AS warp_component_count, + warpRun.magicked + FROM +-- Join the camera and fake information +( select V1.*, + camRun.cam_id,camRun.state as camRun_state, + coalesce(CONVERT(sum(camProcessedExp.quality != 0),SIGNED),0) AS cam_bad_quality, + coalesce(CONVERT(count(camProcessedExp.cam_id),SIGNED),0) AS cam_component_count, + fakeRun.fake_id,fakeRun.state as fakeRun_state FROM +-- Get all the lap, exposure, chip stage information +( SELECT DISTINCT + lap_id,lapRun.tess_id,projection_cell,filter,lapRun.state as lapRun_state, lapRun.registered, lapRun.fault, lapRun.label, lapRun.dist_group, + lapExp.exp_id,lapExp.chip_id,lapExp.pair_id,private,pairwise,active,lapExp.data_state, + chipRun.state as chipRun_state, + coalesce(CONVERT(sum(chipProcessedImfile.quality != 0),SIGNED),0) AS chip_bad_quality, + coalesce(CONVERT(count(chipProcessedImfile.chip_id),SIGNED),0) AS chip_component_count + FROM lapRun JOIN lapExp USING(lap_id) + LEFT JOIN chipRun USING(chip_id) LEFT JOIN chipProcessedImfile USING(chip_id) +WHERE @WHERE@ + GROUP BY lap_id,exp_id + ) AS V1 +-- End lap/exposure/chip + LEFT JOIN camRun USING(chip_id) LEFT JOIN camProcessedExp USING(cam_id) + LEFT JOIN fakeRun USING(cam_id) + GROUP BY lap_id,exp_id + ) AS V2 +-- End camera/fake + LEFT JOIN warpRun USING(fake_id) LEFT JOIN warpSkyfile USING(warp_id) + GROUP BY lap_id,exp_id +) AS V3 +-- End warp +-- LEFT JOIN +-- (SELECT DISTINCT diff_id,warp1,warp2 FROM diffInputSkyfile) AS DI ON +-- (DI.warp1 = warp_id OR DI.warp2 = warp_id) +-- LEFT JOIN diffRun USING(diff_id) +-- LEFT JOIN diffSkyfile USING(diff_id) +GROUP BY lap_id,exp_id + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_filtersforgroup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_filtersforgroup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_filtersforgroup.sql (revision 41104) @@ -0,0 +1,2 @@ +SELECT DISTINCT filter +FROM lapGroup JOIN lapRun USING(seq_id, tess_id, projection_cell) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_inactiveexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_inactiveexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_inactiveexp.sql (revision 41104) @@ -0,0 +1,41 @@ +SELECT DISTINCT + D.*,diffRun.state AS diff_state,coalesce(CONVERT(sum(others.active),SIGNED),0) AS is_in_use FROM ( + SELECT DISTINCT + W.*,IFNULL(diff1.diff_id,diff2.diff_id) AS diff_id FROM ( + SELECT DISTINCT + lapRun.lap_id, lapRun.seq_id, lapRun.tess_id, lapRun.projection_cell, lapRun.filter, lapRun.state, lapRun.label, + lapRun.dist_group, lapRun.registered, lapRun.fault, lapRun.quick_sass_id, lapRun.final_sass_id, + lapExp.exp_id,lapExp.chip_id,lapExp.pair_id,private,pairwise,active,lapExp.data_state, + chipRun.state as chipRun_state, + CONVERT(sum(chipProcessedImfile.fault),SIGNED) as chip_faults, + CONVERT(sum(chipProcessedImfile.quality),SIGNED) as chip_quality, + camRun.cam_id, camRun.state as camRun_state, + CONVERT(sum(camProcessedExp.fault),SIGNED) AS cam_faults, + CONVERT(sum(camProcessedExp.quality),SIGNED) AS cam_quality, + fakeRun.fake_id, fakeRun.state as fakeRun_state, + CONVERT(sum(fakeProcessedImfile.fault),SIGNED) as fake_faults, + warpRun.warp_id, warpRun.state as warpRun_state, + CONVERT(sum(warpSkyfile.fault),SIGNED) as warp_faults, + CONVERT(sum(warpSkyfile.quality),SIGNED) as warp_quality, + warpRun.magicked + FROM lapRun JOIN lapExp USING(lap_id) + LEFT JOIN chipRun USING(chip_id) + LEFT JOIN chipProcessedImfile USING(chip_id) + LEFT JOIN camRun USING(chip_id) LEFT JOIN camProcessedExp USING(cam_id) + LEFT JOIN fakeRun USING(cam_id) LEFT JOIN fakeProcessedImfile USING(fake_id) + LEFT JOIN warpRun USING(fake_id) LEFT JOIN warpSkyfile USING(warp_id) + WHERE lapExp.active = FALSE + AND @WHERE@ + AND (warpSkyfile.quality IS NULL OR + (warpSkyfile.quality != 8007 -- known cases where quality != 0, but everything's fine. + AND warpSkyfile.quality != 3006 -- known cases where quality != 0, but everything's fine. + )) + GROUP BY lap_id,exp_id + ) AS W +-- This was unreasonably slow in testing, so that's why I'm using a subquery here. + LEFT JOIN diffInputSkyfile AS diff1 ON (W.warp_id = diff1.warp1) + LEFT JOIN diffInputSkyfile AS diff2 ON (W.warp_id = diff2.warp2) +) AS D +LEFT JOIN diffRun USING(diff_id) +LEFT JOIN lapExp AS others ON (D.chip_id = others.chip_id AND D.lap_id != others.lap_id) +GROUP BY lap_id,exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_listsequence.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_listsequence.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_listsequence.sql (revision 41104) @@ -0,0 +1,1 @@ +select * from lapSequence Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingchipexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingchipexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingchipexp.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT re_id,projection_cell,registered,reprocRun.state as reproc_state, tess_id, + exp_id,private,active, + chip_id,chipRun.state AS chip_state, + cam_id,camRun.state AS cam_state, + fake_id,fakeRun.state AS fake_state, + warp_id,warpRun.state AS warp_state + FROM reprocRun JOIN reprocExp USING(re_id) + LEFT JOIN chipRun USING(chip_id,tess_id) + LEFT JOIN camRun USING(chip_id,tess_id) + LEFT JOIN fakeRun USING(fake_id,tess_id) + LEFT JOIN warpRun USING(fake_id,tess_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingexp.sql (revision 41104) @@ -0,0 +1,12 @@ +select lapRun.lap_id, lapRun.seq_id, lapRun.tess_id, lapRun.projection_cell, lapRun.filter, + lapRun.state, lapRun.label, + lapRun.dist_group, lapRun.registered, lapRun.fault, lapRun.quick_sass_id, lapRun.final_sass_id, + exp_id, chip_id, pair_id, private, pairwise, active, lapExp.data_state, + dateobs, object, comment, + chipRun.state AS chip_state + FROM lapRun JOIN lapExp USING(lap_id) + JOIN rawExp USING(exp_id,filter) + LEFT JOIN chipRun USING(exp_id,chip_id) +WHERE active IS TRUE AND lapRun.fault = 0 +-- lap_id restriction here. +-- This probably needs to be sorted by dateobs. Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendinggroup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendinggroup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendinggroup.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT DISTINCT lapGroup.*, + lapRun.dist_group +FROM lapGroup JOIN lapRun USING(seq_id, tess_id, projection_cell) +WHERE lapGroup.state ='new' AND lapGroup.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_pendingrun.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * from lapRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_revertgroup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_revertgroup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_revertgroup.sql (revision 41104) @@ -0,0 +1,3 @@ +UPDATE lapGroup +SET fault = 0 +WHERE fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_revertsass.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_revertsass.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_revertsass.sql (revision 41104) @@ -0,0 +1,1 @@ +UPDATE lapRun SET final_sass_id = NULL,quick_sass_id = NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_stacks.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_stacks.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/laptool_stacks.sql (revision 41104) @@ -0,0 +1,103 @@ +SELECT + lapRun.lap_id,lapRun.seq_id,lapRun.tess_id,lapRun.projection_cell,lapRun.filter, + lapRun.state,lapRun.label,lapRun.dist_group,lapRun.registered,lapRun.fault, + skycell_id, + lapRun.quick_sass_id,quick_data_group,quick_stack_id,quick_state,quick_fault,quick_quality, + lapRun.final_sass_id,final_data_group,final_stack_id,final_state,final_fault,final_quality + FROM + lapRun LEFT JOIN + ( +SELECT DISTINCT + lap_id,tess_id,projection_cell,filter,skycell_id, + quick_sass_id,quick_data_group,quick_stack_id,quick_state,quick_fault,quick_quality, + final_sass_id,final_data_group,final_stack_id,final_state,final_fault,final_quality + FROM + (select lapRun.lap_id, + stackAssociation.sass_id AS quick_sass_id, + stackAssociation.data_group AS quick_data_group, + stackAssociation.projection_cell, + stackAssociation.tess_id, + stackAssociation.filter, + stackRun.stack_id AS quick_stack_id, + stackRun.skycell_id, + stackRun.state AS quick_state, + stackSumSkyfile.fault AS quick_fault, + stackSumSkyfile.quality AS quick_quality FROM + lapRun + LEFT JOIN stackAssociation ON (lapRun.quick_sass_id = stackAssociation.sass_id) + LEFT JOIN stackAssociationMap USING (sass_id) + LEFT JOIN stackRun USING(stack_id) + LEFT JOIN stackSumSkyfile USING(stack_id) + WHERE 1 @WHERE@ + AND sass_id = lapRun.quick_sass_id + ) AS quick LEFT JOIN + (select lapRun.lap_id, + stackAssociation.sass_id AS final_sass_id, + stackAssociation.data_group AS final_data_group, + stackAssociation.projection_cell, + stackAssociation.tess_id, + stackAssociation.filter, + stackRun.stack_id AS final_stack_id, + stackRun.skycell_id, + stackRun.state AS final_state, + stackSumSkyfile.fault AS final_fault, + stackSumSkyfile.quality AS final_quality FROM + lapRun + LEFT JOIN stackAssociation ON (lapRun.final_sass_id = stackAssociation.sass_id) + LEFT JOIN stackAssociationMap USING (sass_id) + LEFT JOIN stackRun USING(stack_id) + LEFT JOIN stackSumSkyfile USING(stack_id) + WHERE 1 @WHERE@ + AND sass_id = lapRun.final_sass_id + ) AS final USING(lap_id,projection_cell,tess_id,filter,skycell_id) + UNION +SELECT DISTINCT + lap_id,tess_id,projection_cell,filter,skycell_id, + quick_sass_id,quick_data_group,quick_stack_id,quick_state,quick_fault,quick_quality, + final_sass_id,final_data_group,final_stack_id,final_state,final_fault,final_quality + FROM + (select lapRun.lap_id, + stackAssociation.sass_id AS quick_sass_id, + stackAssociation.data_group AS quick_data_group, + stackAssociation.projection_cell, + stackAssociation.tess_id, + stackAssociation.filter, + stackRun.stack_id AS quick_stack_id, + stackRun.skycell_id, + stackRun.state AS quick_state, + stackSumSkyfile.fault AS quick_fault, + stackSumSkyfile.quality AS quick_quality FROM + lapRun + LEFT JOIN stackAssociation ON (lapRun.quick_sass_id = stackAssociation.sass_id) + LEFT JOIN stackAssociationMap USING (sass_id) + LEFT JOIN stackRun USING(stack_id) + LEFT JOIN stackSumSkyfile USING(stack_id) + WHERE 1 @WHERE@ + AND sass_id = lapRun.quick_sass_id + ) AS quick RIGHT JOIN + (select lapRun.lap_id, + stackAssociation.sass_id AS final_sass_id, + stackAssociation.data_group AS final_data_group, + stackAssociation.projection_cell, + stackAssociation.tess_id, + stackAssociation.filter, + stackRun.stack_id AS final_stack_id, + stackRun.skycell_id, + stackRun.state AS final_state, + stackSumSkyfile.fault AS final_fault, + stackSumSkyfile.quality AS final_quality FROM + lapRun + LEFT JOIN stackAssociation ON (lapRun.final_sass_id = stackAssociation.sass_id) + LEFT JOIN stackAssociationMap USING (sass_id) + LEFT JOIN stackRun USING(stack_id) + LEFT JOIN stackSumSkyfile USING(stack_id) + WHERE 1 @WHERE@ + AND sass_id = lapRun.final_sass_id + ) AS final USING(lap_id,projection_cell,tess_id,filter,skycell_id) + ) stacks ON (stacks.lap_id = lapRun.lap_id AND + stacks.projection_cell = lapRun.projection_cell AND + stacks.tess_id = lapRun.tess_id AND + stacks.filter = lapRun.filter AND + (lapRun.quick_sass_id IS NULL OR lapRun.quick_sass_id = stacks.quick_sass_id) AND + (lapRun.final_sass_id IS NULL OR lapRun.final_sass_id = stacks.final_sass_id)) +WHERE 1 @WHERE@ Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_change_file_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_change_file_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_change_file_data_state.sql (revision 41104) @@ -0,0 +1,8 @@ +-- handle changes in data_state. +UPDATE magicDSFile + JOIN magicDSRun USING(magic_ds_id) +SET + magicDSFile.data_state = '%s' +WHERE + magic_ds_id = %lld + AND component = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_change_run_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_change_run_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_change_run_state.sql (revision 41104) @@ -0,0 +1,15 @@ +-- change state of magicDSRun from goto_cleaned to cleaned or goto_purged to purged +-- when all of the constituant imfiles are in the right state +-- arguments are new state (cleaned or purged) chip_id and new state again for +-- the chipProcessedImfile sub query +UPDATE magicDSRun + SET magicDSRun.state = '%s' +WHERE + magicDSRun.magic_ds_id = %lld + AND (SELECT + COUNT(magic_ds_id) + FROM magicDSFile + WHERE + magicDSFile.magic_ds_id = magicDSRun.magic_ds_id + AND data_state != '%s' + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_clearstatefaults.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_clearstatefaults.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_clearstatefaults.sql (revision 41104) @@ -0,0 +1,1 @@ +UPDATE magicDSRun SET state = '%s', fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_completed_runs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_completed_runs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_completed_runs.sql (revision 41104) @@ -0,0 +1,117 @@ +SELECT DISTINCT + magic_ds_id, + magicked, + re_place, + label +FROM + ( +-- raw stage +SELECT + magicDSRun.*, + rawExp.magicked + FROM magicDSRun + JOIN rawImfile ON stage_id = rawImfile.exp_id + JOIN rawExp using(exp_id) + LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = rawImfile.class_id + WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'raw' + GROUP BY + magicDSRun.magic_ds_id, + stage_id + HAVING + COUNT(rawImfile.class_id) = COUNT(magicDSFile.component) + AND SUM(magicDSFile.fault) = 0 +UNION +-- chip stage +SELECT + magicDSRun.*, + chipRun.magicked + FROM magicDSRun + JOIN chipRun ON stage_id = chip_id + JOIN chipProcessedImfile USING(chip_id) + LEFT JOIN magicDSFile + ON magicDSFile.magic_ds_id = magicDSRun.magic_ds_id + AND magicDSFile.component = chipProcessedImfile.class_id + WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'chip' + AND chipProcessedImfile.quality = 0 + GROUP BY + magic_ds_id, + stage_id + HAVING + COUNT(chipProcessedImfile.class_id) = COUNT(magicDSFile.component) + AND SUM(magicDSFile.fault) = 0 +UNION +-- camera stage +SELECT + magicDSRun.*, + camRun.magicked + FROM magicDSRun + JOIN camRun ON stage_id = camRun.cam_id + JOIN camProcessedExp ON camRun.cam_id = camProcessedExp.cam_id + LEFT JOIN magicDSFile + ON magicDSFile.magic_ds_id = magicDSRun.magic_ds_id + AND magicDSFile.component IS NOT NULL + WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'camera' + AND camProcessedExp.quality = 0 + AND magicDSFile.fault = 0 + GROUP BY + magic_ds_id, + stage_id +UNION +-- warp stage +SELECT + magicDSRun.*, + warpRun.magicked + FROM magicDSRun + JOIN warpRun on stage_id = warp_id + JOIN warpSkyfile USING(warp_id) + LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = warpSkyfile.skycell_id + WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'warp' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + GROUP BY + magicDSRun.magic_ds_id, + stage_id + HAVING + COUNT(warpSkyfile.skycell_id) = COUNT(magicDSFile.component) + AND SUM(magicDSFile.fault) = 0 +UNION +-- diff stage +SELECT DISTINCT + magicDSRun.*, + diffRun.magicked + FROM magicDSRun + JOIN magicRun USING (magic_id) + JOIN magicInputSkyfile USING(magic_id) + JOIN diffRun + ON magicRun.diff_id = diffRun.diff_id + JOIN diffSkyfile + ON diffRun.diff_id = diffSkyfile.diff_id + AND magicInputSkyfile.node = diffSkyfile.skycell_id + LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = diffSkyfile.skycell_id + WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'diff' + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 + GROUP BY + magicDSRun.magic_ds_id, + magicRun.magic_id + HAVING + COUNT(magicInputSkyfile.node) = COUNT(magicDSFile.component) + AND SUM(magicDSFile.fault) = 0 + + ) as magicDSRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_completedrevert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_completedrevert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_completedrevert.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + magic_ds_id, + state +FROM magicDSRun +LEFT JOIN magicDSFile USING(magic_ds_id) +WHERE + magicDSRun.state = 'goto_censored' OR magicDSRun.state = 'goto_restored' +GROUP BY magic_ds_id +HAVING COUNT(component) = 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_camera.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + exp_id, + 'camera' AS stage, + camRun.cam_id AS stage_id, + camRun.data_group, + camRun.cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id, exp_id) + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'camera' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n 1 " if rerun else "\n 0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND camRun.magicked = 0 + AND chipRun.magicked + AND chipRun.state = 'full' + AND camRun.state = 'full' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_chip.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + exp_id, + 'chip' AS stage, + chip_id AS stage_id, + chipRun.data_group, + camRun.cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id, exp_id) + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'chip' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND chipRun.magicked = 0 + AND chipRun.state = 'full' + AND camRun.state = 'full' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_chip_bg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_chip_bg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_chip_bg.sql (revision 41104) @@ -0,0 +1,33 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + exp_id, + 'chip_bg' AS stage, + chip_bg_id AS stage_id, + chipBackgroundRun.data_group, + camRun.cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun +JOIN magicMask USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffInputSkyfile USING(diff_id) +JOIN warpRun + ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id, exp_id) +JOIN chipBackgroundRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN magicDSRun + ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'chip_bg' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND chipBackgroundRun.magicked = 0 + AND chipBackgroundRun.state = 'full' + AND camRun.state = 'full' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_diff.sql (revision 41104) @@ -0,0 +1,71 @@ +SELECT * +FROM ( +SELECT + magicRun.magic_id, + exp_id, + 'diff' AS stage, + diff_id AS stage_id, + diffRun.data_group, + 0 AS cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + LEFT JOIN magicDSRun ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'diff' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND diffRun.bothways = 0 + AND diffRun.magicked = 0 + AND diffRun.state = 'full' + -- WHERE hook %s +UNION +SELECT + forwardRun.magic_id, + forwardRun.exp_id, + 'diff' AS stage, + forwardRun.diff_id AS stage_id, + forwardRun.data_group, + 0 AS cam_id, + forwardRun.label, + forwardRun.workdir, + inverseRun.magic_id AS inv_magic_id, + inverseRun.exp_id AS inv_exp_id +FROM ( + SELECT + magicRun.magic_id, + exp_id, + diff_id, + diffRun.data_group, + magicRun.label, + magicRun.workdir + FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + LEFT JOIN magicDSRun ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'diff' + WHERE magicRun.state = 'full' + AND magicRun.inverse = 0 + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND diffRun.state = 'full' + AND diffRun.bothways + AND diffRun.magicked = 0 + -- WHERE hook %s +) AS forwardRun +JOIN ( + SELECT + magic_id, + exp_id, + diff_id + FROM magicRun + JOIN magicMask USING(magic_id) + WHERE magicRun.state = 'full' + AND magicRun.inverse = 1 +) AS inverseRun USING(diff_id) + +) AS Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_raw.sql (revision 41104) @@ -0,0 +1,29 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + exp_id, + 'raw' AS stage, + exp_id AS stage_id, + chipRun.data_group, + camRun.cam_id AS cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id, exp_id) + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage_id = exp_id + AND magicDSRun.stage = 'raw' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND rawExp.magicked = 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_warp.sql (revision 41104) @@ -0,0 +1,29 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + exp_id, + 'warp' AS stage, + warp_id AS stage_id, + warpRun.data_group, + 0 AS cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id, exp_id) + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'warp' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND warpRun.magicked = 0 + AND warpRun.state = 'full' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_warp_bg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_warp_bg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definebyquery_warp_bg.sql (revision 41104) @@ -0,0 +1,32 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + exp_id, + 'warp_bg' AS stage, + warp_bg_id AS stage_id, + warpBackgroundRun.data_group, + 0 AS cam_id, + magicRun.label, + magicRun.workdir, + CAST(NULL AS SIGNED) AS inv_magic_id, + CAST(NULL AS SIGNED) AS inv_exp_id +FROM magicRun +JOIN magicMask USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffInputSkyfile USING(diff_id) +JOIN warpRun + ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) +JOIN warpBackgroundRun USING(warp_bg_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id, exp_id) +JOIN rawExp USING(exp_id) +LEFT JOIN magicDSRun + ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'warp_bg' +WHERE magicRun.state = 'full' + AND ( -- rerun HOOK magicdstool sends "\n1 " if rerun else "\n0 " %s + OR magicDSRun.magic_ds_id IS NULL) + AND warpBackgroundRun.magicked = 0 + AND warpBackgroundRun.state = 'full' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_camera.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + chipRun.exp_id, + 'camera' AS stage, + camRun.cam_id AS stage_id, + camRun.data_group, + camRun.cam_id, + camRun.label, + CAST(NULL AS SIGNED) AS inv_magic_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun AS oldWarpRun + ON (!magicRun.inverse AND diffInputSkyfile.warp1 = oldWarpRun.warp_id) + OR ( magicRun.inverse AND diffInputSkyfile.warp2 = oldWarpRun.warp_id) + -- This ends the usual JOIN down to the raw level + JOIN rawExp + ON rawExp.exp_id = magicRun.exp_id + JOIN chipRun + ON chipRun.exp_id = rawExp.exp_id + JOIN camRun + ON camRun.chip_id = chipRun.chip_id + LEFT JOIN magicDSRun + ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'camera' + AND magicDSRun.label = '%s' +WHERE magicRun.state = 'full' + -- HOOK: NEWLINE HERE TO ACTIVATE %s AND magicDSRun.magic_ds_id IS NULL + AND camRun.magicked = 0 + AND camRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_chip.sql (revision 41104) @@ -0,0 +1,35 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + chipRun.exp_id, + 'chip' AS stage, + chipRun.chip_id AS stage_id, + chipRun.data_group, + camRun.cam_id, + chipRun.label, + CAST(NULL AS SIGNED) AS inv_magic_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun + ON (!magicRun.inverse AND warp1 = warp_id) + OR ( magicRun.inverse AND warp2 = warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun AS oldCamRun USING(cam_id) + -- This ends the usual JOIN down to the raw level + JOIN chipRun + ON chipRun.exp_id = magicRun.exp_id + AND chipRun.chip_id != oldCamRun.chip_id -- Not allowed to copy yourself + JOIN camRun ON + camRun.chip_id = chipRun.chip_id + JOIN rawExp + ON rawExp.exp_id = chipRun.exp_id + LEFT JOIN magicDSRun + ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'chip' + AND magicDSRun.label = '%s' +WHERE magicRun.state = 'full' + -- HOOK: NEWLINE HERE TO ACTIVATE %s AND magicDSRun.magic_ds_id IS NULL + AND chipRun.magicked = 0 + AND chipRun.state = 'full' + AND camRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_definecopy_warp.sql (revision 41104) @@ -0,0 +1,36 @@ +SELECT DISTINCT -- DISTINCT because we are going though diffInputSkyfile + magicRun.magic_id, + chipRun.exp_id, + 'warp' AS stage, + warpRun.warp_id AS stage_id, + warpRun.data_group, + camRun.cam_id, + warpRun.label, + CAST(NULL AS SIGNED) AS inv_magic_id +FROM magicRun + JOIN magicMask USING(magic_id) + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun AS oldWarpRun + ON (!magicRun.inverse AND diffInputSkyfile.warp1 = oldWarpRun.warp_id) + OR ( magicRun.inverse AND diffInputSkyfile.warp2 = oldWarpRun.warp_id) + -- This ends the usual JOIN down to the raw level + JOIN rawExp + ON rawExp.exp_id = magicRun.exp_id + JOIN chipRun + ON chipRun.exp_id = rawExp.exp_id + JOIN camRun + ON camRun.chip_id = chipRun.chip_id + JOIN fakeRun + ON fakeRun.cam_id = camRun.cam_id + JOIN warpRun + ON warpRun.fake_id = fakeRun.fake_id + AND warpRun.warp_id != oldWarpRun.warp_id -- Not allowed to copy yourself + LEFT JOIN magicDSRun + ON magicRun.magic_id = magicDSRun.magic_id + AND magicDSRun.stage = 'warp' + AND magicDSRun.label = '%s' +WHERE magicRun.state = 'full' + -- HOOK: NEWLINE HERE TO ACTIVATE %s AND magicDSRun.magic_ds_id IS NULL + AND warpRun.magicked = 0 + AND warpRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_destreakedfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_destreakedfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_destreakedfile.sql (revision 41104) @@ -0,0 +1,17 @@ +SELECT + magicDSFile.*, + CONCAT_WS('.', CONCAT(outroot, '/', exp_id), 'mds', magic_ds_id, stage_id, component) AS path_base, + magicDSRun.magic_id, + magicDSRun.inv_magic_id, + magicDSRun.state, + magicDSRun.stage, + magicDSRun.stage_id, + magicDSRun.cam_id, + magicDSRun.label, + magicDSRun.data_group, + magicDSRun.outroot, + magicDSRun.re_place, + magicDSRun.fault as run_fault, + magicDSRun.note +FROM + magicDSRun JOIN magicDSFile USING(magic_ds_id) JOIN magicRun USING(magic_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_getrunids.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_getrunids.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_getrunids.sql (revision 41104) @@ -0,0 +1,23 @@ +-- find the inputs for a new magicDSRun from an existing magicRun +SELECT DISTINCT + magic_id, + magicRun.exp_id, + chip_id, + cam_id, + warp_id, + diffSkyfile.diff_id +FROM magicRun +JOIN magicInputSkyfile USING(magic_id) +JOIN diffSkyfile + ON magicRun.diff_id = diffSkyfile.diff_id + AND magicInputSkyfile.node = diffSkyfile.skycell_id +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffSkyfile.diff_id + AND diffInputSkyfile.skycell_id = diffSkyfile.skycell_id +JOIN warpRun + ON (diffInputSkyfile.warp1 = warp_id AND !magicRun.inverse) + OR (diffInputSkyfile.warp2 = warp_id AND magicRun.inverse) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +WHERE magic_id = %ld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_getskycells.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_getskycells.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_getskycells.sql (revision 41104) @@ -0,0 +1,64 @@ +-- Input warps +SELECT DISTINCT + diffSkyfile.diff_id, + diffRun.tess_id, + diffSkyfile.skycell_id, + diffSkyfile.path_base, + diffSkyfile.data_state, + diffSkyfile.quality +FROM magicDSRun +JOIN magicRun USING(magic_id) +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffSkyfile + ON magicRun.diff_id = diffSkyfile.diff_id + AND magicInputSkyfile.node = diffSkyfile.skycell_id +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffSkyfile.diff_id + AND diffInputSkyfile.skycell_id = diffSkyfile.skycell_id + -- Want input warps only + AND diffInputSkyfile.warp1 IS NOT NULL + AND magicRun.inverse = 0 +JOIN warpSkyCellMap + ON warpSkyCellMap.warp_id = diffInputSkyfile.warp1 + AND warpSkyCellMap.skycell_id = diffInputSkyfile.skycell_id +JOIN warpSkyfile + ON warpSkyfile.warp_id = warpSkyCellMap.warp_id + AND warpSkyfile.skycell_id = warpSkyCellMap.skycell_id + AND warpSkyfile.quality = 0 +WHERE + diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 + -- WHERE hook %s +UNION +-- Reference warps +SELECT DISTINCT + diffSkyfile.diff_id, + diffRun.tess_id, + diffSkyfile.skycell_id, + diffSkyfile.path_base, + diffSkyfile.data_state, + diffSkyfile.quality +FROM magicDSRun +JOIN magicRun USING(magic_id) +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffSkyfile + ON magicRun.diff_id = diffSkyfile.diff_id + AND magicInputSkyfile.node = diffSkyfile.skycell_id +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffSkyfile.diff_id + AND diffInputSkyfile.skycell_id = diffSkyfile.skycell_id + AND diffInputSkyfile.warp2 IS NOT NULL + AND magicRun.inverse = 1 +JOIN warpSkyCellMap + ON warpSkyCellMap.warp_id = diffInputSkyfile.warp2 + AND warpSkyCellMap.skycell_id = diffInputSkyfile.skycell_id +JOIN warpSkyfile + ON warpSkyfile.warp_id = warpSkyCellMap.warp_id + AND warpSkyfile.skycell_id = warpSkyCellMap.skycell_id + AND warpSkyfile.quality = 0 +WHERE + diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 + -- WHERE hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_listrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_listrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_listrun.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT + magicDSRun.* +FROM + magicDSRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_revertdestreakedfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_revertdestreakedfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_revertdestreakedfile.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM magicDSFile USING magicDSFile, magicDSRun +WHERE (magicDSRun.magic_ds_id = magicDSFile.magic_ds_id) + AND ((magicDSRun.state = 'new' AND magicDSFile.fault != 0) + OR (magicDSRun.state = 'goto_restored' + OR magicDSRun.state = 'goto_censored' + ) + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_revertupdated.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_revertupdated.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_revertupdated.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE magicDSRun + JOIN magicDSFile using(magic_ds_id) +SET magicDSFile.fault = 0 +WHERE + magicDSRun.state = 'update' + AND magicDSFile.data_state = 'update' + AND magicDSFile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_setfiletoupdate.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_setfiletoupdate.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_setfiletoupdate.sql (revision 41104) @@ -0,0 +1,9 @@ +UPDATE magicDSRun + JOIN magicDSFile USING(magic_ds_id) +SET magicDSRun.state = 'update', + magicDSFile.data_state = 'update', + magicDSFile.fault = 0 + -- set hook %s +WHERE magic_ds_id = %lld + AND (magicDSRun.state = 'cleaned' OR magicDSRun.state = 'update') + AND (magicDSFile.data_state = 'cleaned' OR magicDSFile.data_state ='update') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_tocleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_tocleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_tocleanup.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + magic_ds_id, + stage, + magicDSRun.state, + magicDSRun.outroot, + camera +FROM magicDSRun + JOIN magicRun USING(magic_id) + JOIN rawExp USING(exp_id) +WHERE magicDSRun.state = 'goto_cleaned' +-- XXX: need to add fault to magicDSRun +-- XXX: the database has been updated, but fault isn't yet used +-- AND magicDSRun.fault = 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_camera.sql (revision 41104) @@ -0,0 +1,46 @@ +SELECT DISTINCT + magicDSRun.magic_ds_id, + magicDSRun.magic_id, + magicDSRun.state, + chipRun.exp_id, + camRun.magicked, + magicDSRun.label, + camera, + magicMask.uri AS streaks_uri, + magicMask.path_base AS streaks_path_base, + CAST(NULL AS CHAR(255)) AS inv_streaks_uri, + CAST(NULL AS CHAR(255)) AS inv_streaks_path_base, + stage, + stage_id, + 'exposure' AS component, + 0 AS mismatched_tess, + CAST(NULL AS CHAR(255)) AS uri, + camProcessedExp.path_base, + magicRun.inverse, + camProcessedExp.path_base AS cam_path_base, + camRun.reduction AS cam_reduction, + outroot, + recoveryroot, + re_place, + remove, + IFNULL(Label.priority, 10000) AS priority +FROM magicDSRun +JOIN magicMask USING (magic_id) +JOIN magicRun USING(magic_id) +JOIN camRun ON magicDSRun.stage_id = camRun.cam_id +JOIN camProcessedExp ON camRun.cam_id = camProcessedExp.cam_id +JOIN chipRun USING(chip_id) +JOIN rawExp ON chipRun.exp_id = rawExp.exp_id +LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id +LEFT JOIN Label ON magicDSRun.label = Label.label +WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'camera' + AND camRun.state = 'full' + AND ((chipRun.state = 'full' AND chipRun.magicked > 0) + OR (chipRun.state = 'cleaned' AND chipRun.magicked < 0)) + AND camProcessedExp.fault = 0 + AND camProcessedExp.quality = 0 + AND magicDSFile.component IS NULL + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_chip.sql (revision 41104) @@ -0,0 +1,48 @@ +SELECT DISTINCT + magicDSRun.magic_ds_id, + magicDSRun.magic_id, + magicDSRun.state, + chipRun.exp_id, + chipRun.magicked, + magicDSRun.label, + camera, + magicMask.uri AS streaks_uri, + magicMask.path_base AS streaks_path_base, + CAST(NULL AS CHAR(255)) AS inv_streaks_uri, + CAST(NULL AS CHAR(255)) AS inv_streaks_path_base, + stage, + stage_id, + class_id AS component, + 0 AS mismatched_tess, + chipProcessedImfile.uri, + chipProcessedImfile.path_base, + magicRun.inverse, + camProcessedExp.path_base AS cam_path_base, + camRun.reduction AS cam_reduction, + outroot, + recoveryroot, + re_place, + remove, + IFNULL(Label.priority, 10000) AS priority +FROM magicDSRun +JOIN magicMask USING (magic_id) +JOIN magicRun USING(magic_id) +JOIN camRun USING(cam_id) +JOIN camProcessedExp USING(cam_id) +JOIN chipRun ON chipRun.chip_id = stage_id +JOIN chipProcessedImfile ON chipRun.chip_id = chipProcessedImfile.chip_id +JOIN rawExp ON chipRun.exp_id = rawExp.exp_id +LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = chipProcessedImfile.class_id +LEFT JOIN Label + ON magicDSRun.label = Label.label +WHERE + ((magicDSRun.state = 'new' AND magicDSFile.component IS NULL) + OR (magicDSRun.state = 'update' AND magicDSFile.data_state = 'update' + AND magicDSFile.fault = 0)) + AND magicDSRun.stage = 'chip' + AND (chipRun.state = 'full' OR (chipRun.state = 'update' and chipProcessedImfile.data_state = 'full')) + AND chipProcessedImfile.fault = 0 + AND chipProcessedImfile.quality = 0 + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_diff.sql (revision 41104) @@ -0,0 +1,100 @@ +SELECT * FROM ( +SELECT DISTINCT + magicDSRun.magic_ds_id, + magicRun.magic_id, + magicDSRun.state, + magicRun.exp_id, + magicDSRun.label, + rawExp.camera, + magicMask.uri AS streaks_uri, + magicMask.path_base AS streaks_path_base, + CAST(NULL AS CHAR(255)) AS inv_streaks_uri, + CAST(NULL AS CHAR(255)) AS inv_streaks_path_base, + stage, + magicRun.diff_id AS stage_id, + diffRun.magicked, + diffSkyfile.skycell_id AS component, + 0 AS mismatched_tess, + CAST(NULL AS CHAR(255)) AS uri, + diffSkyfile.path_base, + magicRun.inverse, + CAST(NULL AS CHAR(255)) AS cam_path_base, + CAST(NULL AS CHAR(255)) AS cam_reduction, + outroot, + recoveryroot, + re_place, + remove, + IFNULL(Label.priority, 10000) AS priority +FROM rawExp +JOIN magicRun USING (exp_id) +JOIN magicMask USING (magic_id) +JOIN magicDSRun USING(magic_id) +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffSkyfile + ON magicRun.diff_id = diffSkyfile.diff_id + AND magicInputSkyfile.node = diffSkyfile.skycell_id +LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = diffSkyfile.skycell_id +LEFT JOIN Label ON magicDSRun.label = Label.label +WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'diff' + AND diffRun.bothways = 0 + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 + AND magicDSFile.component IS NULL + AND (Label.active OR Label.active IS NULL) +-- bothways diffSkyfiles +UNION +SELECT DISTINCT + magicDSRun.magic_ds_id, + magicRun.magic_id, + magicDSRun.state, + magicRun.exp_id, + magicDSRun.label, + rawExp.camera, + magicMask.uri AS streaks_uri, + magicMask.path_base AS streaks_path_base, + (SELECT uri from magicMask where magic_id = inv_magic_id) AS inv_streaks_uri, + (SELECT path_base from magicMask where magic_id = inv_magic_id) AS inv_streaks_path_base, + stage, + magicRun.diff_id AS stage_id, + diffRun.magicked, + diffSkyfile.skycell_id AS component, + 0 AS mismatched_tess, + CAST(NULL AS CHAR(255)) AS uri, + diffSkyfile.path_base, + magicRun.inverse, + CAST(NULL AS CHAR(255)) AS cam_path_base, + CAST(NULL AS CHAR(255)) AS cam_reduction, + outroot, + recoveryroot, + re_place, + remove, + IFNULL(Label.priority, 10000) AS priority +FROM rawExp +JOIN magicRun USING (exp_id) +JOIN magicMask USING (magic_id) +JOIN magicDSRun USING(magic_id) +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffSkyfile + ON magicRun.diff_id = diffSkyfile.diff_id + AND magicInputSkyfile.node = diffSkyfile.skycell_id +LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = diffSkyfile.skycell_id +LEFT JOIN Label ON magicDSRun.label = Label.label +WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'diff' + AND diffRun.bothways + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 + AND magicDSFile.component IS NULL + AND (Label.active OR Label.active IS NULL) +) AS magicDSRun +-- we need the following so this query is compatible with the other stages +WHERE magic_ds_id IS NOT NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_raw.sql (revision 41104) @@ -0,0 +1,44 @@ +SELECT DISTINCT + magicDSRun.magic_ds_id, + magicRun.magic_id, + magicDSRun.state, + magicRun.exp_id, + magicDSRun.label, + rawExp.camera, + magicMask.uri as streaks_uri, + magicMask.path_base as streaks_path_base, + CAST(NULL AS CHAR(255)) AS inv_streaks_uri, + CAST(NULL AS CHAR(255)) AS inv_streaks_path_base, + stage, + stage_id, + rawExp.magicked, + class_id as component, + 0 AS mismatched_tess, + rawImfile.uri AS uri, + -- XXX: replace this with rawImfile.path_base once it exists + TRIM(TRAILING '.fits' FROM rawImfile.uri) AS path_base, + magicRun.inverse, + camProcessedExp.path_base as cam_path_base, + camRun.reduction AS cam_reduction, + outroot, + recoveryroot, + re_place, + remove, + IFNULL(Label.priority, 10000) AS priority +FROM magicDSRun +JOIN magicMask USING (magic_id) +JOIN magicRun USING (magic_id) +JOIN camRun USING(cam_id) +JOIN camProcessedExp USING(cam_id) +JOIN rawExp ON magicRun.exp_id = rawExp.exp_id +JOIN rawImfile ON rawExp.exp_id = rawImfile.exp_id +LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = rawImfile.class_id +LEFT JOIN Label ON magicDSRun.label = Label.label +WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'raw' + AND magicDSFile.component IS NULL + AND (Label.active OR Label.active IS NULL) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_todestreak_warp.sql (revision 41104) @@ -0,0 +1,46 @@ +SELECT DISTINCT + magicDSRun.magic_ds_id, + magicRun.magic_id, + magicDSRun.state, + magicRun.exp_id, + magicDSRun.label, + camera, + magicMask.uri as streaks_uri, + magicMask.path_base as streaks_path_base, + CAST(NULL AS CHAR(255)) AS inv_streaks_uri, + CAST(NULL AS CHAR(255)) AS inv_streaks_path_base, + stage, + stage_id, + warpRun.magicked, + warpSkyfile.skycell_id as component, + diffRun.tess_id != warpRun.tess_id AS mismatched_tess, + diffRun.tess_id as diff_tess_id, + warpSkyfile.uri, + warpSkyfile.path_base, + magicRun.inverse, + CAST(NULL AS CHAR(255)) as cam_path_base, + CAST(NULL AS CHAR(255)) as cam_reduction, + outroot, + recoveryroot, + re_place, + remove, + IFNULL(Label.priority, 10000) AS priority +FROM magicDSRun +JOIN magicMask USING (magic_id) +JOIN magicRun USING (magic_id) +JOIN diffRun USING(diff_id) +JOIN warpRun ON warp_id = stage_id +JOIN warpSkyfile USING(warp_id) +JOIN rawExp ON magicRun.exp_id = rawExp.exp_id +LEFT JOIN magicDSFile + ON magicDSRun.magic_ds_id = magicDSFile.magic_ds_id + AND magicDSFile.component = warpSkyfile.skycell_id +LEFT JOIN Label ON magicDSRun.label = Label.label +WHERE + magicDSRun.state = 'new' + AND magicDSRun.stage = 'warp' + AND warpRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + AND magicDSFile.component IS NULL + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_toremove.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_toremove.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_toremove.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + magic_ds_id, + stage, + stage_id, + component, + backup_path_base +FROM magicDSFile +JOIN magicDSRun USING(magic_ds_id) +WHERE magicDSRun.remove + AND magicDSRun.state = 'full' + AND backup_path_base IS NOT NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_camera.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT + magic_ds_id, + magicDSRun.state, + exp_id, + re_place, + camera, + stage, + stage_id, + component, + outroot, + camProcessedExp.path_base, + camProcessedExp.path_base AS cam_path_base, + recovery_path_base, + 0 AS bothways, + 0 AS bytes, + 0 AS md5sum, + camRun.magicked +FROM magicDSRun + JOIN magicDSFile using(magic_ds_id) + JOIN camProcessedExp ON stage_id = camProcessedExp.cam_id + JOIN camRun ON camRun.cam_id = camProcessedExp.cam_id + JOIN chipRun USING(chip_id) + JOIN rawExp using(exp_id) +WHERE magicDSRun.stage = 'camera' + AND camProcessedExp.fault = 0 + AND camRun.state = 'full' + AND ((magicDSRun.state = 'new' AND magicDSFile.fault > 0) + OR ((magicDSRun.state = 'goto_censored' OR magicDSRun.state = 'goto_restored') + AND (backup_path_base IS NOT NULL) + ) + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_chip.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT + magic_ds_id, + magicDSRun.state, + rawExp.exp_id, + re_place, + camera, + stage, + stage_id, + component, + outroot, + chipProcessedImfile.path_base, + camProcessedExp.path_base AS cam_path_base, + recovery_path_base, + 0 AS bothways, + 0 AS bytes, + 0 AS md5sum, + chipRun.magicked +FROM magicDSRun + JOIN magicDSFile using(magic_ds_id) + JOIN camProcessedExp using(cam_id) + JOIN chipRun ON (stage_id = chipRun.chip_id) + JOIN chipProcessedImfile ON (stage_id = chipProcessedImfile.chip_id AND component = class_id) + JOIN rawExp ON chipRun.exp_id = rawExp.exp_id +WHERE magicDSRun.stage = 'chip' + AND (((magicDSRun.state = 'new' OR (magicDSRun.state = 'update')) + AND magicDSFile.fault > 0) + -- why don't we require a fault for these states? + OR ((magicDSRun.state = 'goto_censored' OR magicDSRun.state = 'goto_restored') + AND ((backup_path_base IS NOT NULL) OR (recovery_path_base IS NOT NULL)) + ) + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_diff.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT + magic_ds_id, + magicDSRun.state, + exp_id, + re_place, + camera, + stage, + stage_id, + component, + outroot, + diffSkyfile.path_base, + magicDSFile.backup_path_base, + magicDSFile.recovery_path_base, + "NULL" AS cam_path_base, + CAST(diffRun.bothways AS SIGNED) AS bothways, + 0 AS bytes, + 0 AS md5sum, + diffRun.magicked +FROM magicDSRun + JOIN magicRun using(magic_id) + JOIN magicDSFile USING(magic_ds_id) + JOIN diffSkyfile ON (stage_id = diffSkyfile.diff_id AND component = skycell_id) + JOIN diffRun ON diffSkyfile.diff_id = diffRun.diff_id + JOIN rawExp USING(exp_id) +WHERE magicDSRun.stage = 'diff' + AND ((magicDSRun.state = 'new' AND magicDSFile.fault > 0) + OR ((magicDSRun.state = 'goto_censored' OR magicDSRun.state = 'goto_restored') + AND ((backup_path_base IS NOT NULL) OR (recovery_path_base IS NOT NULL)) + ) + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_raw.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT + magic_ds_id, + magicDSRun.state, + exp_id, + re_place, + rawImfile.camera, + stage, + stage_id, + component, + outroot, + rawImfile.uri AS path_base, + CAST(NULL AS CHAR(255)) AS cam_path_base, + recovery_path_base, + 0 AS bothways, + bytes, + md5sum, + 0 AS magicked +FROM magicDSRun + JOIN magicDSFile using(magic_ds_id) + JOIN rawImfile ON (stage_id = exp_id AND component = class_id) +WHERE magicDSRun.stage = 'raw' + AND ((magicDSRun.state = 'new' AND magicDSFile.fault > 0) + OR ((magicDSRun.state = 'goto_censored' OR magicDSRun.state = 'goto_restored') + AND ((backup_path_base IS NOT NULL) OR (recovery_path_base IS NOT NULL)) + ) + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magicdstool_torevert_warp.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT + magic_ds_id, + magicDSRun.state, + exp_id, + re_place, + camera, + stage, + stage_id, + component, + outroot, + warpSkyfile.path_base, + "NULL" AS cam_path_base, + recovery_path_base, + 0 AS bothways, + 0 AS bytes, + 0 AS md5sum, + warpRun.magicked +FROM magicDSRun + JOIN magicDSFile USING(magic_ds_id) + JOIN warpSkyfile ON (stage_id = warp_id AND component = skycell_id) + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun ON fakeRun.cam_id = camRun.cam_id + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) +WHERE magicDSRun.stage = 'warp' + AND ((magicDSRun.state = 'new' AND magicDSFile.fault > 0) + OR ((magicDSRun.state = 'goto_censored' OR magicDSRun.state = 'goto_restored') + AND ((backup_path_base IS NOT NULL) OR (recovery_path_base IS NOT NULL)) + ) + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_chipprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_chipprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_chipprocessedimfile.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT DISTINCT + chip_id, + class_id, + chipProcessedImfile.uri, + chipProcessedImfile.path_base, + camProcessedExp.path_base as cam_path_base +FROM magicRun +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffRun.diff_id + AND diffInputSkyfile.skycell_id = diffRun.skycell_id + -- Want input warps only + AND diffInputSkyfile.warp_id IS NOT NULL + AND diffInputSkyfile.template = 0 +JOIN warpRun USING(warp_id) +JOIN warpSkyCellMap + ON warpSkyCellMap.warp_id = warpRun.warp_id + AND warpSkyCellMap.skycell_id = diffRun.skycell_id +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN camProcessedExp USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN chipProcessedImfile USING(chip_id,class_id) +WHERE + chipRun.state = 'full' + AND chipProcessedImfile.fault = 0 + AND chipProcessedImfile.quality = 0 +-- AND magicRun.state = 'full' +-- AND magicMask.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_create_tmp_warpcomplete.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_create_tmp_warpcomplete.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_create_tmp_warpcomplete.sql (revision 41104) @@ -0,0 +1,3 @@ +CREATE TEMPORARY TABLE warpComplete + (warp_id BIGINT, skycell_id VARCHAR(64), tess_id VARCHAR(64), + PRIMARY KEY(warp_id, skycell_id, tess_id)) ENGINE=MEMORY; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_insert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_insert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_insert.sql (revision 41104) @@ -0,0 +1,10 @@ +-- Insert the best list of diffs as magic inputs +INSERT INTO magicInputSkyfile +SELECT + @MAGIC_ID@, -- Update this with the appropriate magic_id + skycell_id +FROM diffSkyfile +WHERE + diff_id = @DIFF_ID@ -- Update this with the appropriate diff_id + AND fault = 0 + AND quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_select.sql (revision 41104) @@ -0,0 +1,52 @@ +-- Get a list of exposures on which magic may be performed +SELECT DISTINCT + chipRun.exp_id, + MAX(diffWarps.diff_id) AS diff_id, + -- The following trick pulls out the 'inverse' value for the maximum diff_id + CONVERT(SUBSTRING_INDEX(GROUP_CONCAT(diffWarps.inverse ORDER BY diffWarps.diff_id DESC), ',', 1), UNSIGNED) AS inverse, + diff_data_group +FROM ( + -- Forward diffs + SELECT + diffRun.diff_id, + warp1 AS warp_id, + 0 AS inverse, + diffRun.data_group AS diff_data_group + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + JOIN diffSkyfile USING(diff_id, skycell_id) + WHERE diffInputSkyfile.warp1 IS NOT NULL + AND diffRun.exposure = 1 + AND diffRun.magicked = 0 + AND diffSkyfile.quality = 0 + -- diff WHERE hook %s + UNION + -- Backward diffs + SELECT + diffRun.diff_id, + warp2 AS warp_id, + 1 AS inverse, + diffRun.data_group AS diff_data_group + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + JOIN diffSkyfile USING(diff_id, skycell_id) + WHERE diffInputSkyfile.warp2 IS NOT NULL + AND diffRun.exposure = 1 + AND diffRun.bothways = 1 + AND diffRun.magicked = 0 + AND diffSkyfile.quality = 0 + -- diff WHERE hook %s + ) AS diffWarps +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN + (SELECT magic_id, exp_id, label + FROM magicRun + -- rerun hook %s + ) AS oldMagicRun + ON oldMagicRun.exp_id = chipRun.exp_id +-- WHERE hook %s +GROUP BY chipRun.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_select_multidiff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_select_multidiff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_definebyquery_select_multidiff.sql (revision 41104) @@ -0,0 +1,53 @@ +-- Get a list of exposures on which magic may be performed +SELECT DISTINCT + chipRun.exp_id, + MAX(diffWarps.diff_id) AS diff_id, + -- The following trick pulls out the 'inverse' value for the maximum diff_id + CONVERT(SUBSTRING_INDEX(GROUP_CONCAT(diffWarps.inverse ORDER BY diffWarps.diff_id DESC), ',', 1), UNSIGNED) AS inverse, + diff_data_group +FROM ( + -- Forward diffs + SELECT + diffRun.diff_id, + warp1 AS warp_id, + 0 AS inverse, + diffRun.data_group AS diff_data_group + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + JOIN diffSkyfile USING(diff_id, skycell_id) + WHERE diffInputSkyfile.warp1 IS NOT NULL + AND diffRun.exposure = 1 + AND diffRun.magicked = 0 + AND diffSkyfile.quality = 0 + -- diff WHERE hook %s + UNION + -- Backward diffs + SELECT + diffRun.diff_id, + warp2 AS warp_id, + 1 AS inverse, + diffRun.data_group AS diff_data_group + FROM diffRun + JOIN diffInputSkyfile USING(diff_id) + JOIN diffSkyfile USING(diff_id, skycell_id) + WHERE diffInputSkyfile.warp2 IS NOT NULL + AND diffRun.exposure = 1 + AND diffRun.bothways = 1 + AND diffRun.magicked = 0 + AND diffSkyfile.quality = 0 + -- diff WHERE hook %s + ) AS diffWarps +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN + (SELECT magic_id, exp_id, diff_id, label + FROM magicRun + -- rerun hook %s + ) AS oldMagicRun + ON oldMagicRun.exp_id = chipRun.exp_id + AND oldMagicRun.diff_id = diffWarps.diff_id +-- WHERE hook %s +GROUP BY chipRun.exp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_deletemask.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_deletemask.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_deletemask.sql (revision 41104) @@ -0,0 +1,9 @@ +DELETE magicMask +FROM magicRun + JOIN magicNodeResult USING(magic_id) + JOIN magicMask USING(magic_id) +WHERE + magicRun.state = 'new' + AND magicNodeResult.node = 'root' + AND magicNodeResult.fault != 0 + AND magicRun.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_diffskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_diffskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_diffskyfile.sql (revision 41104) @@ -0,0 +1,28 @@ +SELECT DISTINCT + diffSkyfile.diff_id, + diffRun.skycell_id, + diffSkyfile.uri, + diffSkyfile.path_base +FROM magicRun +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffSkyfile USING(diff_id) +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffRun.diff_id + AND diffInputSkyfile.skycell_id = diffRun.skycell_id + -- Want input warps only + AND diffInputSkyfile.warp_id IS NOT NULL + AND diffInputSkyfile.template = 0 +JOIN warpSkyCellMap + ON warpSkyCellMap.warp_id = diffInputSkyfile.warp_id + AND warpSkyCellMap.skycell_id = diffInputSkyfile.skycell_id +JOIN warpSkyfile + ON warpSkyfile.warp_id = warpSkyCellMap.warp_id + AND warpSkyfile.skycell_id = warpSkyCellMap.skycell_id + AND warpSkyfile.quality = 0 +WHERE + diffRun.state = 'full' + AND diffSkyfile.fault = 0 + AND diffSkyfile.quality = 0 +-- AND magicRun.state = 'full' +-- AND magicMask.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_exposure.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_exposure.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_exposure.sql (revision 41104) @@ -0,0 +1,54 @@ +SELECT * +FROM ( + -- Forward diff + SELECT DISTINCT + magicRun.magic_id, + rawExp.*, + camProcessedExp.path_base, + camProcessedExp.zpt_obs, + camProcessedExp.zpt_stdev, + camProcessedExp.fwhm_major, + camProcessedExp.fwhm_minor, + camProcessedExp.iq_m2, + camProcessedExp.iq_m3, + camProcessedExp.iq_m4 + FROM + magicRun + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + LEFT JOIN warpRun + ON warpRun.warp_id = diffInputSkyfile.warp1 + AND magicRun.inverse = 0 + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN camProcessedExp USING(cam_id) + JOIN chipRun USING(chip_id,exp_id) + JOIN rawExp USING(exp_id) + -- WHERE hook %s + UNION + -- Backward diff + SELECT DISTINCT + magicRun.magic_id, + rawExp.*, + camProcessedExp.path_base, + camProcessedExp.zpt_obs, + camProcessedExp.zpt_stdev, + camProcessedExp.fwhm_major, + camProcessedExp.fwhm_minor, + camProcessedExp.iq_m2, + camProcessedExp.iq_m3, + camProcessedExp.iq_m4 + FROM + magicRun + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + LEFT JOIN warpRun + ON warpRun.warp_id = diffInputSkyfile.warp2 + AND magicRun.inverse = 1 + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN camProcessedExp USING(cam_id) + JOIN chipRun USING(chip_id,exp_id) + JOIN rawExp USING(exp_id) + -- WHERE hook %s +) AS magicExposures Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_inputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_inputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_inputs.sql (revision 41104) @@ -0,0 +1,73 @@ +-- Get input details for a magic node +SELECT + magicRun.magic_id, + magicRun.inverse, -- Using the inverse subtraction? + magicRun.diff_id, + magicInputs.node, + -- Only one of diff_path_base and magic_path_base should be non-NULL + -- If diff_path_base is non-NULL, then only one of warp_path_base and stack_path_base should be non-NULL + magicInputs.diff_path_base, -- path_base for the diff (if any) + magicInputs.warp_path_base, -- path_base for the template warp (if any) + magicInputs.stack_path_base, -- path_base for the template stack (if any) + magicInputs.magic_path_base -- path_base for child nodes (if any) +FROM magicRun +JOIN ( + -- Single skycells: have uri=NULL + SELECT + magic_id, + magicRun.diff_id, + skycell_id AS node, + diffSkyfile.path_base AS diff_path_base, + warpSkyfile.path_base AS warp_path_base, + stackSumSkyfile.path_base AS stack_path_base, + NULL AS magic_path_base + FROM magicRun + JOIN magicTree USING(magic_id) + JOIN magicInputSkyfile USING(magic_id, node) + JOIN diffRun USING(diff_id) + JOIN diffSkyfile + ON diffSkyfile.diff_id = magicRun.diff_id + AND diffSkyfile.skycell_id = magicInputSkyfile.node + JOIN ( + -- Template for non-inverse + SELECT + magic_id, + skycell_id, + warp2 AS warp_id, + stack2 AS stack_id + FROM magicRun + JOIN diffInputSkyfile USING(diff_id) + WHERE magicRun.inverse = 0 + -- WHERE hook (magicRun.magic_id, diffInputSkyfile.skycell_id) %s + UNION + -- Template for inverse + SELECT + magic_id, + skycell_id, + warp1 AS warp_id, + stack1 AS stack_id + FROM magicRun + JOIN diffInputSkyfile USING(diff_id) + WHERE magicRun.inverse = 1 + -- WHERE hook (magicRun.magic_id, diffInputSkyfile.skycell_id) %s + ) AS diffTemplates USING(magic_id, skycell_id) + LEFT JOIN warpSkyfile USING(warp_id, skycell_id) + LEFT JOIN stackSumSkyfile USING(stack_id) + -- WHERE hook (magicRun.magic_id, magicTree.node) %s + UNION + -- Merged skycells: have diff_id=0, various_path_base=NULL + SELECT + magicTree.magic_id, + 0 AS diff_id, + magicTree.dep, + NULL AS diff_path_base, + NULL AS warp_path_base, + NULL AS stack_path_base, + magicNodeResult.path_base AS magic_path_base + FROM magicTree + JOIN magicRun USING(magic_id) + JOIN magicNodeResult + ON magicTree.magic_id = magicNodeResult.magic_id + AND magicTree.dep = magicNodeResult.node + -- WHERE hook (magicRun.magic_id, magicTree.node) %s + ) AS magicInputs USING(magic_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_inputskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_inputskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_inputskyfile.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + magicInputSkyfile.*, + magicRun.inverse, + diffSkyfile.path_base +FROM magicRun +JOIN magicInputSkyfile USING(magic_id) +JOIN diffSkyfile + ON diffSkyfile.diff_id = magicRun.diff_id + AND diffSkyfile.skycell_id = magicInputSkyfile.node +WHERE + magicRun.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_mask.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_mask.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_mask.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + magicMask.*, + magicRun.exp_id +FROM magicMask +JOIN magicRun + USING(magic_id) +WHERE + magicRun.state = 'full' + AND magicMask.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_rawimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_rawimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_rawimfile.sql (revision 41104) @@ -0,0 +1,27 @@ +SELECT DISTINCT + exp_id, + class_id, + rawImfile.uri, + camProcessedExp.path_base as cam_path_base +FROM magicRun +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffRun.diff_id + -- Want input warps only + AND diffInputSkyfile.warp_id IS NOT NULL + AND diffInputSkyfile.template = 0 +JOIN warpRun USING(warp_id) +JOIN warpSkyCellMap + ON warpSkyCellMap.warp_id = warpRun.warp_id + AND warpSkyCellMap.skycell_id = diffRun.skycell_id +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN camProcessedExp USING(cam_id) +JOIN chipRun USING(chip_id,exp_id) +JOIN rawExp USING(exp_id) +JOIN rawImfile USING(exp_id,class_id) +WHERE + rawImfile.fault = 0 +-- AND magicRun.state = 'full' +-- AND magicMask.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_camera.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_camera.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_camera.sql (revision 41104) @@ -0,0 +1,6 @@ +UPDATE magicRun + JOIN magicDSRun USING(magic_id) + JOIN camRun ON stage = 'camera' AND stage_id = camRun.cam_id +SET camRun.magicked = 0, + magicDSRun.state = '@NEW_STATE@' +WHERE re_place Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_chip.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE magicRun + JOIN magicDSRun USING(magic_id) + JOIN chipRun ON stage = 'chip' AND stage_id = chip_id + JOIN chipProcessedImfile USING(chip_id) +SET chipProcessedImfile.magicked = 0, + chipRun.magicked = 0, + magicDSRun.state = '@NEW_STATE@' +WHERE re_place Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_diff.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE magicRun + JOIN magicDSRun USING(magic_id) + JOIN diffRun ON stage = 'diff' AND stage_id = diffRun.diff_id + JOIN diffSkyfile ON diffRun.diff_id = diffSkyfile.diff_id +SET diffRun.magicked = 0, + diffSkyfile.magicked = 0, + magicDSRun.state = '@NEW_STATE@' +WHERE re_place Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_raw.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_raw.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_raw.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE magicRun + JOIN magicDSRun USING(magic_id) + JOIN rawExp ON stage = 'raw' AND stage_id = rawExp.exp_id + JOIN rawImfile ON rawExp.exp_id = rawImfile.exp_id +SET rawImfile.magicked = 0, + rawExp.magicked = 0, + magicDSRun.state = '@NEW_STATE@' +WHERE re_place Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_restore_warp.sql (revision 41104) @@ -0,0 +1,8 @@ +UPDATE magicRun + JOIN magicDSRun USING(magic_id) + JOIN warpRun ON stage = 'warp' AND stage_id = warp_id + JOIN warpSkyfile using(warp_id) +SET warpRun.magicked = 0, + warpSkyfile.magicked = 0, + magicDSRun.state = '@NEW_STATE@' +WHERE re_place Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_revertnode.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_revertnode.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_revertnode.sql (revision 41104) @@ -0,0 +1,5 @@ +DELETE magicNodeResult +FROM magicNodeResult +JOIN magicRun USING(magic_id) +WHERE magicNodeResult.fault != 0 + AND magicRun.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_setfull.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_setfull.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_setfull.sql (revision 41104) @@ -0,0 +1,6 @@ +UPDATE + magicRun +SET + state = 'full' +WHERE + state != 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_setgotocleaned.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_setgotocleaned.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_setgotocleaned.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE magicRun + SET workdir_state = 'goto_cleaned' +WHERE workdir_state = 'dirty' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_tocleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_tocleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_tocleanup.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT magic_id, + magicRun.exp_id, + rawExp.camera, + magicRun.workdir, + IFNULL(priority, 10000) as priority +FROM magicRun + JOIN rawExp USING(exp_id) + LEFT JOIN Label ON magicRun.label = Label.label +WHERE magicRun.workdir_state = 'goto_cleaned' + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_tomask.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_tomask.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_tomask.sql (revision 41104) @@ -0,0 +1,18 @@ +SELECT + magic_id, + exp_id, + camera, + magicRun.workdir, + uri +FROM magicRun +JOIN rawExp USING(exp_id) +JOIN magicTree USING(magic_id) +LEFT JOIN magicNodeResult USING(magic_id, node) +WHERE + magicRun.state = 'new' + AND magicNodeResult.node = 'root' + AND magicNodeResult.fault = 0 +GROUP BY + magic_id +HAVING + COUNT(magicTree.node) = COUNT(magicNodeResult.node) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_inputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_inputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_inputs.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT + magicTree.*, + magicRun.workdir, + rawExp.exp_id, + rawExp.exp_name, + rawExp.workdir AS raw_workdir, + rawExp.camera, + -- convert magic_id into a boolean value (1 or 0) + -- note that the type stays a 64 bit int + magicNodeResult.magic_id IS TRUE AS done, + magicNodeResult.fault IS TRUE AS bad, + IFNULL(Label.priority, 10000) AS priority +FROM magicRun +JOIN magicTree USING(magic_id) +JOIN magicInputSkyfile USING(magic_id, node) +JOIN rawExp USING(exp_id) +JOIN diffSkyfile -- only get nodes that match a skycell + ON diffSkyfile.diff_id = magicRun.diff_id + AND diffSkyfile.skycell_id = magicInputSkyfile.node +JOIN diffRun + ON diffRun.diff_id = diffSkyfile.diff_id +LEFT JOIN magicNodeResult USING(magic_id, node) +LEFT JOIN Label ON magicRun.label = Label.label +WHERE + magicRun.state = 'new' + AND magicNodeResult.magic_id IS NULL + AND magicNodeResult.node IS NULL + AND diffRun.state = 'full' + AND (Label.active OR Label.active IS NULL) +-- WHERE hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_runs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_runs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_runs.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + magic_id, + IFNULL(priority, 10000) AS priority +FROM magicTree +JOIN magicRun USING(magic_id) +LEFT JOIN magicNodeResult USING(magic_id, node) +LEFT JOIN Label ON magicRun.label = Label.label +WHERE + magicRun.state = 'new' +AND magicNodeResult.magic_id IS NULL +-- WHERE hook %s +ORDER BY + priority DESC, magicRun.magic_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_tree.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_tree.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toprocess_tree.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT + magicTree.*, + magicRun.workdir, + rawExp.exp_id, + rawExp.exp_name, + rawExp.workdir AS raw_workdir, + rawExp.camera, + -- convert magic_id into a boolean value (1 or 0) + -- note that the type stays a 64 bit int + magicNodeResult.magic_id IS TRUE AS done, + magicNodeResult.fault IS TRUE AS bad, + IFNULL(Label.priority, 10000) AS priority +FROM magicTree +JOIN magicRun USING(magic_id) +JOIN rawExp USING(exp_id) +LEFT JOIN magicNodeResult USING(magic_id, node) +LEFT JOIN Label ON magicRun.label = Label.label +WHERE + magicRun.state = 'new' + AND (Label.active OR Label.active IS NULL) +-- WHERE hook %s +ORDER BY + priority, magicRun.magic_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toskyfilemask.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toskyfilemask.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_toskyfilemask.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT + magicMask.* +FROM magicRun +JOIN magicMask + USING(magic_id) +LEFT JOIN magicSkyfileMask + USING(magic_id) +WHERE + magicRun.state = 'new' + AND magicSkyfileMask.magic_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_totree.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_totree.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_totree.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + magic_id, + exp_id, + camera, + magicRun.workdir, + ra, + decl, + diffRun.tess_id, + IFNULL(priority, 10000) AS priority +FROM magicRun +JOIN diffRun USING(diff_id) +JOIN rawExp USING(exp_id) +LEFT JOIN magicTree + USING(magic_id) +LEFT JOIN Label ON magicRun.label = Label.label +WHERE + magicRun.state = 'new' + AND magicTree.node IS NULL + AND magicRun.fault = 0 + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_warpskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_warpskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/magictool_warpskyfile.sql (revision 41104) @@ -0,0 +1,25 @@ +SELECT DISTINCT + warpSkyfile.warp_id, + warpSkyfile.skycell_id, + warpSkyfile.uri, + warpSkyfile.path_base +FROM magicRun +JOIN magicInputSkyfile USING(magic_id) +JOIN diffRun USING(diff_id) +JOIN diffInputSkyfile + ON diffInputSkyfile.diff_id = diffRun.diff_id + AND diffInputSkyfile.skycell_id = diffRun.skycell_id + -- Want input warps only + AND diffInputSkyfile.warp_id IS NOT NULL + AND diffInputSkyfile.template = 0 +JOIN warpSkyfile + ON warpSkyfile.warp_id = diffInputSkyfile.warp_id + AND warpSkyfile.skycell_id = diffRun.skycell_id +JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id +WHERE + warpRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 +-- AND magicRun.state = 'full' +-- AND magicMask.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_merge_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_merge_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_merge_id_dvo.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT mergedvodbRun.* FROM mergedvodbRun +JOIN mergedvodbProcessed USING(merge_id) +WHERE ( mergedvodbRun.state = 'merged' OR mergedvodbRun.state = 'full') + AND mergedvodbProcessed.fault = 0 + AND merge_id NOT IN (SELECT merge_id + FROM mergedvodbCopy + JOIN mergedvodbRun USING(mergedvodb_id) + WHERE %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_mergedvodbcopy.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_mergedvodbcopy.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_mergedvodbcopy.sql (revision 41104) @@ -0,0 +1,7 @@ +SELECT + mergedvodbCopy.*, + mergedvodbRun.mergedvodb_path +FROM mergedvodbCopy + JOIN mergedvodbRun using (mergedvodb_id) + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_minidvodb_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_minidvodb_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_minidvodb_id.sql (revision 41104) @@ -0,0 +1,8 @@ +SELECT minidvodbRun.* FROM minidvodbRun +JOIN minidvodbProcessed USING(minidvodb_id) +WHERE minidvodbRun.state = 'merged' and minidvodbProcessed.fault = 0 + AND minidvodb_id NOT IN (SELECT minidvodb_id + FROM mergedvodbRun + JOIN minidvodbRun USING(minidvodb_id) + WHERE %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_pending_merge.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_pending_merge.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_pending_merge.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT + mergedvodbRun.*, + minidvodbRun.minidvodb_path +FROM minidvodbRun +JOIN mergedvodbRun + USING(minidvodb_id) +LEFT JOIN mergedvodbProcessed + USING(merge_id) +WHERE + (mergedvodbRun.state = 'new' AND mergedvodbProcessed.merge_id IS NULL) + AND mergedvodbRun.mergedvodb IS NOT NULL + AND mergedvodbRun.mergedvodb_path IS NOT NULL + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_pendingmergeprocess.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_pendingmergeprocess.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_pendingmergeprocess.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT mergedvodbRun.* +FROM mergedvodbRun +LEFT JOIN mergedvodbProcessed +USING(merge_id) +WHERE (mergedvodbRun.state = 'new' AND mergedvodbProcessed.merge_id IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_processed.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_processed.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_find_processed.sql (revision 41104) @@ -0,0 +1,7 @@ +SELECT + mergedvodbProcessed.*, + mergedvodbRun.mergedvodb_path +FROM mergedvodbProcessed +JOIN mergedvodbRun + USING(merge_id) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_merge_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_merge_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_merge_id.sql (revision 41104) @@ -0,0 +1,11 @@ +INSERT INTO mergedvodbRun + SELECT + 0, -- merge_id + minidvodb_id, -- minidvodb_id + '%s', -- mergedvodb + '%s', -- mergedvodb_path + '%s', -- state + FROM minidvodbRun join minidvodbProcessed using (minidvodb_id) + WHERE + minidvodbRun.state = 'merged' and minidvodbProcessed.fault = 0 + AND minidvodbRun.minidvodb_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_mergecopy_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_mergecopy_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_mergecopy_id.sql (revision 41104) @@ -0,0 +1,16 @@ +INSERT INTO mergedvodbCopy + SELECT + 0, -- mergedvodbcopy_id + merge_id, -- merge_id + '%s', -- mergedvodb_rsync_path + '%s', -- destination_host + %d, -- fault + '%s', -- state + %s, -- epoch + %f -- dtime + + FROM mergedvodbRun + WHERE + (mergedvodbRun.state = 'merged' + OR mergedvodbRun.state = 'full') + AND mergedvodbRun.merge_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_minidvodb_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_minidvodb_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_queue_minidvodb_id.sql (revision 41104) @@ -0,0 +1,15 @@ +INSERT INTO minidvodbCopy + SELECT + 0, -- minidvodbcopy_id + minidvodb_id, -- minidvodb_id + '%s', -- minidvodb_rsync_path + '%s', -- destination_host + %d, -- fault + '%s', -- state + %s, -- epoch + %f -- dtime + + FROM minidvodbRun + WHERE + minidvodbRun.state = 'merged' + AND minidvodbRun.minidvodb_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_revertmergedvodbcopy.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_revertmergedvodbcopy.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_revertmergedvodbcopy.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE mergedvodbCopy + -- SET minidvodbCopy.state = 'new', + SET mergedvodbCopy.fault = 0 +WHERE + mergedvodbCopy.state = 'new' + AND mergedvodbCopy.fault >0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_revertmergedvodbprocessed.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_revertmergedvodbprocessed.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/mergetool_revertmergedvodbprocessed.sql (revision 41104) @@ -0,0 +1,6 @@ +DELETE FROM mergedvodbProcessed +USING mergedvodbProcessed, mergedvodbRun +WHERE + mergedvodbProcessed.merge_id = mergedvodbRun.merge_id + AND mergedvodbProcessed.fault != 0 + AND mergedvodbRun.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_find_minidvodb_id_dvo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_find_minidvodb_id_dvo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_find_minidvodb_id_dvo.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT minidvodbRun.* FROM minidvodbRun +JOIN minidvodbProcessed USING(minidvodb_id) +WHERE minidvodbRun.state = 'merged' + AND minidvodbProcessed.fault = 0 + AND minidvodb_id NOT IN (SELECT minidvodb_id + FROM minidvodbCopy + JOIN minidvodbRun USING(minidvodb_id) + WHERE %s + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_find_minidvodbcopy.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_find_minidvodbcopy.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_find_minidvodbcopy.sql (revision 41104) @@ -0,0 +1,7 @@ +SELECT + minidvodbCopy.*, + minidvodbRun.minidvodb_path +FROM minidvodbCopy + JOIN minidvodbRun using (minidvodb_id) + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_queue_minidvodb_id.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_queue_minidvodb_id.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_queue_minidvodb_id.sql (revision 41104) @@ -0,0 +1,15 @@ +INSERT INTO minidvodbCopy + SELECT + 0, -- minidvodbcopy_id + minidvodb_id, -- minidvodb_id + '%s', -- minidvodb_rsync_path + '%s', -- destination_host + %d, -- fault + '%s', -- state + %s, -- epoch + %f -- dtime + + FROM minidvodbRun + WHERE + minidvodbRun.state = 'merged' + AND minidvodbRun.minidvodb_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_revertminidvodbcopy.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_revertminidvodbcopy.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/minidvodbtool_revertminidvodbcopy.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE minidvodbCopy + -- SET minidvodbCopy.state = 'new', + SET minidvodbCopy.fault = 0 +WHERE + minidvodbCopy.state = 'new' + AND minidvodbCopy.fault >0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/parse_pxadmin_create_tables.pl =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/parse_pxadmin_create_tables.pl (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/parse_pxadmin_create_tables.pl (revision 41104) @@ -0,0 +1,34 @@ +#!/usr/bin/env perl + +use warnings; +use strict; + +my $ctr=0; +my $old=""; +while (<>) { + s/^\s+//; #remove whitespace from front + s/\s*--.*\n$//; # remove all comments + s/\s+/ /g; # kill the middle whitespace + s/\s*$//; # kill the whitespace at the end of the line +# if ($_ !~ /;$/ ) {chomp $_}; + s/;$/;\n/; # add back in the \n that was removed as whitespace but only for ; + s/,$/, /; # add space at end of , if at end of line + if (m/CREATE\s+TABLE\s+receive/) {$ctr = 1;} + if (m/;/) {$ctr = 0;} + if ($ctr == 0) { + s/AUTO_INCREMENT// ; + if (/FOREIGN\s+KEY/) { + if (!/, $/) { #if FOREIGN KEY line doesn't have , + $old =~ s/,\s*$//; #remove , from previous line + print $old; + $old=""; + } + } else { + print $old; + $old=$_; + } + } else { print $_; + } +} +print $old; +print "-- This comment line is here to avoid empty query error.\n"; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_addaccesslevel.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_addaccesslevel.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_addaccesslevel.sql (revision 41104) @@ -0,0 +1,4 @@ +INSERT + INTO pstampAccessLevel + SELECT proj_id, %d, %f, %f FROM pstampProject + WHERE name = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_completedreq.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_completedreq.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_completedreq.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT pstampRequest.* , + IFNULL(Label.priority, 0) AS priority +FROM pstampRequest + LEFT JOIN Label USING(label) +WHERE state = 'run' + AND pstampRequest.fault = 0 + AND ( + SELECT count(*) FROM pstampJob + WHERE pstampJob.req_id = pstampRequest.req_id + AND pstampJob.state != 'stop' + AND pstampJob.state != 'cancel' + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_datastore.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_datastore.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_datastore.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT pstampDataStore.* +FROM pstampDataStore +WHERE + state = 'enabled' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_getdependent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_getdependent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_getdependent.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT DISTINCT pstampDependent.* +FROM pstampDependent +WHERE (pstampDependent.state = 'new' OR pstampDependent.state = 'hold') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listfile.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + pstampFile.*, + pstampJob.req_id +FROM pstampFile + JOIN pstampJob using(job_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listjob.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listjob.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listjob.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT + pstampJob.*, + pstampRequest.name, + pstampRequest.outProduct +FROM pstampJob + JOIN pstampRequest USING(req_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listuser.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listuser.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_listuser.sql (revision 41104) @@ -0,0 +1,10 @@ +SELECT userName, + domainName, + pstampUser.accessLevel, + IFNULL(pstampUserDomain.accessLevel, 0) AS domainAccessLevel, + pstampUser.defaultProduct as userProduct, + pstampUser.defaultLabel as userLabel, + pstampUserDomain.defaultProduct as domainProduct, + pstampUserDomain.defaultLabel as domainLabel +FROM pstampUser + LEFT JOIN pstampUserDomain USING(domainName) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingcleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingcleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingcleanup.sql (revision 41104) @@ -0,0 +1,9 @@ +-- postage stamp requests pending cleanup +SELECT + pstampRequest.*, + IFNULL(Label.priority, 10000) AS priority +FROM pstampRequest + LEFT JOIN Label ON pstampRequest.label = Label.label +WHERE pstampRequest.state = 'goto_cleaned' + AND pstampRequest.fault = 0 + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingdependent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingdependent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingdependent.sql (revision 41104) @@ -0,0 +1,14 @@ +SELECT DISTINCT + pstampDependent.*, + pstampRequest.label, + MIN(req_id) as first_req_id, + concat( pstampDependent.outdir, '/checkdep.', dep_id, '.log') as logfile, + IFNULL(Label.priority, 10000) AS priority +FROM pstampDependent +JOIN pstampJob USING(dep_id) +JOIN pstampRequest USING(req_id) +LEFT JOIN Label ON pstampRequest.label = Label.label +WHERE pstampDependent.state = 'new' + AND pstampJob.state = 'run' + AND pstampRequest.state = 'run' + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingjob.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingjob.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingjob.sql (revision 41104) @@ -0,0 +1,48 @@ +SELECT * FROM +(SELECT pstampJob.*, + IFNULL(Label.priority, 10000) AS priority, + 0 AS myParent, + 0 AS runningChildren +FROM pstampJob + JOIN pstampRequest USING(req_id) + LEFT JOIN pstampDependent USING(dep_id) + LEFT JOIN Label ON pstampRequest.label = Label.label +WHERE pstampRequest.state = 'run' + AND pstampRequest.fault = 0 + AND pstampJob.state = 'run' + AND pstampJob.fault = 0 + AND NOT is_parent + AND (dep_id = 0 OR pstampDependent.state = 'full') + AND (Label.active OR Label.active IS NULL) + -- where hook 1 %s +UNION +SELECT * FROM + (SELECT pstampJob.*, + IFNULL(Label.priority, 10000) AS priority + FROM pstampJob + JOIN pstampRequest USING(req_id) + LEFT JOIN pstampDependent USING(dep_id) + LEFT JOIN Label ON pstampRequest.label = Label.label + WHERE pstampRequest.state = 'run' + AND pstampRequest.fault = 0 + AND pstampJob.state = 'run' + AND pstampJob.fault = 0 + AND is_parent + AND (dep_id = 0 OR pstampDependent.state = 'full') + AND (Label.active OR Label.active IS NULL) + -- where hook 2 %s + ) as parentJob + LEFT JOIN + (SELECT parent_id AS myParent, + count(job_id) as runningChildren + FROM pstampRequest JOIN pstampJob USING(req_id) + WHERE pstampRequest.state ='run' + AND pstampJob.state = 'run' + AND pstampJob.parent_id + GROUP BY parent_id + ) as childJobs + ON childJobs.myParent = parentJob.job_id + WHERE runningChildren IS NULL +) AS theQuery + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingreq.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingreq.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_pendingreq.sql (revision 41104) @@ -0,0 +1,12 @@ +-- postage stamp requests ready for download and parsing +SELECT + pstampRequest.*, + pstampDataStore.outProduct AS ds_outProduct, + IFNULL(pstampDataStore.need_magic, 1) AS need_magic, + IFNULL(Label.priority, 10000) AS priority +FROM pstampRequest + LEFT JOIN pstampDataStore USING(ds_id) + LEFT JOIN Label ON pstampRequest.label = Label.label +WHERE pstampRequest.state = 'new' + AND pstampRequest.fault = 0 + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_project.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_project.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_project.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * FROM pstampProject Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertdependent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertdependent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertdependent.sql (revision 41104) @@ -0,0 +1,9 @@ +UPDATE pstampDependent + JOIN pstampJob USING(dep_id) + JOIN pstampRequest USING(req_id) +SET pstampDependent.fault = 0 +-- fault count hook %s +WHERE pstampRequest.state ='run' + AND pstampJob.state ='run' + AND pstampDependent.state = 'new' + AND (pstampDependent.fault > 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertjob.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertjob.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertjob.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE pstampJob + JOIN pstampRequest USING(req_id) +SET pstampJob.fault = 0 + -- clear fault count clause goes here %s +WHERE pstampRequest.state = 'run' + AND pstampJob.state = 'run' + -- fault clause goes here %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertreq.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertreq.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertreq.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE pstampRequest + SET fault = 0 +WHERE pstampRequest.state != 'stop' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertreq_deletejobs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertreq_deletejobs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_revertreq_deletejobs.sql (revision 41104) @@ -0,0 +1,6 @@ +-- delete jobs that got added by an incomplete pstamp parser run +DELETE pstampJob +FROM pstampJob + JOIN pstampRequest USING(req_id) +WHERE pstampRequest.state = 'new' + AND pstampRequest.fault > 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_stopdependentjob.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_stopdependentjob.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_stopdependentjob.sql (revision 41104) @@ -0,0 +1,10 @@ +-- stop jobs which have a faulted dependent +UPDATE pstampDependent + JOIN pstampJob USING(dep_id) + JOIN pstampRequest USING(req_id) +SET pstampJob.state = 'stop', + pstampJob.fault = %d, + pstampDependent.fault = %d, + pstampDependent.state = '%s' +WHERE pstampJob.state = 'run' + AND pstampDependent.state = 'new' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_updatejob.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_updatejob.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pstamptool_updatejob.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE pstampJob + JOIN pstampRequest USING(req_id) + LEFT JOIN pstampDependent USING(dep_id) +SET Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_definerun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_definerun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_definerun.sql (revision 41104) @@ -0,0 +1,74 @@ +-- Get runs to publish +SELECT DISTINCT + client_id, + stage_id, + src_label, + output_format +FROM ( + -- Get diffs to publish + SELECT DISTINCT + client_id, + diff_id AS stage_id, + diffRun.label AS src_label, + output_format + FROM publishClient + JOIN diffRun + JOIN diffInputSkyfile USING(diff_id) + JOIN diffSkyfile using(diff_id) + JOIN warpRun ON warpRun.warp_id = diffInputSkyfile.warp1 -- Only JOINing input, not reference! + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun ON magicDSRun.stage = 'diff' AND magicDSRun.stage_id = diff_id + WHERE publishClient.stage = 'diff' + AND publishClient.active = 1 + AND diffRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND ( diffRun.diff_mode = 4 + OR (publishClient.magicked AND diffRun.magicked != 0) + OR (publishClient.magicked = 0 AND + (diffRun.magicked = 0 OR (diffRun.magicked != 0 AND magic_ds_id IS NOT NULL)) + ) + ) + AND diffSkyfile.quality = 0 + -- WHERE hook %s + UNION + -- Get cameras to publish + SELECT + client_id, + cam_id AS stage_id, + camRun.label AS src_label, + output_format + FROM publishClient + JOIN camRun + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE publishClient.stage = 'camera' + AND publishClient.active = 1 + AND camRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND (camRun.magicked != 0 OR publishClient.magicked = 0) + -- WHERE hook %s + UNION + -- Get diffphots to publish + SELECT DISTINCT + client_id, + diff_phot_id AS stage_id, + diffPhotRun.label AS src_label, + output_format + FROM publishClient + JOIN diffPhotRun + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + JOIN warpRun ON warpRun.warp_id = diffInputSkyfile.warp1 -- Only JOINing input, not reference! + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE publishClient.stage = 'diffphot' + AND publishClient.active = 1 + AND diffPhotRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND (diffPhotRun.magicked != 0 OR diffRun.diff_mode = 4 OR publishClient.magicked = 0) + -- WHERE hook %s + ) AS publishToDo +-- Only get stuff that hasn't been published +LEFT JOIN publishRun USING(client_id, stage_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_pending.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_pending.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_pending.sql (revision 41104) @@ -0,0 +1,87 @@ +SELECT + publishToDo.* +FROM ( + -- Get the diffs + -- The following is only appropriate for a diff where warp1 is set; otherwise it's more difficult to get the camera name + SELECT DISTINCT + publishRun.pub_id, + publishClient.product, + publishClient.stage, + publishClient.workdir, + publishClient.output_format, + publishClient.magicked AS need_magic, + diffRun.diff_id AS stage_id, + rawExp.camera, + rawExp.exp_id + FROM publishRun + JOIN publishClient USING(client_id) + JOIN diffRun + ON diffRun.diff_id = publishRun.stage_id + JOIN diffInputSkyfile USING(diff_id) + -- Need to do something fancy here to get the camera name for a stack + LEFT JOIN warpRun ON warpRun.warp_id = diffInputSkyfile.warp1 + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE publishClient.stage = 'diff' + AND publishClient.active = 1 + AND publishRun.state = 'new' + AND diffRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND (diffRun.magicked != 0 OR diffRun.diff_mode = 4 OR publishClient.magicked = 0) + -- WHERE hook %s + UNION + SELECT + publishRun.pub_id, + publishClient.product, + publishClient.stage, + publishClient.workdir, + publishClient.output_format, + publishClient.magicked AS need_magic, + camRun.cam_id AS stage_id, + rawExp.camera, + rawExp.exp_id + FROM publishRun + JOIN publishClient USING(client_id) + JOIN camRun + ON camRun.cam_id = publishRun.stage_id + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE publishClient.stage = 'camera' + AND publishClient.active = 1 + AND publishRun.state ='new' + AND camRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND (camRun.magicked != 0 OR publishClient.magicked = 0) + -- WHERE hook %s + UNION + SELECT DISTINCT + publishRun.pub_id, + publishClient.product, + publishClient.stage, + publishClient.workdir, + publishClient.output_format, + publishClient.magicked AS need_magic, + diffPhotRun.diff_phot_id AS stage_id, + rawExp.camera, + rawExp.exp_id + FROM publishRun + JOIN publishClient USING(client_id) + JOIN diffPhotRun + ON diffPhotRun.diff_phot_id = publishRun.stage_id + JOIN diffRun USING(diff_id) + JOIN diffInputSkyfile USING(diff_id) + -- Need to do something fancy here to get the camera name for a stack + LEFT JOIN warpRun ON warpRun.warp_id = diffInputSkyfile.warp1 + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE publishClient.stage = 'diffphot' + AND publishClient.active = 1 + AND publishRun.state = 'new' + AND diffPhotRun.state IN ('full', 'cleaned', 'goto_cleaned') + AND (diffPhotRun.magicked != 0 OR diffRun.diff_mode = 4 OR publishClient.magicked = 0) + -- WHERE hook %s +) AS publishToDo +LEFT JOIN publishDone USING(pub_id) +WHERE publishDone.pub_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_revert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_revert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pubtool_revert.sql (revision 41104) @@ -0,0 +1,7 @@ +DELETE FROM publishDone +USING publishDone, publishRun, publishClient +WHERE publishDone.pub_id = publishRun.pub_id + AND publishRun.state = 'new' + AND publishRun.client_id = publishClient.client_id + AND publishClient.active = 1 + AND publishDone.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_create_tables.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_create_tables.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_create_tables.sql (revision 41104) @@ -0,0 +1,2430 @@ +CREATE TABLE dbversion ( + schema_version VARCHAR(64), + updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pzDataStore ( + camera VARCHAR(64), + telescope VARCHAR(64), + uri VARCHAR(255), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + use_compress SMALLINT, + PRIMARY KEY(camera, telescope) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE summitExp ( + summit_id BIGINT NOT NULL AUTO_INCREMENT, + exp_name VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + dateobs DATETIME, + exp_type VARCHAR(64), + uri VARCHAR(255), + imfiles INT, + fault SMALLINT NOT NULL, + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY(summit_id), + KEY(exp_name, camera, telescope), + KEY(fault) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE summitImfile ( + summit_id BIGINT, + exp_name VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + file_id VARCHAR(64), + bytes INT, + md5sum VARCHAR(32), + class VARCHAR(64), + class_id VARCHAR(64), + uri VARCHAR(255), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY(summit_id, class, class_id), + KEY(exp_name, camera, telescope, class, class_id), + KEY(file_id), + FOREIGN KEY(summit_id) REFERENCES summitExp(summit_id), + FOREIGN KEY(exp_name, camera, telescope) REFERENCES summitExp(exp_name, camera, telescope) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pzDownloadExp ( + summit_id BIGINT, + exp_name VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + state VARCHAR(64), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY(summit_id), + KEY(exp_name, camera, telescope), + KEY(state), + FOREIGN KEY(summit_id) REFERENCES summitExp(summit_id), + FOREIGN KEY(exp_name, camera, telescope) REFERENCES summitExp(exp_name, camera, telescope) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pzDownloadImfile ( + summit_id BIGINT, + exp_name VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + class VARCHAR(64), + class_id VARCHAR(64), + uri VARCHAR(255), + fault SMALLINT NOT NULL, + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + hostname VARCHAR(64), + bytes INT, + md5sum VARCHAR(32), + PRIMARY KEY(summit_id, class, class_id), + KEY(exp_name, camera, telescope, class, class_id), + KEY(fault), + FOREIGN KEY(summit_id) REFERENCES pzDownloadExp(summit_id), + FOREIGN KEY(summit_id, class, class_id) REFERENCES summitImfile(summit_id, class, class_id), + FOREIGN KEY (exp_name, camera, telescope) REFERENCES pzDownloadExp(exp_name, camera, telescope), + FOREIGN KEY(exp_name, camera, telescope, class, class_id) REFERENCES summitImfile(exp_name, camera, telescope, class, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE newExp ( + exp_id BIGINT AUTO_INCREMENT, + summit_id BIGINT, + tmp_exp_name VARCHAR(64), + tmp_camera VARCHAR(64), + tmp_telescope VARCHAR(64), + state VARCHAR(64), + workdir VARCHAR(255), + workdir_state VARCHAR(64), + reduction VARCHAR(64), + dvodb VARCHAR(255), + tess_id VARCHAR(64), + end_stage VARCHAR(64), + label VARCHAR(64), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY(exp_id), + KEY(exp_id), + KEY(summit_id), + KEY(tmp_exp_name), + KEY(tmp_camera), + KEY(tmp_telescope), + KEY(state), + KEY(workdir_state), + KEY(end_stage), + KEY(label) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE newImfile ( + exp_id BIGINT, + tmp_class_id VARCHAR(64), + uri VARCHAR(255), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + bytes INT, + md5sum VARCHAR(32), + PRIMARY KEY(exp_id, tmp_class_id), + FOREIGN KEY(exp_id) REFERENCES newExp(exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE rawExp ( + exp_id BIGINT, + exp_name VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + dateobs DATETIME, + exp_tag VARCHAR(255), + exp_type VARCHAR(64), + filelevel VARCHAR(64), + workdir VARCHAR(255), + state VARCHAR(64), + reduction VARCHAR(64), + dvodb VARCHAR(255), + tess_id VARCHAR(64), + end_stage VARCHAR(64), + filter VARCHAR(64), + comment VARCHAR(80), + obs_mode VARCHAR(64), + obs_group VARCHAR(64), + airmass FLOAT, + ra DOUBLE, + decl DOUBLE, + exp_time FLOAT, + sat_pixel_frac FLOAT, + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + alt DOUBLE, + az DOUBLE, + ccd_temp FLOAT, + posang DOUBLE, + m1_x FLOAT, + m1_y FLOAT, + m1_z FLOAT, + m1_tip FLOAT, + m1_tilt FLOAT, + m2_x FLOAT, + m2_y FLOAT, + m2_z FLOAT, + m2_tip FLOAT, + m2_tilt FLOAT, + env_temperature FLOAT, + env_humidity FLOAT, + env_wind_speed FLOAT, + env_wind_dir FLOAT, + teltemp_m1 FLOAT, + teltemp_m1cell FLOAT, + teltemp_m2 FLOAT, + teltemp_spider FLOAT, + teltemp_truss FLOAT, + teltemp_extra FLOAT, + pon_time FLOAT, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + object VARCHAR(64), + sun_angle FLOAT, + sun_alt FLOAT, + moon_angle FLOAT, + moon_alt FLOAT, + moon_phase FLOAT, + hostname VARCHAR(64), + fault SMALLINT NOT NULL, + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + magicked BIGINT, + PRIMARY KEY(exp_id), + KEY(end_stage), + KEY(fault), + KEY(state), + KEY(exp_name), + FOREIGN KEY(exp_id) REFERENCES newExp(exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE rawImfile ( + exp_id BIGINT, + exp_name VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + dateobs DATETIME, + tmp_class_id VARCHAR(64), + class_id VARCHAR(64), + uri VARCHAR(255), + data_state VARCHAR(64), + exp_type VARCHAR(64), + filelevel VARCHAR(64), + filter VARCHAR(64), + comment VARCHAR(80), + obs_mode VARCHAR(64), + obs_group VARCHAR(64), + airmass FLOAT, + ra DOUBLE, + decl DOUBLE, + exp_time FLOAT, + sat_pixel_frac FLOAT, + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + alt DOUBLE, + az DOUBLE, + ccd_temp FLOAT, + posang DOUBLE, + m1_x FLOAT, + m1_y FLOAT, + m1_z FLOAT, + m1_tip FLOAT, + m1_tilt FLOAT, + m2_x FLOAT, + m2_y FLOAT, + m2_z FLOAT, + m2_tip FLOAT, + m2_tilt FLOAT, + env_temperature FLOAT, + env_humidity FLOAT, + env_wind_speed FLOAT, + env_wind_dir FLOAT, + teltemp_m1 FLOAT, + teltemp_m1cell FLOAT, + teltemp_m2 FLOAT, + teltemp_spider FLOAT, + teltemp_truss FLOAT, + teltemp_extra FLOAT, + pon_time FLOAT, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + object VARCHAR(64), + sun_angle FLOAT, + sun_alt FLOAT, + moon_angle FLOAT, + moon_alt FLOAT, + moon_phase FLOAT, + ignored TINYINT DEFAULT 0, + hostname VARCHAR(64), + quality SMALLINT NOT NULL DEFAULT 0, + fault SMALLINT NOT NULL, + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + raw_image_id BIGINT AUTO_INCREMENT, + magicked BIGINT, + bytes INT, + md5sum VARCHAR(32), + burntool_state SMALLINT, + video_cells TINYINT DEFAULT 0, + PRIMARY KEY(exp_id, class_id), + KEY(tmp_class_id), + KEY(fault), + KEY(raw_image_id), + KEY(quality), + KEY(dateobs), + UNIQUE KEY(exp_id, tmp_class_id), + FOREIGN KEY(exp_id, tmp_class_id) REFERENCES newImfile(exp_id, tmp_class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE chipRun ( + chip_id BIGINT AUTO_INCREMENT, + exp_id BIGINT, + state VARCHAR(64), + workdir VARCHAR(255), + workdir_state VARCHAR(64), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + reduction VARCHAR(64), + expgroup VARCHAR(64), + dvodb VARCHAR(255), + tess_id VARCHAR(64), + end_stage VARCHAR(64), + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_npix FLOAT, + maskfrac_static FLOAT, + maskfrac_dynamic FLOAT, + maskfrac_magic FLOAT, + maskfrac_advisory FLOAT, + update_mode SMALLINT DEFAULT 0, + note VARCHAR(255), + PRIMARY KEY(chip_id), + KEY(chip_id), KEY(exp_id), + KEY(state), + KEY(workdir_state), + KEY(label), + KEY(data_group), + KEY(dist_group), + KEY(expgroup), + KEY(end_stage), + INDEX(chip_id, exp_id), + FOREIGN KEY(exp_id) REFERENCES rawExp(exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE chipImfile ( + chip_id BIGINT, + class_id VARCHAR(64), + chip_imfile_id BIGINT AUTO_INCREMENT, + PRIMARY KEY(chip_id, class_id), + KEY(chip_imfile_id), + FOREIGN KEY(chip_id) REFERENCES chipRun(chip_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE chipProcessedImfile ( + chip_id BIGINT, + exp_id BIGINT, + class_id VARCHAR(64), + data_state VARCHAR(64), + uri VARCHAR(255), + bg FLOAT, + bg_stdev FLOAT, + bg_mean_stdev FLOAT, + bias FLOAT, + bias_stdev FLOAT, + fringe_0 FLOAT, + fringe_1 FLOAT, + fringe_2 FLOAT, + ap_resid FLOAT, + ap_resid_stdev FLOAT, + fwhm_major FLOAT, + fwhm_major_lq FLOAT, + fwhm_major_uq FLOAT, + fwhm_minor FLOAT, + fwhm_minor_lq FLOAT, + fwhm_minor_uq FLOAT, + iq_fwhm_major FLOAT, + iq_fwhm_major_err FLOAT, + iq_fwhm_minor FLOAT, + iq_fwhm_minor_err FLOAT, + iq_m2 FLOAT, + iq_m2_err FLOAT, + iq_m2_lq FLOAT, + iq_m2_uq FLOAT, + iq_m2c FLOAT, + iq_m2c_err FLOAT, + iq_m2c_lq FLOAT, + iq_m2c_uq FLOAT, + iq_m2s FLOAT, + iq_m2s_err FLOAT, + iq_m2s_lq FLOAT, + iq_m2s_uq FLOAT, + iq_m3 FLOAT, + iq_m3_err FLOAT, + iq_m3_lq FLOAT, + iq_m3_uq FLOAT, + iq_m4 FLOAT, + iq_m4_err FLOAT, + iq_m4_lq FLOAT, + iq_m4_uq FLOAT, + dtime_detrend FLOAT, + dtime_photom FLOAT, + dtime_total FLOAT, + dtime_script FLOAT, + hostname VARCHAR(64), + n_stars INT, + n_psfstars INT, + n_iqstars INT, + n_extended INT, + n_cr INT, + path_base VARCHAR(255), + quality SMALLINT NOT NULL DEFAULT 0, + fault SMALLINT NOT NULL, + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_npix INT, + maskfrac_static FLOAT, + maskfrac_dynamic FLOAT, + maskfrac_magic FLOAT, + maskfrac_advisory FLOAT, + deteff_magref FLOAT, + PRIMARY KEY(chip_id, exp_id, class_id), + KEY(data_state), + KEY(fault), + KEY(quality), + KEY(data_state), + FOREIGN KEY(chip_id, exp_id) REFERENCES chipRun(chip_id, exp_id), + FOREIGN KEY(exp_id, class_id) REFERENCES rawImfile(exp_id, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE chipMask ( + label VARCHAR(64), + PRIMARY KEY(label) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE camRun ( + cam_id BIGINT AUTO_INCREMENT, + chip_id BIGINT, + state VARCHAR(64), + workdir VARCHAR(255), + workdir_state VARCHAR(64), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + reduction VARCHAR(64), + expgroup VARCHAR(64), + dvodb VARCHAR(255), + tess_id VARCHAR(64), + end_stage VARCHAR(64), + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_ref_npix FLOAT, + maskfrac_ref_static FLOAT, + maskfrac_ref_dynamic FLOAT, + maskfrac_ref_magic FLOAT, + maskfrac_ref_advisory FLOAT, + maskfrac_max_npix FLOAT, + maskfrac_max_static FLOAT, + maskfrac_max_dynamic FLOAT, + maskfrac_max_magic FLOAT, + maskfrac_max_advisory FLOAT, + note VARCHAR(255), + PRIMARY KEY(cam_id), + KEY(cam_id), + KEY(chip_id), + KEY(state), + KEY(workdir_state), + KEY(label), + KEY(data_group), + KEY(dist_group), + KEY(expgroup), + KEY(end_stage), + INDEX(cam_id, chip_id), + FOREIGN KEY(chip_id) REFERENCES chipRun(chip_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE camProcessedExp ( + cam_id BIGINT, + uri VARCHAR(255), + bg FLOAT, + bg_stdev FLOAT, + bg_mean_stdev FLOAT, + bias FLOAT, + bias_stdev FLOAT, + fringe_0 FLOAT, + fringe_1 FLOAT, + fringe_2 FLOAT, + sigma_ra FLOAT, + sigma_dec FLOAT, + ap_resid FLOAT, + ap_resid_stdev FLOAT, + zpt_obs FLOAT, + zpt_stdev FLOAT, + zpt_lq FLOAT, + zpt_uq FLOAT, + fwhm_major FLOAT, + fwhm_major_lq FLOAT, + fwhm_major_uq FLOAT, + fwhm_minor FLOAT, + fwhm_minor_lq FLOAT, + fwhm_minor_uq FLOAT, + iq_fwhm_major FLOAT, + iq_fwhm_major_err FLOAT, + iq_fwhm_minor FLOAT, + iq_fwhm_minor_err FLOAT, + iq_m2 FLOAT, + iq_m2_err FLOAT, + iq_m2_lq FLOAT, + iq_m2_uq FLOAT, + iq_m2c FLOAT, + iq_m2c_err FLOAT, + iq_m2c_lq FLOAT, + iq_m2c_uq FLOAT, + iq_m2s FLOAT, + iq_m2s_err FLOAT, + iq_m2s_lq FLOAT, + iq_m2s_uq FLOAT, + iq_m3 FLOAT, + iq_m3_err FLOAT, + iq_m3_lq FLOAT, + iq_m3_uq FLOAT, + iq_m4 FLOAT, + iq_m4_err FLOAT, + iq_m4_lq FLOAT, + iq_m4_uq FLOAT, + dtime_script FLOAT, + dtime_astrom FLOAT, + dtime_addstar FLOAT, + hostname VARCHAR(64), + n_stars INT, + n_psfstars INT, + n_iqstars INT, + n_extended INT, + n_cr INT, + n_astrom INT, + path_base VARCHAR(255), + fault SMALLINT NOT NULL, + software_ver VARCHAR(16), + maskfrac_ref_npix INT, + maskfrac_ref_static FLOAT, + maskfrac_ref_dynamic FLOAT, + maskfrac_ref_magic FLOAT, + maskfrac_ref_advisory FLOAT, + maskfrac_max_npix INT, + maskfrac_max_static FLOAT, + maskfrac_max_dynamic FLOAT, + maskfrac_max_magic FLOAT, + maskfrac_max_advisory FLOAT, + deteff FLOAT, + deteff_err FLOAT, + deteff_lq FLOAT, + deteff_uq FLOAT, + quality SMALLINT NOT NULL DEFAULT 0, + background_model SMALLINT, + astrom_chips BIGINT, + ast_r0 FLOAT, + ast_d0 FLOAT, + ast_t0 FLOAT, + ast_s0 FLOAT, + ast_rs FLOAT, + ast_ds FLOAT, + PRIMARY KEY(cam_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(cam_id) REFERENCES camRun(cam_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE camMask ( + label VARCHAR(64), + PRIMARY KEY(label) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE addRun ( + add_id BIGINT AUTO_INCREMENT, + stage VARCHAR(64), + stage_id BIGINT, + stage_extra1 INT, + state VARCHAR(64), + workdir VARCHAR(255), + workdir_state VARCHAR(64), + reduction VARCHAR(64), + label VARCHAR(64), + data_group VARCHAR(64), + dvodb VARCHAR(255), + note VARCHAR(255), + image_only TINYINT, + minidvodb TINYINT, + minidvodb_group VARCHAR(64), + minidvodb_name VARCHAR(64), + PRIMARY KEY(add_id), + KEY(add_id), + KEY(stage_id), + KEY(stage), + KEY(state), + KEY(workdir_state), + KEY(label), + KEY(data_group) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE addProcessedExp ( + add_id BIGINT AUTO_INCREMENT, + dtime_addstar FLOAT, + path_base VARCHAR(255), + dvodb_path VARCHAR(255), + fault SMALLINT NOT NULL, + PRIMARY KEY(add_id), + FOREIGN KEY(add_id) REFERENCES addRun(add_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE addMask ( + label VARCHAR(64), + PRIMARY KEY(label) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE fakeRun ( + fake_id BIGINT AUTO_INCREMENT, + cam_id BIGINT, + state VARCHAR(64), + workdir VARCHAR(255), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + reduction VARCHAR(64), + expgroup VARCHAR(64), + dvodb VARCHAR(255), + tess_id VARCHAR(64), + end_stage VARCHAR(64), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + note VARCHAR(255), + PRIMARY KEY(fake_id), + KEY(cam_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(dist_group), + KEY(expgroup), + KEY(end_stage), + INDEX(fake_id, cam_id), + FOREIGN KEY(cam_id) REFERENCES camRun(cam_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE fakeProcessedImfile ( + fake_id BIGINT AUTO_INCREMENT, + exp_id BIGINT, + class_id VARCHAR(64), + uri VARCHAR(255), + dtime_fake FLOAT, + dtime_script FLOAT, + hostname VARCHAR(64), + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY(fake_id, exp_id, class_id), + KEY(fault), + FOREIGN KEY(fake_id) REFERENCES fakeRun(fake_id), + FOREIGN KEY(exp_id, class_id) REFERENCES rawImfile(exp_id, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE fakeMask ( + label VARCHAR(64), + PRIMARY KEY(label) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detRun ( + det_id BIGINT AUTO_INCREMENT, + iteration INT, + det_type VARCHAR(64), + mode VARCHAR(64), + state VARCHAR(64), + filelevel VARCHAR(64), + workdir VARCHAR(255), + camera VARCHAR(64), + telescope VARCHAR(64), + exp_type VARCHAR(64), + reduction VARCHAR(64), + filter VARCHAR(64), + airmass_min FLOAT, + airmass_max FLOAT, + exp_time_min FLOAT, + exp_time_max FLOAT, + ccd_temp_min FLOAT, + ccd_temp_max FLOAT, + posang_min DOUBLE, + posang_max DOUBLE, + registered DATETIME, + time_begin DATETIME, + time_end DATETIME, + use_begin DATETIME, + use_end DATETIME, + solang_min FLOAT, + solang_max FLOAT, + label VARCHAR(64), + ref_det_id BIGINT, + ref_iter INT, + PRIMARY KEY(det_id), + KEY(det_id), + KEY(iteration), + KEY(det_type), + KEY(mode), + KEY(state), + KEY(label), + INDEX(det_id, iteration) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detInputExp ( + det_id BIGINT, + iteration INT, + exp_id BIGINT, + include TINYINT, + PRIMARY KEY(det_id, iteration, exp_id), + INDEX(det_id, exp_id), + INDEX(det_id, iteration), + FOREIGN KEY(det_id) REFERENCES detRun(det_id), + FOREIGN KEY(exp_id) REFERENCES rawExp(exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detProcessedImfile ( + det_id BIGINT, + exp_id BIGINT, + class_id VARCHAR(64), + uri VARCHAR(255), + recipe VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + fringe_0 DOUBLE, + fringe_1 DOUBLE, + fringe_2 DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, exp_id, class_id), + KEY(fault), + INDEX(det_id, class_id), + INDEX(det_id, exp_id), + FOREIGN KEY(det_id, exp_id) REFERENCES detInputExp(det_id, exp_id), + FOREIGN KEY(exp_id, class_id) REFERENCES rawImfile(exp_id, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detProcessedExp ( + det_id BIGINT, + exp_id BIGINT, + recipe VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + fringe_0 DOUBLE, + fringe_1 DOUBLE, + fringe_2 DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, exp_id), + KEY(fault), + FOREIGN KEY(det_id, exp_id) REFERENCES detInputExp(det_id, exp_id), + FOREIGN KEY(det_id, exp_id) REFERENCES detProcessedImfile(det_id, exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detStackedImfile ( + det_id BIGINT, + iteration INT, + class_id VARCHAR(64), + uri VARCHAR(255), + recipe VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration, class_id), + KEY(fault), + FOREIGN KEY(det_id, iteration) REFERENCES detInputExp(det_id, iteration), + FOREIGN KEY(det_id, class_id) REFERENCES detProcessedImfile(det_id, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detNormalizedStatImfile ( + det_id BIGINT, + iteration INT, + class_id VARCHAR(64), + norm FLOAT, + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration, class_id), + KEY(fault), + FOREIGN KEY(det_id, iteration) REFERENCES detInputExp(det_id, iteration), + FOREIGN KEY(det_id, iteration, class_id) REFERENCES detStackedImfile(det_id, iteration, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detNormalizedImfile ( + det_id BIGINT, + iteration INT, + class_id VARCHAR(64), + uri VARCHAR(255), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration, class_id), + KEY(fault), + INDEX(det_id, iteration), + FOREIGN KEY(det_id) REFERENCES detInputExp(det_id), + FOREIGN KEY(det_id, iteration, class_id) REFERENCES detNormalizedStatImfile(det_id, iteration, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detNormalizedExp ( + det_id BIGINT, + iteration INT, + recipe VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration), + KEY(fault), + FOREIGN KEY(det_id, iteration) REFERENCES detInputExp(det_id, iteration), + FOREIGN KEY(det_id, iteration) REFERENCES detNormalizedImfile(det_id, iteration) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detResidImfile ( + det_id BIGINT, + iteration INT, + ref_det_id BIGINT, + ref_iter INT, + exp_id BIGINT, + class_id VARCHAR(64), + uri VARCHAR(255), + recipe VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + bg_skewness DOUBLE, + bg_kurtosis DOUBLE, + bin_stdev DOUBLE, + fringe_0 DOUBLE, + fringe_1 DOUBLE, + fringe_2 DOUBLE, + fringe_resid_0 DOUBLE, + fringe_resid_1 DOUBLE, + fringe_resid_2 DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration, exp_id, class_id), + KEY(fault), + INDEX(det_id, iteration, exp_id), + FOREIGN KEY (det_id, iteration, exp_id) REFERENCES detInputExp(det_id, iteration, exp_id), + FOREIGN KEY (det_id, exp_id, class_id) REFERENCES detProcessedImfile(det_id, exp_id, class_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detResidExp ( + det_id BIGINT, + iteration INT, + exp_id BIGINT, + recipe VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + bg_skewness DOUBLE, + bg_kurtosis DOUBLE, + bin_stdev DOUBLE, + fringe_0 DOUBLE, + fringe_1 DOUBLE, + fringe_2 DOUBLE, + fringe_resid_0 DOUBLE, + fringe_resid_1 DOUBLE, + fringe_resid_2 DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + accept TINYINT, + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration, exp_id), + KEY(fault), + INDEX(det_id, iteration), + FOREIGN KEY(det_id, iteration, exp_id) REFERENCES detInputExp(det_id, iteration, exp_id), + FOREIGN KEY(det_id, iteration, exp_id) REFERENCES detResidImfile(det_id, iteration, exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detRunSummary ( + det_id BIGINT, + iteration INT, + data_state VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + accept TINYINT, + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration), + KEY(fault), + FOREIGN KEY(det_id, iteration) REFERENCES detInputExp(det_id, iteration), + FOREIGN KEY(det_id, iteration) REFERENCES detResidExp(det_id, iteration) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE detRegisteredImfile ( + det_id BIGINT, + iteration INT, + class_id VARCHAR(64), + uri VARCHAR(255), + bg DOUBLE, + bg_stdev DOUBLE, + bg_mean_stdev DOUBLE, + user_1 DOUBLE, + user_2 DOUBLE, + user_3 DOUBLE, + user_4 DOUBLE, + user_5 DOUBLE, + path_base VARCHAR(255), + data_state VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(det_id, iteration, class_id), + KEY(fault), + FOREIGN KEY(det_id, iteration) REFERENCES detRun(det_id, iteration) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE warpRun ( + warp_id BIGINT AUTO_INCREMENT, + fake_id BIGINT, + mode VARCHAR(64), + state VARCHAR(64), + workdir VARCHAR(255), + workdir_state VARCHAR(64), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + dvodb VARCHAR(255), + tess_id VARCHAR(64), + reduction VARCHAR(64), + end_stage VARCHAR(64), + registered DATETIME, + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_npix FLOAT, + maskfrac_static FLOAT, + maskfrac_dynamic FLOAT, + maskfrac_magic FLOAT, + maskfrac_advisory FLOAT, + note VARCHAR(255), + PRIMARY KEY(warp_id), + KEY(warp_id), + KEY(fake_id), + KEY(mode), + KEY(state), + KEY(workdir_state), + KEY(label), + KEY(data_group), + KEY(dist_group), + KEY(end_stage), + INDEX(warp_id, fake_id), + FOREIGN KEY(fake_id) REFERENCES fakeRun(fake_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE warpSkyCellMap ( + warp_id BIGINT, + skycell_id VARCHAR(64), + tess_id VARCHAR(64), + class_id VARCHAR(64), + fault SMALLINT, + PRIMARY KEY(warp_id, skycell_id, tess_id, class_id), + KEY(fault), + FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE warpImfile ( + warp_id BIGINT, + skycell_id VARCHAR(64), + warp_skyfile_id BIGINT AUTO_INCREMENT, + PRIMARY KEY(warp_id, skycell_id), + KEY(warp_skyfile_id), + FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + +CREATE TABLE warpSkyfile ( + warp_id BIGINT, + skycell_id VARCHAR(64), + tess_id VARCHAR(64), + uri VARCHAR(255), + path_base VARCHAR(255), + data_state VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + dtime_warp FLOAT, + dtime_script FLOAT, + hostname VARCHAR(64), + good_frac FLOAT, + xmin INT, + xmax INT, + ymin INT, + ymax INT, + quality SMALLINT NOT NULL DEFAULT 0, + fault SMALLINT, + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_npix INT, + maskfrac_static FLOAT, + maskfrac_dynamic FLOAT, + maskfrac_magic FLOAT, + maskfrac_advisory FLOAT, + background_model SMALLINT, + PRIMARY KEY(warp_id, skycell_id, tess_id), + KEY(good_frac), + KEY(fault), + KEY(quality), + KEY(data_state), + FOREIGN KEY(warp_id, skycell_id, tess_id) REFERENCES warpSkyCellMap(warp_id, skycell_id, tess_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE warpMask ( + label VARCHAR(64), + PRIMARY KEY(label) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE warpSummary ( + warp_id BIGINT, + projection_cell VARCHAR(64) NOT NULL, + path_base VARCHAR(255) NOT NULL, + PRIMARY KEY(warp_id, projection_cell), + FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE stackRun ( + stack_id BIGINT AUTO_INCREMENT, + state VARCHAR(64), + workdir VARCHAR(255), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + reduction VARCHAR(64), + dvodb VARCHAR(255), + registered DATETIME, + skycell_id VARCHAR(64), + tess_id VARCHAR(64), + filter VARCHAR(64), + software_ver VARCHAR(16), + note VARCHAR(255), + PRIMARY KEY(stack_id), + KEY(stack_id), + KEY(state), + KEY(skycell_id), + KEY(tess_id), + KEY(label), + KEY(data_group), + KEY(dist_group) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE stackInputSkyfile ( + stack_id BIGINT, + warp_id BIGINT, + PRIMARY KEY(stack_id, warp_id), + FOREIGN KEY(stack_id) REFERENCES stackRun(stack_id), + FOREIGN KEY(warp_id) REFERENCES warpSkyfile(warp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE stackSumSkyfile ( + stack_id BIGINT, + uri VARCHAR(255), + path_base VARCHAR(255), + bg DOUBLE, + bg_stdev DOUBLE, + dtime_stack FLOAT, + dtime_match_mean FLOAT, + dtime_match_stdev FLOAT, + dtime_convolve FLOAT, + dtime_initial FLOAT, + dtime_reject FLOAT, + dtime_final FLOAT, + dtime_phot FLOAT, + dtime_script FLOAT, + match_mean FLOAT, + match_stdev FLOAT, + match_rms FLOAT, + stamps_mean FLOAT, + stamps_stdev FLOAT, + stamps_min INT, + reject_images INT, + reject_pix_mean FLOAT, + reject_pix_stdev FLOAT, + sources INT, + hostname VARCHAR(64), + good_frac FLOAT, + mjd_obs DOUBLE, + fault SMALLINT, + software_ver VARCHAR(16), + background_model SMALLINT, + quality SMALLINT NOT NULL DEFAULT 0, + PRIMARY KEY(stack_id), + KEY(dtime_stack), + KEY(good_frac), + KEY(fault), + KEY(quality), + FOREIGN KEY(stack_id) REFERENCES stackRun(stack_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE stackAssociation ( + sass_id BIGINT AUTO_INCREMENT, + data_group VARCHAR(64) NOT NULL, + projection_cell VARCHAR(64) NOT NULL, + tess_id VARCHAR(64) NOT NULL, + filter VARCHAR(64) NOT NULL, + PRIMARY KEY(sass_id), + KEY(data_group), + KEY(projection_cell), + KEY(tess_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE stackAssociationMap ( + sass_id BIGINT, + stack_id BIGINT, + PRIMARY KEY(sass_id, stack_id), + FOREIGN KEY(sass_id) REFERENCES stackAssociation(sass_id), + FOREIGN KEY(stack_id) REFERENCES stackRun(stack_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE stackSummary ( + sass_id BIGINT, + projection_cell VARCHAR(64) NOT NULL, + path_base VARCHAR(255) NOT NULL, + PRIMARY KEY(sass_id, projection_cell), + FOREIGN KEY(sass_id) REFERENCES stackAssociation(sass_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE diffRun ( + diff_id BIGINT AUTO_INCREMENT, + state VARCHAR(64), + workdir VARCHAR(255), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + reduction VARCHAR(64), + dvodb VARCHAR(255), + registered DATETIME, + tess_id VARCHAR(64), + bothways TINYINT DEFAULT 0, + exposure TINYINT DEFAULT 0, + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_npix FLOAT, + maskfrac_static FLOAT, + maskfrac_dynamic FLOAT, + maskfrac_magic FLOAT, + maskfrac_advisory FLOAT, + diff_mode SMALLINT NOT NULL, + note VARCHAR(255), + PRIMARY KEY(diff_id), + KEY(diff_id), + KEY(state), + KEY(tess_id), + KEY(label), + KEY(data_group), + KEY(dist_group) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE diffInputSkyfile ( + diff_id BIGINT, + skycell_id VARCHAR(64), + warp1 BIGINT, + stack1 BIGINT, + warp2 BIGINT, + stack2 BIGINT, + tess_id VARCHAR(64), + diff_skyfile_id BIGINT AUTO_INCREMENT, + PRIMARY KEY(diff_skyfile_id), + KEY(diff_id, skycell_id), + KEY(warp1), + KEY(warp2), + KEY(stack1), + KEY(stack2), + KEY(skycell_id), + KEY(tess_id), + FOREIGN KEY(diff_id) REFERENCES diffRun(diff_id), + FOREIGN KEY(warp1, skycell_id, tess_id) REFERENCES warpSkyfile(warp_id, skycell_id, tess_id), + FOREIGN KEY(warp2, skycell_id, tess_id) REFERENCES warpSkyfile(warp_id, skycell_id, tess_id), + FOREIGN KEY(stack1) REFERENCES stackSumSkyfile(stack_id), + FOREIGN KEY(stack2) REFERENCES stackSumSkyfile(stack_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE diffSkyfile ( + diff_id BIGINT, + skycell_id VARCHAR(64), + path_base VARCHAR(255), + data_state VARCHAR(64), + bg DOUBLE, + bg_stdev DOUBLE, + stamps_num INT, + stamps_mean FLOAT, + stamps_rms FLOAT, + norm FLOAT, + bg_diff FLOAT, + kernel_x FLOAT, + kernel_y FLOAT, + kernel_xx FLOAT, + kernel_xy FLOAT, + kernel_yy FLOAT, + deconv_max FLOAT, + sources INT, + dtime_diff FLOAT, + dtime_match FLOAT, + dtime_phot FLOAT, + dtime_script FLOAT, + hostname VARCHAR(64), + good_frac FLOAT, + quality SMALLINT NOT NULL DEFAULT 0, + fault SMALLINT, + magicked BIGINT, + software_ver VARCHAR(16), + maskfrac_npix INT, + maskfrac_static FLOAT, + maskfrac_dynamic FLOAT, + maskfrac_magic FLOAT, + maskfrac_advisory FLOAT, + PRIMARY KEY(diff_id, skycell_id), + KEY(good_frac), + KEY(fault), + KEY(quality), + KEY(data_state), + FOREIGN KEY(diff_id) REFERENCES diffRun(diff_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE diffSummary ( + diff_id BIGINT, + projection_cell VARCHAR(64) NOT NULL, + path_base VARCHAR(255) NOT NULL, + PRIMARY KEY(diff_id, projection_cell), + FOREIGN KEY(diff_id) REFERENCES diffRun(diff_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicRun ( + magic_id BIGINT AUTO_INCREMENT, + exp_id BIGINT, + diff_id BIGINT, + inverse TINYINT NOT NULL DEFAULT 0, + state VARCHAR(64), + workdir VARCHAR(255), + workdir_state VARCHAR(255), + label VARCHAR(64), + data_group VARCHAR(64), + dvodb VARCHAR(255), + registered DATETIME, + fault SMALLINT, + note VARCHAR(255), + PRIMARY KEY(magic_id), + KEY(magic_id), + KEY(state), + KEY(workdir_state), + KEY(label), + KEY(data_group), + KEY(fault), + FOREIGN KEY(exp_id) REFERENCES rawExp(exp_id), + FOREIGN KEY(diff_id) REFERENCES diffRun(diff_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicInputSkyfile ( + magic_id BIGINT, + node VARCHAR(64), + PRIMARY KEY(magic_id, node), + FOREIGN KEY(magic_id) REFERENCES magicRun(magic_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicTree ( + magic_id BIGINT, + node VARCHAR(64), + dep VARCHAR(64), + KEY(magic_id), + KEY(node), + KEY(dep), + INDEX(magic_id, node), + FOREIGN KEY(magic_id) REFERENCES magicRun(magic_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicNodeResult ( + magic_id BIGINT, + node VARCHAR(64), + path_base VARCHAR(255), + fault SMALLINT, + PRIMARY KEY(magic_id, node), + FOREIGN KEY(magic_id) REFERENCES magicRun(magic_id), + FOREIGN KEY(magic_id, node) REFERENCES magicTree(magic_id, node), + KEY(fault) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicMask ( + magic_id BIGINT, + uri VARCHAR(255), + path_base VARCHAR(255), + streaks INT, + fault SMALLINT, + PRIMARY KEY(magic_id), + FOREIGN KEY(magic_id) REFERENCES magicRun(magic_id), + KEY(fault) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicDSRun ( + magic_ds_id BIGINT AUTO_INCREMENT, + magic_id BIGINT, + inv_magic_id BIGINT, + state VARCHAR(64), + stage VARCHAR(64), + stage_id BIGINT, + cam_id BIGINT, + label VARCHAR(64), + data_group VARCHAR(64), + outroot VARCHAR(255), + recoveryroot VARCHAR(255), + re_place TINYINT, + remove TINYINT, + fault SMALLINT, + note VARCHAR(255), + PRIMARY KEY(magic_ds_id), + KEY(magic_ds_id), + KEY(state), + KEY(magic_id), + KEY(label), + KEY(fault), + KEY(stage), + KEY(stage_id), + FOREIGN KEY(magic_id) REFERENCES magicRun(magic_id), + FOREIGN KEY(inv_magic_id) REFERENCES magicRun(magic_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE magicDSFile ( + magic_ds_id BIGINT, + component VARCHAR(64), + backup_path_base VARCHAR(255), + recovery_path_base VARCHAR(255), + streak_frac FLOAT, + nondiff_frac FLOAT, + run_time FLOAT, + fault SMALLINT, + data_state VARCHAR(64), + PRIMARY KEY(magic_ds_id, component), + KEY(fault), + KEY(data_state), + FOREIGN KEY(magic_ds_id) REFERENCES magicDSRun(magic_ds_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE calDB ( + cal_id BIGINT AUTO_INCREMENT, + dvodb VARCHAR(64), + state VARCHAR(64), + PRIMARY KEY(cal_id), + KEY(cal_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE calRun ( + cal_id BIGINT AUTO_INCREMENT, + region VARCHAR(64), + last_step VARCHAR(64), + state VARCHAR(64), + PRIMARY KEY(cal_id), + KEY(cal_id), + KEY(last_step), + FOREIGN KEY(cal_id) REFERENCES calDB(cal_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE flatcorrRun ( + corr_id BIGINT AUTO_INCREMENT, + det_type VARCHAR(64), + dvodb VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + epoch TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + filter VARCHAR(64), + state VARCHAR(64), + make_corr TINYINT, + workdir VARCHAR(255), + label VARCHAR(64), + reduction VARCHAR(64), + region VARCHAR(64), + hostname VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(corr_id), + KEY(corr_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- these three tables link the flatcorrRun to the associated chip, camera, and addstar runs +CREATE TABLE flatcorrChipLink ( + corr_id BIGINT, + chip_id BIGINT, + include TINYINT, + PRIMARY KEY(corr_id, chip_id), + FOREIGN KEY(corr_id) REFERENCES flatcorrRun(corr_id), + FOREIGN KEY(chip_id) REFERENCES chipRun(chip_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE flatcorrCamLink ( + corr_id BIGINT, + chip_id BIGINT, + cam_id BIGINT, + include TINYINT, + PRIMARY KEY(corr_id, cam_id), + FOREIGN KEY(corr_id) REFERENCES flatcorrRun(corr_id), + FOREIGN KEY(chip_id) REFERENCES chipRun(chip_id), + FOREIGN KEY(cam_id) REFERENCES camRun(cam_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE flatcorrAddstarLink ( + corr_id BIGINT, + cam_id BIGINT, + add_id BIGINT, + include TINYINT, + PRIMARY KEY(corr_id, cam_id, add_id), + FOREIGN KEY (corr_id) REFERENCES flatcorrRun(corr_id), + FOREIGN KEY (cam_id) REFERENCES camRun(cam_id), + FOREIGN KEY (add_id) REFERENCES addRun(add_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampDataStore ( + ds_id BIGINT AUTO_INCREMENT, + state VARCHAR(64), + lastFileset VARCHAR(64), + timestamp DATETIME, + label VARCHAR(64), + outProduct VARCHAR(64) UNIQUE, + uri VARCHAR(255), + pollInterval INTEGER DEFAULT 60, + need_magic TINYINT, + PRIMARY KEY(ds_id), + KEY(ds_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampProject ( + proj_id BIGINT AUTO_INCREMENT, + name VARCHAR(64) UNIQUE, + state VARCHAR(64), + dbname VARCHAR(64), + dvodb VARCHAR(64), + camera VARCHAR(64), + telescope VARCHAR(64), + need_magic TINYINT, + PRIMARY KEY(proj_id), + KEY(proj_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampRequest ( + req_id BIGINT AUTO_INCREMENT, + ds_id BIGINT, + state VARCHAR(64), + name VARCHAR(64) UNIQUE, + reqType VARCHAR(16), + label VARCHAR(64), + outProduct VARCHAR(64), + uri VARCHAR(255), + outdir VARCHAR(255), + username VARCHAR(255), + proj_id BIGINT, + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + fault SMALLINT, + PRIMARY KEY(req_id), + KEY(req_id), + KEY(state), + KEY(fault), + KEY(label), + KEY(username) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampJob ( + job_id BIGINT AUTO_INCREMENT, + req_id BIGINT, + rownum VARCHAR(64), + state VARCHAR(64), + jobType VARCHAR(16), + fault SMALLINT, + exp_id BIGINT, + outputBase VARCHAR(255), + options BIGINT, + dep_id BIGINT, + fault_count INT, + parent_id BIGINT DEFAULT NULL, + is_parent TINYINT DEFAULT 0, + PRIMARY KEY(job_id, req_id), + KEY(job_id), + KEY(req_id), + KEY(state), + KEY(fault), + KEY(dep_id), + KEY(parent_id), + KEY(is_parent), + FOREIGN KEY(req_id) REFERENCES pstampRequest(req_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampDependent ( + dep_id BIGINT AUTO_INCREMENT, + state VARCHAR(64), + stage VARCHAR(64), + stage_id BIGINT, + component VARCHAR(64), + imagedb VARCHAR(64), + outdir VARCHAR(255), + rlabel VARCHAR(64), + need_magic TINYINT, + fault SMALLINT, + fault_count INT, + PRIMARY KEY(dep_id), + KEY(state), + KEY(stage), + KEY(stage_id), + KEY(component), + KEY(imagedb), + KEY(fault) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampFile ( + file_id BIGINT AUTO_INCREMENT, + job_id BIGINT NOT NULL, + path VARCHAR(255), + PRIMARY KEY(file_id), + KEY(job_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampUserDomain ( + domainName VARCHAR(64), + accessLevel INT, + defaultProduct VARCHAR(255), + defaultLabel VARCHAR(64), + PRIMARY KEY(domainName) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampUser ( + userName VARCHAR(64), + domainName VARCHAR(64), + accessLevel INT, + defaultProduct VARCHAR(255), + defaultLabel VARCHAR(64), + PRIMARY KEY (userName, domainName) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampAccessLevel ( + proj_id BIGINT, + accessLevel INT, + mjd_min FLOAT DEFAULT 0, + mjd_max FLOAT DEFAULT 0, + PRIMARY KEY (proj_id, accessLevel), + UNIQUE KEY (proj_id, accessLevel), + FOREIGN KEY(proj_id) REFERENCES pstampProject(proj_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE pstampWebRequest ( + num BIGINT AUTO_INCREMENT, + PRIMARY KEY(num) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE distTarget ( + target_id BIGINT AUTO_INCREMENT, + dist_group VARCHAR(64), + filter VARCHAR(64), + stage VARCHAR(64), + clean TINYINT, + state VARCHAR(64), + comment VARCHAR(255), + PRIMARY KEY(target_id), + KEY(stage), + KEY(dist_group), + KEY(filter), + KEY(state), + CONSTRAINT UNIQUE (dist_group, filter, stage, clean) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE distRun ( + dist_id BIGINT AUTO_INCREMENT, + target_id BIGINT, + stage VARCHAR(64), + stage_id BIGINT, + magic_ds_id BIGINT, + label VARCHAR(64), + outroot VARCHAR(255), + outdir VARCHAR(255), + clean TINYINT, + no_magic TINYINT, + alternate TINYINT, + state VARCHAR(64), + time_stamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + fault SMALLINT, + data_group VARCHAR(64), + note VARCHAR(255), + PRIMARY KEY(dist_id), + KEY(state), + KEY(label), + KEY(stage), + KEY(stage_id), + KEY(magic_ds_id), + KEY(fault), + KEY(data_group), + FOREIGN KEY(target_id) REFERENCES distTarget(target_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE distComponent ( + dist_id BIGINT, + component VARCHAR(64), + bytes INT, + md5sum VARCHAR(32), + state VARCHAR(64), + outdir VARCHAR(255), + name VARCHAR(255), + fault SMALLINT, + PRIMARY KEY(dist_id, component), + KEY(state), + FOREIGN KEY(dist_id) REFERENCES distRun(dist_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + +CREATE TABLE rcDestination ( + dest_id BIGINT AUTO_INCREMENT, + name VARCHAR(64), + status_uri VARCHAR(255), + comment VARCHAR(255), + last_fileset VARCHAR(255), + dbname VARCHAR(64), + dbhost VARCHAR(64), + state VARCHAR(64), + PRIMARY KEY(dest_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE rcInterest ( + int_id BIGINT AUTO_INCREMENT, + dest_id BIGINT, + target_id BIGINT, + state VARCHAR(64), + PRIMARY KEY(int_id), + FOREIGN KEY(dest_id) REFERENCES rcDestination(dest_id), + FOREIGN KEY(target_id) REFERENCES distTarget(target_id), + CONSTRAINT UNIQUE (dest_id, target_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE rcDSFileset ( + fs_id BIGINT AUTO_INCREMENT, + dist_id BIGINT, + dest_id BIGINT, + name VARCHAR(255), + state VARCHAR(64), + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + fault SMALLINT DEFAULT 0, + PRIMARY KEY(dist_id, dest_id), + KEY(fs_id), + FOREIGN KEY(dist_id) REFERENCES distRun(dist_id), + FOREIGN KEY(dest_id) REFERENCES rcDestination(dest_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE rcRun ( + rc_id BIGINT AUTO_INCREMENT, + fs_id BIGINT, + dest_id BIGINT, + state VARCHAR(64), + status_fs_name VARCHAR(64), + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + fault SMALLINT DEFAULT 0, + PRIMARY KEY(rc_id), + FOREIGN KEY(fs_id) REFERENCES rcDSFileset(fs_id), + FOREIGN KEY(dest_id) REFERENCES rcDestination(dest_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Sources from which to receive files +CREATE TABLE receiveSource ( + source_id BIGINT AUTO_INCREMENT, -- unique identifier + source VARCHAR(128) NOT NULL, -- source URI + product VARCHAR(64) NOT NULL, -- product of interest + workdir VARCHAR(255) NOT NULL, -- where to extract + state VARCHAR(64) NOT NULL, -- state 'enabled' or 'disabled' + comment VARCHAR(255), -- for human memory + fileset_last VARCHAR(128), -- last fileset seen + status_product VARCHAR(64), -- status data store product + ds_dbname VARCHAR(64), -- status data store's database name + ds_dbhost VARCHAR(64), -- status data store's host name + PRIMARY KEY(source_id), + KEY(source), + KEY(product), + KEY(comment) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Filesets to receive +CREATE TABLE receiveFileset ( + fileset_id BIGINT AUTO_INCREMENT, -- unique identifier + source_id BIGINT NOT NULL, -- link to receiveSource + fileset VARCHAR(128) NOT NULL, -- fileset to receive + state VARCHAR(64), -- new or full + dirinfo VARCHAR(255), -- uri for directory info file for this run + dbinfo VARCHAR(255), -- uri for database dump file for this run + fault SMALLINT NOT NULL DEFAULT 0, -- Fault code + PRIMARY KEY(fileset_id), + KEY(source_id), + CONSTRAINT UNIQUE(source_id, fileset), + FOREIGN KEY(source_id) REFERENCES receiveSource(source_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Files to receive +CREATE TABLE receiveFile ( + file_id BIGINT AUTO_INCREMENT, -- unique identifier + fileset_id BIGINT NOT NULL, -- link to receiveFileset + file VARCHAR(128) NOT NULL, -- file to receive + bytes BIGINT, + md5sum VARCHAR(255), + file_type VARCHAR(64), + component VARCHAR(64), + PRIMARY KEY(file_id), + KEY(fileset_id), + CONSTRAINT UNIQUE(fileset_id, file), + FOREIGN KEY(fileset_id) REFERENCES receiveFileset(fileset_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Result of receiving files +CREATE TABLE receiveResult ( + file_id BIGINT AUTO_INCREMENT, -- link to receiveFile + dtime_copy FLOAT, -- Time to copy + dtime_extract FLOAT, -- Time to extract + fault SMALLINT NOT NULL DEFAULT 0, -- Fault code + PRIMARY KEY(file_id), + KEY(fault), + FOREIGN KEY(file_id) REFERENCES receiveFile(file_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + + +-- Tables to support publishing of detections to a Science Client + +-- Clients to which we send stuff +CREATE TABLE publishClient ( + client_id BIGINT AUTO_INCREMENT, -- unique identifier + active TINYINT DEFAULT 0, -- whether we should worry about this or not + product VARCHAR(64), -- product name + stage VARCHAR(64) NOT NULL, -- stage of interest (chip, camera, diff, etc.) + magicked TINYINT DEFAULT 1, -- Require magicked data? + workdir VARCHAR(255) NOT NULL, -- working directory + comment VARCHAR(255), -- for human memory + name varchar(64) default NULL, -- unique client_id verbose identifier + output_format SMALLINT NOT NULL default 1, -- format output versioning + PRIMARY KEY(client_id), + UNIQUE KEY name (name) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Publishing a set of data (e.g., a specific diffRun) +CREATE TABLE publishRun ( + pub_id BIGINT AUTO_INCREMENT, -- unique identifier + client_id BIGINT NOT NULL, -- link to publishClient + stage_id BIGINT NOT NULL, -- link to various stage tables + label VARCHAR(64), -- label for run + state VARCHAR(64), -- state of run (new, full, etc.) + PRIMARY KEY(pub_id), + KEY(client_id), + KEY(stage_id), + KEY(label), + KEY(state), + FOREIGN KEY(client_id) REFERENCES publishClient(client_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Publishing a file within a set +CREATE TABLE publishDone ( + pub_id BIGINT AUTO_INCREMENT, -- link to publishRun + path_base VARCHAR(255), -- base path of output + hostname VARCHAR(64), -- name of host + dtime_script FLOAT, -- run time for script + fault SMALLINT NOT NULL DEFAULT 0, -- Fault code + PRIMARY KEY(pub_id), + KEY(fault), + FOREIGN KEY(pub_id) REFERENCES publishRun(pub_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + +-- Tables for static sky analysis + +-- A static sky analysis set +CREATE TABLE staticskyRun ( + sky_id BIGINT AUTO_INCREMENT, -- unique identifier + state VARCHAR(64) NOT NULL, -- state of run (new, full, etc.) + workdir VARCHAR(255) NOT NULL, -- working directory + label VARCHAR(64), -- processing label + data_group VARCHAR(64), -- group for data + dist_group VARCHAR(64), -- group for distribution + reduction VARCHAR(64), -- reduction class (for altering recipe) + note VARCHAR(255), -- note + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + PRIMARY KEY(sky_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(dist_group) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Inputs for static sky analysis +CREATE TABLE staticskyInput ( + sky_id BIGINT, -- static sky identifier + stack_id BIGINT, -- stack identifier + PRIMARY KEY(sky_id,stack_id), + KEY(stack_id), + FOREIGN KEY(sky_id) REFERENCES staticskyRun(sky_id), + FOREIGN KEY(stack_id) REFERENCES stackSumSkyfile(stack_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Result of static sky analysis +CREATE TABLE staticskyResult ( + sky_id BIGINT, -- static sky identifier + path_base VARCHAR(255) NOT NULL, -- root name for outputs + dtime_phot FLOAT, -- elapsed time for photometry + dtime_script FLOAT, -- elapsed time for script + sources INT, -- number of sources + num_inputs INT, -- number of input images + hostname VARCHAR(64) NOT NULL, -- host that executed script + good_frac FLOAT, -- good fraction of skycell + quality SMALLINT NOT NULL, -- bad quality flag + fault SMALLINT NOT NULL, -- fault code + PRIMARY KEY(sky_id), + KEY(good_frac), + KEY(fault), + KEY(quality), + FOREIGN KEY(sky_id) REFERENCES staticskyRun(sky_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + +-- Processing labels with their priorities +CREATE TABLE Label ( + label VARCHAR(64), + priority INT, + active TINYINT, + comment VARCHAR(80), + PRIMARY KEY(label), + KEY(priority), + KEY(active) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Tables to support background restoration + +-- Background replacement on a chipRun +CREATE TABLE chipBackgroundRun ( + chip_bg_id BIGINT AUTO_INCREMENT, -- unique identifier + chip_id BIGINT NOT NULL, -- link to chipRun + cam_id BIGINT DEFAULT 0, -- optional link to camRun + state VARCHAR(64) NOT NULL, -- state of run (new, full, etc.) + workdir VARCHAR(255) NOT NULL, -- working directory + label VARCHAR(64), -- processing label + data_group VARCHAR(64), -- group for data + dist_group VARCHAR(64), -- group for distribution + reduction VARCHAR(64), -- reduction class (for altering recipe) + note VARCHAR(255), -- note + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + magicked BIGINT DEFAULT 0 NOT NULL, -- magic status + PRIMARY KEY(chip_bg_id), + KEY(chip_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(dist_group), + FOREIGN KEY(chip_id) REFERENCES chipRun(chip_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Results of background replacement from chipBackgroundRun +CREATE TABLE chipBackgroundImfile ( + chip_bg_id BIGINT NOT NULL, -- unique identifier + class_id VARCHAR(64) NOT NULL, -- class (component) identifier + path_base VARCHAR(255) NOT NULL, -- root name for outputs + magicked BIGINT, -- magic_id if magicked + dtime_script FLOAT, -- elapsed time for script + hostname VARCHAR(64) NOT NULL, -- host that executed script + quality SMALLINT NOT NULL, -- bad quality flag + fault SMALLINT NOT NULL, -- fault code + software_ver VARCHAR(16), -- software version + bg FLOAT, -- background level + bg_stdev FLOAT, -- stdev of background + maskfrac_npix FLOAT, -- Number of pixels masked + maskfrac_static FLOAT, -- Fraction masked static + maskfrac_dynamic FLOAT, -- Fraction masked dynamic + maskfrac_magic FLOAT, -- Fraction masked magic + maskfrac_advisory FLOAT, -- Fraction masked advisory + PRIMARY KEY(chip_bg_id,class_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(chip_bg_id) REFERENCES chipBackgroundRun(chip_bg_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Background replacement on a warpRun (utilising chipBackgroundRun) +CREATE TABLE warpBackgroundRun ( + warp_bg_id BIGINT AUTO_INCREMENT, -- unique identifier + warp_id BIGINT NOT NULL, -- link to warpRun + chip_bg_id BIGINT NOT NULL, -- link to chipBackgroundRun + state VARCHAR(64) NOT NULL, -- state of run (new, full, etc.) + workdir VARCHAR(255) NOT NULL, -- working directory + label VARCHAR(64), -- processing label + data_group VARCHAR(64), -- group for data + dist_group VARCHAR(64), -- group for distribution + reduction VARCHAR(64), -- reduction class (for altering recipe) + note VARCHAR(255), -- note + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + magicked BIGINT DEFAULT 0 NOT NULL, -- magic status + PRIMARY KEY(warp_bg_id), + KEY(warp_id), + KEY(chip_bg_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(dist_group), + FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id), + FOREIGN KEY(chip_bg_id) REFERENCES chipBackgroundRun(chip_bg_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Results of background replacement from warpBackgroundRun +CREATE TABLE warpBackgroundSkyfile ( + warp_bg_id BIGINT NOT NULL, -- unique identifier + skycell_id VARCHAR(64) NOT NULL, -- skycell identifier + path_base VARCHAR(255) NOT NULL, -- root name for outputs + magicked BIGINT, -- magic_id if magicked + dtime_script FLOAT, -- elapsed time for script + hostname VARCHAR(64) NOT NULL, -- host that executed script + quality SMALLINT NOT NULL, -- bad quality flag + fault SMALLINT NOT NULL, -- fault code + software_ver VARCHAR(16), -- software version + bg FLOAT, -- background level + bg_stdev FLOAT, -- stdev of background + maskfrac_npix FLOAT, -- Number of pixels masked + maskfrac_static FLOAT, -- Fraction masked static + maskfrac_dynamic FLOAT, -- Fraction masked dynamic + maskfrac_magic FLOAT, -- Fraction masked magic + maskfrac_advisory FLOAT, -- Fraction masked advisory + PRIMARY KEY(warp_bg_id,skycell_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(warp_bg_id) REFERENCES warpBackgroundRun(warp_bg_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + +-- Tables to support (re-)photometry of a diff + +CREATE TABLE diffPhotRun ( + diff_phot_id BIGINT AUTO_INCREMENT, -- Identifier for diffPhotRun + diff_id BIGINT NOT NULL, -- Identifier for diffRun + state VARCHAR(64) NOT NULL, -- State of run + workdir VARCHAR(255) NOT NULL, -- working directory + label VARCHAR(64), -- processing label + data_group VARCHAR(64), -- group for data + reduction VARCHAR(64), -- reduction class (for altering recipe) + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + note VARCHAR(255), -- note + magicked BIGINT NOT NULL DEFAULT 0, -- magic mask applied + PRIMARY KEY(diff_phot_id), + KEY(diff_id), + KEY(state), + KEY(label), + KEY(data_group), + FOREIGN KEY(diff_id) REFERENCES diffRun(diff_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE diffPhotSkyfile ( + diff_phot_id BIGINT AUTO_INCREMENT, -- Identifier for diffPhotRun + skycell_id VARCHAR(64) NOT NULL, -- Skycell identifier + path_base VARCHAR(255) NOT NULL, -- Base of path for output + dtime_script FLOAT, -- elapsed time for script + hostname VARCHAR(64) NOT NULL, -- host that executed script + fault SMALLINT NOT NULL, -- fault code + quality SMALLINT NOT NULL, -- bad quality flag + software_ver VARCHAR(16), -- software version + magicked BIGINT NOT NULL DEFAULT 0, -- magic mask applied + PRIMARY KEY(diff_phot_id, skycell_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(diff_phot_id) REFERENCES diffPhotRun(diff_phot_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Tables for large area processing + +CREATE TABLE lapSequence ( + seq_id BIGINT AUTO_INCREMENT, -- Identifier for the processing sequence + name VARCHAR(64) NOT NULL, -- short name of the sequence + description VARCHAR(255) NOT NULL, -- longer description of the sequence + PRIMARY KEY(seq_id), + KEY(name) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE lapRun ( + lap_id BIGINT AUTO_INCREMENT, -- Identifier for the processing run + seq_id BIGINT NOT NULL, -- Identifier to match to the sequence + tess_id VARCHAR(64) NOT NULL, -- tessellation id to use + projection_cell VARCHAR(64) NOT NULL, -- projection cell from the tessellation to consider + filter VARCHAR(64) NOT NULL, -- filter + state VARCHAR(64) NOT NULL, -- state of run + label VARCHAR(64) NOT NULL, -- processing label + dist_group VARCHAR(64) NOT NULL, -- distribution group for products of this run + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + fault SMALLINT NOT NULL, -- fault code + quick_sass_id BIGINT, -- stackAssociation id for quick stack + final_sass_id BIGINT, -- stackAssociation id for final stack + PRIMARY KEY(lap_id), + KEY(seq_id), + KEY(projection_cell), + KEY(filter), + KEY(state), + KEY(label), + KEY(fault), + FOREIGN KEY(seq_id) REFERENCES lapSequence(seq_id), + FOREIGN KEY(quick_sass_id) REFERENCES stackAssociation(sass_id), + FOREIGN KEY(final_sass_id) REFERENCES stackAssociation(sass_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE lapExp ( + lap_id BIGINT NOT NULL, -- Link back to processing run + exp_id BIGINT NOT NULL, -- exposure definition + chip_id BIGINT, -- processing id from chipRun + pair_id BIGINT, -- companion chip_id + private TINYINT DEFAULT 0, -- denotes this exposure is private + pairwise TINYINT DEFAULT 0, -- denotes if this exposure should be pairwise diffed + active TINYINT DEFAULT 0, -- denotes if this exposure is currently in use + data_state VARCHAR(64) NOT NULL, -- state of exposure + PRIMARY KEY (lap_id), + KEY (exp_id), + KEY (chip_id), + KEY (pair_id), + KEY (data_state), + FOREIGN KEY (lap_id) REFERENCES lapRun(lap_id), + FOREIGN KEY (exp_id) REFERENCES rawExp(exp_id), + FOREIGN KEY (chip_id,exp_id) REFERENCES chipRun(chip_id,exp_id), + FOREIGN KEY (pair_id) REFERENCES chipRun(chip_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE minidvodbRun ( +minidvodb_id BIGINT AUTO_INCREMENT, + minidvodb_name VARCHAR(64), + minidvodb_group VARCHAR(64) NOT NULL, + minidvodb_path VARCHAR(255) NOT NULL, + state VARCHAR(64) NOT NULL, + creation_date TIMESTAMP, + PRIMARY KEY(minidvodb_id), KEY(minidvodb_name), INDEX(minidvodb_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE minidvodbProcessed ( + minidvodb_id BIGINT, + dtime_resort FLOAT, + dtime_relphot FLOAT, + dtime_script FLOAT, + epoch timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + fault SMALLINT NOT NULL, + UNIQUE KEY (minidvodb_id), + KEY(minidvodb_id), + FOREIGN KEY(minidvodb_id) REFERENCES minidvodbRun(minidvodb_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + + +CREATE TABLE minidvodbCopy ( + minidvodbcopy_id BIGINT AUTO_INCREMENT, + minidvodb_id BIGINT, + minidvodb_rsync_path VARCHAR(255) NOT NULL, + destination_host VARCHAR(255) NOT NULL, + fault SMALLINT NOT NULL, + state VARCHAR(64) NOT NULL, + epoch TIMESTAMP, + dtime FLOAT, + PRIMARY KEY(minidvodbcopy_id), + KEY(minidvodb_id), + FOREIGN KEY(minidvodb_id) REFERENCES minidvodbRun(minidvodb_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE mergedvodbRun ( + merge_id BIGINT AUTO_INCREMENT, + minidvodb_id BIGINT, + mergedvodb VARCHAR(64) NOT NULL, + mergedvodb_path VARCHAR(255) NOT NULL, + state VARCHAR(64) NOT NULL, + creation_date TIMESTAMP, + PRIMARY KEY(merge_id), KEY(minidvodb_id), KEY(mergedvodb), INDEX(merge_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE mergedvodbProcessed ( + merge_id BIGINT, + merge_order BIGINT, + dtime_verify FLOAT, + dtime_merge FLOAT, + dtime_script FLOAT, + epoch timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + fault SMALLINT NOT NULL, + UNIQUE KEY (merge_id), + KEY(merge_id), + FOREIGN KEY(merge_id) REFERENCES mergedvodbRun(merge_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE mergedvodbCopy ( + mergedvodbcopy_id BIGINT AUTO_INCREMENT, + merge_id BIGINT, + mergedvodb_rsync_path VARCHAR(255) NOT NULL, + destination_host VARCHAR(255) NOT NULL, + fault SMALLINT NOT NULL, + state VARCHAR(64) NOT NULL, + epoch TIMESTAMP, + dtime FLOAT, + PRIMARY KEY(mergedvodbcopy_id), + KEY(merge_id), + FOREIGN KEY(merge_id) REFERENCES mergedvodbRun(merge_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE vpRun ( + vp_id BIGINT AUTO_INCREMENT, + exp_id BIGINT, + state VARCHAR(64), + label VARCHAR(64), + data_group VARCHAR(64), + workdir VARCHAR(255), + note VARCHAR(255), + dest_id BIGINT, + outroot VARCHAR(255), + dtime_script FLOAT, + hostname VARCHAR(64), + fault SMALLINT NOT NULL DEFAULT 0, + PRIMARY KEY(vp_id), + KEY(exp_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(fault), + INDEX(vp_id, exp_id), + FOREIGN KEY(exp_id) REFERENCES rawExp(exp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE vpProcessedCell ( + vp_id BIGINT, + class_id VARCHAR(64), + cell_id VARCHAR(64), + dtime_photom FLOAT, + quality SMALLINT NOT NULL DEFAULT 0, + path_base VARCHAR(255), + fault SMALLINT NOT NULL DEFAULT 0, + KEY(quality), + KEY(fault), + FOREIGN KEY(vp_id) REFERENCES vpRun(vp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE skycalRun ( + skycal_id BIGINT AUTO_INCREMENT, -- unique identifier + sky_id BIGINT, + stack_id BIGINT, + state VARCHAR(64) NOT NULL, -- state of run (new, full, etc.) + workdir VARCHAR(255) NOT NULL, -- working directory + label VARCHAR(64), -- processing label + data_group VARCHAR(64), -- group for data + dist_group VARCHAR(64), -- group for distribution + reduction VARCHAR(64), -- reduction class (for altering recipe) + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + note VARCHAR(255), -- note + PRIMARY KEY(skycal_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(dist_group), + KEY(sky_id), + KEY(stack_id), + FOREIGN KEY(sky_id) REFERENCES staticskyRun(sky_id), + FOREIGN KEY(stack_id) REFERENCES stackRun(stack_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- Result of sky calibration analysis +CREATE TABLE skycalResult ( + skycal_id BIGINT, + path_base VARCHAR(255) NOT NULL, + dtime_script FLOAT, + dtime_astrom FLOAT, + sigma_ra FLOAT, + sigma_dec FLOAT, + n_astrom INT, + n_detections INT, + n_extended INT, + n_forced INT, + zpt_obs FLOAT, + zpt_stdev FLOAT, + fwhm_major FLOAT, + fwhm_minor FLOAT, + quality SMALLINT NOT NULL, + hostname VARCHAR(64) NOT NULL, + software_ver VARCHAR(64), + fault SMALLINT NOT NULL, + PRIMARY KEY(skycal_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(skycal_id) REFERENCES skycalRun(skycal_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE skycell ( + tess_id VARCHAR(64), + skycell_id VARCHAR(64), + radeg float, + decdeg float, + glong float, + glat float, + width float, + height float, + PRIMARY KEY (tess_id, skycell_id), + KEY(radeg), + KEY(decdeg), + KEY(glong), + KEY(glat) +) ENGINE=InnoDB DEFAULT CHARSET=latin1; + +CREATE TABLE survey ( + surveyID INT, + surveyName VARCHAR(16) NOT NULL, + description VARCHAR(255), + PRIMARY KEY(surveyID), + UNIQUE KEY(surveyName) +) ENGINE=InnoDB CHARSET=latin1; + +CREATE TABLE ippRelease ( + rel_id INT AUTO_INCREMENT, + surveyID INT, + release_name VARCHAR(64), + release_state VARCHAR(16), -- active, pending, archive, drop + dataRelease INT, -- PSPS dataRelease + priority INT, + dvodb VARCHAR(255), + ubercal_file VARCHAR(255), + accessLevelMin INT DEFAULT 0, + PRIMARY KEY(rel_id), + UNIQUE KEY(surveyID, release_name), + KEY(release_name), + KEY(release_state), + KEY(priority), + FOREIGN KEY(surveyID) REFERENCES survey(surveyID) +) ENGINE=Innodb DEFAULT CHARSET=latin1; + +CREATE TABLE relExp ( + relexp_id BIGINT AUTO_INCREMENT, + rel_id INT, + exp_id BIGINT, + chip_id BIGINT, -- links to the runs that supplied the data + cam_id BIGINT, -- for this release + group_id INT, -- id of relGroup that this exposure is contained in + state VARCHAR(16), -- released, pending, archive, drop + flags INT UNSIGNED, -- flags for relphot, relastro, ?? + zpt_obs FLOAT, -- calibrated zero point for this release of + zpt_stdev FLOAT, -- this exposure + mcal FLOAT, -- this exposure + ubercal_dist INT, -- ubercal distance (from dvo calibration) + path_base VARCHAR(255), -- path_base of any supporting files for this + -- release of this exposure. + fault SMALLINT NOT NULL, + registered DATETIME, -- insertion time for this row + time_stamp DATETIME, -- time of last update for this row + PRIMARY KEY (relexp_id), + FOREIGN KEY(rel_id) REFERENCES ippRelease(rel_id), + UNIQUE KEY(rel_id, exp_id), + KEY (state), + KEY (fault), + KEY (group_id), + FOREIGN KEY(exp_id) REFERENCES rawExp(exp_id), + FOREIGN KEY(chip_id) REFERENCES chipRun(chip_id), + FOREIGN KEY(cam_id) REFERENCES camRun(cam_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE relStack ( + relstack_id BIGINT AUTO_INCREMENT, + rel_id INT, + stack_id BIGINT, -- id of the stackRun + skycal_id BIGINT, -- id of the sky calibration run that supplied the calibration + -- for this release of this skycell. + skycell_id VARCHAR(64), + tess_id VARCHAR(64), + filter VARCHAR(16), + state VARCHAR(16), -- released, pending, archive, drop + flags INT UNSIGNED, + stack_type VARCHAR(16), -- nightly, deep, reference + zpt_obs FLOAT, -- calibrated zero point for this release of this skycell + zpt_stdev FLOAT, + mjd_obs INT UNSIGNED, -- for nightly stacks the mjd_obs, zero for deep and reference + path_base VARCHAR(255), -- path_base of any supporting files, + fault SMALLINT NOT NULL, + registered DATETIME, + time_stamp DATETIME, + PRIMARY KEY (relstack_id), + UNIQUE KEY (rel_id, tess_id, skycell_id, filter, stack_type, mjd_obs), + KEY (tess_id, skycell_id), + KEY (stack_type), + KEY (mjd_obs), + KEY (state), + KEY (fault), + FOREIGN KEY(rel_id) REFERENCES ippRelease(rel_id), + FOREIGN KEY(stack_id) REFERENCES stackRun(stack_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE relGroup ( + group_id INT AUTO_INCREMENT, + rel_id INT, + group_type VARCHAR(16), + lap_id BIGINT, + group_name VARCHAR(16), + state VARCHAR(16), -- new full + label VARCHAR(64), + exp_list_path VARCHAR(255), + fault SMALLINT NOT NULL, + registered DATETIME, + PRIMARY KEY (group_id), + KEY (rel_id, group_name), + KEY (group_type), + KEY (lap_id), + KEY (group_name), + KEY (state), + KEY (label), + KEY (fault), + FOREIGN KEY(rel_id) REFERENCES ippRelease(rel_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE lapGroup ( + seq_id BIGINT, + tess_id VARCHAR(64), + projection_cell VARCHAR(64), + state VARCHAR(16), + label VARCHAR(64), + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time group was registered + fault SMALLINT, + PRIMARY KEY(seq_id, tess_id, projection_cell), + KEY(state), + KEY(fault), + FOREIGN KEY(seq_id) REFERENCES lapSequence(seq_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE dqstatsRun ( + dqstats_id BIGINT NOT NULL AUTO_INCREMENT, + state VARCHAR(64) DEFAULT NULL, + registered TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + label VARCHAR(64) DEFAULT NULL, + fault SMALLINT NOT NULL, + PRIMARY KEY (dqstats_id), + KEY label (label), + KEY fault (fault) +) ENGINE=InnoDB DEFAULT CHARSET=latin1; + +CREATE TABLE dqstatsContent ( + dqstats_id BIGINT, + exp_id BIGINT, + chip_id BIGINT, + cam_id BIGINT, + warp_id BIGINT, + invalid TINYINT DEFAULT NULL, + KEY dqstats_id (dqstats_id), + KEY exp_id (exp_id), + KEY chip_id (chip_id), + KEY cam_id (cam_id), + KEY warp_id (warp_id), + FOREIGN KEY (dqstats_id) REFERENCES dqstatsRun (dqstats_id), + FOREIGN KEY (exp_id) REFERENCES rawExp (exp_id), + FOREIGN KEY (chip_id) REFERENCES chipRun (chip_id), + FOREIGN KEY (cam_id) REFERENCES camRun (cam_id), + FOREIGN KEY (warp_id) REFERENCES warpRun (warp_id) +) ENGINE=InnoDB DEFAULT CHARSET=latin1; + +CREATE TABLE fullForceRun ( + ff_id BIGINT NOT NULL AUTO_INCREMENT, + skycal_id BIGINT, + sources_path_base VARCHAR(255), + state VARCHAR(64), + workdir VARCHAR(255), + label VARCHAR(64), + data_group VARCHAR(64), + dist_group VARCHAR(64), + note VARCHAR(255), + reduction VARCHAR(64), + registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered + PRIMARY KEY(ff_id), + KEY(state), + KEY(label), + KEY(data_group), + KEY(skycal_id), + FOREIGN KEY(skycal_id) REFERENCES skycalRun(skycal_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE fullForceInput ( + ff_id BIGINT, + warp_id BIGINT, + PRIMARY KEY(ff_id, warp_id), + FOREIGN KEY(ff_id) REFERENCES fullForceRun(ff_id), + FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE fullForceResult ( + ff_id BIGINT, + warp_id BIGINT, + path_base VARCHAR(255) NOT NULL, + dtime_script FLOAT, + quality SMALLINT NOT NULL, + hostname VARCHAR(64) NOT NULL, + software_ver VARCHAR(16), + fault SMALLINT NOT NULL, + PRIMARY KEY(ff_id, warp_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(ff_id) REFERENCES fullForceRun(ff_id), + FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE fullForceSummary ( + ff_id BIGINT, + path_base VARCHAR(255) NOT NULL, + dtime_script FLOAT, + quality SMALLINT NOT NULL, + hostname VARCHAR(64) NOT NULL, + software_ver VARCHAR(16), + fault SMALLINT NOT NULL, + PRIMARY KEY(ff_id), + KEY(fault), + KEY(quality), + FOREIGN KEY(ff_id) REFERENCES fullForceRun(ff_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE remoteRun ( + remote_id BIGINT NOT NULL AUTO_INCREMENT, + state VARCHAR(64) NOT NULL, + stage VARCHAR(64) NOT NULL, + label VARCHAR(64) NOT NULL, + path_base VARCHAR(255) NOT NULL, + job_id BIGINT, + last_poll TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + fault SMALLINT NOT NULL, + PRIMARY KEY (remote_id), + KEY (state), + KEY (stage), + KEY (label), + KEY (job_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +CREATE TABLE remoteComponent ( + remote_id BIGINT, + stage_id BIGINT, + jobs INT, + state VARCHAR(64) NOT NULL, + path_base VARCHAR(255), + KEY (stage_id), + FOREIGN KEY (remote_id) REFERENCES remoteRun(remote_id) +) ENGINE=innodb DEFAULT CHARSET=latin1; + +-- These comment lines are here to avoid an empty query error. +-- Another way to avoid that problem is to omit the semicolon above but I think that is untidy. Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_drop_tables.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_drop_tables.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_drop_tables.sql (revision 41104) @@ -0,0 +1,133 @@ +SET FOREIGN_KEY_CHECKS=0; + +DROP TABLE IF EXISTS dbversion; +DROP TABLE IF EXISTS pzDataStore; +DROP TABLE IF EXISTS summitExp; +DROP TABLE IF EXISTS summitImfile; +DROP TABLE IF EXISTS pzDownloadExp; +DROP TABLE IF EXISTS pzDownloadImfile; +DROP TABLE IF EXISTS newExp; +DROP TABLE IF EXISTS newImfile; +DROP TABLE IF EXISTS rawExp; +DROP TABLE IF EXISTS rawImfile; +DROP TABLE IF EXISTS chipRun; +DROP TABLE IF EXISTS chipProcessedImfile; +DROP TABLE IF EXISTS chipImfile; +DROP TABLE IF EXISTS chipMask; +DROP TABLE IF EXISTS camRun; +DROP TABLE IF EXISTS camProcessedExp; +DROP TABLE IF EXISTS camMask; +DROP TABLE IF EXISTS detRun; +DROP TABLE IF EXISTS detInputExp; +DROP TABLE IF EXISTS addRun; +DROP TABLE IF EXISTS addProcessedExp; +DROP TABLE IF EXISTS addMask; +DROP TABLE IF EXISTS fakeRun; +DROP TABLE IF EXISTS fakeProcessedImfile; +DROP TABLE IF EXISTS fakeMask; +DROP TABLE IF EXISTS detProcessedImfile; +DROP TABLE IF EXISTS detProcessedExp; +DROP TABLE IF EXISTS detStackedImfile; +DROP TABLE IF EXISTS detNormalizedStatImfile; +DROP TABLE IF EXISTS detNormalizedImfile; +DROP TABLE IF EXISTS detNormalizedExp; +DROP TABLE IF EXISTS detResidImfile; +DROP TABLE IF EXISTS detResidExp; +DROP TABLE IF EXISTS detRunSummary; +DROP TABLE IF EXISTS detRegisteredImfile; +DROP TABLE IF EXISTS warpRun; +DROP TABLE IF EXISTS warpSkyCellMap; +DROP TABLE IF EXISTS warpImfile; +DROP TABLE IF EXISTS warpSkyfile; +DROP TABLE IF EXISTS warpMask; +DROP TABLE IF EXISTS warpSummary; +DROP TABLE IF EXISTS diffRun; +DROP TABLE IF EXISTS diffInputSkyfile; +DROP TABLE IF EXISTS diffSkyfile; +DROP TABLE IF EXISTS diffSummary; +DROP TABLE IF EXISTS stackRun; +DROP TABLE IF EXISTS stackInputSkyfile; +DROP TABLE IF EXISTS stackSumSkyfile; +DROP TABLE IF EXISTS stackSummary; +DROP TABLE IF EXISTS stackAssociation; +DROP TABLE IF EXISTS stackAssociationMap; +DROP TABLE IF EXISTS magicRun; +DROP TABLE IF EXISTS magicInputSkyfile; +DROP TABLE IF EXISTS magicTree; +DROP TABLE IF EXISTS magicNodeResult; +DROP TABLE IF EXISTS magicMask; +DROP TABLE IF EXISTS magicDSRun; +DROP TABLE IF EXISTS magicDSFile; +DROP TABLE IF EXISTS calDB; +DROP TABLE IF EXISTS calRun; +DROP TABLE IF EXISTS flatcorrRun; +DROP TABLE IF EXISTS flatcorrExp; +DROP TABLE IF EXISTS flatcorrChipLink; +DROP TABLE IF EXISTS flatcorrCamLink; +DROP TABLE IF EXISTS flatcorrAddstarLink; +DROP TABLE IF EXISTS pstampDataStore; +DROP TABLE IF EXISTS pstampProject; +DROP TABLE IF EXISTS pstampRequest; +DROP TABLE IF EXISTS pstampDependent; +DROP TABLE IF EXISTS pstampWebRequest; +DROP TABLE IF EXISTS pstampFile; +DROP TABLE IF EXISTS pstampJob; +DROP TABLE IF EXISTS pstampUserDomain; +DROP TABLE IF EXISTS pstampUser; +DROP TABLE IF EXISTS pstampAccessLevel; +DROP TABLE IF EXISTS distRun; +DROP TABLE IF EXISTS distTarget; +DROP TABLE IF EXISTS distComponent; +DROP TABLE IF EXISTS rcDSProduct; +DROP TABLE IF EXISTS rcDestination; +DROP TABLE IF EXISTS rcInterest; +DROP TABLE IF EXISTS rcDSFileset; +DROP TABLE IF EXISTS rcRun; +DROP TABLE IF EXISTS receiveSource; +DROP TABLE IF EXISTS receiveFileset; +DROP TABLE IF EXISTS receiveFile; +DROP TABLE IF EXISTS receiveResult; +DROP TABLE IF EXISTS publishClient; +DROP TABLE IF EXISTS publishRun; +DROP TABLE IF EXISTS publishDone; +DROP TABLE IF EXISTS staticskyInput; +DROP TABLE IF EXISTS staticskyResult; +DROP TABLE IF EXISTS staticskyRun; +DROP TABLE IF EXISTS Label; +DROP TABLE IF EXISTS chipBackgroundRun; +DROP TABLE IF EXISTS chipBackgroundImfile; +DROP TABLE IF EXISTS warpBackgroundRun; +DROP TABLE IF EXISTS warpBackgroundSkyfile; +DROP TABLE IF EXISTS diffPhotRun; +DROP TABLE IF EXISTS diffPhotSkyfile; +DROP TABLE IF EXISTS lapExp; +DROP TABLE IF EXISTS lapRun; +DROP TABLE IF EXISTS lapSequence; +DROP TABLE IF EXISTS minidvodbRun; +DROP TABLE IF EXISTS minidvodbCopy; +DROP TABLE IF EXISTS minidvodbProcessed; +DROP TABLE IF EXISTS vpProcessedCell; +DROP TABLE IF EXISTS vpRun; +DROP TABLE IF EXISTS skycalRun; +DROP TABLE IF EXISTS skycalResult; +DROP TABLE IF EXISTS skycell; +DROP TABLE IF EXISTS relStack; +DROP TABLE IF EXISTS relExp; +DROP TABLE IF EXISTS relGroup; +DROP TABLE IF EXISTS ippRelease; +DROP TABLE IF EXISTS survey; +DROP TABLE IF EXISTS lapGroup; +DROP TABLE IF EXISTS mergedvodbCopy; +DROP TABLE IF EXISTS mergedvodbProcessed; +DROP TABLE IF EXISTS mergedvodbRun; +DROP TABLE IF EXISTS dqstatsContent; +DROP TABLE IF EXISTS dqstatsRun; +DROP TABLE IF EXISTS fullForceRun; +DROP TABLE IF EXISTS fullForceInput; +DROP TABLE IF EXISTS fullForceResult; +DROP TABLE IF EXISTS fullForceSummary; +DROP TABLE IF EXISTS remoteComponent; +DROP TABLE IF EXISTS remoteRun; + + +SET FOREIGN_KEY_CHECKS=1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_update_version.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_update_version.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pxadmin_update_version.sql (revision 41104) @@ -0,0 +1,1 @@ +UPDATE dbversion SET schema_version = '%s', updated = current_timestamp() Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_find_completed_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_find_completed_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_find_completed_exp.sql (revision 41104) @@ -0,0 +1,34 @@ +SELECT DISTINCT + summit_id, + exp_name, + camera, + telescope, + dateobs, + state +FROM ( + SELECT + pzDownloadImfile.*, + pzDownloadExp.state, + summitExp.imfiles, + summitExp.dateobs + FROM pzDownloadExp + JOIN summitExp + USING(summit_id) + LEFT JOIN pzDownloadImfile + USING(summit_id) + LEFT JOIN newExp + USING(summit_id) + WHERE + pzDownloadExp.state = 'run' + AND newExp.summit_id IS NULL + AND newExp.tmp_exp_name IS NULL + AND newExp.tmp_camera IS NULL + AND newExp.tmp_telescope IS NULL + GROUP BY + pzDownloadExp.summit_id + -- it doesn't matter which field in pzDownloadImfile we count as we've + -- already Download a group by + HAVING + COUNT(pzDownloadImfile.exp_name) = summitExp.imfiles + AND SUM(pzDownloadImfile.fault) = 0 + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_pendingimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_pendingimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_pendingimfile.sql (revision 41104) @@ -0,0 +1,46 @@ +SELECT * +FROM ( + SELECT * FROM + (SELECT + summitImfile.*, + summitExp.dateobs + FROM summitImfile + JOIN pzDownloadExp + USING(summit_id) + LEFT JOIN pzDownloadImfile + USING(summit_id, class, class_id) + JOIN summitExp + USING(summit_id) + WHERE + pzDownloadExp.state = 'run' + AND pzDownloadImfile.summit_id IS NULL + AND pzDownloadImfile.exp_name IS NULL + AND pzDownloadImfile.camera IS NULL + AND pzDownloadImfile.telescope IS NULL + AND pzDownloadImfile.class IS NULL + AND pzDownloadImfile.class_id IS NULL + AND summitImfile.exp_name NOT LIKE 'c%' + ORDER BY dateobs) AS partA +UNION + SELECT * FROM + (SELECT + summitImfile.*, + summitExp.dateobs + FROM summitImfile + JOIN pzDownloadExp + USING(summit_id) + LEFT JOIN pzDownloadImfile + USING(summit_id, class, class_id) + JOIN summitExp + USING(summit_id) + WHERE + pzDownloadExp.state = 'run' + AND pzDownloadImfile.summit_id IS NULL + AND pzDownloadImfile.exp_name IS NULL + AND pzDownloadImfile.camera IS NULL + AND pzDownloadImfile.telescope IS NULL + AND pzDownloadImfile.class IS NULL + AND pzDownloadImfile.class_id IS NULL + AND summitImfile.exp_name LIKE 'c%' + ORDER BY dateobs) AS partB +) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revert_downloadimfile_faults.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revert_downloadimfile_faults.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revert_downloadimfile_faults.sql (revision 41104) @@ -0,0 +1,21 @@ +DELETE FROM pzDownloadImfile +WHERE + fault != 0 + -- fault was logged in the last 3 days + AND ABS(TIMESTAMPDIFF(SECOND, epoch, NOW())) < 86400 * 3 + AND ( + -- HTTP 503: try again + fault = 203 + -- HTTP 500: timeout + OR fault = 200 + -- HTTP 404: unknown datastore internal problem + OR fault = 104 + -- ipptool errors + OR fault = 115 + -- nebulous timout / ipptool errors + OR fault < 100 + -- perl untrapped die() + OR fault = 255 + -- HTTP 502 Proxy Error (seen now at ITC) + OR fault = 202 + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revert_fileset_faults.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revert_fileset_faults.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revert_fileset_faults.sql (revision 41104) @@ -0,0 +1,19 @@ +UPDATE summitExp SET fault = 0 +WHERE + fault != 0 + -- fault is older than 10 minutes + AND ABS(TIMESTAMPDIFF(SECOND, epoch, NOW())) > 60 * 10 + -- fault was logged in the last 3 days + AND ABS(TIMESTAMPDIFF(SECOND, epoch, NOW())) < 86400 * 3 + AND ( + -- HTTP 500: timeout + fault = 200 + -- HTTP 404: unknown datastore internal problem + OR fault = 104 + -- ipptool errors + OR fault < 100 + -- perl untrapped die() + OR fault = 255 + -- HTTP 502 Proxy Error (seen now at ITC) + OR fault = 202 + ) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revertcopied.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revertcopied.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/pztool_revertcopied.sql (revision 41104) @@ -0,0 +1,3 @@ +DELETE FROM pzDownloadImfile +WHERE + fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/rcserver_updatercrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/rcserver_updatercrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/rcserver_updatercrun.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE rcRun +JOIN rcDestination USING(dest_id) +JOIN rcDSFileset using(dest_id, fs_id) +SET Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_addfileset.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_addfileset.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_addfileset.sql (revision 41104) @@ -0,0 +1,4 @@ +SELECT fileset_id +FROM receiveFileset +WHERE source_id = %s + AND fileset = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_list.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_list.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_list.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT * +FROM receiveSource +WHERE receiveSource.state = 'enabled' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_pendingfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_pendingfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_pendingfile.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + receiveFile.file_id, + source, + product, + workdir, + dirinfo, + fileset, + fileset_id, + file, + bytes, + md5sum, + file_type, + component +FROM receiveFile +JOIN receiveFileset USING(fileset_id) +JOIN receiveSource USING(source_id) +LEFT JOIN receiveResult + ON receiveResult.file_id = receiveFile.file_id +WHERE + receiveSource.state = 'enabled' + AND ((receiveFileset.state = 'new') + OR (receiveFileset.state = 'listed' AND receiveFile.component = 'dirinfo' AND receiveFileset.dirinfo IS NULL)) + AND receiveFileset.fault = 0 + AND receiveResult.file_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_pendingfileset.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_pendingfileset.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_pendingfileset.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT receiveFileset.*, + receiveSource.source, + receiveSource.product, + receiveSource.workdir +FROM receiveFileset +JOIN receiveSource USING(source_id) +LEFT JOIN receiveFile USING(fileset_id) +WHERE receiveSource.state = 'enabled' + AND receiveFileset.state = 'reg' + AND receiveFileset.fault = 0 + AND receiveFile.fileset_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_revert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_revert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_revert.sql (revision 41104) @@ -0,0 +1,6 @@ +DELETE FROM receiveResult +USING receiveResult, receiveFile, receiveFileset, receiveSource +WHERE receiveResult.file_id = receiveFile.file_id + AND receiveFile.fileset_id = receiveFileset.fileset_id + AND receiveFileset.source_id = receiveSource.source_id + AND receiveResult.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_toadvance.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_toadvance.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/receivetool_toadvance.sql (revision 41104) @@ -0,0 +1,17 @@ +SELECT + receiveFileset.fileset_id, + receiveFileset.fileset, + receiveFileset.dbinfo, + receiveSource.status_product, + receiveSource.ds_dbname, + receiveSource.ds_dbhost +FROM receiveFileset + JOIN receiveSource USING(source_id) + JOIN receiveFile USING(fileset_id) + LEFT JOIN receiveResult USING(file_id) +WHERE receiveSource.state = 'enabled' + AND receiveFileset.state = 'new' + AND receiveFileset.fault = 0 +GROUP BY fileset_id +HAVING COUNT(receiveFile.file_id) = COUNT(receiveResult.file_id) + AND SUM(receiveResult.fault) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_checkburntoolimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_checkburntoolimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_checkburntoolimfile.sql (revision 41104) @@ -0,0 +1,34 @@ +SELECT DISTINCT + summitExp.exp_name, + rawImfile.uri, + summitExp.dateobs AS registered, + summitExp.imfiles, + summitExp.fault AS summit_fault, + pzDownloadExp.state AS download_state, + newExp.state AS newExp_state, + rawExp.state AS rawExp_state, + rawImfile.data_state AS imfile_state, + newExp.exp_id, + summitExp.exp_type, + rawImfile.dateobs, + rawImfile.obs_mode AS obs_mode, + rawImfile.object AS object, + rawImfile.burntool_state AS burntool_state, + rawImfile.class_id, + summitImfile.class_id AS summit_class_id, + (pzDownloadExp.state = 'stop') AS is_downloaded, + (rawImfile.burntool_state IS NOT NULL) AS is_registered + FROM + summitExp + JOIN summitImfile USING(summit_id) + LEFT JOIN pzDownloadExp USING(summit_id) + LEFT JOIN newExp USING(summit_id) + LEFT JOIN newImfile ON (summitImfile.class_id = newImfile.tmp_class_id AND newExp.exp_id = newImfile.exp_id) + LEFT JOIN rawExp ON rawExp.exp_id = newExp.exp_id + LEFT JOIN rawImfile ON (rawImfile.exp_id = newExp.exp_id AND rawImfile.tmp_class_id = newImfile.tmp_class_id) + WHERE + summitExp.dateobs >= '@DATEOBS_BEGIN@' + AND summitExp.dateobs <= '@DATEOBS_END@' + AND summitImfile.class_id = '@CLASS_ID@' + AND (summitExp.exp_name <= '@EXP_NAME@') +ORDER BY summitExp.dateobs Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_checkstatus.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_checkstatus.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_checkstatus.sql (revision 41104) @@ -0,0 +1,29 @@ +SELECT DISTINCT + summitExp.exp_name, + rawImfile.uri, + summitExp.dateobs AS registered, + summitExp.imfiles, + summitExp.fault AS summit_fault, + pzDownloadExp.state AS download_state, + newExp.state AS newExp_state, + rawExp.state AS rawExp_state, + rawImfile.data_state AS imfile_state, + newExp.exp_id, + summitExp.exp_type, + rawExp.dateobs, + rawExp.obs_mode AS obs_mode, + rawExp.object AS object, + rawImfile.burntool_state AS burntool_state, + rawImfile.class_id, + summitImfile.class_id AS summit_class_id, + (pzDownloadExp.state = 'stop') AS is_downloaded, + (rawImfile.burntool_state IS NOT NULL) AS is_registered + FROM summitExp + JOIN summitImfile USING(summit_id) + LEFT JOIN pzDownloadExp USING(summit_id) + LEFT JOIN newExp USING(summit_id) + LEFT JOIN newImfile ON (summitImfile.class_id = newImfile.tmp_class_id AND newExp.exp_id = newImfile.exp_id) + LEFT JOIN rawExp ON rawExp.exp_id = newExp.exp_id + LEFT JOIN rawImfile ON (rawImfile.exp_id = newExp.exp_id AND rawImfile.tmp_class_id = newImfile.tmp_class_id) + WHERE + summitImfile.class_id = '@CLASS_ID@' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_create_dup_table.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_create_dup_table.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_create_dup_table.sql (revision 41104) @@ -0,0 +1,2 @@ +CREATE TEMPORARY TABLE duplicate (exp_id BIGINT, PRIMARY KEY(exp_id)) +ENGINE=MEMORY Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_export_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_export_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_export_exp.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * FROM rawExp Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_export_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_export_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_export_imfile.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * FROM rawImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_finishcompressexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_finishcompressexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_finishcompressexp.sql (revision 41104) @@ -0,0 +1,82 @@ +select DISTINCT + exp_id, + exp_name, + camera, + telescope, + dateobs, + exp_tag, + exp_type, + filelevel, + workdir, + state, + reduction, + dvodb, + tess_id, + end_stage, + filter, + comment, + obs_mode, + obs_group, + airmass, + ra, + decl, + exp_time, + sat_pixel_frac, + bg, + bg_stdev, + bg_mean_stdev, + alt, + az, + ccd_temp, + posang, + m1_x, + m1_y, + m1_z, + m1_tip, + m1_tilt, + m2_x, + m2_y, + m2_z, + m2_tip, + m2_tilt, + env_temperature, + env_humidity, + env_wind_speed, + env_wind_dir, + teltemp_m1, + teltemp_m1cell, + teltemp_m2, + teltemp_spider, + teltemp_truss, + teltemp_extra, + pon_time, + user_1, + user_2, + user_3, + user_4, + user_5, + object, + sun_angle, + sun_alt, + moon_angle, + moon_alt, + moon_phase, + hostname, + fault, + epoch, + magicked +FROM + (SELECT V.*, + count(class_id) AS complete + FROM + (SELECT rawExp.*,count(newImfile.tmp_class_id) AS total + FROM rawExp + LEFT JOIN newImfile ON rawExp.exp_id = newImfile.exp_id + WHERE (rawExp.state = 'goto_compressed' OR rawExp.state = 'goto_lossy') + GROUP BY rawExp.exp_id) AS V + JOIN rawImfile ON V.exp_id = rawImfile.exp_id + WHERE ((V.state = 'goto_compressed' AND rawImfile.data_state = 'compressed') OR + (V.state = 'goto_lossy' AND rawImfile.data_state = 'lossy')) + GROUP BY V.exp_id) as foo +WHERE (foo.complete = foo.total) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingburntoolimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingburntoolimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingburntoolimfile.sql (revision 41104) @@ -0,0 +1,34 @@ +SELECT DISTINCT + summitExp.exp_name, + rawImfile.uri, + summitExp.dateobs AS registered, + summitExp.imfiles, + summitExp.fault AS summit_fault, + pzDownloadExp.state AS download_state, + newExp.state AS newExp_state, + rawExp.state AS rawExp_state, + rawImfile.data_state AS imfile_state, + newExp.exp_id, + newImfile.tmp_class_id, + summitExp.exp_type, + rawImfile.camera AS camera, + rawImfile.dateobs, + rawImfile.obs_mode AS obs_mode, + rawImfile.object AS object, + rawImfile.burntool_state AS burntool_state, + rawImfile.class_id, + summitImfile.class_id AS summit_class_id, + (pzDownloadExp.state = 'stop') AS is_downloaded, + (rawImfile.burntool_state IS NOT NULL) AS is_registered + FROM + summitExp + JOIN summitImfile USING(summit_id) + LEFT JOIN pzDownloadExp USING(summit_id) + LEFT JOIN newExp USING(summit_id) + LEFT JOIN newImfile ON (summitImfile.class_id = newImfile.tmp_class_id AND newExp.exp_id = newImfile.exp_id) + LEFT JOIN rawExp ON rawExp.exp_id = newExp.exp_id + LEFT JOIN rawImfile ON (rawImfile.exp_id = newExp.exp_id AND rawImfile.tmp_class_id = newImfile.tmp_class_id) + WHERE + summitExp.dateobs > '@DATEOBS_BEGIN@' + AND summitExp.dateobs <= '@DATEOBS_END@' +ORDER BY summitImfile.class_id,summitExp.dateobs Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingcompressimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingcompressimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingcompressimfile.sql (revision 41104) @@ -0,0 +1,7 @@ +SELECT rawImfile.*,rawExp.workdir,rawExp.exp_tag from rawImfile + JOIN rawExp USING(exp_id) + WHERE 1 +-- ((data_state = 'goto_compressed' AND state = 'goto_compressed') +-- OR (data_state = 'goto_lossy' AND state = 'goto_lossy')) +-- where hook %s +-- limit hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingexp.sql (revision 41104) @@ -0,0 +1,56 @@ +SELECT DISTINCT + summit_id, + exp_id, + tmp_exp_name, + tmp_camera, + tmp_telescope, + state, + workdir, + workdir_state, + reduction, + dvodb, + tess_id, + end_stage, + label, + camera, + filter, + obs_mode, + obs_group, + epoch, + dateobs +FROM + ( + select DISTINCT * from + ( + select newExp.*,count(newImfile.tmp_class_id) AS new_count + from newExp + LEFT JOIN newImfile USING (exp_id) + LEFT JOIN rawExp USING (exp_id) + where + newExp.state = 'run' AND rawExp.exp_id IS NULL + GROUP BY newExp.exp_id + ) AS NEWEXPOSURES + JOIN + ( + select DISTINCT newExp.exp_id, + rawImfile.camera, + rawImfile.filter, + rawImfile.dateobs, + rawImfile.obs_mode, + rawImfile.obs_group, + count(rawImfile.tmp_class_id) AS raw_count + from newExp + LEFT JOIN rawExp USING (exp_id) + LEFT JOIN rawImfile USING (exp_id) + where + newExp.state = 'run' AND rawExp.exp_id IS NULL + AND rawImfile.data_state = 'full' + -- where hook %s + GROUP BY newExp.exp_id + HAVING SUM(rawImfile.fault) = 0 + LIMIT 256 + ) AS RAWEXPOSURES + USING (exp_id) + WHERE raw_count = new_count + ) AS Foo +-- limit hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_pendingimfile.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT DISTINCT + newExp.exp_id, + newExp.tmp_exp_name, + newExp.tmp_camera, + newExp.tmp_telescope, + newExp.workdir, + newImfile.tmp_class_id, + newImfile.uri, + newImfile.bytes, + newImfile.md5sum, + summitExp.dateobs AS summit_dateobs +FROM newImfile +JOIN newExp + USING(exp_id) +LEFT JOIN summitExp USING(summit_id) +LEFT JOIN rawImfile + ON newExp.exp_id = rawImfile.exp_id + AND newImfile.tmp_class_id = rawImfile.tmp_class_id +WHERE + newExp.state = 'run' + AND rawImfile.tmp_class_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_populate_dup_table.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_populate_dup_table.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_populate_dup_table.sql (revision 41104) @@ -0,0 +1,13 @@ +INSERT INTO duplicate +SELECT exp_id FROM + ( + SELECT + MAX(exp_id) AS exp_id, +-- tmp_exp_name, +-- tmp_camera, +-- tmp_telescope, + count(exp_id) AS count + FROM newExp + GROUP BY tmp_exp_name, tmp_camera, tmp_telescope + HAVING count > 1 + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_processedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_processedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_processedexp.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + *, + DEGREES(ra) / 15. AS rahours, + DEGREES(ra) AS radeg, + DEGREES(decl) AS decdeg +FROM rawExp +-- bogus where clause so there is already a where statement to append too +WHERE rawExp.exp_id IS NOT NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_processedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_processedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_processedimfile.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + rawImfile.*, + newExp.tmp_exp_name, + newExp.tmp_camera, + newExp.tmp_telescope +FROM rawImfile +JOIN newExp + USING(exp_id) +WHERE rawImfile.ignored = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_revertprocessedexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_revertprocessedexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_revertprocessedexp.sql (revision 41104) @@ -0,0 +1,3 @@ +DELETE FROM rawExp +WHERE + fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_revertprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_revertprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_revertprocessedimfile.sql (revision 41104) @@ -0,0 +1,3 @@ +DELETE FROM rawImfile +WHERE + fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updatebyquery.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updatebyquery.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updatebyquery.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE rawExp + SET state = '%s' +WHERE 1 +-- where hook %s +-- limit hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updatebyqueryimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updatebyqueryimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updatebyqueryimfile.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE rawImfile JOIN rawExp USING(exp_id) + SET data_state = '%s' +WHERE 1 +-- where hook %s +-- limit hook %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updateprocessedimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updateprocessedimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/regtool_updateprocessedimfile.sql (revision 41104) @@ -0,0 +1,6 @@ +UPDATE rawImfile + SET %s + WHERE + rawImfile.exp_id = %lld + AND + rawImfile.class_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelexp.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + ippRelease.rel_id, + ippRelease.release_name, + rawExp.exp_name, + rawExp.filter, + rawExp.dateobs, + rawExp.exp_id, + camRun.chip_id, + camRun.cam_id, + camProcessedExp.zpt_obs, + camProcessedExp.zpt_stdev +FROM ippRelease +JOIN camRun +JOIN camProcessedExp USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN relExp AS previousrelexp USING(exp_id, rel_id) + +WHERE previousrelexp.relexp_id IS NULL + AND camRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_data_group.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_data_group.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_data_group.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + ippRelease.rel_id, + count(exp_id) AS num_exposures +FROM ippRelease + LEFT JOIN relExp using(rel_id) + LEFT JOIN camRun using(cam_id) + WHERE relExp.group_id = 0 + AND relExp.state != 'drop' + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_exp_data_group.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_exp_data_group.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_exp_data_group.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT relExp.relexp_id +FROM relExp + JOIN camRun USING(cam_id) +WHERE + relExp.group_id = 0 + AND relExp.state != 'drop' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_exp_lap.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_exp_lap.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_exp_lap.sql (revision 41104) @@ -0,0 +1,6 @@ +SELECT relExp.relexp_id +FROM relGroup + JOIN lapRun USING(lap_id) + JOIN lapExp USING(lap_id) + JOIN relExp using(rel_id, exp_id) +WHERE relExp.group_id = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_lap.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_lap.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelgroup_select_lap.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + lapRun.seq_id, + lapRun.lap_id, + ippRelease.rel_id +FROM lapRun + JOIN ippRelease + LEFT JOIN relGroup USING(lap_id) +WHERE (lapRun.state = 'done' or lapRun.state = 'full') + AND relGroup.group_id IS NULL +-- AND lapRun.seq_id = xxx + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelstack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelstack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelstack.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT + ippRelease.rel_id, + stackRun.stack_id, + 0 as skycal_id, + stackRun.skycell_id, + stackRun.tess_id, + stackRun.filter, + stackSumSkyfile.mjd_obs, + 0 AS zpt_obs, + 0 AS zpt_stdev + +FROM ippRelease +JOIN stackRun +JOIN stackSumSkyfile using(stack_id) +LEFT JOIN relStack AS previousRelStack + ON previousRelStack.rel_id = ippRelease.rel_id + AND previousRelStack.tess_id = stackRun.tess_id + AND previousRelStack.skycell_id = stackRun.skycell_id + AND previousRelStack.filter = stackRun.filter + -- JOIN hook %s +WHERE previousRelStack.relstack_id IS NULL + AND stackRun.state ='full' + AND stackSumSkyfile.quality = 0 + AND stackSumSkyfile.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelstack_with_skycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelstack_with_skycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_definerelstack_with_skycal.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT + ippRelease.rel_id, + stackRun.stack_id, + IFNULL(skycalRun.skycal_id, 0) as skycal_id, + stackRun.skycell_id, + stackRun.tess_id, + stackRun.filter, + stackSumSkyfile.mjd_obs, + skycalResult.zpt_obs, + skycalResult.zpt_stdev + +FROM ippRelease +JOIN stackRun +JOIN stackSumSkyfile USING(stack_id) +JOIN skycalRun ON skycalRun.stack_id = stackRun.stack_id +JOIN skycalResult ON skycalRun.skycal_id = skycalResult.skycal_id +-- LEFT JOIN relStack AS previousRelStack USING(rel_id, tess_id, skycell_id, filter) +LEFT JOIN relStack AS previousRelStack + ON ippRelease.rel_id = previousRelStack.rel_id + AND stackRun.tess_id = previousRelStack.tess_id + AND stackRun.skycell_id = previousRelStack.skycell_id + AND stackRun.filter = previousRelStack.filter +-- JOIN Hook %s + +WHERE previousRelStack.relstack_id IS NULL + AND skycalRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_deleterelexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_deleterelexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_deleterelexp.sql (revision 41104) @@ -0,0 +1,4 @@ +DELETE FROM relExp USING relExp, ippRelease, rawExp, camRun +WHERE relExp.rel_id = ippRelease.rel_id + AND relExp.exp_id = rawExp.exp_id + AND relExp.cam_id = camRun.cam_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_deleterelstack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_deleterelstack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_deleterelstack.sql (revision 41104) @@ -0,0 +1,3 @@ +DELETE FROM relStack USING relStack, ippRelease, stackRun +WHERE relStack.rel_id = ippRelease.rel_id + AND relStack.stack_id = stackRun.stack_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelease.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelease.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelease.sql (revision 41104) @@ -0,0 +1,1 @@ +select * from ippRelease JOIN survey USING(surveyID) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelexp.sql (revision 41104) @@ -0,0 +1,45 @@ +SELECT + relexp_id, + relExp.state, + relExp.fault, + ippRelease.release_name, + ippRelease.release_state, + ippRelease.rel_id, + ippRelease.priority, + ippRelease.dvodb, + ippRelease.ubercal_file, + survey.surveyName, + rawExp.exp_name, + relExp.flags, + relExp.zpt_obs, + relExp.zpt_stdev, + relExp.mcal, + relExp.ubercal_dist, + relExp.path_base, + relExp.registered, + relExp.time_stamp, + relExp.exp_id, + relExp.chip_id, + chipRun.state as chip_state, + relExp.cam_id, + camRun.state as cam_state, + camProcessedExp.path_base as cam_path_base, + camProcessedExp.fwhm_major, + camProcessedExp.fwhm_minor, + warpRun.warp_id, + warpRun.state as warp_state, + warpRun.tess_id, + rawExp.filter, + rawExp.dateobs, + TRUNCATE(DEGREES(rawExp.ra), 4) AS radeg, + TRUNCATE(DEGREES(rawExp.decl), 4) AS decdeg, + rawExp.comment +FROM relExp +JOIN ippRelease USING(rel_id) +JOIN survey USING(surveyID) +JOIN rawExp using(exp_id) +JOIN chipRun using(chip_id, exp_id) +JOIN camRun using(cam_id, chip_id) +JOIN camProcessedExp using(cam_id) +JOIN fakeRun using(cam_id) +JOIN warpRun using(fake_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelstack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelstack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listrelstack.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT + release_name, + surveyName, + relStack.*, + stackSumSkyfile.path_base AS stack_path_base, + stackSumSkyfile.quality, + stackRun.state AS stack_state, + stackRun.data_group AS stack_data_group, + skycalResult.fwhm_major, + skycalResult.quality as skycal_quality, + skycalResult.path_base AS skycal_path_base, + skycalRun.data_group AS skycal_data_group, + staticskyResult.sky_id AS sky_id, + staticskyResult.path_base AS staticsky_path_base, + 'GPC1' AS camera +FROM relStack +JOIN ippRelease USING(rel_id) +JOIN survey USING(surveyID) +JOIN stackRun USING(stack_id, filter, tess_id, skycell_id) +JOIN stackSumSkyfile USING(stack_id) +LEFT JOIN skycalRun USING(skycal_id) +LEFT JOIN skycalResult USING(skycal_id) +LEFT JOIN staticskyResult USING(sky_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listsurvey.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listsurvey.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_listsurvey.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * from survey Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_pendingrelgroup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_pendingrelgroup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_pendingrelgroup.sql (revision 41104) @@ -0,0 +1,7 @@ +SELECT + relGroup.*, + ippRelease.release_name +FROM relGroup + JOIN ippRelease USING(rel_id) +WHERE relGroup.state = 'new' + AND relGroup.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_setrelstackcalibratedfromskycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_setrelstackcalibratedfromskycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_setrelstackcalibratedfromskycal.sql (revision 41104) @@ -0,0 +1,10 @@ +UPDATE relStack + JOIN ippRelease USING(rel_id) + JOIN skycell USING(tess_id, skycell_id) + JOIN stackRun USING(stack_id, tess_id, skycell_id) + JOIN skycalRun USING(stack_id) -- join to skycal using stack_id not skycal_id ... + JOIN skycalResult ON skycalRun.skycal_id = skycalResult.skycal_id +SET relStack.skycal_id = skycalRun.skycal_id, -- ... because we set skycal_id here + relStack.zpt_obs = skycalResult.zpt_obs, + relStack.zpt_stdev = skycalResult.zpt_stdev, + relStack.state = 'calibrated' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_stacksummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_stacksummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_stacksummary.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT DISTINCT sass_id, + stackAssociation.tess_id, + stackAssociation.projection_cell, + stackAssociation.filter, + stackSummary.path_base, + stackAssociation.data_group, + survey.surveyName, + ippRelease.release_name, + ippRelease.rel_id + +FROM relStack + JOIN ippRelease USING(rel_id) + JOIN survey USING(surveyID) + JOIN stackAssociationMap USING(stack_id) + JOIN stackAssociation USING(sass_id, filter) + JOIN stackSummary using(sass_id, projection_cell) + +-- WHERE rel_id = 37 and skycell_id like 'skycell.0635.045' and filter = 'r.00000'\G +-- WHERE rel_id = 37 and projection_cell = 'skycell.0635' and filter = 'r.00000'\G Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_summary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_summary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_summary.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT DISTINCT + release_name, + surveyName, + sass_id, + stackAssociation.tess_id, + stackAssociation.projection_cell, + stackAssociation.filter, + stackSummary.path_base, + stackAssociation.data_group, + rawExp.camera +FROM relStack + JOIN ippRelease USING(rel_id) + JOIN survey USING(surveyID) + JOIN stackRun USING(stack_id, filter, tess_id, skycell_id) + JOIN stackAssociationMap USING(stack_id) + JOIN stackAssociation USING(sass_id, filter) + JOIN stackSummary using(sass_id, projection_cell) + JOIN stackInputSkyfile using(stack_id) + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_tocalibexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_tocalibexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/releasetool_tocalibexp.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT relExp.*, +ippRelease.release_name, +ippRelease.release_state +FROM relExp JOIN ippRelease USING(rel_id) +WHERE relExp.state = 'goto_calib' AND relExp.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/reltool_pendinggroup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/reltool_pendinggroup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/reltool_pendinggroup.sql (revision 41104) @@ -0,0 +1,8 @@ +-- relGroups to process +SELECT relGroup.*, + ippRelease.release_id, + ippRelease.release_name +FROM relGroup + JOIN ippRelease USING(rel_id) +WHERE relGroup.state ='new' + AND relGroup.fault = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_cam.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_cam.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_cam.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT stage_id FROM + +( SELECT cam_id AS stage_id + FROM camRun +WHERE +camRun.state = 'new' +-- where hook %s +GROUP BY cam_id +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING (remote_id) +WHERE +remoteRun.stage = 'camera' and remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING (stage_id) +WHERE R.remote_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_chip.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_chip.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_chip.sql (revision 41104) @@ -0,0 +1,26 @@ +SELECT stage_id FROM + +( SELECT chip_id AS stage_id + FROM chipRun +WHERE +chipRun.state = 'new' +-- where hook %s +GROUP BY chip_id +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING (remote_id) +WHERE +remoteRun.stage = 'chip' and remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING (stage_id) +WHERE R.remote_id IS NULL + + + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_diff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_diff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_diff.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT stage_id FROM + +( SELECT diff_id AS stage_id + FROM diffRun +WHERE +diffRun.state = 'new' +-- where hook %s +GROUP BY diff_id +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING(remote_id) +WHERE +remoteRun.stage = 'diff' AND remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING(stage_id) +WHERE R.remote_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_ff.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_ff.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_ff.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT stage_id FROM + +( SELECT ff_id AS stage_id + FROM fullForceRun +WHERE +fullForceRun.state = 'new' +-- where hook %s +GROUP BY ff_id +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING(remote_id) +WHERE +remoteRun.stage = 'ff' AND remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING(stage_id) +WHERE R.remote_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_stack.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_stack.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_stack.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT stage_id FROM + +( SELECT stack_id AS stage_id + FROM stackRun +WHERE +stackRun.state = 'new' +-- where hook %s +GROUP BY stack_id +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING (remote_id) +WHERE +remoteRun.stage = 'stack' and remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING (stage_id) +WHERE R.remote_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_staticsky.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_staticsky.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_staticsky.sql (revision 41104) @@ -0,0 +1,21 @@ +SELECT stage_id FROM + +( SELECT sky_id AS stage_id + FROM staticskyRun +WHERE +staticskyRun.state = 'new' +-- where hook %s +GROUP BY sky_id +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING(remote_id) +WHERE +remoteRun.stage = 'staticsky' AND remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING(stage_id) +WHERE R.remote_id IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_warp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_warp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_definebyquery_warp.sql (revision 41104) @@ -0,0 +1,24 @@ +SELECT stage_id FROM + +( SELECT warp_id AS stage_id, count(warpSkyCellMap.skycell_id) AS N + FROM warpRun + LEFT JOIN warpSkyCellMap USING(warp_id) +WHERE +warpRun.state = 'new' +-- where hook %s +GROUP BY warp_id +HAVING N != 0 +) AS W + +LEFT JOIN + +(SELECT remote_id,remoteRun.stage,remoteRun.label,remoteComponent.stage_id,remoteRun.state + FROM remoteRun + JOIN remoteComponent USING (remote_id) +WHERE +remoteRun.stage = 'warp' and remoteComponent.state != 'retry' +-- where hook %s +) AS R +USING (stage_id) +WHERE R.remote_id IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_doneprep.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_doneprep.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_doneprep.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT * FROM + (SELECT + remote_id, + remoteRun.path_base, + stage, + count(stage_id) as Ndone +FROM remoteRun +JOIN remoteComponent +USING (remote_id) +WHERE +((remoteComponent.state = 'prep_done') OR (remoteComponent.state = 'prep_fail') OR (remoteComponent.state = 'fail')) +-- and hook %s +GROUP BY remote_id) +AS DONE + +JOIN + +(SELECT +remote_id, +count(stage_id) as Njobs +FROM remoteRun +JOIN remoteComponent +USING (remote_id) +-- where hook %s +GROUP BY remote_id) +AS JOBS + +using (remote_id) + +where Ndone = Njobs Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_dropcomponent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_dropcomponent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_dropcomponent.sql (revision 41104) @@ -0,0 +1,4 @@ +DELETE FROM remoteComponent +WHERE +remote_id = %ld AND +stage_id = %ld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_listcomponent.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_listcomponent.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_listcomponent.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT DISTINCT + remoteComponent.remote_id, + remoteComponent.stage_id, + remoteComponent.jobs, + remoteComponent.state, + remoteComponent.path_base, + remoteRun.stage, + remoteRun.path_base as run_path_base +FROM remoteRun +JOIN remoteComponent + USING(remote_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_updatepoll.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_updatepoll.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/remotetool_updatepoll.sql (revision 41104) @@ -0,0 +1,3 @@ +UPDATE remoteRun SET last_poll = CURRENT_TIMESTAMP +WHERE +%s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/sctool_list.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/sctool_list.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/sctool_list.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + skycell.tess_id, + skycell.skycell_id, + TRUNCATE(skycell.radeg/15., 4) AS rahours, + skycell.radeg, + skycell.decdeg, + skycell.glong, + skycell.glat, + skycell.width, + skycell.height +FROM skycell Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_addsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_addsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_addsummary.sql (revision 41104) @@ -0,0 +1,5 @@ +INSERT INTO stackSummary (sass_id,projection_cell,path_base) VALUES ( + %lld, -- sass_id + '%s', -- projection_cell + '%s' -- path_base +) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_associationdefine_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_associationdefine_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_associationdefine_select.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT + sass_id, + data_group, + projection_cell, + tess_id, + filter + FROM stackAssociation +RIGHT JOIN ( + SELECT + data_group, + tess_id, + filter, + CASE WHEN LOCATE('.',skycell_id,9) > 0 THEN + SUBSTRING_INDEX(skycell_id,'.',2) ELSE + SUBSTRING_INDEX(skycell_id,'.',1) END + AS projection_cell + FROM stackRun + WHERE stackRun.stack_id = @STACK_ID@ + ) AS RUN USING (data_group,tess_id,filter,projection_cell) LIMIT 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert.sql (revision 41104) @@ -0,0 +1,25 @@ +-- Define the inputs to a defined stack +INSERT INTO + stackInputSkyfile(stack_id, warp_id) +SELECT + @STACK_ID@, -- This should be replaced with the stack_id + warpRun.warp_id +FROM warpSkyfile +JOIN warpRun + USING(warp_id) +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN camProcessedExp + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +WHERE + warpSkyfile.skycell_id = '@SKYCELL_ID@' + AND warpRun.state = 'full' + AND rawExp.filter = '@FILTER@' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert_random_part1.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert_random_part1.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert_random_part1.sql (revision 41104) @@ -0,0 +1,34 @@ +-- This is the part 1 of 2 of a query to use a random set of warps for +-- the inputs to a defined stack. +-- stacktool_definebyquery_insert_random_part2.sql should be appended. +INSERT INTO + stackInputSkyfile(stack_id, warp_id) +SELECT + @STACK_ID@, -- This should be replaced with the stack_id + warp_id +FROM ( + -- Sub-select to get the random set of warps + SELECT + warpSkyfile.*, + rand() AS rnd_num + FROM warpSkyfile + JOIN warpRun + USING(warp_id) + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN camProcessedExp + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN rawExp + USING(exp_id) + WHERE + skycell_id = '@SKYCELL_ID@' + AND warpRun.state = 'full' + AND rawExp.filter = '@FILTER@' -- the result of the query is grouped by filter and inserted for one at a time + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 +-- Put additional constraints here +-- stacktool_definebyquery_insert_random_part2.sql goes here Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert_random_part2.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert_random_part2.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_insert_random_part2.sql (revision 41104) @@ -0,0 +1,8 @@ +-- This is the part 2 of 2 of a query to use a random set of warps for +-- the inputs to a defined stack. +-- stacktool_definebyquery_insert_random_part1.sql should be prepended. + -- Sort by the random number, and take the first N + -- to get a random set of N. + ORDER BY rnd_num + LIMIT @RANDOM_LIMIT@ +) AS randomWarps Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_select.sql (revision 41104) @@ -0,0 +1,54 @@ +-- This is a query to get a list of skycells with warps to be stacked, +-- along with the number of warps already in a stack. + +SELECT + tess_id, + skycell_id, + filter, + num_warp, + MAX(num_stack) AS num_stack +FROM (( + -- Number of stack-ready warps as a function of skycell and filter + SELECT + warpSkyfile.tess_id as tess_id, + skycell_id, + rawExp.filter, + COUNT(warpSkyfile.skycell_id) AS num_warp -- number of warps that can be stacked + FROM warpRun + JOIN warpSkyfile USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN camProcessedExp USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE + warpRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 + -- WHERE hook %s + GROUP BY + warpSkyfile.tess_id, + skycell_id, + filter + ) AS warpsToStack +LEFT JOIN ( + -- Number of stack inputs as a function of skycell and filter + SELECT + tess_id, + skycell_id, + filter, + COUNT(stackInputSkyfile.warp_id) as num_stack -- number of warps in a stack + FROM stackRun + JOIN stackInputSkyfile USING(stack_id) + -- WHERE hook %s + GROUP BY + stack_id + ) AS stackSizes +-- JOINing the warpsToStack and stackSizes tables + USING(tess_id, skycell_id, filter) + ) +GROUP BY + tess_id, + skycell_id, + filter + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_test.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_test.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_definebyquery_test.sql (revision 41104) @@ -0,0 +1,59 @@ +-- This is the combination of the two parts of the query to get a list +-- of skycells with warps to be stacked, along with the number of +-- warps already in a stack. + +-------------------------------------------------------------------------------- +-- THIS FILE IS INTENDED FOR TESTING ONLY! IT IS NOT USED BY stacktool. +-- CHANGES SHOULD BE MADE TO stacktool_definebyquery_part1.sql +-- AND stacktool_definebyquery_part2.sql +-------------------------------------------------------------------------------- + +-- stacktool_definebyquery_part1.sql +SELECT + skycell_id, + tess_id, + filter, + num_warp, + MAX(num_stack) AS num_stack +FROM (( + -- Number of stack-ready warps as a function of skycell and filter + SELECT + skycell_id, + tess_id, + filter, + COUNT(warpSkyfile.skycell_id) AS num_warp -- number of warps that can be stacked + FROM warpRun + JOIN warpSkyfile USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + WHERE + warpRun.state = 'full' + AND warpSkyfile.quality = 0 + AND warpSkyfile.fault = 0 + -- Any additional selection on warps/exposures goes here +-- stacktool_definebyquery_part2.sql + GROUP BY + skycell_id, + filter + ) AS warpsToStack + LEFT JOIN ( + -- Number of stack inputs as a function of skycell and filter + SELECT + skycell_id, + tess_id, + filter, + COUNT(stackInputSkyfile.warp_id) as num_stack -- number of warps in a stack + FROM stackRun + JOIN stackInputSkyfile USING(stack_id) + GROUP BY + stack_id + ) AS stackSizes + -- JOINing the warpsToStack and stackSizes tables + USING(skycell_id, filter) + ) + GROUP BY + skycell_id, + filter +; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + stackRun.* +FROM stackRun +WHERE + stackRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_input_skyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_input_skyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_input_skyfile.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT stackInputSkyfile.* FROM stackInputSkyfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_run.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT stackRun.* from stackRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_sum_skyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_sum_skyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_export_sum_skyfile.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT stackSumSkyfile.* FROM stackSumSkyfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_find_complete_warps.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_find_complete_warps.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_find_complete_warps.sql (revision 41104) @@ -0,0 +1,46 @@ +SELECT + warpSkyfile.skycell_id, + warpSkyfile.tess_id, + stackRun.stack_id, + COUNT(warpSkyfile.skycell_id) AS num_avail, + COUNT(stackRun.stack_id) AS num_extant +FROM warpRun +JOIN warpSkyfile + USING(warp_id) +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +LEFT JOIN stackInputSkyfile + ON warpSkyfile.warp_id = stackInputSkyfile.warp_id +LEFT JOIN stackRun + ON stackRun.skycell_id = warpSkyfile.skycell_id + AND stackRun.stack_id = stackInputSkyfile.stack_id +WHERE + warpRun.state = 'full' + AND warpSkyfile.fault = 0 + AND warpSkyfile.quality = 0 +GROUP BY + warpSkyfile.skycell_id, stackRun.stack_id +HAVING + num_avail > num_extant + +-- It seems like we should be grouping the results here but in fact we don't +-- want to do that as each warp_id may contain multiple skycells. So a warp_id +-- may end up in more than one stackRun (for different skycells) +-- GROUP BY +-- warp_id + + +INSERT INTO + stackInputSkyfile(stack_id, warp_id) +SELECT + 13, + warp_id +FROM warpSkyfile +WHERE + skycell_id = 'new'; Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_inputskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_inputskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_inputskyfile.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + warpSkyfile.*, + rawExp.camera, + rawExp.exp_id, + rawExp.exp_name, + rawExp.exp_time, + rawExp.object, + rawExp.dateobs, + rawExp.comment +FROM stackRun +JOIN stackInputSkyfile USING(stack_id) +JOIN warpRun USING(warp_id) +JOIN warpSkyfile + ON warpSkyfile.warp_id = warpRun.warp_id + AND warpSkyfile.tess_id = stackRun.tess_id + AND warpSkyfile.skycell_id = stackRun.skycell_id +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,31 @@ +-- results in too many runs adding DISTINCT fixes this +-- is the camera needed for this? YES I think so that update can define the camera +-- is the camera unique for this? YES. Well, sort of. +-- stack_skycell.pl insures that the files +-- we're here to clean come from the same camera so +-- if the camera is not unique we have nothing to do +SELECT DISTINCT + stackRun.stack_id, + rawExp.camera, + stackRun.state, + stackSumSkyfile.path_base +FROM stackRun +JOIN stackSumSkyfile using(stack_id) +JOIN stackInputSkyfile + USING(stack_id) +JOIN warpSkyfile + ON stackInputSkyfile.warp_id = warpSkyfile.warp_id + AND stackRun.skycell_id = warpSkyfile.skycell_id + AND stackRun.tess_id = warpSkyfile.tess_id +JOIN warpRun + ON warpRun.warp_id = warpSkyfile.warp_id +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING (exp_id) +WHERE + (stackRun.state = 'goto_cleaned' OR stackRun.state = 'goto_scrubbed' OR stackRun.state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_pendingcleanupskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_pendingcleanupskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_pendingcleanupskyfile.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + stackSumSkyfile.*, + stackRun.state, + stackRun.workdir, + stackRun.dvodb, + stackRun.tess_id, + stackRun.skycell_id +FROM stackRun +JOIN stackSumSkyfile + USING(stack_id) +WHERE + (stackRun.state = 'goto_cleaned' OR stackRun.state = 'goto_scrubbed' OR stackRun.state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_revertsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_revertsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_revertsummary.sql (revision 41104) @@ -0,0 +1,3 @@ +DELETE FROM stackSummary + WHERE + sass_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_revertsumskyfile_delete.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_revertsumskyfile_delete.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_revertsumskyfile_delete.sql (revision 41104) @@ -0,0 +1,5 @@ +DELETE FROM stackSumSkyfile +USING stackSumSkyfile, stackRun +WHERE stackSumSkyfile.stack_id = stackRun.stack_id + AND stackRun.state = 'new' + AND fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_sassskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_sassskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_sassskyfile.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT + stackAssociation.sass_id, + stackAssociation.projection_cell, + stackSumSkyfile.*, + stackRun.state, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + stackRun.workdir, + stackRun.label, + (SELECT rawExp.camera FROM + stackInputSkyfile + JOIN warpRun USING(warp_id) + JOIN fakeRun ON warpRun.fake_id = fakeRun.fake_id + JOIN camRun ON camRun.cam_id = fakeRun.cam_id + JOIN chipRun ON camRun.chip_id = chipRun.chip_id + JOIN rawExp ON chipRun.exp_id = rawExp.exp_id + where stack_id = stackRun.stack_id limit 1 + ) as camera +FROM stackAssociation +JOIN stackAssociationMap USING(sass_id) +JOIN stackRun USING(stack_id) +LEFT JOIN stackSumSkyfile USING(stack_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_summary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_summary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_summary.sql (revision 41104) @@ -0,0 +1,17 @@ +SELECT DISTINCT sass_id, + stackAssociation.tess_id, + stackAssociation.projection_cell, + stackAssociation.filter, + stackSummary.path_base, + stackAssociation.data_group, + rawExp.camera +FROM stackRun + JOIN stackAssociationMap USING(stack_id) + JOIN stackAssociation USING(sass_id, filter) + JOIN stackSummary using(sass_id, projection_cell) + JOIN stackInputSkyfile using(stack_id) + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_sumskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_sumskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_sumskyfile.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + stackSumSkyfile.*, + stackRun.state, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + stackRun.workdir, + stackRun.label, + (SELECT rawExp.camera FROM + stackInputSkyfile + JOIN warpRun USING(warp_id) + JOIN fakeRun ON warpRun.fake_id = fakeRun.fake_id + JOIN camRun ON camRun.cam_id = fakeRun.cam_id + JOIN chipRun ON camRun.chip_id = chipRun.chip_id + JOIN rawExp ON chipRun.exp_id = rawExp.exp_id + where stack_id = stackRun.stack_id limit 1 + ) as camera +FROM stackRun +JOIN stackSumSkyfile +USING(stack_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tobkg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tobkg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tobkg.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT + stackRun.stack_id, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.workdir, + stackRun.reduction, + stackRun.label, + stackRun.state, + stackSumSkyfile.path_base, + rawExp.camera, + IFNULL(Label.priority, 10000) AS priority +FROM stackRun +JOIN stackInputSkyfile USING(stack_id) +JOIN warpRun USING(warp_id) +JOIN warpSkyfile USING(warp_id,skycell_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN stackSumSkyfile USING(stack_id) +LEFT JOIN Label ON Label.label = stackRun.label +WHERE + ((stackRun.state = 'full' AND stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0 AND stackSumSkyfile.background_model != 1)) + AND (Label.active OR Label.active IS NULL) + -- WHERE hook %s +GROUP BY stack_id +HAVING ((SUM(IF(warpRun.state = 'cleaned', 1, 0)) = COUNT(stackInputSkyfile.warp_id) OR + SUM(IF(warpRun.state = 'full', 1, 0)) = COUNT(stackInputSkyfile.warp_id)) AND + SUM(IF(warpSkyfile.background_model = 1, 1, 0)) = COUNT(stackInputSkyfile.warp_id)) + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tosum.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tosum.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tosum.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + stackRun.stack_id, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.workdir, + stackRun.reduction, + stackRun.label, + stackRun.state, + stackSumSkyfile.path_base, + IFNULL(Label.priority, 10000) AS priority +FROM stackRun +JOIN stackInputSkyfile USING(stack_id) +JOIN warpRun USING(warp_id) +LEFT JOIN stackSumSkyfile USING(stack_id) +LEFT JOIN Label ON Label.label = stackRun.label +WHERE + ((stackRun.state = 'new' AND stackSumSkyfile.stack_id IS NULL) + OR (stackRun.state = 'update' AND stackSumSkyfile.fault = 0 AND stackSumSkyfile.quality = 0)) + AND (Label.active OR Label.active IS NULL) + -- WHERE hook %s +GROUP BY stack_id +HAVING SUM(IF(warpRun.state = 'full', 1, 0)) = COUNT(stackInputSkyfile.warp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tosummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tosummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/stacktool_tosummary.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT sass_id,camera,workdir,tess_id,state FROM +(SELECT sass_id,rawExp.camera,stackRun.workdir,stackRun.tess_id,stackRun.state, + sum((stackRun.state = 'full')) AS is_done, sum(1) AS is_defined + FROM stackRun + JOIN stackInputSkyfile ON stackRun.stack_id = stackInputSkyfile.stack_id + JOIN stackAssociationMap ON stackRun.stack_id = stackAssociationMap.stack_id + JOIN stackAssociation USING(sass_id) + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + LEFT JOIN stackSummary USING(sass_id) +WHERE stackRun.state = 'full' AND + stackSummary.projection_cell IS NULL +-- WHERE HOOK %s +GROUP BY sass_id +) AS T +WHERE is_done = is_defined Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + tess_id, + skycell_id, + num_filter +FROM ( + SELECT + tess_id, + skycell_id, + COUNT(DISTINCT filter) AS num_filter + FROM stackRun + JOIN stackSumSkyfile USING(stack_id) + WHERE stackRun.state = 'full' + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + -- WHERE hook 1 %s + -- restrict by stackSumSkyfile.good_frac, stackRun.skycell_id, stackRun.label + GROUP BY + tess_id, + skycell_id + ) AS TMP +WHERE num_filter == %d + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_inputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_inputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_inputs.sql (revision 41104) @@ -0,0 +1,16 @@ + SELECT + MAX(stack_id) as stack_id, + tess_id, + skycell_id, + filter + FROM stackRun + JOIN stackSumSkyfile USING(stack_id) + JOIN skycell USING(tess_id, skycell_id) + WHERE stackRun.state = 'full' + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + AND stackRun.tess_id = '%s' + AND stackRun.skycell_id = '%s' + -- WHERE hook 1 %s + -- WHERE hook 2 %s + GROUP BY stackRun.filter Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_select.sql (revision 41104) @@ -0,0 +1,41 @@ +SELECT DISTINCT + tess_id, + skycell_id, + num_filter +FROM ( + SELECT + tess_id, + skycell_id, + COUNT(DISTINCT filter) AS num_filter + FROM stackRun + JOIN stackSumSkyfile USING(stack_id) + JOIN skycell using(tess_id, skycell_id) + WHERE stackRun.state = 'full' + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + -- WHERE hook 1 %s + -- restrict by selected filters, stackSumSkyfile.good_frac, stackRun.skycell_id, stackRun.label + GROUP BY + tess_id, + skycell_id + ) AS Stacks +LEFT JOIN ( + SELECT DISTINCT sky_id, + stackRun.skycell_id, + stackRun.tess_id, + stackRun.data_group, + COUNT(stack_id) AS num_filter + FROM staticskyRun + JOIN staticskyInput USING(sky_id) + JOIN stackRun USING(stack_id) + JOIN stackSumSkyfile USING(stack_id) + JOIN skycell USING(tess_id, skycell_id) + WHERE 1 + -- restrict by selected filters, stackSumSkyfile.good_frac, stackRun.skycell_id, stackRun.label + -- and unless -rerun is supplied that no existing staticskyRun exists + -- WHERE hook 2 %s + GROUP BY sky_id + ) AS ExistingRunsInputs +USING(tess_id, skycell_id, num_filter) +WHERE num_filter = %d +-- WHERE hook 3 %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_select_by_dg.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_select_by_dg.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_definebyquery_select_by_dg.sql (revision 41104) @@ -0,0 +1,43 @@ +SELECT DISTINCT + tess_id, + skycell_id, + Stacks.data_group, + num_filter +FROM ( + SELECT + tess_id, + skycell_id, + label, + data_group, + COUNT(DISTINCT filter) AS num_filter + FROM stackRun + JOIN stackSumSkyfile USING(stack_id) + WHERE stackRun.state = 'full' + AND stackSumSkyfile.fault = 0 + AND stackSumSkyfile.quality = 0 + -- WHERE hook 1 %s + -- restrict by selected filters, stackSumSkyfile.good_frac, stackRun.skycell_id, stackRun.label + GROUP BY + tess_id, + skycell_id, + data_group + ) AS Stacks +LEFT JOIN ( + SELECT DISTINCT sky_id, + stackRun.skycell_id, + stackRun.tess_id, + stackRun.data_group, + COUNT(stack_id) AS num_filter + FROM staticskyRun + JOIN staticskyInput USING(sky_id) + JOIN stackRun USING(stack_id) + JOIN stackSumSkyfile USING(stack_id) + WHERE 1 + -- WHERE hook 2 %s + -- restrict by selected filters, stackSumSkyfile.good_frac, stackRun.skycell_id, stackRun.label + GROUP BY sky_id + ) AS ExistingRunsInputs +USING(tess_id, skycell_id, data_group, num_filter) +WHERE num_filter = %d + AND Stacks.data_group = ExistingRunsInputs.data_group +-- WHERE hook 3 %s Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_defineskycalrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_defineskycalrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_defineskycalrun.sql (revision 41104) @@ -0,0 +1,19 @@ +SELECT + staticskyRun.sky_id, + stackRun.stack_id, + skycell_id, + filter, + staticskyRun.workdir, + staticskyRun.label, + staticskyRun.data_group, + staticskyRun.dist_group +FROM staticskyRun + JOIN staticskyResult USING(sky_id) + JOIN staticskyInput USING(sky_id) + JOIN stackRun USING(stack_id) + JOIN stackSumSkyfile USING(stack_id) + JOIN skycell USING(tess_id, skycell_id) + LEFT JOIN skycalRun ON staticskyRun.sky_id = skycalRun.sky_id AND stackRun.stack_id = skycalRun.stack_id -- join hook %s +WHERE staticskyRun.state = 'full' + AND staticskyResult.quality = 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_input.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_input.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_input.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT staticskyInput.* FROM staticskyInput Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_result.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_result.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_result.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT staticskyResult.* FROM staticskyResult Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_run.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT staticskyRun.* from staticskyRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_skycalresult.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_skycalresult.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_skycalresult.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * from skycalResult Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_skycalrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_skycalrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_export_skycalrun.sql (revision 41104) @@ -0,0 +1,1 @@ +SELECT * from skycalRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_inputs.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_inputs.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_inputs.sql (revision 41104) @@ -0,0 +1,9 @@ +SELECT + staticskyRun.*, + staticskyInput.stack_id, + stackSumSkyfile.path_base, + stackRun.filter +FROM staticskyRun +JOIN staticskyInput USING(sky_id) +JOIN stackSumSkyfile USING (stack_id) +JOIN stackRun using(stack_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_pendingskycalrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_pendingskycalrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_pendingskycalrun.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT DISTINCT + skycalRun.*, + stackRun.tess_id, + stackRun.skycell_id, + stackRun.filter, + staticskyResult.path_base, + rawExp.camera, + staticskyResult.num_inputs = 1 AS singlefilter +FROM skycalRun +JOIN staticskyRun USING (sky_id) +JOIN staticskyResult USING(sky_id) +JOIN stackRun USING (stack_id) +JOIN stackInputSkyfile USING(stack_id) +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING (chip_id) +JOIN rawExp USING(exp_id) +LEFT JOIN skycalResult USING(skycal_id) +WHERE + ((skycalRun.state = 'new' AND skycalResult.skycal_id IS NULL) + OR (skycalRun.state ='update' and staticskyRun.state ='full' and skycalResult.fault = 0)) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_result.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_result.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_result.sql (revision 41104) @@ -0,0 +1,18 @@ +SELECT + staticskyResult.*, + staticskyRun.state, + staticskyRun.workdir, + staticskyRun.label, + stackRun.tess_id, + stackRun.skycell_id, + count(stackRun.filter) AS num_filters, + TRUNCATE(skycell.radeg/15., 4) AS rahours, + skycell.radeg, + skycell.decdeg, + skycell.glong, + skycell.glat +FROM staticskyRun +JOIN staticskyResult USING(sky_id) +JOIN staticskyInput USING(sky_id) +JOIN stackRun USING(stack_id) +LEFT JOIN skycell USING(tess_id, skycell_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revert.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revert.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revert.sql (revision 41104) @@ -0,0 +1,9 @@ +DELETE FROM staticskyResult +USING staticskyResult, staticskyRun, staticskyInput, stackRun, skycell +WHERE staticskyRun.sky_id = staticskyResult.sky_id + AND staticskyRun.sky_id = staticskyInput.sky_id + AND staticskyInput.stack_id = stackRun.stack_id + AND stackRun.tess_id = skycell.tess_id + AND stackRun.skycell_id = skycell.skycell_id + AND staticskyRun.state = 'new' + AND staticskyResult.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revert_update.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revert_update.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revert_update.sql (revision 41104) @@ -0,0 +1,11 @@ +UPDATE staticskyRun + JOIN staticskyResult USING(sky_id) JOIN staticskyInput USING(sky_id) + JOIN stackRun USING(stack_id) JOIN skycell USING(tess_id, skycell_id) +SET staticskyResult.fault = 0 +WHERE staticskyRun.sky_id = staticskyResult.sky_id + AND staticskyRun.sky_id = staticskyInput.sky_id + AND staticskyInput.stack_id = stackRun.stack_id + AND stackRun.tess_id = skycell.tess_id + AND stackRun.skycell_id = skycell.skycell_id + AND staticskyRun.state = 'update' + AND staticskyResult.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revertskycal.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revertskycal.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_revertskycal.sql (revision 41104) @@ -0,0 +1,8 @@ +DELETE FROM skycalResult +USING skycalResult, skycalRun, stackRun, skycell +WHERE skycalRun.skycal_id = skycalResult.skycal_id + AND stackRun.stack_id = skycalRun.stack_id + AND skycalRun.state = 'new' + AND skycalResult.fault != 0 + AND stackRun.tess_id = skycell.tess_id + AND stackRun.skycell_id = skycell.skycell_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_skycalresult.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_skycalresult.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_skycalresult.sql (revision 41104) @@ -0,0 +1,15 @@ +SELECT + skycalResult.*, + skycalRun.stack_id, + skycalRun.workdir, + stackRun.filter, + skycalRun.state, + skycalRun.label, + skycalRun.data_group, + skycalRun.dist_group, + skycalRun.sky_id, + skycell.* +FROM skycalRun +JOIN skycalResult USING(skycal_id) +JOIN stackRun USING(stack_id) +JOIN skycell USING(tess_id, skycell_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_todo.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_todo.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/staticskytool_todo.sql (revision 41104) @@ -0,0 +1,31 @@ +SELECT + staticskyRun.sky_id, + staticskyRun.workdir, + staticskyRun.reduction, + staticskyRun.label, + staticskyRun.state, + stackRun.tess_id, + stackRun.skycell_id, + TRUNCATE(skycell.radeg/15., 4) as rahours, + skycell.radeg, + skycell.decdeg, + skycell.glong, + skycell.glat, + staticskyResult.path_base, + IFNULL(Label.priority, 10000) AS priority +FROM staticskyRun +JOIN staticskyInput USING (sky_id) +JOIN stackRun USING (stack_id) +JOIN skycell USING(tess_id, skycell_id) +LEFT JOIN staticskyResult USING(sky_id) +LEFT JOIN Label ON staticskyRun.label = Label.label +WHERE + ((staticskyRun.state = 'new' AND staticskyResult.sky_id IS NULL) + OR (staticskyRun.state = 'update' AND staticskyResult.fault = 0)) + + AND (Label.active OR Label.active IS NULL) + + -- WHERE hook %s +GROUP BY sky_id +HAVING SUM(IF(stackRun.state = 'full', 1, 0)) = COUNT(staticskyInput.sky_id) +ORDER BY priority DESC, sky_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_find_rawexp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_find_rawexp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_find_rawexp.sql (revision 41104) @@ -0,0 +1,12 @@ +-- this query is used to find potental rawExps to be queued for vptool +-- processeing +SELECT DISTINCT + rawExp.*, + newExp.label +FROM rawExp +JOIN newExp using (exp_id) +JOIN rawImfile using(exp_id) +LEFT JOIN vpRun USING (exp_id) +WHERE + rawExp.fault = 0 + AND rawImfile.video_cells Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_pendingimfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_pendingimfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_pendingimfile.sql (revision 41104) @@ -0,0 +1,12 @@ +SELECT + vpRun.*, + rawImfile.class_id, + rawImfile.uri +FROM vpRun + JOIN rawExp USING(exp_id) + JOIN rawImfile USING(exp_id) + LEFT JOIN vpProcessedCell USING(vp_id, class_id) +WHERE rawImfile.video_cells + AND vpRun.state = 'new' + AND vpProcessedCell.vp_id IS NULL + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_pendingrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_pendingrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_pendingrun.sql (revision 41104) @@ -0,0 +1,17 @@ +SELECT vp_id, + rawExp.camera, + CONCAT_WS('.', exp_name, exp_id) AS exp_tag, + vpRun.label, + vpRun.workdir, + vpRun.dest_id, + rcDestination.name AS dest_name, + rcDestination.dbname AS ds_dbname, + rcDestination.dbhost AS ds_dbhost, + IFNULL(Label.priority, 10000) as priority +FROM vpRun + JOIN rawExp USING(exp_id) + LEFT JOIN rcDestination USING(dest_id) + LEFT JOIN Label ON vpRun.label = Label.label +WHERE vpRun.state = 'new' + AND vpRun.fault = 0 + AND (Label.active OR Label.active IS NULL) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_processedcell.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_processedcell.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_processedcell.sql (revision 41104) @@ -0,0 +1,22 @@ +SELECT + vpProcessedCell.*, + vpRun.state, + vpRun.label, + vpRun.data_group, + vpRun.outroot, + rawExp.exp_id, + rawExp.exp_tag, + rawExp.exp_name, + rawExp.filter, + rawExp.camera, + rawExp.telescope, + rawExp.filelevel, + rawExp.dateobs, + rawExp.obs_mode, + rawExp.comment, + rawExp.ra, + rawExp.decl, + rawExp.exp_time +FROM vpRun + JOIN vpProcessedCell USING(vp_id) + JOIN rawExp USING(exp_id) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_revertrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_revertrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/vptool_revertrun.sql (revision 41104) @@ -0,0 +1,4 @@ +UPDATE vpRun + SET fault = 0 +WHERE vpRun.state = 'new' + AND fault > 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_addsummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_addsummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_addsummary.sql (revision 41104) @@ -0,0 +1,5 @@ +INSERT INTO warpSummary (warp_id,projection_cell,path_base) VALUES( + %lld, -- warp_id + '%s', -- projection_cell + '%s' -- path_base +) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_change_run_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_change_run_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_change_run_state.sql (revision 41104) @@ -0,0 +1,20 @@ +-- change state of warpRun from goto_cleaned to cleaned or goto_purged to purged +-- when all of the consituant skyfiles are in the end state +-- arguments are new state (cleaned or purged) warp_id and new state again for +-- the chipProcessedImfile sub query +UPDATE warpRun +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) + SET warpRun.state = '%s' + -- set magicked hook %s + WHERE + warpRun.warp_id = %lld + AND (SELECT + COUNT(warp_id) + FROM warpSkyfile + WHERE + warpSkyfile.warp_id = warpRun.warp_id + AND warpSkyfile.data_state != '%s' + AND warpSkyfile.quality = 0 + ) = 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_change_skyfile_data_state.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_change_skyfile_data_state.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_change_skyfile_data_state.sql (revision 41104) @@ -0,0 +1,14 @@ +-- handle changes in warpSkyfile.data_state. +-- Used for the modes tofullskyfile, tocleanedskyfile and topurgedskyfile +-- arguments are are new data_state, string for magic hook, warp_id, skycell_id +UPDATE warpSkyfile +JOIN warpRun USING(warp_id) +JOIN fakeRun USING(fake_id) +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) + SET + data_state = '%s' + -- set magicked hook %s +WHERE + warp_id = %lld + AND skycell_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_coalesce_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_coalesce_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_coalesce_run.sql (revision 41104) @@ -0,0 +1,1 @@ +select software_ver,maskfrac_npix,maskfrac_static,maskfrac_dynamic,maskfrac_magic,maskfrac_advisory from warpSkyfile where warp_id = %lld Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_definebyquery.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_definebyquery.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_definebyquery.sql (revision 41104) @@ -0,0 +1,33 @@ +SELECT DISTINCT + fakeRun.*, + chipRun.chip_id, + rawExp.camera, + rawExp.telescope, + rawExp.dateobs, + rawExp.exp_id, + rawExp.exp_tag, + rawExp.exp_name, + rawExp.exp_type, + rawExp.filelevel, + rawExp.filter, + rawExp.airmass, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + rawExp.sat_pixel_frac, + rawExp.bg, + rawExp.bg_stdev, + rawExp.bg_mean_stdev, + rawExp.alt, + rawExp.az, + rawExp.ccd_temp, + rawExp.posang, + rawExp.object, + rawExp.sun_angle +FROM fakeRun +JOIN camRun USING(cam_id) +JOIN chipRun USING(chip_id) +JOIN rawExp USING(exp_id) +WHERE fakeRun.state = 'full' +AND camRun.state = 'full' +AND chipRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_donecleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_donecleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_donecleanup.sql (revision 41104) @@ -0,0 +1,5 @@ +SELECT + warpRun.* +FROM warpRun +WHERE + warpRun.state = 'cleaned' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_exp.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_exp.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_exp.sql (revision 41104) @@ -0,0 +1,8 @@ +SELECT + fakeRun.* +FROM warpRun +JOIN fakeRun + USING(fake_id) +WHERE + warpRun.state = 'new' + AND fakeRun.state = 'full' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_imfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + warpImfile.* +FROM warpImfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_run.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + warpRun.* +FROM warpRun Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_skycell_map.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_skycell_map.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_skycell_map.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + warpSkyCellMap.* +FROM warpSkyCellMap Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_skyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_skyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_export_skyfile.sql (revision 41104) @@ -0,0 +1,3 @@ +SELECT + warpSkyfile.* +FROM warpSkyfile Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_finish_run.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_finish_run.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_finish_run.sql (revision 41104) @@ -0,0 +1,5 @@ +UPDATE warpRun + SET state = 'full', + magicked = %lld +WHERE warp_id = %lld + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_finished_run_select.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_finished_run_select.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_finished_run_select.sql (revision 41104) @@ -0,0 +1,23 @@ +SELECT + warp_id, + magicked +FROM + (SELECT DISTINCT + warpRun.warp_id, + warpRun.label as label, + warpSkyCellMap.warp_id as foo, + warpSkyfile.warp_id as bar, + warpSkyfile.magicked as magicked + FROM warpRun + JOIN warpSkyCellMap + USING(warp_id) + LEFT JOIN warpSkyfile + USING(warp_id, skycell_id) + WHERE + warpRun.state = 'new' + GROUP BY + warpRun.warp_id + HAVING + COUNT(warpSkyCellMap.warp_id) = COUNT(warpSkyfile.warp_id) + AND SUM(warpSkyfile.fault > 0) = 0 + ) as Foo Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_imfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_imfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_imfile.sql (revision 41104) @@ -0,0 +1,30 @@ +SELECT DISTINCT + rawImfile.*, + warpRun.fake_id, + camRun.cam_id as cam_id, + chipProcessedImfile.uri as chip_uri, + chipProcessedImfile.path_base as chip_path_base, + camProcessedExp.path_base as cam_path_base +FROM warpRun +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN camProcessedExp + USING(cam_id) +JOIN chipRun + ON camRun.chip_id = chipRun.chip_id +JOIN chipProcessedImfile + ON chipRun.chip_id = chipProcessedImfile.chip_id +JOIN rawImfile -- is there any reason not to refer back to rawimfiles? + ON chipProcessedImfile.exp_id = rawImfile.exp_id + AND chipProcessedImfile.class_id = rawImfile.class_id + AND rawImfile.ignored = 0 +WHERE + warpRun.state = 'new' + AND fakeRun.state = 'full' + AND camRun.state = 'full' + AND chipRun.state = 'full' + AND chipProcessedImfile.quality = 0 + and camProcessedExp.quality = 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_listrun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_listrun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_listrun.sql (revision 41104) @@ -0,0 +1,25 @@ +SELECT + warpRun.*, + camRun.cam_id, + chipRun.chip_id, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.filter, + rawExp.dateobs, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + magicDSRun.state AS dsRun_state, + IFNULL(magicDSRun.magic_ds_id, 0) AS magic_ds_id +FROM warpRun +JOIN fakeRun + ON warpRun.fake_id = fakeRun.fake_id +JOIN camRun + ON camRun.cam_id = fakeRun.cam_id +JOIN chipRun + ON camRun.chip_id = chipRun.chip_id +JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id +LEFT JOIN magicDSRun + ON stage = 'warp' AND stage_id = warp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanuprun.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanuprun.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanuprun.sql (revision 41104) @@ -0,0 +1,17 @@ +SELECT + warpRun.warp_id, + warpRun.workdir, + rawExp.camera, + rawExp.exp_tag, + warpRun.state +FROM warpRun +JOIN fakeRun +USING (fake_id) +JOIN camRun +USING (cam_id) +JOIN chipRun +USING (chip_id) +JOIN rawExp +USING (exp_id) +WHERE + (warpRun.state = 'goto_cleaned' OR warpRun.state = 'goto_scrubbed' OR warpRun.state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanupskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanupskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanupskyfile.sql (revision 41104) @@ -0,0 +1,20 @@ +SELECT + warpSkyfile.*, + warpRun.state, + warpRun.workdir, + warpRun.label, + warpRun.dvodb, + warpRun.end_stage +FROM warpRun +JOIN warpSkyfile + USING(warp_id) +WHERE + ((warpRun.state = 'goto_cleaned' + AND (warpSkyfile.data_state = 'full' + OR warpSkyfile.data_state = 'error_cleaned' + OR warpSkyfile.data_state = 'update') + AND warpSkyfile.quality = 0) + OR + (warpRun.state = 'goto_scrubbed' AND warpSkyfile.data_state != 'scrubbed') + OR + (warpRun.state = 'goto_purged' AND warpSkyfile.data_state != 'purged')) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanupskyfile_all.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanupskyfile_all.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_pendingcleanupskyfile_all.sql (revision 41104) @@ -0,0 +1,11 @@ +SELECT + warpSkyfile.*, + warpRun.state, + warpRun.workdir, + warpRun.label, + warpRun.dvodb, + warpRun.end_stage +FROM warpRun +JOIN warpSkyfile + USING(warp_id) +WHERE (warpRun.state = 'goto_cleaned' OR warpRun.state = 'goto_scrubbed' OR warpRun.state = 'goto_purged') Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertcleanup.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertcleanup.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertcleanup.sql (revision 41104) @@ -0,0 +1,7 @@ +UPDATE warpSkyfile + JOIN warpRun using(warp_id) +SET warpSkyfile.data_state = 'full', + warpRun.state = '%s' +WHERE + warpRun.state = '%s' + AND warpSkyfile.data_state = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertoverlap.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertoverlap.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertoverlap.sql (revision 41104) @@ -0,0 +1,10 @@ +DELETE FROM warpSkyCellMap +USING warpSkyCellMap, warpRun, fakeRun, camRun, chipRun, rawExp +WHERE warpSkyCellMap.warp_id = warpRun.warp_id + AND warpRun.fake_id = fakeRun.fake_id + AND fakeRun.cam_id = camRun.cam_id + AND camRun.chip_id = chipRun.chip_id + AND chipRun.exp_id = rawExp.exp_id + AND warpRun.state = 'new' + AND warpSkyCellMap.fault != 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertwarped_delete.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertwarped_delete.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertwarped_delete.sql (revision 41104) @@ -0,0 +1,10 @@ +DELETE FROM warpSkyfile +USING warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp +WHERE warpSkyfile.warp_id = warpRun.warp_id + AND warpRun.fake_id = fakeRun.fake_id + AND fakeRun.cam_id = camRun.cam_id + AND camRun.chip_id = chipRun.chip_id + AND chipRun.exp_id = rawExp.exp_id + AND warpRun.state = 'new' + AND warpSkyfile.fault != 0 + Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertwarped_updated.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertwarped_updated.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_revertwarped_updated.sql (revision 41104) @@ -0,0 +1,10 @@ +UPDATE warpSkyfile + JOIN warpRun USING(warp_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) +SET warpSkyfile.fault = 0 +WHERE warpRun.state = 'update' + AND warpSkyfile.data_state = 'update' + AND warpSkyfile.fault != 0 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_runstate.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_runstate.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_runstate.sql (revision 41104) @@ -0,0 +1,16 @@ +SELECT + warp_id, + warpRun.label, + warpRun.state, + warpRun.magicked, + chip_id, + chipRun.state AS chip_state, + exp_id, + magic_ds_id, + magicDSRun.state AS magic_ds_state +FROM warpRun + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + LEFT JOIN magicDSRun ON stage = 'warp' AND stage_id = warp_id AND re_place Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_scmap.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_scmap.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_scmap.sql (revision 41104) @@ -0,0 +1,44 @@ +SELECT + warpSkyCellMap.*, + chipProcessedImfile.uri, + chipProcessedImfile.path_base as chip_path_base, + camProcessedExp.path_base as cam_path_base, + camProcessedExp.fault as cam_fault, + camProcessedExp.background_model AS cam_background_model, + chipProcessedImfile.chip_id, + chipRun.state, + chipRun.data_group as chip_data_group, + chipProcessedImfile.data_state, + chipProcessedImfile.fault AS chip_fault, + chipProcessedImfile.magicked, + camRun.state as camState, + rawImfile.magicked AS raw_magicked, + IFNULL(magicDSRun.magic_ds_id, 0) AS magic_ds_id, + IFNULL(magicDSRun.state, 0) AS dsRun_state, + IFNULL(magicDSFile.data_state, 0) AS dsFile_data_state, + IFNULL(magicDSFile.fault, 0) as dsFile_fault +FROM warpRun +JOIN warpSkyCellMap + USING(warp_id) +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN camProcessedExp + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN chipProcessedImfile + ON chipRun.chip_id = chipProcessedImfile.chip_id + AND warpSkyCellMap.class_id = chipProcessedImfile.class_id + AND chipProcessedImfile.quality = 0 +JOIN rawImfile + ON chipRun.exp_id = rawImfile.exp_id + AND chipProcessedImfile.class_id = rawImfile.class_id +LEFT JOIN magicDSRun + ON chipRun.chip_id = magicDSRun.stage_id AND magicDSRun.stage = 'chip' +LEFT JOIN magicDSFile + ON magicDSFile.magic_ds_id = magicDSRun.magic_ds_id + AND chipProcessedImfile.class_id = magicDSFile.component +WHERE + 1 Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_setskyfiletoupdate.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_setskyfiletoupdate.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_setskyfiletoupdate.sql (revision 41104) @@ -0,0 +1,10 @@ +UPDATE warpRun + JOIN warpSkyfile USING(warp_id) +SET warpRun.state = 'update', + warpSkyfile.data_state = 'update', + warpSkyfile.fault = 0 + -- set hook %s +WHERE warp_id = %lld + AND (warpRun.state = 'cleaned' OR warpRun.state = 'update') + AND (warpSkyfile.data_state = 'cleaned' OR warpSkyfile.data_state = 'update') + AND (warpSkyfile.quality = 0) Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_tooverlap.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_tooverlap.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_tooverlap.sql (revision 41104) @@ -0,0 +1,29 @@ +SELECT + warpRun.warp_id, + warpRun.fake_id, + warpRun.workdir, + warpRun.tess_id, + warpRun.label, + rawExp.camera, + exp_id, + exp_tag +FROM warpRun +JOIN fakeRun + USING(fake_id) +JOIN camRun + USING(cam_id) +JOIN chipRun + USING(chip_id) +JOIN rawExp + USING(exp_id) +LEFT JOIN warpSkyCellMap + USING(warp_id) +LEFT JOIN warpMask + ON warpRun.label = warpMask.label +WHERE + warpRun.state = 'new' + AND fakeRun.state = 'full' + AND camRun.state = 'full' + AND chipRun.state = 'full' + AND warpSkyCellMap.warp_id IS NULL + AND warpMask.label IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_tosummary.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_tosummary.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_tosummary.sql (revision 41104) @@ -0,0 +1,7 @@ +SELECT DISTINCT warp_id,rawExp.camera,warpRun.workdir,warpRun.tess_id,rawExp.exp_tag,warpRun.state + FROM warpRun + JOIN fakeRun USING(fake_id) JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) + LEFT JOIN warpSummary USING(warp_id) +WHERE warpRun.state = 'full' AND + warpSummary.projection_cell IS NULL Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_towarped.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_towarped.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_towarped.sql (revision 41104) @@ -0,0 +1,111 @@ +SELECT + warp_id, + warp_skyfile_id, + skycell_id, + tess_id, + fake_id, + state, + reduction, + cam_id, + camera, + exp_tag, + workdir, + label, + magicked, + path_base, + priority +FROM ( + (SELECT DISTINCT + warpSkyCellMap.warp_id, + warpImfile.warp_skyfile_id, + warpSkyCellMap.skycell_id, + warpSkyCellMap.tess_id, + warpRun.fake_id, + warpRun.state, + warpRun.reduction, + warpRun.label, + camRun.cam_id, + rawExp.camera, + rawExp.exp_tag, + warpRun.workdir, + chipRun.magicked, + CAST(NULL AS CHAR(255)) AS path_base, + IFNULL(Label.priority, 10000) AS priority + FROM warpRun + JOIN warpSkyCellMap + USING(warp_id) + JOIN warpImfile + ON warpSkyCellMap.warp_id = warpImfile.warp_id + AND warpSkyCellMap.skycell_id = warpImfile.skycell_id + JOIN fakeRun + USING(fake_id) + JOIN camRun + USING(cam_id) + JOIN chipRun + USING(chip_id) + JOIN chipProcessedImfile + USING(chip_id) + JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id + LEFT JOIN warpSkyfile + ON warpRun.warp_id = warpSkyfile.warp_id + AND warpSkyCellMap.skycell_id = warpSkyfile.skycell_id + AND warpSkyCellMap.tess_id = warpSkyfile.tess_id + LEFT JOIN warpMask + ON warpRun.label = warpMask.label + LEFT JOIN Label ON warpRun.label = Label.label + WHERE + warpRun.state = 'new' + AND warpSkyfile.warp_id IS NULL + AND warpSkyfile.skycell_id IS NULL + AND warpSkyfile.tess_id IS NULL + AND fakeRun.state = 'full' + AND camRun.state = 'full' + AND chipRun.state = 'full' + AND warpMask.label IS NULL + AND warpSkyCellMap.fault = 0 + AND (Label.active OR Label.active IS NULL) + -- where hook 1 %s + -- limit hook 1 %s + ) +UNION + (SELECT + warpSkyCellMap.warp_id, + warpImfile.warp_skyfile_id, + warpSkyCellMap.skycell_id, + warpSkyCellMap.tess_id, + warpRun.fake_id, + warpRun.state, + warpRun.reduction, + warpRun.label, + camRun.cam_id, + rawExp.camera, + rawExp.exp_tag, + warpRun.workdir, + MIN(chipProcessedImfile.magicked) AS magicked, + warpSkyfile.path_base, + IFNULL(Label.priority, 10000) AS priority + FROM warpRun + JOIN warpImfile USING(warp_id) + JOIN warpSkyCellMap USING(warp_id, skycell_id) + JOIN warpSkyfile USING(warp_id, skycell_id) + JOIN fakeRun USING(fake_id) + JOIN camRun USING(cam_id) + JOIN chipRun USING(chip_id) + JOIN rawExp USING(exp_id) + LEFT JOIN chipProcessedImfile USING(chip_id, class_id) + LEFT JOIN Label ON warpRun.label = Label.label + WHERE warpRun.state = 'update' + AND warpSkyfile.data_state = 'update' + AND warpSkyfile.fault = 0 + AND camRun.state = 'full' + AND chipProcessedImfile.quality = 0 + -- Maybe add this to solve the if any chip is bad a whole warp can't be updated problem? + -- AND (chipProcessedImfile.fault = 0 OR chipProcessedImfile.fault = 26) + AND (Label.active OR Label.active IS NULL) + -- where hook 2 %s + GROUP BY warp_id, skycell_id + HAVING COUNT(warpSkyCellMap.class_id) = SUM(IF(chipProcessedImfile.data_state ='full', 1, 0)) + -- limit hook 2 %s + ) +) as towarped Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_towarped_labels.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_towarped_labels.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_towarped_labels.sql (revision 41104) @@ -0,0 +1,13 @@ +SELECT DISTINCT + warpRun.label, + Label.label AS label_label, + IFNULL(Label.priority, 10000) AS priority, + count(warpRun.warp_id) + FROM warpRun + LEFT JOIN Label ON warpRun.label = Label.label + WHERE + (warpRun.state = 'new' OR warpRun.state = 'update') + AND (Label.active OR Label.active IS NULL) + -- where hook %s + group by warpRun.label + ORDER BY priority DESC, warp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_updateskyfile.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_updateskyfile.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_updateskyfile.sql (revision 41104) @@ -0,0 +1,1 @@ +UPDATE warpSkyfile SET fault = %d WHERE warp_id = %lld AND skycell_id = '%s' Index: /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_warped.sql =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_warped.sql (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/share/warptool_warped.sql (revision 41104) @@ -0,0 +1,33 @@ +SELECT + warpSkyfile.*, + warpRun.state, + warpRun.workdir, + warpRun.label, + warpRun.data_group, + warpImfile.warp_skyfile_id, + rawExp.exp_id, + rawExp.exp_name, + rawExp.camera, + rawExp.filter, + rawExp.dateobs, + rawExp.ra, + rawExp.decl, + rawExp.exp_time, + camRun.cam_id, + magicDSRun.state AS dsRun_state, + IFNULL(magicDSRun.magic_ds_id, 0) AS magic_ds_id +FROM warpRun +JOIN warpSkyfile + USING(warp_id) +JOIN warpImfile + USING(warp_id, skycell_id) +JOIN fakeRun + ON warpRun.fake_id = fakeRun.fake_id +JOIN camRun + ON camRun.cam_id = fakeRun.cam_id +JOIN chipRun + ON camRun.chip_id = chipRun.chip_id +JOIN rawExp + ON chipRun.exp_id = rawExp.exp_id +LEFT JOIN magicDSRun + ON stage = 'warp' AND stage_id = warp_id Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/Makefile.am =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/Makefile.am (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/Makefile.am (revision 41104) @@ -0,0 +1,372 @@ +bin_PROGRAMS = \ + addtool \ + bgtool \ + caltool \ + camtool \ + chiptool \ + detselect \ + dettool \ + difftool \ + disttool \ + flatcorr \ + dqstatstool \ + labeltool \ + magictool \ + magicdstool \ + pstamptool \ + pxadmin \ + pxinject \ + faketool \ + pzgetexp \ + pzgetimfiles \ + pztool \ + regtool \ + stacktool \ + staticskytool \ + warptool \ + receivetool \ + pubtool \ + diffphottool \ + minidvodbtool \ + mergetool \ + laptool \ + vptool \ + sctool \ + releasetool \ + fftool \ + remotetool + +pkginclude_HEADERS = \ + pxadd.h \ + pxadmin.h \ + pxcam.h \ + pxchip.h \ + pxconfig.h \ + pxdata.h \ + pxfake.h \ + pxdqstats.h + pxmagic.h \ + pxminidvodb.h \ + pxmerge.h \ + pxregister.h \ + pxspace.h \ + pxtag.h \ + pxtools.h \ + pxtoolsErrorCodes.h \ + pxtree.h \ + pxwarp.h + +noinst_HEADERS = \ + addtool.h \ + bgtool.h \ + caltool.h \ + camtool.h \ + chiptool.h \ + detselect.h \ + dettool.h \ + difftool.h \ + disttool.h \ + flatcorr.h \ + faketool.h \ + dqstatstool.h \ + labeltool.h \ + magictool.h \ + magicdstool.h \ + pstamptool.h \ + pxinject.h \ + pzgetexp.h \ + pzgetimfiles.h \ + pztool.h \ + regtool.h \ + stacktool.h \ + staticskytool.h \ + warptool.h \ + pubtool.h \ + diffphottool.h \ + minidvodbtool.h \ + mergetool.h \ + laptool.h \ + vptool.h \ + sctool.h \ + releasetool.h \ + fftool.h \ + remotetool.h + +lib_LTLIBRARIES = libpxtools.la +libpxtools_la_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +libpxtools_la_LIBS = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) +libpxtools_la_LDFLAGS = -release $(PACKAGE_VERSION) +libpxtools_la_SOURCES = \ + pxadd.c \ + pxcam.c \ + pxchip.c \ + pxconfig.c \ + pxdata.c \ + pxerrors.c \ + pxfake.c \ + pxfault.c \ + pxdqstats.c \ + pxmagic.c \ + pxminidvodb.c \ + pxmerge.c \ + pxregister.c \ + pxspace.c \ + pxtag.c \ + pxtools.c \ + pxtoolsErrorCodes.c \ + pxtree.c \ + pxwarp.c + +# for pxtools.h +AM_CPPFLAGS = -I$(top_srcdir)/src$ + +pztool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pztool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pztool_SOURCES = \ + pztool.c \ + pztoolConfig.c + +pstamptool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pstamptool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pstamptool_SOURCES = \ + pstamptool.c \ + pstamptoolConfig.c + +addtool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +addtool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +addtool_SOURCES = \ + addtool.c \ + addtoolConfig.c + +bgtool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +bgtool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +bgtool_SOURCES = \ + bgtool.c \ + bgtoolConfig.c + +caltool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +caltool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +caltool_SOURCES = \ + caltool.c \ + caltoolConfig.c + +flatcorr_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +flatcorr_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +flatcorr_SOURCES = \ + flatcorr.c \ + flatcorrConfig.c + +regtool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +regtool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +regtool_SOURCES = \ + regtool.c \ + regtoolConfig.c + +chiptool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +chiptool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +chiptool_SOURCES = \ + chiptool.c \ + chiptoolConfig.c + +camtool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +camtool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +camtool_SOURCES = \ + camtool.c \ + camtoolConfig.c + +faketool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +faketool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +faketool_SOURCES = \ + faketool.c \ + faketoolConfig.c + +dqstatstool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +dqstatstool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +dqstatstool_SOURCES = \ + dqstatstool.c \ + dqstatstoolConfig.c + +labeltool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +labeltool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +labeltool_SOURCES = \ + labeltool.c \ + labeltoolConfig.c + +magictool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +magictool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +magictool_SOURCES = \ + magictool.c \ + magictoolConfig.c + +magicdstool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +magicdstool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +magicdstool_SOURCES = \ + magicdstool.c \ + magicdstoolConfig.c + +warptool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +warptool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +warptool_SOURCES = \ + warptool.c \ + warptoolConfig.c + +difftool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +difftool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +difftool_SOURCES = \ + difftool.c \ + difftoolConfig.c + +disttool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +disttool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +disttool_SOURCES = \ + disttool.c \ + disttoolConfig.c + +stacktool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +stacktool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +stacktool_SOURCES = \ + stacktool.c \ + stacktoolConfig.c + +staticskytool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +staticskytool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +staticskytool_SOURCES = \ + staticskytool.c \ + staticskytoolConfig.c + +pxadmin_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pxadmin_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pxadmin_SOURCES = \ + pxadmin.c \ + pxadminConfig.c + +pzgetexp_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pzgetexp_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pzgetexp_SOURCES = \ + pzgetexp.c \ + pzgetexpConfig.c + +pzgetimfiles_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pzgetimfiles_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pzgetimfiles_SOURCES = \ + pzgetimfiles.c \ + pzgetimfilesConfig.c + +dettool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +dettool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +dettool_SOURCES = \ + dettool.c \ + dettoolConfig.c \ + dettool_correction.c \ + dettool_detrunsummary.c \ + dettool_normalizedexp.c \ + dettool_normalizedimfile.c \ + dettool_normalizedstat.c \ + dettool_processedexp.c \ + dettool_processedimfile.c \ + dettool_residexp.c \ + dettool_residimfile.c \ + dettool_stack.c + +detselect_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +detselect_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +detselect_SOURCES = \ + detselect.c \ + detselectConfig.c + +pxinject_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pxinject_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pxinject_SOURCES = \ + pxinject.c \ + pxinjectConfig.c + + +receivetool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +receivetool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +receivetool_SOURCES = \ + receivetool.c \ + receivetoolConfig.c + +pubtool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +pubtool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +pubtool_SOURCES = \ + pubtool.c \ + pubtoolConfig.c + +diffphottool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +diffphottool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +diffphottool_SOURCES = \ + diffphottool.c \ + diffphottoolConfig.c + +minidvodbtool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +minidvodbtool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +minidvodbtool_SOURCES = \ + minidvodbtool.c \ + minidvodbtoolConfig.c + +mergetool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +mergetool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +mergetool_SOURCES = \ + mergetool.c \ + mergetoolConfig.c + +laptool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +laptool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +laptool_SOURCES = \ + laptool.c \ + laptoolConfig.c + +vptool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +vptool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +vptool_SOURCES = \ + vptool.c \ + vptoolConfig.c + +sctool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +sctool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +sctool_SOURCES = \ + sctool.c \ + sctoolConfig.c + +releasetool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +releasetool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +releasetool_SOURCES = \ + releasetool.c \ + releasetoolConfig.c + +fftool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +fftool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +fftool_SOURCES = \ + fftool.c \ + fftoolConfig.c + +remotetool_CFLAGS = $(PSLIB_CFLAGS) $(PSMODULES_CFLAGS) $(IPPDB_CFLAGS) +remotetool_LDADD = $(PSLIB_LIBS) $(PSMODULES_LIBS) $(IPPDB_LIBS) libpxtools.la +remotetool_SOURCES = \ + remotetool.c \ + remotetoolConfig.c + +clean-local: + -rm -f TAGS +# Tags for emacs +tags: + etags `find . -name \*.[ch] -print` + +# Error codes. +BUILT_SOURCES = pxtoolsErrorCodes.h pxtoolsErrorCodes.c +CLEANFILES = pxtoolsErrorCodes.h pxtoolsErrorCodes.c +EXTRA_DIST = pxtoolsErrorCodes.dat pxtoolsErrorCodes.c.in pxtoolsErrorCodes.h.in fakemagic + +pxtoolsErrorCodes.h : pxtoolsErrorCodes.dat pxtoolsErrorCodes.h.in + $(ERRORCODES) --data=pxtoolsErrorCodes.dat --outdir=. pxtoolsErrorCodes.h + +pxtoolsErrorCodes.c : pxtoolsErrorCodes.dat pxtoolsErrorCodes.c.in pxtoolsErrorCodes.h + $(ERRORCODES) --data=pxtoolsErrorCodes.dat --outdir=. pxtoolsErrorCodes.c + +BUILT_SOURCES += pxdata.c +EXTRA_DIST += pxdata.c.template + +pxdata.c: $(srcdir)/pxdata.c.template $(srcdir)/Makefile + $(PERL) -pe 's|\@PKGDATADIR\@|$(pkgdatadir)|' $< > $@ + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/addtool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/addtool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/addtool.c (revision 41104) @@ -0,0 +1,2375 @@ +/* + * addtool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxadd.h" +#include "addtool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool pendingexpMode(pxConfig *config); +static bool addprocessedexpMode(pxConfig *config); +static bool processedexpMode(pxConfig *config); +static bool revertprocessedexpMode(pxConfig *config); +static bool updateprocessedexpMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); +static bool addminidvodbrunMode(pxConfig *config); +static bool updateminidvodbrunMode(pxConfig *config); +static bool listminidvodbrunMode(pxConfig *config); +static bool flipminidvodbrunMode(pxConfig *config); +static bool checkminidvodbrunaddrunMode(pxConfig *config); +static bool addminidvodbprocessedMode(pxConfig *config); +static bool listminidvodbprocessedMode(pxConfig *config); +static bool revertminidvodbprocessedMode(pxConfig *config); +static bool updateminidvodbprocessedMode(pxConfig *config); + + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = addtoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(ADDTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(ADDTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(ADDTOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(ADDTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(ADDTOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(ADDTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpMode); + MODECASE(ADDTOOL_MODE_UPDATEPROCESSEDEXP, updateprocessedexpMode); + MODECASE(ADDTOOL_MODE_BLOCK, blockMode); + MODECASE(ADDTOOL_MODE_MASKED, maskedMode); + MODECASE(ADDTOOL_MODE_UNBLOCK, unblockMode); + MODECASE(ADDTOOL_MODE_ADDMINIDVODBRUN, addminidvodbrunMode); + MODECASE(ADDTOOL_MODE_UPDATEMINIDVODBRUN, updateminidvodbrunMode); + MODECASE(ADDTOOL_MODE_LISTMINIDVODBRUN, listminidvodbrunMode); + MODECASE(ADDTOOL_MODE_FLIPMINIDVODBRUN, flipminidvodbrunMode); + MODECASE(ADDTOOL_MODE_CHECKMINIDVODBRUNADDRUN, checkminidvodbrunaddrunMode); + MODECASE(ADDTOOL_MODE_ADDMINIDVODBPROCESSED,addminidvodbprocessedMode); + MODECASE(ADDTOOL_MODE_LISTMINIDVODBPROCESSED,listminidvodbprocessedMode); + MODECASE(ADDTOOL_MODE_REVERTMINIDVODBPROCESSED,revertminidvodbprocessedMode); + MODECASE(ADDTOOL_MODE_UPDATEMINIDVODBPROCESSED,updateminidvodbprocessedMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + // pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sky_id", "staticskyRun.sky_id", "=="); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalRun.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + + // PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_STR(minidvodb_name, config->args, "-set_minidvodb_name", false, false); + PXOPT_LOOKUP_STR(minidvodb_group, config->args, "-set_minidvodb_group", false, false); + PXOPT_LOOKUP_BOOL(image_only, config->args, "-image_only", false); + PXOPT_LOOKUP_BOOL(minidvodb, config->args, "-set_minidvodb", false); + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + PXOPT_LOOKUP_BOOL(uncensored, config->args, "-uncensored", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + PXOPT_LOOKUP_F32(minra, config->args, "-set_minra", false, false); + PXOPT_LOOKUP_F32(maxra, config->args, "-set_maxra", false, false); + PXOPT_LOOKUP_F32(mindec, config->args, "-set_mindec", false, false); + PXOPT_LOOKUP_F32(maxdec, config->args, "-set_maxdec", false, false); + + // Handle stages + PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + if (strcmp(stage, "cam") == 0) { + pxcamGetSearchArgs (config, where); + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "=="); // define using camRun label + pxAddLabelSearchArgs (config, where, "-data_group","camRun.data_group", "=="); // define using camRun label + PXOPT_COPY_STR(config->args, where, "-reduction", "camRun.reduction", "=="); + } + else if (strcmp(stage, "stack") == 0) { + pxAddLabelSearchArgs (config, where, "-label", "stackRun.label", "=="); // define using camRun label + pxAddLabelSearchArgs (config, where, "-data_group","stackRun.data_group", "=="); // define using camRun label + PXOPT_COPY_STR(config->args, where, "-reduction", "stackRun.reduction", "=="); + pxAddLabelSearchArgs (config, where, "-filter", "stackRun.filter", "=="); // define using camRun label + } + else if (strcmp(stage, "staticsky") == 0) { + pxAddLabelSearchArgs (config, where, "-label", "staticskyRun.label", "=="); // define using camRun label + pxAddLabelSearchArgs (config, where, "-data_group","staticskyRun.data_group", "=="); // define using camRun label + PXOPT_COPY_STR(config->args, where, "-reduction", "staticskyRun.reduction", "=="); + //no filter here + } + else if (strcmp(stage, "skycal") == 0) { + pxAddLabelSearchArgs (config, where, "-label", "skycalRun.label", "=="); //define using skycalRun label + pxAddLabelSearchArgs (config, where, "-data_group","skycalRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "skycalRun.reduction", "=="); + } + else if (strcmp(stage, "diff") == 0) { + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "=="); //define using diffRun label + pxAddLabelSearchArgs (config, where, "-data_group","diffRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "diffRun.reduction", "=="); + } + else if (strcmp(stage, "fullforce") == 0) { + pxAddLabelSearchArgs (config, where, "-label", "fullForceRun.label", "=="); //define using fullForceRun label + pxAddLabelSearchArgs (config, where, "-data_group","fullForceRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "fullForceRun.reduction", "=="); + } + else if (strcmp(stage, "fullforce_summary")==0) { + //should be nearly identical to fullforce (uses the same tables) + pxAddLabelSearchArgs (config, where, "-label", "fullForceRun.label", "=="); + pxAddLabelSearchArgs (config, where, "-data_group","fullForceRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "fullForceRun.reduction", "=="); + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // Select either destreaked or uncensored data. CZW: Should this be defaulted to uncensored? + if (!(destreaked && uncensored) && (destreaked || uncensored)) { + //if one of destreaked and uncensored is true, but not both + if (destreaked) { + psTrace("addtool.c", PS_LOG_INFO, "using -destreaked\n"); + } else { + psTrace("addtool.c", PS_LOG_INFO, "using -unscensored\n"); + } + } else { + if (destreaked && uncensored) { + psError(PS_ERR_UNKNOWN, false, "Both -destreaked and -uncensored are selected. Please use one or the other, not both, \n"); + return false; + } + if (!destreaked && !uncensored) { + psError(PS_ERR_UNKNOWN, false, "Neither -destreaked or -uncensored are selected. Please use one.\n"); + return false; + } + } + if ((strcmp(stage,"diff") == 0) || + (strcmp(stage,"fullforce") == 0) || + (strcmp(stage,"fullforce_summary") == 0)) { + if (! (dvodb)) { + psError(PS_ERR_UNKNOWN, false, "The SQL is not correctly written to handle this case (-set_dvodb not specified), so stopping here."); + return(false); + } + } + + + // prevent queueing an addRun if a given exposure has already been added to + // the given dvo database + psString dvodb_string = NULL; + psString bare_query = NULL; + + if (strcmp(stage,"cam") == 0 ) { + if (dvodb) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_cam_id_dvo.sql\n%s\n", dvodb,stage); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_cam_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_find_cam_id.sql\n%s\n",stage); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_cam_id.sql"); + // inherit dvodb from camRun, avoid matching NULL + psStringAppend(&dvodb_string, "(camRun.dvodb IS NOT NULL AND previous_dvodb = camRun.dvodb)"); + } + } + + else if (strcmp(stage,"stack") == 0) { + if (dvodb) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_stack_id_dvo.sql\n%s\n", dvodb,stage); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_stack_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_find_stack_id.sql\n%s\n",stage); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_stack_id.sql"); + // inherit dvodb from camRun, avoid matching NULL + psStringAppend(&dvodb_string, "(stackRun.dvodb IS NOT NULL AND previous_dvodb = stackRun.dvodb)"); + } + } + + else if (strcmp(stage,"staticsky") == 0) { + if (dvodb ) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_sky_id_multi_dvo.sql\n%s\n", dvodb,stage); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_sky_id_multi_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_find_sky_id_multi.sql\n%s\n",stage); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_sky_id_multi.sql"); + // inherit dvodb from camRun, avoid matching NULL + psStringAppend(&dvodb_string, "(staticskyRun.dvodb IS NOT NULL AND previous_dvodb = staticskyRun.dvodb)"); + } + } + + else if (strcmp(stage,"skycal") == 0) { + if (dvodb) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_skycal_id_dvo.sql\n%s\n", dvodb,stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_skycal_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_find_skycal_id.sql\n%s\n",stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_skycal_id.sql"); + // inherit dvodb from skycalRun, avoid matching NULL + psStringAppend(&dvodb_string, "(skycalRun.dvodb IS NOT NULL AND previous_dvodb = skycalRun.dvodb)"); + // this is silly, there is no dvodb in skycalRun...? + } + } + + else if (strcmp(stage,"diff") == 0) { + if (dvodb) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_diff_id_dvo.sql\n%s\n", dvodb,stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_diff_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_find_diff_id.sql\n%s\n",stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_diff_id.sql"); + // inherit dvodb from skycalRun, avoid matching NULL + psStringAppend(&dvodb_string, "(diffRun.dvodb IS NOT NULL AND previous_dvodb = diffRun.dvodb)"); + // this is silly, there is no dvodb in skycalRun...? + } + } + + else if (strcmp(stage,"fullforce") == 0) { + if (dvodb) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_ff_id_dvo.sql\n%s\n", dvodb,stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_ff_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_ff_id.sql\n%s\n",stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_ff_id.sql"); + // inherit dvodb from skycalRun, avoid matching NULL + psStringAppend(&dvodb_string, "(fullForceRun.dvodb IS NOT NULL AND previous_dvodb = fullForceRun.dvodb)"); + // this is silly, there is no dvodb in skycalRun...? + } + } + else if (strcmp(stage,"fullforce_summary") == 0) { + if (dvodb) { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument found (%s) using addtool_find_ffsummary_id_dvo.sql\n%s\n", dvodb,stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_ffsummary_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "addRun.dvodb = '%s'", dvodb); + } else { + psTrace("addtool.c", PS_LOG_INFO, "dvodb argument not found using addtool_ffsummary_id.sql\n%s\n",stage); + // find the skycal_id of all the exposures that we want to queue up. + bare_query = pxDataGet("addtool_find_ffsummary_id.sql"); + // inherit dvodb from skycalRun, avoid matching NULL + psStringAppend(&dvodb_string, "(fullForceRun.dvodb IS NOT NULL AND previous_dvodb = fullForceRun.dvodb)"); + // this is silly, there is no dvodb in skycalRun...? + } + } + + if (!bare_query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + psFree(where); + return false; + } + + // Take the bare query and add the dvodb selector + psString query = NULL; + psStringAppend(&query, bare_query, dvodb_string); + psFree(dvodb_string); + psFree(bare_query); + + // use psDBGenerateWhereConditionSQL because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + + if ((strcmp(stage,"diff") == 0)|| + (strcmp(stage,"fullforce") == 0)|| + (strcmp(stage,"fullforce_summary") == 0)) { + //diff needs the ra /deg stuff + //ff needs the ra /deg stuff + //ff summary needs the ra /deg stuff + psStringAppend(&query, " AND radeg >= %f", minra); + psStringAppend(&query, " AND radeg <= %f", maxra); + psStringAppend(&query, " AND decdeg >= %f", mindec); + psStringAppend(&query, " AND decdeg <= %f", maxdec); + } + + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + psFree(where); + + if (destreaked) { + //This picks the magicked/censored ones + if ((strcmp(stage,"cam") == 0)|| + (strcmp(stage,"stack") == 0)) { + psStringAppend(&query, " AND (camRun.magicked > 0)"); + } + // staticSky/skycal have no magicked column. + } else { + //This picks only the unmagicked/uncensored ones + //if (strcmp(stage,"cam") == 0) { + //we can now properly handle the magicked case in ippScripts, so we queue camRuns in any magic state now if -uncensored. + //psStringAppend(&query, " AND (camRun.magicked = 0)"); + //} + if (strcmp(stage,"stack") == 0) { + psStringAppend(&query, " AND (stackRun.magicked = 0)"); + } + } + + // if we grab a group of camRun/stackRuns that have multiple camRun/stackRun pointing to the same exp_id/stack_id, and that exp_id/stack_id has never before been added to addRun, we need to group by exp_id/stack_id to ensure that only 1 of the camRun/stackRun for that exp_id is added to the addRun stage. + + if (strcmp(stage,"cam") == 0) { + psStringAppend(&query, " GROUP BY exp_id"); + } + else if (strcmp(stage,"stack") == 0) { + psStringAppend(&query, " GROUP BY stack_id"); + } + else if (strcmp(stage,"staticsky") == 0) { + psStringAppend(&query, " GROUP BY sky_id, stack_id"); //some reason it needs this + } + else if (strcmp(stage,"skycal") == 0) { + psStringAppend(&query, " GROUP BY skycal_id, sky_id, stack_id "); //this needs checking, but I think it shoul be fine? it groups by lots of stuff (including stack - we only want one of each stack in there + } + //needs to be checked HAF xxx + else if (strcmp(stage,"diff") == 0) { + psStringAppend(&query, " GROUP BY diff_id, diff_skyfile_id "); //this needs checking, but I think it shoul be fine? it groups by lots of stuff (including stack - we only want one of each stack in there + } + //needs to be checked HAF xxx + else if (strcmp(stage,"fullforce") == 0) { + psStringAppend(&query, " GROUP BY ff_id, warp_id "); //this needs checking, but I think it shoul be fine? it groups by lots of stuff (including stack - we only want one of each stack in there + } + else if (strcmp(stage,"fullforce_summary") == 0) { + psStringAppend(&query, " GROUP BY ff_id "); //needs to be checked, but should be fine: want 1 ff summary cmf. + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + psError(PS_ERR_UNKNOWN, false, "query: %s\n", query); + + psTrace("addtool.c", PS_LOG_INFO,"query: \n\n%s\n\n",query); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error, \n%s\n", query); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "addRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + // loop over our list of camRun rows to check the supplied and selected dvodb and workdir values: + if (strcmp(stage,"cam") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + camRunRow *row = camRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + if (!dvodb && !row->dvodb) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, cam_id %" PRId64, row->label, row->cam_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, cam_id %" PRId64, row->label, row->cam_id); + psFree(output); + return false; + } + + psFree(row); + } + } + else if (strcmp(stage,"stack") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + stackRunRow *row = stackRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + if (!dvodb && !row->dvodb) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, stack_id %" PRId64, row->label, row->stack_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, stack_id %" PRId64, row->label, row->stack_id); + psFree(output); + return false; + } + + psFree(row); + } + } + else if (strcmp(stage,"staticsky") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + bool status = false; + psS32 num_inputs = psMetadataLookupS32(&status, md, "num_inputs"); + if (!status) { + psError(PS_ERR_UNKNOWN, true, "failed to lookup value for item num_inputs"); + return false; + } + if (num_inputs < 0) { + psError(PS_ERR_UNKNOWN, true, "invalid value for num_inputs"); + return false; + } + + staticskyRunRow *row = staticskyRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + if (!dvodb) { //there's no staticsky.dvodb + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, sky_id %" PRId64, row->label, row->sky_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, sky_id %" PRId64, row->label, row->sky_id); + psFree(output); + return false; + } + + psFree(row); + } + } + else if (strcmp(stage,"skycal") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + skycalRunRow *row = skycalRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into skycalRun"); + psFree(output); + return false; + } + + if (!dvodb) { // there's no skycalRun.dvodb + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, skycal_id %" PRId64, row->label, row->skycal_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, skycal_id %" PRId64, row->label, row->skycal_id); + psFree(output); + return false; + } + + psFree(row); + } + } + else if (strcmp(stage,"diff") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + diffRunRow *row = diffRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into diffRun"); + psFree(output); + return false; + } + + if (!dvodb) { // there's no skycalRun.dvodb + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, diff_id %" PRId64, row->label, row->diff_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, diff_id %" PRId64, row->label, row->diff_id); + psFree(output); + return false; + } + + psFree(row); + } + } + else if (strcmp(stage,"fullforce") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + fullForceRunRow *row = fullForceRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into fullforceRun"); + psFree(output); + return false; + } + + if (!dvodb) { // there's no skycalRun.dvodb + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, ff_id %" PRId64, row->label, row->ff_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, ff_id %" PRId64, row->label, row->ff_id); + psFree(output); + return false; + } + + psFree(row); + } + } + else if (strcmp(stage,"fullforce_summary") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + //i believe this is correct for ff_summary as well: + fullForceRunRow *row = fullForceRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into fullforceRun"); + psFree(output); + return false; + } + + if (!dvodb) { // there's no skycalRun.dvodb + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined dvodb: label: %s, ff_id %" PRId64, row->label, row->ff_id); + psFree(output); + return false; + } + if (!workdir && !row->workdir) { + psError(PS_ERR_UNKNOWN, false, "cannot queue addstar run without a defined workdir: label: %s, ff_id %" PRId64, row->label, row->ff_id); + psFree(output); + return false; + } + + psFree(row); + } + } + + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // would could do this "all in the database" if we didn't want the option + // of changing the label/reduction/expgroup/dvodb/etc. So we're pulling the + // data out so we have the option of changing these values or leaving the + // old values in place (i.e., passing the values through). + + + if (strcmp(stage,"cam") == 0) { + // loop over our list of camRun rows + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + + camRunRow *row = camRunObjectFromMetadata(md); + stage_id = row->cam_id; + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByCamID(config, + stage, + stage_id, + 0, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : row->dvodb, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + } + else if (strcmp(stage,"stack") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + + stackRunRow *row = stackRunObjectFromMetadata(md); + stage_id = row->stack_id; + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByCamID(config, + stage, + stage_id, + 0, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : row->dvodb, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + } + else if (strcmp(stage,"staticsky") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + bool status = false; + //psS32 num_inputs = psMetadataLookupS32(&status, md, "num_inputs"); + psS32 stack_id = psMetadataLookupS32(&status, md, "stack_id"); + + if (!status) { + psError(PS_ERR_UNKNOWN, true, "failed to lookup value for item stack_id"); + return false; + } + staticskyRunRow *row = staticskyRunObjectFromMetadata(md); + stage_id = row->sky_id; + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + // queue the exp + //for (int filter_id = 0; filter_id < num_inputs; filter_id++) { + if (!pxaddQueueByCamID(config, + stage, + stage_id, + stack_id, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : NULL, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + //} + psFree(row); + } + } + else if (strcmp(stage,"skycal") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + + skycalRunRow *row = skycalRunObjectFromMetadata(md); + stage_id = row->skycal_id; + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into skycalRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByCamID(config, + stage, + stage_id, + 0, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : NULL, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + } + else if (strcmp(stage,"diff") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + // psS64 stage_extra1 = 0; + diffRunRow *row = diffRunObjectFromMetadata(md); + stage_id = row->diff_id; + bool status = false; + psS32 stage_extra1 = psMetadataLookupS32(&status, md, "diff_skyfile_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, true, "failed to lookup value for item diff_skyfile_id"); + return false; + } + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into skycalRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByCamID(config, + stage, + stage_id, + stage_extra1, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : NULL, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + } + else if (strcmp(stage,"fullforce") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + + fullForceRunRow *row = fullForceRunObjectFromMetadata(md); + stage_id = row->ff_id; + bool status = false; + psS32 stage_extra1 = psMetadataLookupS32(&status, md, "warp_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, true, "failed to lookup value for item warp_id"); + return false; + } + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into skycalRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByCamID(config, + stage, + stage_id, + stage_extra1, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : NULL, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + } + else if (strcmp(stage,"fullforce_summary") == 0) { + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id =0; + + fullForceRunRow *row = fullForceRunObjectFromMetadata(md); + stage_id = row->ff_id; + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into fullforceRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByCamID(config, + stage, + stage_id, + 0, + workdir ? workdir : row->workdir, + reduction ? reduction : row->reduction, + label ? label : row->label, + data_group ? data_group : (row->data_group ? row->data_group : (label ? label : row->label)), + dvodb ? dvodb : NULL, + note ? note : NULL, + image_only, + minidvodb, + minidvodb_group, + minidvodb_name + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue stage %s %" PRId64,stage, stage_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-add_id", "addRun.add_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sky_id", "staticskyRun.sky_id", "=="); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalRun.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + + PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + pxcamGetSearchArgs (config, where); // most search arguments based on camera + PXOPT_COPY_STR(config->args, where, "-label", "addRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "addRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "addRun.reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-dvodb", "addRun.dvodb", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psString query = NULL; + if (strcmp(stage, "cam") == 0) { + query = psStringCopy("UPDATE addRun JOIN camRun on (cam_id = stage_id and stage = 'cam') JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id)"); + } + else if (strcmp(stage, "stack") == 0) { + query = psStringCopy("UPDATE addRun JOIN stackRun on (stack_id = stage_id and stage = 'stack')"); + } + else if (strcmp(stage, "staticsky") == 0) { + query = psStringCopy("UPDATE addRun JOIN staticskyRun on (sky_id = stage_id and stage = 'staticsky')"); + } + else if (strcmp(stage, "skycal") == 0) { + query = psStringCopy("UPDATE addRun JOIN skycalRun on (skycal_id = stage_id and stage = 'skycal')"); + } + else if (strcmp(stage, "diff") == 0) { + query = psStringCopy("UPDATE addRun JOIN diffInputSkyfile on (diff_id = stage_id and diff_skyfile_id = stage_extra1 and stage = 'diff') JOIN diffSkyfile on (diff_id, skycell_id) JOIN diffRun using (diff_id)"); + } + else if (strcmp(stage, "fullforce") == 0) { + query = psStringCopy("UPDATE addRun JOIN fullForceResult on (ff_id = stage_id and warp_id = stage_extra1 and stage = 'fullforce') JOIN fullForceRun on (ff_id)"); + } + else if (strcmp(stage, "fullforce_summary") == 0) { + query = psStringCopy("UPDATE addRun JOIN fullForceSummary on (ff_id = stage_id and stage = 'fullforce_summary') JOIN fullForceRun on (ff_id)"); + } + + + // pxUpdateRun gets parameters from config->args and runs the update query + bool result = pxUpdateRun(config, where, &query, "addRun", "add_id", + "addProcessedExp", false, false); + + psFree(query); + psFree(where); + + return result; +} + + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-add_id", "addRun.add_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sky_id", "staticskyRun.sky_id", "=="); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalRun.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_extra1", "addRun.stage_extra1", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "addRun.stage_id", "=="); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + pxcamGetSearchArgs (config, where); + pxAddLabelSearchArgs (config, where, "-label", "addRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "addRun.reduction", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(multiadd, config->args, "-multiadd", false); + + psString query = NULL; + + if (strcmp(stage, "cam") == 0) { + query = pxDataGet("addtool_find_pendingexp_cam.sql"); + } + else if (strcmp(stage, "stack") == 0) { + query = pxDataGet("addtool_find_pendingexp_stack.sql"); + } + else if (strcmp(stage, "staticsky") == 0) { + query = pxDataGet("addtool_find_pendingexp_staticsky_multi.sql"); + } + else if (strcmp(stage, "skycal") == 0) { + query = pxDataGet("addtool_find_pendingexp_skycal.sql"); + } + else if (strcmp(stage, "diff") == 0) { + query = pxDataGet("addtool_find_pendingexp_diff.sql"); + } + else if (strcmp(stage, "fullforce") == 0) { + query = pxDataGet("addtool_find_pendingexp_ff.sql"); + } + else if (strcmp(stage, "fullforce_summary") == 0) { + query = pxDataGet("addtool_find_pendingexp_ffsummary.sql"); + } + + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + if (strcmp(stage, "stack") == 0) { + //this group by is needed to join against all the warps (to get camera) + psStringAppend(&query, " GROUP BY %s", "stack_id"); + } + else if (strcmp(stage, "skycal") == 0) { + //this group by is needed to join against all the warps (to get camera) + psStringAppend(&query, " GROUP BY %s", "skycal_id"); + } + else if (strcmp(stage, "staticsky") == 0) { + //this group by is needed to join against all the warps (to get camera) + psStringAppend(&query, " GROUP BY %s", "sky_id, stage_extra1"); + } + else if (strcmp(stage, "diff") == 0) { + //this group by is needed to join against all the warps (to get camera) + if (multiadd) { + psStringAppend(&query, " GROUP BY %s", "diff_id"); + } else { + psStringAppend(&query, " GROUP BY %s", "diff_id, stage_extra1"); + } + } + else if (strcmp(stage, "fullforce") == 0) { + //this group by is needed to join against all the warps (to get camera) + if (multiadd) { + psStringAppend(&query, "GROUP BY %s", "ff_id"); + } else { + psStringAppend(&query, " GROUP BY %s", "ff_id, stage_extra1"); + } + } + else if (strcmp(stage, "fullforce_summary") == 0) { + //this group by is needed to join against all the warps (to get camera) + psStringAppend(&query, "GROUP BY %s", "ff_id"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + // printf ("%s\n", query); + //psTrace("addtool.c", PS_LOG_INFO, "used this query:\n %s\n",query); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "addPendingExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + // PXOPT_LOOKUP_S64(add_id, config->args, "-add_id", true, false); + + // optional + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_F32(dtime_addstar, config->args, "-dtime_addstar", false, false); + PXOPT_LOOKUP_STR(dvodb_path, config->args, "-dvodb_path", false, false); + PXOPT_LOOKUP_STR(minidvodb_name, config->args, "-minidvodb_name", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S64(stage_extra1, config->args, "-stage_extra1", false, false); + + PXOPT_LOOKUP_BOOL(multiadd, config->args, "-multiadd", false); + // generate restrictions + psMetadata *where = psMetadataAlloc(); + if (multiadd) { + // NOTE: these tests are not needed because the PXOPT_COPY_* calls below return FALSE on failure to find the item + // required, check that they are there + // PXOPT_LOOKUP_STR(multiaddlabel, config->args, "-multiaddlabel", true, false); + // PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + // PXOPT_LOOKUP_S64(stage_id, config->args, "-stage_id", true, false); + + // do the where + PXOPT_COPY_STR(config->args, where, "-multiaddlabel", "addRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "addRun.stage","=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "addRun.stage_id", "=="); + + //PXOPT_LOOKUP_S64(stage_id, config->args, "-stage_id", false, false); + } else { + //if not multiadd mode, then do it by add_id + // required, check for add_id + // PXOPT_LOOKUP_S64(add_id, config->args, "-add_id",true,false); + PXOPT_COPY_S64(config->args, where, "-add_id", "addRun.add_id", "=="); + + } + psString query = pxDataGet("addtool_find_pendingexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClaus = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClaus); + psFree(whereClaus); + } + psFree(where); + + if (multiadd) { + // optional -stage_extra1_list extra1,extra1,extra1 to restrict + PXOPT_LOOKUP_STR(stage_extra1_list, config->args, "-stage_extra1_list", false, false); + if (stage_extra1_list) { + psStringAppend(&query, " AND stage_extra1 IN (%s)", stage_extra1_list); + } + } + + + //if (multiadd) { + // psStringAppend(&query, "group by %s", "stage_extra1"); + //} + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i =0; idata[i]); + + addProcessedExpRow *row = addProcessedExpRowAlloc( + pendingRow->add_id, + dtime_addstar, + path_base, + dvodb_path, + fault + ); + + if (!addProcessedExpInsertObject(config->dbh, row)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + psFree(pendingRow); + return false; + } + + //if there is a minidvodb_name, set it in addRun (it's not known until it is processed) + if (minidvodb_name) { + psString setName = NULL; + psStringAppend (&setName, "UPDATE addRun set minidvodb_name = '%s' where add_id = %" PRId64, minidvodb_name, row->add_id); + if (!p_psDBRunQuery(config->dbh, setName)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + + return false; + } + } + //if there is a stage_extra1, set it in addRun (it's not known until it is processed) + if (stage_extra1) { + psString setExtra = NULL; + psStringAppend (&setExtra, "UPDATE addRun set stage_extra1 = %" PRId64, stage_extra1); + psStringAppend (&setExtra, " where add_id = %" PRId64, row->add_id); + + if (!p_psDBRunQuery(config->dbh, setExtra)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + + return false; + } + } + + + + + // since there is only one exp per 'new' set addRun.state = 'full' + // but check to make sure there are no faults + + if (!fault) { + if (!pxaddRunSetState(config, row->add_id, "full")) { + psError(PS_ERR_UNKNOWN, false, "failed to change addRun.state for add_id: %" PRId64, row->add_id); + psFree(row); + psFree(pendingRow); + return false; + } + } + psFree(row); + psFree(pendingRow); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + return true; +} + + +static bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // generate restrictions + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-add_id", "addRun.add_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "addRun.stage_id", "=="); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + pxcamGetSearchArgs (config, where); + pxAddLabelSearchArgs (config, where, "-label", "addRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "addRun.reduction", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + if (!psListLength(where->list) && + !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters (or -all) are required"); + return false; + } + psString query = NULL; + + if (strcmp (stage,"cam") == 0) { + query = pxDataGet("addtool_find_processedexp_cam.sql"); + } else if (strcmp (stage,"stack") == 0) { + query = pxDataGet("addtool_find_processedexp_stack.sql"); + } else if (strcmp (stage,"staticsky") == 0) { + query = pxDataGet("addtool_find_processedexp_staticsky_multi.sql"); + } else if (strcmp (stage,"skycal") == 0) { + query = pxDataGet("addtool_find_processedexp_skycal.sql"); + } else if (strcmp (stage,"diff") == 0) { + query = pxDataGet("addtool_find_processedexp_diff.sql"); + } else if (strcmp (stage,"fullforce") == 0) { + query = pxDataGet("addtool_find_processedexp_ff.sql"); + } else if (strcmp (stage,"fullforce_summary") == 0) { + query = pxDataGet("addtool_find_processedexp_ffsummary.sql"); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "can't find sql file for stage %s", stage); + return false; + } + + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + // we either add AND (condition) or WHERE (condition): + if (where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " AND addProcessedExp.fault != 0"); + } + if (where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND addProcessedExp.fault = 0"); + } + if (!where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " WHERE addProcessedExp.fault != 0"); + } + if (!where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " WHERE addProcessedExp.fault = 0"); + } + psStringAppend(&query, " AND stage = '%s'", stage); + + psFree(where); + + // order by add_id so that the postage stamp parser can easliy find the 'latest' astrometry + psStringAppend(&query, " ORDER BY add_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "addProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool revertprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-add_id", "addRun.add_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "addRun.stage_id", "=="); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + pxcamGetSearchArgs (config, where); + pxAddLabelSearchArgs (config, where, "-label", "addRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "addRun.reduction", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + { + psString query = NULL; + if (strcmp(stage, "cam") == 0) { + query = pxDataGet("addtool_revertprocessedexp_cam.sql"); + } + else if (strcmp(stage, "stack") == 0) { + query = pxDataGet("addtool_revertprocessedexp_stack.sql"); + } + else if (strcmp(stage, "staticsky") == 0) { + query = pxDataGet("addtool_revertprocessedexp_staticsky_multi.sql"); + } + else if (strcmp(stage, "skycal") == 0) { + query = pxDataGet("addtool_revertprocessedexp_skycal.sql"); + } + else if (strcmp(stage, "diff") == 0) { + query = pxDataGet("addtool_revertprocessedexp_diff.sql"); + } + else if (strcmp(stage, "fullforce") == 0) { + query = pxDataGet("addtool_revertprocessedexp_ff.sql"); + } + else if (strcmp(stage, "fullforce_summary") == 0) { + query = pxDataGet("addtool_revertprocessedexp_ffsummary.sql"); + } + + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(where); + return false; + } + psFree(query); + } + psFree(where); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool updateprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-add_id", "add_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + + if (!pxSetFaultCode(config->dbh, "addProcessedExp", where, fault, 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + if (!addMaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM addMask"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "addMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + char *query = "DELETE FROM addMask WHERE label = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool addminidvodbrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // required + PXOPT_LOOKUP_STR(minidvodb_group, config->args, "-set_minidvodb_group", true, false); + + //optional + PXOPT_LOOKUP_STR(minidvodb_name, config->args, "-set_minidvodb_name", false, false); + PXOPT_LOOKUP_STR(minidvodb_path, config->args, "-set_minidvodb_path", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString minidvodbpath = "NULL"; + + // I don't know how to get around the complaints of minidvodb_path can't be null. this 'fixes' it, but someone smarter can fix it properly. + if (minidvodb_path) { + minidvodbpath = minidvodb_path; + } else { + psError(PS_ERR_UNKNOWN, false, "require minidvodb_path"); + return false; + } + + if (!minidvodbRunInsert(config->dbh, + 0, // job_id + minidvodb_name, + minidvodb_group, + minidvodbpath, + "new", + 0 + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, + "should have affected one row but %" PRIu64 " rows were modified", + affected); + return false; + } + + psS64 minidvodb_id = psDBLastInsertID(config->dbh); + printf("%" PRId64 "\n", minidvodb_id); + + + if (!minidvodb_name) { + psStringAppend(&minidvodb_name, "%s.%" PRIu64,minidvodb_group,minidvodb_id); + } + if (minidvodb_path) { + psStringAppend(&minidvodb_path,"/%s",minidvodb_name); + } + + psString query = NULL; + + psStringAppend(&query, "UPDATE minidvodbRun SET minidvodb_path = '%s', minidvodb_name = '%s' where minidvodb_id = %" PRIu64";", minidvodb_path, minidvodb_name, minidvodb_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updateminidvodbrunMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_name", "minidvodb_name", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_path", "minidvodb_path", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodb_group", "=="); + + PXOPT_LOOKUP_STR(minidvodb_name, config->args, "-set_minidvodb_name", false, false); + PXOPT_LOOKUP_STR(minidvodb_path, config->args, "-set_minidvodb_path", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(minidvodb_group, config->args, "-set_minidvodb_group", false, false); + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE minidvodbRun SET "); + int cnt = 0; + psString comma = ","; + if (minidvodb_name) { + psStringAppend(&query, " minidvodb_name = '%s'", minidvodb_name); + cnt++; + } + + if (minidvodb_path) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " minidvodb_path = '%s'", minidvodb_path); + cnt++; + } + + if (state) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " state = '%s'", state); + cnt++; + } + + if (minidvodb_group) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " minidvodb_group = '%s'", minidvodb_group); + cnt++; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psFree(query); + psFree(where); + + return true; + +} + +static bool flipminidvodbrunMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_LOOKUP_STR(minidvodb_group, config->args, "-minidvodb_group",true, false); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodb_group", "=="); + +//this flips the new - > active +// and the active - > waiting in one action + + +// the first query looks to find things that are new and where all the fields are filled (ie, ready to be flipped to active) + +// start a transaction eraly so it will contain any row level locks + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString firstquery = psStringCopy("SELECT * from minidvodbRun where state = 'new' and minidvodb_name is NOT NULL and minidvodb_group IS NOT NULL and minidvodb_path IS NOT NULL"); + + psString firstwhereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&firstquery, " AND %s", firstwhereClause); + + + + if (!p_psDBRunQuery(config->dbh, firstquery)) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(firstquery); + + //we don't care what the stuff is that is found, just that there is stuff. This is a check to see that there is something in the 'new' state, before flipping (so that if there is nothing in new, it won't flip the active to waiting. the flipminidvo is just to make it easy to flip from new -> active. + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + + //ok, there's something, so flip active -> waiting + + psString query = psStringCopy("UPDATE minidvodbRun SET state = 'waiting' WHERE state = 'active' "); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + + //now flip new -> active + psString query2 = psStringCopy("UPDATE minidvodbRun SET state = 'active' WHERE state = 'new' AND minidvodb_name is NOT NULL and minidvodb_group IS NOT NULL and minidvodb_path IS NOT NULL "); + psStringAppend(&query2, " AND minidvodb_group = '%s' limit 1;", minidvodb_group); + if (!p_psDBRunQuery(config->dbh, query2)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(query); + return false; + } + + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(query); + return false; + } + psFree(query2); + psFree(where); + + return true; + +} + + +static bool checkminidvodbrunaddrunMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + //this checks to see if a minidvod_group/name is has completed addRun processing + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodbRun.minidvodb_group", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_name", "minidvodbRun.minidvodb_name", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "minidvodbRun.state", "=="); + PXOPT_LOOKUP_STR(minidvodb_group, config->args, "-minidvodb_group", false, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(all_addrun_states, config->args, "-all_addrun_states", false); + //this doesn't care what state the addRun is in (useful for counting addRuns) + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!minidvodb_group) { + psError(PXTOOLS_ERR_CONFIG, false, "minidvodb_group required"); + return false; + } + + + psString bare_query = pxDataGet("addtool_checkminidvodbaddrun.sql"); + + if (!bare_query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString where_string = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&where_string, " WHERE %s", whereClause); + psFree(whereClause); + //search params are defined above + } + + psString query = NULL; + psStringAppend(&query, bare_query, minidvodb_group, minidvodb_group); + psStringAppend(&query," %s ", where_string); + + if (!all_addrun_states) { + psStringAppend(&query, " AND (cnt2 = cnt) "); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error,\n\n%s", query); + psFree(query); + return false; + } + psFree(query); + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "minidvodbRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + + +static bool listminidvodbrunMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-minidvodb_name", "minidvodbRun.minidvodb_name", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodbRun.minidvodb_group", "=="); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbRun.minidvodb_id", "=="); + + PXOPT_COPY_STR(config->args, where, "-state", "minidvodbRun.state", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("addtool_find_minidvodbrun.sql"); + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psFree(query); + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "minidvodbRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + +return true; +} + +static bool addminidvodbprocessedMode(pxConfig *config) { + + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + // PXOPT_LOOKUP_U64(minidvodb_id, config->args, "-minidvodb_id", true, false); + PXOPT_LOOKUP_STR(minidvodb_group, config->args, "-minidvodb_group", true, false); + + // optional + PXOPT_LOOKUP_F32(dtime_relphot, config->args, "-dtime_relphot", false, false); + PXOPT_LOOKUP_F32(dtime_resort, config->args, "-dtime_resort", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + + PXOPT_LOOKUP_TIME(epoch, config->args, "-epoch", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + //generate restrictions + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbRun.minidvodb_id", "=="); + + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + psString query = pxDataGet("addtool_find_pendingmergeprocess.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClaus = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClaus); + psFree(whereClaus); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + minidvodbRunRow *pendingRow = minidvodbRunObjectFromMetadata(output->data[0]); + psFree(output); + minidvodbProcessedRow *row = minidvodbProcessedRowAlloc( + pendingRow->minidvodb_id, + dtime_resort, + dtime_relphot, + dtime_script, + epoch, + fault + ); + + if (!minidvodbProcessedInsertObject(config->dbh, row)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + psFree(pendingRow); + return false; + } + + psString query2 = NULL ; + psStringAppend(&query2, "UPDATE minidvodbRun SET state = 'merged' WHERE minidvodb_id = %'" PRIu64, row->minidvodb_id); + + if (!p_psDBRunQuery(config->dbh, query2)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query2); + return false; + } + psFree(row); + psFree(pendingRow); + + //commit the changes + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; +} + + + +static bool listminidvodbprocessedMode(pxConfig *config) { + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbProcessed.minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_name", "minidvodbRun.minidvodb_name", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodbRun.minidvodb_group", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("addtool_find_minidvodbprocessed.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + +if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + +// we either add AND (condition) or WHERE (condition): + if (where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " AND minidvodbProcessed.fault != 0"); + } + if (where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND minidvodbProcessed.fault = 0"); + } + if (!where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " WHERE minidvodbProcessed.fault != 0"); + } + if (!where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " WHERE minidvodbProcessed.fault = 0"); + } + psFree(where); + + // order by epoch + psStringAppend(&query, " ORDER BY minidvodb_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error "); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "minidvodbProcessed", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + +return true; +} + + + +static bool revertminidvodbprocessedMode(pxConfig *config) { + psMetadata *where = psMetadataAlloc(); + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbProcessed.minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "addRun.minidvodb_group", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + { + psString query = pxDataGet("addtool_revertminidvodbprocessed.sql"); + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(where); + return false; + } + psFree(query); + } + psFree(where); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + + +static bool updateminidvodbprocessedMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + // PXOPT_LOOKUP_U64(minidvodb_id, config->args, "-minidvodb_id", true, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_F32(dtime_relphot, config->args, "-set_dtime_relphot", false, false); + PXOPT_LOOKUP_F32(dtime_resort, config->args, "-set_dtime_resort", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-set_dtime_script", false, false); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbProcessed.minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_name", "minidvodbRun.minidvodb_name", "=="); + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE minidvodbProcessed JOIN minidvodbRun USING (minidvodb_id) SET "); + int cnt = 0; + psString comma = ","; + if (fault) { + psStringAppend(&query, " fault = %d", fault); + cnt++; + } + + if (dtime_relphot) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " dtime_relphot = %f", dtime_relphot); + cnt++; + } + + if (dtime_resort) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " dtime_resort = %f", dtime_resort); + cnt++; + } + + if (dtime_script) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " dtime_script = %f", dtime_script); + cnt++; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error \n%s\n",query ); + psFree(query); + return false; + } + + psFree(query); + psFree(where); + + return true; +} + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/addtool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/addtool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/addtool.h (revision 41104) @@ -0,0 +1,55 @@ +/* + * addtool.h + * + * Copyright (C) 2006 Joshua Hoblitt, Christopher Waters + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef ADDTOOL_H +#define ADDTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + ADDTOOL_MODE_NONE = 0x0, + ADDTOOL_MODE_DEFINEBYQUERY, + ADDTOOL_MODE_UPDATERUN, + ADDTOOL_MODE_PENDINGEXP, + ADDTOOL_MODE_ADDPROCESSEDEXP, + ADDTOOL_MODE_PROCESSEDEXP, + ADDTOOL_MODE_REVERTPROCESSEDEXP, + ADDTOOL_MODE_UPDATEPROCESSEDEXP, + ADDTOOL_MODE_BLOCK, + ADDTOOL_MODE_MASKED, + ADDTOOL_MODE_UNBLOCK, + ADDTOOL_MODE_PENDINGCLEANUPRUN, + ADDTOOL_MODE_PENDINGCLEANUPEXP, + ADDTOOL_MODE_DONECLEANUP, + ADDTOOL_MODE_EXPORTRUN, + ADDTOOL_MODE_IMPORTRUN, + ADDTOOL_MODE_ADDMINIDVODBRUN, + ADDTOOL_MODE_UPDATEMINIDVODBRUN, + ADDTOOL_MODE_LISTMINIDVODBRUN, + ADDTOOL_MODE_FLIPMINIDVODBRUN, + ADDTOOL_MODE_CHECKMINIDVODBRUNADDRUN, + ADDTOOL_MODE_ADDMINIDVODBPROCESSED, + ADDTOOL_MODE_LISTMINIDVODBPROCESSED, + ADDTOOL_MODE_REVERTMINIDVODBPROCESSED, + ADDTOOL_MODE_UPDATEMINIDVODBPROCESSED +} addtoolMode; + +pxConfig *addtoolConfig(pxConfig *config, int argc, char **argv); + +#endif // ADDTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/addtoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/addtoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/addtoolConfig.c (revision 41104) @@ -0,0 +1,300 @@ +/* + * addtoolConfig.c + * + * Copyright (C) 2009 Joshua Hoblitt, Chris Waters + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include + +#include "pxtools.h" +#include "pxadd.h" +#include "addtool.h" + +pxConfig *addtoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-stage", 0, "set the stage (required)", NULL); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-sky_id", 0, "search by sky_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal_id", 0); + pxcamSetSearchArgs(definebyqueryArgs); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-ff_id", 0, "search by ff_id", 0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by camRun label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by camRun data_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-filter", PS_META_DUPLICATE_OK, "search by filter", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-reduction", 0, "search by camRun reduction class", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-destreaked", 0, "queue destreaked runs", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-uncensored", 0, "queue uncensored runs", false); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-image_only", 0, "addstar image metadata but not detections", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(definebyqueryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-set_minidvodb", 0, "use minidvodb", false); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_minidvodb_group", 0, "define minidvodb_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_minidvodb_name", 0, "define minidvodb_bname", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-set_minra", 0, "min ra for query", 0); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-set_maxra", 0, "max ra for query", 360); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-set_mindec", 0, "min dec for query", -90); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-set_maxdec", 0, "max dec for query", 90); + + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-stage", 0, "set the stage (required)", NULL); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-add_id", 0, "search by add_id", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack_id", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-sky_id", 0, "search by sky_id", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal_id", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff_id",0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-ff_id", 0, "search by _ff_id",0); + pxcamSetSearchArgs(updaterunArgs); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by addRun label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by addRun state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-reduction", 0, "search by addRun reduction class", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-dvodb", 0, "search by dvodb", NULL); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dvodb", 0, "set dvodb", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-stage", 0, "set the stage (required)", NULL); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-add_id", 0, "search by add_id", 0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack_id", 0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-sky_id", 0, "search by sky_id", 0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal_id", 0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff_id",0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-ff_id", 0, "search by ff_id",0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-stage_extra1", 0, "search by stage_extra1",0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-stage_id", 0, "search by stage_id",0); + pxcamSetSearchArgs(pendingexpArgs); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by addRun label", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-reduction", 0, "search by addRun reduction class", NULL); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-multiadd", 0, "for addstar multi mode, group by stage_id ", false); + + // -addprocessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-add_id", 0, "define addtool ID (required)", 0); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-dvodb_path", 0, "define base output location", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-minidvodb_name", 0, "define minidvodb_name", NULL); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-dtime_addstar", 0, "define elapsed time for DVO insertion (seconds)", NAN); + psMetadataAddS16(addprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-stage_extra1", 0, "set stage_extra1", 0); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-stage_extra1_list", 0, "restrict multiadd ops to a set of stage_extra1 values", 0); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-stage_id", 0, "set stage_id", 0); + psMetadataAddBool(addprocessedexpArgs, PS_LIST_TAIL, "-multiadd", 0, "for addstar multi mode, group by stage_id ", false); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-multiaddlabel", 0, "label for multiadd", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-stage", 0, "select stage", NULL); + // -processedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + psMetadataAddS64(processedexpArgs, PS_LIST_TAIL, "-add_id", 0, "search by add_id", 0); + psMetadataAddS64(processedexpArgs, PS_LIST_TAIL, "-stage_id", 0, "search by stage_id", 0); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-stage", 0, "set the stage", "cam"); + pxcamSetSearchArgs(processedexpArgs); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by addRun label", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-reduction",0, "search by addRun reduction class", NULL); + + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-all", 0, "list everything without restriction", false); + + // -revertprocessedexp + psMetadata *revertprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-add_id", 0, "search by add_id", 0); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-stage_id", 0, "search by stage_id", 0); + psMetadataAddStr(revertprocessedexpArgs, PS_LIST_TAIL, "-stage", 0, "set the stage", NULL); + pxcamSetSearchArgs(revertprocessedexpArgs); + psMetadataAddStr(revertprocessedexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by addRun label", NULL); + psMetadataAddStr(revertprocessedexpArgs, PS_LIST_TAIL, "-reduction",0, "search by addRun reduction class", NULL); + + psMetadataAddBool(revertprocessedexpArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + + // -updateprocessedexp + psMetadata *updateprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedexpArgs, PS_LIST_TAIL, "-add_id", 0, "search by addtool ID", 0); + psMetadataAddS64(updateprocessedexpArgs, PS_LIST_TAIL, "-stage_id", 0, "search by stage_id", 0); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-stage", 0, "set the stage", NULL); + psMetadataAddS16(updateprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to mask out (required)", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to unmask (required)", NULL); + + // -addminidvodbruns + psMetadata *addminidvodbrunArgs = psMetadataAlloc(); + psMetadataAddStr(addminidvodbrunArgs, PS_LIST_TAIL, "-set_minidvodb_name", 0, "define minidvodb_name", NULL); + psMetadataAddStr(addminidvodbrunArgs, PS_LIST_TAIL, "-set_minidvodb_group", 0, "define minidvodb_group (required)", NULL); + psMetadataAddStr(addminidvodbrunArgs, PS_LIST_TAIL, "-set_minidvodb_path", 0, "define path for minidvodb", NULL); + psMetadataAddStr(addminidvodbrunArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + + // -updateminidvodbruns + psMetadata *updateminidvodbrunArgs = psMetadataAlloc(); + psMetadataAddU64(updateminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id ", 0); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "search by minidvodb_name (LIKE)", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_name", 0, "search by minidvodb_name (LIKE)", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_path", 0, "search by path for minidvodb", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-set_minidvodb_name", 0, "define minidvodb_name", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-set_minidvodb_path", 0, "define path for minidvodb", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-set_minidvodb_group", 0, "define path for the merged dvodb", NULL); + psMetadataAddStr(updateminidvodbrunArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + + // -listminidvodbrunArgs + psMetadata *listminidvodbrunArgs = psMetadataAlloc(); + psMetadataAddU64(listminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(listminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_name", 0, "search by minidvodb_name (LIKE)", NULL); + psMetadataAddStr(listminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "search by minidvodb.minidvodb_group", NULL); + psMetadataAddStr(listminidvodbrunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddU64(listminidvodbrunArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(listminidvodbrunArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + + //psMetadataAddBool(listminidvodbrunArgs, PS_LIST_TAIL, "-finished_addrun", 0, "limit to minidvodbs with completed addRuns (none in new state)", false); + psMetadata *flipminidvodbrunArgs = psMetadataAlloc(); + psMetadataAddStr(flipminidvodbrunArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "for the supplied minidvodb_group (required): flip the current 'new' to 'active', the current 'active' to 'waiting'", NULL); + + psMetadata *checkminidvodbrunaddrunArgs = psMetadataAlloc(); + psMetadataAddStr(checkminidvodbrunaddrunArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "for the supplied minidvodb_group (required): check if the addRun stage is complete (all in addRun.state = full) ", NULL); + psMetadataAddStr(checkminidvodbrunaddrunArgs, PS_LIST_TAIL, "-minidvodb_name", 0, "for the supplied minidvodb_name: check if the addRun stage is complete (all in addRun.state = full) ", NULL); + psMetadataAddStr(checkminidvodbrunaddrunArgs, PS_LIST_TAIL, "-state", 0, "limit by minidvodbRun state ", NULL); + psMetadataAddU64(checkminidvodbrunaddrunArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(checkminidvodbrunaddrunArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + psMetadataAddBool(checkminidvodbrunaddrunArgs, PS_LIST_TAIL, "-all_addrun_states", 0, "list all minidvodbRun.minidvodb_names, not just ones that have complete addRuns", false); + + psMetadata *addminidvodbprocessedArgs = psMetadataAlloc(); + psMetadataAddU64(addminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "define minidvodb_id (required)", 0); + psMetadataAddF32(addminidvodbprocessedArgs, PS_LIST_TAIL, "-dtime_relphot", 0, "define elapsed time for relphot (seconds)", NAN); + psMetadataAddF32(addminidvodbprocessedArgs, PS_LIST_TAIL, "-dtime_resort", 0, "define elapsed time for resort (seconds)", NAN); + psMetadataAddF32(addminidvodbprocessedArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time for script (seconds)", NAN); + psMetadataAddTime(addminidvodbprocessedArgs, PS_LIST_TAIL, "-epoch", 0, "time merge is finished", NULL); + psMetadataAddStr(addminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_group",0, "minidvodb_group", NULL); + psMetadataAddS16(addminidvodbprocessedArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + psMetadata *listminidvodbprocessedArgs = psMetadataAlloc(); + psMetadataAddU64(listminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(listminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_name", 0, "search by minidvodb_name", NULL); + psMetadataAddStr(listminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "search by minidvodb.minidvodb_group", NULL); + psMetadataAddU64(listminidvodbprocessedArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(listminidvodbprocessedArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + psMetadataAddBool(listminidvodbprocessedArgs, PS_LIST_TAIL, "-faulted", 0, "limit to faulted state", false); + + psMetadata *revertminidvodbprocessedArgs = psMetadataAlloc(); + psMetadataAddU64(revertminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(revertminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "search by addRun.minidvodb_group", NULL); + + psMetadata *updateminidvodbprocessedArgs = psMetadataAlloc(); + psMetadataAddU64(updateminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(updateminidvodbprocessedArgs, PS_LIST_TAIL, "-minidvodb_name", 0, "search by minidvodb_name", NULL); + psMetadataAddS16(updateminidvodbprocessedArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + psMetadataAddF32(updateminidvodbprocessedArgs, PS_LIST_TAIL, "-set_dtime_relphot", 0, "define elapsed time for relphot (seconds)", 0); + psMetadataAddF32(updateminidvodbprocessedArgs, PS_LIST_TAIL, "-set_dtime_resort", 0, "define elapsed time for resort (seconds)", 0); + psMetadataAddF32(updateminidvodbprocessedArgs, PS_LIST_TAIL, "-set_dtime_script", 0, "define elapsed time for script (seconds)", 0); + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from cam/skycal/forcedwarp/diff stage", ADDTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "change add run properties", ADDTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-pendingexp", "show pending exps", ADDTOOL_MODE_PENDINGEXP, pendingexpArgs); + PXOPT_ADD_MODE("-addprocessedexp", "add a processed exp", ADDTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXOPT_ADD_MODE("-processedexp", "show processed exps", ADDTOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXOPT_ADD_MODE("-revertprocessedexp", "change procesed exp properties", ADDTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpArgs); + PXOPT_ADD_MODE("-updateprocessedexp", "undo a processed exp", ADDTOOL_MODE_UPDATEPROCESSEDEXP,updateprocessedexpArgs); + PXOPT_ADD_MODE("-block", "set a label block", ADDTOOL_MODE_BLOCK, blockArgs); + PXOPT_ADD_MODE("-masked", "show blocked labels", ADDTOOL_MODE_MASKED, maskedArgs); + PXOPT_ADD_MODE("-addminidvodbrun", "create minidvodbs ", ADDTOOL_MODE_ADDMINIDVODBRUN, addminidvodbrunArgs); + PXOPT_ADD_MODE("-updateminidvodbrun", "change minidvodb properties", ADDTOOL_MODE_UPDATEMINIDVODBRUN, updateminidvodbrunArgs); + PXOPT_ADD_MODE("-listminidvodbrun", "list minidvodbs", ADDTOOL_MODE_LISTMINIDVODBRUN, listminidvodbrunArgs); + PXOPT_ADD_MODE("-flipminidvodbrun", "flip minidvodbs", ADDTOOL_MODE_FLIPMINIDVODBRUN, flipminidvodbrunArgs); + PXOPT_ADD_MODE("-checkminidvodbrunaddrun", "check minidvodbs to see if addRuns are completed", ADDTOOL_MODE_CHECKMINIDVODBRUNADDRUN, checkminidvodbrunaddrunArgs); + PXOPT_ADD_MODE("-addminidvodbprocessed","add a processed minidvodb", ADDTOOL_MODE_ADDMINIDVODBPROCESSED, addminidvodbprocessedArgs); + PXOPT_ADD_MODE("-listminidvodbprocessed","list processed minidvodbs", ADDTOOL_MODE_LISTMINIDVODBPROCESSED, listminidvodbprocessedArgs); + PXOPT_ADD_MODE("-revertminidvodbprocessed","revert processed minidvobs", ADDTOOL_MODE_REVERTMINIDVODBPROCESSED, revertminidvodbprocessedArgs); + PXOPT_ADD_MODE("-updateminidvodbprocessed","change processed minidvodb properties",ADDTOOL_MODE_UPDATEMINIDVODBPROCESSED, updateminidvodbprocessedArgs); + + + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtool.c (revision 41104) @@ -0,0 +1,2163 @@ +/* + * bgtool.c + * + * Copyright (C) 2006-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "bgtool.h" + +static bool definechipMode(pxConfig *config); +static bool updatechipMode(pxConfig *config); +static bool tochipMode(pxConfig *config); +static bool chipinputsMode(pxConfig *config); +static bool addchipMode(pxConfig *config); +static bool chipMode(pxConfig *config); +static bool advancechipMode(pxConfig *config); +static bool revertchipMode(pxConfig *config); +static bool listchipMode(pxConfig *config); +static bool definewarpMode(pxConfig *config); +static bool updatewarpMode(pxConfig *config); +static bool towarpMode(pxConfig *config); +static bool warpinputsMode(pxConfig *config); +static bool addwarpMode(pxConfig *config); +static bool warpMode(pxConfig *config); +static bool advancewarpMode(pxConfig *config); +static bool revertwarpMode(pxConfig *config); +static bool listwarpMode(pxConfig *config); + +static bool pendingcleanupchiprunMode(pxConfig *config); +static bool pendingcleanupchipimfileMode(pxConfig *config); +static bool tocleanedchipimfileMode(pxConfig *config); +static bool updatechipimfileMode(pxConfig *config); + +static bool pendingcleanupwarprunMode(pxConfig *config); +static bool pendingcleanupwarpskyfileMode(pxConfig *config); +static bool tocleanedwarpskyfileMode(pxConfig *config); + +static bool exportchipMode(pxConfig *config); +static bool importchipMode(pxConfig *config); +static bool exportwarpMode(pxConfig *config); +static bool importwarpMode(pxConfig *config); + +static bool validDataState(psString data_state); + +// Tables to import/export +typedef struct { + const char *name; // Table name + void* (*parse)(); // Parsing function + bool (*insert)(); // Insertion function +} tableData; +static const tableData chipTables[] = { + { "chipBackgroundRun", (void*)&chipBackgroundRunObjectFromMetadata, &chipBackgroundRunInsertObject }, + { "chipBackgroundImfile", (void*)&chipBackgroundImfileObjectFromMetadata, &chipBackgroundImfileInsertObject }, + { NULL, NULL, NULL } +}; +static const tableData warpTables[] = { + { "warpBackgroundRun", (void*)&warpBackgroundRunObjectFromMetadata, &warpBackgroundRunInsertObject }, + { "warpBackgroundSkyfile", (void*)&warpBackgroundSkyfileObjectFromMetadata, &warpBackgroundSkyfileInsertObject }, + { NULL, NULL, NULL } +}; + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = bgtoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(BGTOOL_MODE_DEFINECHIP, definechipMode); + MODECASE(BGTOOL_MODE_UPDATECHIP, updatechipMode); + MODECASE(BGTOOL_MODE_TOCHIP, tochipMode); + MODECASE(BGTOOL_MODE_CHIPINPUTS, chipinputsMode); + MODECASE(BGTOOL_MODE_ADDCHIP, addchipMode); + MODECASE(BGTOOL_MODE_CHIP, chipMode); + MODECASE(BGTOOL_MODE_ADVANCECHIP, advancechipMode); + MODECASE(BGTOOL_MODE_REVERTCHIP, revertchipMode); + MODECASE(BGTOOL_MODE_LISTCHIP, listchipMode); + MODECASE(BGTOOL_MODE_DEFINEWARP, definewarpMode); + MODECASE(BGTOOL_MODE_UPDATEWARP, updatewarpMode); + MODECASE(BGTOOL_MODE_TOWARP, towarpMode); + MODECASE(BGTOOL_MODE_WARPINPUTS, warpinputsMode); + MODECASE(BGTOOL_MODE_ADDWARP, addwarpMode); + MODECASE(BGTOOL_MODE_WARP, warpMode); + MODECASE(BGTOOL_MODE_ADVANCEWARP, advancewarpMode); + MODECASE(BGTOOL_MODE_REVERTWARP, revertwarpMode); + MODECASE(BGTOOL_MODE_LISTWARP, listwarpMode); + MODECASE(BGTOOL_MODE_PENDINGCLEANUPCHIPRUN, pendingcleanupchiprunMode); + MODECASE(BGTOOL_MODE_PENDINGCLEANUPCHIPIMFILE, pendingcleanupchipimfileMode); + MODECASE(BGTOOL_MODE_TOCLEANEDCHIPIMFILE, tocleanedchipimfileMode); + MODECASE(BGTOOL_MODE_UPDATECHIPIMFILE, updatechipimfileMode); + MODECASE(BGTOOL_MODE_PENDINGCLEANUPWARPRUN, pendingcleanupwarprunMode); + MODECASE(BGTOOL_MODE_PENDINGCLEANUPWARPSKYFILE, pendingcleanupwarpskyfileMode); + MODECASE(BGTOOL_MODE_TOCLEANEDWARPSKYFILE, tocleanedwarpskyfileMode); + MODECASE(BGTOOL_MODE_EXPORTCHIP, exportchipMode); + MODECASE(BGTOOL_MODE_IMPORTCHIP, importchipMode); + MODECASE(BGTOOL_MODE_EXPORTWARP, exportwarpMode); + MODECASE(BGTOOL_MODE_IMPORTWARP, importwarpMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// General functions +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + +static bool exportTables(pxConfig *config, // Configuration (with DB handle) + const char *filename, // Filename to which to write + const tableData tables[], // Tables to export (NULL terminated) + const psMetadata *where, // WHERE restrictions + bool clean // Write as cleaned? + ) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + FILE *file = fopen(filename, "w"); + if (!file) { + psError(PXTOOLS_ERR_SYS, true, "failed to open output file %s", filename); + return false; + } + + if (!pxExportVersion(config, file)) { + psError(psErrorCodeLast(), false, "failed to write dbversion to output file %s", filename); + return false; + } + + for (int i = 0; tables[i].name; i++) { + const char *name = tables[i].name; // Name of table + psString query = NULL; + psStringAppend(&query, "SELECT * FROM %s", name); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PXTOOLS_ERR_CONFIG, true, "no rows found"); + psFree(output); + return false; + } + + if (clean && + (strcmp(name, "chipBackgroundRun") == 0 || + strcmp(name, "warpBackgroundRun") == 0) && + !pxSetStateCleaned(name, "state", output)) { + psFree(output); + psError(psErrorCodeLast(), false, "pxSetStateClean failed for table %s", name); + return false; + } + + if (!ippdbPrintMetadatas(file, output, name, true)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + fclose(file); + + return true; +} + +static bool importTables(pxConfig *config, // Configuration + const char *filename, // + const tableData tables[] // Tables to read in + ) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + unsigned int badLines = 0; // Number of bad lines + psMetadata *input = psMetadataConfigRead(NULL, &badLines, filename, false); // Input file contents + if (!input) { + psError(psErrorCodeLast(), false, "Unable to parse input file %s", filename); + return false; + } + if (badLines > 0) { + psWarning("%d bad lines encountered when parsing %s", badLines, filename); + } + + if (!pxCheckImportVersion(config, input)) { + psError(psErrorCodeLast(), false, "pxCheckImportVersion failed"); + return false; + } + + // Import primary table + for (int i = 0; tables[i].name; i++) { + const char *name = tables[i].name; // Name of table + psMetadataItem *item = psMetadataLookup(input, name); // Item from input + psAssert(item, "%s not in input", name); + psAssert(item->type == PS_DATA_METADATA_MULTI, "%s not MULTI type", name); + psAssert(psListLength(item->data.list) == 1, "%s has multiple entries", name); + psMetadataItem *entry = psListGet(item->data.list, PS_LIST_HEAD); // Entry of interest + void *data = tables[i].parse(entry); // Parsed entry + if (!data) { + psError(PXTOOLS_ERR_CONFIG, false, "Unable to parse entry %s", name); + psFree(input); + return false; + } + if (!tables[0].insert(config->dbh, data)) { + psError(psErrorCodeLast(), false, "Unable to insert entry %s", name); + psFree(input); + return false; + } + } + + psFree(input); + + return true; +} + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// Functions for chip stage +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + +static bool definechipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "rawExp.obs_mode", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + PXOPT_COPY_STR(config->args, where, "-label", "chipRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-cam_label", "camRun.label", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, true, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_STR(cam_label, config->args, "-cam_label", false, false); + PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", false, false); + if (!cam_id && !cam_label) { + psError(PXTOOLS_ERR_CONFIG, true, "either cam_id or cam_label is required"); + return false; + } + + // Get chip runs to promote to chipBackgroundRun + + psString query = pxDataGet("bgtool_definechip.sql"); // Query to execute + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (destreaked) { + psStringAppend(&query, " AND chipRun.magicked > 0"); + } + + psString labelHook = psStringCopy(""); + if (!rerun) { + if (label) { + // check for run with the newly specified label + psStringAppend(&labelHook, "\nAND (chipBackgroundRun.label = '%s')", label); + } + psStringAppend(&query, "\nAND chip_bg_id IS NULL"); + } + + if (!psDBTransaction(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, labelHook)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + psFree(labelHook); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(query); + psFree(labelHook); + + psArray *output = p_psDBFetchResult(config->dbh); // Matching rows + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipBackgroundRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + chipBackgroundRunRow *row = chipBackgroundRunObjectFromMetadata(md); + if (!row) { + psError(psErrorCodeLast(), false, "failed to convert metadata into fakeRun"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + if (!chipBackgroundRunInsert(config->dbh, 0, row->chip_id, row->cam_id, "new", + workdir ? workdir : row->workdir, + label ? label : row->label, + data_group ? data_group : row->data_group, + dist_group ? dist_group : row->dist_group, + reduction ? reduction : row->reduction, + note ? note : row->note, + NULL, 0)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(row); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + return true; +} + +static bool updatechipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "chipBackgroundRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "chipBackgroundRun.data_group","LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group", "chipBackgroundRun.dist_group","LIKE"); + pxAddLabelSearchArgs(config, where, "-label", "chipBackgroundRun.label", "LIKE"); + PXOPT_COPY_TIME(config->args, where, "-registered_begin", "chipBackgroundRun.registered", ">="); + PXOPT_COPY_TIME(config->args, where, "-registered_end", "chipBackgroundRun.registered", "<"); + + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + if (destreaked) { + psMetadataAddS64(where, PS_LIST_TAIL, "chipBackgroundRun.magicked", PS_META_DUPLICATE_OK, ">", 0); + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE chipBackgroundRun"); + bool result = pxUpdateRun(config, where, &query, "chipBackgroundRun", "chip_bg_id", + "chipBackgroundImfile", true, true); + + psFree(query); + psFree(where); + + return result; +} + +static bool tochipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "chipBackgroundRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("bgtool_tochip.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString whereStr = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereStr, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereStr)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(whereStr); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "Unable to fetch result of query %s", query); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "pendingchipBackgroundRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool chipinputsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipProcessedImfile.class_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("bgtool_chipinputs.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "pendingchipBackgroundImfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addchipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(chip_bg_id, config->args, "-chip_bg_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + // optional + + PXOPT_LOOKUP_S64(magicked, config->args, "-set_magicked", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_ppbackground, config->args, "-ver_ppbackground", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_S32(maskfrac_npix, config->args, "-maskfrac_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_static, config->args, "-maskfrac_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_dynamic, config->args, "-maskfrac_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_magic, config->args, "-maskfrac_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_advisory, config->args, "-maskfrac_advisory", false, false); + + psString ver_code = pxMergeCodeVersions(ver_pslib, ver_psmodules); + ver_code = pxMergeCodeVersions(ver_code, ver_ppbackground); + ver_code = pxMergeCodeVersions(ver_code, ver_ppstats); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!chipBackgroundImfileInsert(config->dbh, chip_bg_id, class_id, path_base, "full", magicked, dtime_script, + hostname, quality, fault, ver_code, bg, bg_stdev, maskfrac_npix, + maskfrac_static, maskfrac_dynamic, maskfrac_magic, maskfrac_advisory)) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + +static bool chipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipBackgroundRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipBackgroundImfile.class_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "chipBackgroundRun.label", "LIKE"); + pxAddLabelSearchArgs(config, where, "-data_group", "chipBackgroundRun.data_group", "LIKE"); + pxAddLabelSearchArgs(config, where, "-dist_group", "chipBackgroundRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("bgtool_chip.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString magicWhere = NULL; + if (!pxmagicAddWhere(config, &magicWhere, "chipBackgroundImfile")) { + psError(psErrorCodeLast(), false, "pxMagicAddWhere failed"); + return false; + } + if (!pxspaceAddWhere(config, &magicWhere, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all && !magicWhere) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + if (magicWhere) { + psStringAppend(&query, "%s %s", psListLength(where->list) ? "AND" : "WHERE", magicWhere); + } + psFree(magicWhere); + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "chipBackgroundImfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + } + psFree(output); + + return true; +} + + +static bool advancechipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "chipBackgroundRun.label", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString select = pxDataGet("bgtool_advancechip.sql"); + if (!select) { + psError(psErrorCodeLast(), false, "failed to retrieve SQL statement"); + return false; + } + + psString selectWhere = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&selectWhere, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&select, " %s", limitString); + psFree(limitString); + } + + if (!psDBTransaction(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, select, selectWhere)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(select); + psFree(selectWhere); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(select); + psFree(selectWhere); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + bool status = true; // Status of MD lookup + psS64 chip_bg_id = psMetadataLookupS64(&status, row, "chip_bg_id"); + if (!status) { + psError(PXTOOLS_ERR_PROG, true, "failed to look up value for chip_bg_id"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psS64 magicked = psMetadataLookupS64(&status, row, "magicked"); + if (!status) { + psError(PXTOOLS_ERR_PROG, true, "failed to look up value for magicked"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + if (!p_psDBRunQueryF(config->dbh, + "UPDATE chipBackgroundRun " + "SET state = 'full', magicked = %" PRId64 " " + " WHERE chip_bg_id = %" PRId64, + magicked, chip_bg_id)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + psS64 numUpdated = psDBAffectedRows(config->dbh); + if (numUpdated != 1) { + psError(PXTOOLS_ERR_PROG, true, "should have affected 1 row"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + psFree(output); + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + +static bool revertchipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipBackgroundImfile.class_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "chipBackgroundRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "chipBackgroundImfile.fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, true, "search parameters are required"); + return false; + } + + psString query = pxDataGet("bgtool_revertchip.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numDeleted = psDBAffectedRows(config->dbh); + psLogMsg("bgtool", PS_LOG_INFO, "Deleted %d chipBackgroundImfiles", numDeleted); + + return true; +} + +static bool listchipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipProcessedImfile.class_id", "=="); + + // chip_bg_id is required + // PXOPT_LOOKUP_S64(chip_bg_id, config->args, "-chip_bg_id", true, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("bgtool_listchip.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString whereStr = psStringCopy(""); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(whereStr); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "Unable to fetch result of query %s", query); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "chipProcessedImfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool pendingcleanupchiprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs(config, where, "-label", "chipBackgroundRun.label", "=="); + pxAddLabelSearchArgs(config, where, "-data_group", "chipBackgroundRun.data_group", "=="); + + psString query = pxDataGet("bgtool_pendingcleanupchiprun.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipBackgroundRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} +static bool pendingcleanupchipimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(chip_bg_id, config->args, "-chip_bg_id", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundRun.chip_bg_id", "=="); + + psString query = pxDataGet("bgtool_pendingcleanupchipimfile.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipBackgroundImfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool change_imfile_data_state(pxConfig *config, psString data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // chip_id, class_id are required + PXOPT_LOOKUP_S64(chip_bg_id, config->args, "-chip_bg_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + psString query = pxDataGet("bgtool_change_imfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state, chip_bg_id, class_id)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + query = pxDataGet("bgtool_change_chiprun_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, chip_bg_id, data_state)) { + psFree(query); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool tocleanedchipimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "cleaned"); +} + +static bool updatechipimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + // PXOPT_LOOKUP_S64(chip_bg_id, config->args, "-chip_bg_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chipBackgroundImfile.chip_bg_id", "=="); + PXOPT_COPY_S64(config->args, where, "-class_id", "chipBackgroundImfile.class_id", "=="); + + // optional + + PXOPT_LOOKUP_STR(data_state, config->args, "-set_data_state", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + if (!data_state && !fault) { + psError(PS_ERR_UNKNOWN, true, "must supply either -set_fault or -set_data_state"); + return false; + } + char *sep = ""; + psString query = psStringCopy("UPDATE chipBackgroundImfile SET "); + if (data_state) { + if (!validDataState(data_state)) { + return false; + } + psStringAppend(&query, "%s data_state = '%s'", sep, data_state); + sep = ", "; + } + if (fault) { + psStringAppend(&query, "%s fault = %d", sep, fault); + sep = ", "; + } + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psFree(where); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool exportchipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // PXOPT_LOOKUP_S64(chip_bg_id, config->args, "-chip_bg_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chip_bg_id", "=="); + + bool status = exportTables(config, outfile, chipTables, where, clean); + + psFree(where); + return status; +} + +static bool importchipMode(pxConfig *config) +{ + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + return importTables(config, infile, chipTables); +} + +////////////////////////////////////////////////////////////////////////////////////////////////////////////// +// Functions for warp stage +////////////////////////////////////////////////////////////////////////////////////////////////////////////// + +static bool definewarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "rawExp.obs_mode", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + pxAddLabelSearchArgs(config, where, "-warp_label", "warpRun.label", "=="); + pxAddLabelSearchArgs(config, where, "-chip_bg_label", "chipBackgroundRun.label", "=="); + pxAddLabelSearchArgs(config, where, "-chip_label", "chipRun.label", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, true, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + // Get warp runs to promote to warpBackgroundRun + + psString query = pxDataGet("bgtool_definewarp.sql"); // Query to execute + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (destreaked) { + psStringAppend(&query, " AND warpRun.magicked > 0"); + } + + psString labelHook = psStringCopy(""); + if (!rerun) { + if (label) { + // check for run with the newly specified label + psStringAppend(&labelHook, "\nAND (warpBackgroundRun.label = '%s')", label); + } + psStringAppend(&query, "\nAND warp_bg_id IS NULL"); + } + + if (!psDBTransaction(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, labelHook)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(labelHook); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(labelHook); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); // Matching rows + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpBackgroundRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + psS64 chip_bg_id = psMetadataLookupS64(NULL, md, "chip_bg_id"); + + + warpBackgroundRunRow *row = warpBackgroundRunObjectFromMetadata(md); + if (!row) { + psError(psErrorCodeLast(), false, "failed to convert metadata into warpRun"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + if (!warpBackgroundRunInsert(config->dbh, 0, row->warp_id, chip_bg_id, "new", + workdir ? workdir : row->workdir, + label ? label : row->label, + data_group ? data_group : row->data_group, + dist_group ? dist_group : row->dist_group, + reduction ? reduction : row->reduction, + note ? note : row->note, + NULL, 0)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(row); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + return true; +} + +static bool updatewarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "warpBackgroundRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "warpBackgroundRun.data_group","LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group", "warpBackgroundRun.dist_group","LIKE"); + pxAddLabelSearchArgs(config, where, "-label", "warpBackgroundRun.label", "LIKE"); + PXOPT_COPY_TIME(config->args, where, "-registered_begin", "warpBackgroundRun.registered", ">="); + PXOPT_COPY_TIME(config->args, where, "-registered_end", "warpBackgroundRun.registered", "<"); + + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + if (destreaked) { + psMetadataAddS64(where, PS_LIST_TAIL, "warpBackgroundRun.magicked", PS_META_DUPLICATE_OK, ">", 0); + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE warpBackgroundRun"); + bool result = pxUpdateRun(config, where, &query, "warpBackgroundRun", "warp_bg_id", + "warpBackgroundSkyfile", true, true); + + psFree(query); + psFree(where); + + return result; +} + +static bool towarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpBackgroundRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("bgtool_towarp.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString whereStr = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereStr, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereStr)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(whereStr); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "Unable to fetch result of query %s", query); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "warpBackgroundRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool warpinputsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("bgtool_warpinputs.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "warpBackgroundSkyfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addwarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(warp_bg_id, config->args, "-warp_bg_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + // optional + + PXOPT_LOOKUP_S64(magicked, config->args, "-set_magicked", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_pswarp, config->args, "-ver_pswarp", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_S32(maskfrac_npix, config->args, "-maskfrac_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_static, config->args, "-maskfrac_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_dynamic, config->args, "-maskfrac_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_magic, config->args, "-maskfrac_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_advisory, config->args, "-maskfrac_advisory", false, false); + + psString ver_code = pxMergeCodeVersions(ver_pslib, ver_psmodules); + ver_code = pxMergeCodeVersions(ver_code, ver_pswarp); + ver_code = pxMergeCodeVersions(ver_code, ver_ppstats); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!warpBackgroundSkyfileInsert(config->dbh, warp_bg_id, skycell_id, path_base, "full", magicked, dtime_script, + hostname, quality, fault, ver_code, bg, bg_stdev, maskfrac_npix, + maskfrac_static, maskfrac_dynamic, maskfrac_magic, maskfrac_advisory)) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + +static bool warpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpBackgroundRun.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "warpBackgroundRun.chip_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpBackgroundSkyfile.skycell_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "warpBackgroundRun.label", "LIKE"); + pxAddLabelSearchArgs(config, where, "-data_group", "warpBackgroundRun.data_group", "LIKE"); + pxAddLabelSearchArgs(config, where, "-dist_group", "warpBackgroundRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("bgtool_warp.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString magicWhere = NULL; + if (!pxmagicAddWhere(config, &magicWhere, "warpBackgroundSkyfile")) { + psError(psErrorCodeLast(), false, "pxMagicAddWhere failed"); + return false; + } + if (!pxspaceAddWhere(config, &magicWhere, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all && !magicWhere) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + if (magicWhere) { + psStringAppend(&query, "%s %s", psListLength(where->list) ? "AND" : "WHERE", magicWhere); + } + psFree(magicWhere); + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "warpBackgroundSkyfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + } + psFree(output); + + return true; +} + + +static bool advancewarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warp_bg_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "label", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString select = pxDataGet("bgtool_advancewarp.sql"); + if (!select) { + psError(psErrorCodeLast(), false, "failed to retrieve SQL statement"); + return false; + } + + psString selectWhere = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&selectWhere, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&select, " %s", limitString); + psFree(limitString); + } + + if (!psDBTransaction(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, select, selectWhere)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(select); + psFree(selectWhere); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psFree(select); + psFree(selectWhere); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + bool status = true; // Status of MD lookup + psS64 warp_bg_id = psMetadataLookupS64(&status, row, "warp_bg_id"); + if (!status) { + psError(PXTOOLS_ERR_PROG, true, "failed to look up value for warp_bg_id"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + psS64 magicked = psMetadataLookupS64(&status, row, "magicked"); + if (!status) { + psError(PXTOOLS_ERR_PROG, true, "failed to look up value for magicked"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + if (!p_psDBRunQueryF(config->dbh, + "UPDATE warpBackgroundRun " + "SET state = 'full', magicked = %" PRId64 " " + " WHERE warp_bg_id = %" PRId64, + magicked, warp_bg_id)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + return false; + } + + psS64 numUpdated = psDBAffectedRows(config->dbh); + if (numUpdated != 1) { + psError(PXTOOLS_ERR_PROG, true, "should have affected 1 row"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + psFree(output); + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + +static bool revertwarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpBackgroundSkyfile.skycell_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "warpBackgroundRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "warpBackgroundSkyfile.fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, true, "search parameters are required"); + return false; + } + + psString query = pxDataGet("bgtool_revertwarp.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numDeleted = psDBAffectedRows(config->dbh); + psLogMsg("bgtool", PS_LOG_INFO, "Deleted %d warpBackgroundSkyfiles", numDeleted); + + return true; +} +static bool listwarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("bgtool_listwarp.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString whereStr = psStringCopy(""); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(whereStr); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "Unable to fetch result of query %s", query); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "warpSkyfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool pendingcleanupwarprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs(config, where, "-label", "warpBackgroundRun.label", "=="); + + psString query = pxDataGet("bgtool_pendingcleanupwarprun.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpBackgroundRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} +static bool pendingcleanupwarpskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(warp_bg_id, config->args, "-warp_bg_id", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warpBackgroundRun.warp_bg_id", "=="); + + psString query = pxDataGet("bgtool_pendingcleanupwarpskyfile.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("bgtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpBackgroundSkyfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool change_skyfile_data_state(pxConfig *config, psString data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // warp_id, skycell_id are required + PXOPT_LOOKUP_S64(warp_bg_id, config->args, "-warp_bg_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + + psString query = pxDataGet("bgtool_change_skyfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state, warp_bg_id, skycell_id)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + query = pxDataGet("bgtool_change_warprun_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, warp_bg_id, data_state)) { + psFree(query); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool tocleanedwarpskyfileMode(pxConfig *config) +{ + return change_skyfile_data_state(config, "cleaned"); +} + +static bool exportwarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // PXOPT_LOOKUP_S64(warp_bg_id, config->args, "-warp_bg_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warp_bg_id", "=="); + + bool status = exportTables(config, outfile, warpTables, where, clean); + + psFree(where); + return status; +} + +static bool importwarpMode(pxConfig *config) +{ + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + return importTables(config, infile, warpTables); +} + +static bool validDataState(psString data_state) +{ + // NOTE: can't use pxIsValidState because + // update, scrubbed and purged are not supported for the backround stages (yet?) + if (!strcmp(data_state, "new") || + !strcmp(data_state, "full") || + !strcmp(data_state, "drop") || + !strcmp(data_state, "cleaned") || + !strcmp(data_state, "goto_cleaned") || + !strcmp(data_state, "error_cleaned")) { + return true; + } else { + psError(PS_ERR_UNKNOWN, true, "%s is not a valid value data_state", data_state); + return false; + } +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtool.h (revision 41104) @@ -0,0 +1,65 @@ +/* + * bgtool.h + * + * Copyright (C) 2006-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef BGTOOL_H +#define BGTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + BGTOOL_MODE_NONE = 0x0, + // Chip stage + BGTOOL_MODE_DEFINECHIP, + BGTOOL_MODE_UPDATECHIP, + BGTOOL_MODE_TOCHIP, + BGTOOL_MODE_CHIPINPUTS, + BGTOOL_MODE_ADDCHIP, + BGTOOL_MODE_CHIP, + BGTOOL_MODE_ADVANCECHIP, + BGTOOL_MODE_REVERTCHIP, + BGTOOL_MODE_LISTCHIP, + BGTOOL_MODE_UPDATECHIPIMFILE, + // Warp stage + BGTOOL_MODE_DEFINEWARP, + BGTOOL_MODE_UPDATEWARP, + BGTOOL_MODE_TOWARP, + BGTOOL_MODE_WARPINPUTS, + BGTOOL_MODE_ADDWARP, + BGTOOL_MODE_WARP, + BGTOOL_MODE_ADVANCEWARP, + BGTOOL_MODE_REVERTWARP, + BGTOOL_MODE_LISTWARP, + BGTOOL_MODE_UPDATEWARPSKYFILE, + // Cleanups + BGTOOL_MODE_PENDINGCLEANUPCHIPRUN, + BGTOOL_MODE_PENDINGCLEANUPCHIPIMFILE, + BGTOOL_MODE_TOCLEANEDCHIPIMFILE, + BGTOOL_MODE_PENDINGCLEANUPWARPRUN, + BGTOOL_MODE_PENDINGCLEANUPWARPSKYFILE, + BGTOOL_MODE_TOCLEANEDWARPSKYFILE, + // Exporting + BGTOOL_MODE_EXPORTCHIP, + BGTOOL_MODE_IMPORTCHIP, + BGTOOL_MODE_EXPORTWARP, + BGTOOL_MODE_IMPORTWARP, +} bgtoolMode; + +pxConfig *bgtoolConfig(pxConfig *config, int argc, char **argv); + +#endif // BGTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/bgtoolConfig.c (revision 41104) @@ -0,0 +1,487 @@ +/* + * bgtoolConfig.c + * + * Copyright (C) 2006-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "bgtool.h" + +pxConfig *bgtoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definechip + psMetadata *definechipArgs = psMetadataAlloc(); + psMetadataAddS64(definechipArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddS64(definechipArgs, PS_LIST_TAIL, "-cam_id", 0, "choose cam_id", 0); + psMetadataAddS64(definechipArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(definechipArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definechipArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(definechipArgs, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-comment", 0, "search by comment field (LIKE comparison)", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by observation mode", NULL); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "search by min solar angle", NAN); + psMetadataAddF32(definechipArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "search by max solar angle", NAN); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-label", 0, "search on chipRun label", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-cam_label", 0, "search on camRun label", NULL); + psMetadataAddBool(definechipArgs, PS_LIST_TAIL, "-destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddBool(definechipArgs, PS_LIST_TAIL, "-rerun", 0, "re-run data?", false); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definechipArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + + psMetadataAddTime(definechipArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddBool(definechipArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definechipArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + + // -updatechip + psMetadata *updatechipArgs = psMetadataAlloc(); + psMetadataAddS64(updatechipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label (LIKE comparison)", NULL); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-dist_group", 0, "search by dist_group (LIKE comparison)", NULL); + psMetadataAddTime(updatechipArgs, PS_LIST_TAIL, "-registered_begin", 0, "search by registration time (>=)", NULL); + psMetadataAddTime(updatechipArgs, PS_LIST_TAIL, "-registered_end", 0, "search by registration time (<)", NULL); + psMetadataAddBool(updatechipArgs, PS_LIST_TAIL, "-destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddS64(updatechipArgs, PS_LIST_TAIL, "-magicked", 0, "define magicked status", 0); + psMetadataAddS64(updatechipArgs, PS_LIST_TAIL, "-not_destreaked", 0, "define magicked status", 0); + psMetadataAddBool(updatechipArgs, PS_LIST_TAIL, "-pretend", 0, "only pretend to run the query", false); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", 0); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updatechipArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -tochip + psMetadata *tochipArgs = psMetadataAlloc(); + psMetadataAddS64(tochipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddStr(tochipArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddU64(tochipArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tochipArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -chipinputs + psMetadata *chipinputsArgs = psMetadataAlloc(); + psMetadataAddS64(chipinputsArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddStr(chipinputsArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", 0); + psMetadataAddU64(chipinputsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(chipinputsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addchip + psMetadata *addchipArgs = psMetadataAlloc(); + psMetadataAddS64(addchipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "define chip_bg_id (required)", 0); + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-class_id", 0, "define class_id (required)", NULL); + + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location (required)", NULL); + psMetadataAddS64(addchipArgs, PS_LIST_TAIL, "-set_magicked", 0, "define if this skycell has been magicked", 0); + psMetadataAddF32(addchipArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + psMetadataAddS16(addchipArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddS16(addchipArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-ver_ppbackground", 0, "define ppBackground version", NULL); + psMetadataAddStr(addchipArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddF64(addchipArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addchipArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddS32(addchipArgs, PS_LIST_TAIL, "-maskfrac_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(addchipArgs, PS_LIST_TAIL, "-maskfrac_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(addchipArgs, PS_LIST_TAIL, "-maskfrac_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(addchipArgs, PS_LIST_TAIL, "-maskfrac_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(addchipArgs, PS_LIST_TAIL, "-maskfrac_advisory", 0, "define advisory mask fraction", NAN); + + // -chip + psMetadata *chipArgs = psMetadataAlloc(); + psMetadataAddS64(chipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddS64(chipArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddStr(chipArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", NULL); + psMetadataAddS64(chipArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp", 0); + psMetadataAddStr(chipArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", NULL); + psMetadataAddS16(chipArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(chipArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddStr(chipArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by data_group", NULL); + psMetadataAddStr(chipArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by dist_group", NULL); + pxmagicAddArguments(chipArgs); + pxspaceAddArguments(chipArgs); + psMetadataAddBool(chipArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(chipArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(chipArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -advancechip + psMetadata *advancechipArgs = psMetadataAlloc(); + psMetadataAddS64(advancechipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddStr(advancechipArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label ", NULL); + psMetadataAddU64(advancechipArgs, PS_LIST_TAIL, "-limit", 0, "search limit", 0); + + // -revertchip + psMetadata *revertchipArgs = psMetadataAlloc(); + psMetadataAddS64(revertchipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddStr(revertchipArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", NULL); + psMetadataAddStr(revertchipArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS16(revertchipArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertchipArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + + // -listchip + psMetadata *listchipArgs = psMetadataAlloc(); + psMetadataAddS64(listchipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id (required)", 0); + psMetadataAddStr(listchipArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", NULL); + psMetadataAddBool(listchipArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(listchipArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupchiprun + psMetadata *pendingcleanupchiprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupchiprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list runs for specified label", NULL); + psMetadataAddStr(pendingcleanupchiprunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "list runs for specified data_group", NULL); + psMetadataAddBool(pendingcleanupchiprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupchiprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupchipimfile + psMetadata *pendingcleanupchipimfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingcleanupchipimfileArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id (required)", 0); + psMetadataAddBool(pendingcleanupchipimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupchipimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tocleanedchipimfile + psMetadata *tocleanedchipimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedchipimfileArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip ID (required)", 0); + psMetadataAddStr(tocleanedchipimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", NULL); + psMetadataAddBool(tocleanedchipimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(tocleanedchipimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -updatechipimfile + psMetadata *updatechipimfileArgs = psMetadataAlloc(); + psMetadataAddS64(updatechipimfileArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip ID (required)", 0); + psMetadataAddStr(updatechipimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id (required)", NULL); + psMetadataAddStr(updatechipimfileArgs, PS_LIST_TAIL, "-set_data_state", 0, "new value for data_state", NULL); + psMetadataAddS16(updatechipimfileArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + + // -exportchip + psMetadata *exportchipArgs = psMetadataAlloc(); + psMetadataAddS64(exportchipArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "export this chip_bg_id (required)", 0); + psMetadataAddStr(exportchipArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddBool(exportchipArgs, PS_LIST_TAIL, "-clean", 0, "export run in cleaned state", false); + + // -importchip + psMetadata *importchipArgs = psMetadataAlloc(); + psMetadataAddStr(importchipArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + + // -definewarp + psMetadata *definewarpArgs = psMetadataAlloc(); + psMetadataAddS64(definewarpArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp_id", 0); + psMetadataAddS64(definewarpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(definewarpArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definewarpArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(definewarpArgs, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-comment", 0, "search by comment field (LIKE comparison)", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by observation mode", NULL); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "search by min solar angle", NAN); + psMetadataAddF32(definewarpArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "search by max solar angle", NAN); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-warp_label", PS_META_DUPLICATE_OK, "search on warpRun label", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-chip_label", PS_META_DUPLICATE_OK, "search on chipRun label", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-chip_bg_label", PS_META_DUPLICATE_OK, "search on chipBackgroundRun label", NULL); + psMetadataAddBool(definewarpArgs, PS_LIST_TAIL, "-destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddBool(definewarpArgs, PS_LIST_TAIL, "-rerun", 0, "rerun data?", false); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definewarpArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + + psMetadataAddTime(definewarpArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddBool(definewarpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definewarpArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + + // -updatewarp + psMetadata *updatewarpArgs = psMetadataAlloc(); + psMetadataAddS64(updatewarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label (LIKE comparison)", NULL); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-dist_group", 0, "search by dist_group (LIKE comparison)", NULL); + psMetadataAddTime(updatewarpArgs, PS_LIST_TAIL, "-registered_begin", 0, "search by registration time (>=)", NULL); + psMetadataAddTime(updatewarpArgs, PS_LIST_TAIL, "-registered_end", 0, "search by registration time (<)", NULL); + psMetadataAddBool(updatewarpArgs, PS_LIST_TAIL, "-destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddBool(updatewarpArgs, PS_LIST_TAIL, "-not_destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddS64(updatewarpArgs, PS_LIST_TAIL, "-magicked", 0, "define magicked status", 0); + psMetadataAddBool(updatewarpArgs, PS_LIST_TAIL, "-pretend", 0, "only pretend to run the query", false); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", 0); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updatewarpArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -towarp + psMetadata *towarpArgs = psMetadataAlloc(); + psMetadataAddS64(towarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddStr(towarpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddU64(towarpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(towarpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -warpinputs + psMetadata *warpinputsArgs = psMetadataAlloc(); + psMetadataAddS64(warpinputsArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddStr(warpinputsArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", 0); + psMetadataAddU64(warpinputsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(warpinputsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addwarp + psMetadata *addwarpArgs = psMetadataAlloc(); + psMetadataAddS64(addwarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "define warp_bg_id (required)", 0); + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell_id (required)", NULL); + + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", 0); + psMetadataAddS64(addwarpArgs, PS_LIST_TAIL, "-set_magicked", 0, "define if this skycell has been magicked", 0); + psMetadataAddF32(addwarpArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddS16(addwarpArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddS16(addwarpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-ver_pswarp", 0, "define pswarp version", NULL); + psMetadataAddStr(addwarpArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddF64(addwarpArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addwarpArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddS32(addwarpArgs, PS_LIST_TAIL, "-maskfrac_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(addwarpArgs, PS_LIST_TAIL, "-maskfrac_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(addwarpArgs, PS_LIST_TAIL, "-maskfrac_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(addwarpArgs, PS_LIST_TAIL, "-maskfrac_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(addwarpArgs, PS_LIST_TAIL, "-maskfrac_advisory", 0, "define advisory mask fraction", NAN); + + // -warp + psMetadata *warpArgs = psMetadataAlloc(); + psMetadataAddS64(warpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddS64(warpArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp_id", 0); + psMetadataAddS64(warpArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddStr(warpArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + psMetadataAddS16(warpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(warpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddStr(warpArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by data_group", NULL); + psMetadataAddStr(warpArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by dist_group", NULL); + pxmagicAddArguments(warpArgs); + pxspaceAddArguments(warpArgs); + psMetadataAddBool(warpArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(warpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(warpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -advancewarp + psMetadata *advancewarpArgs = psMetadataAlloc(); + psMetadataAddS64(advancewarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddStr(advancewarpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label ", NULL); + psMetadataAddU64(advancewarpArgs, PS_LIST_TAIL, "-limit", 0, "search limit", 0); + + // -revertwarp + psMetadata *revertwarpArgs = psMetadataAlloc(); + psMetadataAddS64(revertwarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddStr(revertwarpArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + psMetadataAddStr(revertwarpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS16(revertwarpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertwarpArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + + // -listwarp + psMetadata *listwarpArgs = psMetadataAlloc(); + psMetadataAddS64(listwarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id (required)", 0); + psMetadataAddStr(listwarpArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + psMetadataAddBool(listwarpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(listwarpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupwarprun + psMetadata *pendingcleanupwarprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupwarprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddStr(pendingcleanupwarprunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "list blocks for specified data_group", NULL); + psMetadataAddBool(pendingcleanupwarprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupwarprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupwarpskyfile + psMetadata *pendingcleanupwarpskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingcleanupwarpskyfileArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id (required)", 0); + psMetadataAddBool(pendingcleanupwarpskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupwarpskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tocleanwarpskyfile + psMetadata *tocleanedwarpskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedwarpskyfileArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id (required)", 0); + psMetadataAddStr(tocleanedwarpskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id (required)", 0); + psMetadataAddStr(tocleanedwarpskyfileArgs, PS_LIST_TAIL, "-state", 0, "cleaned state to set", NULL); + + // -updatewarpskyfile + psMetadata *updatewarpskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(updatewarpskyfileArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by chip ID (required)", 0); + psMetadataAddStr(updatewarpskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id (required)", NULL); + psMetadataAddStr(updatewarpskyfileArgs, PS_LIST_TAIL, "-set_state", 0, "new value for data_state", NULL); + psMetadataAddS16(updatewarpskyfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -exportwarp + psMetadata *exportwarpArgs = psMetadataAlloc(); + psMetadataAddS64(exportwarpArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "export this warp_bg_id (required)", 0); + psMetadataAddStr(exportwarpArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddBool(exportwarpArgs, PS_LIST_TAIL, "-clean", 0, "export run in cleaned state", false); + + // -importwarp + psMetadata *importwarpArgs = psMetadataAlloc(); + psMetadataAddStr(importwarpArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + + psFree(now); + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definechip", "", BGTOOL_MODE_DEFINECHIP, definechipArgs); + PXOPT_ADD_MODE("-updatechip", "", BGTOOL_MODE_UPDATECHIP, updatechipArgs); + PXOPT_ADD_MODE("-tochip", "", BGTOOL_MODE_TOCHIP, tochipArgs); + PXOPT_ADD_MODE("-chipinputs", "", BGTOOL_MODE_CHIPINPUTS, chipinputsArgs); + PXOPT_ADD_MODE("-addchip", "", BGTOOL_MODE_ADDCHIP, addchipArgs); + PXOPT_ADD_MODE("-chip", "", BGTOOL_MODE_CHIP, chipArgs); + PXOPT_ADD_MODE("-advancechip", "", BGTOOL_MODE_ADVANCECHIP, advancechipArgs); + PXOPT_ADD_MODE("-revertchip", "", BGTOOL_MODE_REVERTCHIP, revertchipArgs); + PXOPT_ADD_MODE("-listchip", "", BGTOOL_MODE_LISTCHIP, listchipArgs); + PXOPT_ADD_MODE("-pendingcleanupchiprun", "", BGTOOL_MODE_PENDINGCLEANUPCHIPRUN, pendingcleanupchiprunArgs); + PXOPT_ADD_MODE("-pendingcleanupchipimfile", "", BGTOOL_MODE_PENDINGCLEANUPCHIPIMFILE, pendingcleanupchipimfileArgs); + PXOPT_ADD_MODE("-tocleanedchipimfile", "", BGTOOL_MODE_TOCLEANEDCHIPIMFILE, tocleanedchipimfileArgs); + PXOPT_ADD_MODE("-updatechipimfile", "", BGTOOL_MODE_UPDATECHIPIMFILE, updatechipimfileArgs); + PXOPT_ADD_MODE("-exportchip", "", BGTOOL_MODE_EXPORTCHIP, exportchipArgs); + PXOPT_ADD_MODE("-importchip", "", BGTOOL_MODE_IMPORTCHIP, importchipArgs); + + PXOPT_ADD_MODE("-definewarp", "", BGTOOL_MODE_DEFINEWARP, definewarpArgs); + PXOPT_ADD_MODE("-updatewarp", "", BGTOOL_MODE_UPDATEWARP, updatewarpArgs); + PXOPT_ADD_MODE("-towarp", "", BGTOOL_MODE_TOWARP, towarpArgs); + PXOPT_ADD_MODE("-warpinputs", "", BGTOOL_MODE_WARPINPUTS, warpinputsArgs); + PXOPT_ADD_MODE("-addwarp", "", BGTOOL_MODE_ADDWARP, addwarpArgs); + PXOPT_ADD_MODE("-warp", "", BGTOOL_MODE_WARP, warpArgs); + PXOPT_ADD_MODE("-advancewarp", "", BGTOOL_MODE_ADVANCEWARP, advancewarpArgs); + PXOPT_ADD_MODE("-revertwarp", "", BGTOOL_MODE_REVERTWARP, revertwarpArgs); + PXOPT_ADD_MODE("-listwarp", "", BGTOOL_MODE_LISTWARP, listwarpArgs); + PXOPT_ADD_MODE("-pendingcleanupwarprun", "", BGTOOL_MODE_PENDINGCLEANUPWARPRUN, pendingcleanupwarprunArgs); + PXOPT_ADD_MODE("-pendingcleanupwarpskyfile", "", BGTOOL_MODE_PENDINGCLEANUPWARPSKYFILE, pendingcleanupwarpskyfileArgs); + PXOPT_ADD_MODE("-tocleanedwarpskyfile", "", BGTOOL_MODE_TOCLEANEDWARPSKYFILE, tocleanedwarpskyfileArgs); + PXOPT_ADD_MODE("-updatewarpskyfile", "", BGTOOL_MODE_UPDATEWARPSKYFILE, updatewarpskyfileArgs); + PXOPT_ADD_MODE("-exportwarp", "", BGTOOL_MODE_EXPORTWARP, exportwarpArgs); + PXOPT_ADD_MODE("-importwarp", "", BGTOOL_MODE_IMPORTWARP, importwarpArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/caltool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/caltool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/caltool.c (revision 41104) @@ -0,0 +1,261 @@ +/* + * caltool.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include + +#include "pxtools.h" +#include "caltool.h" + +static bool adddbMode(pxConfig *config); +static bool dbsMode(pxConfig *config); +static bool addrunMode(pxConfig *config); +static bool runsMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = caltoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(CALTOOL_MODE_ADDDB, adddbMode); + MODECASE(CALTOOL_MODE_DBS, dbsMode); + MODECASE(CALTOOL_MODE_ADDRUN, addrunMode); + MODECASE(CALTOOL_MODE_RUNS, runsMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool adddbMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", true, false); + + if (!calDBInsert(config->dbh, + 0, // cal_id + dvodb, + "active" // state + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + + } + + return false; +} + + +static bool dbsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM calDB"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("caltool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "calDB", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool addrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(cal_id, config->args, "-cal_id", true, false); + PXOPT_LOOKUP_STR(region, config->args, "-region", true, false); + PXOPT_LOOKUP_STR(last_step, config->args, "-last_step", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + + if (!calRunInsert(config->dbh, + (psS64)atoll(cal_id), + region, + last_step, + state + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + + } + + return false; +} + + +static bool runsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-cal_id", "cal_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM calRun"); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "WHERE fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "WHERE fault = 0"); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " AND %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("caltool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "calRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/caltool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/caltool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/caltool.h (revision 41104) @@ -0,0 +1,35 @@ +/* + * caltool.h + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef CALTOOL_H +#define CALTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + CALTOOL_MODE_NONE = PXTOOL_MODE_NONE, + CALTOOL_MODE_DBS, + CALTOOL_MODE_ADDDB, + CALTOOL_MODE_ADDRUN, + CALTOOL_MODE_RUNS +} caltoolMode; + +pxConfig *caltoolConfig(pxConfig *config, int argc, char **argv); + +#endif // CALTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/caltoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/caltoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/caltoolConfig.c (revision 41104) @@ -0,0 +1,104 @@ +/* + * caltoolConfig.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "caltool.h" + +pxConfig *caltoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *time = psTimeGetNow(PS_TIME_TAI); + psString now = psTimeToISO(time); + psFree(time); + + // -adddb + psMetadata *adddbArgs = psMetadataAlloc(); + psMetadataAddStr(adddbArgs, PS_LIST_TAIL, "-dvodb", 0, "define DVO db (required)", NULL); + + // -dbs + psMetadata *dbsArgs = psMetadataAlloc(); + psMetadataAddU64(dbsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(dbsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addrun + psMetadata *addrunArgs = psMetadataAlloc(); + psMetadataAddStr(addrunArgs, PS_LIST_TAIL, "-cal_id", 0, "define calibration DB ID (required)", NULL); + psMetadataAddStr(addrunArgs, PS_LIST_TAIL, "-region", 0, "define region (required)", NULL); + psMetadataAddStr(addrunArgs, PS_LIST_TAIL, "-last_step", 0, "define last step (required)", NULL); + psMetadataAddStr(addrunArgs, PS_LIST_TAIL, "-state", 0, "define state (required)", NULL); + + // -runs + psMetadata *runsArgs = psMetadataAlloc(); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-cal_id", 0, "search for calibration ID", NULL); + psMetadataAddU64(runsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(runsArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(runsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-adddb", "add a DVO calibration DB", CALTOOL_MODE_ADDDB, adddbArgs); + PXOPT_ADD_MODE("-dbs", "list DVO calibration DBs", CALTOOL_MODE_DBS, dbsArgs); + PXOPT_ADD_MODE("-addrun", "add the results of a calibration run", CALTOOL_MODE_ADDRUN, addrunArgs); + PXOPT_ADD_MODE("-runs", "list the results of calibration runs", CALTOOL_MODE_RUNS, runsArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/camtool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/camtool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/camtool.c (revision 41104) @@ -0,0 +1,1463 @@ +/* + * camtool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxcam.h" +#include "camtool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool pendingexpMode(pxConfig *config); +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedexpMode(pxConfig *config); +static bool processedexpMode(pxConfig *config); +static bool revertprocessedexpMode(pxConfig *config); +static bool updateprocessedexpMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); +static bool pendingcleanuprunMode(pxConfig *config); +static bool pendingcleanupexpMode(pxConfig *config); +static bool donecleanupMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = camtoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(CAMTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(CAMTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(CAMTOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(CAMTOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(CAMTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(CAMTOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(CAMTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpMode); + MODECASE(CAMTOOL_MODE_UPDATEPROCESSEDEXP, updateprocessedexpMode); + MODECASE(CAMTOOL_MODE_BLOCK, blockMode); + MODECASE(CAMTOOL_MODE_MASKED, maskedMode); + MODECASE(CAMTOOL_MODE_UNBLOCK, unblockMode); + MODECASE(CAMTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunMode); + MODECASE(CAMTOOL_MODE_PENDINGCLEANUPEXP, pendingcleanupexpMode); + MODECASE(CAMTOOL_MODE_DONECLEANUP, donecleanupMode); + MODECASE(CAMTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(CAMTOOL_MODE_IMPORTRUN, importrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "=="); // define using chipRun label + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "chipRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "rawExp.obs_mode", "=="); + + if (!psListLength(where->list) && + !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-set_end_stage", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + // default + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("camtool_find_chip_id.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // would could do this "all in the database" if we didn't want the option + // of changing the label/reduction/expgroup/dvodb/etc. So we're pulling the + // data out so we have the option of changing these values or leaving the + // old values in place (i.e., passing the values through). + + // if end_stage is warp (or NULL), check for valid tess_id + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status; + char *end_stage = psMetadataLookupStr(&status, md, "end_stage"); + if (end_stage && strcasecmp(end_stage, "warp")) continue; + + char *raw_tess_id = psMetadataLookupStr(&status, md, "tess_id"); + if (raw_tess_id || tess_id) continue; + + char *label = psMetadataLookupStr(&status, md, "label"); + psS64 exp_id = psMetadataLookupS64(&status, md, "exp_id"); + + if (!status) { + psError(PS_ERR_UNKNOWN, false, "cannot queue analysis to WARP without a defined tess id: label: %s, exp_id %" PRId64, label, exp_id); + psFree(output); + return false; + } + } + + // loop over our list of chipRun rows + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + chipRunRow *row = chipRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into chipRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxcamQueueByChipID(config, + row->chip_id, + workdir ? workdir : row->workdir, + label ? label : row->label, + data_group ? data_group: row->data_group, + dist_group ? dist_group: row->dist_group, + reduction ? reduction : row->reduction, + expgroup ? expgroup : row->expgroup, + dvodb ? dvodb : row->dvodb, + tess_id ? tess_id : row->tess_id, + end_stage ? end_stage : row->end_stage, + row->magicked, + note + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue chip_id: %" PRId64, row->chip_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "camRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group","camRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "camRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "camRun.reduction", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psString query = psStringCopy("UPDATE camRun JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id)"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "camRun", "cam_id", "camProcessedExp", true, true); + if (!result) { + psError(psErrorCodeLast(), false, "pxUpdateRun failed"); + } + + psFree(query); + psFree(where); + + return result; +} + + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "camRun.reduction", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("camtool_pendingexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, cam_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camPendingExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "camRun.reduction", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "camRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "camProcessedExp.class_id", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("camtool_find_pendingimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", true, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + + // optional + PXOPT_LOOKUP_F32(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F32(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F32(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F32(bias, config->args, "-bias", false, false); + PXOPT_LOOKUP_F32(bias_stdev, config->args, "-bias_stdev", false, false); + PXOPT_LOOKUP_F32(fringe_0, config->args, "-fringe_0", false, false); + PXOPT_LOOKUP_F32(fringe_1, config->args, "-fringe_1", false, false); + PXOPT_LOOKUP_F32(fringe_2, config->args, "-fringe_2", false, false); + PXOPT_LOOKUP_F32(sigma_ra, config->args, "-sigma_ra", false, false); + PXOPT_LOOKUP_F32(sigma_dec, config->args, "-sigma_dec", false, false); + PXOPT_LOOKUP_F32(ap_resid, config->args, "-ap_resid", false, false); + PXOPT_LOOKUP_F32(ap_resid_stdev, config->args, "-ap_resid_stdev", false, false); + + PXOPT_LOOKUP_F32(zpt_obs, config->args, "-zpt_obs", false, false); + PXOPT_LOOKUP_F32(zpt_err, config->args, "-zpt_err", false, false); + PXOPT_LOOKUP_F32(zpt_uq, config->args, "-zpt_uq", false, false); + PXOPT_LOOKUP_F32(zpt_lq, config->args, "-zpt_lq", false, false); + + PXOPT_LOOKUP_F32(fwhm_major, config->args, "-fwhm_major", false, false); + PXOPT_LOOKUP_F32(fwhm_major_lq, config->args, "-fwhm_major_lq", false, false); + PXOPT_LOOKUP_F32(fwhm_major_uq, config->args, "-fwhm_major_uq", false, false); + + PXOPT_LOOKUP_F32(fwhm_minor, config->args, "-fwhm_minor", false, false); + PXOPT_LOOKUP_F32(fwhm_minor_lq, config->args, "-fwhm_minor_lq", false, false); + PXOPT_LOOKUP_F32(fwhm_minor_uq, config->args, "-fwhm_minor_uq", false, false); + + PXOPT_LOOKUP_F32(iq_fwhm_major, config->args, "-iq_fwhm_major", false, false); + PXOPT_LOOKUP_F32(iq_fwhm_major_err, config->args, "-iq_fwhm_major_err", false, false); + PXOPT_LOOKUP_F32(iq_fwhm_minor, config->args, "-iq_fwhm_minor", false, false); + PXOPT_LOOKUP_F32(iq_fwhm_minor_err, config->args, "-iq_fwhm_minor_err", false, false); + + PXOPT_LOOKUP_F32(iq_m2, config->args, "-iq_m2", false, false); + PXOPT_LOOKUP_F32(iq_m2_err, config->args, "-iq_m2_err", false, false); + PXOPT_LOOKUP_F32(iq_m2_uq, config->args, "-iq_m2_uq", false, false); + PXOPT_LOOKUP_F32(iq_m2_lq, config->args, "-iq_m2_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m2c, config->args, "-iq_m2c", false, false); + PXOPT_LOOKUP_F32(iq_m2c_err, config->args, "-iq_m2c_err", false, false); + PXOPT_LOOKUP_F32(iq_m2c_uq, config->args, "-iq_m2c_uq", false, false); + PXOPT_LOOKUP_F32(iq_m2c_lq, config->args, "-iq_m2c_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m2s, config->args, "-iq_m2s", false, false); + PXOPT_LOOKUP_F32(iq_m2s_err, config->args, "-iq_m2s_err", false, false); + PXOPT_LOOKUP_F32(iq_m2s_uq, config->args, "-iq_m2s_uq", false, false); + PXOPT_LOOKUP_F32(iq_m2s_lq, config->args, "-iq_m2s_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m3, config->args, "-iq_m3", false, false); + PXOPT_LOOKUP_F32(iq_m3_err, config->args, "-iq_m3_err", false, false); + PXOPT_LOOKUP_F32(iq_m3_uq, config->args, "-iq_m3_uq", false, false); + PXOPT_LOOKUP_F32(iq_m3_lq, config->args, "-iq_m3_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m4, config->args, "-iq_m4", false, false); + PXOPT_LOOKUP_F32(iq_m4_err, config->args, "-iq_m4_err", false, false); + PXOPT_LOOKUP_F32(iq_m4_uq, config->args, "-iq_m4_uq", false, false); + PXOPT_LOOKUP_F32(iq_m4_lq, config->args, "-iq_m4_lq", false, false); + + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_F32(dtime_astrom, config->args, "-dtime_astrom", false, false); + PXOPT_LOOKUP_F32(dtime_addstar, config->args, "-dtime_addstar", false, false); + + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_S32(n_stars, config->args, "-n_stars", false, false); + PXOPT_LOOKUP_S32(n_psfstars, config->args, "-n_psfstars", false, false); + PXOPT_LOOKUP_S32(n_iqstars, config->args, "-n_iqstars", false, false); + PXOPT_LOOKUP_S32(n_extended, config->args, "-n_extended", false, false); + PXOPT_LOOKUP_S32(n_cr, config->args, "-n_cr", false, false); + PXOPT_LOOKUP_S32(n_astrom, config->args, "-n_astrom", false, false); + + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + PXOPT_LOOKUP_S16(background_model, config->args, "-background_model", false, false); + PXOPT_LOOKUP_S64(astrom_chips, config->args, "-astrom_chips", false, false); + + + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_STR(ver_psastro, config->args, "-ver_psastro", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_STR(ver_ppimage, config->args, "-ver_ppimage", false, false); + PXOPT_LOOKUP_STR(ver_streaks, config->args, "-ver_streaks", false, false); + + PXOPT_LOOKUP_S32(maskfrac_ref_npix, config->args, "-maskfrac_ref_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_ref_static, config->args, "-maskfrac_ref_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_ref_dynamic, config->args, "-maskfrac_ref_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_ref_magic, config->args, "-maskfrac_ref_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_ref_advisory, config->args, "-maskfrac_ref_advisory", false, false); + + PXOPT_LOOKUP_S32(maskfrac_max_npix, config->args, "-maskfrac_max_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_max_static, config->args, "-maskfrac_max_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_max_dynamic, config->args, "-maskfrac_max_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_max_magic, config->args, "-maskfrac_max_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_max_advisory, config->args, "-maskfrac_max_advisory", false, false); + + PXOPT_LOOKUP_F32(ast_r0, config->args, "-ast_r0", false, false); + PXOPT_LOOKUP_F32(ast_d0, config->args, "-ast_d0", false, false); + PXOPT_LOOKUP_F32(ast_t0, config->args, "-ast_t0", false, false); + PXOPT_LOOKUP_F32(ast_s0, config->args, "-ast_s0", false, false); + PXOPT_LOOKUP_F32(ast_rs, config->args, "-ast_rs", false, false); + PXOPT_LOOKUP_F32(ast_ds, config->args, "-ast_ds", false, false); + + // we store actual detection efficiency by adding in zpt_obs + PXOPT_LOOKUP_F32(deteff_inst, config->args, "-deteff_inst", false, false); + PXOPT_LOOKUP_F32(deteff_inst_lq, config->args, "-deteff_inst_lq", false, false); + PXOPT_LOOKUP_F32(deteff_inst_uq, config->args, "-deteff_inst_uq", false, false); + // error is dd + PXOPT_LOOKUP_F32(deteff_err, config->args, "-deteff_inst_err", false, false); + psF32 deteff = NAN; + psF32 deteff_uq = NAN; + psF32 deteff_lq = NAN; + if (isfinite(zpt_obs)) { + if (isfinite(deteff_inst)) { + deteff = deteff_inst + zpt_obs; + } + if (isfinite(deteff_inst_uq)) { + deteff_uq = deteff_inst_uq + zpt_obs; + } + if (isfinite(deteff_inst_lq)) { + deteff_lq = deteff_inst_lq + zpt_obs; + } + } + + psString software_ver = NULL; + if ((ver_pslib)&&(ver_psmodules)) { + software_ver = pxMergeCodeVersions(ver_pslib,ver_psmodules); + } + if (ver_psphot) { + software_ver = pxMergeCodeVersions(software_ver,ver_psphot); + } + if (ver_psastro) { + software_ver = pxMergeCodeVersions(software_ver,ver_psastro); + } + if (ver_ppstats) { + software_ver = pxMergeCodeVersions(software_ver,ver_ppstats); + } + if (ver_ppimage) { + software_ver = pxMergeCodeVersions(software_ver,ver_ppimage); + } + if (ver_streaks) { + software_ver = pxMergeCodeVersions(software_ver,ver_streaks); + } + +// Get this from the chipRun +// PXOPT_LOOKUP_S64(magicked, config->args, "-magicked", false, false); + + // generate restrictions + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + + psString query = pxDataGet("camtool_addprocessedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClaus = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClaus); + psFree(whereClaus); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + bool status; + psS64 chip_magicked = psMetadataLookupS64(&status, output->data[0], "chip_magicked"); + + camRunRow *pendingRow = camRunObjectFromMetadata(output->data[0]); + psFree(output); + camProcessedExpRow *row = camProcessedExpRowAlloc( + pendingRow->cam_id, + uri, + bg, + bg_stdev, + bg_mean_stdev, + bias, + bias_stdev, + fringe_0, + fringe_1, + fringe_2, + sigma_ra, + sigma_dec, + ap_resid, + ap_resid_stdev, + zpt_obs, + zpt_err, + zpt_lq, + zpt_uq, + fwhm_major, + fwhm_major_lq, + fwhm_major_uq, + fwhm_minor, + fwhm_minor_lq, + fwhm_minor_uq, + + iq_fwhm_major, + iq_fwhm_major_err, + iq_fwhm_minor, + iq_fwhm_minor_err, + + iq_m2, + iq_m2_err, + iq_m2_lq, + iq_m2_uq, + iq_m2c, + iq_m2c_err, + iq_m2c_lq, + iq_m2c_uq, + iq_m2s, + iq_m2s_err, + iq_m2s_lq, + iq_m2s_uq, + iq_m3, + iq_m3_err, + iq_m3_lq, + iq_m3_uq, + iq_m4, + iq_m4_err, + iq_m4_lq, + iq_m4_uq, + dtime_script, + dtime_astrom, + dtime_addstar, + hostname, + n_stars, + n_psfstars, + n_iqstars, + n_extended, + n_cr, + n_astrom, + path_base, + fault, + software_ver, + maskfrac_ref_npix, + maskfrac_ref_static, + maskfrac_ref_dynamic, + maskfrac_ref_magic, + maskfrac_ref_advisory, + maskfrac_max_npix, + maskfrac_max_static, + maskfrac_max_dynamic, + maskfrac_max_magic, + maskfrac_max_advisory, + deteff, + deteff_err, + deteff_lq, + deteff_uq, + quality, + background_model, + astrom_chips, + + ast_r0, + ast_d0, + ast_t0, + ast_s0, + ast_rs, + ast_ds + ); + + if (!camProcessedExpInsertObject(config->dbh, row)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + psFree(pendingRow); + return false; + } + + if (fault) { + psFree(row); + psFree(pendingRow); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + // else continue on... + + + // Set camRun.software_ver to the appropriate value + if (!pxSetRunSoftware(config, "camRun", "cam_id", cam_id, software_ver)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to set camRun.software_ver for cam_id: %" PRId64, + cam_id); + psFree(output); + return(false); + } + // Set chipRun.maskfrac* to the appropriate values. + if (maskfrac_ref_npix) { + if (!pxCamSetRunMaskfrac(config, "camRun", "cam_id",cam_id, + (float) maskfrac_ref_npix, maskfrac_ref_static, + maskfrac_ref_dynamic, maskfrac_ref_magic, maskfrac_ref_advisory, + (float) maskfrac_max_npix, maskfrac_max_static, + maskfrac_max_dynamic, maskfrac_max_magic, maskfrac_max_advisory)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to set camRun.maskstats for cam_id: %" PRId64, + cam_id); + psFree(output); + return(false); + } + } + + // since there is only one exp per 'new' set camRun.state = 'full' + // propagate magicked state from chipRun + if (!pxcamRunSetState(config, row->cam_id, "full", chip_magicked)) { + psError(PS_ERR_UNKNOWN, false, "failed to change camRun.state for cam_id: %" PRId64, row->cam_id); + psFree(row); + psFree(pendingRow); + return false; + } + + psFree(row); + + // EAM: NULL for end_stage means go as far as possible + // Also, we can run fake even if tess_id is not defined + // but stop if quality is non-zero. + if ((quality > 0) || (pendingRow->end_stage && psStrcasestr(pendingRow->end_stage, "cam"))) { + psFree(pendingRow); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + + if (!pxfakeQueueByCamID(config, + pendingRow->cam_id, + pendingRow->workdir, + pendingRow->label, + pendingRow->data_group, + pendingRow->dist_group, + pendingRow->reduction, + pendingRow->expgroup, + pendingRow->dvodb, + pendingRow->tess_id, + pendingRow->end_stage, + NULL // note does not propagate + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to queue new fakeRun"); + psFree(pendingRow); + return false; + } + + psFree(pendingRow); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + + // generate restrictions + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "=="); + pxAddLabelSearchArgs (config, where, "-data_group", "camRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-reduction", "camRun.reduction", "=="); + PXOPT_COPY_S16(config->args, where, "-background_model", "camProcessedExp.background_model", "=="); + + psString where2 = NULL; + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + if (!pxmagicAddWhere(config, &where2, "chipRun")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + if (!psListLength(where->list) && !where2 && + !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters (or -all) are required"); + return false; + } + + psString query = pxDataGet("camtool_find_processedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + // we either add AND (condition) or WHERE (condition): + if (where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " AND camProcessedExp.fault != 0"); + } + if (where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND camProcessedExp.fault = 0"); + } + if (!where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " WHERE camProcessedExp.fault != 0"); + } + if (!where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " WHERE camProcessedExp.fault = 0"); + } + psFree(where); + if (where2) { + psStringAppend(&query, " %s", where2); + psFree(where2); + } + + + if (pstamp_order) { + // put runs in order of exposure id with newest chip Runs first + // The postage stamp parser depends on this behavior + psStringAppend(&query, " ORDER BY exp_id, cam_id DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool revertprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "camRun.reduction", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "camProcessedExp.fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + psString whereClause = NULL; + { + psString query = pxDataGet("camtool_revertprocessedexp.sql"); + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(where); + return false; + } + psFree(query); + } + psFree(where); + + int numDeleted = psDBAffectedRows(config->dbh); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psLogMsg("camtool", PS_LOG_INFO, "Deleted %d camProcessedExps", numDeleted); + + { + psString query = pxDataGet("camtool_revertupdatedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (whereClause) { + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + } + int numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("camtool", PS_LOG_INFO, "Updated %d camProcessedExps", numUpdated); + + return true; +} + + +static bool updateprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + PXOPT_LOOKUP_S16(background_model, config->args, "-set_background_model", false, false); + if (background_model) { + // CZW Copied from warptool.c + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + PXOPT_COPY_S16(config->args, values, "-set_background_model", "background_model", "=="); + long rows = psDBUpdateRows(config->dbh, "camProcessedExp", where, values); + psFree(values); + psFree(where); + if (!rows) { + psError(PS_ERR_UNKNOWN, true, "no rows changed"); + } + } + else { + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class", "class", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + if (!pxSetFaultCode(config->dbh, "camProcessedExp", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + } + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + if (!camMaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM camMask"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + char *query = "DELETE FROM camMask WHERE label = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "=="); + + psString query = pxDataGet("camtool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camPendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool pendingcleanupexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", false, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + if (cam_id) { + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + } + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + psString query = pxDataGet("camtool_pendingcleanupexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camPendingCleanupExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psString query = pxDataGet("camtool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "camDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 2; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-cam_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + + ExportTable tables [] = { + {"camRun", "camtool_export_run.sql"}, + {"camProcessedExp", "camtool_export_processed_exp.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + if (!strcmp(tables[i].tableName, "camRun")) { + if (!pxSetStateCleaned("camRun", "state", output)) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + +#ifdef notdef + fprintf (stderr, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); +#endif + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + + psMetadataItem *item = psMetadataLookup (input, "camRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + camRunRow *camRun = camRunObjectFromMetadata (entry->data.md); + camRunInsertObject (config->dbh, camRun); + + // fprintf (stdout, "---- cam run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + item = psMetadataLookup (input, "camProcessedExp"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + for (int i = 0; i < item->data.list->n; i++) { + psMetadataItem *entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + camProcessedExpRow *camProcessedExp = camProcessedExpObjectFromMetadata (entry->data.md); + camProcessedExpInsertObject (config->dbh, camProcessedExp); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/camtool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/camtool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/camtool.h (revision 41104) @@ -0,0 +1,47 @@ +/* + * camtool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef CAMTOOL_H +#define CAMTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + CAMTOOL_MODE_NONE = 0x0, + CAMTOOL_MODE_DEFINEBYQUERY, + CAMTOOL_MODE_UPDATERUN, + CAMTOOL_MODE_PENDINGEXP, + CAMTOOL_MODE_PENDINGIMFILE, + CAMTOOL_MODE_ADDPROCESSEDEXP, + CAMTOOL_MODE_PROCESSEDEXP, + CAMTOOL_MODE_REVERTPROCESSEDEXP, + CAMTOOL_MODE_UPDATEPROCESSEDEXP, + CAMTOOL_MODE_BLOCK, + CAMTOOL_MODE_MASKED, + CAMTOOL_MODE_UNBLOCK, + CAMTOOL_MODE_PENDINGCLEANUPRUN, + CAMTOOL_MODE_PENDINGCLEANUPEXP, + CAMTOOL_MODE_DONECLEANUP, + CAMTOOL_MODE_EXPORTRUN, + CAMTOOL_MODE_IMPORTRUN +} camtoolMode; + +pxConfig *camtoolConfig(pxConfig *config, int argc, char **argv); + +#endif // CAMTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/camtoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/camtoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/camtoolConfig.c (revision 41104) @@ -0,0 +1,345 @@ +/* + * camtoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include + +#include "pxtools.h" +#include "pxcam.h" +#include "camtool.h" + +pxConfig *camtoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + // XXX need to allow multiple chip_ids + // XXX need to allow multiple exp_ids + psMetadata *definebyqueryArgs = psMetadataAlloc(); + pxcamSetSearchArgs(definebyqueryArgs); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by chipRun label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-reduction", 0, "search by chipRun reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-data_group", 0, "search by chipRun data_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by rawExp obs_mode", NULL); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_expgroup", 0, "define exposure group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tess ID", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_end_stage", 0, "define end stage", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actual modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + // XXX need to allow multiple cam_ids + // XXX need to allow multiple chip_ids + // XXX need to allow multiple exp_ids + psMetadata *updaterunArgs = psMetadataAlloc(); + pxcamSetSearchArgs(updaterunArgs); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by camRun label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by camRun data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by camRun state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-reduction", 0, "search by camRun reduction class", NULL); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + pxmagicAddArguments(updaterunArgs); + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + pxcamSetSearchArgs(pendingexpArgs); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by camRun label", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-reduction", 0, "search by camRun reduction class", NULL); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + pxcamSetSearchArgs(pendingimfileArgs); + psMetadataAddS64(pendingimfileArgs, PS_LIST_TAIL, "-cam_id", 0, "search by camtool ID", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by camRun label", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-reduction",0, "search by camRun reduction class", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // XXX is this used? psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class", 0, "search by class", NULL); + + // -addprocessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-cam_id", 0, "define camtool ID (required)", 0); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-uri", 0, "define URI (required)", NULL); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-bias", 0, "define bias", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-bias_stdev", 0, "define bias stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fringe_0", 0, "define fringe term 0", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fringe_1", 0, "define fringe term 1", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fringe_2", 0, "define fringe term 2", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-sigma_ra", 0, "define exposure E ra", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-sigma_dec", 0, "define exposure E dec", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ap_resid", 0, "define aperture residual", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ap_resid_stdev", 0, "define aperture residual stdev", NAN); + + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-zpt_obs", 0, "define observed zero point", 0); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-zpt_err", 0, "define observed zero point error", 0); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-zpt_uq", 0, "define observed zero point upper quartile", 0); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-zpt_lq", 0, "define observed zero point lower quartile", 0); + + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fwhm_major", 0, "define fwhm (major axis; arcsec)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fwhm_major_lq", 0, "define fwhm (major axis; arcsec)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fwhm_major_uq", 0, "define fwhm (major axis; arcsec)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fwhm_minor", 0, "define fwhm (minor axis; arcsec)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fwhm_minor_lq", 0, "define fwhm (minor axis; arcsec)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-fwhm_minor_uq", 0, "define fwhm (minor axis; arcsec)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_fwhm_major", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_fwhm_major_err", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_fwhm_minor", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_fwhm_minor_err", 0, "define moment value m2 mean", NAN); + + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2_err", 0, "define moment value m2 stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2_lq", 0, "define moment value m2 lower quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2_uq", 0, "define moment value m2 upper quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2c", 0, "define moment value m2c mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2c_err", 0, "define moment value m2c stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2c_lq", 0, "define moment value m2c lower quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2c_uq", 0, "define moment value m2c upper quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2s", 0, "define moment value m2s mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2s_err", 0, "define moment value m2s stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2s_lq", 0, "define moment value m2s lower quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m2s_uq", 0, "define moment value m2s upper quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m3", 0, "define moment value m3 mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m3_err", 0, "define moment value m3 stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m3_lq", 0, "define moment value m3 lower quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m3_uq", 0, "define moment value m3 upper quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m4", 0, "define moment value m4 mean", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m4_err", 0, "define moment value m4 stdev", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m4_lq", 0, "define moment value m4 lower quartile", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-iq_m4_uq", 0, "define moment value m4 upper quartile", NAN); + + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-dtime_astrom", 0, "define elapsed time for astrometry (seconds)", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-dtime_addstar", 0, "define elapsed time for DVO insertion (seconds)", NAN); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-n_stars", 0, "define number of stars", 0); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-n_psfstars", 0, "define number of PSF stars", 0); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-n_iqstars", 0, "define number of IQ stars", 0); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-n_extended", 0, "define number of extended objects", 0); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-n_cr", 0, "define number of cosmic rays", 0); + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-n_astrom", 0, "define number of astrometry reference objects", 0); + + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddS16(addprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(addprocessedexpArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddBool(addprocessedexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-magicked", 0, "set magicked", 0); + + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_psastro", 0, "define psastro version", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_ppimage", 0, "define ppImage version", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-ver_streaks", 0, "define streaksremove version", NULL); + + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_ref_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_ref_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_ref_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_ref_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_ref_advisory", 0, "define advisory mask fraction", NAN); + + psMetadataAddS32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_max_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_max_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_max_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_max_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-maskfrac_max_advisory", 0, "define advisory mask fraction", NAN); + + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-deteff_inst", 0, "define deteff", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-deteff_inst_err", 0, "define deteff_err", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-deteff_inst_lq", 0, "define deteff_lq", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-deteff_inst_uq", 0, "define deteff_uq", NAN); + psMetadataAddS16(addprocessedexpArgs, PS_LIST_TAIL, "-background_model", 0, "set background_model value", 0); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-astrom_chips", 0, "chips with successful astrom", 0); + + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ast_r0", 0, "define boresite offset in RA", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ast_d0", 0, "define boresite offset in DEC", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ast_t0", 0, "define boresite angle", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ast_s0", 0, "define bosite scale", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ast_rs", 0, "define bosite scatter in RA", NAN); + psMetadataAddF32(addprocessedexpArgs, PS_LIST_TAIL, "-ast_ds", 0, "define bosite scatter in DEC", NAN); + + // -processedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + pxcamSetSearchArgs(processedexpArgs); + psMetadataAddS64(processedexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by camRun label", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by camRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-reduction",0, "search by camRun reduction class", NULL); + pxspaceAddArguments(processedexpArgs); + pxmagicAddArguments(processedexpArgs); + + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order the results for the postage stamp server", false); + + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-all", 0, "list everything without restriction", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddS16(processedexpArgs, PS_LIST_TAIL, "-background_model", 0, "search by background_model value", 0); + + // -revertprocessedexp + // XXX need to allow multiple cam_ids + // XXX need to allow multiple chip_ids + // XXX need to allow multiple exp_ids + psMetadata *revertprocessedexpArgs = psMetadataAlloc(); + pxcamSetSearchArgs(revertprocessedexpArgs); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddStr(revertprocessedexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by camRun label", NULL); + psMetadataAddStr(revertprocessedexpArgs, PS_LIST_TAIL, "-reduction",0, "search by camRun reduction class", NULL); + psMetadataAddS16(revertprocessedexpArgs, PS_LIST_TAIL, "-code", 0, "search by fault code", 0); + + psMetadataAddBool(revertprocessedexpArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + psMetadataAddS16(revertprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -updateprocessedexp + // XXX allow full search options? + psMetadata *updateprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by camtool ID", 0); + psMetadataAddS64(updateprocessedexpArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chiptool ID", 0); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-class", 0, "search by class", NULL); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(updateprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", INT16_MAX); + psMetadataAddS16(updateprocessedexpArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality", 0); + psMetadataAddS16(updateprocessedexpArgs, PS_LIST_TAIL, "-set_background_model", 0, "set background_model value", 0); + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to mask out (required)", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to unmask (required)", NULL); + + // -pendingcleanuprun + // XXX allow full search options? + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupexp + // XXX allow full search options? + psMetadata *pendingcleanupexpArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by camera ID", 0); + psMetadataAddStr(pendingcleanupexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", NULL); + psMetadataAddBool(pendingcleanupexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-cam_id", 0, "export this camera ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "export tables as cleaned", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from chip stage", CAMTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "change cam run properties", CAMTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-pendingexp", "show pending exposures", CAMTOOL_MODE_PENDINGEXP, pendingexpArgs); + PXOPT_ADD_MODE("-pendingimfile", "show pending imfiles", CAMTOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXOPT_ADD_MODE("-addprocessedexp", "add a processed exposure", CAMTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXOPT_ADD_MODE("-processedexp", "show processed exposures", CAMTOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXOPT_ADD_MODE("-revertprocessedexp", "undo a processed exposure", CAMTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpArgs); + PXOPT_ADD_MODE("-updateprocessedexp", "changed processed exp properties", CAMTOOL_MODE_UPDATEPROCESSEDEXP,updateprocessedexpArgs); + PXOPT_ADD_MODE("-block", "set a label block", CAMTOOL_MODE_BLOCK, blockArgs); + PXOPT_ADD_MODE("-masked", "show blocked labels", CAMTOOL_MODE_MASKED, maskedArgs); + PXOPT_ADD_MODE("-unblock", "remove a label block", CAMTOOL_MODE_UNBLOCK, unblockArgs); + PXOPT_ADD_MODE("-pendingcleanuprun", "show runs that need to be cleaned up", CAMTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunArgs); + PXOPT_ADD_MODE("-pendingcleanupexp", "show exposures for cleanup runs", CAMTOOL_MODE_PENDINGCLEANUPEXP, pendingcleanupexpArgs); + PXOPT_ADD_MODE("-donecleanup", "show runs that have been cleaned", CAMTOOL_MODE_DONECLEANUP, donecleanupArgs); + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", CAMTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", CAMTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptool.c (revision 41104) @@ -0,0 +1,2167 @@ +/* + * chiptool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "pxchip.h" + +#include "chiptool.h" +#include "camtool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool definecopyMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedimfileMode(pxConfig *config); +static bool processedimfileMode(pxConfig *config); +static bool revertprocessedimfileMode(pxConfig *config); +static bool updateprocessedimfileMode(pxConfig *config); +static bool dropprocessedimfileMode(pxConfig *config); +static bool advanceexpMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unmaskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); +static bool pendingcleanuprunMode(pxConfig *config); +static bool pendingcleanupimfileMode(pxConfig *config); +static bool revertcleanupMode(pxConfig *config); +static bool donecleanupMode(pxConfig *config); +static bool runMode(pxConfig *config); +static bool tocleanedimfileMode(pxConfig *config); +static bool tofullimfileMode(pxConfig *config); +static bool topurgedimfileMode(pxConfig *config); +static bool toscrubbedimfileMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); +static bool runstateMode(pxConfig *config); +static bool setimfiletoupdateMode(pxConfig *config); +static bool listrunMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = chiptoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(CHIPTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(CHIPTOOL_MODE_DEFINECOPY, definecopyMode); + MODECASE(CHIPTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(CHIPTOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(CHIPTOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileMode); + MODECASE(CHIPTOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(CHIPTOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileMode); + MODECASE(CHIPTOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileMode); + MODECASE(CHIPTOOL_MODE_DROPPROCESSEDIMFILE, dropprocessedimfileMode); + MODECASE(CHIPTOOL_MODE_ADVANCEEXP, advanceexpMode); + MODECASE(CHIPTOOL_MODE_BLOCK, blockMode); + MODECASE(CHIPTOOL_MODE_MASKED, maskedMode); + MODECASE(CHIPTOOL_MODE_UNMASKED, unmaskedMode); + MODECASE(CHIPTOOL_MODE_UNBLOCK, unblockMode); + MODECASE(CHIPTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunMode); + MODECASE(CHIPTOOL_MODE_PENDINGCLEANUPIMFILE, pendingcleanupimfileMode); + MODECASE(CHIPTOOL_MODE_REVERTCLEANUP, revertcleanupMode); + MODECASE(CHIPTOOL_MODE_DONECLEANUP, donecleanupMode); + MODECASE(CHIPTOOL_MODE_RUN, runMode); + MODECASE(CHIPTOOL_MODE_TOCLEANEDIMFILE, tocleanedimfileMode); + MODECASE(CHIPTOOL_MODE_TOFULLIMFILE, tofullimfileMode); + MODECASE(CHIPTOOL_MODE_TOPURGEDIMFILE, topurgedimfileMode); + MODECASE(CHIPTOOL_MODE_TOSCRUBBEDIMFILE, toscrubbedimfileMode); + MODECASE(CHIPTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(CHIPTOOL_MODE_IMPORTRUN, importrunMode); + MODECASE(CHIPTOOL_MODE_RUNSTATE, runstateMode); + MODECASE(CHIPTOOL_MODE_SETIMFILETOUPDATE, setimfiletoupdateMode); + MODECASE(CHIPTOOL_MODE_LISTRUN, listrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +// Queue exposures for chip processing +static bool queue_exposures(pxConfig *config, // Configuration + const psArray *exps, // Exposures information + const char *workdir, // Working directory, or NULL to inherit + const char *label, // Label, or NULL to inherit + const char *data_group, // Data group, or NULL to inherit + const char *dist_group, // Distribution group, or NULL to inherit + const char *reduction, // Reduction class, or NULL to inherit + const char *expgroup, // Exposure group + const char *dvodb, // DVO database, or NULL to inherit + const char *tess_id, // Tessellation identifier, or NULL to inherit + const char *end_stage, // End stage, or NULL to inherit + const char *note // Note + ) +{ + // start a transaction so we don't end up with an exp without any associated + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // if end_stage is warp (or NULL), check for valid tess_id + for (long i = 0; i < psArrayLength(exps); i++) { + psMetadata *md = exps->data[i]; + + bool status; + char *end_stage = psMetadataLookupStr(&status, md, "end_stage"); + if (end_stage && strcasecmp(end_stage, "warp")) continue; + + char *raw_tess_id = psMetadataLookupStr(&status, md, "tess_id"); + if (raw_tess_id || tess_id) continue; + + char *label = psMetadataLookupStr(&status, md, "label"); + psS64 exp_id = psMetadataLookupS64(&status, md, "exp_id"); + + if (!status) { + psError(PS_ERR_UNKNOWN, false, + "cannot queue analysis to WARP without a defined tess id: label: %s, exp_id %" PRId64, + label, exp_id); + return false; + } + } + + +# define GET_VALUE(PTYPE,CTYPE,VALUE,NAME) \ + PTYPE VALUE; \ + { \ + bool status; \ + VALUE = psMetadataLookup##CTYPE(&status, md, NAME); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", NAME); \ + return false; \ + } \ + } + + // loop over our list of exp_ids + for (long i = 0; i < psArrayLength(exps); i++) { + psMetadata *md = exps->data[i]; + + rawExpRow *row = rawExpObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into chipRun"); + return false; + } + + GET_VALUE (psS64, S64, exp_id, "exp_id"); + GET_VALUE (psString, Str, raw_workdir, "workdir"); +#ifdef notyet + GET_VALUE (psString, Str, old_workdir, "old_workdir"); + GET_VALUE (psString, Str, old_data_group, "old_data_group"); +#else + const char *old_workdir = NULL; + const char *old_data_group = NULL; +#endif + GET_VALUE (psString, Str, raw_label, "label"); + GET_VALUE (psString, Str, raw_reduction, "reduction"); + // GET_VALUE (psString, Str, raw_expgroup, "expgroup"); + GET_VALUE (psString, Str, raw_dvodb, "dvodb"); + GET_VALUE (psString, Str, raw_tess_id, "tess_id"); + GET_VALUE (psString, Str, raw_end_stage, "end_stage"); + + if (!row->exp_id) { + psError(PS_ERR_UNKNOWN, false, "failed to find value for exp_id"); + return false; + } + + // queue the exp + if (!pxchipQueueByExpTag(config, + exp_id, + workdir ? workdir : (old_workdir ? old_workdir : raw_workdir), + label ? label : raw_label, + data_group ? data_group : (old_data_group ? old_data_group : (label ? label : raw_label)), + dist_group, + reduction ? reduction : raw_reduction, + // expgroup ? expgroup : raw_expgroup, + // XXX how does expgroup get defined? + expgroup, + dvodb ? dvodb : raw_dvodb, + tess_id ? tess_id : raw_tess_id, + end_stage ? end_stage : raw_end_stage, + note + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue exp_id: %" PRId64, exp_id); + return false; + } + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // rawExp only + pxAddLabelSearchArgs (config, where, "-label", "newExp.label", "LIKE"); // define using newExp label + + // psListLength(where->list) is at least 1 because exp_type defaults to "object" + // so we require a list longer than 1 entry + if ((psListLength(where->list) <= 1) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-set_end_stage", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // default + PXOPT_LOOKUP_BOOL(unique, config->args, "-unique", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + PXOPT_LOOKUP_BOOL(randomOrder, config->args, "-random", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("chiptool_find_rawexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + + psFree(whereClause); + psFree(where); + + psString labelHook = NULL; + if (unique) { + psStringAppend(&labelHook, "\nAND chipRun.label = '%s'", label); + psStringAppend(&query, " AND chip_id IS NULL"); + } + + // treat limit == 0 as "no limit" + if (randomOrder) { + psStringAppend(&query, " %s", "ORDER BY RAND()"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, labelHook ? labelHook : "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!queue_exposures(config, output, workdir, label, data_group, dist_group, reduction, expgroup, + dvodb, tess_id, end_stage, note)) { + psError(psErrorCodeLast(), false, "Unable to queue exposures for chip."); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool definecopyMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // rawExp only + pxAddLabelSearchArgs (config, where, "-label", "chipOld.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipOld.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "chipOld.state", "=="); + + // psListLength(where->list) is at least 1 because exp_type defaults to "object" + // so we require a list longer than 1 entry + if ((psListLength(where->list) <= 1) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-set_end_stage", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + // default + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("chiptool_definecopy.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!queue_exposures(config, output, workdir, label, data_group, dist_group, reduction, expgroup, + dvodb, tess_id, end_stage, note)) { + psError(psErrorCodeLast(), false, "Unable to queue exposures for chip."); + psFree(output); + return false; + } + psFree(output); + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // rawExp, chipRun + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + // we only allow a single label to match (do not use pxAddLabelSearchArgs here) + PXOPT_COPY_STR(config->args, where, "-label", "chipRun.label", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "chipRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "chipRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group", "chipRun.dist_group", "LIKE"); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psString query = psStringCopy("UPDATE chipRun JOIN rawExp USING(exp_id) -- join hook %s\n"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "chipRun", "chip_id", "chipProcessedImfile", true, true); + if (!result) { + psError(psErrorCodeLast(), false, "pxUpdateRun failed"); + } + + psFree(query); + psFree(where); + + return result; +} + + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); //chipRun, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "rawImfile.class_id", "=="); + + psString query = pxDataGet("chiptool_pendingimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, chip_id, class_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipPendingImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // chip_id, exp_tag, class_id are required + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + // optional + PXOPT_LOOKUP_STR(uri, config->args, "-uri", false, false); + PXOPT_LOOKUP_F32(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F32(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F32(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F32(bias, config->args, "-bias", false, false); + PXOPT_LOOKUP_F32(bias_stdev, config->args, "-bias_stdev", false, false); + PXOPT_LOOKUP_F32(fringe_0, config->args, "-fringe_0", false, false); + PXOPT_LOOKUP_F32(fringe_1, config->args, "-fringe_1", false, false); + PXOPT_LOOKUP_F32(fringe_2, config->args, "-fringe_2", false, false); + PXOPT_LOOKUP_F32(ap_resid, config->args, "-ap_resid", false, false); + PXOPT_LOOKUP_F32(ap_resid_stdev, config->args, "-ap_resid_stdev", false, false); + + PXOPT_LOOKUP_F32(fwhm_major, config->args, "-fwhm_major", false, false); + PXOPT_LOOKUP_F32(fwhm_major_lq, config->args, "-fwhm_major_lq", false, false); + PXOPT_LOOKUP_F32(fwhm_major_uq, config->args, "-fwhm_major_uq", false, false); + + PXOPT_LOOKUP_F32(fwhm_minor, config->args, "-fwhm_minor", false, false); + PXOPT_LOOKUP_F32(fwhm_minor_lq, config->args, "-fwhm_minor_lq", false, false); + PXOPT_LOOKUP_F32(fwhm_minor_uq, config->args, "-fwhm_minor_uq", false, false); + + PXOPT_LOOKUP_F32(iq_fwhm_major, config->args, "-iq_fwhm_major", false, false); + PXOPT_LOOKUP_F32(iq_fwhm_major_err, config->args, "-iq_fwhm_major_err", false, false); + PXOPT_LOOKUP_F32(iq_fwhm_minor, config->args, "-iq_fwhm_minor", false, false); + PXOPT_LOOKUP_F32(iq_fwhm_minor_err, config->args, "-iq_fwhm_minor_err", false, false); + + PXOPT_LOOKUP_F32(iq_m2, config->args, "-iq_m2", false, false); + PXOPT_LOOKUP_F32(iq_m2_err, config->args, "-iq_m2_err", false, false); + PXOPT_LOOKUP_F32(iq_m2_uq, config->args, "-iq_m2_uq", false, false); + PXOPT_LOOKUP_F32(iq_m2_lq, config->args, "-iq_m2_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m2c, config->args, "-iq_m2c", false, false); + PXOPT_LOOKUP_F32(iq_m2c_err, config->args, "-iq_m2c_err", false, false); + PXOPT_LOOKUP_F32(iq_m2c_uq, config->args, "-iq_m2c_uq", false, false); + PXOPT_LOOKUP_F32(iq_m2c_lq, config->args, "-iq_m2c_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m2s, config->args, "-iq_m2s", false, false); + PXOPT_LOOKUP_F32(iq_m2s_err, config->args, "-iq_m2s_err", false, false); + PXOPT_LOOKUP_F32(iq_m2s_uq, config->args, "-iq_m2s_uq", false, false); + PXOPT_LOOKUP_F32(iq_m2s_lq, config->args, "-iq_m2s_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m3, config->args, "-iq_m3", false, false); + PXOPT_LOOKUP_F32(iq_m3_err, config->args, "-iq_m3_err", false, false); + PXOPT_LOOKUP_F32(iq_m3_uq, config->args, "-iq_m3_uq", false, false); + PXOPT_LOOKUP_F32(iq_m3_lq, config->args, "-iq_m3_lq", false, false); + + PXOPT_LOOKUP_F32(iq_m4, config->args, "-iq_m4", false, false); + PXOPT_LOOKUP_F32(iq_m4_err, config->args, "-iq_m4_err", false, false); + PXOPT_LOOKUP_F32(iq_m4_uq, config->args, "-iq_m4_uq", false, false); + PXOPT_LOOKUP_F32(iq_m4_lq, config->args, "-iq_m4_lq", false, false); + + PXOPT_LOOKUP_F32(dtime_detrend, config->args, "-dtime_detrend", false, false); + PXOPT_LOOKUP_F32(dtime_photom, config->args, "-dtime_photom", false, false); + PXOPT_LOOKUP_F32(dtime_total, config->args, "-dtime_total", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_F32(n_stars, config->args, "-n_stars", false, false); + PXOPT_LOOKUP_F32(n_psfstars, config->args, "-n_psfstars", false, false); + PXOPT_LOOKUP_F32(n_iqstars, config->args, "-n_iqstars", false, false); + PXOPT_LOOKUP_F32(n_extended, config->args, "-n_extended", false, false); + PXOPT_LOOKUP_F32(n_cr, config->args, "-n_cr", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_S64(magicked, config->args, "-magicked", false, false); + + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_STR(ver_psastro, config->args, "-ver_psastro", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_STR(ver_ppimage, config->args, "-ver_ppimage", false, false); + PXOPT_LOOKUP_STR(ver_streaks, config->args, "-ver_streaks", false, false); + + PXOPT_LOOKUP_S32(maskfrac_npix, config->args, "-maskfrac_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_static, config->args, "-maskfrac_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_dynamic, config->args, "-maskfrac_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_magic, config->args, "-maskfrac_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_advisory, config->args, "-maskfrac_advisory", false, false); + PXOPT_LOOKUP_F32(deteff_magref, config->args, "-deteff_magref", false, false); + + psTrace("czw.test",1,"Received versions: pslib %s psmodules %s psphot %s psastro %s ppstats %s ppImage %s streaks %s\n", + ver_pslib,ver_psmodules,ver_psphot,ver_psastro,ver_ppstats,ver_ppimage,ver_streaks); + psString ver_code = NULL; + if ((ver_pslib)&&(ver_psmodules)) { + ver_code = pxMergeCodeVersions(ver_pslib,ver_psmodules); + } + if (ver_psphot) { + ver_code = pxMergeCodeVersions(ver_code,ver_psphot); + } + if (ver_psastro) { + ver_code = pxMergeCodeVersions(ver_code,ver_psastro); + } + if (ver_ppstats) { + ver_code = pxMergeCodeVersions(ver_code,ver_ppstats); + } + if (ver_ppimage) { + ver_code = pxMergeCodeVersions(ver_code,ver_ppimage); + } + if (ver_streaks) { + ver_code = pxMergeCodeVersions(ver_code,ver_streaks); + } + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!chipProcessedImfileInsert(config->dbh, + chip_id, + exp_id, + class_id, + "full", + uri, + bg, + bg_stdev, + bg_mean_stdev, + bias, + bias_stdev, + fringe_0, + fringe_1, + fringe_2, + ap_resid, + ap_resid_stdev, + + fwhm_major, + fwhm_major_lq, + fwhm_major_uq, + fwhm_minor, + fwhm_minor_lq, + fwhm_minor_uq, + + iq_fwhm_major, + iq_fwhm_major_err, + iq_fwhm_minor, + iq_fwhm_minor_err, + + iq_m2, + iq_m2_err, + iq_m2_lq, + iq_m2_uq, + + iq_m2c, + iq_m2c_err, + iq_m2c_lq, + iq_m2c_uq, + + iq_m2s, + iq_m2s_err, + iq_m2s_lq, + iq_m2s_uq, + + iq_m3, + iq_m3_err, + iq_m3_lq, + iq_m3_uq, + + iq_m4, + iq_m4_err, + iq_m4_lq, + iq_m4_uq, + + dtime_detrend, + dtime_photom, + dtime_total, + dtime_script, + hostname, + n_stars, + n_psfstars, + n_iqstars, + n_extended, + n_cr, + path_base, + fault, + quality, + magicked, + ver_code, + maskfrac_npix, + maskfrac_static, + maskfrac_dynamic, + maskfrac_magic, + maskfrac_advisory, + deteff_magref + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(allfiles, config->args, "-allfiles", false); + if (allfiles) { + faulted = false; + } + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipRun, chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_imfile_id", "chipImfile.chip_imfile_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipProcessedImfile.class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipRun.data_group", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "chipProcessedImfile.magicked", "=="); + + psString where2 = NULL; + pxmagicAddWhere(config, &where2, "chipProcessedImfile"); + // add cuts on ra and decl if supplied + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + psString query = pxDataGet("chiptool_processedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s %s", whereClause, where2 ? where2 : ""); + psFree(whereClause); + } else if (psMetadataLookupBool(NULL, config->args, "-all") || (faulted || where2)) { + psStringAppend(&query, " WHERE chipRun.chip_id IS NOT NULL %s", where2 ? where2 : ""); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters (or -all) are required"); + return false; + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND chipProcessedImfile.fault != 0"); + } else if (!allfiles) { + // don't list faulted rows + psStringAppend(&query, " %s", "AND chipProcessedImfile.fault = 0"); + } + + if (pstamp_order) { + // put runs in order of exposure id with newest chip Runs first + // The postage stamp parser depends on this behavior + psStringAppend(&query, "\nORDER by exp_id, chip_id DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool revertprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipProcessedImfile.class_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "chipProcessedImfile.fault", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psListLength(where->list) + && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("chiptool_revertprocessedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + psString query_update = pxDataGet("chiptool_revertupdatedimfile.sql"); + if (!query_update) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psStringAppend(&query_update, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + int numDeleted = psDBAffectedRows(config->dbh); + psLogMsg("chiptool", PS_LOG_INFO, "Deleted %d chipProcessedImfiles", numDeleted); + + if (!fault) { + // If fault has not been supplied, don't revert update faults with the magic value + // We don't do this for new runs because then they would never complete + // quality should be used to drop bad components + psStringAppend(&query_update, " AND (chipProcessedImfile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + } + if (!p_psDBRunQuery(config->dbh, query_update)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query_update); + return false; + } + psFree(query_update); + int numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("chiptool", PS_LOG_INFO, "Updated %d chipProcessedImfiles", numUpdated); + + + return true; +} +static bool revertcleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipRun.data_group", "LIKE"); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + + char* newState = NULL; + if (!state) { + state = "error_cleaned"; + } + if (!strcmp(state, "error_cleaned")) { + newState = "goto_cleaned"; + } else if (!strcmp(state, "error_purged")) { + newState = "goto_purged"; + } else if (!strcmp(state, "error_scrubbed")) { + newState = "goto_scrubbed"; + } else { + psError(PXTOOLS_ERR_CONFIG, true, "-state must be either error_cleaned, error_purged, or error_scrubbed"); + return false; + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("chiptool_revertcleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, newState, state, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool updateprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + if (psListLength(where->list) == 0) { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + + if (!state) { + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + if (!pxSetFaultCode(config->dbh, "chipProcessedImfile", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + return false; + } + psFree(where); + } + else { + if (!pxchipProcessedImfileSetStateByQuery(config,where,state)) { + psError(PS_ERR_UNKNOWN, false, "failed to set chipProcessedImfile state"); + return(false); + } + } + + + return true; +} + +// "drop" previously processed components as though the original processing +// had poor quality. Sets quality, data_state, clears fault and magicked +// Deletes corresponding magicDSFile (if one exists) +static bool dropprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipProcessedImfile.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "chipProcessedImfile.class_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + + // PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", false, false); + // if cam_id is not supplied chip_id is required + // PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", cam_id ? false : true, false); + // PXOPT_LOOKUP_S64(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", true, false); + + if (!quality) { + psError(PXTOOLS_ERR_CONFIG, true, "non-zero quality value is required"); + return false; + } + + if (psListLength(where->list) == 0) { + // won't get here + psError(PXTOOLS_ERR_CONFIG, true, "-chip_id (or -cam_id) and -class_id are required"); + return false; + } + + psString query = pxDataGet("chiptool_dropprocessedimfile_update.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString delete_query = pxDataGet("chiptool_dropprocessedimfile_delete.sql"); + if (!delete_query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(where); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, quality)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error on rollback"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psStringAppend(&delete_query, " AND %s", whereClause); + psFree(whereClause); + if (!p_psDBRunQuery(config->dbh, delete_query)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error on rollback"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete_query); + return false; + } + psFree(delete_query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + if (!chipMaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (where->list->n < 1) { + psFree(where); + where = NULL; + } + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM chipMask"); + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psFree(where); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// return the list of labels which are NOT blocked +static bool unmaskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (where->list->n < 1) { + psFree(where); + where = NULL; + } + + psString query = pxDataGet("chiptool_unmasked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, "chipUnmask"); + psFree(where); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipUnmask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + char *query = "DELETE FROM chipMask WHERE label = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "=="); + + psString query = pxDataGet("chiptool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, chip_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipPendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingcleanupimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", false, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + if (chip_id) { + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + } + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + char *sql_file = all ? "chiptool_pendingcleanupimfile_all.sql" : "chiptool_pendingcleanupimfile.sql"; + psString query = pxDataGet(sql_file); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipPendingCleanupImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psString query = pxDataGet("chiptool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool runMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // make sure that the state string is valid + if (!pxIsValidState(state)) { + psError(PXTOOLS_ERR_CONFIG, false, "%s is not a valid state", state); + return false; + } + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipRun, chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "chipRun.state", "=="); + + psString query = pxDataGet("chiptool_run.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool advanceexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + // look for completed chipPendingExp + // migrate them to chipProccessedExp & camPendingExp + psString query = pxDataGet("chiptool_completely_processed_exp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + psTrace("chiptool", 10, "%s\n", query); + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + psS64 chip_id = psMetadataLookupS64(NULL,row,"chip_id"); + + psString software_ver = NULL; + psS64 maskfrac_npix = 0; + psF32 maskfrac_static = 0; + psF32 maskfrac_dynamic = 0; + psF32 maskfrac_magic = 0; + psF32 maskfrac_advisory = 0; + + // Calculate run level masking and software state + if (!pxCoalesceRunStatus(config,"chiptool_coalesce_run.sql",chip_id, + &software_ver,&maskfrac_npix, + &maskfrac_static,&maskfrac_dynamic, + &maskfrac_magic,&maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to generate run level statistics"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + psMetadataAddStr(row, PS_LIST_TAIL, "software_ver", 0, "Software version", software_ver); + psMetadataAddS32(row, PS_LIST_TAIL, "maskfrac_npix", 0, "Number of pixels used for mask stats", (psS32) maskfrac_npix); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_static", 0, "Fraction of static masked pixels", maskfrac_static); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_dynamic", 0, "Fraction of dynamic masked pixels", maskfrac_dynamic); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_magic", 0, "Fraction of magic masked pixels", maskfrac_magic); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_advisory", 0, "Fraction of advisory pixels", maskfrac_advisory); +/* psWarning("ADVANCE %ld %s %d %f %f %f %f\n",chip_id,software_ver,maskfrac_npix,maskfrac_static,maskfrac_dynamic,maskfrac_magic,maskfrac_advisory); */ + chipRunRow *chipRun = chipRunObjectFromMetadata(row); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // Set chipRun.software_ver to the appropriate value + if (software_ver) { + if (!pxSetRunSoftware(config, "chipRun", "chip_id", chip_id, software_ver)) { + psError(PS_ERR_UNKNOWN, false, "failed to set chipRun.software_ver for chip_id: %" PRId64, + chip_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // Set chipRun.maskfrac* to the appropriate values. + if (maskfrac_npix) { + if (!pxSetRunMaskfrac(config, "chipRun", "chip_id",chip_id, maskfrac_npix, maskfrac_static, + maskfrac_dynamic, maskfrac_magic, maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to set chipRun.software_ver for chip_id: %" PRId64, + chip_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // set chipRun.state to 'stop' and update the magicked state + if (!pxchipRunSetState(config, chipRun->chip_id, "full", chipRun->magicked)) { + psError(PS_ERR_UNKNOWN, false, "failed to change chipRun.state for chip_id: %" PRId64, chipRun->chip_id); + psFree(chipRun); + psFree(output); + return false; + } + + // should we stop here or proceed on to the cam stage? + // NULL for end_stage means go as far as possible + if (chipRun->end_stage && psStrcasestr(chipRun->end_stage, "chip")) { + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(chipRun); + continue; + } + // else continue on... + + // camQueueChipID() can only be run after chipRun.state has been set to + // stop + if (!pxcamQueueByChipID(config, + chipRun->chip_id, + chipRun->workdir, + chipRun->label, + chipRun->data_group, + chipRun->dist_group, + chipRun->reduction, + chipRun->expgroup, + chipRun->dvodb, + chipRun->tess_id, + chipRun->end_stage, + chipRun->magicked, + NULL // note does not propagate + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to queue camPendingExp"); + psFree(chipRun); + psFree(output); + return false; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(chipRun); + } + + psFree(output); + + return true; +} + +// update chipProcessedImfile.data_state to given value. +// afterwards, if all imfiles in the exposure have the new state, update the state for the exposure as well +// shared code for the modes -tocleanedimfile -tofullimfile -topurgedimfile + +static bool change_imfile_data_state(pxConfig *config, psString data_state, psString run_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // chip_id, class_id are required + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + psString query = pxDataGet("chiptool_change_imfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString set_magicked_imfile = psStringCopy(""); + psString set_magicked_run = psStringCopy(""); + if (!strcmp(data_state, "full")) { + // if (chipProcessedImfile.magicked < 0 and rawImfile.magicked = 0) leave magicked unchanged. This will + // block warp processing until destreaking has been done + // otherwise copy magicked from the rawImfile + // Same thing for chipRun/rawExp + psStringAppend(&set_magicked_imfile, "\n , chipProcessedImfile.magicked = IF((chipProcessedImfile.magicked < 0" + " AND rawImfile.magicked = 0), chipProcessedImfile.magicked, rawImfile.magicked)"); + psStringAppend(&set_magicked_run, "\n , chipRun.magicked = IF((chipRun.magicked < 0 AND rawExp.magicked = 0), " + " chipRun.magicked, rawExp.magicked)"); + + } else if (!strcmp(data_state, "cleaned") || !strcmp(data_state, "purged")) { + // if magicked is non-zero set it to -1 + // Once one imfile has been cleaned, the chipRun is no longer 'magicked' + psStringAppend(&set_magicked_imfile, "\n, chipProcessedImfile.magicked = IF(chipProcessedImfile.magicked = 0, 0, -1)," + " chipRun.magicked = IF(chipRun.magicked = 0, 0, -1)"); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_imfile, chip_id, class_id)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + psFree(set_magicked_imfile); + psFree(set_magicked_run); + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + query = pxDataGet("chiptool_change_exp_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_run, chip_id, data_state)) { + psFree(query); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool tocleanedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "cleaned", "goto_cleaned"); +} +static bool tofullimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "full", "update"); +} +static bool topurgedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "purged", "goto_purged"); +} +static bool toscrubbedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "scrubbed", "goto_scrubbed"); +} +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 3; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(dummy, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + + ExportTable tables [] = { + {"chipRun", "chiptool_export_run.sql"}, + {"chipImfile", "chiptool_export_imfile.sql"}, + {"chipProcessedImfile", "chiptool_export_processed_imfile.sql"}, + }; + + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + bool success = true; + if (!strcmp(tables[i].tableName, "chipRun")) { + success = pxSetStateCleaned("chipRun", "state", output); + } else if (!strcmp(tables[i].tableName, "chipProcessedImfile")) { + success = pxSetStateCleaned("chipProcessedImfile", "data_state", output); + } + if (!success) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + unsigned int nFail = 0; // Number of failed lines + psMetadata *input = psMetadataConfigRead(NULL, &nFail, infile, false); + if (nFail) { + psError(PS_ERR_IO, false, "%d failed lines in input", nFail); + psFree(input); + return false; + } + + psVector *identifiers = psVectorAllocEmpty(16, PS_TYPE_U64); // Identifiers inserted + + psMetadataIterator *iter = psMetadataIteratorAlloc(input, PS_LIST_HEAD, NULL); // Iterator + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + // first item is the dbversion, skip it + psMetadataItem *dbversion = psMetadataGetAndIncrement(iter); + (void) dbversion; + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psMetadataItem *item; // Item from iteration + while ((item = psMetadataGetAndIncrement(iter))) { + PXMIRROR_PRIMARY(item, "chipRun", chipRunRow, chipRunObjectFromMetadata, identifiers, chip_id, + chipRunInsertObject, config->dbh, + { psFree(iter); psFree(identifiers); psFree(input); }); + + PXMIRROR_OTHER(item, "chipImfile", chipImfileRow, chipImfileObjectFromMetadata, identifiers, chip_id, + chipImfileInsertObject, config->dbh, + { psFree(iter); psFree(identifiers); psFree(input); }); + + PXMIRROR_OTHER(item, "chipProcessedImfile", chipProcessedImfileRow, + chipProcessedImfileObjectFromMetadata, identifiers, chip_id, + chipProcessedImfileInsertObject, config->dbh, + { psFree(iter); psFree(identifiers); psFree(input); }); + } + psFree(iter); + psFree(input); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psLogMsg("chiptool", PS_LOG_INFO, "%ld chipRuns added", identifiers->n); + psFree(identifiers); + + return true; +} + +static bool runstateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + +// PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + // PXOPT_LOOKUP_BOOL(no_magic, config->args, "-no_magic", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("chiptool_runstate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipRunState", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +// a very specfic function to queue a cleaned chipProcessedImfile to be updated +static bool setimfiletoupdateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_S16(update_mode, config->args, "-set_update_mode", false, false); + + psString query = pxDataGet("chiptool_setimfiletoupdate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString setHook = psStringCopy(""); + if (label) { + psStringAppend(&setHook, "\n , chipRun.label = '%s'", label); + } + // always set update_mode. If user has not supplied it zero is the right answer. + psStringAppend(&setHook, "\n, chipRun.update_mode = %d", update_mode); + + if (class_id) { + psStringAppend(&query, " AND (chipProcessedImfile.class_id = '%s')", class_id); + } + + // we do not update components with the magic fault value. They are non-updateable + psStringAppend(&query, " AND (chipProcessedImfile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + + if (!p_psDBRunQueryF(config->dbh, query, setHook, chip_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(setHook); + psFree(query); + + // if we are updating a whole chipRun set lingering chips with poor quality to full data_state + if (!class_id) { + query = "UPDATE chipProcessedImfile SET data_state ='full', fault = 0 WHERE chip_id = %" PRId64 " AND quality != 0 AND (data_state ='cleaned' OR data_state = 'update')"; + + if (!p_psDBRunQueryF(config->dbh, query, chip_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + return true; +} + +static bool listrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipRun, chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "chipRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "chipRun.tess_id", "LIKE"); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipRun.data_group", "LIKE"); + pxAddLabelSearchArgs (config, where, "-dist_group", "chipRun.dist_group", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "chipRun.magicked", "=="); + + psString where2 = NULL; + pxmagicAddWhere(config, &where2, "chipRun"); + // add cuts on ra and decl if supplied + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + psString query = pxDataGet("chiptool_listrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s %s", whereClause, where2 ? where2 : ""); + psFree(whereClause); + } else if (psMetadataLookupBool(NULL, config->args, "-all") || where2) { + psStringAppend(&query, " WHERE chipRun.chip_id IS NOT NULL %s", where2 ? where2 : ""); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters (or -all) are required"); + return false; + } + psFree(where); + + if (pstamp_order) { + // put runs in order of exposure id with newest chip Runs first + // The postage stamp parser depends on this behavior + psStringAppend(&query, "\nORDER by exp_id, chip_id DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptool.h (revision 41104) @@ -0,0 +1,60 @@ +/* + * chiptool.h + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef CHIPTOOL_H +#define CHIPTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + CHIPTOOL_MODE_NONE = 0x0, + CHIPTOOL_MODE_DEFINEBYQUERY, + CHIPTOOL_MODE_DEFINECOPY, + CHIPTOOL_MODE_UPDATERUN, + CHIPTOOL_MODE_PENDINGIMFILE, + CHIPTOOL_MODE_ADDPROCESSEDIMFILE, + CHIPTOOL_MODE_PROCESSEDIMFILE, + CHIPTOOL_MODE_REVERTPROCESSEDIMFILE, + CHIPTOOL_MODE_UPDATEPROCESSEDIMFILE, + CHIPTOOL_MODE_DROPPROCESSEDIMFILE, + CHIPTOOL_MODE_SETIMFILETOUPDATE, + CHIPTOOL_MODE_ADVANCEEXP, + CHIPTOOL_MODE_BLOCK, + CHIPTOOL_MODE_MASKED, + CHIPTOOL_MODE_UNMASKED, + CHIPTOOL_MODE_UNBLOCK, + CHIPTOOL_MODE_RETRYPROCESSEDIMFILE, + CHIPTOOL_MODE_PENDINGCLEANUPRUN, + CHIPTOOL_MODE_PENDINGCLEANUPIMFILE, + CHIPTOOL_MODE_REVERTCLEANUP, + CHIPTOOL_MODE_DONECLEANUP, + CHIPTOOL_MODE_RUN, + CHIPTOOL_MODE_TOCLEANEDIMFILE, + CHIPTOOL_MODE_TOFULLIMFILE, + CHIPTOOL_MODE_TOPURGEDIMFILE, + CHIPTOOL_MODE_TOSCRUBBEDIMFILE, + CHIPTOOL_MODE_EXPORTRUN, + CHIPTOOL_MODE_IMPORTRUN, + CHIPTOOL_MODE_RUNSTATE, + CHIPTOOL_MODE_LISTRUN +} chiptoolMode; + +pxConfig *chiptoolConfig(pxConfig *config, int argc, char **argv); + +#endif // CHIPTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/chiptoolConfig.c (revision 41104) @@ -0,0 +1,431 @@ +/* + * chiptoolConfig.c + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "chiptool.h" + +pxConfig *chiptoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + pxchipSetSearchArgs (definebyqueryArgs); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by rawExp label (LIKE comparison)", NULL); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-exp_type", PS_META_REPLACE, "search by exp_type", "object"); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_expgroup", 0, "define exposure group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tessellation identifier", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_end_stage", 0, "define end stage", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define distribution group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-unique", 0, "only queue exposures that have no previous chipRun", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(definebyqueryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-random", 0, "randomly sort the output", false); + + // -definecopy + psMetadata *definecopyArgs = psMetadataAlloc(); + pxchipSetSearchArgs (definecopyArgs); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by chipRun label (LIKE comparison)", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by chipRun label (LIKE comparison)", NULL); + + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-exp_type", PS_META_REPLACE, "search by exp_type", "object"); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-state", 0, "search by chipRun.state", NULL); + + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_label", 0, "define label (required)", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_expgroup", 0, "define exposure group", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tessellation identifier", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_end_stage", 0, "define end stage", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define distribution group", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definecopyArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definecopyArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + pxchipSetSearchArgs (updaterunArgs); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-dist_group", 0, "search by dist_group (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "set data group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "set dist group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "set note", NULL); + pxmagicAddArguments(updaterunArgs); + + // -setimfiletoupdate + psMetadata *setimfiletoupdateArgs = psMetadataAlloc(); + psMetadataAddS64(setimfiletoupdateArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID (required)", 0); + psMetadataAddStr(setimfiletoupdateArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", NULL); + psMetadataAddStr(setimfiletoupdateArgs, PS_LIST_TAIL, "-set_label", 0,"new value for label", NULL); + psMetadataAddS16(setimfiletoupdateArgs, PS_LIST_TAIL, "-set_update_mode", 0,"new value for update_mode", 0); + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class_id", 0); + pxchipSetSearchArgs(pendingimfileArgs); + psMetadataAddU64(pendingimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "define chip ID (required)", 0); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "define exposure ID (required)", 0); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-uri", 0, "define URL", NULL); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-bias", 0, "define bias", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-bias_stdev", 0, "define bias stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fringe_0", 0, "define fringe term 0", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fringe_1", 0, "define fringe term 1", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fringe_2", 0, "define fringe term 2", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-ap_resid", 0, "define aperture residual", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-ap_resid_stdev", 0, "define aperture residual stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fwhm_major", 0, "define fwhm (major axis; arcsec)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fwhm_major_lq", 0, "define fwhm (major axis; arcsec)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fwhm_major_uq", 0, "define fwhm (major axis; arcsec)", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fwhm_minor", 0, "define fwhm (minor axis; arcsec)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fwhm_minor_lq", 0, "define fwhm (minor axis; arcsec)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-fwhm_minor_uq", 0, "define fwhm (minor axis; arcsec)", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_fwhm_major", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_fwhm_major_err", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_fwhm_minor", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_fwhm_minor_err", 0, "define moment value m2 mean", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2", 0, "define moment value m2 mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2_err", 0, "define moment value m2 stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2_lq", 0, "define moment value m2 lower quartile", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2_uq", 0, "define moment value m2 upper quartile", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2c", 0, "define moment value m2c mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2c_err", 0, "define moment value m2c stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2c_lq", 0, "define moment value m2c lower quartile", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2c_uq", 0, "define moment value m2c upper quartile", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2s", 0, "define moment value m2s mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2s_err", 0, "define moment value m2s stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2s_lq", 0, "define moment value m2s lower quartile", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m2s_uq", 0, "define moment value m2s upper quartile", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m3", 0, "define moment value m3 mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m3_err", 0, "define moment value m3 stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m3_lq", 0, "define moment value m3 lower quartile", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m3_uq", 0, "define moment value m3 upper quartile", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m4", 0, "define moment value m4 mean", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m4_err", 0, "define moment value m4 stdev", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m4_lq", 0, "define moment value m4 lower quartile", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-iq_m4_uq", 0, "define moment value m4 upper quartile", NAN); + + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-dtime_detrend", 0, "define elapsed time for detrend (seconds)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-dtime_photom", 0, "define elapsed time for photometry (seconds)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-dtime_total", 0, "define total elapsed time (seconds)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + psMetadataAddS32(addprocessedimfileArgs, PS_LIST_TAIL, "-n_stars", 0, "define number of stars in image", 0); + psMetadataAddS32(addprocessedimfileArgs, PS_LIST_TAIL, "-n_psfstars", 0, "define number of stars used for psf", 0); + psMetadataAddS32(addprocessedimfileArgs, PS_LIST_TAIL, "-n_iqstars", 0, "define number of stars used for IQ stats", 0); + psMetadataAddS32(addprocessedimfileArgs, PS_LIST_TAIL, "-n_extended", 0, "define number of extended objects in image", 0); + psMetadataAddS32(addprocessedimfileArgs, PS_LIST_TAIL, "-n_cr", 0, "define number of cosmic rays in image", 0); + + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddS16(addprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(addprocessedimfileArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-magicked", 0, "define magicked status", 0); + + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_psastro", 0, "define psastro version", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_ppimage", 0, "define ppImage version", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-ver_streaks", 0, "define streaksremove version", NULL); + + psMetadataAddS32(addprocessedimfileArgs, PS_LIST_TAIL, "-maskfrac_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-maskfrac_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-maskfrac_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-maskfrac_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-maskfrac_advisory", 0, "define advisory mask fraction", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-deteff_magref", 0, "define deteff_magref", NAN); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + pxchipSetSearchArgs(processedimfileArgs); + psMetadataAddS64(processedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddS64(processedimfileArgs, PS_LIST_TAIL, "-chip_imfile_id", 0, "search by chip_file_id", 0); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by chipRun label (LIKE comparison)", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by chipRun data_group (LIKE comparison)", NULL); + pxmagicAddArguments(processedimfileArgs); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + + psMetadataAddU64(processedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-all", 0, "list everything without search terms", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-allfiles", 0, "only all matching imfiles regardless of fault status", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + pxspaceAddArguments(processedimfileArgs); + + // -revertprocessedimfile + psMetadata *revertprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertprocessedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by chipRun label (LIKE comparison)", NULL); + pxchipSetSearchArgs(revertprocessedimfileArgs); + psMetadataAddBool(revertprocessedimfileArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + psMetadataAddS16(revertprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -updateprocessedimfile + psMetadata *updateprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(updateprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddS16(updateprocessedimfileArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality", 0); + + // -dropprocessedimfile + psMetadata *dropprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(dropprocessedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddS64(dropprocessedimfileArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam ID", 0); + psMetadataAddStr(dropprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID (required)", NULL); + psMetadataAddS16(dropprocessedimfileArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality (required)", 0); + + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + pxchipSetSearchArgs(listrunArgs); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by chipRun label (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-state", 0, "search by chipRun state", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-tess_id", 0, "search by chipRun tess_id (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by chipRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by chipRun dist_group (LIKE comparison)", NULL); + pxmagicAddArguments(listrunArgs); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + psMetadataAddU64(listrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-all", 0, "list everything without search terms", false); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + pxspaceAddArguments(listrunArgs); + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to mask out (required)", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddStr(maskedArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(maskedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -unmasked + psMetadata *unmaskedArgs = psMetadataAlloc(); + psMetadataAddStr(unmaskedArgs, PS_LIST_TAIL, "-label", 0, "restrict to specified label", NULL); + psMetadataAddBool(unmaskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(unmaskedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to unmask (required)", NULL); + + // -pendingcleanuprun + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupimfile + psMetadata *pendingcleanupimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(pendingcleanupimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", NULL); + psMetadataAddBool(pendingcleanupimfileArgs, PS_LIST_TAIL, "-all", 0, "list all components regardless of data_state", false); + psMetadataAddBool(pendingcleanupimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -revertcleanup + psMetadata *revertcleanupArgs = psMetadataAlloc(); + psMetadataAddS64(revertcleanupArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by chipRun label (LIKE comparison)", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by chipRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-state", 0, "search by current state", NULL); + psMetadataAddS16(revertcleanupArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -run + psMetadata *runArgs = psMetadataAlloc(); + psMetadataAddStr(runArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddBool(runArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(runArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddStr(runArgs, PS_LIST_TAIL, "-state", 0, "search by state (required)", NULL); + pxchipSetSearchArgs(runArgs); + psMetadataAddS64(runArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(runArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + + // -advanceexp + psMetadata *advanceexpArgs = psMetadataAlloc(); + psMetadataAddS64(advanceexpArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddStr(advanceexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "advance exposures for specified label", NULL); + psMetadataAddU64(advanceexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tocleanedimfile + psMetadata *tocleanedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "chip ID to update", 0); + psMetadataAddStr(tocleanedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -tofullimfile + psMetadata *tofullimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tofullimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "chip ID to update", 0); + psMetadataAddStr(tofullimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -topurgedimfile + psMetadata *topurgedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(topurgedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "chip ID to update", 0); + psMetadataAddStr(topurgedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -toscrubbedimfile + psMetadata *toscrubbedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(toscrubbedimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "chip ID to update", 0); + psMetadataAddStr(toscrubbedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + // pxchipSetSearchArgs (exportrunArgs); XXX include search terms? + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-chip_id", 0, "export this chip ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + // -runstate + psMetadata *runstateArgs = psMetadataAlloc(); + psMetadataAddS64(runstateArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip ID", 0); + psMetadataAddS64(runstateArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure tag", 0); + psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure tag", 0); + psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddBool(runstateArgs, PS_LIST_TAIL, "-no_magic", 0, "magic is not necessary for result", false); + + psMetadataAddU64(runstateArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(runstateArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from raw exposures", CHIPTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definecopy", "create copy runs", CHIPTOOL_MODE_DEFINECOPY, definecopyArgs); + PXOPT_ADD_MODE("-updaterun", "change chip run properties", CHIPTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-setimfiletoupdate", "set imfile state to update", CHIPTOOL_MODE_SETIMFILETOUPDATE, setimfiletoupdateArgs); + PXOPT_ADD_MODE("-pendingimfile", "show pending imfiles", CHIPTOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXOPT_ADD_MODE("-addprocessedimfile", "add a processed imfile", CHIPTOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileArgs); + PXOPT_ADD_MODE("-processedimfile", "show processed imfiles", CHIPTOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXOPT_ADD_MODE("-updateprocessedimfile","change procesed imfile properties", CHIPTOOL_MODE_UPDATEPROCESSEDIMFILE,updateprocessedimfileArgs); + PXOPT_ADD_MODE("-dropprocessedimfile","drop previously procesed imfile", CHIPTOOL_MODE_DROPPROCESSEDIMFILE, dropprocessedimfileArgs); + PXOPT_ADD_MODE("-revertprocessedimfile","undo a processed imfile", CHIPTOOL_MODE_REVERTPROCESSEDIMFILE,revertprocessedimfileArgs); + PXOPT_ADD_MODE("-runstate", "list the states of chip run", CHIPTOOL_MODE_RUNSTATE, runstateArgs); + PXOPT_ADD_MODE("-listrun", "list chipRuns", CHIPTOOL_MODE_LISTRUN, listrunArgs); + PXOPT_ADD_MODE("-advanceexp", "advance completed exposures", CHIPTOOL_MODE_ADVANCEEXP, advanceexpArgs); + PXOPT_ADD_MODE("-block", "set a label block", CHIPTOOL_MODE_BLOCK, blockArgs); + PXOPT_ADD_MODE("-masked", "show blocked labels", CHIPTOOL_MODE_MASKED, maskedArgs); + PXOPT_ADD_MODE("-unmasked", "", CHIPTOOL_MODE_UNMASKED, unmaskedArgs); + PXOPT_ADD_MODE("-unblock", "remove a label block", CHIPTOOL_MODE_UNBLOCK, unblockArgs); + PXOPT_ADD_MODE("-pendingcleanuprun", "show runs that need to be cleaned up", CHIPTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunArgs); + PXOPT_ADD_MODE("-pendingcleanupimfile", "show runs that need to be cleaned up", CHIPTOOL_MODE_PENDINGCLEANUPIMFILE, pendingcleanupimfileArgs); + PXOPT_ADD_MODE("-donecleanup", "show runs that have been cleaned", CHIPTOOL_MODE_DONECLEANUP, donecleanupArgs); + PXOPT_ADD_MODE("-revertcleanup", "show runs that have been cleaned", CHIPTOOL_MODE_REVERTCLEANUP, revertcleanupArgs); + PXOPT_ADD_MODE("-run", "show runs", CHIPTOOL_MODE_RUN, runArgs); + PXOPT_ADD_MODE("-tocleanedimfile", "set imfile state to cleaned", CHIPTOOL_MODE_TOCLEANEDIMFILE, tocleanedimfileArgs); + PXOPT_ADD_MODE("-tofullimfile", "set imfile state to full", CHIPTOOL_MODE_TOFULLIMFILE, tofullimfileArgs); + PXOPT_ADD_MODE("-topurgedimfile", "set imfile state to purged", CHIPTOOL_MODE_TOPURGEDIMFILE, topurgedimfileArgs); + PXOPT_ADD_MODE("-toscrubbedimfile", "set imfile state to scrubbed", CHIPTOOL_MODE_TOSCRUBBEDIMFILE, toscrubbedimfileArgs); + + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", CHIPTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", CHIPTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/detselect.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/detselect.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/detselect.c (revision 41104) @@ -0,0 +1,307 @@ +/* + * detselect.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "detselect.h" + +static bool searchMode(pxConfig *config); +static bool selectMode(pxConfig *config); +static bool showMode(pxConfig *config); + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = detselectConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(DETSELECT_MODE_SEARCH, searchMode); + MODECASE(DETSELECT_MODE_SELECT, selectMode); + MODECASE(DETSELECT_MODE_SHOW, showMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +# define PXOPT_COPY_NULLTEST_F32(from, to, oldname, newname, comment) \ +{ \ + bool status = false; \ + psF32 var = psMetadataLookupF32(&status, from, oldname); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for " oldname); \ + return false; \ + } \ + if (!isnan(var)) { \ + if (!psMetadataAddF32(to, PS_LIST_TAIL, newname, PS_META_DUPLICATE_OK, comment, var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " newname); \ + psFree(to); \ + return false; \ + } \ + if (!psMetadataAddTime(to, PS_LIST_TAIL, newname, PS_META_DUPLICATE_OK, "==", NULL)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add NULL test " newname); \ + psFree(to); \ + return false; \ + } \ + } \ +} + +static bool searchMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(unlimit, config->args, "-unlimit", false); + + PXOPT_LOOKUP_TIME(time, config->args, "-time", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-det_type", "det_type", "=="); + PXOPT_COPY_STR(config->args, where, "-type", "det_type", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + + // airmass_min < airmass < airmass_max + PXOPT_COPY_NULLTEST_F32(config->args, where, "-airmass", "airmass_min", "<="); + PXOPT_COPY_NULLTEST_F32(config->args, where, "-airmass", "airmass_max", ">="); + + // exp_time_min < exp_time < exp_time_max + PXOPT_COPY_NULLTEST_F32(config->args, where, "-exp_time", "exp_time_min", "<="); + PXOPT_COPY_NULLTEST_F32(config->args, where, "-exp_time", "exp_time_max", ">="); + + // ccd_temp_min < ccd_temp < ccd_temp_max + PXOPT_COPY_NULLTEST_F32(config->args, where, "-ccd_temp", "ccd_temp_min", "<="); + PXOPT_COPY_NULLTEST_F32(config->args, where, "-ccd_temp", "ccd_temp_max", ">="); + + PXOPT_COPY_F64(config->args, where, "-posang", "posang_min", "<="); + PXOPT_COPY_F64(config->args, where, "-posang", "posang_max", ">="); + + // time_begin < time < time_end + // the == NULL tests invokes some psDB magic to make an OR + // conditional query + psMetadataAddTime(where, PS_LIST_TAIL, "time_begin", 0, "<=", time); + psMetadataAddTime(where, PS_LIST_TAIL, "time_begin", PS_META_DUPLICATE_OK, "==", NULL); + psMetadataAddTime(where, PS_LIST_TAIL, "time_end", 0, ">=", time); + psMetadataAddTime(where, PS_LIST_TAIL, "time_end", PS_META_DUPLICATE_OK, "==", NULL); + + psString query = pxDataGet("detselect_search.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // we choose the single detrend image which matches all criteria and has + // the latest insertion date + + // unless explicitly specified by the user, list all possible matches + if (!unlimit) { + psStringAppend(&query, " ORDER BY registered DESC, iteration DESC LIMIT 1"); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detrend exposures found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool selectMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString query = pxDataGet("detselect_select.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detNormalizedImfile rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool showMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-det_type", "det_type", "=="); + PXOPT_COPY_STR(config->args, where, "-type", "det_type", "=="); + + psString query = pxDataGet("detselect_show.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // we choose the single detrend image which matches all criteria and has + // the latest insertion date + + psStringAppend(&query, " ORDER BY registered DESC, iteration DESC"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detRun rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + return(true); +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/detselect.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/detselect.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/detselect.h (revision 41104) @@ -0,0 +1,34 @@ +/* + * detselect.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef DETSELECT_H +#define DETSELECT_H 1 + +#include "pxtools.h" + +typedef enum { + DETSELECT_MODE_NONE = 0x0, + DETSELECT_MODE_SEARCH, + DETSELECT_MODE_SELECT, + DETSELECT_MODE_SHOW +} detselectMode; + +pxConfig *detselectConfig(pxConfig *config, int argc, char **argv); + +#endif // DETSELECT_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/detselectConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/detselectConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/detselectConfig.c (revision 41104) @@ -0,0 +1,103 @@ +/* + * detselectConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "pxtools.h" +#include "detselect.h" + +pxConfig *detselectConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -search + psMetadata *searchArgs = psMetadataAlloc(); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-inst", 0, "search by camera", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-telescope", 0, "search by telescope", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-det_type", 0, "search by detrend type", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-type", 0, "search by detrend type (alias for -det_type)", NULL); + psMetadataAddStr(searchArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddF32(searchArgs, PS_LIST_TAIL, "-airmass", 0, "define airmass", NAN); + psMetadataAddF32(searchArgs, PS_LIST_TAIL, "-exp_time", 0, "search by exposure time", NAN); + psMetadataAddF32(searchArgs, PS_LIST_TAIL, "-ccd_temp", 0, "search by ccd tempature", NAN); + psMetadataAddF64(searchArgs, PS_LIST_TAIL, "-posang", 0, "search by rotator position angle", NAN); + psMetadataAddTime(searchArgs, PS_LIST_TAIL, "-time", 0, "define time for desired detrend data", NULL); + psMetadataAddBool(searchArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(searchArgs, PS_LIST_TAIL, "-unlimit", 0, "list all possible detruns, not just the best match", false); + + // -select + psMetadata *selectArgs = psMetadataAlloc(); + psMetadataAddStr(selectArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", NULL); + psMetadataAddS32(selectArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(selectArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddBool(selectArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -show + psMetadata *showArgs = psMetadataAlloc(); + psMetadataAddStr(showArgs, PS_LIST_TAIL, "-inst", 0, "search by camera", NULL); + psMetadataAddStr(showArgs, PS_LIST_TAIL, "-telescope", 0, "search by telescope", NULL); + psMetadataAddStr(showArgs, PS_LIST_TAIL, "-det_type", 0, "search by detrend type", NULL); + psMetadataAddStr(showArgs, PS_LIST_TAIL, "-type", 0, "search by detrend type (alias for -det_type)", NULL); + psMetadataAddBool(showArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-search", "search for an appropriate detrend", DETSELECT_MODE_SEARCH, searchArgs); + PXOPT_ADD_MODE("-select", "retreive detrend information", DETSELECT_MODE_SELECT, selectArgs); + PXOPT_ADD_MODE("-show", "show all active detrends of this type with constraints", DETSELECT_MODE_SHOW, showArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool.c (revision 41104) @@ -0,0 +1,2501 @@ +/* + * dettool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +static bool pendingMode(pxConfig *config); +static bool definebytagMode(pxConfig *config); +static bool definebyqueryMode(pxConfig *config); +static bool definebydetrunMode(pxConfig *config); +static bool runsMode(pxConfig *config); +static bool childlessrunMode(pxConfig *config); +static bool inputMode(pxConfig *config); +static bool rawMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +// run +static bool updatedetrunMode(pxConfig *config); +static bool rerunMode(pxConfig *config); +// register +static bool register_detrendMode(pxConfig *config); + +//static psArray *validDetInputClassIds(pxConfig *config, const char *det_id); +//static psArray *searchInputImfiles(pxConfig *config, const char *det_id); +static detInputExpRow *rawDetrenTodetInputExpRow(rawExpRow *rawExp, psS64 det_id, psS32 iteration); +static psS32 incrementIteration(pxConfig *config, psS64 det_id); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; +/* +typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; +} ExportTable; +*/ +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = dettoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(DETTOOL_MODE_PENDING, pendingMode); + MODECASE(DETTOOL_MODE_DEFINEBYTAG, definebytagMode); + MODECASE(DETTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(DETTOOL_MODE_DEFINEBYDETRUN, definebydetrunMode); + MODECASE(DETTOOL_MODE_RUNS, runsMode); + MODECASE(DETTOOL_MODE_CHILDLESSRUN, childlessrunMode); + MODECASE(DETTOOL_MODE_INPUT, inputMode); + MODECASE(DETTOOL_MODE_RAW, rawMode); + // correction + MODECASE(DETTOOL_MODE_MAKECORRECTION, makecorrectionMode); + MODECASE(DETTOOL_MODE_TOCORRECTEXP, tocorrectexpMode); + MODECASE(DETTOOL_MODE_TOCORRECTIMFILE, tocorrectimfileMode); + MODECASE(DETTOOL_MODE_ADDCORRECTIMFILE, addcorrectimfileMode); + // imfile + MODECASE(DETTOOL_MODE_TOPROCESSEDIMFILE,toprocessedimfileMode); + MODECASE(DETTOOL_MODE_ADDPROCESSEDIMFILE,addprocessedimfileMode); + MODECASE(DETTOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(DETTOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileMode); + MODECASE(DETTOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_PROCESSEDIMFILE, pendingcleanup_processedimfileMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_PROCESSEDIMFILE, donecleanup_processedimfileMode); + // exp + MODECASE(DETTOOL_MODE_TOPROCESSEDEXP, toprocessedexpMode); + MODECASE(DETTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(DETTOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(DETTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpMode); + MODECASE(DETTOOL_MODE_UPDATEPROCESSEDEXP, updateprocessedexpMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_PROCESSEDEXP, pendingcleanup_processedexpMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_PROCESSEDEXP, donecleanup_processedexpMode); + MODECASE(DETTOOL_MODE_UPDATESTATE_PROCESSED, updatestateprocessedMode); + // stacked + MODECASE(DETTOOL_MODE_TOSTACKED, tostackedMode); + MODECASE(DETTOOL_MODE_ADDSTACKED, addstackedMode); + MODECASE(DETTOOL_MODE_STACKED, stackedMode); + MODECASE(DETTOOL_MODE_REVERTSTACKED, revertstackedMode); + MODECASE(DETTOOL_MODE_UPDATESTACKED, updatestackedMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_STACKED, pendingcleanup_stackedMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_STACKED, donecleanup_stackedMode); + // normalizedstat + MODECASE(DETTOOL_MODE_TONORMALIZEDSTAT, tonormalizedstatMode); + MODECASE(DETTOOL_MODE_ADDNORMALIZEDSTAT,addnormalizedstatMode); + MODECASE(DETTOOL_MODE_NORMALIZEDSTAT, normalizedstatMode); + MODECASE(DETTOOL_MODE_REVERTNORMALIZEDSTAT, revertnormalizedstatMode); + MODECASE(DETTOOL_MODE_UPDATENORMALIZEDSTAT, updatenormalizedstatMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDSTAT, pendingcleanup_normalizedstatMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_NORMALIZEDSTAT, donecleanup_normalizedstatMode); + // normalizedimfile + MODECASE(DETTOOL_MODE_TONORMALIZE, tonormalizeMode); + MODECASE(DETTOOL_MODE_ADDNORMALIZEDIMFILE,addnormalizedimfileMode); + MODECASE(DETTOOL_MODE_NORMALIZEDIMFILE, normalizedimfileMode); + MODECASE(DETTOOL_MODE_REVERTNORMALIZEDIMFILE, revertnormalizedimfileMode); + MODECASE(DETTOOL_MODE_UPDATENORMALIZEDIMFILE, updatenormalizedimfileMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDIMFILE, pendingcleanup_normalizedimfileMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_NORMALIZEDIMFILE, donecleanup_normalizedimfileMode); + // normalizedexp + MODECASE(DETTOOL_MODE_TONORMALIZEDEXP, tonormalizedexpMode); + MODECASE(DETTOOL_MODE_ADDNORMALIZEDEXP, addnormalizedexpMode); + MODECASE(DETTOOL_MODE_NORMALIZEDEXP, normalizedexpMode); + MODECASE(DETTOOL_MODE_REVERTNORMALIZEDEXP, revertnormalizedexpMode); + MODECASE(DETTOOL_MODE_UPDATENORMALIZEDEXP, updatenormalizedexpMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDEXP, pendingcleanup_normalizedexpMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_NORMALIZEDEXP, donecleanup_normalizedexpMode); + // residimfile + MODECASE(DETTOOL_MODE_TORESIDIMFILE, toresidimfileMode); + MODECASE(DETTOOL_MODE_ADDRESIDIMFILE, addresidimfileMode); + MODECASE(DETTOOL_MODE_RESIDIMFILE, residimfileMode); + MODECASE(DETTOOL_MODE_REVERTRESIDIMFILE,revertresidimfileMode); + MODECASE(DETTOOL_MODE_UPDATERESIDIMFILE, updateresidimfileMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_RESIDIMFILE, pendingcleanup_residimfileMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_RESIDIMFILE, donecleanup_residimfileMode); + // residexp + MODECASE(DETTOOL_MODE_TORESIDEXP, toresidexpMode); + MODECASE(DETTOOL_MODE_ADDRESIDEXP, addresidexpMode); + MODECASE(DETTOOL_MODE_RESIDEXP, residexpMode); + MODECASE(DETTOOL_MODE_REVERTRESIDEXP, revertresidexpMode); + MODECASE(DETTOOL_MODE_UPDATERESIDEXP, updateresidexpMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_RESIDEXP, pendingcleanup_residexpMode); + MODECASE(DETTOOL_MODE_DONECLEANUP_RESIDEXP, donecleanup_residexpMode); + MODECASE(DETTOOL_MODE_UPDATESTATE_RESID, updatestateresidMode); + // detrunsummary + MODECASE(DETTOOL_MODE_TODETRUNSUMMARY, todetrunsummaryMode); + MODECASE(DETTOOL_MODE_ADDDETRUNSUMMARY, adddetrunsummaryMode); + MODECASE(DETTOOL_MODE_DETRUNSUMMARY, detrunsummaryMode); + MODECASE(DETTOOL_MODE_REVERTDETRUNSUMMARY, revertdetrunsummaryMode); + MODECASE(DETTOOL_MODE_UPDATEDETRUNSUMMARY, updatedetrunsummaryMode); + MODECASE(DETTOOL_MODE_UPDATEDETRUN, updatedetrunMode); + MODECASE(DETTOOL_MODE_RERUN, rerunMode); + MODECASE(DETTOOL_MODE_PENDINGCLEANUP_DETRUNSUMMARY, pendingcleanup_detrunsummaryMode); + // register + MODECASE(DETTOOL_MODE_REGISTER_DETREND, register_detrendMode); + MODECASE(DETTOOL_MODE_REGISTER_DETREND_IMFILE, register_detrend_imfileMode); + MODECASE(DETTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(DETTOOL_MODE_IMPORTRUN, importrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool pendingMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_pending.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool definebytagMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // what type of detRun is this? + // required + PXOPT_LOOKUP_STR(det_type, config->args, "-det_type", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + + // optional + PXOPT_LOOKUP_STR(filelevel, config->args, "-filelevel", false, false); + + // check mode + PXOPT_LOOKUP_STR(mode, config->args, "-mode", true, false); // default ('master') is supplied + if (!isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalid mode"); + return false; + } + + // get -ref_det_id and -ref_iter : required for 'verify' mode / disallowed otherwise + PXOPT_LOOKUP_S64(ref_det_id, config->args, "-ref_det_id", false, false); + PXOPT_LOOKUP_S32(ref_iter, config->args, "-ref_iter", false, false); + if (!strcmp(mode, "verify") && ((ref_det_id == 0) || (ref_iter == -1))) { + psError(PS_ERR_UNKNOWN, false, "verify mode requires both -ref_det_id and -ref_iter"); + return false; + } + if (strcmp(mode, "verify") && ((ref_det_id != 0) || (ref_iter != -1))) { + psError(PS_ERR_UNKNOWN, false, "master mode cannot have -ref_det_id or -ref_iter set"); + return false; + } + + PXOPT_LOOKUP_STR(camera, config->args, "-inst", false, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", false, false); + PXOPT_LOOKUP_STR(exp_type, config->args, "-exp_type", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", false, false); + + PXOPT_LOOKUP_F32(airmass_min, config->args, "-airmass_min", false, false); + PXOPT_LOOKUP_F32(airmass_max, config->args, "-airmass_max", false, false); + PXOPT_LOOKUP_F32(exp_time_min, config->args, "-exp_time_min", false, false); + PXOPT_LOOKUP_F32(exp_time_max, config->args, "-exp_time_max", false, false); + PXOPT_LOOKUP_F32(ccd_temp_min, config->args, "-ccd_temp_min", false, false); + PXOPT_LOOKUP_F32(ccd_temp_max, config->args, "-ccd_temp_max", false, false); + PXOPT_LOOKUP_F64(posang_min, config->args, "-posang_min", false, false); + PXOPT_LOOKUP_F64(posang_max, config->args, "-posang_max", false, false); + PXOPT_LOOKUP_F64(solang_min, config->args, "-sun_angle_min", false, false); + PXOPT_LOOKUP_F64(solang_max, config->args, "-sun_angle_max", false, false); + PXOPT_LOOKUP_TIME(time_begin, config->args, "-time_begin", false, false); + PXOPT_LOOKUP_TIME(time_end, config->args, "-time_end", false, false); + PXOPT_LOOKUP_TIME(use_begin, config->args, "-use_begin", false, false); + PXOPT_LOOKUP_TIME(use_end, config->args, "-use_end", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + + // default + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + + // we have to support multiple exp_ids + psMetadataItem *item = psMetadataLookup(config->args, "-exp_id"); + if (!item) { + // this shouldn't actually happen when using psArgs + psError(PS_ERR_UNKNOWN, true, "at least one -exp_id is required"); + return false; + } + psMetadata *where = psMetadataAlloc(); + + if ((item->type != PS_DATA_METADATA_MULTI) && (item->type != PS_DATA_S64)) { + psAbort("-exp_id was not parsed correctly (this should not happen"); + } + + // make sure that -exp_id was parsed correctly + if (item->type == PS_DATA_METADATA_MULTI) { + psListIterator *iter = psListIteratorAlloc(item->data.list, 0, false); + psMetadataItem *mItem = NULL; + while ((mItem = psListGetAndIncrement(iter))) { + psS64 exp_id = mItem->data.S64; + if (!psMetadataAddS64(where, PS_LIST_TAIL, "exp_id", PS_META_DUPLICATE_OK, "==", exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(iter); + psFree(where); + return false; + } + } + psFree(iter); + } + if (item->type == PS_DATA_S64) { + psS64 exp_id = item->data.S64; + if (!psMetadataAddS64(where, PS_LIST_TAIL, "exp_id", PS_META_DUPLICATE_OK, "==", exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(where); + return false; + } + } + + if (psListLength(where->list) < 1) { + psFree(where); + where = NULL; + } + + // check that the specified exp_ids actually exist + psArray *detrendExps = rawExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!detrendExps) { + psError(PS_ERR_UNKNOWN, false, "no rawExp rows found"); + return false; + } + + // we should have one rawExp row per exp_id specified + if (psListLength(item->data.list) != psArrayLength(detrendExps)) { + psAbort( "an -exp_id matched more then one rawExp (this should not happen"); + + } + + // check to see if -filelevel was set on the command line + if (!filelevel) { + filelevel = psStringCopy(((rawExpRow *)detrendExps->data[0])->filelevel); + } + + // start a transaction so we don't end up with childlessed det_ids + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detrendExps); + return false; + } + + // the first iteration is always 0 + // XXX the camera name is set from the first inputExp + // XXX det_id + detRunInsert(config->dbh, + 0, + 0, + det_type, + mode, + "run", + filelevel, + workdir, + camera, + telescope, + exp_type, + reduction, + filter, + airmass_min, + airmass_max, + exp_time_min, + exp_time_max, + ccd_temp_min, + ccd_temp_max, + posang_min, + posang_max, + registered, + time_begin, + time_end, + use_begin, + use_end, + solang_min, + solang_max, + label, + ref_det_id, + ref_iter + ); + psS64 det_id = psDBLastInsertID(config->dbh); + + // create new detInputExp row(s) from the rawExp row(s) + psArray *inputExps = psArrayAllocEmpty(psArrayLength(detrendExps)); + for (long i = 0; i < psArrayLength(detrendExps); i++) { + detInputExpRow *inputExp = rawDetrenTodetInputExpRow( + detrendExps->data[i], + det_id, + 0 // the first iteration is explicitly 0 + ); + psArrayAdd(inputExps, 0, inputExp); + psFree(inputExp); + } + + psFree(detrendExps); + + // insert detInputExp objects into the database + if (!detInputExpInsertObjects(config->dbh, inputExps)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(inputExps); + return false; + } + psFree(inputExps); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // print the new det_id + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", det_id); + detRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check results + if (psArrayLength(detRuns) != 1) { + psAbort("found more then one detRun matching det_id %" PRId64 " (this should not happen)", det_id); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, detRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detRuns); + return false; + } + psFree(detRuns); + + return true; +} + +#if 0 +// This function is used to convert the det_id from an int, as it is used +// internally, to be a string for external use. The rationale being that we may +// want to change how det_id is generated in the future and don't want to +// external programs to become depending on this value being an int. +static bool convertDetIdToStr(psArray *mds) +{ + PS_ASSERT_PTR_NON_NULL(mds, false); + + for (long i = 0; i < psArrayLength(mds); i++) { + psMetadata *md = mds->data[i]; + bool status = false; + psS32 det_id = psMetadataLookupS32(&status, md, "det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for det_id"); + return false; + } + psMetadataRemoveKey(md, "det_id"); + psString det_idStr = psDBIntToString((psU64)det_id); + psMetadataAddStr(mds->data[i], PS_LIST_HEAD, "det_id", 0, NULL, det_idStr); + psFree(det_idStr); + } + + return true; +} +#endif + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(det_type, config->args, "-det_type", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + + // check mode + PXOPT_LOOKUP_STR(mode, config->args, "-mode", true, false); // default ('master') is supplied + if (!isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalid mode"); + return false; + } + + // get -ref_det_id and -ref_iter : required for 'verify' mode / disallowed otherwise + PXOPT_LOOKUP_S64(ref_det_id, config->args, "-ref_det_id", false, false); + PXOPT_LOOKUP_S32(ref_iter, config->args, "-ref_iter", false, false); + if (!strcmp(mode, "verify") && ((ref_det_id == 0) || (ref_iter == -1))) { + psError(PS_ERR_UNKNOWN, false, "verify mode requires both -ref_det_id and -ref_iter"); + return false; + } + if (strcmp(mode, "verify") && ((ref_det_id != 0) || (ref_iter != -1))) { + psError(PS_ERR_UNKNOWN, false, "master mode cannot have -ref_det_id or -ref_iter set"); + return false; + } + + PXOPT_LOOKUP_STR(filelevel, config->args, "-filelevel", false, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", false, false); + PXOPT_LOOKUP_F32(airmass_min, config->args, "-airmass_min", false, false); + PXOPT_LOOKUP_F32(airmass_max, config->args, "-airmass_max", false, false); + PXOPT_LOOKUP_F32(exp_time_min, config->args, "-exp_time_min", false, false); + PXOPT_LOOKUP_F32(exp_time_max, config->args, "-exp_time_max", false, false); + PXOPT_LOOKUP_F32(ccd_temp_min, config->args, "-ccd_temp_min", false, false); + PXOPT_LOOKUP_F32(ccd_temp_max, config->args, "-ccd_temp_max", false, false); + PXOPT_LOOKUP_F64(posang_min, config->args, "-posang_min", false, false); + PXOPT_LOOKUP_F64(posang_max, config->args, "-posang_max", false, false); + PXOPT_LOOKUP_F64(solang_min, config->args, "-sun_angle_min", false, false); + PXOPT_LOOKUP_F64(solang_max, config->args, "-sun_angle_max", false, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + PXOPT_LOOKUP_BOOL(randomSubset, config->args, "-random_subset", false); + PXOPT_LOOKUP_S32(randomLimit, config->args, "-random_limit", false, false); + + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_TIME(time_begin, config->args, "-time_begin", false, false); + PXOPT_LOOKUP_TIME(time_end, config->args, "-time_end", false, false); + PXOPT_LOOKUP_TIME(use_begin, config->args, "-use_begin", false, false); + PXOPT_LOOKUP_TIME(use_end, config->args, "-use_end", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-select_exp_type", "exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-select_inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-select_telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-select_filter", "filter", "=="); + PXOPT_COPY_STR(config->args, where, "-select_uri", "uri", "=="); + PXOPT_COPY_TIME(config->args, where, "-select_dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-select_dateobs_end", "dateobs", "<="); + PXOPT_COPY_F32(config->args, where, "-select_airmass_min", "airmass", ">="); + PXOPT_COPY_F32(config->args, where, "-select_airmass_max", "airmass", "<="); + PXOPT_COPY_F32(config->args, where, "-select_sat_pixel_frac_max", "sat_pixel_frac", "<="); + PXOPT_COPY_F32(config->args, where, "-select_sat_pixel_frac_min", "sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-select_exp_time_min", "exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-select_exp_time_max", "exp_time", "<="); + PXOPT_COPY_F64(config->args, where, "-select_ccd_temp_min", "ccd_temp", ">="); + PXOPT_COPY_F64(config->args, where, "-select_ccd_temp_max", "ccd_temp", "<="); + PXOPT_COPY_F64(config->args, where, "-select_pon_time_min", "pon_time", ">="); + PXOPT_COPY_F64(config->args, where, "-select_pon_time_max", "pon_time", "<="); + PXOPT_COPY_F64(config->args, where, "-select_posang_min", "posang", ">="); + PXOPT_COPY_F64(config->args, where, "-select_posang_max", "posang", "<="); + PXOPT_COPY_F64(config->args, where, "-select_sun_angle_min", "sun_angle", ">="); + PXOPT_COPY_F64(config->args, where, "-select_sun_angle_max", "sun_angle", "<="); + PXOPT_COPY_F64(config->args, where, "-select_sun_alt_min", "sun_alt", ">="); + PXOPT_COPY_F64(config->args, where, "-select_sun_alt_max", "sun_alt", "<="); + + PXOPT_COPY_F64(config->args, where, "-select_moon_angle_min", "moon_angle", ">="); + PXOPT_COPY_F64(config->args, where, "-select_moon_angle_max", "moon_angle", "<="); + PXOPT_COPY_F64(config->args, where, "-select_moon_alt_min", "moon_alt", ">="); + PXOPT_COPY_F64(config->args, where, "-select_moon_alt_max", "moon_alt", "<="); + PXOPT_COPY_F64(config->args, where, "-select_moon_phase_min", "moon_phase", ">="); + PXOPT_COPY_F64(config->args, where, "-select_moon_phase_max", "moon_phase", "<="); + + PXOPT_COPY_STR(config->args, where, "-select_state", "state", "="); + PXOPT_COPY_STR(config->args, where, "-comment", "comment", "LIKE"); + + if (!psListLength(where->list)) { + psFree(where); + where = NULL; + } + + // there is some namespace overlap between the names of the fields we'd + // like to search by to setup a detrun and the names of the fields we'd + // like to assign values to so I've separated them but prepending set- to + // the assigned values + + // search for rawExps with the specified options + psArray *detrendExps = rawExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + // make sure that we found at least one rawExp + if (!detrendExps) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(detrendExps)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(detrendExps); + return true; + } + + if (randomSubset && (randomLimit < detrendExps->n)) { + // generate a random-valued vector, return an index sorted by the random values + psVector *randomVector = psVectorAlloc(detrendExps->n, PS_TYPE_F32); // random values + /* + * change due to PAP work on random number generator? + * psRandom *rng = psRandomAlloc(PS_RANDOM_TAUS, 0); + */ + psRandom *rng = psRandomAlloc(PS_RANDOM_TAUS); + for (int i = 0; i < randomVector->n; i++) { + randomVector->data.F32[i] = psRandomUniform(rng); + } + psVector *indexVector = psVectorSortIndex(NULL, randomVector); + + // accept for first n of the sequence + psArray *subset = psArrayAlloc (randomLimit); + for (int i = 0; i < randomLimit; i++ ){ + int j = indexVector->data.S32[i]; + subset->data[i] = psMemIncrRefCounter (detrendExps->data[j]); + } + psFree (detrendExps); + detrendExps = subset; + } + + // check to see if -filelevel was set on the command line + if (!filelevel) { + filelevel = psStringCopy(((rawExpRow *)detrendExps->data[0])->filelevel); + } + + if (pretend) { + // negative simple so the default is true + if (!rawExpPrintObjects(stdout, detrendExps, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detrendExps); + return false; + } + psFree(detrendExps); + return true; + } + + // start a transaction so we don't end up with childlessed det_ids + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detrendExps); + return false; + } + + // the first iteration is always 0 + // XXX det_id + detRunInsert(config->dbh, + 0, // det_id + 0, // iteration + det_type, + mode, + "run", // state + filelevel, + workdir, + camera, + telescope, + "NA", + reduction, + filter, + airmass_min, + airmass_max, + exp_time_min, + exp_time_max, + ccd_temp_min, + ccd_temp_max, + posang_min, + posang_max, + registered, + time_begin, + time_end, + use_begin, + use_end, + solang_min, + solang_max, + label, + ref_det_id, + ref_iter + ); + psS64 det_id = psDBLastInsertID(config->dbh); + + // create new detInputExp row(s) from the rawExp row(s) + psArray *inputExps = psArrayAllocEmpty(psArrayLength(detrendExps)); + for (long i = 0; i < psArrayLength(detrendExps); i++) { + detInputExpRow *inputExp = rawDetrenTodetInputExpRow( + detrendExps->data[i], + det_id, + 0 // the first iteration is explicitly 0 + ); + psArrayAdd(inputExps, 0, inputExp); + psFree(inputExp); + } + + psFree(detrendExps); + + // insert detInputExp objects into the database + if (!detInputExpInsertObjects(config->dbh, inputExps)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(inputExps); + return false; + } + psFree(inputExps); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + // print the new det_id + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", det_id); + detRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check results + if (psArrayLength(detRuns) != 1) { + psAbort("found more then one detRun matching det_id %" PRId64 " (this should not happen)", det_id); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, detRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detRuns); + return false; + } + psFree(detRuns); + + return true; +} + +static bool definebydetrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // XXX this mode is not well-tested: probably need to specify iteration here + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_STR(det_type, config->args, "-set_det_type", false, false); // optional + + PXOPT_LOOKUP_STR(mode, config->args, "-set_mode", false, false); // optional + if (!isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalid mode"); + return false; + } + + // get -ref_det_id and -ref_iter : required for 'verify' mode / disallowed otherwise + PXOPT_LOOKUP_S64(ref_det_id, config->args, "-ref_det_id", false, false); + PXOPT_LOOKUP_S32(ref_iter, config->args, "-ref_iter", false, false); + if (!strcmp(mode, "verify") && ((ref_det_id == 0) || (ref_iter == -1))) { + psError(PS_ERR_UNKNOWN, false, "verify mode requires both -ref_det_id and -ref_iter"); + return false; + } + if (strcmp(mode, "verify") && ((ref_det_id != 0) || (ref_iter != -1))) { + psError(PS_ERR_UNKNOWN, false, "master mode cannot have -ref_det_id or -ref_iter set"); + return false; + } + + // the new detRun may have different values for these limits: + PXOPT_LOOKUP_STR(exp_type, config->args, "-set_exp_type", false, false); + PXOPT_LOOKUP_STR(filelevel, config->args, "-set_filelevel", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-set_filter", false, false); + PXOPT_LOOKUP_F32(airmass_min, config->args, "-set_airmass_min", false, false); + PXOPT_LOOKUP_F32(airmass_max, config->args, "-set_airmass_max", false, false); + PXOPT_LOOKUP_F32(exp_time_min, config->args, "-set_exp_time_min", false, false); + PXOPT_LOOKUP_F32(exp_time_max, config->args, "-set_exp_time_max", false, false); + PXOPT_LOOKUP_F64(ccd_temp_min, config->args, "-set_ccd_temp_min", false, false); + PXOPT_LOOKUP_F64(ccd_temp_max, config->args, "-set_ccd_temp_max", false, false); + PXOPT_LOOKUP_F64(posang_min, config->args, "-set_posang_min", false, false); + PXOPT_LOOKUP_F64(posang_max, config->args, "-set_posang_max", false, false); + PXOPT_LOOKUP_F64(solang_min, config->args, "-set_sun_angle_min", false, false); + PXOPT_LOOKUP_F64(solang_max, config->args, "-set_sun_angle_max", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + PXOPT_LOOKUP_TIME(time_begin, config->args, "-set_time_begin", false, false); + PXOPT_LOOKUP_TIME(time_end, config->args, "-set_time_end", false, false); + PXOPT_LOOKUP_TIME(use_begin, config->args, "-set_use_begin", false, false); + PXOPT_LOOKUP_TIME(use_end, config->args, "-set_use_end", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); // optional + + // reference iteration to use (otherwise last iteration) + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", false, false); // for specifying input exp restrictions + + // lookup the detRun that we will be basing this one on + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", det_id); + detRuns = detRunSelectRowObjects(config->dbh, where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "database error: no matching det_id found"); + return false; + } + + // sanity check the result... we should have only found one det_id + if (psArrayLength(detRuns) != 1) { + psAbort("found more then one detRun matching det_id %" PRId64 " (this should not happen)", det_id); + } + + // pull the detRun object out the result array + detRunRow *detRun = psMemIncrRefCounter(detRuns->data[0]); + + // discard the resultarray + psFree(detRuns); + + if (iteration < 0) { + iteration = detRun->iteration; + } + + detRun->det_id = 0; // set the det_id to 0/NULL so the database can assign it + detRun->iteration = 0; // reset the iteration to 0 + + detRun->ref_det_id = ref_det_id; // not inherited: may only be set for 'verify' run + detRun->ref_iter = ref_iter; // not inherited: may only be set for 'verify' run + + // reset the state to "run" + psFree(detRun->state); + detRun->state = psStringCopy("run"); + + // walk through the optional values and update the detRun as required. Otherwise, these + // value are inherited from the specified detRun + if (det_type) { + psFree(detRun->det_type); + detRun->det_type = psStringCopy(det_type); + } + + if (mode) { + psFree(detRun->mode); + detRun->mode = psStringCopy(mode); + } + + if (exp_type) { + psFree(detRun->exp_type); + detRun->exp_type = psStringCopy(exp_type); + } + + if (filelevel) { + psFree(detRun->filelevel); + detRun->filelevel = psStringCopy(filelevel); + } + + if (workdir) { + psFree(detRun->workdir); + detRun->workdir = psStringCopy(workdir); + } + + if (filter) { + psFree(detRun->filter); + detRun->filter = psStringCopy(filter); + } + + if (!isnan(airmass_min)) { + detRun->airmass_min = airmass_min; + } + + if (!isnan(airmass_max)) { + detRun->airmass_max = airmass_max; + } + + if (!isnan(exp_time_min)) { + detRun->exp_time_min = exp_time_min; + } + + if (!isnan(exp_time_max)) { + detRun->exp_time_max = exp_time_max; + } + + if (!isnan(ccd_temp_min)) { + detRun->ccd_temp_min = ccd_temp_min; + } + + if (!isnan(ccd_temp_max)) { + detRun->ccd_temp_max = ccd_temp_max; + } + + if (!isnan(posang_min)) { + detRun->posang_min = posang_min; + } + + if (!isnan(posang_max)) { + detRun->posang_max = posang_max; + } + + if (!isnan(solang_min)) { + detRun->solang_min = solang_min; + } + + if (!isnan(solang_max)) { + detRun->solang_max = solang_max; + } + + if (label) { + psFree(detRun->label); + detRun->label = label; + } + + if (registered) { + psFree(detRun->registered); + detRun->registered = psMemIncrRefCounter(registered); + } + + if (time_begin) { + psFree(detRun->time_begin); + detRun->time_begin = psMemIncrRefCounter(time_begin); + } + + if (time_end) { + psFree(detRun->time_end); + detRun->time_end = psMemIncrRefCounter(time_end); + } + + if (use_begin) { + psFree(detRun->use_begin); + detRun->use_begin = psMemIncrRefCounter(use_begin); + } + + if (use_end) { + psFree(detRun->use_end); + detRun->use_end = psMemIncrRefCounter(use_end); + } + + if (reduction) { + psFree(detRun->reduction); + detRun->reduction = psStringCopy(reduction); + } + + // create a metadata to restrict detInputExp's be in in the specified range + + psMetadata *input_filter = psMetadataAlloc(); + + // additional restriction on the detInputExp's to be selected + PXOPT_LOOKUP_TIME(input_begin, config->args, "-set_input_begin", false, false); + if (input_begin) { + PXOPT_COPY_TIME(config->args, input_filter, "-set_input_begin", "dateobs", ">="); + } + PXOPT_LOOKUP_TIME(input_end, config->args, "-set_input_end", false, false); + if (input_end) { + PXOPT_COPY_TIME(config->args, input_filter, "-set_input_end", "dateobs", "<"); + } + PXOPT_LOOKUP_BOOL(only_accepted, config->args, "-only_accepted", false); // optional + + // start a transaction so we don't end up with childlessed det_ids + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(input_filter); + psFree(detRun); + return false; + } + + if (!detRunInsertObject(config->dbh, detRun)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(input_filter); + psFree(detRun); + return false; + } + psFree(detRun); + + // get the det_id + psS64 newDet_id = psDBLastInsertID(config->dbh); + + psString query = pxDataGet("dettool_definebydetrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (input_filter->list->n) { + psString whereClause = psDBGenerateWhereConditionSQL(input_filter, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(input_filter); + + if (only_accepted) { + psStringAppend(&query, " AND accept = 1"); + } + + if (!p_psDBRunQueryF(config->dbh, query, (long long) newDet_id, (long long) det_id, iteration)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // print the new det_id + psArray *newDetRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", newDet_id); + newDetRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!newDetRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check the result... we should have only found one det_id + if (psArrayLength(newDetRuns) != 1) { + psAbort("found more then one detRun matching det_id %" PRId64 " (this should not happen)", newDet_id); + return false; // unreachable + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, newDetRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(newDetRuns); + return false; + } + psFree(newDetRuns); + + return true; +} + +static bool runsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-det_type", "det_type", "=="); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-mode", "mode", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + + // airmass_min < airmass < airmass_max + PXOPT_COPY_F32(config->args, where, "-airmass", "airmass_min", "<="); + PXOPT_COPY_F32(config->args, where, "-airmass", "airmass_max", ">="); + + // exp_time_min < exp_time < exp_time_max + PXOPT_COPY_F32(config->args, where, "-exp_time", "exp_time_min", "<="); + PXOPT_COPY_F32(config->args, where, "-exp_time", "exp_time_max", ">="); + + // ccd_temp_min < ccd_temp < ccd_temp_max + PXOPT_COPY_F32(config->args, where, "-ccd_temp", "ccd_temp_min", "<="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp", "ccd_temp_max", ">="); + + PXOPT_COPY_F64(config->args, where, "-posang", "posang_min", "<="); + PXOPT_COPY_F64(config->args, where, "-posang", "posang_max", ">="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(active, config->args, "-active", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("dettool_runs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list) && active) { + psString whereClause = psDBGenerateWhereSQL(where, "detRun"); + psStringAppend(&query, " %s AND (detRun.state = 'run' OR detRun.state = 'stop' OR detRun.state = 'register')", whereClause); + psFree(whereClause); + } + if (psListLength(where->list) && !active) { + psString whereClause = psDBGenerateWhereSQL(where, "detRun"); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + if (!psListLength(where->list) && active) { + psStringAppend(&query, " WHERE (detRun.state = 'run' OR detRun.state = 'stop' OR detRun.state = 'register')"); + } + psFree (where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *runs = p_psDBFetchResult(config->dbh); + if (!runs) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(runs)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(runs); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, runs, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(runs); + return false; + } + + psFree(runs); + + return true; +} + +static bool childlessrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-det_type", "det_type", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("dettool_childlessrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detRun"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree (where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static detInputExpRow *rawDetrenTodetInputExpRow(rawExpRow *rawExp, psS64 det_id, psS32 iteration) +{ + PS_ASSERT_PTR_NON_NULL(rawExp, NULL); + + return detInputExpRowAlloc( + det_id, + iteration, + rawExp->exp_id, + true // use + ); +} + +static bool inputMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + psString query = pxDataGet("dettool_input.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, "detInputExp"); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree (where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detInputExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool rawMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + + psString query = pxDataGet("dettool_raw.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree (where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool updatedetrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool updating = false; + long rows = 0; + psMetadata *where = psMetadataAlloc(); + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + + PXOPT_LOOKUP_BOOL(again, config->args, "-again", false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + + psMetadata *values = psMetadataAlloc(); + PXOPT_LOOKUP_TIME(time_begin, config->args, "-set_time_begin", false, false); + if (time_begin) { + updating = true; + PXOPT_COPY_TIME(config->args, values, "-set_time_begin", "time_begin", "=="); + } + PXOPT_LOOKUP_TIME(time_end, config->args, "-set_time_end", false, false); + if (time_end) { + updating = true; + PXOPT_COPY_TIME(config->args, values, "-set_time_end", "time_end", "=="); + } + PXOPT_LOOKUP_TIME(use_begin, config->args, "-set_use_begin", false, false); + if (use_begin) { + updating = true; + PXOPT_COPY_TIME(config->args, values, "-set_use_begin", "use_begin", "=="); + } + PXOPT_LOOKUP_TIME(use_end, config->args, "-set_use_end", false, false); + if (use_end) { + updating = true; + PXOPT_COPY_TIME(config->args, values, "-set_use_end", "use_end", "=="); + } + if (state) { + updating = true; + if (!isValidDetRunState (state)) return false; + PXOPT_COPY_STR(config->args, values, "-state", "state", "=="); + } + + PXOPT_LOOKUP_STR(det_type, config->args, "-set_det_type", false, false); + if (det_type) { + updating = true; + PXOPT_COPY_STR(config->args, values, "-set_det_type", "det_type", "=="); + } + + + // either -rerun or -state must be specified + if (!(again || updating)) { + psError(PS_ERR_UNKNOWN, true, "either -again or update parameters must be specified"); + return false; + } + if (again && updating) { + psError(PS_ERR_UNKNOWN, true, "-again and update parameters are exclusive"); + return false; + } + + if (updating) { + rows = psDBUpdateRows(config->dbh,"detRun",where,values); + if (rows) { + return(true); + } + else { + return(false); + } + } + + // else + // -again + if (!startNewIteration(config, det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to start new iteration"); + return false; + } + + return true; +} + +// used by updatedetrunMode +bool startNewIteration(pxConfig *config, psS64 det_id) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = pxDataGet("dettool_start_new_iteration.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // restrict the update only to the specified det_id + psStringAppend(&query, " WHERE det_id = %" PRId64 , det_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "det_id %" PRId64 " not found", det_id); + psFree(output); + return false; + } + + // start a transaction so we don't end up with an incremented iteration + // count but no detInputExps + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // up the detRuns iteration count for for the single det_id we are + // operating on + // XXX this will have to changed in order to support multiple det_ids with + // a single invocation of this functions + psS32 newIteration = incrementIteration(config, det_id); + if (!newIteration) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + bool status = false; + psS64 exp_id = psMetadataLookupS64(&status, row, "exp_id"); + if (!status) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for exp_id"); + psFree(output); + return false; + } + bool accept = psMetadataLookupBool(&status, row, "accept"); + if (!status) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for accept"); + psFree(output); + return false; + } + + // detResidExp.include is used to set detInputExp.include + if (!detInputExpInsert( + config->dbh, + det_id, + newIteration, + exp_id, + accept + ) + ) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + } + + psFree(output); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool rerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id is required + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + + // we have to support multipe exp_ids + psMetadataItem *item = psMetadataLookup(config->args, "-exp_id"); + if (!item) { + // this shouldn't actually happen when using psArgs + psError(PS_ERR_UNKNOWN, true, "-exp_id is required"); + return false; + } + + psList *exp_id_list = item->data.list; + psMetadata *where = psMetadataAlloc(); + + // make sure that -exp_id was parsed correctly + if (item->type == PS_DATA_METADATA_MULTI) { + psListIterator *iter = psListIteratorAlloc(item->data.list, 0, false); + psMetadataItem *mItem = NULL; + while ((mItem = psListGetAndIncrement(iter))) { + psS64 exp_id = mItem->data.S64; + if (!psMetadataAddS64(where, PS_LIST_TAIL, "exp_id", PS_META_DUPLICATE_OK, "==", exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(iter); + psFree(where); + return false; + } + } + psFree(iter); + } + if (item->type == PS_DATA_S64) { + psS64 exp_id = item->data.S64; + if (!psMetadataAddS64(where, PS_LIST_TAIL, "exp_id", PS_META_DUPLICATE_OK, "==", exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(where); + return false; + } + } + +# if (0) + // make sure that -exp_id was parsed correctly + // XXX this can be removed someday + if (item->type == PS_DATA_METADATA_MULTI) { + psListIterator *iter = psListIteratorAlloc(item->data.list, 0, false); + psMetadataItem *mItem = NULL; + while ((mItem = psListGetAndIncrement(iter))) { + psString exp_id = mItem->data.V; + // if exp_id is NULL then it means that -exp_id has not been + // specified + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, + "at least one -exp_id is required"); + psFree(where); + return false; + } + + if (!psMetadataAddStr(where, PS_LIST_TAIL, "exp_id", + PS_META_DUPLICATE_OK, "==", exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(iter); + psFree(where); + return false; + } + } + psFree(iter); + } else { + psAbort("-exp_id was not parsed correctly (this should not happen"); + } +# endif + + // check that the specified exp_ids actually exist in the iteration zero + // detInputExp set + + // add the det_id & iteration == 0 to the where clause + if (!psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return false; + } + if (!psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item iteration"); + psFree(where); + return false; + } + + psArray *detrendExps = detInputExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!detrendExps) { + psError(PS_ERR_UNKNOWN, false, "no rawExp rows found"); + psFree(where); + return false; + } + + // build a hash for the valid exp_ids + psHash *valid_exp_ids = psHashAlloc(psArrayLength(detrendExps)); + for (long i = 0; i < psArrayLength(detrendExps); i++) { + psString exp_idStr = psDBIntToString(((detInputExpRow *)detrendExps->data[i])->exp_id); + psHashAdd(valid_exp_ids, exp_idStr, detrendExps->data[i]); + psFree(exp_idStr); + } + psFree(detrendExps); + + // start a transaction so we don't end up with an incremented iteration + // count but no detInputExps + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(valid_exp_ids); + return false; + } + + // up the detRuns iteration count + psS32 newIteration = incrementIteration(config, det_id); + if (!newIteration) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(valid_exp_ids); + return false; + } + + // check exp_ids and build up an array of new detInputExp rows at the same + // time + psListIterator *iter = psListIteratorAlloc(exp_id_list, 0, false); + psMetadataItem *mItem = NULL; + psArray *newInputExps = psArrayAllocEmpty(psListLength(exp_id_list)); + while ((mItem = psListGetAndIncrement(iter))) { + psString exp_idStr = psDBIntToString(mItem->data.S64); + detInputExpRow *inputExp = psHashLookup(valid_exp_ids, exp_idStr); + if (!inputExp) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + // invalid exp_id + psError(PS_ERR_UNKNOWN, false, "exp_id %" PRId64 " is invalid for det_id %" PRId64, mItem->data.S64, det_id); + psFree(iter); + psFree(valid_exp_ids); + return false; + } + detInputExpRow *newInputExp = detInputExpRowAlloc( + det_id, + newIteration, + inputExp->exp_id, + true // use + ); + psArrayAdd(newInputExps, 0, newInputExp); + psFree(newInputExp); + } + psFree(iter); + psFree(valid_exp_ids); + + for (long i = 0; i < psArrayLength(newInputExps); i++) { + if (!detInputExpInsertObject(config->dbh, newInputExps->data[i])) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(newInputExps); + return false; + } + } + psFree(newInputExps); + + // point of no return for det_id creation + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool register_detrendMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(det_type, config->args, "-det_type", true, false); // required + PXOPT_LOOKUP_STR(filelevel, config->args, "-filelevel", true, false); // required + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", false, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", false, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", false, false); + PXOPT_LOOKUP_STR(exp_type, config->args, "-exp_type", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", false, false); + PXOPT_LOOKUP_F32(airmass_min, config->args, "-airmass_min", false, false); + PXOPT_LOOKUP_F32(airmass_max, config->args, "-airmass_max", false, false); + PXOPT_LOOKUP_F32(exp_time_min, config->args, "-exp_time_min", false, false); + PXOPT_LOOKUP_F32(exp_time_max, config->args, "-exp_time_max", false, false); + PXOPT_LOOKUP_F32(ccd_temp_min, config->args, "-ccd_temp_min", false, false); + PXOPT_LOOKUP_F32(ccd_temp_max, config->args, "-ccd_temp_max", false, false); + PXOPT_LOOKUP_F64(posang_min, config->args, "-posang_min", false, false); + PXOPT_LOOKUP_F64(posang_max, config->args, "-posang_max", false, false); + PXOPT_LOOKUP_F64(solang_min, config->args, "-sun_angle_min", false, false); + PXOPT_LOOKUP_F64(solang_max, config->args, "-sun_angle_max", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_TIME(time_begin, config->args, "-time_begin", false, false); + PXOPT_LOOKUP_TIME(time_end, config->args, "-time_end", false, false); + PXOPT_LOOKUP_TIME(use_begin, config->args, "-use_begin", false, false); + PXOPT_LOOKUP_TIME(use_end, config->args, "-use_end", false, false); + PXOPT_LOOKUP_S64(ref_det_id, config->args, "-ref_det_id", false, false); + PXOPT_LOOKUP_S32(ref_iter, config->args, "-ref_iter", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!detRunInsert(config->dbh, + 0, // det_id + 0, // the iteration is fixed at 0 + det_type, + "register", // mode is required to be "register" + "register", // state + filelevel, + workdir, + camera, + telescope, + exp_type, + NULL, + filter, + airmass_min, + airmass_max, + exp_time_min, + exp_time_max, + ccd_temp_min, + ccd_temp_max, + posang_min, + posang_max, + registered, + time_begin, + time_end, + use_begin, + use_end, + solang_min, + solang_max, + label, + ref_det_id, + ref_iter + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + // print the new detRun + psS64 det_id = psDBLastInsertID(config->dbh); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *detRuns = NULL; + { + psMetadata *where = psMetadataAlloc(); + psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", det_id); + detRuns = psDBSelectRows(config->dbh, "detRun", where, 0); + psFree(where); + } + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check results + if (psArrayLength(detRuns) != 1) { + psAbort("found more then one detRun matching det_id %" PRId64 "(this should not happen)", det_id); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, detRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detRuns); + return false; + } + psFree(detRuns); + + return true; +} + +// NOTE : this function is also used by addcorrectimfileMode +bool register_detrend_imfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); // required + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); // required + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + if (!detRegisteredImfileInsert(config->dbh, + det_id, + 0, // the iteration is fixed at 0 + class_id, + uri, + bg, + bg_stdev, + bg_mean_stdev, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + 0 // fault code + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static psS32 incrementIteration(pxConfig *config, psS64 det_id) +{ + // this function returns zero on error + PS_ASSERT_PTR_NON_NULL(config, 0); + + char *query = "UPDATE detRun SET iteration = iteration + 1 WHERE det_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, det_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to increment iteration for det_id %" PRId64, det_id); + return 0; + } + + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return 0; + } + + psArray *detRuns = detRunSelectRowObjects(config->dbh, where, 0); + psFree(where); + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "no detRun rows found"); + return 0; + } + + // sanity check the database + if (psArrayLength(detRuns) != 1) { + // this should no happen + psAbort( "database query return too many rows (this should not happen"); + } + + psS32 newIteration = ((detRunRow *)detRuns->data[0])->iteration; + psFree(detRuns); + + return newIteration; +} + +bool setDetRunState(pxConfig *config, psS64 det_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + if (!isValidDetRunState (state)) return false; + + char *query = "UPDATE detRun SET state = '%s' WHERE det_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, det_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for det_id %" PRId64, det_id); + return false; + } + + return true; +} + +// the detRun states are a superset of the data states below +bool isValidDetRunState (const char *state) { + + // check that state is a valid string value + if (!strcmp(state, "run")) return true; + if (!strcmp(state, "stop")) return true; + if (!strcmp(state, "drop")) return true; + if (!strcmp(state, "wait")) return true; + if (!strcmp(state, "test")) return true; + if (!strcmp(state, "ignore")) return true; + if (!strcmp(state, "register")) return true; + + psError(PS_ERR_UNKNOWN, true, "invalid detRun state: %s", state); + return false; +} + +bool isValidDataState (const char *data_state) { + + // check that state is a valid string value + if (!strcmp(data_state, "run")) return true; + if (!strcmp(data_state, "stop")) return true; + if (!strcmp(data_state, "drop")) return true; + if (!strcmp(data_state, "register")) return true; + // These are valid data states, and are necessary for the cleanup to work correctly. + if (!strcmp(data_state, "full")) return true; + if (!strcmp(data_state, "goto_cleaned")) return true; + if (!strcmp(data_state, "goto_scrubbed")) return true; + if (!strcmp(data_state, "goto_purged")) return true; + if (!strcmp(data_state, "cleaned")) return true; + if (!strcmp(data_state, "scrubbed")) return true; + if (!strcmp(data_state, "purged")) return true; + if (!strcmp(data_state, "error_cleaned")) return true; + if (!strcmp(data_state, "error_scrubbed")) return true; + if (!strcmp(data_state, "error_purged")) return true; + + psError(PS_ERR_UNKNOWN, true, "invalid data state: %s", data_state); + return false; +} + +bool isValidMode(pxConfig *config, const char *mode) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(mode, false); + + // check that state is a valid string value + if (!strcmp(mode, "master")) return true; + if (!strcmp(mode, "verify")) return true; + + psError(PS_ERR_UNKNOWN, false, "invalid detRun mode: %s", mode); + return false; +} + + +bool setProcessedImfileDataState(pxConfig *config, psMetadata *where, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(where, false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + psString query = psStringCopy("UPDATE detProcessedImfile SET data_state = '%s'"); + if (where && psListLength(where->list) > 0) { + psString whereClause = psDBGenerateWhereSQL(where,NULL); + psStringAppend(&query," %s",whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return(false); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + psFree(query); + return(true); +} + +bool setProcessedExpDataState(pxConfig *config, psMetadata *where, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(where, false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + psString query = psStringCopy("UPDATE detProcessedExp SET data_state = '%s'"); + if (where && psListLength(where->list) > 0) { + psString whereClause = psDBGenerateWhereSQL(where,NULL); + psStringAppend(&query," %s",whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return(false); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + psFree(query); + return(true); +} + +bool setResidImfileDataState(pxConfig *config, psMetadata *where, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(where,false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + psString query = psStringCopy("UPDATE detResidImfile SET data_state = '%s'"); + if (where && psListLength(where->list) > 0) { + psString whereClause = psDBGenerateWhereSQL(where,NULL); + psStringAppend(&query," %s",whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return(false); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + psFree(query); + return(true); +} +// This function apparently not used anymore. +bool setResidExpDataState(pxConfig *config, psMetadata *where, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(where,false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + psString query = psStringCopy("UPDATE detResidExp SET data_state = '%s'"); + if (where && psListLength(where->list) > 0) { + psString whereClause = psDBGenerateWhereSQL(where,NULL); + psStringAppend(&query," %s",whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return(false); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + psFree(query); + return(true); +} + +// Not yet updated for cleaning +bool setStackedImfileDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *class_id, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(class_id, false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + char *query = "UPDATE detStackedImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32 + " AND class_id = '%s'"; + if (!p_psDBRunQueryF(config->dbh, query, data_state, det_id, iteration, class_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for det_id %" PRId64 ", iteration %" PRId32 "class_id %s", + det_id, iteration, class_id); + return false; + } + + return true; +} + +bool setNormStatImfileDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *class_id, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(class_id, false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + char *query = "UPDATE detNormalizedStatImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32 + " AND class_id = '%s'"; +/* fprintf(stderr,"DETTOOL SAYS: %s\n",query); */ + if (!p_psDBRunQueryF(config->dbh, query, data_state, det_id, iteration,class_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for det_id %" PRId64 ", iteration %" PRId32, + det_id, iteration); + return false; + } + + return true; +} + +bool setNormImfileDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *class_id, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(class_id, false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + char *query = "UPDATE detNormalizedImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32 + " AND class_id = '%s'"; + if (!p_psDBRunQueryF(config->dbh, query, data_state, det_id, iteration, class_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for det_id %" PRId64 ", iteration %" PRId32 " class %s", + det_id, iteration, class_id); + return false; + } + + return true; +} + +bool setNormExpDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(data_state, false); + + if (!isValidDataState (data_state)) return false; + + char *query = "UPDATE detNormalizedExp SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query, data_state, det_id, iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for det_id %" PRId64 ", iteration %" PRId32, + det_id, iteration); + return false; + } + + return true; +} +// End not yet modified for cleanup. + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 12; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // replaced below PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + + ExportTable tables [] = { + {"detRun", "dettool_export_run.sql"}, + {"detInputExp", "dettool_export_input_exp.sql"}, + {"detNormalizedExp", "dettool_export_normalized_exp.sql"}, + {"detNormalizedImfile", "dettool_export_normalized_imfile.sql"}, + {"detNormalizedStatImfile", "dettool_export_normalized_stat_imfile.sql"}, + {"detProcessedExp", "dettool_export_processed_exp.sql"}, + {"detProcessedImfile", "dettool_export_processed_imfile.sql"}, + {"detRegisteredImfile", "dettool_export_registered_imfile.sql"}, + {"detResidExp", "dettool_export_resid_exp.sql"}, + {"detResidImfile", "dettool_export_resid_imfile.sql"}, + {"detRunSummary", "dettool_export_run_summary.sql"}, + {"detStackedImfile", "dettool_export_stacked_imfile.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + int numImportTables = 11; + + char tables[11] [80] = {"detInputExp", "detNormalizedExp", + "detNormalizedImfile", "detNormalizedStatImfile", "detProcessedExp", + "detProcessedImfile", "detRegisteredImfile", "detResidExp", + "detResidImfile", "detRunSummary", "detStackedImfile"}; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + + fprintf (stdout, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); + + psMetadataItem *item = psMetadataLookup (input, "detRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detRunRow *detRun = detRunObjectFromMetadata (entry->data.md); + detRunInsertObject (config->dbh, detRun); + + // fprintf (stdout, "---- det run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + for (int i = 0; i < numImportTables; i++) { + item = psMetadataLookup (input, tables[i]); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + switch (i) { + case 0: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detInputExpRow *detInputExp = detInputExpObjectFromMetadata (entry->data.md); + detInputExpInsertObject (config->dbh, detInputExp); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 1: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detNormalizedExpRow *detNormalizedExp = detNormalizedExpObjectFromMetadata (entry->data.md); + detNormalizedExpInsertObject (config->dbh, detNormalizedExp); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 2: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detNormalizedStatImfileRow *detNormalizedStatImfile = detNormalizedStatImfileObjectFromMetadata (entry->data.md); + detNormalizedStatImfileInsertObject (config->dbh, detNormalizedStatImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 3: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detProcessedExpRow *detProcessedExp = detProcessedExpObjectFromMetadata (entry->data.md); + detProcessedExpInsertObject (config->dbh, detProcessedExp); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 4: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detProcessedImfileRow *detProcessedImfile = detProcessedImfileObjectFromMetadata (entry->data.md); + detProcessedImfileInsertObject (config->dbh, detProcessedImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 5: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detRegisteredImfileRow *detRegisteredImfile = detRegisteredImfileObjectFromMetadata (entry->data.md); + detRegisteredImfileInsertObject (config->dbh, detRegisteredImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 6: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detResidExpRow *detResidExp = detResidExpObjectFromMetadata (entry->data.md); + detResidExpInsertObject (config->dbh, detResidExp); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 7: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detResidImfileRow *detResidImfile = detResidImfileObjectFromMetadata (entry->data.md); + detResidImfileInsertObject (config->dbh, detResidImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 8: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detNormalizedImfileRow *detNormalizedImfile = detNormalizedImfileObjectFromMetadata (entry->data.md); + detNormalizedImfileInsertObject (config->dbh, detNormalizedImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 9: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detRunSummaryRow *detRunSummary = detRunSummaryObjectFromMetadata (entry->data.md); + detRunSummaryInsertObject (config->dbh, detRunSummary); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 10: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + detStackedImfileRow *detStackedImfile = detStackedImfileObjectFromMetadata (entry->data.md); + detStackedImfileInsertObject (config->dbh, detStackedImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + } + } + + return true; +} + +#if 0 +// XXX this function was left in commented as this method may be useful in the +// future +static psArray *validDetInputClassIds(pxConfig *config, const char *det_id) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + // det_id is input as a string because the fact that it is an integer + // is just a database impliementation detail. + PS_ASSERT_PTR_NON_NULL(det_id, NULL); + + psArray *rawImfiles = searchInputImfiles(config, det_id); + if (!rawImfiles) { + return NULL; + } + + psArray *valid_class_ids = NULL; + { + // All this jumping through hoops is so that we end up with unique + // values. PP thinks that making multiple passes through this array and + // deleting matched elements would end up being more expensive then a + // double sort and stagger scheme. JH thinks it would be cheaper to + // just do a unqiue sort and delete. So this is really just a cheap + // hack to avoid implimenting a unique sort function but at least it's + // stable. + psHash *hash = psHashAlloc(psArrayLength(rawImfiles)); + for (long i = 0; i < psArrayLength(rawImfiles); i++) { + psHashAdd(hash, + ((rawImfileRow *)rawImfiles->data[i])->class_id, + ((rawImfileRow *)rawImfiles->data[i])->class_id + ); + } + valid_class_ids = psHashToArray(hash); + psFree(hash); + } + psFree(rawImfiles); + + return valid_class_ids; +} + +static psArray *searchInputImfiles(pxConfig *config, const char *det_id) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + // det_id is input as a string because the fact that it is an integer + // is just a database impliementation detail. + PS_ASSERT_PTR_NON_NULL(det_id, NULL); + + psArray *inputExps = NULL; + { + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddS32(where, PS_LIST_TAIL, "det_id", 0, "==", + (psS32)atoi(det_id))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item det_id"); + psFree(where); + return NULL; + } + inputExps = detInputExpSelectRowObjects(config->dbh, where, 0); + psFree(where); + } + if (!inputExps) { + psError(PS_ERR_UNKNOWN, false, "no detInputExp rows found"); + return NULL; + } + + // find rawImfiles associated with detInputExps + psArray *rawImfiles = NULL; + { + psMetadata *where = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(inputExps); i++) { + detInputExpRow *row = (detInputExpRow *row) inputExps->data[i]; + if (!psMetadataAddS64(where, PS_LIST_TAIL, "exp_id", PS_META_DUPLICATE_OK, "==", row->exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(inputExps); + psFree(where); + return NULL; + } + } + psFree(inputExps); + rawImfiles = rawImfileSelectRowObjects(config->dbh, where, 0); + // XXX this really should be sorted for uniqueness + psFree(where); + } + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + return NULL; + } + + return rawImfiles; +} +#endif + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool.h (revision 41104) @@ -0,0 +1,236 @@ +/* + * dettool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#ifndef DETTOOL_H +#define DETTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + DETTOOL_MODE_NONE = PXTOOL_MODE_NONE, + DETTOOL_MODE_PENDING, + DETTOOL_MODE_DEFINEBYTAG, + DETTOOL_MODE_DEFINEBYQUERY, + DETTOOL_MODE_DEFINEBYDETRUN, + DETTOOL_MODE_MAKECORRECTION, + DETTOOL_MODE_TOCORRECTEXP, + DETTOOL_MODE_TOCORRECTIMFILE, + DETTOOL_MODE_ADDCORRECTIMFILE, + DETTOOL_MODE_RUNS, + DETTOOL_MODE_CHILDLESSRUN, + DETTOOL_MODE_INPUT, + DETTOOL_MODE_RAW, + + DETTOOL_MODE_TOPROCESSEDIMFILE, + DETTOOL_MODE_ADDPROCESSEDIMFILE, + DETTOOL_MODE_PROCESSEDIMFILE, + DETTOOL_MODE_REVERTPROCESSEDIMFILE, + DETTOOL_MODE_UPDATEPROCESSEDIMFILE, + DETTOOL_MODE_PENDINGCLEANUP_PROCESSEDIMFILE, + DETTOOL_MODE_DONECLEANUP_PROCESSEDIMFILE, + + DETTOOL_MODE_TOPROCESSEDEXP, + DETTOOL_MODE_ADDPROCESSEDEXP, + DETTOOL_MODE_PROCESSEDEXP, + DETTOOL_MODE_REVERTPROCESSEDEXP, + DETTOOL_MODE_UPDATEPROCESSEDEXP, + DETTOOL_MODE_PENDINGCLEANUP_PROCESSEDEXP, + DETTOOL_MODE_DONECLEANUP_PROCESSEDEXP, + DETTOOL_MODE_UPDATESTATE_PROCESSED, + + DETTOOL_MODE_TOSTACKED, + DETTOOL_MODE_ADDSTACKED, + DETTOOL_MODE_STACKED, + DETTOOL_MODE_REVERTSTACKED, + DETTOOL_MODE_UPDATESTACKED, + DETTOOL_MODE_PENDINGCLEANUP_STACKED, + DETTOOL_MODE_DONECLEANUP_STACKED, + + DETTOOL_MODE_TONORMALIZEDSTAT, + DETTOOL_MODE_ADDNORMALIZEDSTAT, + DETTOOL_MODE_NORMALIZEDSTAT, + DETTOOL_MODE_REVERTNORMALIZEDSTAT, + DETTOOL_MODE_UPDATENORMALIZEDSTAT, + DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDSTAT, + DETTOOL_MODE_DONECLEANUP_NORMALIZEDSTAT, + + DETTOOL_MODE_TONORMALIZE, + DETTOOL_MODE_ADDNORMALIZEDIMFILE, + DETTOOL_MODE_NORMALIZEDIMFILE, + DETTOOL_MODE_REVERTNORMALIZEDIMFILE, + DETTOOL_MODE_UPDATENORMALIZEDIMFILE, + DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDIMFILE, + DETTOOL_MODE_DONECLEANUP_NORMALIZEDIMFILE, + + DETTOOL_MODE_TONORMALIZEDEXP, + DETTOOL_MODE_ADDNORMALIZEDEXP, + DETTOOL_MODE_NORMALIZEDEXP, + DETTOOL_MODE_REVERTNORMALIZEDEXP, + DETTOOL_MODE_UPDATENORMALIZEDEXP, + DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDEXP, + DETTOOL_MODE_DONECLEANUP_NORMALIZEDEXP, + + DETTOOL_MODE_TORESIDIMFILE, + DETTOOL_MODE_ADDRESIDIMFILE, + DETTOOL_MODE_RESIDIMFILE, + DETTOOL_MODE_REVERTRESIDIMFILE, + DETTOOL_MODE_UPDATERESIDIMFILE, + DETTOOL_MODE_PENDINGCLEANUP_RESIDIMFILE, + DETTOOL_MODE_DONECLEANUP_RESIDIMFILE, + + DETTOOL_MODE_TORESIDEXP, + DETTOOL_MODE_ADDRESIDEXP, + DETTOOL_MODE_RESIDEXP, + DETTOOL_MODE_REVERTRESIDEXP, + DETTOOL_MODE_UPDATERESIDEXP, + DETTOOL_MODE_PENDINGCLEANUP_RESIDEXP, + DETTOOL_MODE_DONECLEANUP_RESIDEXP, + DETTOOL_MODE_UPDATESTATE_RESID, + + DETTOOL_MODE_TODETRUNSUMMARY, + DETTOOL_MODE_ADDDETRUNSUMMARY, + DETTOOL_MODE_DETRUNSUMMARY, + DETTOOL_MODE_REVERTDETRUNSUMMARY, + DETTOOL_MODE_UPDATEDETRUNSUMMARY, + DETTOOL_MODE_PENDINGCLEANUP_DETRUNSUMMARY, + DETTOOL_MODE_UPDATEDETRUN, + DETTOOL_MODE_RERUN, + DETTOOL_MODE_REGISTER_DETREND, + DETTOOL_MODE_REGISTER_DETREND_IMFILE, + + DETTOOL_MODE_EXPORTRUN, + DETTOOL_MODE_IMPORTRUN +} dettoolMode; + +pxConfig *dettoolConfig(pxConfig *config, int argc, char **argv); + +// correct +bool makecorrectionMode(pxConfig *config); +bool tocorrectexpMode(pxConfig *config); +bool tocorrectimfileMode(pxConfig *config); +bool addcorrectimfileMode(pxConfig *config); + +// register +bool register_detrend_imfileMode(pxConfig *config); + +// processedimfile +bool toprocessedimfileMode(pxConfig *config); +bool addprocessedimfileMode(pxConfig *config); +bool processedimfileMode(pxConfig *config); +bool revertprocessedimfileMode(pxConfig *config); +bool updateprocessedimfileMode(pxConfig *config); +bool pendingcleanup_processedimfileMode(pxConfig *config); +bool donecleanup_processedimfileMode(pxConfig *config); + +// processedexp +bool toprocessedexpMode(pxConfig *config); +bool addprocessedexpMode(pxConfig *config); +bool processedexpMode(pxConfig *config); +bool revertprocessedexpMode(pxConfig *config); +bool updateprocessedexpMode(pxConfig *config); +bool pendingcleanup_processedexpMode(pxConfig *config); +bool donecleanup_processedexpMode(pxConfig *config); +bool updatestateprocessedMode(pxConfig *config); + +// stackedimfile +bool tostackedMode(pxConfig *config); +bool addstackedMode(pxConfig *config); +bool stackedMode(pxConfig *config); +bool revertstackedMode(pxConfig *config); +bool updatestackedMode(pxConfig *config); +bool pendingcleanup_stackedMode(pxConfig *config); +bool donecleanup_stackedMode(pxConfig *config); + +// normalizedstat +bool tonormalizedstatMode(pxConfig *config); +bool addnormalizedstatMode(pxConfig *config); +bool normalizedstatMode(pxConfig *config); +bool revertnormalizedstatMode(pxConfig *config); +bool updatenormalizedstatMode(pxConfig *config); +bool pendingcleanup_normalizedstatMode(pxConfig *config); +bool donecleanup_normalizedstatMode(pxConfig *config); + +// normalizedimfile +bool tonormalizeMode(pxConfig *config); +bool addnormalizedimfileMode(pxConfig *config); +bool normalizedimfileMode(pxConfig *config); +bool revertnormalizedimfileMode(pxConfig *config); +bool updatenormalizedimfileMode(pxConfig *config); +bool pendingcleanup_normalizedimfileMode(pxConfig *config); +bool donecleanup_normalizedimfileMode(pxConfig *config); + +// normalizedexp +bool tonormalizedexpMode(pxConfig *config); +bool addnormalizedexpMode(pxConfig *config); +bool normalizedexpMode(pxConfig *config); +bool revertnormalizedexpMode(pxConfig *config); +bool updatenormalizedexpMode(pxConfig *config); +bool pendingcleanup_normalizedexpMode(pxConfig *config); +bool donecleanup_normalizedexpMode(pxConfig *config); + +// residimfile +bool toresidimfileMode(pxConfig *config); +bool addresidimfileMode(pxConfig *config); +bool residimfileMode(pxConfig *config); +bool revertresidimfileMode(pxConfig *config); +bool updateresidimfileMode(pxConfig *config); +bool pendingcleanup_residimfileMode(pxConfig *config); +bool donecleanup_residimfileMode(pxConfig *config); + +// residexp +bool toresidexpMode(pxConfig *config); +bool addresidexpMode(pxConfig *config); +bool residexpMode(pxConfig *config); +bool revertresidexpMode(pxConfig *config); +bool updateresidexpMode(pxConfig *config); +bool pendingcleanup_residexpMode(pxConfig *config); +bool donecleanup_residexpMode(pxConfig *config); +bool updatestateresidMode(pxConfig *config); + +// detrunsummary +bool todetrunsummaryMode(pxConfig *config); +bool adddetrunsummaryMode(pxConfig *config); +bool detrunsummaryMode(pxConfig *config); +bool revertdetrunsummaryMode(pxConfig *config); +bool updatedetrunsummaryMode(pxConfig *config); +bool pendingcleanup_detrunsummaryMode(pxConfig *config); + +// other utilities +bool startNewIteration(pxConfig *config, psS64 det_id); +bool setDetRunState(pxConfig *config, psS64 det_id, const char *state); +bool isValidDetRunState (const char *state); +bool isValidDataState (const char *data_state); +bool isValidMode(pxConfig *config, const char *mode); + +// functions to set the 'data_state' for stages +bool setProcessedImfileDataState(pxConfig *config, psMetadata *where, const char *data_state); +bool setProcessedExpDataState(pxConfig *config, psMetadata *where, const char *data_state); +bool setResidImfileDataState(pxConfig *config, psMetadata *where, const char *data_state); +bool setResidExpDataState(pxConfig *config, psMetadata *where, const char *data_state); + +bool setStackedImfileDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *class_id, const char *data_state); +bool setNormStatImfileDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *class_id, const char *data_state); +bool setNormImfileDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *class_id, const char *data_state); +bool setNormExpDataState(pxConfig *config, psS64 det_id, psS32 iteration, const char *data_state); + +#endif // DETTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettoolConfig.c (revision 41104) @@ -0,0 +1,1038 @@ +/* + * dettoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +pxConfig *dettoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -pending + // XXX EAM : is this used? does it make sense? + psMetadata *pendingArgs = psMetadataAlloc(); + psMetadataAddS64(pendingArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exposure type", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-inst", 0, "search by camera", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-telescope", 0, "search by telescope", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-uri", 0, "search by URL", NULL); + psMetadataAddBool(pendingArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -definebytag + psMetadata *definebytagArgs = psMetadataAlloc(); + psMetadataAddS64(definebytagArgs, PS_LIST_TAIL, "-exp_id", PS_META_DUPLICATE_OK, "include this exposure (multiple OK, required)", 0); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-det_type", 0, "define the type of detrend run (required)", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-mode", 0, "define the mode of this detrend run (master, verify)", "master"); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-filelevel", 0, "define filelevel", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-inst", 0, "define camera", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-exp_type", 0, "define exposure type", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-filter", 0, "define filter ", NULL); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(definebytagArgs, PS_LIST_TAIL, "-exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF64(definebytagArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + + psMetadataAddTime(definebytagArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddTime(definebytagArgs, PS_LIST_TAIL, "-time_begin", 0, "detrend applies to exposures taken during this period", NULL); + psMetadataAddTime(definebytagArgs, PS_LIST_TAIL, "-time_end", 0, "detrend applies to exposures taken during this period", NULL); + psMetadataAddTime(definebytagArgs, PS_LIST_TAIL, "-use_begin", 0, "start of detrend run applicable period", NULL); + psMetadataAddTime(definebytagArgs, PS_LIST_TAIL, "-use_end", 0, "end of detrend run applicable period", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-reduction", 0, "define reduction class for processing", NULL); + psMetadataAddStr(definebytagArgs, PS_LIST_TAIL, "-label", 0, "define detrun label", NULL); + psMetadataAddBool(definebytagArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadataAddS64(definebytagArgs, PS_LIST_TAIL, "-ref_det_id", 0, "define reference detrend (verify mode only)", 0); + psMetadataAddS32(definebytagArgs, PS_LIST_TAIL, "-ref_iter", 0, "define reference detrend (verify mode only)", -1); + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-det_type", 0, "define the type of detrend run (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-mode", 0, "define the mode of this detrend run (master, verify)", "master"); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-filelevel", 0, "define filelevel", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-inst", 0, "define camera (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-filter", 0, "define filter ", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-time_begin", 0, "detrend applies to exposures taken during this period (required)", NULL); + // i'm requiring time_begin because otherwise if not specified detselect will pick the wrong detrend for old date ranges (it will pick the most recent one with NULL) + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-time_end", 0, "detrend applies to exposures taken during this period", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-use_begin", 0, "start of detrend run applicable period", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-use_end", 0, "end of detrend run applicable period", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_type", 0, "search for exp_type", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_inst", 0, "search for camera", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_telescope", 0, "search for telescope", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_filter", 0, "search for filter", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_uri", 0, "search for uri", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-select_dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-select_dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_airmass_max", 0, "define max airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_sat_pixel_frac_max", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_sat_pixel_frac_min", 0, "define min fraction of saturated pixels", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_pon_time_min", 0, "define min power-on time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_pon_time_max", 0, "define max power-on time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_posang_max", 0, "define max rotator position angle", NAN); + + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_sun_angle_max", 0, "define max solar angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_sun_alt_min", 0, "define min solar altitude", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_sun_alt_max", 0, "define max solar altitude", NAN); + + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_moon_angle_min", 0, "define min moon angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_moon_angle_max", 0, "define max moon angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_moon_alt_min", 0, "define min moon alt", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_moon_alt_max", 0, "define max moon alt", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_moon_phase_min", 0, "define min moon phase", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_moon_phase_max", 0, "define max moon phase", NAN); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_state", 0, "select by state (default is 'full')", "full"); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-comment", 0, "search by comment field (LIKE comparison)", NULL); + + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "print the exposures that would be included in the detrend run and exit", false); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-reduction", 0, "define reduction class for processing", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", 0, "define detrun label", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-random_subset", 0, "use a random subset of elements", false); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-random_limit", 0, "use this number of random elements", 20); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-ref_det_id", 0, "define reference detrend (verify mode only)", 0); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-ref_iter", 0, "define reference detrend (verify mode only)", -1); + + // -definebydetrun + psMetadata *definebydetrunArgs = psMetadataAlloc(); + psMetadataAddS64(definebydetrunArgs, PS_LIST_TAIL, "-det_id", 0, "det ID to base a new detRun on (required)", 0); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_det_type", 0, "define the type of detrend run", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_mode", 0, "define the mode of this detrend run", "master"); + psMetadataAddS64(definebydetrunArgs, PS_LIST_TAIL, "-ref_det_id", 0, "reference det ID ('verify' run only)", 0); + psMetadataAddS32(definebydetrunArgs, PS_LIST_TAIL, "-ref_iter", 0, "reference iteration ('verify' run only)", -1); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_exp_type", 0, "define exposure type", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_filelevel", 0, "define filelevel", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_filter", 0, "define filter ", NULL); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_airmass_min", 0, "define airmass", NAN); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_airmass_max", 0, "define airmass", NAN); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_exp_time_min", 0, "define exposure time", NAN); + psMetadataAddF32(definebydetrunArgs, PS_LIST_TAIL, "-set_exp_time_max", 0, "define exposure time", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_ccd_temp_min", 0, "define ccd tempature", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_ccd_temp_max", 0, "define ccd tempature", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_posang_min", 0, "define rotator position angle", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_posang_max", 0, "define rotator position angle", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_sun_angle_min", 0, "define solar angle", NAN); + psMetadataAddF64(definebydetrunArgs, PS_LIST_TAIL, "-set_sun_angle_max", 0, "define solar angle", NAN); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_time_begin", 0, "start of period to apply detrend too", NULL); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_time_end", 0, "end of period to apply detrend too", NULL); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_use_begin", 0, "start of detrend run applicable period", NULL); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_use_end", 0, "end of detrend run applicable period", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class for processing", NULL); + psMetadataAddStr(definebydetrunArgs, PS_LIST_TAIL, "-set_label", 0, "define detrun label", NULL); + psMetadataAddBool(definebydetrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_input_begin", 0, "input exp must be in this period", NULL); + psMetadataAddTime(definebydetrunArgs, PS_LIST_TAIL, "-set_input_end", 0, "input exp must be in this period", NULL); + psMetadataAddBool(definebydetrunArgs, PS_LIST_TAIL, "-only_accepted", 0, "limit input exp to those accepted in the source detRun", false); + psMetadataAddS32(definebydetrunArgs, PS_LIST_TAIL, "-iteration", 0, "det ID to base a new detRun on", -1); + + // *** these are in dettool_correction.c *** + // -makecorrection + psMetadata *makecorrectionArgs = psMetadataAlloc(); + psMetadataAddStr(makecorrectionArgs, PS_LIST_TAIL, "-det_type", 0, "set detrend type of corrected detRun", NULL); + psMetadataAddS64(makecorrectionArgs, PS_LIST_TAIL, "-det_id", 0, "detRun det_id to be corrected (required)", 0); + psMetadataAddS32(makecorrectionArgs, PS_LIST_TAIL, "-iteration", 0, "detRun iteration to be corrected (required)", -1);; + psMetadataAddBool(makecorrectionArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadataAddStr(makecorrectionArgs, PS_LIST_TAIL, "-workdir", 0, "set workdir", NULL); + psMetadataAddStr(makecorrectionArgs, PS_LIST_TAIL, "-reduction", 0, "set reduction", NULL); + psMetadataAddStr(makecorrectionArgs, PS_LIST_TAIL, "-label", 0, "set label", NULL); + + // -tocorrectexp + psMetadata *tocorrectexpArgs = psMetadataAlloc(); + psMetadataAddU64(tocorrectexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tocorrectexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -tocorrectimfile + psMetadata *tocorrectimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocorrectimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddStr(tocorrectimfileArgs, PS_LIST_TAIL, "-det_type", 0, "search by type of corrected detRun", NULL); + psMetadataAddU64(tocorrectimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tocorrectimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addcorrectimfile + psMetadata *addcorrectimfileArgs = psMetadataAlloc(); + psMetadataAddS64(addcorrectimfileArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddStr(addcorrectimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID (required)", NULL); + psMetadataAddStr(addcorrectimfileArgs, PS_LIST_TAIL, "-uri", 0, "define resid file URI", NULL); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addcorrectimfileArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addcorrectimfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + + // -runs + psMetadata *runsArgs = psMetadataAlloc(); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-det_type", 0, "search for type of detrend run", NULL); + psMetadataAddS64(runsArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-inst", 0, "search for camera (instrument)", NULL); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(runsArgs, PS_LIST_TAIL, "-airmass", 0, "match airmass", NAN); + psMetadataAddF32(runsArgs, PS_LIST_TAIL, "-exp_time", 0, "match exp time", NAN); + psMetadataAddF32(runsArgs, PS_LIST_TAIL, "-ccd_temp", 0, "match ccd temp", NAN); + psMetadataAddF64(runsArgs, PS_LIST_TAIL, "-posang", 0, "match posang", NAN); + psMetadataAddStr(runsArgs, PS_LIST_TAIL, "-mode", 0, "search for det run by mode", NULL); + psMetadataAddBool(runsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(runsArgs, PS_LIST_TAIL, "-active", 0, "only return active detRuns", false); + psMetadataAddU64(runsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -childlessrun + psMetadata *childlessrunArgs = psMetadataAlloc(); + psMetadataAddStr(childlessrunArgs, PS_LIST_TAIL, "-det_type", 0, "search for type of detrend run", NULL); + psMetadataAddU64(childlessrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(childlessrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -input + psMetadata *inputArgs = psMetadataAlloc(); + psMetadataAddS64(inputArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(inputArgs, PS_LIST_TAIL, "-iteration", 0, "search for iteration number (default 0)", 0); + psMetadataAddS64(inputArgs, PS_LIST_TAIL, "-exp_id", 0, "search for exp ID", 0); + psMetadataAddBool(inputArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -raw + psMetadata *rawArgs = psMetadataAlloc(); + psMetadataAddBool(rawArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -toprocessedimfile + psMetadata *toprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(toprocessedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS64(toprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search for exp ID", 0); + psMetadataAddStr(toprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID", NULL); + psMetadataAddU64(toprocessedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toprocessedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "define exp ID (required)", 0); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddS16(addprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-uri", 0, "define URI", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-recip", 0, "define recipe", NULL); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-fringe_0", 0, "define fringe slope (0th term)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-fringe_1", 0, "define fringe slope (1st term)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-fringe_2", 0, "define fringe slope (2nd term)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addprocessedimfileArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(processedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS64(processedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search for exp ID", 0); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-select_state", 0, "search for state", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-select_mode", 0, "search for mode", NULL); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-included", 0, "restrict results to exposures 'includeded' in the current iteration", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddU64(processedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertprocessedimfile + psMetadata *revertprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertprocessedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS64(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(revertprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertprocessedimfileArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + // -updateprocessedimfile + psMetadata *updateprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(updateprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_processedimfile + psMetadata *pendingcleanup_processedimfileArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_processedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_processedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_processedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(pendingcleanup_processedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(pendingcleanup_processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + + // -donecleanup_processedimfile + psMetadata *donecleanup_processedimfileArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_processedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_processedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_processedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(donecleanup_processedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(donecleanup_processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + + // -toprocessedexp + psMetadata *toprocessedexpArgs = psMetadataAlloc(); + psMetadataAddU64(toprocessedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toprocessedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addproccessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "define detrend ID (required)", 0); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-recip", 0, "define recipe", NULL); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-fringe_0", 0, "define fringe slope (0th term)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-fringe_1", 0, "define fringe slope (1st term)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-fringe_2", 0, "define fringe slope (2nd term)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addprocessedexpArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddS16(addprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -proccessedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + psMetadataAddS64(processedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS64(processedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp ID", 0); + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertprocessedexp + psMetadata *revertprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddS16(revertprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertprocessedexpArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + // -updateprocessedexp + psMetadata *updateprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(updateprocessedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updateprocessedexpArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_processedexp + psMetadata *pendingcleanup_processedexpArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_processedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_processedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_processedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(pendingcleanup_processedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + + // -donecleanup_processedexp + psMetadata *donecleanup_processedexpArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_processedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_processedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_processedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(donecleanup_processedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + + // -updatestate_processed + psMetadata *updatestateprocessedArgs = psMetadataAlloc(); + psMetadataAddS64(updatestateprocessedArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(updatestateprocessedArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updatestateprocessedArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -tostacked + psMetadata *tostackedArgs = psMetadataAlloc(); + psMetadataAddU64(tostackedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tostackedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addstacked + psMetadata *addstackedArgs = psMetadataAlloc(); + psMetadataAddS64(addstackedArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(addstackedArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number (required)", 0); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddS16(addstackedArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-uri", 0, "define URI", NULL); + psMetadataAddStr(addstackedArgs, PS_LIST_TAIL, "-recip", 0, "define recipe", NULL); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addstackedArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + + // -stacked + psMetadata *stackedArgs = psMetadataAlloc(); + psMetadataAddS64(stackedArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(stackedArgs, PS_LIST_TAIL, "-iteration", 0, "search for iteration number", 0); + psMetadataAddStr(stackedArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID", NULL); + psMetadataAddStr(stackedArgs, PS_LIST_TAIL, "-recip", 0, "search for recipe", NULL); + psMetadataAddU64(stackedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(stackedArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(stackedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertstacked + psMetadata *revertstackedArgs= psMetadataAlloc(); + psMetadataAddS64(revertstackedArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(revertstackedArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(revertstackedArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(revertstackedArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertstackedArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + + // -updatestacked + psMetadata *updatestackedArgs = psMetadataAlloc(); + psMetadataAddS64(updatestackedArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(updatestackedArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(updatestackedArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddStr(updatestackedArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_stacked + psMetadata *pendingcleanup_stackedArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_stackedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_stackedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_stackedArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(pendingcleanup_stackedArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(pendingcleanup_stackedArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + + // -donecleanup_stacked + psMetadata *donecleanup_stackedArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_stackedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_stackedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_stackedArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(donecleanup_stackedArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(donecleanup_stackedArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + + // -tonormalizedstat + psMetadata *tonormalizedstatArgs = psMetadataAlloc(); + psMetadataAddU64(tonormalizedstatArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tonormalizedstatArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addnormalizedstat + psMetadata *addnormstatArgs = psMetadataAlloc(); + psMetadataAddS64(addnormstatArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(addnormstatArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddStr(addnormstatArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddF32(addnormstatArgs, PS_LIST_TAIL, "-norm", 0, "define normal value (required)", NAN); + psMetadataAddS16(addnormstatArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -normalizedstat + psMetadata *normalizedstatArgs = psMetadataAlloc(); + psMetadataAddS64(normalizedstatArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(normalizedstatArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(normalizedstatArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddU64(normalizedstatArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(normalizedstatArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(normalizedstatArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertnormalizedstat + psMetadata *revertnormalizedstatArgs= psMetadataAlloc(); + psMetadataAddS64(revertnormalizedstatArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(revertnormalizedstatArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(revertnormalizedstatArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(revertnormalizedstatArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertnormalizedstatArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + // -updatenormalizedstat + psMetadata *updatenormalizedstatArgs = psMetadataAlloc(); + psMetadataAddS64(updatenormalizedstatArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(updatenormalizedstatArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(updatenormalizedstatArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + psMetadataAddStr(updatenormalizedstatArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_normalizedstat + psMetadata *pendingcleanup_normalizedstatArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_normalizedstatArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_normalizedstatArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_normalizedstatArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(pendingcleanup_normalizedstatArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(pendingcleanup_normalizedstatArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + + // -donecleanup_normalizedstat + psMetadata *donecleanup_normalizedstatArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_normalizedstatArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_normalizedstatArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_normalizedstatArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(donecleanup_normalizedstatArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(donecleanup_normalizedstatArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + + // -tonormalize + psMetadata *tonormalizeArgs = psMetadataAlloc(); + psMetadataAddU64(tonormalizeArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tonormalizeArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addnormalizedimfile + psMetadata *addnormalizedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(addnormalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(addnormalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddS16(addnormalizedimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-uri", 0, "define URI", NULL); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addnormalizedimfileArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addnormalizedimfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + + // -normalizedimfile + psMetadata *normalizedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(normalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(normalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search for iteration number", 0); + psMetadataAddStr(normalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID", NULL); + psMetadataAddStr(normalizedimfileArgs, PS_LIST_TAIL, "-recip", 0, "search for recipe", NULL); + psMetadataAddU64(normalizedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(normalizedimfileArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(normalizedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertnormalizedimfile + psMetadata *revertnormalizedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertnormalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(revertnormalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(revertnormalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(revertnormalizedimfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertnormalizedimfileArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + // -updatenormalizedimfile + psMetadata *updatenormalizedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(updatenormalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(updatenormalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(updatenormalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + psMetadataAddStr(updatenormalizedimfileArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_normalizedimfile + psMetadata *pendingcleanup_normalizedimfileArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_normalizedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_normalizedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_normalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(pendingcleanup_normalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(pendingcleanup_normalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + + // -donecleanup_normalizedimfile + psMetadata *donecleanup_normalizedimfileArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_normalizedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_normalizedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_normalizedimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(donecleanup_normalizedimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(donecleanup_normalizedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + + // -tonormalizedexp + psMetadata *tonormalizedexpArgs = psMetadataAlloc(); + psMetadataAddU64(tonormalizedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tonormalizedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addnormalizedexp + psMetadata *addnormalizedexpArgs = psMetadataAlloc(); + psMetadataAddS64(addnormalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(addnormalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddS16(addnormalizedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addnormalizedexpArgs, PS_LIST_TAIL, "-recip", 0, "search for recipe", NULL); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addnormalizedexpArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addnormalizedexpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + + // -normalizedexp + psMetadata *normalizedexpArgs = psMetadataAlloc(); + psMetadataAddS64(normalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(normalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(normalizedexpArgs, PS_LIST_TAIL, "-recip", 0, "search for recipe", NULL); + psMetadataAddU64(normalizedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(normalizedexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(normalizedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertnormalizedexp + psMetadata *revertnormalizedexpArgs = psMetadataAlloc(); + psMetadataAddS64(revertnormalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(revertnormalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS16(revertnormalizedexpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertnormalizedexpArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + + // -updatenormalizedexp + psMetadata *updatenormalizedexpArgs = psMetadataAlloc(); + psMetadataAddS64(updatenormalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(updatenormalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddStr(updatenormalizedexpArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_normalizedexp + psMetadata *pendingcleanup_normalizedexpArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_normalizedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_normalizedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_normalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(pendingcleanup_normalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + + // -donecleanup_normalizedexp + psMetadata *donecleanup_normalizedexpArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_normalizedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_normalizedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_normalizedexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(donecleanup_normalizedexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + + // -toresidimfile + psMetadata *toresidimfileArgs = psMetadataAlloc(); + psMetadataAddU64(toresidimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toresidimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addresidimfile + // XXX does a default value of 0 mean the entry is not instatiated? what about a supplied value of 0? + psMetadata *addresidimfileArgs = psMetadataAlloc(); + psMetadataAddS64(addresidimfileArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(addresidimfileArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddS64(addresidimfileArgs, PS_LIST_TAIL, "-ref_det_id", 0, "define reference detrend ID (required)", 0); + psMetadataAddS32(addresidimfileArgs, PS_LIST_TAIL, "-ref_iter", 0, "define reference iteration number (required)", -1); + psMetadataAddS64(addresidimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "define detrend ID (required)", 0); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddS16(addresidimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-uri", 0, "define resid file URI", NULL); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-recip", 0, "define recipe", NULL); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg_skewness", 0, "define exposure background skewness", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bg_kurtosis", 0, "define exposure background kurtosis", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-bin_stdev", 0, "define exposure background binned stdev", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-fringe_0", 0, "define fringe slope (0th term)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-fringe_1", 0, "define fringe slope (1st term)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-fringe_2", 0, "define fringe slope (2nd term)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-fringe_resid_0", 0, "define fringe residual (0th term)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-fringe_resid_1", 0, "define fringe residual (1st term)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-fringe_resid_2", 0, "define fringe residual (2nd term)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addresidimfileArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addresidimfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + + // -residimfile + psMetadata *residimfileArgs = psMetadataAlloc(); + psMetadataAddS64(residimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(residimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search for iteration number", 0); + psMetadataAddS64(residimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by detrend ID", 0); + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID", NULL); + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-recip", 0, "search for recipe", NULL); + psMetadataAddU64(residimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(residimfileArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(residimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(residimfileArgs, PS_LIST_TAIL, "-included", 0, "use only files included in ths detrun", false); + // XXX is this option used? + psMetadataAddStr(residimfileArgs, PS_LIST_TAIL, "-select_state", 0, "search for state", NULL); + + // -revertresidimfile + psMetadata *revertresidimfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertresidimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(revertresidimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(revertresidimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by detrend ID", 0); + psMetadataAddStr(revertresidimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID", NULL); + psMetadataAddS16(revertresidimfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertresidimfileArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + + // -updateresidimfile + psMetadata *updateresidimfileArgs = psMetadataAlloc(); + psMetadataAddS64(updateresidimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(updateresidimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(updateresidimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updateresidimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + psMetadataAddStr(updateresidimfileArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_residimfile + psMetadata *pendingcleanup_residimfileArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_residimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_residimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_residimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(pendingcleanup_residimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(pendingcleanup_residimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(pendingcleanup_residimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + + // -donecleanup_residimfile + psMetadata *donecleanup_residimfileArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_residimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_residimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_residimfileArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS32(donecleanup_residimfileArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(donecleanup_residimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(donecleanup_residimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by exp_name", NULL); + + // -toresidexp + psMetadata *toresidexpArgs = psMetadataAlloc(); + psMetadataAddU64(toresidexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toresidexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addresidexp + psMetadata *addresidexpArgs = psMetadataAlloc(); + psMetadataAddS64(addresidexpArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(addresidexpArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddS64(addresidexpArgs, PS_LIST_TAIL, "-exp_id", 0, "define detrend ID (required)", 0); + psMetadataAddS16(addresidexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-recip", 0, "define recipe", NULL); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg_skewness", 0, "define exposure background skewness", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bg_kurtosis", 0, "define exposure background kurtosis", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-bin_stdev", 0, "define exposure background binned stdev", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-fringe_0", 0, "define fringe slope (0th term)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-fringe_1", 0, "define fringe slope (1st term)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-fringe_2", 0, "define fringe slope (2nd term)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-fringe_resid_0", 0, "define fringe residual (0th term)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-fringe_resid_1", 0, "define fringe residual (1st term)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-fringe_resid_2", 0, "define fringe residual (2nd term)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(addresidexpArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(addresidexpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddBool(addresidexpArgs, PS_LIST_TAIL, "-reject", 0, "exposure is not to be stacked in the next iteration", false); + + // -residexp + psMetadata *residexpArgs = psMetadataAlloc(); + psMetadataAddS64(residexpArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(residexpArgs, PS_LIST_TAIL, "-iteration", 0, "search for iteration number", 0); + psMetadataAddS64(residexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search for exp ID", 0); + psMetadataAddStr(residexpArgs, PS_LIST_TAIL, "-recip", 0, "search for recipe", NULL); + psMetadataAddBool(residexpArgs, PS_LIST_TAIL, "-reject", 0, "search for acceptable residuals", false); + psMetadataAddU64(residexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(residexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(residexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertresidexp + psMetadata *revertresidexpArgs = psMetadataAlloc(); + psMetadataAddS64(revertresidexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(revertresidexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(revertresidexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by detrend ID", 0); + psMetadataAddS16(revertresidexpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertresidexpArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + // -updateresidexp + psMetadata *updateresidexpArgs = psMetadataAlloc(); + psMetadataAddS64(updateresidexpArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID", 0); + psMetadataAddS32(updateresidexpArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddS64(updateresidexpArgs, PS_LIST_TAIL, "-exp_id", 0, "define exp ID", 0); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-recip", 0, "define recipe", NULL); + psMetadataAddF64(updateresidexpArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(updateresidexpArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(updateresidexpArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddBool(updateresidexpArgs, PS_LIST_TAIL, "-reject", 0, "exposure is not to be stacked in the next iteration", false); + psMetadataAddStr(updateresidexpArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -pendingcleanup_residexp + psMetadata *pendingcleanup_residexpArgs = psMetadataAlloc(); + psMetadataAddBool(pendingcleanup_residexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanup_residexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(pendingcleanup_residexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID (required)", 0); + psMetadataAddS32(pendingcleanup_residexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(pendingcleanup_residexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by detrend ID", 0); + + // -donecleanup_residexp + psMetadata *donecleanup_residexpArgs = psMetadataAlloc(); + psMetadataAddBool(donecleanup_residexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanup_residexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddS64(donecleanup_residexpArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID (required)", 0); + psMetadataAddS32(donecleanup_residexpArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS64(donecleanup_residexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by detrend ID", 0); + + // -updatestate_resid + psMetadata *updatestateresidArgs = psMetadataAlloc(); + psMetadataAddS64(updatestateresidArgs, PS_LIST_TAIL, "-det_id", 0, "search by det ID", 0); + psMetadataAddS64(updatestateresidArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updatestateresidArgs, PS_LIST_TAIL, "-data_state", 0, "set state", NULL); + + // -todetrunsummary + psMetadata *todetrunsummaryArgs = psMetadataAlloc(); + psMetadataAddBool(todetrunsummaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(todetrunsummaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -adddetrunsummary + psMetadata *adddetrunsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(adddetrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddS32(adddetrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, "define iteration number", 0); + psMetadataAddF64(adddetrunsummaryArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(adddetrunsummaryArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(adddetrunsummaryArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddS16(adddetrunsummaryArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddBool(adddetrunsummaryArgs, PS_LIST_TAIL, "-accept", 0, "declare that this detrun iteration is accepted as a master", false); + psMetadataAddBool(adddetrunsummaryArgs, PS_LIST_TAIL, "-again", 0, "start a new iteration of this detrend run", false); + + // -detrunsummary + psMetadata *detrunsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(detrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend ID", 0); + psMetadataAddS32(detrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, "search for iteration number", 0); + psMetadataAddU64(detrunsummaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(detrunsummaryArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(detrunsummaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(detrunsummaryArgs, PS_LIST_TAIL, "-reject", 0, "search for acceptable residuals", false); + + // -revertdetrunsummary + psMetadata *revertdetrunsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(revertdetrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID", 0); + psMetadataAddS32(revertdetrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddS16(revertdetrunsummaryArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertdetrunsummaryArgs, PS_LIST_TAIL, "-all-run", 0, "revert all in state 'run'", false); + + // -updatedetrunsummary + psMetadata *updatedetrunsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(updatedetrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend master for detrend ID (required)", 0); + psMetadataAddBool(updatedetrunsummaryArgs, PS_LIST_TAIL, "-accept", 0, "declare that this detrun iteration is accepted as a master", false); + psMetadataAddBool(updatedetrunsummaryArgs, PS_LIST_TAIL, "-reject", 0, "reject this detrun iteration as a master", false); + psMetadataAddStr(updatedetrunsummaryArgs, PS_LIST_TAIL, "-data_state", 0, "set the data state", NULL); + psMetadataAddS32(updatedetrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + + // -pendingcleanup_detrunsummary + psMetadata *pendingcleanup_detrunsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(pendingcleanup_detrunsummaryArgs, PS_LIST_TAIL, "-det_id", 0, "search by detrend ID (required)", 0); + psMetadataAddS32(pendingcleanup_detrunsummaryArgs, PS_LIST_TAIL, "-iteration", 0, "search by iteration number", 0); + psMetadataAddU64(pendingcleanup_detrunsummaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingcleanup_detrunsummaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updatedetrun + psMetadata *updatedetrunArgs = psMetadataAlloc(); + psMetadataAddS64(updatedetrunArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend master for detrend ID (required)", 0); + psMetadataAddBool(updatedetrunArgs, PS_LIST_TAIL, "-again", 0, "start a new iteration of this detrend run", false); + psMetadataAddStr(updatedetrunArgs, PS_LIST_TAIL, "-state", 0, "set the state of this detrend run", false); + psMetadataAddStr(updatedetrunArgs, PS_LIST_TAIL, "-set_det_type", 0, "set the det_type", NULL); + psMetadataAddTime(updatedetrunArgs, PS_LIST_TAIL, "-set_time_begin", 0, "start of period to apply detrend too", NULL); + psMetadataAddTime(updatedetrunArgs, PS_LIST_TAIL, "-set_time_end", 0, "end of period to apply detrend too", NULL); + psMetadataAddTime(updatedetrunArgs, PS_LIST_TAIL, "-set_use_begin", 0, "start of detrend run applicable period", NULL); + psMetadataAddTime(updatedetrunArgs, PS_LIST_TAIL, "-set_use_end", 0, "end of detrend run applicable period", NULL); + + // -rerun + psMetadata *rerunArgs = psMetadataAlloc(); + psMetadataAddS64(rerunArgs, PS_LIST_TAIL, "-det_id", 0, "search for detrend master for detrend ID (required)", 0); + psMetadataAddS64(rerunArgs, PS_LIST_TAIL, "-exp_id", PS_META_DUPLICATE_OK, "include this exposure (multiple OK, required)", 0); + + // -register_detrend + psMetadata *register_detrendArgs = psMetadataAlloc(); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-det_type", 0, "define the type of detrend run (required)", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-filelevel", 0, "define filelevel (required)", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-inst", 0, "define camera", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-exp_type", 0, "define exposure type", NULL); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-filter", 0, "define filter ", NULL); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF32(register_detrendArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF64(register_detrendArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + psMetadataAddTime(register_detrendArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddTime(register_detrendArgs, PS_LIST_TAIL, "-time_begin", 0, "detrend applies to exposures taken during this period", NULL); + psMetadataAddTime(register_detrendArgs, PS_LIST_TAIL, "-time_end", 0, "detrend applies to exposures taken during this period", NULL); + psMetadataAddTime(register_detrendArgs, PS_LIST_TAIL, "-use_begin", 0, "start of detrend run applicable period", NULL); + psMetadataAddTime(register_detrendArgs, PS_LIST_TAIL, "-use_end", 0, "end of detrend run applicable period", NULL); + psMetadataAddS64(register_detrendArgs, PS_LIST_TAIL, "-ref_det_id", 0, "define reference det_id", 0); + psMetadataAddS32(register_detrendArgs, PS_LIST_TAIL, "-ref_iter", 0, "define reference iteration", -1); + psMetadataAddStr(register_detrendArgs, PS_LIST_TAIL, "-label", 0, "define detrun label", NULL); + psMetadataAddBool(register_detrendArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -register_detrend_imfile + psMetadata *register_detrend_imfileArgs = psMetadataAlloc(); + psMetadataAddS64(register_detrend_imfileArgs, PS_LIST_TAIL, "-det_id", 0, "define detrend ID (required)", 0); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-class_id", 0, "search for class ID (required)", NULL); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-uri", 0, "define resid file URI (required)", NULL); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-bg_mean_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-user_1", 0, "define user statistic (1)", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-user_2", 0, "define user statistic (2)", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-user_3", 0, "define user statistic (3)", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-user_4", 0, "define user statistic (4)", NAN); + psMetadataAddF64(register_detrend_imfileArgs, PS_LIST_TAIL, "-user_5", 0, "define user statistic (5)", NAN); + psMetadataAddStr(register_detrend_imfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-det_id", 0, "export this detrend ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-pending", "list active detruns", DETTOOL_MODE_PENDING, pendingArgs); + PXOPT_ADD_MODE("-definebytag", "", DETTOOL_MODE_DEFINEBYTAG, definebytagArgs); + PXOPT_ADD_MODE("-definebyquery", "", DETTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definebydetrun", "", DETTOOL_MODE_DEFINEBYDETRUN, definebydetrunArgs); + PXOPT_ADD_MODE("-makecorrection", "", DETTOOL_MODE_MAKECORRECTION, makecorrectionArgs); + PXOPT_ADD_MODE("-tocorrectexp", "", DETTOOL_MODE_TOCORRECTEXP, tocorrectexpArgs); + PXOPT_ADD_MODE("-tocorrectimfile", "", DETTOOL_MODE_TOCORRECTIMFILE, tocorrectimfileArgs); + PXOPT_ADD_MODE("-addcorrectimfile", "", DETTOOL_MODE_ADDCORRECTIMFILE, addcorrectimfileArgs); + PXOPT_ADD_MODE("-raw", "", DETTOOL_MODE_RAW, rawArgs); + PXOPT_ADD_MODE("-runs", "", DETTOOL_MODE_RUNS, runsArgs); + PXOPT_ADD_MODE("-childlessrun", "", DETTOOL_MODE_CHILDLESSRUN, childlessrunArgs); + PXOPT_ADD_MODE("-input", "", DETTOOL_MODE_INPUT, inputArgs); + + PXOPT_ADD_MODE("-toprocessedimfile", "", DETTOOL_MODE_TOPROCESSEDIMFILE, toprocessedimfileArgs); + PXOPT_ADD_MODE("-addprocessedimfile", "", DETTOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileArgs); + PXOPT_ADD_MODE("-processedimfile", "", DETTOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXOPT_ADD_MODE("-revertprocessedimfile", "", DETTOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileArgs); + PXOPT_ADD_MODE("-updateprocessedimfile", "", DETTOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileArgs); + PXOPT_ADD_MODE("-pendingcleanup_processedimfile", "", DETTOOL_MODE_PENDINGCLEANUP_PROCESSEDIMFILE, pendingcleanup_processedimfileArgs); + PXOPT_ADD_MODE("-donecleanup_processedimfile", "", DETTOOL_MODE_DONECLEANUP_PROCESSEDIMFILE, donecleanup_processedimfileArgs); + + PXOPT_ADD_MODE("-toprocessedexp", "", DETTOOL_MODE_TOPROCESSEDEXP, toprocessedexpArgs); + PXOPT_ADD_MODE("-addprocessedexp", "", DETTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXOPT_ADD_MODE("-processedexp", "", DETTOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXOPT_ADD_MODE("-revertprocessedexp", "", DETTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpArgs); + PXOPT_ADD_MODE("-updateprocessedexp", "", DETTOOL_MODE_UPDATEPROCESSEDEXP, updateprocessedexpArgs); + PXOPT_ADD_MODE("-pendingcleanup_processedexp", "", DETTOOL_MODE_PENDINGCLEANUP_PROCESSEDEXP, pendingcleanup_processedexpArgs); + PXOPT_ADD_MODE("-donecleanup_processedexp", "", DETTOOL_MODE_DONECLEANUP_PROCESSEDEXP, donecleanup_processedexpArgs); + PXOPT_ADD_MODE("-updatestate_processed", "", DETTOOL_MODE_UPDATESTATE_PROCESSED, updatestateprocessedArgs); + + PXOPT_ADD_MODE("-tostacked", "", DETTOOL_MODE_TOSTACKED, tostackedArgs); + PXOPT_ADD_MODE("-addstacked", "", DETTOOL_MODE_ADDSTACKED, addstackedArgs); + PXOPT_ADD_MODE("-stacked", "", DETTOOL_MODE_STACKED, stackedArgs); + PXOPT_ADD_MODE("-revertstacked", "", DETTOOL_MODE_REVERTSTACKED, revertstackedArgs); + PXOPT_ADD_MODE("-updatestacked", "", DETTOOL_MODE_UPDATESTACKED, updatestackedArgs); + PXOPT_ADD_MODE("-pendingcleanup_stacked", "", DETTOOL_MODE_PENDINGCLEANUP_STACKED, pendingcleanup_stackedArgs); + PXOPT_ADD_MODE("-donecleanup_stacked", "", DETTOOL_MODE_DONECLEANUP_STACKED, donecleanup_stackedArgs); + + PXOPT_ADD_MODE("-tonormalizedstat", "", DETTOOL_MODE_TONORMALIZEDSTAT, tonormalizedstatArgs); + PXOPT_ADD_MODE("-addnormalizedstat", "", DETTOOL_MODE_ADDNORMALIZEDSTAT, addnormstatArgs); + PXOPT_ADD_MODE("-normalizedstat", "", DETTOOL_MODE_NORMALIZEDSTAT, normalizedstatArgs); + PXOPT_ADD_MODE("-revertnormalizedstat", "", DETTOOL_MODE_REVERTNORMALIZEDSTAT, revertnormalizedstatArgs); + PXOPT_ADD_MODE("-updatenormalizedstat", "", DETTOOL_MODE_UPDATENORMALIZEDSTAT, updatenormalizedstatArgs); + PXOPT_ADD_MODE("-pendingcleanup_normalizedstat", "", DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDSTAT, pendingcleanup_normalizedstatArgs); + PXOPT_ADD_MODE("-donecleanup_normalizedstat", "", DETTOOL_MODE_DONECLEANUP_NORMALIZEDSTAT, donecleanup_normalizedstatArgs); + + PXOPT_ADD_MODE("-tonormalize", "", DETTOOL_MODE_TONORMALIZE, tonormalizeArgs); + PXOPT_ADD_MODE("-addnormalizedimfile", "", DETTOOL_MODE_ADDNORMALIZEDIMFILE,addnormalizedimfileArgs); + PXOPT_ADD_MODE("-normalizedimfile","", DETTOOL_MODE_NORMALIZEDIMFILE, normalizedimfileArgs); + PXOPT_ADD_MODE("-revertnormalizedimfile","", DETTOOL_MODE_REVERTNORMALIZEDIMFILE, revertnormalizedimfileArgs); + PXOPT_ADD_MODE("-updatenormalizedimfile","", DETTOOL_MODE_UPDATENORMALIZEDIMFILE, updatenormalizedimfileArgs); + PXOPT_ADD_MODE("-pendingcleanup_normalizedimfile", "", DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDIMFILE, pendingcleanup_normalizedimfileArgs); + PXOPT_ADD_MODE("-donecleanup_normalizedimfile", "", DETTOOL_MODE_DONECLEANUP_NORMALIZEDIMFILE, donecleanup_normalizedimfileArgs); + + PXOPT_ADD_MODE("-tonormalizedexp", "", DETTOOL_MODE_TONORMALIZEDEXP, tonormalizedexpArgs); + PXOPT_ADD_MODE("-addnormalizedexp", "", DETTOOL_MODE_ADDNORMALIZEDEXP, addnormalizedexpArgs); + PXOPT_ADD_MODE("-normalizedexp", "", DETTOOL_MODE_NORMALIZEDEXP, normalizedexpArgs); + PXOPT_ADD_MODE("-revertnormalizedexp","", DETTOOL_MODE_REVERTNORMALIZEDEXP, revertnormalizedexpArgs); + PXOPT_ADD_MODE("-updatenormalizedexp","", DETTOOL_MODE_UPDATENORMALIZEDEXP, updatenormalizedexpArgs); + PXOPT_ADD_MODE("-pendingcleanup_normalizedexp", "", DETTOOL_MODE_PENDINGCLEANUP_NORMALIZEDEXP, pendingcleanup_normalizedexpArgs); + PXOPT_ADD_MODE("-donecleanup_normalizedexp", "", DETTOOL_MODE_DONECLEANUP_NORMALIZEDEXP, donecleanup_normalizedexpArgs); + + PXOPT_ADD_MODE("-toresidimfile", "", DETTOOL_MODE_TORESIDIMFILE, toresidimfileArgs); + PXOPT_ADD_MODE("-addresidimfile", "", DETTOOL_MODE_ADDRESIDIMFILE, addresidimfileArgs); + PXOPT_ADD_MODE("-residimfile", "", DETTOOL_MODE_RESIDIMFILE, residimfileArgs); + PXOPT_ADD_MODE("-revertresidimfile", "", DETTOOL_MODE_REVERTRESIDIMFILE, revertresidimfileArgs); + PXOPT_ADD_MODE("-updateresidimfile", "", DETTOOL_MODE_UPDATERESIDIMFILE, updateresidimfileArgs); + PXOPT_ADD_MODE("-pendingcleanup_residimfile", "", DETTOOL_MODE_PENDINGCLEANUP_RESIDIMFILE, pendingcleanup_residimfileArgs); + PXOPT_ADD_MODE("-donecleanup_residimfile", "", DETTOOL_MODE_DONECLEANUP_RESIDIMFILE, donecleanup_residimfileArgs); + + PXOPT_ADD_MODE("-toresidexp", "", DETTOOL_MODE_TORESIDEXP, toresidexpArgs); + PXOPT_ADD_MODE("-addresidexp", "", DETTOOL_MODE_ADDRESIDEXP, addresidexpArgs); + PXOPT_ADD_MODE("-residexp", "", DETTOOL_MODE_RESIDEXP, residexpArgs); + PXOPT_ADD_MODE("-revertresidexp", "", DETTOOL_MODE_REVERTRESIDEXP, revertresidexpArgs); + PXOPT_ADD_MODE("-updateresidexp", "", DETTOOL_MODE_UPDATERESIDEXP, updateresidexpArgs); + PXOPT_ADD_MODE("-pendingcleanup_residexp", "", DETTOOL_MODE_PENDINGCLEANUP_RESIDEXP, pendingcleanup_residexpArgs); + PXOPT_ADD_MODE("-donecleanup_residexp", "", DETTOOL_MODE_DONECLEANUP_RESIDEXP, donecleanup_residexpArgs); + PXOPT_ADD_MODE("-updatestate_resid", "", DETTOOL_MODE_UPDATESTATE_RESID, updatestateresidArgs); + + PXOPT_ADD_MODE("-todetrunsummary", "", DETTOOL_MODE_TODETRUNSUMMARY, todetrunsummaryArgs); + PXOPT_ADD_MODE("-adddetrunsummary", "", DETTOOL_MODE_ADDDETRUNSUMMARY,adddetrunsummaryArgs); + PXOPT_ADD_MODE("-detrunsummary", "", DETTOOL_MODE_DETRUNSUMMARY,detrunsummaryArgs); + PXOPT_ADD_MODE("-revertdetrunsummary", "", DETTOOL_MODE_REVERTDETRUNSUMMARY, revertdetrunsummaryArgs); + PXOPT_ADD_MODE("-updatedetrunsummary", "", DETTOOL_MODE_UPDATEDETRUNSUMMARY, updatedetrunsummaryArgs); + + PXOPT_ADD_MODE("-pendingcleanup_detrunsummary", "", DETTOOL_MODE_PENDINGCLEANUP_DETRUNSUMMARY, pendingcleanup_detrunsummaryArgs); + + PXOPT_ADD_MODE("-updatedetrun", "", DETTOOL_MODE_UPDATEDETRUN, updatedetrunArgs); + PXOPT_ADD_MODE("-rerun", "", DETTOOL_MODE_RERUN, rerunArgs); + PXOPT_ADD_MODE("-register_detrend", "", DETTOOL_MODE_REGISTER_DETREND, register_detrendArgs); + PXOPT_ADD_MODE("-register_detrend_imfile", "", DETTOOL_MODE_REGISTER_DETREND_IMFILE, register_detrend_imfileArgs); + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", DETTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", DETTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_correction.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_correction.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_correction.c (revision 41104) @@ -0,0 +1,266 @@ +/* + * dettool_correction.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +bool makecorrectionMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(ref_det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_S32(ref_iter, config->args, "-iteration", true, false); // required + PXOPT_LOOKUP_STR(det_type, config->args, "-det_type", true, false); // required + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); // optional + + // optional modifications + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + + // build the needed where + psMetadata *where = psMetadataAlloc(); + psMetadataAddS64(where, PS_LIST_TAIL, "det_id", 0, "==", ref_det_id); + psMetadataAddS32(where, PS_LIST_TAIL, "iteration", 0, "==", ref_iter); + + // select the detRun that matches + psArray *runs = detRunSelectRowObjects(config->dbh, where, 1); + psFree (where); + + if (!psArrayLength(runs)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + detRunRow *detRun = runs->data[0]; + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + char *use_workdir = (workdir) ? workdir : detRun->workdir; + char *use_reduction = (reduction) ? reduction : detRun->reduction; + char *use_label = (label) ? label : detRun->label; + + detRunInsert(config->dbh, + 0, // det_id + 0, // iteration + det_type, + "correction", // mode + "run", // state + detRun->filelevel, + use_workdir, + detRun->camera, + detRun->telescope, + detRun->exp_type, + use_reduction, + detRun->filter, + detRun->airmass_min, + detRun->airmass_max, + detRun->exp_time_min, + detRun->exp_time_max, + detRun->ccd_temp_min, + detRun->ccd_temp_max, + detRun->posang_min, + detRun->posang_max, + detRun->registered, + detRun->time_begin, + detRun->time_end, + detRun->use_begin, + detRun->use_end, + detRun->solang_min, + detRun->solang_max, + use_label, + detRun->det_id, // ref_det_id + detRun->iteration // ref_iter + ); + psFree(runs); + + // print the new detRun + psS64 new_det_id = psDBLastInsertID(config->dbh); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psMetadata *where_new = psMetadataAlloc(); + psMetadataAddS64(where_new, PS_LIST_TAIL, "det_id", 0, "==", new_det_id); + psArray *detRuns = psDBSelectRows(config->dbh, "detRun", where_new, 0); + psFree(where_new); + + if (!detRuns) { + psError(PS_ERR_UNKNOWN, false, "can't find the detRun we just created"); + return false; + } + // sanity check results + if (psArrayLength(detRuns) != 1) { + psAbort("found more then one detRun matching det_id %" PRId64 "(this should not happen)", new_det_id); + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, detRuns, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(detRuns); + return false; + } + psFree(detRuns); + + return true; +} + +bool tocorrectimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_STR(config->args, where, "-det_type", "det_type", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_tocorrectimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + // NOTE the SQL uses an intermediate table 'det1' for this query + psString whereClause = psDBGenerateWhereConditionSQL(where, "det1"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree (where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingCorrectImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool tocorrectexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_tocorrectexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// NOTE : the command-line args are parsed by register_detrend_imfileMode +bool addcorrectimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // insert the row into detRegisterImfile + if (!register_detrend_imfileMode(config)) { + return false; + } + + // automatically stop completed 'correct' detRuns + psString query = pxDataGet("dettool_stop_completed_correct_runs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_detrunsummary.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_detrunsummary.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_detrunsummary.c (revision 41104) @@ -0,0 +1,526 @@ +/* + * dettool_detrunsummary.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +static detRunSummaryRow *mdToDetRunSummary(pxConfig *config, psMetadata *row); + +/* The SQL returns a list of detrend runs (with detrend id, iteration and detrend type) which + * have completed all residexps. + */ + +bool todetrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_todetrunsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detRejectExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// this function is used to generate the detrunSummary entries used below +// XXX why is this a separate function? roll it back into adddetrunsummary? +static detRunSummaryRow *mdToDetRunSummary(pxConfig *config, psMetadata *row) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + bool status = false; + // from row + psS64 det_id = psMetadataLookupS64(&status, row, "det_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for det_id"); + return false; + } + psS32 iteration = psMetadataLookupS32(&status, row, "iteration"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for iteration"); + return false; + } + + // optional + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_BOOL(accept, config->args, "-accept", false); + + return detRunSummaryRowAlloc( + det_id, + iteration, + "full", + bg, + bg_stdev, + bg_mean_stdev, + accept, + fault + ); +} + +// XXX data_state should be set to 'full' here +bool adddetrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // build a query to search by det_id, iteration, exp_id + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_BOOL(again, config->args, "-again", false); + + // The values supplied as arguments on the command (eg, -bg) are parsed + // by mdToDetRunSummary below. + // XXX why is there ever more than one? + + psString query = pxDataGet("dettool_find_completed_runs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // XXX why is there ever more than one result here? + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + // convert metadata into a detResidExp object + detRunSummaryRow *runSummary = mdToDetRunSummary(config, row); + if (!runSummary) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata to detResidExp"); + psFree(output); + return false; + } + // insert detResidExp object into the database + if (!detRunSummaryInsertObject(config->dbh, runSummary)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(runSummary); + psFree(output); + return false; + } + psFree(runSummary); + } + + psFree(output); + + // XXX this logic does not deal with the case of -fault being set + // XXX it should be an error for -again and -fault to both be set + if (again) { + if (!startNewIteration(config, det_id)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to start new iteration"); + return false; + } + } else { + // set detRun.state to stop + if (!setDetRunState(config, det_id, "stop")) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to set detRun.state"); + return false; + } + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; +} + +bool detrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + psString query = pxDataGet("dettool_detrunsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "NULL"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detRunSummary.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detRunSummary.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawDetrendImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool revertdetrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertdetrunsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detRunSummary"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool pendingcleanup_detrunsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_detrunsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detRunSummary"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + // fprintf(stderr,">>>%s<<<\n",query); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return(true); + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pendingCleanupDetRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return(false); + } + psFree(output); + + return(true); +} + + +// preliminary code now. +bool updatedetrunsummaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_BOOL(accept, config->args, "-accept", false); + PXOPT_LOOKUP_BOOL(reject, config->args, "-reject", false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", ((accept == 0)&&(reject == 0)), false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", ((accept == 0)&&(reject == 0)), false); + + + if (accept && reject) { + psError(PS_ERR_UNKNOWN, true, "-accept and -reject are exclusive"); + return false; + } + + if (!(accept || reject || (data_state != NULL))) { + psError(PS_ERR_UNKNOWN, true, "either -accept or -reject is required if -data_state is not supplied"); + return false; + } + + if (accept || reject) { + char *query = "UPDATE detRunSummary SET accept = %d WHERE det_id = %"PRId64; + if (!p_psDBRunQueryF(config->dbh, query, accept, det_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + else { + PS_ASSERT_PTR_NON_NULL(data_state,false); + +/* if (!isValidDataState(data_state)) return false; */ + + + + char *query_detRunSummary = "UPDATE detRunSummary SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detRunSummary, data_state, det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return false; + } + + /* This check allows the one update to flag everything for cleanup. The check for full is only temporary while I test for bugs. */ + if ((!strncmp(data_state,"goto_",5) + // || (!strcmp(data_state,"full")) + // || (!strcmp(data_state,"cleaned")) + )) { + char *query_detProcessedImfile = "UPDATE detProcessedImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query_detProcessedImfile,data_state,det_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detProcessedImfile det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detProcessedExp = "UPDATE detProcessedExp SET data_state = '%s'" + " WHERE det_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query_detProcessedExp,data_state,det_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detProcessedExp det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detNormalizedImfile = "UPDATE detNormalizedImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detNormalizedImfile,data_state,det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detNormalizedImfile det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detNormalizedStatImfile = "UPDATE detNormalizedStatImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detNormalizedStatImfile,data_state,det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detNormalizedStatImfile det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detNormalizedExp = "UPDATE detNormalizedExp SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detNormalizedExp,data_state,det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detNormalizedExp det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detResidImfile = "UPDATE detResidImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detResidImfile,data_state,det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detResidImfile det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detResidExp = "UPDATE detResidExp SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detResidExp,data_state,det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detResidExp det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + + char *query_detStackedImfile = "UPDATE detStackedImfile SET data_state = '%s'" + " WHERE det_id = %" PRId64 + " AND iteration = %" PRId32; + if (!p_psDBRunQueryF(config->dbh, query_detStackedImfile,data_state,det_id,iteration)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for detStackedImfile det_id %" PRId64 ", iteration %" PRId32, det_id,iteration); + return(false); + } + } + /* End if */ + + } + + return true; +} + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedexp.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedexp.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedexp.c (revision 41104) @@ -0,0 +1,390 @@ +/* + * dettool_normalizedexp.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +bool tonormalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_tonormalizedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingNormExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +bool addnormalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(recipe, config->args, "-recip", (fault == 0), false); // Required if fault == 0 + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + psString query = pxDataGet("dettool_tonormalizedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // XXX in some other places, we put the arguments in the .sql and supply them to RunQuery + psStringAppend(&query, " WHERE det_id = %" PRId64 " AND iteration = %d", det_id, iteration); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + psFree(output); + + // insert the new row into the detProcessedImfile table + if (!detNormalizedExpInsert( + config->dbh, + det_id, + iteration, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + fault + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +bool normalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-recip", "recipe", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + psString query = pxDataGet("dettool_normalizedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detNormalizedExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detNormalizedExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + + +bool revertnormalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertnormalizedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detNormalizedExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updatenormalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + if (!setNormExpDataState(config, det_id, iteration, data_state)) { + return false; + } + return true; +} + +bool pendingcleanup_normalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detNormalizedExp.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detNormalizedExp.iteration", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_normalizedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupNormExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool donecleanup_normalizedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detNormalizedExp.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detNormalizedExp.iteration", "=="); + + psString query = pxDataGet("dettool_donecleanup_normalizedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_normalizedexp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedimfile.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedimfile.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedimfile.c (revision 41104) @@ -0,0 +1,373 @@ +/* + * dettool_normalizedimfile.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +// XXX this one is inconsistent - rename 'tonormalizedimfile' and change ipp_serial_detrend.pl and detrend.norm.pro +bool tonormalizeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_tonormalize.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingNormImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool addnormalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", false, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // Required if fault == 0 + PXOPT_LOOKUP_STR(uri, config->args, "-uri", (fault == 0), false); + + // optional + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + if (!detNormalizedImfileInsert( + config->dbh, + det_id, + iteration, + class_id, + uri, + bg, + bg_stdev, + bg_mean_stdev, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + fault + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +bool normalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-recip", "recipe", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + psString query = pxDataGet("dettool_normalizedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detNormalizedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detNormalizedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detNormalizedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool revertnormalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertnormalizedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detNormalizedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updatenormalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + if (!setNormImfileDataState(config, det_id, iteration, class_id, data_state)) { + return false; + } + return true; +} + +bool pendingcleanup_normalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detNormalizedImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detNormalizedImfile.iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detNormalizedImfile.det_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_normalizedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupNormImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool donecleanup_normalizedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detNormalizedImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detNormalizedImfile.iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detNormalizedImfile.det_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_normalizedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_normalizedimfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedstat.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedstat.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_normalizedstat.c (revision 41104) @@ -0,0 +1,353 @@ +/* + * dettool_normalizedstat.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +bool tonormalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_tonormalizedstat.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingNormStatImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool addnormalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // required + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + // optional + PXOPT_LOOKUP_F32(norm, config->args, "-norm", false, false); + + // default + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", false, false); + PXOPT_LOOKUP_S16(code, config->args, "-fault", false, false); + + detNormalizedStatImfileRow *row = detNormalizedStatImfileRowAlloc + (det_id, + iteration, + class_id, + norm, + "full", + code); + + if (!detNormalizedStatImfileInsertObject(config->dbh, row)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +bool normalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_normalizedstat.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "WHERE fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "WHERE fault = 0"); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detNormalizedStatImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool revertnormalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertnormalizedstat.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detNormalizedStatImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updatenormalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + if (!setNormStatImfileDataState(config, det_id, iteration, class_id, data_state)) { + return false; + } + return true; +} + +bool pendingcleanup_normalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detNormalizedStatImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detNormalizedStatImfile.iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detNormalizedStatImfile.det_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_normalizedstat.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupNormStatImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool donecleanup_normalizedstatMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detNormalizedStatImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detNormalizedStatImfile.iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detNormalizedStatImfile.det_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_normalizedstat.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_normalizedstat", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_processedexp.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_processedexp.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_processedexp.c (revision 41104) @@ -0,0 +1,425 @@ +/* + * dettool_processedexp.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +bool toprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_toprocessedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, exp_id, recip, -bg, -bg_stdev + // are required + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // Required if fault == 0 + PXOPT_LOOKUP_STR(recipe, config->args, "-recip", (fault == 0), false); + + // optional + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(fringe_0, config->args, "-fringe_0", false, false); + PXOPT_LOOKUP_F64(fringe_1, config->args, "-fringe_1", false, false); + PXOPT_LOOKUP_F64(fringe_2, config->args, "-fringe_2", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + psString query = pxDataGet("dettool_addprocessedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, (long long) det_id, (long long) exp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + psFree(output); + + // create a new detProcessedImfile object + detProcessedExpRow *detRow = detProcessedExpRowAlloc( + det_id, + exp_id, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + fringe_0, + fringe_1, + fringe_2, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + fault + ); + + // insert the new row into the detProcessedImfile table + if (!detProcessedExpInsertObject(config->dbh, detRow)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detRow); + return false; + } + + psFree(detRow); + + return true; +} + +bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM detProcessedExp"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detProcessedExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detProcessedExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool revertprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertprocessedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detProcessedExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updateprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args,where,"-det_id","det_id","=="); + PXOPT_COPY_S64(config->args,where,"-exp_id","exp_id","=="); + + if (!setProcessedExpDataState(config, where, data_state)) { + return false; + } + psFree(where); + return true; +} + +bool updatestateprocessedMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + // PXOPT_LOOKUP_S64(det_id,config->args, "-det_id",true,false); + PXOPT_LOOKUP_STR(data_state,config->args, "-data_state", true, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args,where,"-det_id","det_id","=="); + PXOPT_COPY_S64(config->args,where,"-exp_id","exp_id","=="); + + if (!setProcessedExpDataState(config,where,data_state)) { + return(false); + } + if (!setProcessedImfileDataState(config,where,data_state)) { + return(false); + } + psFree(where); + return(true); +} + +bool pendingcleanup_processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_processedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// XXX SQL missing +bool donecleanup_processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_processedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanupProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_processedimfile.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_processedimfile.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_processedimfile.c (revision 41104) @@ -0,0 +1,447 @@ +/* + * dettool_processedimfile.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +bool toprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_toprocessedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree (where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, exp_id, class_id, uri, recipe, -bg, -bg_stdev are required + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // Required if fault == 0 + PXOPT_LOOKUP_STR(uri, config->args, "-uri", (fault == 0), false); + PXOPT_LOOKUP_STR(recipe, config->args, "-recip", (fault == 0), false); + + // optional + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(fringe_0, config->args, "-fringe_0", false, false); + PXOPT_LOOKUP_F64(fringe_1, config->args, "-fringe_1", false, false); + PXOPT_LOOKUP_F64(fringe_2, config->args, "-fringe_2", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + // find the matching rawImfile by exp_id/class_id + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psArray *rawImfiles = rawImfileSelectRowObjects(config->dbh, where, 0); + psFree(where); + + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found "); + return false; + } + + // create a new detProcessedImfile object + detProcessedImfileRow *detRow = detProcessedImfileRowAlloc( + det_id, + exp_id, + class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + fringe_0, + fringe_1, + fringe_2, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + fault + ); + psFree(rawImfiles); + + // insert the new row into the detProcessedImfile table + if (!detProcessedImfileInsertObject(config->dbh, detRow)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(detRow); + return false; + } + + psFree(detRow); + + return true; +} + +bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detProcessedImfile.det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "detProcessedImfile.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detProcessedImfile.class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-select_state", "detRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-select_mode", "detRun.mode", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(included, config->args, "-included", false); + + psString query = pxDataGet("dettool_processedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + bool hasWhere = false; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + hasWhere = true; + } + psFree (where); + + // restrict search to included imfiles + if (included) { + if (hasWhere) { + psStringAppend(&query, " AND detInputExp.include = 1"); + } else { + psStringAppend(&query, " WHERE detInputExp.include = 1"); + } + hasWhere = true; + } + + if (hasWhere) { + psStringAppend(&query, " AND"); + } else { + psStringAppend(&query, " WHERE"); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " detProcessedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", " detProcessedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +bool revertprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertprocessedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detProcessedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updateprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args,where,"-det_id", "det_id","=="); + PXOPT_COPY_S64(config->args,where,"-exp_id", "exp_id","=="); + PXOPT_COPY_STR(config->args,where,"-class_id","class_id","=="); + + if (!setProcessedImfileDataState(config, where, data_state)) { + return false; + } + psFree(where); + return true; +} + +bool pendingcleanup_processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); +/* int i; */ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_processedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } +/* fprintf(stderr,"DETTOOL:procimfile: %s\n",query); */ + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); +/* fprintf(stderr,"WTF !output?\n"); */ + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); +/* fprintf(stderr,"WTF no rows??\n"); */ + psFree(output); + return true; + } + + // negative simple so the default is true +/* i = (int) ippdbPrintMetadatas(stdout, output, "detCleanupProcessedImfile", !simple); */ +/* fprintf(stderr,">>%d<<\n",i); */ + if (!ippdbPrintMetadatas(stdout, output, "detCleanupProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } +/* fprintf(stderr,"DETTOOL:procimfile: %s\n",output); */ +/* psFree(output); */ + + psFree(output); + + return true; +} + +// XXX SQL missing +bool donecleanup_processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_processedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_processedimfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_residexp.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_residexp.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_residexp.c (revision 41104) @@ -0,0 +1,500 @@ +/* + * dettool_residexp.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +/* + The SQL returns a list of exposures for which all component class ids have had residuals + created. This list includes the detrend id, iteration, exposure id, detrend type, and + whether the exposure was included in the stack for this iteration. +*/ + +bool toresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_toresidexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingResidExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +/* + The SQL returns a list of exposures for which all component class ids have had residuals + created. This list includes the detrend id, iteration, exposure id, detrend type, and + whether the exposure was included in the stack for this iteration. +*/ + +bool addresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // limit search by det_id, iteration, exp_id + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", false, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); // required + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(recipe, config->args, "-recip", (fault == 0), false); // Required if fault == 0 + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(bg_skewness, config->args, "-bg_skewness", false, false); + PXOPT_LOOKUP_F64(bg_kurtosis, config->args, "-bg_kurtosis", false, false); + PXOPT_LOOKUP_F64(bin_stdev, config->args, "-bin_stdev", false, false); + PXOPT_LOOKUP_F64(fringe_0, config->args, "-fringe_0", false, false); + PXOPT_LOOKUP_F64(fringe_1, config->args, "-fringe_1", false, false); + PXOPT_LOOKUP_F64(fringe_2, config->args, "-fringe_2", false, false); + PXOPT_LOOKUP_F64(fringe_resid_0, config->args, "-fringe_resid_0", false, false); + PXOPT_LOOKUP_F64(fringe_resid_1, config->args, "-fringe_resid_1", false, false); + PXOPT_LOOKUP_F64(fringe_resid_2, config->args, "-fringe_resid_2", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_BOOL(reject, config->args, "-reject", false); + + psString query = pxDataGet("dettool_toresidexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + // XXX check psError here + psError(PS_ERR_UNKNOWN, false, "no detResidImfile rows found"); + psFree(output); + return false; + } + psFree(output); + + if (!detResidExpInsert( + config->dbh, + det_id, + iteration, + exp_id, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + bg_skewness, + bg_kurtosis, + bin_stdev, + fringe_0, + fringe_1, + fringe_2, + fringe_resid_0, + fringe_resid_1, + fringe_resid_2, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + !reject, + fault + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +bool residexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-recip", "recipe", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(reject, config->args, "-reject", false); + + psString query = pxDataGet("dettool_residexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detResidExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detResidExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detResidExp.fault = 0"); + } + + // list only accepted rows + // XXX the usage of this flag is unclear : -reject means "accepted"? + if (reject) { + psStringAppend(&query, " %s", "AND detResidExp.accept != 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detResidExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool revertresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertresidexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detResidExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + + +bool updateresidexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(reject, config->args, "-reject", false); + + // build a query to search by det_id, iteration, exp_id + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + // find the values we're going to set + // copy everything but det_id, iteration, & exp_id from the args and + // remove the '-' prefix + psMetadata *set = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, set, "-recip", "recipe", "=="); + PXOPT_COPY_F64(config->args, set, "-bg", "bg", "=="); + PXOPT_COPY_F64(config->args, set, "-bg_stdev", "bg_stdev", "=="); + PXOPT_COPY_F64(config->args, set, "-bg_mean_stdev", "bg_mean_stdev", "=="); + PXOPT_COPY_STR(config->args, set, "-path_base", "path_base", "=="); + PXOPT_COPY_STR(config->args, set, "-data_state", "data_state", "=="); + + // double-check the allowed data_state values + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", false, false); + if (data_state) { + if (!isValidDataState (data_state)) return false; + } + + // this can't be PXOPT_ macro-ized as reject is !'d + if (!psMetadataAddBool(set, PS_LIST_TAIL, "accept", 0, "==", !reject)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item accept"); + psFree(set); + psFree(where); + return false; + } + + long changed = psDBUpdateRows(config->dbh, "detResidExp", where, set); + psFree(set); + psFree(where); + + if (changed < 0) { + psError(PS_ERR_UNKNOWN, false, "no rows were updated"); + return false; + } + + return true; +} + +bool updatestateresidMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + // PXOPT_LOOKUP_S64(det_id,config->args, "-det_id",true,false); + PXOPT_LOOKUP_STR(data_state,config->args, "-data_state", true, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args,where,"-det_id","det_id","=="); + PXOPT_COPY_S64(config->args,where,"-exp_id","exp_id","=="); + + if (!setResidExpDataState(config,where,data_state)) { + return(false); + } + if (!setResidImfileDataState(config,where,data_state)) { + return(false); + } + psFree(where); + return(true); +} + +bool pendingcleanup_residexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_residexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupResidExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool donecleanup_residexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_residexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_residexp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_residimfile.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_residimfile.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_residimfile.c (revision 41104) @@ -0,0 +1,428 @@ +/* + * dettool_residimfile.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +bool toresidimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_toresidimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingResidImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool addresidimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); // required + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", false, false); + + PXOPT_LOOKUP_S64(ref_det_id, config->args, "-ref_det_id", true, false); // required + PXOPT_LOOKUP_S32(ref_iter, config->args, "-ref_iter", true, false); + + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); // required + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); // required + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", (fault == 0), false); // Required if fault == 0 + PXOPT_LOOKUP_STR(recipe, config->args, "-recip", (fault == 0), false); // Required if fault == 0 + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(bg_skewness, config->args, "-bg_skewness", false, false); + PXOPT_LOOKUP_F64(bg_kurtosis, config->args, "-bg_kurtosis", false, false); + PXOPT_LOOKUP_F64(bin_stdev, config->args, "-bin_stdev", false, false); + PXOPT_LOOKUP_F64(fringe_0, config->args, "-fringe_0", false, false); + PXOPT_LOOKUP_F64(fringe_1, config->args, "-fringe_1", false, false); + PXOPT_LOOKUP_F64(fringe_2, config->args, "-fringe_2", false, false); + PXOPT_LOOKUP_F64(fringe_resid_0, config->args, "-fringe_resid_0", false, false); + PXOPT_LOOKUP_F64(fringe_resid_1, config->args, "-fringe_resid_1", false, false); + PXOPT_LOOKUP_F64(fringe_resid_2, config->args, "-fringe_resid_2", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + if (!detResidImfileInsert( + config->dbh, + det_id, + iteration, + ref_det_id, + ref_iter, + exp_id, + class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + bg_skewness, + bg_kurtosis, + bin_stdev, + fringe_0, + fringe_1, + fringe_2, + fringe_resid_0, + fringe_resid_1, + fringe_resid_2, + user_1, + user_2, + user_3, + user_4, + user_5, + path_base, + "full", + fault + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +bool residimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detResidImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detResidImfile.iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "detResidImfile.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detResidImfile.class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-recip", "detResidImfile.recipe", "=="); + PXOPT_COPY_STR(config->args, where, "-select_state", "detRun.state", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(included, config->args, "-included", false); + + psString query = pxDataGet("dettool_residimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + bool hasWhere = false; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + hasWhere = true; + } + psFree(where); + + // restrict search to included imfiles + if (included) { + if (hasWhere) { + psStringAppend(&query, " AND detInputExp.include = 1"); + } else { + psStringAppend(&query, " WHERE detInputExp.include = 1"); + } + hasWhere = true; + } + + if (hasWhere) { + psStringAppend(&query, " AND"); + } else { + psStringAppend(&query, " WHERE"); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " detResidImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", " detResidImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawResidImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool revertresidimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertresidimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detResidImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updateresidimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args,where,"-det_id","det_id","=="); + PXOPT_COPY_S32(config->args,where,"-iteration", "iteration","=="); + PXOPT_COPY_S64(config->args,where,"-exp_id","exp_id","=="); + PXOPT_COPY_STR(config->args,where,"-class_id","class_id","=="); + + if (!setResidImfileDataState(config, where, data_state)) { + return false; + } + return true; +} + +bool pendingcleanup_residimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_residimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupResidImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool donecleanup_residimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_residimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_residimfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_stack.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_stack.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dettool_stack.c (revision 41104) @@ -0,0 +1,458 @@ +/* + * dettool_stack.c + * + * Copyright (C) 2006 Joshua Hoblitt & EAM; IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#include "dettool.h" + +static psArray *searchRawImfiles(pxConfig *config); + +bool tostackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_tostacked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detPendingStackedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// this is NOT a command-line mode : it is used by 'addstackedMode' +static psArray *searchRawImfiles(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + + // select exp_ids from detInputExp matching det_id & iteration + // where query should be pre-generated + psArray *detInputExp = detInputExpSelectRowObjects(config->dbh, where, 0); + psFree (where); + + if (!detInputExp) { + psError(PS_ERR_UNKNOWN, false, "no rawExp rows found"); + return NULL; + } + + // generate where query with just the exp_ids + psMetadata *where_exp_ids = psMetadataAlloc(); + for (long i = 0; i < psArrayLength(detInputExp); i++) { + detInputExpRow *row = detInputExp->data[i]; + if (!psMetadataAddS64(where_exp_ids, PS_LIST_TAIL, "exp_id", + PS_META_DUPLICATE_OK, "==", row->exp_id) + ) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(detInputExp); + psFree(where_exp_ids); + return NULL; + } + } + psFree(detInputExp); + + // select rawImfiles with matching exp_ids + psArray *rawImfiles = + rawImfileSelectRowObjects(config->dbh, where_exp_ids, 0); + psFree(where_exp_ids); + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "no rawImfile rows found"); + return NULL; + } + + return rawImfiles; +} + +bool addstackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // det_id, iteration, class_id, uri, & recipe are required + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // Required if fault == 0 + PXOPT_LOOKUP_STR(uri, config->args, "-uri", (fault == 0), false); + PXOPT_LOOKUP_STR(recipe, config->args, "-recip", (fault == 0), false); + + // optional + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + + // correlate the class_id against the input exposure(s) + // searchRawImfiles defines a where clause to search by det_id and iteration + psArray *rawImfiles = searchRawImfiles(config); + + if (!rawImfiles) { + psError(PS_ERR_UNKNOWN, false, "failed to get rawImfiles from db"); + return false; + } + + bool valid_class_id = false; + if (rawImfiles) { + for (long i = 0; i < psArrayLength(rawImfiles); i++) { + if (!rawImfiles->data[i]) { + fprintf (stderr, "*"); + continue; + } + if (strcmp(class_id, ((rawImfileRow *)rawImfiles->data[i])->class_id) == 0) { + valid_class_id = true; + break; + } + } + psFree(rawImfiles); + } + + if (!valid_class_id) { + psError(PS_ERR_UNKNOWN, true, + "class_id can not be correlated with the input exposures"); + return false; + } + + // create a new detStackedImfile object + detStackedImfileRow *stackedImfile = detStackedImfileRowAlloc( + det_id, + iteration, + class_id, + uri, + recipe, + bg, + bg_stdev, + bg_mean_stdev, + user_1, + user_2, + user_3, + user_4, + user_5, + "full", + fault + ); + + // insert the new row into the detProcessedImfile table + if (!detStackedImfileInsertObject(config->dbh, stackedImfile)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(stackedImfile); + return false; + } + + psFree(stackedImfile); + + return true; +} + +bool stackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-recip", "recipe", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("dettool_stacked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detStackedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND detStackedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND detStackedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +bool revertstackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all-run")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("dettool_revertstacked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "detStackedImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +bool updatestackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(det_id, config->args, "-det_id", true, false); + PXOPT_LOOKUP_S32(iteration, config->args, "-iteration", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", true, false); + + if (!setStackedImfileDataState(config, det_id, iteration, class_id, data_state)) { + return false; + } + return true; +} + +bool pendingcleanup_stackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detStackedImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detStackedImfile.iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detStackedImfile.class_id", "=="); + + psString query = pxDataGet("dettool_pendingcleanup_stacked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detCleanupStackedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// XXX SQL missing +bool donecleanup_stackedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-det_id", "detStackedImfile.det_id", "=="); + PXOPT_COPY_S32(config->args, where, "-iteration", "detStackedImfile.iteration", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "detStackedImfile.class_id", "=="); + + psString query = pxDataGet("dettool_donecleanup_stacked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dettool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "detDoneCleanup_stacked", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottool.c (revision 41104) @@ -0,0 +1,555 @@ +/* + * diffphottool.c + * + * Copyright (C) 2007-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "diffphottool.h" + +static bool definerunMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool inputMode(pxConfig *config); +static bool pendingMode(pxConfig *config); +static bool doneMode(pxConfig *config); +static bool advanceMode(pxConfig *config); +static bool revertMode(pxConfig *config); +static bool dataMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = diffphottoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(DIFFPHOTTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(DIFFPHOTTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(DIFFPHOTTOOL_MODE_INPUT, inputMode); + MODECASE(DIFFPHOTTOOL_MODE_PENDING, pendingMode); + MODECASE(DIFFPHOTTOOL_MODE_DONE, doneMode); + MODECASE(DIFFPHOTTOOL_MODE_ADVANCE, advanceMode); + MODECASE(DIFFPHOTTOOL_MODE_REVERT, revertMode); + MODECASE(DIFFPHOTTOOL_MODE_DATA, dataMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // Options + PXOPT_LOOKUP_STR(set_workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(set_data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(set_reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(set_note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // Selections + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs(config, where, "-label", "diffRun.label", "LIKE"); + pxAddLabelSearchArgs(config, where, "-data_group", "diffRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + + + psString query = pxDataGet("diffphottool_definerun.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", clause); + psFree(clause); + } + psFree(where); + + if (!psDBTransaction(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "Unable to run query: %s", query); + psFree(where); + return false; + } + psFree(where); + + psArray *results = p_psDBFetchResult(config->dbh); // Results of query + if (!results) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!psArrayLength(results)) { + psTrace("diffphottool", 1, "no rows found"); + psFree(results); + return true; + } + + if (pretend) { + if (!ippdbPrintMetadatas(stdout, results, "diffPhotRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(results); + return false; + } + psFree(results); + return true; + } + + for (int i = 0; i < results->n; i++) { + psMetadata *row = results->data[i]; // Output row from query + bool mdok; // Status of MD lookup + psS64 diff_id = psMetadataLookupS64(&mdok, row, "diff_id"); + const char *workdir = psMetadataLookupStr(&mdok, row, "workdir"); + const char *label = psMetadataLookupStr(&mdok, row, "data_group"); + const char *data_group = psMetadataLookupStr(&mdok, row, "data_group"); + const char *reduction = psMetadataLookupStr(&mdok, row, "reduction"); + const char *note = psMetadataLookupStr(&mdok, row, "note"); + + diffPhotRunRow *run = diffPhotRunRowAlloc(0, diff_id, "new", + set_workdir ? set_workdir : workdir, + set_label ? set_label : label, + set_data_group ? set_data_group : data_group, + set_reduction ? set_reduction : reduction, + registered, + set_note ? set_note : note, 0); + if (!diffPhotRunInsertObject(config->dbh, run)) { + psError(psErrorCodeLast(), false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + psFree(run); + psFree(results); + return false; + } + + run->diff_phot_id = psDBLastInsertID(config->dbh); + + if (!diffPhotRunPrintObject(stdout, run, !simple)) { + psError(psErrorCodeLast(), false, "failed to print object"); + psFree(run); + psFree(results); + return false; + } + psFree(run); + } + psFree(results); + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-diff_phot_id", "diff_phot_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-data_group", "data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE diffPhotRun"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "diffPhotRun", "diff_phot_id", "diffPhotSkyfile", true, false); + + psFree(query); + psFree(where); + + return result; +} + + +static bool inputMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-diff_phot_id", "diff_phot_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("diffphottool_input.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", clause); + psFree(clause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("diffphottool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "diffSkyfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + + +static bool pendingMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_phot_id", "diff_phot_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "diffPhotRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("diffphottool_pending.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", clause); + psFree(clause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("diffphottool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "diffPhotRun", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + + +static bool doneMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(diff_phot_id, config->args, "-diff_phot_id", true, false); // required + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", true, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_S64(magicked, config->args, "-magicked", false, false); + + psString version = pxMergeCodeVersions(ver_pslib, ver_psmodules); + version = pxMergeCodeVersions(version, ver_ppstats); + version = pxMergeCodeVersions(version, ver_psphot); + + if (!diffPhotSkyfileInsert(config->dbh, diff_phot_id, skycell_id, path_base, dtime_script, hostname, + fault, quality, version, magicked)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + +static bool advanceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_phot_id", "diff_phot_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffPhotRun.label", "=="); + + psString query = pxDataGet("diffphottool_advance.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psStringCopy(""); + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereClause, "\nAND %s", clause); + psFree(clause); + } + psFree(where); + + if (!psDBTransaction(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + psFree(whereClause); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + for (int i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row of interest + psS64 diff_phot_id = psMetadataLookupS64(NULL, row, "diff_phot_id"); + psS64 magicked = psMetadataLookupS64(NULL, row, "magicked"); + + const char *query = "UPDATE diffPhotRun SET state = 'full', magicked = %" PRId64 + " WHERE diff_phot_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, magicked, diff_phot_id)) { + psError(psErrorCodeLast(), false, + "failed to change state for diff_phot_id %" PRId64, diff_phot_id); + if (!psDBRollback(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + } + psFree(output); + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + + +static bool revertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_phot_id", "diff_phot_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + if (!psListLength(where->list) && !all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + psFree(where); + return false; + } + + psString query = pxDataGet("diffphottool_revert.sql"); + if (!query) { + psError(psErrorCodeLast(), false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", clause); + psFree(clause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool dataMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-diff_phot_id", "diff_phot_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("diffphottool_data.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", clause); + psFree(clause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("diffphottool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "diffPhotSkyfile", !simple)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottool.h (revision 41104) @@ -0,0 +1,39 @@ +/* + * diffphottool.h + * + * Copyright (C) 2007-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef DIFFPHOTTOOL_H +#define DIFFPHOTTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + DIFFPHOTTOOL_MODE_NONE = 0x0, + DIFFPHOTTOOL_MODE_DEFINERUN, + DIFFPHOTTOOL_MODE_UPDATERUN, + DIFFPHOTTOOL_MODE_INPUT, + DIFFPHOTTOOL_MODE_PENDING, + DIFFPHOTTOOL_MODE_DONE, + DIFFPHOTTOOL_MODE_ADVANCE, + DIFFPHOTTOOL_MODE_REVERT, + DIFFPHOTTOOL_MODE_DATA, +} diffphottoolMode; + +pxConfig *diffphottoolConfig(pxConfig *config, int argc, char **argv); + +#endif // DIFFPHOTTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/diffphottoolConfig.c (revision 41104) @@ -0,0 +1,161 @@ +/* + * diffphottoolConfig.c + * + * Copyright (C) 2007-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "diffphottool.h" + +pxConfig *diffphottoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by data_group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-comment", 0, "search for comment (LIKE)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "search by diffphot ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label (LIKE comparison)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -input + psMetadata *inputArgs = psMetadataAlloc(); + psMetadataAddS64(inputArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "search by diffphot ID", 0); + psMetadataAddStr(inputArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddU64(inputArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(inputArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pending + psMetadata *pendingArgs = psMetadataAlloc(); + psMetadataAddS64(pendingArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "search by diffphot ID", 0); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddU64(pendingArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -done + psMetadata *doneArgs = psMetadataAlloc(); + psMetadataAddS64(doneArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "define diffphot ID (required)", 0); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell of file (required)", NULL); + psMetadataAddS16(doneArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(doneArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location (required)", NULL); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-hostname", 0, "set hostname (required)", NULL); + psMetadataAddF32(doneArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(doneArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddS64(doneArgs, PS_LIST_TAIL, "-magicked", 0, "define magicked state", 0); + + // -advance + psMetadata *advanceArgs = psMetadataAlloc(); + psMetadataAddS64(advanceArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "select by diffphot ID", 0); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "select by label", NULL); + psMetadataAddS32(advanceArgs, PS_LIST_TAIL, "-limit", 0, "limit number of results", 0); + + // -revert + psMetadata *revertArgs = psMetadataAlloc(); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "search by diffphot ID", 0); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS16(revertArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertArgs, PS_LIST_TAIL, "-all", 0, "allow no search terms", 0); + + // -data + psMetadata *dataArgs = psMetadataAlloc(); + psMetadataAddS64(dataArgs, PS_LIST_TAIL, "-diff_phot_id", 0, "search by diffphot ID", 0); + psMetadataAddStr(dataArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddU64(dataArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(dataArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definerun", "", DIFFPHOTTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-updaterun", "", DIFFPHOTTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-input", "", DIFFPHOTTOOL_MODE_INPUT, inputArgs); + PXOPT_ADD_MODE("-pending", "", DIFFPHOTTOOL_MODE_PENDING, pendingArgs); + PXOPT_ADD_MODE("-done", "", DIFFPHOTTOOL_MODE_DONE, doneArgs); + PXOPT_ADD_MODE("-advance", "", DIFFPHOTTOOL_MODE_ADVANCE, advanceArgs); + PXOPT_ADD_MODE("-revert", "", DIFFPHOTTOOL_MODE_REVERT, revertArgs); + PXOPT_ADD_MODE("-data", "", DIFFPHOTTOOL_MODE_DATA, dataArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/difftool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/difftool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/difftool.c (revision 41104) @@ -0,0 +1,3660 @@ +/* + * difftool.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "difftool.h" + +static bool definerunMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool addinputskyfileMode(pxConfig *config); +static bool inputskyfileMode(pxConfig *config); +static bool todiffskyfileMode(pxConfig *config); +static bool adddiffskyfileMode(pxConfig *config); +static bool advanceMode(pxConfig *config); +static bool diffskyfileMode(pxConfig *config); +static bool revertdiffskyfileMode(pxConfig *config); +static bool definepoprunMode(pxConfig *config); +static bool definewarpstackMode(pxConfig *config); +//static bool definewarpstackOldMethodMode(pxConfig *config); +static bool definewarpwarpMode(pxConfig *config); +static bool definestackstackMode(pxConfig *config); +static bool tosummaryMode(pxConfig *config); +static bool addsummaryMode(pxConfig *config); +static bool pendingcleanuprunMode(pxConfig *config); +static bool pendingcleanupskyfileMode(pxConfig *config); +static bool revertcleanupMode(pxConfig *config); +static bool donecleanupMode(pxConfig *config); +static bool updatediffskyfileMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +static bool setdiffRunState(pxConfig *config, psS64 diff_id, const char *state, psS64 magicked); +static bool change_skyfile_data_state(pxConfig *config, psString data_state, psString run_state); +static bool tocleanedskyfileMode(pxConfig *config); +static bool topurgedskyfileMode(pxConfig *config); +static bool toscrubbedskyfileMode(pxConfig *config); +static bool tofullskyfileMode(pxConfig *config); +static bool listrunMode(pxConfig *config); +static bool listssrunMode(pxConfig *config); +static bool setskyfiletoupdateMode(pxConfig *config); + + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = difftoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(DIFFTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(DIFFTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(DIFFTOOL_MODE_ADDINPUTSKYFILE, addinputskyfileMode); + MODECASE(DIFFTOOL_MODE_INPUTSKYFILE, inputskyfileMode); + MODECASE(DIFFTOOL_MODE_TODIFFSKYFILE, todiffskyfileMode); + MODECASE(DIFFTOOL_MODE_ADDDIFFSKYFILE, adddiffskyfileMode); + MODECASE(DIFFTOOL_MODE_ADVANCE, advanceMode); + MODECASE(DIFFTOOL_MODE_DIFFSKYFILE, diffskyfileMode); + MODECASE(DIFFTOOL_MODE_REVERTDIFFSKYFILE, revertdiffskyfileMode); + MODECASE(DIFFTOOL_MODE_DEFINEPOPRUN, definepoprunMode); + MODECASE(DIFFTOOL_MODE_DEFINEWARPSTACK, definewarpstackMode); + // MODECASE(DIFFTOOL_MODE_DEFINEWARPSTACKOLDMETHOD, definewarpstackOldMethodMode); + MODECASE(DIFFTOOL_MODE_DEFINEWARPWARP, definewarpwarpMode); + MODECASE(DIFFTOOL_MODE_DEFINESTACKSTACK, definestackstackMode); + MODECASE(DIFFTOOL_MODE_TOSUMMARY, tosummaryMode); + MODECASE(DIFFTOOL_MODE_ADDSUMMARY, addsummaryMode); + MODECASE(DIFFTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunMode); + MODECASE(DIFFTOOL_MODE_PENDINGCLEANUPSKYFILE, pendingcleanupskyfileMode); + MODECASE(DIFFTOOL_MODE_REVERTCLEANUP, revertcleanupMode); + MODECASE(DIFFTOOL_MODE_DONECLEANUP, donecleanupMode); + MODECASE(DIFFTOOL_MODE_UPDATEDIFFSKYFILE, updatediffskyfileMode); + MODECASE(DIFFTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(DIFFTOOL_MODE_IMPORTRUN, importrunMode); + MODECASE(DIFFTOOL_MODE_TOCLEANEDSKYFILE, tocleanedskyfileMode); + MODECASE(DIFFTOOL_MODE_TOPURGEDSKYFILE, topurgedskyfileMode); + MODECASE(DIFFTOOL_MODE_TOSCRUBBEDSKYFILE, toscrubbedskyfileMode); + MODECASE(DIFFTOOL_MODE_TOFULLSKYFILE, tofullskyfileMode); + MODECASE(DIFFTOOL_MODE_LISTRUN, listrunMode); + MODECASE(DIFFTOOL_MODE_LISTSSRUN, listssrunMode); + MODECASE(DIFFTOOL_MODE_SETSKYFILETOUPDATE, setskyfiletoupdateMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); + PXOPT_LOOKUP_BOOL(bothways, config->args, "-bothways", false); + PXOPT_LOOKUP_BOOL(exposure, config->args, "-exposure", false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_S16(diff_mode, config->args, "-set_diff_mode", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + // default + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + diffRunRow *run = diffRunRowAlloc( + 0, // ID + "reg", // state + workdir, + label, + data_group ? data_group : label, + dist_group, + reduction, + NULL, // dvodb + registered, + tess_id, + bothways, + exposure, + false, + NULL, // software version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + diff_mode, + note + ); + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc diffRun object"); + return false; + } + if (!diffRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return false; + } + + // get the assigned diff_id + run->diff_id = psDBLastInsertID(config->dbh); + + if (!diffRunPrintObject(stdout, run, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + psFree(run); + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-data_group","data_group","LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group","dist_group","LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE diffRun"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "diffRun", "diff_id", "diffSkyfile", true, true); + + psFree(query); + psFree(where); + + return result; +} + + +static bool addinputskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", true, false); + + // optional + PXOPT_LOOKUP_S64(stack1, config->args, "-stack1", false, false); + PXOPT_LOOKUP_S64(stack2, config->args, "-stack2", false, false); + PXOPT_LOOKUP_S64(warp1, config->args, "-warp1", false, false); + PXOPT_LOOKUP_S64(warp2, config->args, "-warp2", false, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + + // must provide either stack1 or warp1 but not BOTH + if (!(stack1 || warp1)) { + psError(PS_ERR_UNKNOWN, true, "either -stack1 or -warp1 must be specified"); + return false; + } + if (stack1 && warp1) { + psError(PS_ERR_UNKNOWN, true, "either -stack1 or -warp1 must be specified"); + return false; + } + // must provide either stack2 or warp2 but not BOTH + if (!(stack2 || warp2)) { + psError(PS_ERR_UNKNOWN, true, "either -stack2 or -warp2 must be specified"); + return false; + } + if (stack2 && warp2) { + psError(PS_ERR_UNKNOWN, true, "either -stack2 or -warp2 must be specified"); + return false; + } + + // if a warp1 was provided we need to lookup the and tess_id from the diffRun + psString tess_id = NULL; + if (warp1) { + if (!p_psDBRunQueryF(config->dbh, "SELECT * from diffRun WHERE diff_id = %" PRId64, diff_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "diff_id %" PRId64 " not found", diff_id); + psFree(output); + return false; + } + + diffRunRow *run = diffRunObjectFromMetadata(output->data[0]); + tess_id = run->tess_id; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!diffInputSkyfileInsert(config->dbh, + diff_id, + skycell_id, + warp1 ? warp1 : PS_MAX_S64, // defined or NULL + stack1 ? stack1 : PS_MAX_S64, // defined or NULL + warp2 ? warp2 : PS_MAX_S64, // defined or NULL + stack2 ? stack2 : PS_MAX_S64, // defined or NULL + tess_id, + 0 // diff_skyfile_id + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, "SELECT count(diff_id) FROM diffRun JOIN diffInputSkyfile USING(diff_id) WHERE diff_id = %" PRId64, diff_id)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "diff_id %" PRId64 " not found", diff_id); + psFree(output); + return false; + } + + bool status; + psS32 count = psMetadataLookupS32(&status, output->data[0], "count(diff_id)"); + + if (count == 2) { + if (!setdiffRunState(config, diff_id, "new", false)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool inputskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "diffInputSkyfile.skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "tess_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(template, config->args, "-template", false); + PXOPT_LOOKUP_BOOL(input, config->args, "-input", false); + + if (template && input) { + // User apparently wants both, which is the default behaviour + template = false; + input = false; + } + + // find all rawImfiles matching the default query + psString query = pxDataGet("difftool_inputskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psStringCopy(""); + if (psListLength(where->list)) { + whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringPrepend(&whereClause, "\n WHERE "); + } + psFree(where); + + // Add condition to get only templates or only inputs + psString templateClause = psStringCopy(""); + { + if (template) { + psStringAppend(&templateClause, "\n WHERE %s", " template != 0"); + } else if (input) { + psStringAppend(&templateClause, "\n WHERE %s", " template = 0"); + } + } + + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause, whereClause, whereClause, whereClause, templateClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(templateClause); + psFree(whereClause); + psFree(query); + return false; + } + psFree(templateClause); + psFree(whereClause); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffInputSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool todiffskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("difftool_todiffskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER by priority DESC, diff_id, skycell_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool adddiffskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", true, false); // required + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", (fault == 0), false); + + // optional + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F32(dtime_diff, config->args, "-dtime_diff", false, false); + PXOPT_LOOKUP_F32(dtime_match, config->args, "-dtime_match", false, false); + PXOPT_LOOKUP_F32(dtime_phot, config->args, "-dtime_phot", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_S32(stamps_num, config->args, "-stamps_num", false, false); + PXOPT_LOOKUP_F32(stamps_mean, config->args, "-stamps_mean", false, false); + PXOPT_LOOKUP_F32(stamps_rms, config->args, "-stamps_rms", false, false); + PXOPT_LOOKUP_F32(norm, config->args, "-norm", false, false); + PXOPT_LOOKUP_F32(bg_diff, config->args, "-bg_diff", false, false); + PXOPT_LOOKUP_F32(kernel_x, config->args, "-kernel_x", false, false); + PXOPT_LOOKUP_F32(kernel_y, config->args, "-kernel_y", false, false); + PXOPT_LOOKUP_F32(kernel_xx, config->args, "-kernel_xx", false, false); + PXOPT_LOOKUP_F32(kernel_xy, config->args, "-kernel_xy", false, false); + PXOPT_LOOKUP_F32(kernel_yy, config->args, "-kernel_yy", false, false); + PXOPT_LOOKUP_F32(deconv_max, config->args, "-deconv_max", false, false); + PXOPT_LOOKUP_S32(sources, config->args, "-sources", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_F32(good_frac, config->args, "-good_frac", false, false); + PXOPT_LOOKUP_S64(magicked, config->args, "-magicked", false, false); + + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_STR(ver_ppsub, config->args, "-ver_ppsub", false, false); + PXOPT_LOOKUP_STR(ver_streaks, config->args, "-ver_streaks", false, false); + + PXOPT_LOOKUP_S32(maskfrac_npix, config->args, "-maskfrac_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_static, config->args, "-maskfrac_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_dynamic, config->args, "-maskfrac_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_magic, config->args, "-maskfrac_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_advisory, config->args, "-maskfrac_advisory", false, false); + + psTrace("czw.test",1,"Received versions: pslib %s psmodules %s psphot %s ppstats %s ppsub %s streaks %s\n", + ver_pslib,ver_psmodules,ver_psphot,ver_ppstats,ver_ppsub,ver_streaks); + psString ver_code = NULL; + if ((ver_pslib)&&(ver_psmodules)) { + ver_code = pxMergeCodeVersions(ver_pslib,ver_psmodules); + } + if (ver_psphot) { + ver_code = pxMergeCodeVersions(ver_code,ver_psphot); + } + if (ver_ppstats) { + ver_code = pxMergeCodeVersions(ver_code,ver_ppstats); + } + if (ver_ppsub) { + ver_code = pxMergeCodeVersions(ver_code,ver_ppsub); + } + if (ver_streaks) { + ver_code = pxMergeCodeVersions(ver_code,ver_streaks); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!diffSkyfileInsert(config->dbh, + diff_id, + skycell_id, + path_base, + "full", + bg, + bg_stdev, + stamps_num, + stamps_mean, + stamps_rms, + norm, + bg_diff, + kernel_x, + kernel_y, + kernel_xx, + kernel_xy, + kernel_yy, + deconv_max, + sources, + dtime_diff, + dtime_match, + dtime_phot, + dtime_script, + hostname, + good_frac, + fault, + quality, + magicked, + ver_code, + maskfrac_npix, + maskfrac_static, + maskfrac_dynamic, + maskfrac_magic, + maskfrac_advisory + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + return true; +} + +static bool advanceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // look for completed diffRuns + psString query = pxDataGet("difftool_completed_runs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereString = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereString, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, whereString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(whereString); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + psFree(whereString); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + psS64 diff_id = psMetadataLookupS64(NULL, row, "diff_id"); + psS64 magicked = psMetadataLookupS64(NULL, row, "magicked"); + + psString software_ver = NULL; + psS64 maskfrac_npix = 0; + psF32 maskfrac_static = 0; + psF32 maskfrac_dynamic = 0; + psF32 maskfrac_magic = 0; + psF32 maskfrac_advisory = 0; + + // Calculate run level masking and software state + if (!pxCoalesceRunStatus(config,"difftool_coalesce_run.sql",diff_id, + &software_ver,&maskfrac_npix, + &maskfrac_static,&maskfrac_dynamic, + &maskfrac_magic,&maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to generate run level statistics"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + // Set diffRun.software_ver to the appropriate value + if (software_ver) { + if (!pxSetRunSoftware(config, "diffRun", "diff_id", diff_id, software_ver)) { + psError(PS_ERR_UNKNOWN, false, "failed to set diffRun.software_ver for diff_id: %" PRId64, + diff_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // Set diffRun.maskfrac* to the appropriate values. + if (maskfrac_npix) { + if (!pxSetRunMaskfrac(config, "diffRun", "diff_id", diff_id, maskfrac_npix, maskfrac_static, + maskfrac_dynamic, maskfrac_magic, maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to set diffRun.software_ver for diff_id: %" PRId64, + diff_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // set diffRun.state to 'full' + if (!setdiffRunState(config, diff_id, "full", magicked)) { + psError(PS_ERR_UNKNOWN, false, "failed to change diffRun.state for diff_id: %" PRId64, + diff_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool diffskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffSkyfile.diff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "diffInputSkyfile.skycell_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_skyfile_id", "diffInputSkyfile.diff_skyfile_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "diffRun.tess_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "diffSkyfile.fault", "=="); + PXOPT_COPY_S64(config->args, where, "-magicked", "diffSkyfile.magicked", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "diffRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + PXOPT_LOOKUP_BOOL(template, config->args, "-template", false); + if (!template) { + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawInput.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawInput.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-warp_id", "warpInput.warp_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawInput.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawInput.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawInput.filter", "LIKE"); + } else { + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawTemplate.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawTemplate.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-warp_id", "warpTemplate.warp_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawTemplate.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawTemplate.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawTemplate.filter", "LIKE"); + } + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + + psString where2 = NULL; + pxmagicAddWhere(config, &where2, "diffSkyfile"); + pxspaceAddWhere(config, &where2, template ? "rawTemplate" : "rawInput"); + psString query = pxDataGet("difftool_skyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (where2) { + psStringAppend(&query, " WHERE diffRun.diff_id is not null %s", where2); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + psFree(where); + + if (pstamp_order) { + if (template) { + psStringAppend(&query, " ORDER BY rawTemplate.exp_id, diff_id DESC"); + } else { + psStringAppend(&query, " ORDER BY rawInput.exp_id, diff_id DESC"); + } + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool revertdiffskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffSkyfile.diff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "diffSkyfile.skycell_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "diffRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + { + psString query = pxDataGet("difftool_revertdiffskyfile_delete.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString query_updated = pxDataGet("difftool_revertdiffskyfile_updated.sql"); + if (!query_updated) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psStringAppend(&query_updated, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + psLogMsg("difftool", PS_LOG_INFO, "Deleted %" PRIu64 " rows", psDBAffectedRows(config->dbh)); + + if (!fault) { + // If fault has not been supplied, don't revert update faults with the magic value + // We don't do this for new runs because then they would never complete + // quality should be used to drop bad components + psStringAppend(&query_updated, " AND (diffSkyfile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + } + if (!p_psDBRunQuery(config->dbh, query_updated)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query_updated); + return false; + } + psFree(query_updated); + + psLogMsg("difftool", PS_LOG_INFO, "Updated %" PRIu64 " rows", psDBAffectedRows(config->dbh)); + } + + psFree(where); + + + return true; +} + + +static bool setdiffRunState(pxConfig *config, psS64 diff_id, const char *state, psS64 magicked) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid diffRun state: %s", state); + return false; + } + + if (magicked) { + char *query = "UPDATE diffRun SET state = '%s', magicked = %" PRId64 " WHERE diff_id = %"PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, state, magicked, diff_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for diff_id %"PRId64, diff_id); + return false; + } + } + else { + char *query = "UPDATE diffRun SET state = '%s' WHERE diff_id = %"PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, state, diff_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for diff_id %"PRId64, diff_id); + return false; + } + } + + return true; +} + + +#ifdef notdef +static bool setdiffRunStateByLabel(pxConfig *config, const char *label, const char *state) { + PS_ASSERT_PTR_NON_NULL(state,false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid diffRun state: %s", state); + return false; + } + + char *query = "UPDATE diffRun SET state = '%s' WHERE label = '%s'"; + if (!p_psDBRunQueryF(config->dbh,query,state,label)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for label %s", label); + return(false); + } + + return true; +} +#endif + +// Generate a single populated run +static bool populatedrun(psArray *list, // List of runs, to print + const char *workdir, // Working directory + const char *skycell_id, // Skycell identifier + const char *tess_id, // Tessellation identifier + const char *label, // label + const char *data_group, // data_group + const char *dist_group, // dist_group + const char *reduction, // reduction + const char *note, // note + psS64 input_warp_id, // Warp identifier for input image, PS_MAX_S64 for none + psS64 input_stack_id, // Stack identifier for input image, PS_MAX_S64 for none + psS64 template_warp_id, // Warp identifier for template image, PS_MAX_S64 for none + psS64 template_stack_id, // Stack identifier for template image, PS_MAX_S64 for none + pxConfig *config // Configuration + ) +{ + PS_ASSERT_STRING_NON_EMPTY(workdir, false); + PS_ASSERT_STRING_NON_EMPTY(skycell_id, false); + PS_ASSERT_STRING_NON_EMPTY(tess_id, false); + if ((input_warp_id == PS_MAX_S64 && input_stack_id == PS_MAX_S64) || + (input_warp_id != PS_MAX_S64 && input_stack_id != PS_MAX_S64)) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "One, and only one, input must be defined."); + return false; + } + if ((template_warp_id == PS_MAX_S64 && template_stack_id == PS_MAX_S64) || + (template_warp_id != PS_MAX_S64 && template_stack_id != PS_MAX_S64)) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "One, and only one, template must be defined."); + return false; + } + psS16 diff_mode = IPP_DIFF_MODE_UNDEFINED; + if ((input_warp_id != PS_MAX_S64) && (template_warp_id != PS_MAX_S64)) { + diff_mode = IPP_DIFF_MODE_WARP_WARP; + } + else if ((input_warp_id != PS_MAX_S64) && (template_stack_id != PS_MAX_S64)) { + diff_mode = IPP_DIFF_MODE_WARP_STACK; + } + else if ((input_stack_id != PS_MAX_S64) && (template_warp_id != PS_MAX_S64)) { + diff_mode = IPP_DIFF_MODE_STACK_WARP; + } + else if ((input_stack_id != PS_MAX_S64) && (template_stack_id != PS_MAX_S64)) { + diff_mode = IPP_DIFF_MODE_STACK_STACK; + } + + + // default + // PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + + diffRunRow *run = diffRunRowAlloc( + 0, // ID + "reg", // state + workdir, + label, + data_group ? data_group : label, + dist_group, + reduction, + NULL, // dvodb + registered, + tess_id, + false, + false, + 0, // magicked + NULL, // software version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + diff_mode, // diff_mode + note + ); + + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc diffRun object"); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!diffRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return true; + } + + // get the assigned diff_id + run->diff_id = psDBLastInsertID(config->dbh); + + // Template + if (!diffInputSkyfileInsert(config->dbh, + run->diff_id, + skycell_id, + input_warp_id, + input_stack_id, + template_warp_id, + template_stack_id, + tess_id, + 0 + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + char *query = "UPDATE diffRun SET state = 'new' WHERE diff_id = '%" PRId64 "'"; + if (!p_psDBRunQueryF(config->dbh, query, run->diff_id)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to set state to run for diff_id %" PRId64, run->diff_id); + return false; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (list) { + psArrayAdd(list, list->n, run); + } + + psFree(run); + + return true; +} + + +static bool definepoprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); // required options + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); // required options + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); // required options + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); + PXOPT_LOOKUP_S64(template_warp_id, config->args, "-template_warp_id", false, false); + PXOPT_LOOKUP_S64(template_stack_id, config->args, "-template_stack_id", false, false); + PXOPT_LOOKUP_S64(input_warp_id, config->args, "-input_warp_id", false, false); + PXOPT_LOOKUP_S64(input_stack_id, config->args, "-input_stack_id", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + if (template_stack_id && template_warp_id) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "Only one template can be defined."); + return false; + } + if (!template_stack_id && !template_warp_id) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, + "No template has been defined (-template_stack_id or -template_warp_id)"); + return false; + } + + if (input_stack_id && input_warp_id) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "Only one input can be defined."); + return false; + } + if (!input_stack_id && !input_warp_id) { + psError(PS_ERR_BAD_PARAMETER_VALUE, true, + "No input has been defined (-input_stack_id or -input_warp_id)"); + return false; + } + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + // Populated Run will generate the diff_mode value + if (!populatedrun(list, workdir, skycell_id, tess_id, label, data_group ? data_group : label, dist_group, reduction, note, + input_warp_id ? input_warp_id : PS_MAX_S64, + input_stack_id ? input_stack_id : PS_MAX_S64, + template_warp_id ? template_warp_id : PS_MAX_S64, + template_stack_id ? template_stack_id : PS_MAX_S64, + config)) { + psError(PS_ERR_UNKNOWN, false, "failed to create populated diffRun"); + psFree(list); + return false; + } + + if (!diffRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + psFree(list); + + return true; +} + +static bool definewarpstackMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_LOOKUP_BOOL(lapQuery, config->args, "-lap_query", false); + + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "comment", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-warp_label", "warpRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "warpRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + PXOPT_COPY_F32(config->args, where, "-good_frac", "warpSkyfile.good_frac", ">="); + PXOPT_COPY_STR(config->args, where, "-stack_label", "stackRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-stack_data_group", "stackRun.data_group", "=="); + + + // Add position dependence here. + if (!pxspaceBoxAddWhere(config,where)) { + psError(psErrorCodeLast(), false, "pxSpaceBoxAddWhere failed"); + return false; + } + + // PXOPT_LOOKUP_BOOL(available, config->args, "-available", false); + PXOPT_LOOKUP_BOOL(bothways, config->args, "-bothways", false); + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); // required option + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); // option + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); // option + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + PXOPT_LOOKUP_STR(warp_data_group, config->args, "-data_group", false, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(newTemplates, config->args, "-new-templates", false); + PXOPT_LOOKUP_BOOL(reRun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + + if (!data_group) { + if (warp_data_group) { + data_group = warp_data_group; + } + } + + // Get query file + psString query; + if (lapQuery) { + query = pxDataGet("difftool_definewarpstack.sql"); + } + else { + query = pxDataGet("difftool_definewarpstack_old.sql"); + } + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where,NULL); + psString whereClauseString = psStringCopy(""); + if (lapQuery) { + psString diffWhereClause = psStringCopy(""); + psString joinWhereClause = psStringCopy(""); + + // Don't queue things that already have diffs. + if (! (reRun || newTemplates) ) { + psStringAppend(&joinWhereClause, "\nAND diff_id IS NULL\n"); + } + // Append diff qualifiers, if we have them + if (label) { + psStringAppend(&diffWhereClause, "\nAND ((diffRun.label = '%s') OR (diffRun.label IS NULL))",label); + } + if (data_group) { + psStringAppend(&diffWhereClause, "\nAND ((diffRun.data_group = '%s') OR (diffRun.data_group IS NULL))",data_group); + } + if (reduction) { + psStringAppend(&diffWhereClause, "\nAND ((diffRun.reduction = '%s') OR (diffRun.reduction IS NULL))",reduction); + } + + psStringAppend(&whereClauseString, " \n AND %s ", whereClause); + // fprintf(stderr,query,whereClauseString); + + // This is just a simple query, so we don't need to do a transaction + if (!p_psDBRunQueryF(config->dbh, query, whereClauseString, diffWhereClause,joinWhereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(whereClause); + return(false); + } + + psFree(diffWhereClause); + psFree(joinWhereClause); + + } + else { + // Don't queue things that already have diffs. + if (! (reRun || newTemplates) ) { + psStringAppend(&whereClause, "\nAND diff_id IS NULL\n"); + } + // Append diff qualifiers, if we have them + if (label) { + psStringAppend(&whereClause, "\nAND ((diffRun.label = '%s') OR (diffRun.label IS NULL))",label); + } + if (data_group) { + psStringAppend(&whereClause, "\nAND ((diffRun.data_group = '%s') OR (diffRun.data_group IS NULL))",data_group); + } + if (reduction) { + psStringAppend(&whereClause, "\nAND ((diffRun.reduction = '%s') OR (diffRun.reduction IS NULL))",reduction); + } + + psStringAppend(&whereClauseString, " \n AND %s ", whereClause); + if (!p_psDBRunQueryF(config->dbh, query, whereClauseString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(whereClause); + return(false); + } + + } + + psFree(query); + psFree(whereClause); + psFree(whereClauseString); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + break; + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + break; + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + break; + } + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + long numGood = 0; + psS64 last_exp_id = 0; + psS64 diff_id = 0; + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; + bool mdok; + + psS64 exp_id = psMetadataLookupS64(&mdok, row, "exp_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "exp_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psString warpDataGroup = psMetadataLookupStr(&mdok, row, "warpDataGroup"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "warpDataGroup not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (exp_id != last_exp_id) { + if (diff_id != 0) { // We've added a run already, and are now switching to a new one. + // Set state to new + if (!setdiffRunState(config, diff_id, "new", false)) { + psError(PS_ERR_UNKNOWN, false, "failed to change diffRun.state for diff_id: %" PRId64, diff_id); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + // Commit results + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + } + + // Begin transaction + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + // Add a new diffRun row + psString tess_id = psMetadataLookupStr(&mdok, row, "tess_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "tess_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + diffRunRow *run = diffRunRowAlloc( + 0, // ID + "reg", // state + workdir, + label, + data_group ? data_group : warpDataGroup, + dist_group, + reduction, + NULL, // dvodb + registered, + tess_id, + bothways, // bothways (default is false) + true, // exposure + 0, // magicked + NULL, // software_version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + IPP_DIFF_MODE_WARP_STACK, + note + ); + if (!diffRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + } + diff_id = psDBLastInsertID(config->dbh); + run->diff_id = diff_id; + + psArrayAdd(list, list->n, run); + numGood++; + + last_exp_id = exp_id; + } // End Adding diffRun + + if (exp_id == last_exp_id) { + psString skycell_id = psMetadataLookupStr(&mdok, row, "skycell_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "skycell_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psString tess_id = psMetadataLookupStr(&mdok, row, "tess_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "tess_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 warp_id = psMetadataLookupS64(&mdok, row, "warp_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "warp_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 stack_id = psMetadataLookupS64(&mdok, row, "stack_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "stack_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + // Add a new skyfile row + diffInputSkyfileRow *skyfile = diffInputSkyfileRowAlloc( + diff_id, // ID + skycell_id, + warp_id, // warp1_id + PS_MAX_S64, // stack1 -> NULL + PS_MAX_S64, // warp2_id -> NULL + stack_id, // stack2 + tess_id, + 0 // diff_skyfile_id + ); + // fprintf(stderr,"%"PRId64 " %"PRId64 " %"PRId64 " %s %s\n",diff_id,warp_id,stack_id,skycell_id, tess_id); + if (!diffInputSkyfileInsertObject(config->dbh, skyfile)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } // End Adding skyfile + } // End parsing result set + + // Finish the last run's update and close the connection + if (diff_id != 0) { + if (!setdiffRunState(config, diff_id, "new", false)) { + psError(PS_ERR_UNKNOWN, false, "failed to change diffRun.state for diff_id: %" PRId64, diff_id); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + // Commit results + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + } + + if (numGood && !diffRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return(false); + } + psFree(list); + // Free things + return true; +} + +#if (0) +static bool definewarpstackOldMethodMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *expWhere = psMetadataAlloc(); + psMetadata *warp1Where = psMetadataAlloc(); // First set of restrictions on warp + psMetadata *warp2Where = psMetadataAlloc(); // Second set of restriction on warp + psMetadata *stackWhere = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, expWhere, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, expWhere, "-filter", "filter", "=="); + PXOPT_COPY_STR(config->args, expWhere, "-comment", "comment", "LIKE"); + + PXOPT_COPY_S64(config->args, warp1Where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, warp1Where, "-warp_label", "warpRun.label", "=="); + PXOPT_COPY_STR(config->args, warp1Where, "-tess_id", "warpRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, warp1Where, "-data_group", "warpRun.data_group", "=="); + PXOPT_COPY_S64(config->args, warp2Where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, warp2Where, "-tess_id", "warpRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, warp2Where, "-data_group", "warpRun.data_group", "=="); + PXOPT_COPY_STR(config->args, warp2Where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + PXOPT_COPY_STR(config->args, warp2Where, "-warp_label", "warpRun.label", "=="); + PXOPT_COPY_F32(config->args, warp2Where, "-good_frac", "warpSkyfile.good_frac", ">="); + PXOPT_COPY_STR(config->args, stackWhere, "-stack_label", "stackRun.label", "=="); + PXOPT_COPY_STR(config->args, stackWhere, "-stack_data_group", "stackRun.data_group", "=="); + + PXOPT_LOOKUP_BOOL(bothways, config->args, "-bothways", false); + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); // required option + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); // option + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); // option + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(newTemplates, config->args, "-new-templates", false); + PXOPT_LOOKUP_BOOL(reRun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(available, config->args, "-available", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + // find all things to queue + psString query = pxDataGet("difftool_definewarpstack_part1.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString warp1Query = NULL; + psString warp2Query = NULL; + psString stackQuery = NULL; + psString expQuery = NULL; + + if (psListLength(expWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(expWhere, NULL); + psStringAppend(&expQuery, "\n AND %s", whereClause); + psFree(whereClause); + } else { + expQuery = psStringCopy("\n"); + } + psFree(expWhere); + if (psListLength(warp1Where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(warp1Where, NULL); + psStringAppend(&warp1Query, "\n AND %s", whereClause); + psFree(whereClause); + } else { + warp1Query = psStringCopy("\n"); + } + psFree(warp1Where); + if (psListLength(warp2Where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(warp2Where, NULL); + psStringAppend(&warp2Query, "\n AND %s", whereClause); + psFree(whereClause); + } else { + warp2Query = psStringCopy("\n"); + } + psFree(warp2Where); + + // don't queue for exposures that have already been diff'd unless requested + psString diffQuery = NULL; + if (! (reRun || newTemplates) ) { + psStringAppend(&diffQuery, "\nAND diff_id IS NULL"); + } else { + diffQuery = psStringCopy("\n"); + } + + if (psListLength(stackWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(stackWhere, NULL); + psStringAppend(&stackQuery, "\nAND %s", whereClause); + psFree(whereClause); + } else { + stackQuery = psStringCopy(""); + } + psFree(stackWhere); + + psTrace("difftool", 1, query, warp1Query, warp2Query, diffQuery, expQuery, stackQuery); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, warp1Query, warp2Query, expQuery, diffQuery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + break; + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + break; + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + break; + } + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + // create temporary table + query = pxDataGet("difftool_definewarpstack_temp_create.sql"); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(warp1Query); + psFree(query); + query = NULL; + + psString skycell_query = pxDataGet("difftool_definewarpstack_part2.sql"); + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + long numGood = 0; // Number of good rows added + psS64 last_exp_id = 0; + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Output row from query + bool mdok; // Status of MD lookup + + // Take the first warp for each exposure. + // The list is sorted by exposure id and warp_id and the warps are in descending + // order. + psS64 exp_id = psMetadataLookupS64(&mdok, row, "exp_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "exp_id not found"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (exp_id == last_exp_id) { + continue; + } + last_exp_id = exp_id; + + // clear temporary table + if (!p_psDBRunQuery(config->dbh, "DELETE FROM skycellsToDiff")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 warp_id = psMetadataLookupS64(&mdok, row, "warp_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "warp_id not found --- ignoring row %ld", i); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 skycell_count = psMetadataLookupS64(&mdok, row, "skycell_count"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "skycell_count not found"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psString tess_id = psMetadataLookupStr(&mdok, row, "tess_id"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "tess_id not found"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString filter = psMetadataLookupStr(&mdok, row, "filter"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "filter not found"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString warp_data_group = psMetadataLookupStr(&mdok, row, "data_group"); + if (!mdok) { + psError(PXTOOLS_ERR_PROG, false, "warp data_group not found"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!data_group && warp_data_group) { + data_group = warp_data_group; + } + + if (!p_psDBRunQueryF(config->dbh, skycell_query, stackQuery, warp_id, filter, warp2Query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 num = psDBAffectedRows(config->dbh); + + if (num == 0) { + psTrace("difftool", PS_LOG_INFO, "no skycells with stack found for warp_id %" PRId64, warp_id); + continue; + } + + // CZW force available off. If we're using this very-expensive mode, we need to get the one thing it does correctly. + available = false;; + if (!available && (num != skycell_count)) { + psTrace("difftool", PS_LOG_INFO, "%" PRId64 " skyfiles with stack found for warp_id %" PRId64 + " but need %" PRId64, num, warp_id, skycell_count); + continue; + } + + // ok we've got one create the diffRun + diffRunRow *run = diffRunRowAlloc( + 0, // ID + "reg", // state + workdir, + label, + data_group ? data_group : label, + dist_group, + reduction, + NULL, // dvodb + registered, + tess_id, + bothways, // bothways (default is false) + true, // exposure + 0, // magicked + NULL, // software version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + IPP_DIFF_MODE_WARP_STACK, + note + ); + + if (!diffRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + run->diff_id = psDBLastInsertID(config->dbh); + + psStringAppend(&query, "UPDATE skycellsToDiff SET diff_id = %" PRId64, run->diff_id); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + query = NULL; + psStringAppend(&query, + "INSERT INTO diffInputSkyfile(diff_id, skycell_id, warp1, stack1, warp2, stack2, tess_id) SELECT diff_id, skycell_id, warp1, stack1, warp2, stack2, tess_id from skycellsToDiff"); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + query = NULL; + + if (!setdiffRunState(config, run->diff_id, "new", false)) { + psError(PS_ERR_UNKNOWN, false, "failed to change diffRun.state for diff_id: %" PRId64, + run->diff_id); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psArrayAdd(list, list->n, run); + numGood++; + } + psFree(output); + psFree(warp2Query); + psFree(stackQuery); + psFree(skycell_query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + + if (numGood && !diffRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + psFree(list); + + return true; +} +#endif + +static bool definewarpwarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *selectWhere = psMetadataAlloc(); + psMetadata *insertWhere = psMetadataAlloc(); + + // Restrictions for selecting warps + PXOPT_COPY_S64(config->args, selectWhere, "-warp_id", "inputWarpRun.warp_id", "=="); + PXOPT_COPY_S64(config->args, selectWhere, "-exp_id", "inputRawExp.exp_id", "=="); + PXOPT_COPY_S64(config->args, selectWhere, "-template_exp_id", "templateRawExp.exp_id", "=="); + PXOPT_COPY_S64(config->args, selectWhere, "-template_warp_id", "templateWarpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, selectWhere, "-filter", "inputRawExp.filter", "=="); + PXOPT_COPY_STR(config->args, selectWhere, "-obs_mode", "inputRawExp.obs_mode", "=="); + PXOPT_COPY_STR(config->args, selectWhere, "-obs_mode", "templateRawExp.obs_mode", "=="); + PXOPT_COPY_STR(config->args, selectWhere, "-input_label", "inputWarpRun.label", "=="); + PXOPT_COPY_STR(config->args, selectWhere, "-template_label", "templateWarpRun.label", "=="); + PXOPT_COPY_F32(config->args, selectWhere, "-rotdiff", "ABS(inputRawExp.posang - templateRawExp.posang)", "<="); + PXOPT_COPY_F32(config->args, selectWhere, "-timediff", + "ABS(TIME_TO_SEC(TIMEDIFF(inputRawExp.dateobs, templateRawExp.dateobs)))", "<="); + PXOPT_COPY_F32(config->args, selectWhere, "-mintimediff", + "ABS(TIME_TO_SEC(TIMEDIFF(inputRawExp.dateobs, templateRawExp.dateobs)))", ">="); + + // other where restrictions: + PXOPT_COPY_TIME(config->args, selectWhere, "-dateobs_begin", "inputRawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, selectWhere, "-dateobs_end", "inputRawExp.dateobs", "<="); + PXOPT_COPY_RADEC(config->args, selectWhere, "-ra_min", "inputRawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, selectWhere, "-ra_max", "inputRawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, selectWhere, "-decl_min", "inputRawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, selectWhere, "-decl_max", "inputRawExp.decl", "<"); + PXOPT_COPY_F32(config->args, selectWhere, "-sat_pixel_frac_min", "inputRawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, selectWhere, "-sat_pixel_frac_max", "inputRawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-airmass_min", "inputRawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-airmass_max", "inputRawExp.airmass", "<"); + PXOPT_COPY_F32(config->args, selectWhere, "-exp_time_min", "inputRawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, selectWhere, "-exp_time_max", "inputRawExp.exp_time", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-bg_min", "inputRawExp.bg", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-bg_max", "inputRawExp.bg", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-bg_stdev_min", "inputRawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-bg_stdev_max", "inputRawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-bg_mean_stdev_min", "inputRawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-bg_mean_stdev_max", "inputRawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-alt_min", "inputRawExp.alt", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-alt_max", "inputRawExp.alt", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-az_min", "inputRawExp.az", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-az_max", "inputRawExp.az", "<"); + PXOPT_COPY_F32(config->args, selectWhere, "-ccd_temp_min", "inputRawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, selectWhere, "-ccd_temp_max", "inputRawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, selectWhere, "-posang_min", "inputRawExp.posang", ">="); + PXOPT_COPY_F64(config->args, selectWhere, "-posang_max", "inputRawExp.posang", "<"); + PXOPT_COPY_F32(config->args, selectWhere, "-sun_angle_min", "inputRawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, selectWhere, "-sun_angle_max", "inputRawExp.sun_angle", "<"); + PXOPT_COPY_STR(config->args, selectWhere, "-input_comment", "inputRawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, selectWhere, "-template_comment", "templateRawExp.comment", "LIKE"); + + PXOPT_LOOKUP_BOOL(not_bothways, config->args, "-not-bothways", false); + + // Haversine formula for great circle distance + PXOPT_COPY_F32(config->args, selectWhere, "-distance", + "DEGREES(2*ASIN(SQRT(POW(SIN(inputRawExp.decl - templateRawExp.decl),2) + " + "COS(inputRawExp.decl)*COS(templateRawExp.decl)*" + "POW(SIN(inputRawExp.ra - templateRawExp.ra),2))))", "<="); + + PXOPT_LOOKUP_BOOL(backwards, config->args, "-backwards", false); + + psMetadataAddF32(selectWhere, PS_LIST_TAIL, + "TIME_TO_SEC(TIMEDIFF(inputRawExp.dateobs, templateRawExp.dateobs))", + PS_META_DUPLICATE_OK, backwards ? "<" : ">", 0.0); + + // Restrictions for inserting skycells + PXOPT_COPY_F32(config->args, insertWhere, "-good_frac", "inputWarpSkyfile.good_frac", ">="); + PXOPT_COPY_F32(config->args, insertWhere, "-good_frac", "templateWarpSkyfile.good_frac", ">="); + + // Additional controls + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + + // Settings to apply to defined run + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); // required options + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); // option + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); // option + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); // option + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); // option + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); // option + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(selectWhere); + psFree(insertWhere); + return false; + } + + if (!rerun) { + // Need to build table of exposures with diffs + psString tempCreate = pxDataGet("difftool_definewarpwarp_temp_create.sql"); // Create temp table SQL + if (!tempCreate) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(selectWhere); + psFree(insertWhere); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (!p_psDBRunQuery(config->dbh, tempCreate)) { + psError(PS_ERR_UNKNOWN, false, "Unable to create temp table: %s", tempCreate); + psFree(tempCreate); + psFree(selectWhere); + psFree(insertWhere); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(tempCreate); + + psString tempInsert = pxDataGet("difftool_definewarpwarp_temp_insert.sql"); // Insert to temp table + if (!tempInsert) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(selectWhere); + psFree(insertWhere); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString where = psStringCopy(""); // WHERE for insertion + if (label) { + psStringAppend(&where, "\nAND diffRun.label = '%s'", label); + } + + if (!p_psDBRunQueryF(config->dbh, tempInsert, where, where)) { + psError(PS_ERR_UNKNOWN, false, "Unable to insert into temp table: %s", tempInsert); + psFree(tempInsert); + psFree(where); + psFree(selectWhere); + psFree(insertWhere); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(where); + psFree(tempInsert); + } + + // Get list of warps to diff + psString select = pxDataGet("difftool_definewarpwarp_select.sql"); + if (!select) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(selectWhere); + psFree(insertWhere); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString whereClause = NULL; // The WHERE part of the query + + if (psListLength(selectWhere->list)) { + psString new = psDBGenerateWhereConditionSQL(selectWhere, NULL); + psStringAppend(&whereClause, "\nAND %s", new); + psFree(new); + } + psFree(selectWhere); + + if (!rerun) { + psStringAppend(&whereClause, "\nAND diffs.diff_id IS NULL"); + } + + if (!p_psDBRunQueryF(config->dbh, select, + !rerun ? "\n" : "", // Activate LEFT JOIN against diffs? + whereClause)) { + psError(PS_ERR_UNKNOWN, false, "Unable to run query: %s [WITH] %s", select, whereClause); + psFree(select); + psFree(whereClause); + psFree(insertWhere); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(select); + psFree(whereClause); + + psArray *results = p_psDBFetchResult(config->dbh); // Results of query + if (!results) { + psErrorCode err = psErrorCodeLast(); // Code for error + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + break; + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + break; + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + break; + } + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(insertWhere); + return false; + } + if (!psArrayLength(results)) { + psTrace("difftool", 1, "no rows found"); + psFree(results); + psFree(insertWhere); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, results, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(results); + return false; + } + psFree(results); + return true; + } + + psString insert = pxDataGet("difftool_definewarpwarp_insert.sql"); // Insertion for each new run + + if (psListLength(insertWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(insertWhere, NULL); + psStringAppend(&insert, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(insertWhere); + + psArray *list = psArrayAllocEmpty(16); // List of runs defined, to print + long numGood = 0; // Number of good rows added + for (long i = 0; i < results->n; i++) { + if (limit && numGood >= limit) { + break; + } + psMetadata *row = results->data[i]; // Result row from query + + psS64 input_id = psMetadataLookupS64(NULL, row, "input_warp_id"); + const char *template = psMetadataLookupStr(NULL, row, "template_warp_id"); + const char *tess_id = psMetadataLookupStr(NULL, row, "tess_id"); + psString input_data_group = psMetadataLookupStr(NULL, row, "input_data_group"); + if (!data_group) { + data_group = input_data_group; + } + + if (!input_id || !template || !tess_id) { + psError(PXTOOLS_ERR_PROG, false, "Identifiers not found"); + psFree(list); + psFree(insert); + psFree(results); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + diffRunRow *run = diffRunRowAlloc(0, + "reg", + workdir, + label, + data_group ? data_group : label, + dist_group, + reduction, + NULL, // dvodb + registered, + tess_id, + !not_bothways, // bothways (default is true, ie not_bothways is false) + true, // exposure + false, // magicked + NULL, // software version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + IPP_DIFF_MODE_WARP_WARP, + note); // Run to insert + if (!diffRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + psFree(list); + psFree(insert); + psFree(results); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + run->diff_id = psDBLastInsertID(config->dbh); // Difference run identifier + + // Convert identifiers to string, for insertion into query + psString diff = NULL, input = NULL; // String versions of identifiers + psStringAppend(&diff, "%" PRId64, run->diff_id); + psStringAppend(&input, "%" PRId64, input_id); + + if (!p_psDBRunQueryF(config->dbh, insert, diff, input, template, input, template)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(input); + psFree(diff); + psFree(run); + psFree(list); + psFree(insert); + psFree(results); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 numSkycells = psDBAffectedRows(config->dbh); + if (numSkycells > 0) { + if (!setdiffRunState(config, run->diff_id, "new", false)) { + psError(PS_ERR_UNKNOWN, false, "Failed to change diffRun.state for diff_id: %" PRId64, + run->diff_id); + psFree(input); + psFree(diff); + psFree(run); + psFree(list); + psFree(insert); + psFree(results); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + run->state = psStringCopy("new"); + } else { + // No overlap between the warps. Insert a dummy skycell which will complete this pair of warps + // XXX: we'd like to use diffInputSkyfileRowAlloc and diffInputSkyfileInsert but there doesn't + // seem to be a way to pass in NULL for skycell_id, stack1 and stack2 + psString dummyQuery = NULL; + psStringAppend(&dummyQuery, + "INSERT INTO diffInputSkyfile VALUES(%" PRId64 ", NULL, %s, NULL, %s, NULL, '%s', 0)", + run->diff_id, + input, // warp1 + template, // warp2 + tess_id); + if (!p_psDBRunQuery(config->dbh, dummyQuery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(dummyQuery); + psFree(input); + psFree(diff); + psFree(run); + psFree(list); + psFree(insert); + psFree(results); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(dummyQuery); + psString finishQuery = NULL; + psStringAppend( &finishQuery, + "UPDATE diffRun set state ='full', dist_group = NULL, note = 'dummy run - no overlap' WHERE diff_id = %" PRId64, + run->diff_id); + if (!p_psDBRunQuery(config->dbh, finishQuery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(finishQuery); + psFree(input); + psFree(diff); + psFree(run); + psFree(list); + psFree(insert); + psFree(results); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(finishQuery); + run->state = psStringCopy("full"); + } + psFree(input); + psFree(diff); + + psArrayAdd(list, list->n, run); + psFree(run); // Drop reference + + numGood++; + } + psFree(insert); + psFree(results); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + + if (!diffRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + psFree(list); + + return true; +} + +static bool definestackstackMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *stack1Where = psMetadataAlloc(); + psMetadata *stack2Where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, stack1Where, "-tess_id", "stackRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, stack2Where, "-tess_id", "stackRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, stack1Where, "-filter", "stackRun.filter", "=="); + PXOPT_COPY_STR(config->args, stack2Where, "-filter", "stackRun.filter", "=="); + PXOPT_COPY_STR(config->args, stack1Where, "-skycell_id", "stackRun.skycell_id", "=="); + PXOPT_COPY_STR(config->args, stack2Where, "-skycell_id", "stackRun.skycell_id", "=="); + PXOPT_COPY_STR(config->args, stack1Where, "-input_label", "stackRun.label","=="); + PXOPT_COPY_STR(config->args, stack1Where, "-input_data_group", "stackRun.data_group","=="); + PXOPT_COPY_STR(config->args, stack2Where, "-template_label", "stackRun.label","=="); + + PXOPT_COPY_STR(config->args, stack1Where, "-input_stack_id", "stackRun.stack_id","=="); + PXOPT_COPY_STR(config->args, stack2Where, "-template_stack_id", "stackRun.stack_id","=="); + + PXOPT_COPY_F32(config->args, stack1Where, "-good_frac", "stackSumSkyfile.good_frac", ">="); + PXOPT_COPY_F32(config->args, stack2Where, "-good_frac", "stackSumSkyfile.good_frac", ">="); + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); // required option + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); // option + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); // option + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + PXOPT_LOOKUP_BOOL(bothways, config->args, "-bothways", false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(reRun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(newTemplates,config->args,"-new-templates", false); + + // PXOPT_LOOKUP_BOOL(available, config->args, "-available", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + if (!(label)) { + PXOPT_LOOKUP_STR(label,config->args, "-input_label",true,false); + } + + // Organize the config information into queries. + psString stack1Query = NULL; + psString stack2Query = NULL; + + if (psListLength(stack1Where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(stack1Where, NULL); + psStringAppend(&stack1Query, "AND %s", whereClause); + psFree(whereClause); + } + psFree(stack1Where); + + if (psListLength(stack2Where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(stack2Where, NULL); + psStringAppend(&stack2Query, "AND %s", whereClause); + psFree(whereClause); + } + psFree(stack2Where); + + // don't queue for stacks that have already been diffed unless requested + psString diffQuery0 = NULL; + psString diffQuery1 = NULL; + if (! (reRun || newTemplates) ) { + psStringAppend(&diffQuery0, "AND diffExp.diff_id IS NULL"); + psStringAppend(&diffQuery1, "HAVING n_diff = 0"); + } + + // find the distinct set of data_groups and filters + psString query = pxDataGet("difftool_definestackstack_part0.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + psTrace("difftool",1,query,stack2Query,stack1Query,diffQuery0,stack1Query); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + psStringSubstitute(&query, stack1Query, "@STACK1_QUERY@"); + psStringSubstitute(&query, stack2Query, "@STACK2_QUERY@"); + psStringSubstitute(&query, diffQuery0, "@DIFF0_QUERY@"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + break; + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + break; + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + break; + } + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRunTemp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + psArray *list = psArrayAllocEmpty(16); // List of runs defined, to print + for (long i = 0; i < output->n; i++) { + query = pxDataGet("difftool_definestackstack_part1.sql"); + + + psMetadata *row = output->data[i]; // Output row from query + bool mdok; // Status of MD lookup + + // psMetadataPrint(stderr, row, 1); + + psString tess_id = psMetadataLookupStr(&mdok,row,"INPUT_tess_id"); + psString this_data_group = psMetadataLookupStr(&mdok,row,"INPUT_data_group"); + psString this_dist_group = psMetadataLookupStr(&mdok,row,"INPUT_dist_group"); + psString this_label = psMetadataLookupStr(&mdok,row,"INPUT_label"); + + psString this_stack1Query = psStringCopy(stack1Query); + + psString thisWhere = psDBGenerateWhereConditionSQL(row,NULL); + psStringSubstitute(&thisWhere,"stackRun.","INPUT_"); + psStringAppend(&this_stack1Query,"AND %s", thisWhere); + psFree(thisWhere); + + psStringSubstitute(&query, this_stack1Query, "@STACK1_QUERY@"); + psStringSubstitute(&query, stack2Query, "@STACK2_QUERY@"); + psStringSubstitute(&query, diffQuery1, "@DIFF1_QUERY@"); + + psTrace("difftool", 1, "%s", query); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(this_stack1Query); + + psArray *diff_id_output = p_psDBFetchResult(config->dbh); + if (!diff_id_output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + break; + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + break; + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + break; + } + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psArrayLength(diff_id_output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(diff_id_output); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + + // ok we've got one create the diffRun + diffRunRow *run = diffRunRowAlloc( + 0, // ID + "reg", // state + workdir, + label ? label : this_label, + data_group ? data_group : this_data_group, + dist_group ? dist_group : this_dist_group, + reduction, + NULL, // dvodb + registered, + tess_id, + bothways, // bothways + false, // exposure + 0, // magicked + NULL, // software version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + IPP_DIFF_MODE_STACK_STACK, // diff_mode + note + ); + // Commit to database + if (!diffRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + // diffRunPrintObject(stdout,run,1); + run->diff_id = psDBLastInsertID(config->dbh); + + for (long j = 0; j < diff_id_output->n; j++) { + psMetadata *input_row = diff_id_output->data[j]; // Output row from query + bool mdok; + psString in_skycell_id = psMetadataLookupStr(&mdok,input_row,"skycell_id"); + psS64 in_input_stack_id = psMetadataLookupS64(&mdok,input_row,"stack_id"); + psS64 in_template_stack_id = psMetadataLookupS64(&mdok,input_row,"max_stack_id"); + psString in_tess_id = psMetadataLookupStr(&mdok,input_row,"tess_id"); + psTrace("difftool",1,"%s %" PRId64 " %" PRId64 " %s\n", + in_skycell_id, in_input_stack_id, in_template_stack_id, in_tess_id); + diffInputSkyfileRow *input = diffInputSkyfileRowAlloc( + run->diff_id, // ID + in_skycell_id, + PS_MAX_S64, // warp1_id -> NULL + in_input_stack_id, // stack1 + PS_MAX_S64, // warp2_id -> NULL + in_template_stack_id, // stack2 + in_tess_id, + 0 // diff_skyfile_id + ); + + // Commit to database the input + if (!diffInputSkyfileInsertObject(config->dbh, input)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(input); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + +/* diffInputSkyfilePrintObject(stdout,input,1); */ + psFree(input); + } + + if (!setdiffRunState(config, run->diff_id, "new", false)) { + psError(PS_ERR_UNKNOWN, false, "failed to change diffRun.state for diff_id: %" PRId64, run->diff_id); + psFree(stack1Query); + psFree(stack2Query); + psFree(diffQuery0); + psFree(diffQuery1); + psFree(run); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psArrayAdd(list, list->n, run); + psFree(run); + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + + if (!diffRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + + psFree(query); + psFree(stack1Query); + psFree(stack2Query); + psFree(diffQuery0); + psFree(diffQuery1); + psFree(output); + psFree(list); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + + return(true); +} + +static bool tosummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "diffSkyfile.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "diffSkyfile.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "diffRun.state", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "diffSkyfile.magicked", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "diffRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("difftool_tosummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // generate where strings for arguments that require extra processing + // beyond PXOPT_COPY* + psString where2 = NULL; + if (!pxmagicAddWhere(config, &where2, "diffSkyfile")) { + psError(psErrorCodeLast(), false, "pxMagicAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else if (!all && !where2) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + if (where2) { + if (psListLength(where->list)) { + psStringAppend(&query, " %s", where2); + } else { + psStringAppend(&query, " AND 1 %s", where2); + } + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return(true); +} +static bool addsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", true, false); + PXOPT_LOOKUP_STR(projection_cell, config->args, "-projection_cell", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + psString query = pxDataGet("difftool_addsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + if (!p_psDBRunQueryF(config->dbh, query, diff_id, projection_cell, path_base)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psS64 numUpdated = psDBAffectedRows(config->dbh); + + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + psFree(query); + return false; + } + + psFree(query); + + // Print anything here? + + return(true); +} + + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "=="); + + psString query = pxDataGet("difftool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + // pxDataGet/psStringSubstitute workaround + psString queryCopy = psStringCopy(query); + psFree(query); + query = queryCopy; + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereClause, " AND "); + psStringSubstitute(&query,whereClause,"@INNERCONSTRAINT@"); + psFree(whereClause); + } + else { + psStringSubstitute(&query,NULL,"@INNERCONSTRAINT@"); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psStringSubstitute(&query,limitString,"@INNERLIMITS@"); + psFree(limitString); + + } + else { + psStringSubstitute(&query,NULL,"@INNERLIMITS@"); + } + // fprintf(stderr,"%s",query); + + //fprintf(stderr,"%s",query); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffPendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool pendingcleanupskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", false, false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + if (diff_id) { + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + } + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "=="); + + char * sql_file = all ? "difftool_pendingcleanupskyfile_all.sql": "difftool_pendingcleanupskyfile.sql" ; + psString query = pxDataGet(sql_file); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffPendingCleanupSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + return true; +} + + +static bool revertcleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffSkyfile.diff_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "diffRun.data_group", "LIKE"); + + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + if (!state) { + state = "error_cleaned"; + } + char *newState = NULL; + if (!strcmp(state, "error_cleaned")) { + newState = "goto_cleaned"; + } else if (!strcmp(state, "error_purged")) { + newState = "goto_purged"; + } else if (!strcmp(state, "error_scrubbed")) { + newState = "goto_scrubbed"; + } else { + psError(PXTOOLS_ERR_CONFIG, true, "-state must be either error_cleaned, error_purged, or error_scrubbed"); + return false; + } + + psString query = pxDataGet("difftool_revertcleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, newState, state, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numDeleted = psDBAffectedRows(config->dbh); + + psLogMsg("difftool", PS_LOG_INFO, "Reverted %d diffRuns and diffSkyfiles", numDeleted); + + return true; +} +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psString query = pxDataGet("difftool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + return true; +} + +static bool updatediffskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + if (!state) { + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + if (!pxSetFaultCode(config->dbh, "diffSkyfile", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + } else { + if (strcmp(state,"error_cleaned") == 0) { + change_skyfile_data_state(config,"error_cleaned", "goto_cleaned"); + } + else if (strcmp(state, "error_scrubbed") == 0) { + change_skyfile_data_state(config,"error_scrubbed", "goto_scrubbed"); + } + else if (strcmp(state, "error_purged") == 0) { + change_skyfile_data_state(config,"error_purged", "goto_purged"); + } + else { + psError(PS_ERR_UNKNOWN, false, "unhandled state given"); + return(false); + } + } + + return true; +} + +static bool change_skyfile_data_state(pxConfig *config, psString data_state, psString run_state) { + PS_ASSERT_PTR_NON_NULL(config, false); + + // diff_id, skycell_id are required + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + + + psS64 magicked = 0; + if (!strcmp(data_state, "full")) { + PXOPT_LOOKUP_S64(set_magicked, config->args, "-magicked", 0, false); + magicked = set_magicked; + } + + psString query = pxDataGet("difftool_change_skyfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + psString set_magicked_skyfile = psStringCopy(""); + psString set_magicked_run = psStringCopy(""); + if (magicked != 0 && !strcmp(data_state, "full")) { + psStringAppend(&set_magicked_skyfile, "\n , diffSkyfile.magicked = %" PRId64, magicked); + psStringAppend(&set_magicked_run, "\n , diffRun.magicked = %" PRId64, magicked); + + } else if (!strcmp(data_state, "cleaned") || !strcmp(data_state, "purged")) { + psStringAppend(&set_magicked_skyfile, "\n, diffSkyfile.magicked = IF(diffSkyfile.magicked = 0, 0, -1)"); + psStringAppend(&set_magicked_run, "\n, diffRun.magicked = IF(diffRun.magicked = 0, 0, -1)"); + } + + // Uses the unconstrained (diffRun.state [NEED NOT EQUAL] run_state) version from warptool.c + + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_skyfile, diff_id, skycell_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(set_magicked_skyfile); + return(false); + } + psFree(set_magicked_skyfile); + psFree(query); + + query = pxDataGet("difftool_change_run_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_run, diff_id, data_state)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(set_magicked_run); + return(false); + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(set_magicked_run); + return(false); + } + psFree(set_magicked_run); + + return(true); +} + +static bool tocleanedskyfileMode(pxConfig *config) { + return change_skyfile_data_state(config, "cleaned","goto_cleaned"); +} +static bool topurgedskyfileMode(pxConfig *config) { + return change_skyfile_data_state(config, "purged", "goto_purged"); +} +static bool toscrubbedskyfileMode(pxConfig *config) { + return change_skyfile_data_state(config, "scrubbed", "goto_scrubbed"); +} +static bool tofullskyfileMode(pxConfig *config) { + return change_skyfile_data_state(config, "full", "update"); +} + + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportFiles = 3; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-diff_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + + ExportTable tables [] = { + {"diffRun", "difftool_export_run.sql"}, + {"diffInputSkyfile", "difftool_export_input_skyfile.sql"}, + {"diffSkyfile", "difftool_export_skyfile.sql"}, + }; + + for (int i=0; i < numExportFiles; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + bool success = true; + if (!strcmp(tables[i].tableName, "diffRun")) { + success = pxSetStateCleaned("diffRun", "state", output); +#ifdef notyet + // diffSkyfile doesn't have dataState yet + } else if (!strcmp(tables[i].tableName, "diffSkyfile")) { + success = pxSetStateCleaned("diffSkyfile", "data_state", output); +#endif + } + if (!success) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + int numImportTables = 2; + + char tables[2] [80] = {"diffInputSkyfile", "diffSkyfile"}; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + +#ifdef notdef + fprintf (stderr, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); +#endif + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + psMetadataItem *item = psMetadataLookup (input, "diffRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + diffRunRow *diffRun = diffRunObjectFromMetadata (entry->data.md); + diffRunInsertObject (config->dbh, diffRun); + + // fprintf (stdout, "---- diff run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + for (int i = 0; i < numImportTables; i++) { + item = psMetadataLookup (input, tables[i]); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + switch (i) { + case 0: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + diffInputSkyfileRow *diffInputSkyfile = diffInputSkyfileObjectFromMetadata (entry->data.md); + diffInputSkyfileInsertObject (config->dbh, diffInputSkyfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 1: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + diffSkyfileRow *diffSkyfile = diffSkyfileObjectFromMetadata (entry->data.md); + diffSkyfileInsertObject (config->dbh, diffSkyfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + } + } + + return true; +} + +static bool listrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "diffRun.tess_id", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "diffRun.magicked", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "diffRun.state", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "diffRun.data_group", "LIKE"); + pxAddLabelSearchArgs (config, where, "-dist_group", "diffRun.dist_group", "LIKE"); + + PXOPT_COPY_S64(config->args, where, "-template_exp_id", "rawTemplate.exp_id", "=="); + + PXOPT_LOOKUP_BOOL(template, config->args, "-template", false); + if (!template) { + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawInput.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawInput.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-warp_id", "warpInput.warp_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawInput.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawInput.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawInput.filter", "LIKE"); + } else { + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawTemplate.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawTemplate.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-warp_id", "warpTemplate.warp_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawTemplate.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawTemplate.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawTemplate.filter", "LIKE"); + } + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + PXOPT_LOOKUP_S16(diff_mode, config->args, "-diff_mode", false, false); + if (diff_mode) { + PXOPT_COPY_S16(config->args, where, "-diff_mode", "diffRun.diff_mode", "=="); + } + + psString where2 = NULL; + pxmagicAddWhere(config, &where2, "diffRun"); + pxspaceAddWhere(config, &where2, template ? "rawTemplate" : "rawInput"); + psString query = pxDataGet("difftool_listrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (where2) { + psStringAppend(&query, " WHERE diffRun.diff_id is not null %s", where2); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + psFree(where); + + if (pstamp_order) { + if (template) { + psStringAppend(&query, " ORDER BY rawTemplate.exp_id, diff_id DESC"); + } else { + psStringAppend(&query, " ORDER BY rawInput.exp_id, diff_id DESC"); + } + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +static bool listssrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackInput.stack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-template_stack_id", "stackTemplate.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "diffRun.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "diffRun.state", "=="); + pxAddLabelSearchArgs (config, where, "-label", "diffRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "diffRun.data_group", "LIKE"); + pxAddLabelSearchArgs (config, where, "-dist_group", "diffRun.dist_group", "LIKE"); + + // lookup these so we don't compare to zero if they are not supplied + PXOPT_LOOKUP_F64(mjd_obs_begin, config->args, "-mjd_obs_begin", false, false); + PXOPT_LOOKUP_F64(mjd_obs_end, config->args, "-mjd_obs_end", false, false); + + PXOPT_LOOKUP_BOOL(template, config->args, "-template", false); + + if (!template) { + if (mjd_obs_begin) { + PXOPT_COPY_F64(config->args, where, "-mjd_obs_begin", "stackInput.mjd_obs", ">="); + } + if (mjd_obs_end) { + PXOPT_COPY_F64(config->args, where, "-mjd_obs_end", "stackInput.mjd_obs", "<="); + } + PXOPT_COPY_STR(config->args, where, "-filter", "stackInputRun.filter", "LIKE"); + } else { + if (mjd_obs_begin) { + PXOPT_COPY_F64(config->args, where, "-mjd_obs_begin", "stackTemplate.mjd_obs", ">="); + } + if (mjd_obs_end) { + PXOPT_COPY_F64(config->args, where, "-mjd_obs_end", "stackTemplate.mjd_obs", "<="); + } + PXOPT_COPY_STR(config->args, where, "-filter", "stackTemplateRun.filter", "LIKE"); + } + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + + psString where2 = NULL; + psString query = pxDataGet("difftool_listssrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else if (where2) { + psStringAppend(&query, " AND %s", where2); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + psFree(where); + + if (pstamp_order) { + if (template) { + psStringAppend(&query, " ORDER BY stackTemplate.stack_id, diff_id DESC"); + } else { + psStringAppend(&query, " ORDER BY stackInput.stack_id, diff_id DESC"); + } + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("difftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +// a very specfic function to queue a cleaned warpSkyfile to be updated +static bool setskyfiletoupdateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + + psString query = pxDataGet("difftool_setskyfiletoupdate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString setHook = psStringCopy(""); + if (label) { + psStringAppend(&setHook, "\n , diffRun.label = '%s'", label); + } + + if (skycell_id) { + psStringAppend(&query, " AND (diffSkyfile.skycell_id = '%s')", skycell_id); + } + // we do not update components with the magic fault value. They are non-updateable + // (But can be recovered with "difftool -revertwarped -fault 26" (PXTOOL_DO_NOT_REVERT_FAULT) + psStringAppend(&query, " AND (diffSkyfile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + + if (!p_psDBRunQueryF(config->dbh, query, setHook, diff_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(setHook); + psFree(query); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/difftool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/difftool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/difftool.h (revision 41104) @@ -0,0 +1,61 @@ +/* + * difftool.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef DIFFTOOL_H +#define DIFFTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + DIFFTOOL_MODE_NONE = 0x0, + DIFFTOOL_MODE_DEFINERUN, + DIFFTOOL_MODE_UPDATERUN, + DIFFTOOL_MODE_ADDINPUTSKYFILE, + DIFFTOOL_MODE_INPUTSKYFILE, + DIFFTOOL_MODE_TODIFFSKYFILE, + DIFFTOOL_MODE_ADDDIFFSKYFILE, + DIFFTOOL_MODE_ADVANCE, + DIFFTOOL_MODE_TOSUMMARY, + DIFFTOOL_MODE_ADDSUMMARY, + DIFFTOOL_MODE_DIFFSKYFILE, + DIFFTOOL_MODE_REVERTDIFFSKYFILE, + DIFFTOOL_MODE_DEFINEPOPRUN, + DIFFTOOL_MODE_DEFINEWARPSTACK, + DIFFTOOL_MODE_DEFINEWARPWARP, + DIFFTOOL_MODE_DEFINESTACKSTACK, + DIFFTOOL_MODE_PENDINGCLEANUPRUN, + DIFFTOOL_MODE_PENDINGCLEANUPSKYFILE, + DIFFTOOL_MODE_REVERTCLEANUP, + DIFFTOOL_MODE_DONECLEANUP, + DIFFTOOL_MODE_UPDATEDIFFSKYFILE, + DIFFTOOL_MODE_EXPORTRUN, + DIFFTOOL_MODE_IMPORTRUN, + DIFFTOOL_MODE_TOCLEANEDSKYFILE, + DIFFTOOL_MODE_TOPURGEDSKYFILE, + DIFFTOOL_MODE_TOSCRUBBEDSKYFILE, + DIFFTOOL_MODE_TOFULLSKYFILE, + DIFFTOOL_MODE_LISTRUN, + DIFFTOOL_MODE_LISTSSRUN, + DIFFTOOL_MODE_SETSKYFILETOUPDATE, + // DIFFTOOL_MODE_DEFINEWARPSTACKOLDMETHOD, +} difftoolMode; + +pxConfig *difftoolConfig(pxConfig *config, int argc, char **argv); + +#endif // DIFFTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/difftoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/difftoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/difftoolConfig.c (revision 41104) @@ -0,0 +1,560 @@ +/* + * difftoolConfig.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "difftool.h" + +pxConfig *difftoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-tess_id", 0, "define tessellation ID (required)", NULL); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-bothways", 0, "do the subtraction both ways?", false); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-exposure", 0, "subtraction for entire exposure?", false); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddS16(definerunArgs, PS_LIST_TAIL, "-set_diff_mode", 0, "specify type of difference (WW=1,WS=2,SW=3,SS=4)", 0); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-diff_id", 0, "define diff ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "define by label instead of diff ID (LIKE comparison)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "define by data_group instead of diff ID (LIKE comparison)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-dist_group", 0, "define by dist_group instead of diff ID (LIKE comparison)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-set_diff_mode", 0, "specify type of difference (WW=1,WS=2,SW=3,SS=4)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + pxmagicAddArguments(updaterunArgs); + + // -addinputskyfile + psMetadata *addinputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "define diff ID (required)", 0); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "define stack ID", 0); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "define warp ID", 0); + psMetadataAddBool(addinputskyfileArgs, PS_LIST_TAIL, "-template", 0, "this sky cell file is the subtrahend", false); + + // -inputskyfile + psMetadata *inputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(inputskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddS64(inputskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddStr(inputskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(inputskyfileArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID", NULL); + psMetadataAddBool(inputskyfileArgs, PS_LIST_TAIL, "-template", 0, "find only subtrahend", false); + psMetadataAddBool(inputskyfileArgs, PS_LIST_TAIL, "-input", 0, "find only minuend", false); + psMetadataAddU64(inputskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(inputskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -todiffskyfile + psMetadata *todiffskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(todiffskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(todiffskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddU64(todiffskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(todiffskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -adddiffskyfile + psMetadata *adddiffskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(adddiffskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "define warp ID (required)", 0); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell of file (required)", 0); + psMetadataAddS16(adddiffskyfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(adddiffskyfileArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", 0); + psMetadataAddF64(adddiffskyfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(adddiffskyfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background mean stdev", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-dtime_diff", 0, "define elapsed processing time", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-dtime_match", 0, "define match processing time", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-dtime_phot", 0, "define photometry processing time", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddS32(adddiffskyfileArgs, PS_LIST_TAIL, "-stamps_num", 0, "define subtraction stamp number", 0); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-stamps_mean", 0, "define subtraction stamp mean", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-stamps_rms", 0, "define subtraction stamp rms", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-norm", 0, "define normalisation", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-bg_diff", 0, "define background difference", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-kernel_x", 0, "define kernel x moment", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-kernel_y", 0, "define kernel y moment", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-kernel_xx", 0, "define kernel xx moment", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-kernel_xy", 0, "define kernel xy moment", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-kernel_yy", 0, "define kernel yy moment", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-deconv_max", 0, "define maximum deconvolution fraction", NAN); + psMetadataAddS32(adddiffskyfileArgs, PS_LIST_TAIL, "-sources", 0, "define number of sources", 0); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-good_frac", 0, "define %% of good pixels", NAN); + psMetadataAddS64(adddiffskyfileArgs, PS_LIST_TAIL, "-magicked", 0, "define magicked state", 0); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-ver_ppsub", 0, "define ppSub version", NULL); + psMetadataAddStr(adddiffskyfileArgs, PS_LIST_TAIL, "-ver_streaks", 0, "define streaksremove version", NULL); + psMetadataAddS32(adddiffskyfileArgs, PS_LIST_TAIL, "-maskfrac_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-maskfrac_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-maskfrac_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-maskfrac_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(adddiffskyfileArgs, PS_LIST_TAIL, "-maskfrac_advisory", 0, "define advisory mask fraction", NAN); + + // -advance + psMetadata *advanceArgs = psMetadataAlloc(); + psMetadataAddS64(advanceArgs, PS_LIST_TAIL, "-diff_id", 0, "select by diff ID", 0); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "select by label", NULL); + psMetadataAddS32(advanceArgs, PS_LIST_TAIL, "-limit", 0, "limit number of results", 0); + + // -diffskyfile + psMetadata *diffskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(diffskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(diffskyfileArgs , PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddS64(diffskyfileArgs, PS_LIST_TAIL, "-diff_skyfile_id", 0, "search by diff_skyfile_id ID", 0); + psMetadataAddStr(diffskyfileArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(diffskyfileArgs , PS_LIST_TAIL, "-warp_id", 0, "search by warp_id", NULL); + psMetadataAddBool(diffskyfileArgs, PS_LIST_TAIL, "-template", 0, "apply exposure args to template of bothways diff", false); + psMetadataAddS64(diffskyfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(diffskyfileArgs , PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", NULL); + psMetadataAddTime(diffskyfileArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(diffskyfileArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(diffskyfileArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddStr(diffskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by diffRun label (LIKE comparison)", NULL); + psMetadataAddStr(diffskyfileArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by diffRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(diffskyfileArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by diffRun dist_group (LIKE comparison)", NULL); + psMetadataAddS16(diffskyfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + pxmagicAddArguments(diffskyfileArgs); + pxspaceAddArguments(diffskyfileArgs); + + psMetadataAddBool(diffskyfileArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + psMetadataAddBool(diffskyfileArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(diffskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(diffskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs , PS_LIST_TAIL, "-warp_id", 0, "search by warp_id", NULL); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-template", 0, "apply exposure args to template of bothways diff", false); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(listrunArgs , PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", NULL); + psMetadataAddTime(listrunArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(listrunArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-template_exp_id", 0, "search by exposure ID of template", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by diffRun label (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by diffRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by diffRun dist_group (LIKE comparison)", NULL); + psMetadataAddS16(listrunArgs, PS_LIST_TAIL, "-diff_mode", 0, "search for diff_mode", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + pxmagicAddArguments(listrunArgs); + pxspaceAddArguments(listrunArgs); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(listrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + + // -listssrun + psMetadata *listssrunArgs = psMetadataAlloc(); + psMetadataAddS64(listssrunArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(listssrunArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID (LIKE comparison)", NULL); + psMetadataAddS64(listssrunArgs , PS_LIST_TAIL, "-stack_id", 0, "search by input stack_id", 0); + psMetadataAddS64(listssrunArgs , PS_LIST_TAIL, "-template_stack_id",0, "search by template stack_id", 0); + psMetadataAddBool(listssrunArgs, PS_LIST_TAIL, "-template", 0, "apply stack selectors to template", false); + psMetadataAddF64(listssrunArgs, PS_LIST_TAIL, "-mjd_obs_begin", 0, "search by stack MJD-OBS (>=)", 0); + psMetadataAddF64(listssrunArgs, PS_LIST_TAIL, "-mjd_obs_end", 0, "search by stack MJD-OBS(<=)", 0); + psMetadataAddStr(listssrunArgs, PS_LIST_TAIL, "-filter", 0, "search by stack filter", NULL); + psMetadataAddStr(listssrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by diffRun label (LIKE comparison)", NULL); + psMetadataAddStr(listssrunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by diffRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(listssrunArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by diffRun dist_group (LIKE comparison)", NULL); + psMetadataAddS16(listssrunArgs, PS_LIST_TAIL, "-diff_mode", 0, "search for diff_mode", 0); + psMetadataAddStr(listssrunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + pxmagicAddArguments(listssrunArgs); + pxspaceAddArguments(listssrunArgs); + + psMetadataAddBool(listssrunArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + + psMetadataAddBool(listssrunArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(listssrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listssrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + + // -revertdiffskyfile + psMetadata *revertdiffskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertdiffskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(revertdiffskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + psMetadataAddStr(revertdiffskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS16(revertdiffskyfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertdiffskyfileArgs, PS_LIST_TAIL, "-all", 0, "allow no search terms", 0); + + // -definepoprun + psMetadata *definepoprunArgs = psMetadataAlloc(); + psMetadataAddStr(definepoprunArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddTime(definepoprunArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definepoprunArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell ID (required)", NULL); + psMetadataAddStr(definepoprunArgs, PS_LIST_TAIL, "-tess_id", 0, "define tessellation ID (required)", NULL); + psMetadataAddStr(definepoprunArgs, PS_LIST_TAIL, "-label", 0, "define label", NULL); + psMetadataAddStr(definepoprunArgs, PS_LIST_TAIL, "-reduction", 0, "define reduction class", NULL); + psMetadataAddBool(definepoprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddS64(definepoprunArgs, PS_LIST_TAIL, "-template_warp_id", 0, "define warp ID for template", 0); + psMetadataAddS64(definepoprunArgs, PS_LIST_TAIL, "-template_stack_id", 0, "define stack ID for template", 0); + psMetadataAddS64(definepoprunArgs, PS_LIST_TAIL, "-input_warp_id", 0, "define warp ID for input", 0); + psMetadataAddS64(definepoprunArgs, PS_LIST_TAIL, "-input_stack_id", 0, "define stack ID for input", 0); + + // -definewarpstack + psMetadata *definewarpstackArgs = psMetadataAlloc(); + psMetadataAddS64(definewarpstackArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(definewarpstackArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID", NULL); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-bothways", 0, "do the subtraction both ways?", false); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-comment", 0, "search by comment (LIKE)", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-stack_label", 0, "search by stack label", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-stack_data_group", 0, "search by stack data_group", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-warp_label", 0, "search by warp label", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group", NULL); + psMetadataAddF32(definewarpstackArgs, PS_LIST_TAIL, "-good_frac", 0, "minimum good fraction of skycell", NAN); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddTime(definewarpstackArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definewarpstackArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-new-templates", 0, "also search for diffs with new template", false); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-rerun", 0, "define new run even if one exists", false); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-available", 0, "define new run even if no stacks available for some skycells", false); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-pretend", 0, "list results but to not queue", false); + psMetadataAddBool(definewarpstackArgs, PS_LIST_TAIL, "-lap_query", 0, "Use the updated LAP style SQL code", false); + + pxspaceBoxAddArguments(definewarpstackArgs); + +#if (0) + // -definewarpstackOldMethod + psMetadata *definewarpstackOldMethodArgs = psMetadataAlloc(); + psMetadataAddS64(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID", NULL); + psMetadataAddBool(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-bothways", 0, "do the subtraction both ways?", false); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-comment", 0, "search by comment (LIKE)", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-stack_label", 0, "search by stack label", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-stack_data_group", 0, "search by stack data_group", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-warp_label", 0, "search by warp label", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group", NULL); + psMetadataAddF32(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-good_frac", 0, "minimum good fraction of skycell", NAN); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddTime(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-new-templates", 0, "also search for diffs with new template", false); + psMetadataAddBool(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-rerun", 0, "define new run even if one exists", false); + psMetadataAddBool(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-available", 0, "define new run even if no stacks available for some skycells", false); + psMetadataAddBool(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definewarpstackOldMethodArgs, PS_LIST_TAIL, "-pretend", 0, "list results but to not queue", false); + + pxspaceBoxAddArguments(definewarpstackOldMethodArgs); +#endif + + // -definewarpwarp + psMetadata *definewarpwarpArgs = psMetadataAlloc(); + psMetadataAddS64(definewarpwarpArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(definewarpwarpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddBool(definewarpwarpArgs, PS_LIST_TAIL, "-not-bothways", 0, "only do the single-direction subtraction?", false); + psMetadataAddS64(definewarpwarpArgs, PS_LIST_TAIL, "-template_exp_id", 0, "search by template exposure ID", 0); + psMetadataAddS64(definewarpwarpArgs, PS_LIST_TAIL, "-template_warp_id", 0, "search by template warp ID", 0); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-distance", 0, "limit distance between input and template (deg)", NAN); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by observation mode", NULL); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-timediff", 0, "limit time difference between input and template", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-mintimediff", 0, "limit time difference between input and template to be greater than this", NAN); + psMetadataAddBool(definewarpwarpArgs, PS_LIST_TAIL, "-backwards", 0, "Template comes after input?", false); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-rotdiff", 0, "limit rotator difference between input and template", NAN); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-input_label", 0, "search by warp label for input", NULL); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-template_label", 0, "search by warp label for template", NULL); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-good_frac", 0, "minimum good fraction of skycell", NAN); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddTime(definewarpwarpArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definewarpwarpArgs, PS_LIST_TAIL, "-rerun", 0, "define new run even if one exists", false); + psMetadataAddBool(definewarpwarpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definewarpwarpArgs, PS_LIST_TAIL, "-pretend", 0, "list results but to not queue", false); + psMetadataAddTime(definewarpwarpArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definewarpwarpArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(definewarpwarpArgs, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "search by min solar angle", NAN); + psMetadataAddF32(definewarpwarpArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "search by max solar angle", NAN); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-input_comment", 0, "search by comment field for input exposure(LIKE comparison)", NULL); + psMetadataAddStr(definewarpwarpArgs, PS_LIST_TAIL, "-template_comment", 0, "search by comment field for template exposure(LIKE comparison)", NULL); + psMetadataAddU64(definewarpwarpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -definestackstack + psMetadata *definestackstackArgs = psMetadataAlloc(); + // stack id and exp id searches seem less useful here + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID", NULL); + psMetadataAddBool(definestackstackArgs, PS_LIST_TAIL, "-bothways", 0, "do the subtraction both ways?", false); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-input_label", 0, "search by stack label for input", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-template_label", 0, "search by stack label for template", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-input_stack_id", 0, "search by stack_id for input", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-template_stack_id", 0, "search by stack_id for template", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-input_data_group", 0, "search by stack data_group for input", NULL); + psMetadataAddF32(definestackstackArgs, PS_LIST_TAIL, "-good_frac", 0, "minimum good fraction of skycell", NAN); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddTime(definestackstackArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definestackstackArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(definestackstackArgs, PS_LIST_TAIL, "-rerun", 0, "define new run even if one exists", false); + psMetadataAddBool(definestackstackArgs, PS_LIST_TAIL, "-available", 0, "define new run even if no stacks available for some skycells", false); + psMetadataAddBool(definestackstackArgs, PS_LIST_TAIL, "-new-templates", 0, "also search for diffs with new template and same template label", false); + psMetadataAddBool(definestackstackArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definestackstackArgs, PS_LIST_TAIL, "-pretend", 0, "list results but do not queue", false); + + + // -pendingcleanuprun + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list runs for cleanup with specified label", NULL); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tosummary + psMetadata *tosummaryArgs = psMetadataAlloc(); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(tosummaryArgs , PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", NULL); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + + psMetadataAddTime(tosummaryArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(tosummaryArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-magicked", 0, "search by magic id", 0); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by diffRun label (LIKE comparison)", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by diffRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by diffRun dist_group (LIKE comparison)", NULL); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-not_destreaked", 0, "search for runs that have not been destreaked", false); + + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(tosummaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addsummary + psMetadata *addsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(addsummaryArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-projection_cell", 0, "set projection cell", NULL); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-path_base", 0, "set summary path base", NULL); + + // -pendingcleanupskyfile + psMetadata *pendingcleanupskyfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff ID", 0); + psMetadataAddBool(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-all", 0, "list all skycells regardless of data_state", false); + psMetadataAddBool(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -revertcleanup + psMetadata *revertcleanupArgs = psMetadataAlloc(); + psMetadataAddS64(revertcleanupArgs, PS_LIST_TAIL, "-diff_id", 0, "search by difftool ID", 0); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by diffRun label", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by diffRun data_group", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -updatediffskyfile + psMetadata *updatediffskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(updatediffskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "define diff ID (required)", 0); + psMetadataAddStr(updatediffskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddS16(updatediffskyfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + psMetadataAddS16(updatediffskyfileArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality", 0); + psMetadataAddStr(updatediffskyfileArgs, PS_LIST_TAIL, "-set_state", 0, "set data_state state", NULL); + + // -tocleanedskyfile + psMetadata *tocleanedskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "difftool ID to update", 0); + psMetadataAddStr(tocleanedskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + + // -topurgedskyfile + psMetadata *topurgedskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(topurgedskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "difftool ID to update", 0); + psMetadataAddStr(topurgedskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + + // -toscrubbedskyfile + psMetadata *toscrubbedskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(toscrubbedskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "difftool ID to update", 0); + psMetadataAddStr(toscrubbedskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + + // -tofullskyfile + psMetadata *tofullskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(tofullskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "difftool ID to update", 0); + psMetadataAddStr(tofullskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + psMetadataAddS64(tofullskyfileArgs, PS_LIST_TAIL, "-magicked", 0, "define magicked state", 0); + + // -setskyfiletoupdate + psMetadata *setskyfiletoupdateArgs = psMetadataAlloc(); + psMetadataAddS64(setskyfiletoupdateArgs, PS_LIST_TAIL, "-diff_id", 0, "search by difftool ID (required)", 0); + psMetadataAddStr(setskyfiletoupdateArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(setskyfiletoupdateArgs, PS_LIST_TAIL, "-set_label", 0, "new value for diffRun.label", NULL); + + + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-diff_id", 0, "export this diff ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definerun", "", DIFFTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-updaterun", "", DIFFTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-addinputskyfile", "", DIFFTOOL_MODE_ADDINPUTSKYFILE, addinputskyfileArgs); + PXOPT_ADD_MODE("-inputskyfile", "", DIFFTOOL_MODE_INPUTSKYFILE, inputskyfileArgs); + PXOPT_ADD_MODE("-todiffskyfile", "", DIFFTOOL_MODE_TODIFFSKYFILE, todiffskyfileArgs); + PXOPT_ADD_MODE("-adddiffskyfile", "", DIFFTOOL_MODE_ADDDIFFSKYFILE, adddiffskyfileArgs); + PXOPT_ADD_MODE("-advance", "", DIFFTOOL_MODE_ADVANCE, advanceArgs); + PXOPT_ADD_MODE("-diffskyfile", "", DIFFTOOL_MODE_DIFFSKYFILE, diffskyfileArgs); + PXOPT_ADD_MODE("-revertdiffskyfile","", DIFFTOOL_MODE_REVERTDIFFSKYFILE, revertdiffskyfileArgs); + PXOPT_ADD_MODE("-definepoprun", "", DIFFTOOL_MODE_DEFINEPOPRUN, definepoprunArgs); + PXOPT_ADD_MODE("-definewarpstack", "", DIFFTOOL_MODE_DEFINEWARPSTACK, definewarpstackArgs); + // PXOPT_ADD_MODE("-definewarpstackOldMethod", "", DIFFTOOL_MODE_DEFINEWARPSTACKOLDMETHOD, definewarpstackOldMethodArgs); + PXOPT_ADD_MODE("-definewarpwarp", "", DIFFTOOL_MODE_DEFINEWARPWARP, definewarpwarpArgs); + PXOPT_ADD_MODE("-definestackstack", "", DIFFTOOL_MODE_DEFINESTACKSTACK, definestackstackArgs); + PXOPT_ADD_MODE("-listrun", "list diff runs", DIFFTOOL_MODE_LISTRUN, listrunArgs); + PXOPT_ADD_MODE("-listssrun", "list stack-stack diff runs", DIFFTOOL_MODE_LISTSSRUN, listssrunArgs); + PXOPT_ADD_MODE("-pendingcleanuprun", "show runs that need to be cleaned up", DIFFTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunArgs); + PXOPT_ADD_MODE("-pendingcleanupskyfile", "show runs that need to be cleaned up", DIFFTOOL_MODE_PENDINGCLEANUPSKYFILE, pendingcleanupskyfileArgs); + PXOPT_ADD_MODE("-revertcleanup", "revert cleanup runs with errors", DIFFTOOL_MODE_REVERTCLEANUP, revertcleanupArgs); + PXOPT_ADD_MODE("-donecleanup", "show runs that have been cleaned", DIFFTOOL_MODE_DONECLEANUP, donecleanupArgs); + PXOPT_ADD_MODE("-updatediffskyfile", "update fault code for a diffskyfile", DIFFTOOL_MODE_UPDATEDIFFSKYFILE, updatediffskyfileArgs); + PXOPT_ADD_MODE("-setskyfiletoupdate", "set cleaned skyfile to be updated", DIFFTOOL_MODE_SETSKYFILETOUPDATE, setskyfiletoupdateArgs); + + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", DIFFTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", DIFFTOOL_MODE_IMPORTRUN, importrunArgs); + + PXOPT_ADD_MODE("-tocleanedskyfile", "set skyfile as cleaned", DIFFTOOL_MODE_TOCLEANEDSKYFILE, tocleanedskyfileArgs); + PXOPT_ADD_MODE("-topurgedskyfile", "set skyfile as purged", DIFFTOOL_MODE_TOPURGEDSKYFILE, topurgedskyfileArgs); + PXOPT_ADD_MODE("-toscrubbedskyfile", "set skyfile as scrubbed", DIFFTOOL_MODE_TOSCRUBBEDSKYFILE, toscrubbedskyfileArgs); + PXOPT_ADD_MODE("-tofullskyfile", "set skyfile as full", DIFFTOOL_MODE_TOFULLSKYFILE, tofullskyfileArgs); + PXOPT_ADD_MODE("-tosummary", "show runs that can be summarized", DIFFTOOL_MODE_TOSUMMARY, tosummaryArgs); + PXOPT_ADD_MODE("-addsummary", "add entry to the summary table", DIFFTOOL_MODE_ADDSUMMARY, addsummaryArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/disttool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/disttool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/disttool.c (revision 41104) @@ -0,0 +1,2347 @@ +/* + * disttool.c + * + * Copyright (C) 2008-2009 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "disttool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool revertrunMode(pxConfig *config); +static bool startoverMode(pxConfig *config); +static bool pendingcomponentMode(pxConfig *config); +static bool addprocessedcomponentMode(pxConfig *config); +static bool updateprocessedcomponentMode(pxConfig *config); +static bool revertcomponentMode(pxConfig *config); +static bool revertcomponentMode(pxConfig *config); +static bool processedcomponentMode(pxConfig *config); +static bool toadvanceMode(pxConfig *config); +static bool pendingfilesetMode(pxConfig *config); +static bool addfilesetMode(pxConfig *config); +static bool revertfilesetMode(pxConfig *config); +static bool updatefilesetMode(pxConfig *config); +static bool queuercrunMode(pxConfig *config); +static bool updatercrunMode(pxConfig *config); +static bool revertrcrunMode(pxConfig *config); +static bool pendingdestMode(pxConfig *config); +static bool pendingcleanupMode(pxConfig *config); +static bool listfilesetsMode(pxConfig *config); + +static bool definetargetMode(pxConfig *config); +static bool updatetargetMode(pxConfig *config); +static bool listtargetsMode(pxConfig *config); + +static bool definedestinationMode(pxConfig *config); +static bool updatedestinationMode(pxConfig *config); + +static bool defineinterestMode(pxConfig *config); +static bool updateinterestMode(pxConfig *config); +static bool listinterestsMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = disttoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(DISTTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(DISTTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(DISTTOOL_MODE_REVERTRUN, revertrunMode); + MODECASE(DISTTOOL_MODE_STARTOVER, startoverMode); + MODECASE(DISTTOOL_MODE_PENDINGCOMPONENT, pendingcomponentMode); + MODECASE(DISTTOOL_MODE_ADDPROCESSEDCOMPONENT, addprocessedcomponentMode); + MODECASE(DISTTOOL_MODE_UPDATEPROCESSEDCOMPONENT, updateprocessedcomponentMode); + MODECASE(DISTTOOL_MODE_PROCESSEDCOMPONENT, processedcomponentMode); + MODECASE(DISTTOOL_MODE_REVERTCOMPONENT, revertcomponentMode); + MODECASE(DISTTOOL_MODE_TOADVANCE, toadvanceMode); + MODECASE(DISTTOOL_MODE_PENDINGFILESET, pendingfilesetMode); + MODECASE(DISTTOOL_MODE_PENDINGCLEANUP, pendingcleanupMode); + MODECASE(DISTTOOL_MODE_ADDFILESET, addfilesetMode); + MODECASE(DISTTOOL_MODE_REVERTFILESET, revertfilesetMode); + MODECASE(DISTTOOL_MODE_LISTFILESETS, listfilesetsMode); + MODECASE(DISTTOOL_MODE_UPDATEFILESET, updatefilesetMode); + MODECASE(DISTTOOL_MODE_QUEUERCRUN, queuercrunMode); + MODECASE(DISTTOOL_MODE_UPDATERCRUN, updatercrunMode); + MODECASE(DISTTOOL_MODE_REVERTRCRUN, revertrcrunMode); + MODECASE(DISTTOOL_MODE_PENDINGDEST, pendingdestMode); + MODECASE(DISTTOOL_MODE_DEFINETARGET, definetargetMode); + MODECASE(DISTTOOL_MODE_UPDATETARGET, updatetargetMode); + MODECASE(DISTTOOL_MODE_LISTTARGETS, listtargetsMode); + MODECASE(DISTTOOL_MODE_DEFINEDESTINATION, definedestinationMode); + MODECASE(DISTTOOL_MODE_UPDATEDESTINATION, updatedestinationMode); + MODECASE(DISTTOOL_MODE_DEFINEINTEREST, defineinterestMode); + MODECASE(DISTTOOL_MODE_UPDATEINTEREST, updateinterestMode); + MODECASE(DISTTOOL_MODE_LISTINTERESTS, listinterestsMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + + // optional + PXOPT_LOOKUP_BOOL(use_alternate, config->args, "-use_alternate", false); + PXOPT_LOOKUP_BOOL(magic, config->args, "-magic", false); + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(set_data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(set_note, config->args, "-set_note", false, false); + + PXOPT_LOOKUP_S64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + // select arguments + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-target_id", "distTarget.target_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "==");; + PXOPT_COPY_STR(config->args, where, "-dist_group", "distTarget.dist_group", "==");; + + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + PXOPT_LOOKUP_STR(exp_type, config->args, "-exp_type", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", false, false); + + PXOPT_LOOKUP_BOOL(single, config->args, "-singlefilter", false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + + if (use_alternate) { + if (strcmp(stage, "raw")) { + psError(PXTOOLS_ERR_SYS, true, "alternate inputs only supported for raw stage"); + return false; + } + if (!magic) { + psError(PXTOOLS_ERR_SYS, true, "no_magic forbidden with alternate inputs"); + return false; + } + } + + // We want to possible to distribute unmagicked raw exposures for the purpose + // of distributing detrend inputs + // To make it less likely that unmagicked raw stage exposures from being distributed + // accidentally we require that dist_group (to select target and destination with + // an interest) + // and exp_id to be supplied. We could add -exp_type and dateobs cuts to make it easier + if (!strcmp(stage, "raw") && !magic && !(exp_id && dist_group && exp_type)) { + psError(PXTOOLS_ERR_SYS, true, "exp_id, exp_type, and dist_group are required for raw stage if no_magic"); + return false; + } + + + psString query = NULL; + psString magicRunType = NULL; + psString runJoinStr = NULL; + if (!strcmp(stage, "raw")) { + if (! use_alternate ) { + magicRunType = "rawExp"; + runJoinStr = "rawExp.exp_id"; + } else { + magicRunType = "camRun"; + runJoinStr = "camRun.exp_id"; + } + if (magic) { + query = pxDataGet("disttool_definebyquery_raw.sql"); + } else { + query = pxDataGet("disttool_definebyquery_raw_no_magic.sql"); + } + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (magic) { + if (label) { + psStringAppend(&query, " AND (magicDSRun.label = '%s')", label); + } + // for raw stage we select by camRun.label and dist_group because rawExp + // doesn't have those columns + if (dist_group) { + psStringAppend(&query, " AND (camRun.dist_group = '%s')", dist_group); + } + } + } else if (!strcmp(stage, "chip")) { + magicRunType = "chipRun"; + runJoinStr = "chipRun.chip_id"; + query = pxDataGet("disttool_definebyquery_chip.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (chipRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (chipRun.dist_group = '%s')", dist_group); + } + } else if (!strcmp(stage, "chip_bg")) { + magicRunType = "chipBackgroundRun"; + runJoinStr = "chipBackgroundRun.chip_bg_id"; + query = pxDataGet("disttool_definebyquery_chip_bg.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (chipBackgroundRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (chipBackgroundRun.dist_group = '%s')", dist_group); + } + } else if (!strcmp(stage, "camera")) { + magicRunType = "camRun"; // This is used below to set the magicked business + runJoinStr = "camRun.cam_id"; + query = pxDataGet("disttool_definebyquery_camera.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (camRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (camRun.dist_group = '%s')", dist_group); + } + } else if (!strcmp(stage, "fake")) { + magicRunType = "fakeRun"; + query = pxDataGet("disttool_definebyquery_fake.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (fakeRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (fakeRun.dist_group = '%s')", dist_group); + } + // fake stage doesn't require magic + magic = false; + } else if (!strcmp(stage, "warp")) { + magicRunType = "warpRun"; + runJoinStr = "warpRun.warp_id"; + query = pxDataGet("disttool_definebyquery_warp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (warpRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (warpRun.dist_group = '%s')", dist_group); + } + + } else if (!strcmp(stage, "warp_bg")) { + magicRunType = "warpBackgroundRun"; + runJoinStr = "warpBackgroundRun.warp_bg_id"; + query = pxDataGet("disttool_definebyquery_warp_bg.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (warpBackgroundRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (warpBackgroundRun.dist_group = '%s')", dist_group); + } + + } else if (!strcmp(stage, "diff")) { + magicRunType = "diffRun"; + runJoinStr = "diffRun.diff_id"; + query = pxDataGet("disttool_definebyquery_diff.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (diffRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (diffRun.dist_group = '%s')", dist_group); + } + + } else if (!strcmp(stage, "stack")) { + magicRunType = "stackRun"; + query = pxDataGet("disttool_definebyquery_stack.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (stackRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (stackRun.dist_group = '%s')", dist_group); + } + // stack stage doesn't require magic + magic = false; + } else if (!strcmp(stage, "sky")) { + magicRunType = "staticskyRun"; + if (single) { + query = pxDataGet("disttool_definebyquery_sky_singlefilter.sql"); + } else { + query = pxDataGet("disttool_definebyquery_sky.sql"); + } + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (staticskyRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (staticskyRun.dist_group = '%s')", dist_group); + } + // (static)sky stage doesn't require magic + magic = false; + } else if (!strcmp(stage, "skycal")) { + query = pxDataGet("disttool_definebyquery_skycal.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (label) { + psStringAppend(&query, " AND (skycalRun.label = '%s')", label); + } + if (dist_group) { + psStringAppend(&query, " AND (skycalRun.dist_group = '%s')", dist_group); + } + // skycal stage doesn't require magic + magic = false; + } else if (!strcmp(stage, "SSdiff")) { + magicRunType = "diffRun"; + runJoinStr = "diffRun.diff_id"; + query = pxDataGet("disttool_definebyquery_SSdiff.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + psFree(where); + return(false); + } + + if (label) { + psStringAppend(&query, " AND (diffRun.label = '%s') ", label); + } + if (dist_group) { + psStringAppend(&query, " AND (diffRun.dist_group = '%s') ", dist_group); + } + + magic = false; + } else if (!strcmp(stage, "ff")) { + runJoinStr = "fullForceRun.ff_id"; + query = pxDataGet("disttool_definebyquery_ff.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + psFree(where); + return(false); + } + + if (label) { + psStringAppend(&query, " AND (fullForceRun.label = '%s') ", label); + } + if (dist_group) { + psStringAppend(&query, " AND (fullForceRun.dist_group = '%s') ", dist_group); + } + + magic = false; + } else { + psError(PS_ERR_UNKNOWN, true, "unknown value for stage: %s", stage); + psFree(where); + return false; + } + + if (0) { + fprintf (stderr, "runJoinStr: %s\n", runJoinStr); + } + + if (!strcmp(stage, "raw")) { + if (magic) { + psStringAppend(&query, " AND (magicDSRun.re_place = %d)", !use_alternate); + psStringAppend(&query, " AND (camRun.state = 'full')"); + } + } + if (!rerun) { + psStringAppend(&query, " AND (distRun.dist_id IS NULL)"); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psString joinHook = NULL; + + if (magic) { + psStringAppend(&query, " AND (%s.magicked > 0)", magicRunType); + } + + if (!strcmp(stage, "sky")) { + if (single) { + psStringAppend(&query, "\nGROUP BY sky_id HAVING count(stack_id) = 1"); + } else { + psStringAppend(&query, "\nGROUP BY sky_id HAVING count(stack_id) > 1"); + } + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + psTrace("disttool",2,query,joinHook ? joinHook : ""); + if (!p_psDBRunQueryF(config->dbh, query, joinHook ? joinHook : "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + psFree(joinHook); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + + if (pretend) { + if (!ippdbPrintMetadatas(stdout, output, "newdistRuns", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *list = psArrayAllocEmpty(limit); + for (long i=0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psString stage = psMetadataLookupStr(NULL, md, "stage"); + psString run_tag = psMetadataLookupStr(NULL, md, "run_tag"); + psS64 stage_id = psMetadataLookupS64(NULL, md, "stage_id"); + psS64 magic_ds_id = psMetadataLookupS64(NULL, md, "magicked"); + psS64 target_id = psMetadataLookupS64(NULL, md, "target_id"); + psString stage_label = psMetadataLookupStr(NULL, md, "label"); + psString stage_data_group = psMetadataLookupStr(NULL, md, "data_group"); + bool clean = psMetadataLookupBool(NULL, md, "clean"); + + psString outroot = NULL; + psStringAppend(&outroot, "%s/%s/%s", workdir, run_tag, stage); + + psString new_label; + if (set_label != NULL) { + new_label = set_label; + } else { + new_label = stage_label; + } + psString new_data_group; + if (set_data_group != NULL) { + new_data_group = set_data_group; + } else { + new_data_group = stage_data_group; + } + + distRunRow *row = distRunRowAlloc( + 0, // dist_id + target_id, + stage, + stage_id, + magic_ds_id, + new_label, + outroot, + NULL, // outdir + clean, + !magic, + use_alternate, + "new", + NULL, // time_stamp + 0, // fault + new_data_group, + set_note // note does not propagate + ); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to allocate distRunRow"); + psFree(outroot); + psFree(output); + return false; + } + if (!distRunInsertObject(config->dbh, row)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(outroot); + psFree(output); + return false; + } + psFree(outroot); + psS64 dist_id = psDBLastInsertID(config->dbh); + row->dist_id = dist_id; + psArrayAdd(list, list->n, row); + psFree(row); + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!distRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + + psFree(list); + psFree(output); + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "==");; + PXOPT_COPY_STR(config->args, where, "-state", "distRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-data_group", "distRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group", "distTarget.dist_group", "=="); + PXOPT_COPY_TIME(config->args, where, "-time_stamp_begin", "distRun.time_stamp", ">="); + PXOPT_COPY_TIME(config->args, where, "-time_stamp_end", "distRun.time_stamp", "<="); + PXOPT_COPY_S64(config->args, where, "-dist_id_min", "dist_id", ">="); + PXOPT_COPY_S64(config->args, where, "-dist_id_max", "dist_id", "<="); + + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_BOOL(full, config->args, "-full", false); + if (clean && full) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, true, "-clean and -full are contradictory parameters"); + return false; + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(outdir, config->args, "-set_outdir", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(set_note, config->args, "-set_note", false, false); + + if ((!state) && (!label) && (!fault) &&(!data_group)) { + psError(PXTOOLS_ERR_CONFIG, false, "parameters (-fault or -set_state or -set_label -set_data_group) are required"); + psFree(where); + return false; + } + + psString extraWhere = NULL; + psString query = psStringCopy("UPDATE distRun join distTarget using(target_id, stage) SET distRun.time_stamp = UTC_TIMESTAMP()"); + + if (outdir) { + psStringAppend(&query, " , distRun.outdir = '%s'", outdir); + } + if (state) { + psStringAppend(&query, " , distRun.state = '%s'", state); + if (!strcmp(state, "goto_cleaned")) { + // don't queue for clean up if run has already already cleaned + psStringAppend(&extraWhere, " AND (distRun.state != 'cleaned' AND distRun.state != 'goto_cleaned')"); + } + } + + if (label) { + psStringAppend(&query, " , distRun.label = '%s'", label); + } + + if (fault) { + psStringAppend(&query, " , distRun.fault = %d", fault); + } + + if (data_group) { + psStringAppend(&query, " , distRun.data_group = '%s'", data_group); + } + + if (set_note) { + psStringAppend(&query, " , distRun.note = '%s'", set_note); + } + + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (extraWhere) { + psStringAppend(&query, "%s", extraWhere); + } + if (clean) { + psStringAppend(&query, " AND (distRun.clean)"); + } else if (full) { + psStringAppend(&query, " AND (!distRun.clean)"); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} + +static bool startoverMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "distRun.dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "==");; + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "distRun.data_group", "LIKE"); + + // require data_group or dist_id to be supplied + PXOPT_LOOKUP_STR(data_group, config->args, "-data_group", false, false); + PXOPT_LOOKUP_S64(dist_id, config->args, "-dist_id", false, false); + if (!data_group && !dist_id) { + psError(PXTOOLS_ERR_CONFIG, true, "data_group or dist_id is required"); + return false; + } + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + + psString label_hook = psStringCopy(""); + if (set_label) { + psStringAppend(&label_hook, ", label = '%s'", set_label); + } + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + + // It might be useful to be able to query by the parameters of the underlying runs + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("disttool_rerun_select.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (simple) {/* no option? */} + + if (pretend) { + if (!ippdbPrintMetadatas(stdout, output, "distRunsToUpdate", true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + query = "DELETE FROM distComponent where dist_id = %" PRId64; + char *query2 = "DELETE from rcDSFileset using distRun, rcDSFileset WHERE distRun.dist_id = rcDSFileset.dist_id AND rcDSFileset.state ='cleaned' AND rcDSFileset.dist_id = %" PRId64; + + for (long i=0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 dist_id = psMetadataLookupS64(NULL, md, "dist_id"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // delete any existing distComponents + if (!p_psDBRunQueryF(config->dbh, query, dist_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + // delete any exisiting fileset + if (!p_psDBRunQueryF(config->dbh, query2, dist_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (!p_psDBRunQueryF(config->dbh, + "UPDATE distRun SET state = 'new', fault=0 %s WHERE dist_id =%" PRId64, + label_hook, dist_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + printf("re-running distRun %" PRId64 "\n", dist_id); + } + + psFree(output); + + return true; +} +static bool revertrunMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "distRun.dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "==");; + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + pxAddLabelSearchArgs(config, where, "-label", "distRun.label", "=="); + + PXOPT_COPY_S16(config->args, where, "-fault", "distRun.fault", "=="); + + // It might be useful to be able to query by the parameters of the underlying runs + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("disttool_revertrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + int numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("disttool", PS_LOG_INFO, "Updated %d dist runs", numUpdated); + + return true; +} + +static bool revertcomponentMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "distRun.dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "==");; + PXOPT_COPY_STR(config->args, where, "-component", "component", "==");; + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + pxAddLabelSearchArgs(config, where, "-label", "distRun.label", "=="); + + PXOPT_COPY_S16(config->args, where, "-fault", "distComponent.fault", "=="); + + // It might be useful to be able to query by the parameters of the underlying runs + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("disttool_revertcomponent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + int numDeleted = psDBAffectedRows(config->dbh); + + psLogMsg("disttool", PS_LOG_INFO, "Deleted %d distComponents", numDeleted); + + return true; +} + +static bool pendingcomponentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "distRun.dist_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "distRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString queryFile = NULL; + psStringAppend(&queryFile, "disttool_pending_%s.sql", stage); + psString query = pxDataGet(queryFile); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement from %s", queryFile); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, dist_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pendingcomponent", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool addprocessedcomponentMode(pxConfig *config) +{ + + // required values + PXOPT_LOOKUP_S64(dist_id, config->args, "-dist_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + + // unless fault code is set require filename, bytes, and md5sum + bool require_fileinfo = false; + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + if (!fault) { + require_fileinfo = true; + } + PXOPT_LOOKUP_S32(bytes, config->args, "-bytes", require_fileinfo, false); + PXOPT_LOOKUP_STR(md5sum, config->args, "-md5sum", require_fileinfo, false); + PXOPT_LOOKUP_STR(outdir, config->args, "-outdir", require_fileinfo, false); + PXOPT_LOOKUP_STR(name, config->args, "-name", require_fileinfo, false); + + if (!distComponentInsert(config->dbh, dist_id, component, bytes, md5sum, "full", outdir, name, fault)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool updateprocessedcomponentMode(pxConfig *config) +{ + + // required values + PXOPT_LOOKUP_S64(dist_id, config->args, "-dist_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_BOOL(clearfault, config->args, "-clearfault", false); + PXOPT_LOOKUP_S32(bytes, config->args, "-bytes", false, false); + PXOPT_LOOKUP_STR(md5sum, config->args, "-md5sum", false, false); + PXOPT_LOOKUP_STR(outdir, config->args, "-outdir", false, false); + PXOPT_LOOKUP_STR(name, config->args, "-name", false, false); + + bool setfault = clearfault || fault; + + if (!bytes && !md5sum && !outdir && !name && !setfault && !fault) { + psError(PS_ERR_UNKNOWN, true, "at least one of bytes md5sum outdir name fault or setfault is required"); + return false; + } + + char *sep = ""; + psString query = psStringCopy("UPDATE distComponent SET "); + if (setfault) { + psStringAppend(&query, "%s fault = %d", sep, fault ? 1 : 0); + sep = ", "; + } + if (bytes) { + psStringAppend(&query, "%s bytes = %d", sep, bytes); + sep = ", "; + } + if (md5sum) { + psStringAppend(&query, "%s md5sum = '%s'", sep, md5sum); + sep = ", "; + } + if (outdir) { + psStringAppend(&query, "%s outdir = '%s'", sep, outdir); + sep = ", "; + } + if (name) { + psStringAppend(&query, "%s name = '%s'", sep, name); + sep = ", "; + } + + psStringAppend(&query, "\nWHERE dist_id = %"PRId64 " AND component = '%s'", + dist_id, component); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool toadvanceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("disttool_toadvance.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "toadvance", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +static bool processedcomponentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-component", "component", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("disttool_processedcomponent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "processedcomponent", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool pendingfilesetMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "distRun.stage", "=="); + pxAddLabelSearchArgs (config, where, "-label", "distRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("disttool_pendingfileset.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pendingfileset", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +static bool addfilesetMode(pxConfig *config) +{ + + // required values + PXOPT_LOOKUP_S64(dist_id, config->args, "-dist_id", true, false); + PXOPT_LOOKUP_S64(dest_id, config->args, "-dest_id", true, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // unless fault code is set require name + PXOPT_LOOKUP_STR(name, config->args, "-name", fault == 0, false); + + if (!rcDSFilesetInsert(config->dbh, + 0, // fs_id + dist_id, + dest_id, + name, + "full", + fault)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool revertfilesetMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fs_id", "fs_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dist_id", "rcDSFileset.dist_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "==");; + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + pxAddLabelSearchArgs(config, where, "-label", "label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "rcDSFileset.fault", "=="); + + // It might be useful to be able to query by the parameters of the underlying runs + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("disttool_revertfileset.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + int numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("disttool", PS_LOG_INFO, "deleted %d rcDSFilesets", numUpdated); + + return true; +} + +static bool pendingdestMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dist_id", "=="); +// PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("disttool_pendingdest.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereString = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + // where string gets added by hook, not at the end of the query + psStringAppend(&whereString, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereString ? whereString : "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pendingfileset", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool queuercrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_S64(config->args, where, "-target_id","target_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fs_id", "fs_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "==");; + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("disttool_queuercrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + long numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("disttool", PS_LOG_INFO, "Inserted %ld rcRuns", numUpdated); + + return true; +} + + +static bool updatercrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_LOOKUP_S64(rc_id, config->args, "-rc_id", false, false); + PXOPT_LOOKUP_STR(fs_name, config->args, "-fs_name", false, false); + PXOPT_LOOKUP_STR(status_fs_name, config->args, "-status_fs_name", false, false); + PXOPT_LOOKUP_S64(dest_id, config->args, "-dest_id", false, false); + + // We either need rc_id or (dest_id and fs_name) to identifiy the rcRun + if ((!rc_id) && !(dest_id && fs_name)) { + psError(PXTOOLS_ERR_CONFIG, true, "either -rc_id or (-fs_name and -dest_id) are required"); + return false; + } + + // now that we have done the argument checking + PXOPT_COPY_S64(config->args, where, "-rc_id", "rc_id", "=="); + PXOPT_COPY_STR(config->args, where, "-fs_name", "rcDSFileset.name", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "rcRun.dest_id", "=="); + + if (!psListLength(where->list)) { + // this can't happen because we checked above + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + + PXOPT_LOOKUP_STR(last_fileset, config->args, "-set_last_fileset", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!state && (fault < 0)) { + psError(PXTOOLS_ERR_CONFIG, false, "parameters (-fault or -set_state) are required"); + psFree(where); + return false; + } + + psString query = pxDataGet("disttool_updatercrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString setString = psStringCopy("\n"); + psString separator = ""; + if (state) { + psStringAppend(&setString, " rcRun.state = '%s'", state); + separator = ","; + } + + // default for fault is -1, so set it if it's zero or higher. This allows clearing fault + // without forcing revert first + if (fault >= 0) { + psStringAppend(&setString, "%s rcRun.fault = %d", separator, fault); + } + + if (status_fs_name) { + psStringAppend(&setString, ", rcRun.status_fs_name = '%s'", status_fs_name); + } + + if (last_fileset) { + psStringAppend(&setString, ", rcDestination.last_fileset = '%s'", last_fileset); + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, setString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(setString); + psFree(query); + return false; + } + psFree(setString); + psFree(query); + + long numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("disttool", PS_LOG_INFO, "Updated %ld rows", numUpdated); + + return true; +} + +static bool revertrcrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-rc_id", "rc_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fs_id", "fs_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fault", "fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("disttool_revertrcrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + long numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("disttool", PS_LOG_INFO, "Updated %ld rcRuns", numUpdated); + + return true; +} + +static bool definetargetMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", true, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", true, false); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + + // optional + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(comment, config->args, "-comment", false, false); + + distTargetRow *row = distTargetRowAlloc( + 0, // target_id + dist_group, + filter, + stage, + clean, + state ? state : "enabled", + comment + ); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to allocate distTarget object"); + return false; + } + if (!distTargetInsertObject(config->dbh, row)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + return false; + } + + // get the assigned target_id + row->target_id = psDBLastInsertID(config->dbh); + + if (!distTargetPrintObject(stdout, row, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(row); + return false; + } + + psFree(row); + + return true; +} +static bool updatetargetMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-target_id", "target_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dist_group", "dist_group", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + + psString query = psStringCopy("UPDATE distTarget SET state = '%s'"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + psFree(where); + psFree(query); + return false; + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool listtargetsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-target_id", "target_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dist_group", "dist_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_BOOL(full, config->args, "-full", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (clean && full) { + psError(PS_ERR_UNKNOWN, false, "can't select both -clean and -full"); + return false; + } + + psString query = psStringCopy("SELECT * FROM distTarget"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + if (clean) { + psStringAppend(&query, " AND (clean)"); + } else if (full) { + psStringAppend(&query, " AND (!clean)"); + } + } else if (clean) { + psStringAppend(&query, " WHERE clean"); + } else if (full) { + psStringAppend(&query, " WHERE !clean"); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "distTarget", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool definedestinationMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(name, config->args, "-name", true, false); + PXOPT_LOOKUP_STR(dbname, config->args, "-ds_dbname", true, false); + PXOPT_LOOKUP_STR(dbhost, config->args, "-ds_dbhost", true, false); + + // optional + PXOPT_LOOKUP_STR(status_uri, config->args, "-status_uri", false, false); + PXOPT_LOOKUP_STR(comment, config->args, "-comment", false, false); + PXOPT_LOOKUP_STR(last_fileset, config->args, "-last_fileset", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + // XXX: should we insure that these names do not contatin any whitespace? + + rcDestinationRow *row = rcDestinationRowAlloc( + 0, // dest_id + name, + status_uri, + comment, + last_fileset, + dbname, + dbhost, + state ? state : "enabled" + ); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to allocate rcDestination object"); + return false; + } + if (!rcDestinationInsertObject(config->dbh, row)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + return false; + } + + // get the assigned target_id + row->dest_id = psDBLastInsertID(config->dbh); + + if (!rcDestinationPrintObject(stdout, row, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(row); + return false; + } + + psFree(row); + + return true; +} + +static bool updatedestinationMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); +#ifdef ALLOW_UPDATE_LAST_FILESET + PXOPT_LOOKUP_STR(last_fileset, config->args, "-set_last_fileset", false, false); + if (!(state || last_fileset)) { + psError(PS_ERR_UNKNOWN, true, "one or more of -set_state or -set_last_fileset is required"); +# else + if (!state) { +#endif + psFree(where); + return false; + } + psString query = psStringCopy("UPDATE rcDestination SET"); + psString sep = ""; + if (state) { + psStringAppend(&query, " state = '%s'", state); + sep = ","; + } +#ifdef ALLOW_UPDATE_LAST_FILESET + // last_fileset normally gets set by updatercrunMode + // Allowing it to be set here might cause problems + // especially since we are allowing selection by dest_id + if (last_fileset) { + psStringAppend(&query, " %s last_fileset = '%s'", sep, last_fileset); + } +#else + if (0) { fprintf (stderr, "sep: %s\n", sep); } +#endif + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + psFree(where); + psFree(query); + return false; + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool defineinterestMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // one of these is required + PXOPT_LOOKUP_S64(dest_id, config->args, "-dest_id", false, false); + PXOPT_LOOKUP_STR(dest_name, config->args, "-dest_name", false, false); + if (!dest_id && !dest_name) { + psError(PS_ERR_UNKNOWN, true, "either dest_id or dest_name is required"); + return false; + } + + // either target_id or stage and label are required + PXOPT_LOOKUP_S64(target_id, config->args, "-target_id", false, false); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + if (!target_id) { + bool error = false; + if (!stage) { + psError(PS_ERR_UNKNOWN, true, "stage is required if target_id is not supplied"); + error = true; + } + if (!dist_group) { + psError(PS_ERR_UNKNOWN, !error, "dist_group is required if target_id is not supplied"); + error = true; + } + if (error) { + return false; + } + } + + // optional + PXOPT_LOOKUP_S64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + if (state) { + if (strcmp(state, "enabled") && strcmp(state, "disabled")) { + psError(PS_ERR_PROGRAMMING, true, "state must be enabled or disabled"); + return false; + } + } else { + // default state + state = "enabled"; + } + + // now that we've done all of our argument checking, copy the values to where + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dest_name", "rcDestination.name", "=="); + PXOPT_COPY_S64(config->args, where, "-target_id", "target_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dist_group", "dist_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "LIKE"); + // if stage is all don't add it to the query (match all stages) + if (stage && strcmp(stage, "all")) { + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + } + + psString query = pxDataGet("disttool_defineinterest.sql"); + + if (!psListLength(where->list)) { + // can't get here + psError(PS_ERR_PROGRAMMING, true, "search parameters are required"); + psFree(where); + psFree(query); + return false; + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + psFree(where); + if (clean) { + psStringAppend(&query, " AND (distTarget.clean)"); + } else { + psStringAppend(&query, " AND (!distTarget.clean)"); + } + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + { + // psStringSubstitute fails unless the input is a psString which it determines by + // comparing the memory blocks free function to an expected value. + // pxDataGet uses psSlurp which leaves a different free function on the memory block. + // To work around this make a copy of the query before doing the substitution. + psString queryCopy = psStringCopy(query); + psFree(query); + query = queryCopy; + } + // change the @STATE@ in the sql file to our state + if (!psStringSubstitute(&query, state, "@STATE@")) { + psError(PS_ERR_UNKNOWN, false, "failed to substitute state string"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + int numInserted = psDBAffectedRows(config->dbh); + printf("inserted %d rows into rcInterest\n", numInserted); + + return true; +} + +static bool updateinterestMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-int_id", "int_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_S64(config->args, where, "-target_id", "target_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dest_name", "rcDestination.name", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group", "dist_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_BOOL(full, config->args, "-full", false); + if (full && clean) { + psError(PS_ERR_UNKNOWN, true, "-full and -clean makes no sense, chose one or the other"); + psFree(where); + return false; + } + + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + if (!state) { + psError(PS_ERR_UNKNOWN, true, "-set_state is required"); + psFree(where); + return false; + } + psString query = pxDataGet("disttool_updateinterest.sql"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + psFree(where); + psFree(query); + return false; + } + psFree(where); + + if (clean) { + psStringAppend(&query, " AND distTarget.clean"); + } else { + psStringAppend(&query, " AND NOT distTarget.clean"); + } + + if (!p_psDBRunQueryF(config->dbh, query, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psS64 numUpdated = psDBAffectedRows(config->dbh); + printf("updated %" PRId64 " interests\n", numUpdated); + + return true; +} +static bool listinterestsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-int_id", "int_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dest_name", "name", "=="); + PXOPT_COPY_S64(config->args, where, "-target_id", "target_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-dist_group", "dist_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "rcInterest.state", "=="); + + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_BOOL(full, config->args, "-full", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (clean && full) { + psError(PS_ERR_UNKNOWN, false, "can't select both -clean and -full"); + return false; + } + + psString query = pxDataGet("disttool_listinterests.sql"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + if (clean) { + psStringAppend(&query, " AND (clean)"); + } else if (full) { + psStringAppend(&query, " AND (!clean)"); + } + } else if (clean) { + psStringAppend(&query, " WHERE clean"); + } else if (full) { + psStringAppend(&query, " WHERE !clean"); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "rcInterest", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool pendingcleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + pxAddLabelSearchArgs (config, where, "-label", "distRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + psString query = pxDataGet("disttool_pendingcleanup.sql"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + psFree(where); + } else if (!all) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters or -all are required"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "distToCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool listfilesetsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + PXOPT_COPY_S64(config->args, where, "-int_id", "int_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dest_id", "dest_id", "=="); + PXOPT_COPY_STR(config->args, where, "-dest_name", "name", "=="); + PXOPT_COPY_S64(config->args, where, "-target_id", "target_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-dist_group", "dist_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + + // PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + // PXOPT_LOOKUP_BOOL(full, config->args, "-full", false); + + pxAddLabelSearchArgs (config, where, "-label", "distRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("disttool_listfilesets.sql"); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("disttool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "distFilesets", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool updatefilesetMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fs_id", "fs_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dist_id", "dist_id", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + // We don't use PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); here + // because we want -fault 0 to work + bool gotFault = false; + psS16 fault = psMetadataLookupS16(&gotFault, config->args, "-fault"); + + if ((!state) && (!gotFault)) { + psError(PXTOOLS_ERR_CONFIG, true, "parameters (-fault or -set_state) is required"); + psFree(where); + return false; + } + + psString query = psStringCopy("UPDATE rcDSFileset SET "); + + if (state) { + psStringAppend(&query, " state = '%s'", state); + } + + if (gotFault) { + psStringAppend(&query, "%s fault = %d", state ? ", " : "", fault); + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/disttool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/disttool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/disttool.h (revision 41104) @@ -0,0 +1,59 @@ +/* + * disttool.h + * + * Copyright (C) 2008 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef DISTTOOL_H +#define DISTTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + DISTTOOL_MODE_NONE = 0x0, + DISTTOOL_MODE_DEFINEBYQUERY, + DISTTOOL_MODE_UPDATERUN, + DISTTOOL_MODE_REVERTRUN, + DISTTOOL_MODE_STARTOVER, + DISTTOOL_MODE_PENDINGCOMPONENT, + DISTTOOL_MODE_ADDPROCESSEDCOMPONENT, + DISTTOOL_MODE_REVERTCOMPONENT, + DISTTOOL_MODE_PROCESSEDCOMPONENT, + DISTTOOL_MODE_UPDATEPROCESSEDCOMPONENT, + DISTTOOL_MODE_TOADVANCE, + DISTTOOL_MODE_PENDINGCLEANUP, + DISTTOOL_MODE_PENDINGFILESET, + DISTTOOL_MODE_ADDFILESET, + DISTTOOL_MODE_REVERTFILESET, + DISTTOOL_MODE_LISTFILESETS, + DISTTOOL_MODE_UPDATEFILESET, + DISTTOOL_MODE_QUEUERCRUN, + DISTTOOL_MODE_UPDATERCRUN, + DISTTOOL_MODE_REVERTRCRUN, + DISTTOOL_MODE_PENDINGDEST, + DISTTOOL_MODE_DEFINEDESTINATION, + DISTTOOL_MODE_UPDATEDESTINATION, + DISTTOOL_MODE_DEFINETARGET, + DISTTOOL_MODE_UPDATETARGET, + DISTTOOL_MODE_LISTTARGETS, + DISTTOOL_MODE_DEFINEINTEREST, + DISTTOOL_MODE_UPDATEINTEREST, + DISTTOOL_MODE_LISTINTERESTS, +} disttoolMode; + +pxConfig *disttoolConfig(pxConfig *config, int argc, char **argv); + +#endif // DISTTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/disttoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/disttoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/disttoolConfig.c (revision 41104) @@ -0,0 +1,414 @@ +/* + * disttoolConfig.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "disttool.h" + +pxConfig *disttoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration!\n"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-stage", 0, "define stage for bundle (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-no_magic", 0, "magic is not needed (ignored)", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-magic", 0, "magic is needed", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-use_alternate", 0, "use alternate inputs", false); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label for run", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data_group for run", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "select by dist_group", NULL); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "don't queue runs just display what would be selected", false); + + // select args + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-target_id", 0, "select by target_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-exp_id", 0, "select by exp_id", 0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-exp_type", 0, "select by exp_type", NULL); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-chip_id", 0, "select by chip_id", 0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", 0, "select by run label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-dist_group", 0, "select by dist_group", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-singlefilter", 0, "select single filter runs (sky stage only)", false); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-rerun", 0, "queue a new distRun even if one already exists", false); + psMetadataAddU64(definebyqueryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "new value for state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "new value for label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "new value for label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_outdir", 0, "new value for outdir", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-stage", 0, "value for stage", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "value for state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "limit updates to label (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "limit updates to data_group (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-dist_group", 0, "limit updates to data_group", NULL); + psMetadataAddTime(updaterunArgs, PS_LIST_TAIL, "-time_stamp_begin", 0, "limit updates by time_stamp (>=)", NULL); + psMetadataAddTime(updaterunArgs, PS_LIST_TAIL, "-time_stamp_end", 0, "limit updates by time_stamp (<=)", NULL); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-dist_id_min", 0, "limit updates by dist_id (>=)", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-dist_id_max", 0, "limit updates by dist_id (<=)", 0); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL,"-clean", 0, "limit updates to clean distRuns", false); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL,"-full", 0, "limit updates to not clean distRuns", false); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -startover + psMetadata *startoverArgs = psMetadataAlloc(); + psMetadataAddStr(startoverArgs, PS_LIST_TAIL, "-data_group", 0, "define data_group (LIKE comparison)", NULL); + psMetadataAddS64(startoverArgs, PS_LIST_TAIL, "-dist_id", 0, "search by dist_id", 0); + psMetadataAddStr(startoverArgs, PS_LIST_TAIL, "-stage", 0, "search by stage", NULL); + psMetadataAddS64(startoverArgs, PS_LIST_TAIL, "-stage_id", 0, "search by stage_id", 0); + psMetadataAddStr(startoverArgs, PS_LIST_TAIL, "-label", 0, "search by label", NULL); + psMetadataAddStr(startoverArgs, PS_LIST_TAIL, "-set_label", 0, "define new label", NULL); + psMetadataAddBool(startoverArgs, PS_LIST_TAIL, "-pretend", 0, "don't queue runs just display what would be selected", false); + psMetadataAddU64(startoverArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(startoverArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertrun + psMetadata *revertrunArgs = psMetadataAlloc(); + psMetadataAddS64(revertrunArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(revertrunArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddS64(revertrunArgs, PS_LIST_TAIL, "-stage_id", 0, "define stage_id", 0); + psMetadataAddStr(revertrunArgs, PS_LIST_TAIL, "-state", 0, "define state", NULL); + psMetadataAddStr(revertrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddS16(revertrunArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + psMetadataAddBool(revertrunArgs, PS_LIST_TAIL, "-all", 0, "revert all faulted runs", NULL); + + // -pendingcomponent + psMetadata *pendingcomponentArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcomponentArgs, PS_LIST_TAIL, "-stage", 0, "limit results to runs for stage (required)", NULL); + psMetadataAddS64(pendingcomponentArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(pendingcomponentArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "limit results to label", NULL); + psMetadataAddU64(pendingcomponentArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingcomponentArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingcleanup + psMetadata *pendingcleanupArgs = psMetadataAlloc(); + psMetadataAddS64(pendingcleanupArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(pendingcleanupArgs, PS_LIST_TAIL, "-stage", 0, "limit results to runs for stage", NULL); + psMetadataAddStr(pendingcleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "limit results to label", NULL); + psMetadataAddU64(pendingcleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingcleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(pendingcleanupArgs, PS_LIST_TAIL, "-all", 0, "list all runs pending cleanup", NULL); + + + // -addprocessedcomponent + psMetadata *addprocessedcomponentArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedcomponentArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(addprocessedcomponentArgs, PS_LIST_TAIL, "-component", 0, "define component (required)", NULL); + psMetadataAddStr(addprocessedcomponentArgs, PS_LIST_TAIL, "-outdir", 0, "define output directory", NULL); + psMetadataAddStr(addprocessedcomponentArgs, PS_LIST_TAIL, "-name", 0, "define file name", NULL); + psMetadataAddS32(addprocessedcomponentArgs, PS_LIST_TAIL, "-bytes", 0, "define file size", 0); + psMetadataAddStr(addprocessedcomponentArgs, PS_LIST_TAIL, "-md5sum", 0, "define stage for bundle", NULL); + psMetadataAddS32(addprocessedcomponentArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -revertcomponent + psMetadata *revertcomponentArgs = psMetadataAlloc(); + psMetadataAddS64(revertcomponentArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(revertcomponentArgs, PS_LIST_TAIL, "-component", 0, "define component", NULL); + psMetadataAddStr(revertcomponentArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddS64(revertcomponentArgs, PS_LIST_TAIL, "-stage_id", 0, "define stage_id", 0); + psMetadataAddStr(revertcomponentArgs, PS_LIST_TAIL, "-state", 0, "define state", NULL); + psMetadataAddStr(revertcomponentArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddS16(revertcomponentArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + psMetadataAddBool(revertcomponentArgs, PS_LIST_TAIL, "-all", 0, "revert all faulted runs", NULL); + + // -processedcomponent + psMetadata *processedcomponentArgs = psMetadataAlloc(); + psMetadataAddS64(processedcomponentArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(processedcomponentArgs, PS_LIST_TAIL, "-component", 0, "define component", NULL); + psMetadataAddU64(processedcomponentArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedcomponentArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updateprocessedcomponentArgs + psMetadata *updateprocessedcomponentArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedcomponentArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(updateprocessedcomponentArgs, PS_LIST_TAIL, "-component", 0, "define component (required)", NULL); + psMetadataAddStr(updateprocessedcomponentArgs, PS_LIST_TAIL, "-outdir", 0, "define output directory", NULL); + psMetadataAddStr(updateprocessedcomponentArgs, PS_LIST_TAIL, "-name", 0, "define file name", NULL); + psMetadataAddS32(updateprocessedcomponentArgs, PS_LIST_TAIL, "-bytes", 0, "define file size", 0); + psMetadataAddStr(updateprocessedcomponentArgs, PS_LIST_TAIL, "-md5sum", 0, "define stage for bundle", NULL); + psMetadataAddS32(updateprocessedcomponentArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + psMetadataAddBool(updateprocessedcomponentArgs, PS_LIST_TAIL, "-clearfault", 0, "set fault to zero", false); + + // -toadvance + psMetadata *toadvanceArgs = psMetadataAlloc(); + psMetadataAddS64(toadvanceArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "limit updates to label", NULL); + psMetadataAddU64(toadvanceArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toadvanceArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingfileset + psMetadata *pendingfilesetArgs = psMetadataAlloc(); + psMetadataAddS64(pendingfilesetArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "limit results to label", NULL); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "limit results to dist_group", NULL); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-stage", 0, "limit results to runs for stage", NULL); + psMetadataAddU64(pendingfilesetArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingfilesetArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addfileset + psMetadata *addfilesetArgs = psMetadataAlloc(); + psMetadataAddS64(addfilesetArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddS64(addfilesetArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddStr(addfilesetArgs, PS_LIST_TAIL, "-name", 0, "define file name", NULL); + psMetadataAddS32(addfilesetArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -updatefileset + psMetadata *updatefilesetArgs = psMetadataAlloc(); + psMetadataAddS64(updatefilesetArgs, PS_LIST_TAIL, "-fs_id", 0, "define fs_id", 0); + psMetadataAddS64(updatefilesetArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddStr(updatefilesetArgs, PS_LIST_TAIL, "-set_state",0, "new value for state", NULL); + psMetadataAddS32(updatefilesetArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -pendingdest + psMetadata *pendingdestArgs = psMetadataAlloc(); + psMetadataAddS64(pendingdestArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddU64(pendingdestArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingdestArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddStr(pendingdestArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "limit results to label", NULL); + + // -revertfileset + psMetadata *revertfilesetArgs = psMetadataAlloc(); + psMetadataAddS64(revertfilesetArgs, PS_LIST_TAIL, "-fs_id", 0, "define fs_id", 0); + psMetadataAddS64(revertfilesetArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddS64(revertfilesetArgs, PS_LIST_TAIL, "-dest_id", 0, "define dist_id", 0); + psMetadataAddStr(revertfilesetArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddS64(revertfilesetArgs, PS_LIST_TAIL, "-stage_id",0, "define stage_id", 0); + psMetadataAddStr(revertfilesetArgs, PS_LIST_TAIL, "-state", 0, "define state", NULL); + psMetadataAddStr(revertfilesetArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddS16(revertfilesetArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + psMetadataAddBool(revertfilesetArgs, PS_LIST_TAIL, "-all", 0, "revert all faulted runs", NULL); + + // -listfileset + psMetadata *listfilesetsArgs = psMetadataAlloc(); + psMetadataAddS64(listfilesetsArgs, PS_LIST_TAIL, "-dist_id", 0, "list filesets with dist_id", 0); + psMetadataAddS64(listfilesetsArgs, PS_LIST_TAIL, "-target_id", 0, "list filesets with target_id", 0); + psMetadataAddS64(listfilesetsArgs, PS_LIST_TAIL, "-dest_id", 0, "list filesets with dest_id", 0); + psMetadataAddS64(listfilesetsArgs, PS_LIST_TAIL, "-int_id", 0, "list filesets with int_id", 0); + psMetadataAddStr(listfilesetsArgs, PS_LIST_TAIL, "-dest_name", 0, "list filesets for destinationn name)", NULL); + psMetadataAddStr(listfilesetsArgs, PS_LIST_TAIL, "-dist_group", 0, "list filesets for dist_group (LIKE comparison)", NULL); + psMetadataAddStr(listfilesetsArgs, PS_LIST_TAIL, "-filter", 0, "list filesets by filter (LIKE comparison)", NULL); + psMetadataAddStr(listfilesetsArgs, PS_LIST_TAIL, "-stage", 0, "list filesets for stage", NULL); + psMetadataAddBool(listfilesetsArgs, PS_LIST_TAIL,"-clean", 0, "list clean filesets", false); + psMetadataAddBool(listfilesetsArgs, PS_LIST_TAIL,"-full", 0, "list full filesets", false); + psMetadataAddStr(listfilesetsArgs, PS_LIST_TAIL, "-state", 0, "list filesets in state", NULL); + psMetadataAddStr(listfilesetsArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "limit results to label", NULL); + psMetadataAddU64(listfilesetsArgs, PS_LIST_TAIL, "-limit", 0, "limit number of filesets listed to N", 0); + psMetadataAddBool(listfilesetsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -queuercrun + psMetadata *queuercrunArgs = psMetadataAlloc(); + psMetadataAddS64(queuercrunArgs, PS_LIST_TAIL, "-dist_id", 0, "define dist_id", 0); + psMetadataAddS64(queuercrunArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddS64(queuercrunArgs, PS_LIST_TAIL, "-target_id", 0, "define target_id", 0); + psMetadataAddS64(queuercrunArgs, PS_LIST_TAIL, "-fs_id", 0, "define fs_id", 0); + psMetadataAddStr(queuercrunArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddS64(queuercrunArgs, PS_LIST_TAIL, "-stage_id", 0, "define stage_id", 0); + psMetadataAddStr(queuercrunArgs, PS_LIST_TAIL, "-state", 0, "define state", NULL); + psMetadataAddStr(queuercrunArgs, PS_LIST_TAIL, "-label", 0, "define label", NULL); + psMetadataAddU64(queuercrunArgs, PS_LIST_TAIL, "-limit", 0, "limit number of runs queued to N", 0); + + // -updatercrun + psMetadata *updatercrunArgs = psMetadataAlloc(); + psMetadataAddS64(updatercrunArgs, PS_LIST_TAIL, "-rc_id", 0, "define rc_id", 0); + psMetadataAddS64(updatercrunArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddStr(updatercrunArgs, PS_LIST_TAIL, "-fs_name", 0, "define fileset name", NULL); + psMetadataAddStr(updatercrunArgs, PS_LIST_TAIL, "-status_fs_name", 0, "define status fileset name", NULL); + psMetadataAddStr(updatercrunArgs, PS_LIST_TAIL, "-set_state",0, "new value for state", NULL); + psMetadataAddStr(updatercrunArgs, PS_LIST_TAIL, "-set_last_fileset",0, "new value for last_fileset", NULL); + psMetadataAddS16(updatercrunArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", -1); + + // -revertrcrun + psMetadata *revertrcrunArgs = psMetadataAlloc(); + psMetadataAddS64(revertrcrunArgs, PS_LIST_TAIL, "-rc_id", 0, "define rc_id", 0); + psMetadataAddS64(revertrcrunArgs, PS_LIST_TAIL, "-fs_id", 0, "define fs_id", 0); + psMetadataAddS64(revertrcrunArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddS16(revertrcrunArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + psMetadataAddBool(revertrcrunArgs, PS_LIST_TAIL, "-all", 0, "revert all faulted runs", NULL); + + // -definedestination + psMetadata *definedestinationArgs = psMetadataAlloc(); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-name", 0, "define destination name (required)", NULL); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-ds_dbname",0, "define data store database name (required)", NULL); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-ds_dbhost",0, "define data store database host (required)", NULL); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-status_uri", 0, "define status_uri", NULL); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-comment", 0, "define comment", NULL); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-last_fileset",0, "define last_fileset", NULL); + psMetadataAddStr(definedestinationArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + + // -updatedestination + psMetadata *updatedestinationArgs = psMetadataAlloc(); + psMetadataAddS64(updatedestinationArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddStr(updatedestinationArgs, PS_LIST_TAIL, "-name", 0, "define destination name", NULL); + psMetadataAddStr(updatedestinationArgs, PS_LIST_TAIL, "-status_uri", 0, "define status_uri", NULL); + psMetadataAddStr(updatedestinationArgs, PS_LIST_TAIL, "-comment", 0, "define comment", NULL); + psMetadataAddStr(updatedestinationArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + + // -definetarget + psMetadata *definetargetArgs = psMetadataAlloc(); + psMetadataAddStr(definetargetArgs, PS_LIST_TAIL, "-dist_group", 0, "define dist_group (required)", NULL); + psMetadataAddStr(definetargetArgs, PS_LIST_TAIL, "-filter", 0, "define filter (required)", NULL); + psMetadataAddStr(definetargetArgs, PS_LIST_TAIL, "-stage", 0, "define stage (required)", NULL); + psMetadataAddBool(definetargetArgs, PS_LIST_TAIL,"-clean", 0, "define clean", false); + psMetadataAddStr(definetargetArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + psMetadataAddStr(definetargetArgs, PS_LIST_TAIL, "-comment", 0, "define comment", NULL); + + // -updatetarget + psMetadata *updatetargetArgs = psMetadataAlloc(); + psMetadataAddS64(updatetargetArgs, PS_LIST_TAIL, "-target_id", 0, "define target_id", 0); + psMetadataAddStr(updatetargetArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddStr(updatetargetArgs, PS_LIST_TAIL, "-dist_group", 0, "define dist_group", NULL); + psMetadataAddStr(updatetargetArgs, PS_LIST_TAIL, "-filter", 0, "define filter", NULL); + + psMetadataAddStr(updatetargetArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + + // -listtargets + psMetadata *listtargetsArgs = psMetadataAlloc(); + psMetadataAddS64(listtargetsArgs, PS_LIST_TAIL, "-target_id", 0, "list target with target_id", 0); + psMetadataAddStr(listtargetsArgs, PS_LIST_TAIL, "-dist_group", 0, "list targets for dist_group", NULL); + psMetadataAddStr(listtargetsArgs, PS_LIST_TAIL, "-filter", 0, "define filter", NULL); + psMetadataAddStr(listtargetsArgs, PS_LIST_TAIL, "-stage", 0, "list targets for stage", NULL); + psMetadataAddBool(listtargetsArgs, PS_LIST_TAIL,"-clean", 0, "list clean targets", false); + psMetadataAddBool(listtargetsArgs, PS_LIST_TAIL,"-full", 0, "list full targets", false); + psMetadataAddStr(listtargetsArgs, PS_LIST_TAIL, "-state", 0, "list targets in state", NULL); + psMetadataAddU64(listtargetsArgs, PS_LIST_TAIL, "-limit", 0, "limit number of targets listed to N", 0); + psMetadataAddBool(listtargetsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -defineinterest + psMetadata *defineinterestArgs = psMetadataAlloc(); + psMetadataAddS64(defineinterestArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddStr(defineinterestArgs, PS_LIST_TAIL, "-dest_name", 0, "define destination name (LIKE comparison)", NULL); + psMetadataAddS64(defineinterestArgs, PS_LIST_TAIL, "-target_id", 0, "define target_id", 0); + psMetadataAddStr(defineinterestArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddStr(defineinterestArgs, PS_LIST_TAIL, "-dist_group", 0, "define dist_group", NULL); + psMetadataAddStr(defineinterestArgs, PS_LIST_TAIL, "-filter", 0, "define filter (LIKE comparison)", NULL); + psMetadataAddBool(defineinterestArgs, PS_LIST_TAIL,"-clean", 0, "list clean targets", false); + psMetadataAddStr(defineinterestArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + psMetadataAddU64(defineinterestArgs, PS_LIST_TAIL, "-limit", 0, "limit number of targets listed to N", 0); + + // -updateinterest + psMetadata *updateinterestArgs = psMetadataAlloc(); + psMetadataAddS64(updateinterestArgs, PS_LIST_TAIL, "-int_id", 0, "define int_id", 0); + psMetadataAddS64(updateinterestArgs, PS_LIST_TAIL, "-dest_id", 0, "define dest_id", 0); + psMetadataAddS64(updateinterestArgs, PS_LIST_TAIL, "-target_id", 0, "define target_id", 0); + psMetadataAddStr(updateinterestArgs, PS_LIST_TAIL, "-set_state", 0, "define state (required)", NULL); + psMetadataAddStr(updateinterestArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddStr(updateinterestArgs, PS_LIST_TAIL, "-filter", 0, "define filter (LIKE comparison)", NULL); + psMetadataAddStr(updateinterestArgs, PS_LIST_TAIL, "-dest_name", 0, "define destination name", NULL); + psMetadataAddStr(updateinterestArgs, PS_LIST_TAIL, "-dist_group", 0, "define distribution group", NULL); + psMetadataAddBool(updateinterestArgs, PS_LIST_TAIL,"-clean", 0, "update clean interests", false); + psMetadataAddBool(updateinterestArgs, PS_LIST_TAIL,"-full", 0, "update full interests", false); + + // -listinterests + psMetadata *listinterestsArgs = psMetadataAlloc(); + psMetadataAddS64(listinterestsArgs, PS_LIST_TAIL, "-target_id", 0, "list interests with target_id", 0); + psMetadataAddS64(listinterestsArgs, PS_LIST_TAIL, "-dest_id", 0, "list interests with dest_id", 0); + psMetadataAddS64(listinterestsArgs, PS_LIST_TAIL, "-int_id", 0, "list interests with int_id", 0); + psMetadataAddStr(listinterestsArgs, PS_LIST_TAIL, "-dest_name", 0, "list interests for destinationn name)", NULL); + psMetadataAddStr(listinterestsArgs, PS_LIST_TAIL, "-dist_group", 0, "list interests for dist_group (LIKE comparison)", NULL); + psMetadataAddStr(listinterestsArgs, PS_LIST_TAIL, "-filter", 0, "list interests by filter (LIKE comparison)", NULL); + psMetadataAddStr(listinterestsArgs, PS_LIST_TAIL, "-stage", 0, "list interests for stage", NULL); + psMetadataAddBool(listinterestsArgs, PS_LIST_TAIL,"-clean", 0, "list clean interests", false); + psMetadataAddBool(listinterestsArgs, PS_LIST_TAIL,"-full", 0, "list full interests", false); + psMetadataAddStr(listinterestsArgs, PS_LIST_TAIL, "-state", 0, "list interests in state", NULL); + psMetadataAddU64(listinterestsArgs, PS_LIST_TAIL, "-limit", 0, "limit number of interests listed to N", 0); + psMetadataAddBool(listinterestsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "define distribution run", DISTTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "update distRun", DISTTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-startover", "reprocess a cleaned distribution run", DISTTOOL_MODE_STARTOVER, startoverArgs); + PXOPT_ADD_MODE("-revertrun", "revert distRun.faults", DISTTOOL_MODE_REVERTRUN, revertrunArgs); + PXOPT_ADD_MODE("-pendingcomponent", "", DISTTOOL_MODE_PENDINGCOMPONENT, pendingcomponentArgs); + PXOPT_ADD_MODE("-addprocessedcomponent", "", DISTTOOL_MODE_ADDPROCESSEDCOMPONENT, addprocessedcomponentArgs); + PXOPT_ADD_MODE("-updateprocessedcomponent", "", DISTTOOL_MODE_UPDATEPROCESSEDCOMPONENT, updateprocessedcomponentArgs); + PXOPT_ADD_MODE("-revertcomponent", "revert faulted components", DISTTOOL_MODE_REVERTCOMPONENT, revertcomponentArgs); + PXOPT_ADD_MODE("-processedcomponent", "", DISTTOOL_MODE_PROCESSEDCOMPONENT, processedcomponentArgs); + PXOPT_ADD_MODE("-toadvance", "", DISTTOOL_MODE_TOADVANCE, toadvanceArgs); + PXOPT_ADD_MODE("-pendingcleanup", "", DISTTOOL_MODE_PENDINGCLEANUP, pendingcleanupArgs); + PXOPT_ADD_MODE("-pendingfileset", "", DISTTOOL_MODE_PENDINGFILESET, pendingfilesetArgs); + PXOPT_ADD_MODE("-addfileset", "", DISTTOOL_MODE_ADDFILESET, addfilesetArgs); + PXOPT_ADD_MODE("-updatefileset", "", DISTTOOL_MODE_UPDATEFILESET, updatefilesetArgs); + PXOPT_ADD_MODE("-listfilesets", "", DISTTOOL_MODE_LISTFILESETS, listfilesetsArgs); + PXOPT_ADD_MODE("-revertfileset", "revert faulted filesets", DISTTOOL_MODE_REVERTFILESET, revertfilesetArgs); + PXOPT_ADD_MODE("-queuercrun", "", DISTTOOL_MODE_QUEUERCRUN, queuercrunArgs); + PXOPT_ADD_MODE("-updatercrun", "", DISTTOOL_MODE_UPDATERCRUN, updatercrunArgs); + PXOPT_ADD_MODE("-revertrcrun", "", DISTTOOL_MODE_REVERTRCRUN, revertrcrunArgs); + PXOPT_ADD_MODE("-pendingdest", "", DISTTOOL_MODE_PENDINGDEST, pendingdestArgs); + + PXOPT_ADD_MODE("-definedestination", "", DISTTOOL_MODE_DEFINEDESTINATION, definedestinationArgs); + PXOPT_ADD_MODE("-updatedestination", "", DISTTOOL_MODE_UPDATEDESTINATION, updatedestinationArgs); +// PXOPT_ADD_MODE("-listdestination", "", DISTTOOL_MODE_LISTDESTINATION, listdestinationArgs); + + PXOPT_ADD_MODE("-definetarget", "", DISTTOOL_MODE_DEFINETARGET, definetargetArgs); + PXOPT_ADD_MODE("-updatetarget", "", DISTTOOL_MODE_UPDATETARGET, updatetargetArgs); + PXOPT_ADD_MODE("-listtargets", "", DISTTOOL_MODE_LISTTARGETS, listtargetsArgs); + + PXOPT_ADD_MODE("-defineinterest", "", DISTTOOL_MODE_DEFINEINTEREST, defineinterestArgs); + PXOPT_ADD_MODE("-updateinterest", "", DISTTOOL_MODE_UPDATEINTEREST, updateinterestArgs); + PXOPT_ADD_MODE("-listinterests", "", DISTTOOL_MODE_LISTINTERESTS, listinterestsArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstool.c (revision 41104) @@ -0,0 +1,904 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxdqstats.h" +#include "dqstatstool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool createbundleMode(pxConfig *config); +//static bool cleanbundleMode(pxConfig *config); +static bool pendingbundleMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool revertrunMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = dqstatstoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(DQSTATS_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(DQSTATS_MODE_PENDINGBUNDLE, pendingbundleMode); + MODECASE(DQSTATS_MODE_CREATEBUNDLE, createbundleMode); + // MODECASE(DQSTATS_MODE_CLEANBUNDLE, cleanbundleMode); + MODECASE(DQSTATS_MODE_UPDATERUN, updaterunMode); + MODECASE(DQSTATS_MODE_REVERTRUN, revertrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + //psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_SUCCESS); + + FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + // psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(exit_status); +} + +static bool definebyqueryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config,NULL); + + // optional + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(force, config->args, "-force", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, 0); + + // properties of the raw exposures + psMetadata *where = psMetadataAlloc(); + + pxdqstatsGetSearchArgs(config, where); + + psString query = pxDataGet("dqstatstool_definebyquery.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return(false); + } + + if (label) { + pxAddLabelSearchArgs (config, where, "-label", "camRun.label", "LIKE"); // define using cam label + } + // use psDBGenerateWhereConditionSQL because the SQL ends in a WHERE + if (force) { + psStringAppend(&query, " AND (invalid IS NULL OR invalid = 1 OR invalid = 0) "); + } + else { + psStringAppend(&query, " AND ((warpRun.warp_id IS NOT NULL OR camProcessedExp.quality > 0) AND invalid IS NULL) "); + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + psTrace("debug",3,">> %s",query); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + // printf("%s\n",query); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("dqstatstool", PS_LOG_INFO, "no rows found"); + psWarning("dqstatstool: no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "dqstatsContent", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + // Create the run entry and grab the dqstats_id + dqstatsRunRow *run = dqstatsRunRowAlloc( + 0, // ID + "new", // state + registered, + set_label, + 0 // fault + ); + + if (!dqstatsRunInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(run); + return(false); + } + + psS64 dqstats_id = psDBLastInsertID(config->dbh); + + psArray *list = psArrayAllocEmpty(16); // List of runs to print + psArrayAdd(list, list->n, run); + if (!dqstatsRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return(false); + } + psFree(list); + + // Create the content entry. + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + bool status; + + psS64 exp_id = psMetadataLookupS64(&status, row, "exp_id"); + psS64 chip_id= psMetadataLookupS64(&status, row, "chip_id"); + psS64 cam_id = psMetadataLookupS64(&status, row, "cam_id"); + psS64 warp_id= psMetadataLookupS64(&status, row, "warp_id"); + + if (!dqstatsContentInsert(config->dbh, + dqstats_id, + exp_id, + chip_id, + cam_id, + warp_id, + 0 // do not create automatically invalid contents + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + return(true); +} + +static bool pendingbundleMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config,false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + + psMetadata *where = psMetadataAlloc(); + // pxdqstatsGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-dqstats_id", "dqstatsRun.dqstats_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "dqstatsRun.label", "=="); + + psString query = pxDataGet("dqstatstool_get_run.sql"); // query + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retrieve sql statement (dqstatstool_get_run.sql)"); + return(false); + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psFree(query); // where whereClause + + psArray *output = p_psDBFetchResult(config->dbh); // where whereClause output + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + psTrace("dqstatstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "dqstatsPendingBundle", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return(true); +} +//static bool cleanbundleMode(pxConfig *config) { +// return(true); +//} + + +static bool createbundleMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config,false); + + bool status = true; + PXOPT_LOOKUP_S64(dqstats_id, config->args, "-dqstats_id", true, 0); + PXOPT_LOOKUP_STR(bundleUri, config->args, "-uri", true, 0); + + // Determine the information about this run. + psString runQuery = pxDataGet("dqstatstool_get_run.sql"); + if (!runQuery) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retrieve sql statement (dqstatstool_get_run.sql)"); + return(false); + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dqstats_id", "dqstats_id", "=="); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); // Save for content query + psStringAppend(&runQuery, " AND %s", whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, runQuery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(runQuery); + psFree(whereClause); + return(false); + } + psFree(runQuery); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereClause); + return(false); + } + + if (psArrayLength(output) != 1) { + psError(PS_ERR_UNKNOWN, false, "dqstatstool: single return row not found."); + psFree(output); + psFree(whereClause); + return(false); + } + + psMetadata *run = output->data[0]; + if (strcmp(psMetadataLookupStr(&status,run,"state"),"new") != 0) { + psError(PS_ERR_UNKNOWN, false, "dqstatstool: bundle already created for this run (should not happen)."); + psFree(output); + psFree(whereClause); + return(false); + } + psFree(output); // This held the information about the Run. + + // Read column list from config file + psMetadata *recipe = psMetadataLookupPtr(&status, config->modules->recipes, "DQSTATS"); + psMetadata *columns = psMetadataLookupMetadata(&status,recipe,"STATS.COLUMNS"); + if (!columns) { + psError(PS_ERR_UNKNOWN, false, "Could not read STATS.COLUMN metadata"); + psFree(whereClause); + return(false); + } + + psMetadataIterator *iter = psMetadataIteratorAlloc(columns, PS_LIST_HEAD,NULL); + + psMetadataItem *colItem = NULL; + psString columnlist = NULL; + + psMetadata *colNames = psMetadataAlloc(); + psMetadata *colTypes = psMetadataAlloc(); + while ((colItem = psMetadataGetAndIncrement(iter))) { + if (strcmp(colItem->name, "COLUMN")) continue; + if (colItem->type != PS_DATA_METADATA) { + psError(PS_ERR_UNKNOWN, false, "dqstatstool COLUMN not of type METADATA\n"); + return(NULL); + } + + bool status; + psMetadata *coldef = colItem->data.md; + + if (columnlist) { // Stick intervening commas + psStringAppend(&columnlist, ", "); + } + + psStringAppend(&columnlist, "%s.%s", + psMetadataLookupStr(&status,coldef,"TABLE"), + psMetadataLookupStr(&status,coldef,"ENTRY")); + + psMetadataAddStr(colNames,PS_LIST_TAIL, + psMetadataLookupStr(&status,coldef,"ENTRY"),0,"", + psMetadataLookupStr(&status,coldef,"COLNAME")); + psMetadataAddStr(colTypes,PS_LIST_TAIL, + psMetadataLookupStr(&status,coldef,"ENTRY"),0,"", + psMetadataLookupStr(&status,coldef,"DATATYPE")); + } + psFree(iter); + + if (!columnlist) { + psError(PS_ERR_UNKNOWN, false, "Failed to find a columnlist"); + psFree(whereClause); + psFree(colNames); + psFree(colTypes); + return(false); + } + + // Read decision engine settings from config file. + psMetadata *engine = psMetadataLookupMetadata(&status,recipe,"DECISION.ENGINE"); + if (!engine) { + psError(PS_ERR_UNKNOWN, false, "Could not read DECISION.ENGINE metadata"); + psFree(whereClause); + return(false); + } + psArray *decisionRules = psArrayAllocEmpty(4); // list of screened and verified rules. + psMetadataIterator *iterEngine = psMetadataIteratorAlloc(engine, PS_LIST_HEAD,NULL); + psMetadataItem *engineItem = NULL; + + while ((engineItem = psMetadataGetAndIncrement(iterEngine))) { + if (strcmp(engineItem->name, "RULE")) continue; + if (engineItem->type != PS_DATA_METADATA) { + psError(PS_ERR_UNKNOWN, false, "dqstatstool RULE not of type METADATA\n"); + return(NULL); + } + + bool status; + psMetadata *enginedef = engineItem->data.md; + + if (psMetadataLookupStr(&status,enginedef,"COLNAME")&& + (strcmp(psMetadataLookupStr(&status,enginedef,"RULETYPE"),"STRICT") == 0)&& + (isfinite(psMetadataLookupF32(&status,enginedef,"MINIMUM"))&& + isfinite(psMetadataLookupF32(&status,enginedef,"MAXIMUM")))) { + psArrayAdd(decisionRules,decisionRules->n,enginedef); + } + else if (psMetadataLookupStr(&status,enginedef,"COLNAME")&& + (strcmp(psMetadataLookupStr(&status,enginedef,"RULETYPE"),"CDF") == 0)&& + (isfinite(psMetadataLookupF32(&status,enginedef,"CDF00"))&& + isfinite(psMetadataLookupF32(&status,enginedef,"CDF50"))&& + isfinite(psMetadataLookupF32(&status,enginedef,"CDF100")))) { + psArrayAdd(decisionRules,decisionRules->n, enginedef); + } + else if (strcmp(psMetadataLookupStr(&status,enginedef,"RULETYPE"),"UNUSED") == 0) { + } + else { + psError(PS_ERR_UNKNOWN, false, "Rule invalid."); + } + } + psFree(iterEngine); + + if (decisionRules->n < 1) { + psError(PS_ERR_UNKNOWN, false, "Failed to generate a decision engine"); + psFree(decisionRules); + psFree(engine); + return(false); + } + psFree(engine); + + // Find the contents that comprise this run + psString contentQuery = pxDataGet("dqstatstool_get_contents.sql"); + if (!contentQuery) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retrieve sql statement (dqstatstool_get_contents.sql)"); + psFree(whereClause); + psFree(colNames); + psFree(colTypes); + psFree(columnlist); + return(false); + } + + psStringAppend(&contentQuery, " AND %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, contentQuery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(colNames); + psFree(colTypes); + psFree(columnlist); + psFree(contentQuery); + return(false); + } + psFree(contentQuery); + + psArray *contents = p_psDBFetchResult(config->dbh); + if (!contents) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(colNames); + psFree(colTypes); + psFree(columnlist); + return(false); + } + if (!psArrayLength(contents)) { + psWarning("dqstatstool No rows found"); + psFree(colNames); + psFree(colTypes); + psFree(columnlist); + psFree(contents); + return(false); + } + + // Grab data from the database for a row of the bundle. + psString query = pxDataGet("dqstatstool_createbundle.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retrieve sql statement"); + psFree(colNames); + psFree(colTypes); + psFree(columnlist); + psFree(contents); + return(false); + } + // From disttool.c: + // "psStringSubstitute fails unless the input is a psString which it determines by + // comparing the memory blocks free function to an expected value. + // pxDataGet uses psSlurp which leaves a different free function on the memory block. + // To work around this make a copy of the query before doing the substitution." + psString queryCopy = psStringCopy(query); + psFree(query); + query = queryCopy; + psStringSubstitute(&query,columnlist, "@COLLIST@"); + psFree(columnlist); + + psArray *outTable = psArrayAllocEmpty(contents->n); + for (long i = 0; i < contents->n; i++) { // Loop over content entries. + psMetadata *row = contents->data[i]; + + // Limit it just to the stuff we care about for this row. + bool status; + psString MYquery = psStringCopy(query); + if (!MYquery) { + psTrace("debug",4,"Failed to copy, trying again."); + MYquery = psStringCopy(query); + } + + psMetadata *where = psMetadataAlloc(); + + psMetadataAddS64(where,PS_LIST_TAIL, "rawExp.exp_id", 0, "==", psMetadataLookupS64(&status,row,"exp_id")); + psMetadataAddS64(where,PS_LIST_TAIL, "chipRun.chip_id", 0, "==", psMetadataLookupS64(&status,row,"chip_id")); + psMetadataAddS64(where,PS_LIST_TAIL, "camRun.cam_id", 0, "==", psMetadataLookupS64(&status,row,"cam_id")); + // This may not be real, so we need to check first. Is there a better way to check for a null here? + if (psMetadataLookupS64(&status,row,"warp_id") && (psMetadataLookupS64(&status,row,"warp_id") != PS_MAX_S64)) { + psMetadataAddS64(where,PS_LIST_TAIL, "warpRun.warp_id", 0, "==", psMetadataLookupS64(&status,row,"warp_id")); + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + if (!whereClause) { + psError(PS_ERR_UNKNOWN, false, "Unable to generate Where Condition"); + return(false); + } + psStringAppend(&MYquery, " WHERE %s", whereClause); + +/* psTrace("debug",2,"%ld %ld %ld %ld %s\n",psMetadataLookupS64(&status,row,"exp_id"), */ +/* psMetadataLookupS64(&status,row,"chip_id"), */ +/* psMetadataLookupS64(&status,row,"cam_id"), */ +/* psMetadataLookupS64(&status,row,"warp_id"), */ +/* whereClause); */ + +/* psFree(whereClause); */ +/* psFree(where); */ + // printf("%s\n",MYquery); + + if (!p_psDBRunQuery(config->dbh, MYquery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(MYquery); + psFree(query); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(outTable); + return(false); + } + + psArray *statsoutput = p_psDBFetchResult(config->dbh); + if (!statsoutput) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + psFree(outTable); + return(false); + } + if (!psArrayLength(statsoutput)) { + psWarning("dqstatstool No stats rows found: %ld %s %s",i,MYquery,query); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + psFree(outTable); + psFree(statsoutput); + return(false); + } + psFree(MYquery); + psFree(whereClause); + psFree(where); + + // Magic happens and the fits table is filled. + psMetadata *statsrow = statsoutput->data[0]; // Because there should only be one row. + + psMetadataIterator *statIter = psMetadataIteratorAlloc(statsrow,PS_LIST_HEAD,NULL); + + psMetadataItem *rowItem = NULL; + psMetadata *tableRow = psMetadataAlloc(); + // set args -dbname gpc1 -createbundle -dqstats_id 2 -uri /tmp/czwtest.fits + while ((rowItem = psMetadataGetAndIncrement(statIter))) { + bool success = false; + + psString colType = psMetadataLookupStr(&status,colTypes,rowItem->name); + if (!colType) { + psError(PS_ERR_UNKNOWN,false, "Could not find type for %s",rowItem->name); + return(false); + } + psString colName = psMetadataLookupStr(&status,colNames,rowItem->name); + if (!colName) { + psError(PS_ERR_UNKNOWN,false, "Could not find name for %s",rowItem->name); + return(false); + } + + if (!strcmp(colType,"PS_DATA_BOOL")) { + success = psMetadataAddBool(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.B); + } + else if (!strcmp(colType,"PS_DATA_S8")) { + success = psMetadataAddS8(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.S8); + } + else if (!strcmp(colType,"PS_DATA_S16")) { + success = psMetadataAddS16(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.S16); + } + else if (!strcmp(colType,"PS_DATA_S32")) { + success = psMetadataAddS32(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.S32); + } + else if (!strcmp(colType,"PS_DATA_S64")) { + success = psMetadataAddS64(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.S64); + } + else if (!strcmp(colType,"PS_DATA_U8")) { + success = psMetadataAddU8(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.U8); + } + else if (!strcmp(colType,"PS_DATA_U16")) { + success = psMetadataAddU16(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.U16); + } + else if (!strcmp(colType,"PS_DATA_U32")) { + success = psMetadataAddU32(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.U32); + } + else if (!strcmp(colType,"PS_DATA_U64")) { + success = psMetadataAddU64(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.U64); + } + else if (!strcmp(colType,"PS_DATA_F32")) { + success = psMetadataAddF32(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.F32); + } + else if (!strcmp(colType,"PS_DATA_F64")) { + success = psMetadataAddF64(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.F64); + } + else if (!strcmp(colType,"PS_DATA_STRING")) { + if (rowItem->data.str == NULL) { + // psFits blows up on a string with a null value. + // Arguably I should fix psLib.db should handle this + rowItem->data.str = psStringCopy("NULL"); + } + success = psMetadataAddStr(tableRow,PS_LIST_TAIL,colName,0,"",rowItem->data.str); + } + else if (!strcmp(colType,"PS_DATA_TIME")) { + psString timeString = psTimeToISO(rowItem->data.V); + success = psMetadataAddStr(tableRow,PS_LIST_TAIL,colName,0,"",timeString); + psFree(timeString); + } + + if (!success) { + psError(PS_ERR_UNKNOWN,false, "Error constructing fits table row. %s %s",colType,colName); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + psFree(outTable); + psFree(statsoutput); + psFree(statIter); + psFree(tableRow); + psFree(colType); + psFree(colName); + return(false); + } + } + // Calculate if this exposure was "good" or not. + bool accept = true; + psF32 quality = 1.0; + for (long j = 0; j < decisionRules->n; j++) { + psMetadata *rule = decisionRules->data[j]; + bool status = true; + psString colname = psMetadataLookupStr(&status,rule,"COLNAME"); + if (!status) { + psError(PS_EXIT_CONFIG_ERROR, false, "Unable to find column name"); + return(false); + } + // Not happy with this being set to a F32. Can this ever be something else? + psF32 value = psMetadataLookupF32(&status,tableRow,colname); + if (!status) { + psError(PS_EXIT_CONFIG_ERROR, false, "Unable to find column %s (%f) in compare with rule.", + colname,value); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + psFree(outTable); + psFree(tableRow); + return(false); + } + psString filter = psMetadataLookupStr(&status,rule,"FILTER"); + if (!status) { + status = true; + } + if (filter) { + psString imfilter = psMetadataLookupStr(&status,tableRow,"FILTER"); + if (strcmp(filter,imfilter) != 0) { + continue; + } + } + + if (strcmp(psMetadataLookupStr(&status,rule,"RULETYPE"),"STRICT") == 0) { + psF32 min_value = psMetadataLookupF32(&status,rule,"MINIMUM"); + if (!status) { + psError(PS_EXIT_CONFIG_ERROR, false, "Unable to find min value for %s",colname); + return(false); + } + psF32 max_value = psMetadataLookupF32(&status,rule,"MAXIMUM"); + if (!status) { + psError(PS_EXIT_CONFIG_ERROR, false, "Unable to find max value for %s",colname); + return(false); + } + + if ((value > max_value)||(value < min_value)) { + accept = false; + } + } // End STRICT min/max rule parsing + else if (strcmp(psMetadataLookupStr(&status,rule,"RULETYPE"),"CDF") == 0) { + // I'm just going to implement a quick linear interpolation over the points that exist. + psF32 min_value,max_value = 99; + psF32 min_cdf = 0.0,max_cdf = 1.0; + psF32 cdf; + psF32 try[22] = {0.00,0.05,0.10,0.15,0.20,0.25,0.30,0.35,0.40,0.45,0.50, + 0.55,0.60,0.65,0.70,0.75,0.80,0.85,0.90,0.95,0.99,1.00}; + int i; + int done = 0; + status = true; + min_value = psMetadataLookupF32(&status,rule,"CDF00"); + min_cdf = 0.0; + if (!isfinite(value)) { + quality = 0.0; + continue; + } + if (!status) { + status = true; + psError(PS_EXIT_CONFIG_ERROR, false, "Unable to find CDF00 value for %s",colname); + } + if (value < min_value) { + if (quality > min_cdf) { + quality = 0.0; + continue; + } + } + + for (i = 1; i < 22; i++) { + psString label = psStringCopy(""); + psStringAppend(&label,"CDF%02d",(int) (100 * try[i])); + + max_value = psMetadataLookupF32(&status,rule,label); + max_cdf = try[i]; + if (!status) { + status = true; + psError(PS_EXIT_CONFIG_ERROR, false, "Unable to find %s value for %s",label,colname); + } + if ((value < max_value)&&(value > min_value)) { + cdf = ((value - min_value) * max_cdf + (max_value - value) * min_cdf) / (max_value - min_value); + // fprintf(stderr,"value: %f %f %f %f %f %f %d %d %f\n",value,quality,min_cdf,min_value,max_cdf,max_value, value < max_value, value > min_value,cdf); + if (quality > cdf) { + quality = cdf; + done = 1; + i = 25; + } + } + min_value = max_value; + min_cdf = max_cdf; + } + if (done) { + continue; + } + // Reached end of array without finding the value; + if (value > min_value) { + if (quality > min_cdf) { + quality = 1.0; + } + } + + + } // End CDF rule parsing + } + bool success; + success = psMetadataAddBool(tableRow,PS_LIST_TAIL,"ACCEPT",0,"",accept); + if (!success) { + psError(PS_ERR_UNKNOWN,false, "Error adding exposure quality to fits table row."); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + psFree(outTable); + psFree(statsoutput); + psFree(statIter); + psFree(tableRow); + return(false); + } + success = psMetadataAddF32(tableRow,PS_LIST_TAIL,"QUALITY",0,"",quality); + if (!success) { + psError(PS_ERR_UNKNOWN,false, "Error adding exposure quality to fits table row."); + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + psFree(outTable); + psFree(statsoutput); + psFree(statIter); + psFree(tableRow); + return(false); + } + + // Add the row to the table array. + psArrayAdd(outTable,0,tableRow); + psFree(tableRow); + } + psFree(colNames); + psFree(colTypes); + psFree(contents); + psFree(query); + // Define the fits table here. + + psFits *Table = psFitsOpen(bundleUri,"w"); + if (!Table) { + psError(PS_ERR_UNKNOWN, false, "Could not open the fits table for writing."); + psFree(outTable); + return(false); + } + + psMetadata *header = psMetadataAlloc(); + psMetadataAddS64(header, PS_LIST_TAIL, "DQSTATS_ID", 0, "ID number of DQSTATS run", dqstats_id); + + if (!psFitsWriteTable(Table,header,outTable,"DQSTATS")) { + psError(PS_ERR_UNKNOWN, false, "Unable to write table to fits file."); + psFree(outTable); + psFree(Table); + psFree(header); + return(false); + } + psFree(outTable); + psFree(header); + + // Tear down, cleanup. + psFitsClose(Table); + return(true); +} + + +static bool updaterunMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config,false); + // PXOPT_LOOKUP_S64(dqstats_id, config->args, "-dqstats_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (state && ! pxIsValidState(state)) { + psError(PXTOOLS_ERR_CONFIG, false, "pxIsValidState failed"); + return(false); + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dqstats_id", "dqstats_id", "=="); + PXOPT_COPY_S64(config->args, where, "-state", "state", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return(false); + } + + psString query = psStringCopy("UPDATE dqstatsRun "); + psStringAppend(&query," SET state = '%s' ", state); + if (fault) { + psStringAppend(&query, ", fault = %d ", fault); + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query," WHERE %s ",whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + psFree(query); + psFree(where); + psFree(whereClause); + + return(true); +} + +static bool revertrunMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config,false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-dqstats_id", "dqstats_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return(false); + } + + psString query = psStringCopy("UPDATE dqstatsRun SET fault = 0 "); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query," WHERE fault != 0 AND %s ",whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + psFree(query); + psFree(whereClause); + psFree(where); + + return(true); +} + + + + + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstool.h (revision 41104) @@ -0,0 +1,18 @@ +#ifndef DQSTATS_H +#define DQSTATS_H 1 + +#include "pxtools.h" + +typedef enum { + DQSTATS_MODE_NONE = 0x0, + DQSTATS_MODE_DEFINEBYQUERY, + DQSTATS_MODE_PENDINGBUNDLE, + DQSTATS_MODE_CREATEBUNDLE, + DQSTATS_MODE_CLEANBUNDLE, + DQSTATS_MODE_UPDATERUN, + DQSTATS_MODE_REVERTRUN, +} dqstatstoolMode; + +pxConfig *dqstatstoolConfig(pxConfig *config, int argc, char **argv); + +#endif Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/dqstatstoolConfig.c (revision 41104) @@ -0,0 +1,100 @@ +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include "pxtools.h" +#include "pxdqstats.h" +#include "dqstatstool.h" + +pxConfig *dqstatstoolConfig(pxConfig *config, int argc, char **argv) { + + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, "DQSTATS"); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return(NULL); + } + psTime *now = psTimeGetNow(PS_TIME_TAI); + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by camRun label", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "set run label", NULL); + + pxdqstatsSetSearchArgs(definebyqueryArgs); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-force", 0, "invalidate old entries that match", false); + psMetadataAddU64(definebyqueryArgs, PS_LIST_TAIL, "-limit", 0, "limit bundle size to N rows", 0); + + // -createbundle + psMetadata *createbundleArgs = psMetadataAlloc(); + psMetadataAddS64(createbundleArgs, PS_LIST_TAIL, "-dqstats_id", 0, "search by dqstats_id", 0); + psMetadataAddStr(createbundleArgs, PS_LIST_TAIL, "-uri", 0, "output table URI", 0); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-dqstats_id", 0, "search by dqstats_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by dqstatsRun state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set dqstatsRun stats", NULL); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -pendingbundle + psMetadata *pendingbundleArgs = psMetadataAlloc(); + psMetadataAddS64(pendingbundleArgs, PS_LIST_TAIL, "-dqstats_id", 0, "search by dqstats_id", 0); + psMetadataAddU64(pendingbundleArgs, PS_LIST_TAIL, "-limit", 0, "limit number of bundles to N", 0); + psMetadataAddBool(pendingbundleArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddStr(pendingbundleArgs, PS_LIST_TAIL, "-label", 0, "search by dqstatsRun.label", NULL); + + // -revertrun + psMetadata *revertrunArgs = psMetadataAlloc(); + psMetadataAddStr(revertrunArgs, PS_LIST_TAIL, "-label", 0, "search by dqstatsRun.label", NULL); + psMetadataAddS64(revertrunArgs, PS_LIST_TAIL, "-dqstats_id", 0, "search by dqstats_id", 0); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from cam stage", DQSTATS_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "change dqstats run properties", DQSTATS_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-createbundle", "create the bundle for a run", DQSTATS_MODE_CREATEBUNDLE, createbundleArgs); + PXOPT_ADD_MODE("-pendingbundle", "list bundles needing creation", DQSTATS_MODE_PENDINGBUNDLE, pendingbundleArgs); + PXOPT_ADD_MODE("-revertrun", "revert runs with errors", DQSTATS_MODE_REVERTRUN, revertrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return(NULL); + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return(NULL); + } + + // config = psMetadataConfigRead(config,1,"dqstatsTool.config",1); + + return config; +} + + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/fakemagic =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/fakemagic (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/fakemagic (revision 41104) @@ -0,0 +1,25 @@ +#!/usr/bin/env perl + +# Copyright (C) 2007 Joshua Hoblitt +# +# $Id: fakemagic,v 1.1 2007-11-05 23:29:54 jhoblitt Exp $ + +use strict; +use warnings FATAL => qw( all ); + +use vars qw( $VERSION ); +$VERSION = '0.01'; + +use Getopt::Long qw( GetOptions :config auto_help auto_version pass_through ); +use Pod::Usage qw( pod2usage ); + +my ($output); +GetOptions( + 'output|o=s' => \$output, +) || pod2usage( 2 ); + +pod2usage( -msg => "Required options: --output ", -exitval => 2 ) + unless defined $output; + +open(my $fh, ">$output") or die "can't open file $output: $!"; +close($fh) or die "can't close file $output: $!"; Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/faketool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/faketool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/faketool.c (revision 41104) @@ -0,0 +1,1397 @@ +/* + * faketool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" + +#include "faketool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool pendingexpMode(pxConfig *config); +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedimfileMode(pxConfig *config); +static bool processedimfileMode(pxConfig *config); +static bool revertprocessedimfileMode(pxConfig *config); +static bool updateprocessedimfileMode(pxConfig *config); +static bool advanceexpMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unmaskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); +static bool pendingcleanuprunMode(pxConfig *config); +static bool pendingcleanupimfileMode(pxConfig *config); +static bool donecleanupMode(pxConfig *config); +static bool tocleanedimfileMode(pxConfig *config); +static bool tofullimfileMode(pxConfig *config); +static bool topurgedimfileMode(pxConfig *config); +static bool toscrubbedimfileMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = faketoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(FAKETOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(FAKETOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(FAKETOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(FAKETOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(FAKETOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileMode); + MODECASE(FAKETOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(FAKETOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileMode); + MODECASE(FAKETOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileMode); + MODECASE(FAKETOOL_MODE_ADVANCEEXP, advanceexpMode); + MODECASE(FAKETOOL_MODE_BLOCK, blockMode); + MODECASE(FAKETOOL_MODE_MASKED, maskedMode); + MODECASE(FAKETOOL_MODE_UNMASKED, unmaskedMode); + MODECASE(FAKETOOL_MODE_UNBLOCK, unblockMode); + MODECASE(FAKETOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunMode); + MODECASE(FAKETOOL_MODE_PENDINGCLEANUPIMFILE, pendingcleanupimfileMode); + MODECASE(FAKETOOL_MODE_DONECLEANUP, donecleanupMode); + MODECASE(FAKETOOL_MODE_TOCLEANEDIMFILE, tocleanedimfileMode); + MODECASE(FAKETOOL_MODE_TOFULLIMFILE, tofullimfileMode); + MODECASE(FAKETOOL_MODE_TOPURGEDIMFILE, topurgedimfileMode); + MODECASE(FAKETOOL_MODE_TOSCRUBBEDIMFILE, toscrubbedimfileMode); + MODECASE(FAKETOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(FAKETOOL_MODE_IMPORTRUN, importrunMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "exp_tag", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); // XXX does this choose the right label? + PXOPT_COPY_STR(config->args, where, "-filelevel", "filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "bt", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "bt", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "object", "=="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "sun_angle", "<"); + PXOPT_COPY_STR(config->args, where, "-comment", "comment", "LIKE"); + + if (!psListLength(where->list) + && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-set_end_stage", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + // default + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("faketool_find_camrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (pretend) { + // then stop before running the query + fprintf(stderr, "%s\n", query); + psFree(query); + return true; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // if end_stage is warp (or NULL), check for valid tess_id + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status; + char *end_stage = psMetadataLookupStr(&status, md, "end_stage"); + if (end_stage && strcasecmp(end_stage, "warp")) continue; + + char *raw_tess_id = psMetadataLookupStr(&status, md, "tess_id"); + if (raw_tess_id || tess_id) continue; + + char *label = psMetadataLookupStr(&status, md, "label"); + psS64 exp_id = psMetadataLookupS64(&status, md, "exp_id"); + + if (!status) { + psError(PS_ERR_UNKNOWN, false, "cannot queue analysis to WARP without a defined tess id: label: %s, exp_id %" PRId64, label, exp_id); + psFree(output); + return false; + } + } + + // loop over our list of cam_ids + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status; + psS64 cam_id = psMetadataLookupS64(&status, md, "cam_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for cam_id"); + psFree(output); + return false; + } + + // queue the exp + if (!pxfakeQueueByCamID(config, cam_id, workdir, label, data_group ? data_group : label, dist_group, reduction, expgroup, dvodb, tess_id, end_stage, note)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue cam_id: %" PRId64, cam_id); + psFree(output); + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "bt", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "bt", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "object", "=="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "sun_angle", "<"); + PXOPT_COPY_STR(config->args, where, "-label", "fakeRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "fakeRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "fakeRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-dist_group", "fakeRun.dist_group", "=="); + + if (!psListLength(where->list)) { + psFree(where); + where = NULL; + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE fakeRun JOIN camRun USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id)"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "fakeRun", "fake_id", "fakeProcessedImfile", true, false); + if (!result) { + psError(psErrorCodeLast(), false, "pxUpdateRun failed"); + } + + psFree(query); + psFree(where); + + return result; +} + + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + + psString query = pxDataGet("faketool_find_pendingexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("camtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakePendingExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "rawImfile.class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + + psString query = pxDataGet("faketool_pendingimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "fakeRun"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakePendingImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // fake_id, ext_tag, class_id are required + PXOPT_LOOKUP_S64(fake_id, config->args, "-fake_id", true, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + // optional + PXOPT_LOOKUP_STR(uri, config->args, "-uri", false, false); + + PXOPT_LOOKUP_F32(dtime_fake, config->args, "-dtime_fake", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!fakeProcessedImfileInsert(config->dbh, + fake_id, + exp_id, + class_id, + uri, + dtime_fake, + dtime_script, + hostname, + path_base, + "full", + fault, + NULL // epoch + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "rawImfile.class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + + psString query = pxDataGet("faketool_processedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND fakeProcessedImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND fakeProcessedImfile.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakeProcessedImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool revertprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "fakeRun.label", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fakeRun.fake_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp..camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "bt", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "bt", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "object", "=="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "sun_angle", "<"); + + if (!psListLength(where->list) + && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("faketool_revertprocessedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool updateprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + if (!pxSetFaultCode(config->dbh, "fakeProcessedImfile", where, fault, 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree(where); + return false; + } + psFree(where); + + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + if (!fakeMaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (where->list->n < 1) { + psFree(where); + where = NULL; + } + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM fakeMask"); + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psFree(where); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakeMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// return the list of labels which are NOT blocked +static bool unmaskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (where->list->n < 1) { + psFree(where); + where = NULL; + } + + psString query = pxDataGet("faketool_unmasked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, "fakeUnmask"); + psFree(where); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakeUnmask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + char *query = "DELETE FROM fakeMask WHERE label = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "fakeRun.label", "=="); + + psString query = pxDataGet("faketool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakePendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool pendingcleanupimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(fake_id, config->args, "-fake_id", false, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + if (fake_id) { + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + } + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + psString query = pxDataGet("faketool_pendingcleanupimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakePendingCleanupImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psString query = pxDataGet("faketool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fakeDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool advanceexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // look for completed fakePendingExp + // migrate them to fakeProccessedExp & camPendingExp + psString query = pxDataGet("faketool_completely_processed_exp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("faketool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + fakeRunRow *fakeRun = fakeRunObjectFromMetadata(row); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // set fakeRun.state to 'full' + if (!pxfakeRunSetState(config, fakeRun->fake_id, "full")) { + psError(PS_ERR_UNKNOWN, false, "failed to change fakeRun.state for fake_id: %" PRId64, fakeRun->fake_id); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(fakeRun); + psFree(output); + return false; + } + + // should we stop here or proceed on to the warp stage? + // NULL for end_stage means go as far as possible + if ((fakeRun->end_stage && psStrcasestr(fakeRun->end_stage, "fake")) || !fakeRun->tess_id) { + psFree(fakeRun); + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + continue; + } + // else continue on... + + // pxwarpQueueByFakeID() can only be run after fakeRun.state has been set to stop + if (!pxwarpQueueByFakeID(config, + fakeRun->fake_id, + fakeRun->workdir, + fakeRun->label, + fakeRun->data_group, + fakeRun->dist_group, + fakeRun->dvodb, + fakeRun->tess_id, + fakeRun->reduction, + fakeRun->end_stage, + NULL // note does not propagate + )) { + psError(PS_ERR_UNKNOWN, false, "failed to queue warpRun"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psFree(fakeRun); + psFree(output); + return false; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(fakeRun); + return false; + } + psFree(fakeRun); + } + + psFree(output); + + return true; +} + +// update fakeProcessedImfile.data_state to given value. +// afterwards, if all imfiles in the exposure have the new state, update the state for the exposure as well +// shared code for the modes -tocleanedimfile -tofullimfile -topurgedimfile + +static bool change_imfile_data_state(pxConfig *config, psString data_state, psString run_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // fake_id, class_id are required + PXOPT_LOOKUP_S64(fake_id, config->args, "-fake_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + psString query = pxDataGet("faketool_change_imfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // note only updates if fakeRun.state = run_state + if (!p_psDBRunQueryF(config->dbh, query, data_state, fake_id, class_id, run_state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + query = pxDataGet("faketool_change_exp_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, fake_id, data_state)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool tocleanedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "cleaned", "goto_cleaned"); +} +static bool tofullimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "full", "update"); +} +static bool topurgedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "purged", "goto_purged"); +} +static bool toscrubbedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "scrubbed", "goto_scrubbed"); +} + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 2; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-fake_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + + ExportTable tables [] = { + {"fakeRun", "faketool_export_run.sql"}, + {"fakeProcessedImfile", "faketool_export_processed_imfile.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + if (!strcmp(tables[i].tableName, "fakeRun")) { + if(!pxSetStateCleaned("fakeRun", "state", output)) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + +#ifdef notdef + fprintf (stderr, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); +#endif + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + psMetadataItem *item = psMetadataLookup (input, "fakeRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + fakeRunRow *fakeRun = fakeRunObjectFromMetadata (entry->data.md); + fakeRunInsertObject (config->dbh, fakeRun); + + // fprintf (stdout, "---- fake run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + item = psMetadataLookup (input, "fakeProcessedImfile"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + for (int i = 0; i < item->data.list->n; i++) { + psMetadataItem *entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + fakeProcessedImfileRow *fakeProcessedImfile = fakeProcessedImfileObjectFromMetadata (entry->data.md); + fakeProcessedImfileInsertObject (config->dbh, fakeProcessedImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/faketool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/faketool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/faketool.h (revision 41104) @@ -0,0 +1,54 @@ +/* + * faketool.h + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef FAKETOOL_H +#define FAKETOOL_H 1 + +#include "pxtools.h" + +typedef enum { + FAKETOOL_MODE_NONE = 0x0, + FAKETOOL_MODE_DEFINEBYQUERY, + FAKETOOL_MODE_UPDATERUN, + FAKETOOL_MODE_PENDINGIMFILE, + FAKETOOL_MODE_PENDINGEXP, + FAKETOOL_MODE_ADDPROCESSEDIMFILE, + FAKETOOL_MODE_PROCESSEDIMFILE, + FAKETOOL_MODE_REVERTPROCESSEDIMFILE, + FAKETOOL_MODE_UPDATEPROCESSEDIMFILE, + FAKETOOL_MODE_ADVANCEEXP, + FAKETOOL_MODE_BLOCK, + FAKETOOL_MODE_MASKED, + FAKETOOL_MODE_UNMASKED, + FAKETOOL_MODE_UNBLOCK, + FAKETOOL_MODE_RETRYPROCESSEDIMFILE, + FAKETOOL_MODE_PENDINGCLEANUPRUN, + FAKETOOL_MODE_PENDINGCLEANUPIMFILE, + FAKETOOL_MODE_DONECLEANUP, + FAKETOOL_MODE_TOCLEANEDIMFILE, + FAKETOOL_MODE_TOFULLIMFILE, + FAKETOOL_MODE_TOPURGEDIMFILE, + FAKETOOL_MODE_TOSCRUBBEDIMFILE, + FAKETOOL_MODE_EXPORTRUN, + FAKETOOL_MODE_IMPORTRUN +} FAKETOOLMode; + +pxConfig *faketoolConfig(pxConfig *config, int argc, char **argv); + +#endif // FAKETOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/faketoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/faketoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/faketoolConfig.c (revision 41104) @@ -0,0 +1,379 @@ +/* + * faketoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "faketool.h" + +pxConfig *faketoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *queueArgs = psMetadataAlloc(); + // XXX need to allow multiple exp_ids + psMetadataAddS64(queueArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(queueArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddS64(queueArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(queueArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(queueArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", "object"); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-ra_min", 0, "define min RA (degrees)", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-ra_max", 0, "define max RA (degrees)", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-decl_min", 0, "define min DEC (degrees)", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-decl_max", 0, "define max DEC (degrees)", NAN); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-exp_time_min", 0, "define min", NAN); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-exp_time_max", 0, "define max", NAN); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "define min fraction of saturated pixels", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-bg_min", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-bg_max", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-alt_min", 0, "define min", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-alt_max", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-az_min", 0, "define min", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-az_max", 0, "define max", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(queueArgs, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF32(queueArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_expgroup", 0, "define exposure group", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tessellation identifier", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_end_stage", 0, "define end stage", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(queueArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(queueArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + psMetadataAddStr(queueArgs, PS_LIST_TAIL, "-comment", 0, "search by comment field (LIKE comparison)", NULL); + + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake ID", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(updaterunArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(updaterunArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", "object"); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-ra_min", 0, "define min RA (degrees)", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-ra_max", 0, "define max RA (degrees)", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-decl_min", 0, "define min DEC (degrees)", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-decl_max", 0, "define max DEC (degrees)", NAN); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-exp_time_min", 0, "define min", NAN); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-exp_time_max", 0, "define max", NAN); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "define min fraction of saturated pixels", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-bg_min", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-bg_max", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-alt_min", 0, "define min", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-alt_max", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-az_min", 0, "define min", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-az_max", 0, "define max", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(updaterunArgs, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-dist_group", 0, "search by data_group", NULL); + + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake ID", 0); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-cam_id", 0, "search by camtool ID", 0); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chiptool ID", 0); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake ID", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS64(pendingimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-inst", 0, "search by camera of interest", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-filter", 0, "search by filter of interest", NULL); + psMetadataAddU64(pendingimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "define fake ID (required)", 0); + psMetadataAddS64(addprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "define exposure ID (required)", 0); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-uri", 0, "define URL", NULL); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-dtime_fake", 0, "define elapsed time for detrend (seconds)", NAN); + psMetadataAddF32(addprocessedimfileArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + psMetadataAddStr(addprocessedimfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddS16(addprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(processedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "define fake ID", 0); + psMetadataAddS64(processedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "define exposure ID", 0); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-inst", 0, "define camera of interest", NULL); + psMetadataAddStr(processedimfileArgs, PS_LIST_TAIL, "-filter", 0, "define filter of interest", NULL); + psMetadataAddU64(processedimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(processedimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updateprocessedimfile + psMetadata *updateprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake ID", 0); + psMetadataAddS64(updateprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(updateprocessedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(updateprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + + // -revertprocessedimfile + psMetadata *revertprocessedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertprocessedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake ID", 0); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS64(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(revertprocessedimfileArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(revertprocessedimfileArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", "object"); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-ra_min", 0, "define min RA (degrees)", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-ra_max", 0, "define max RA (degrees)", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-decl_min", 0, "define min DEC (degrees)", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-decl_max", 0, "define max DEC (degrees)", NAN); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_time_min", 0, "define min", NAN); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-exp_time_max", 0, "define max", NAN); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "define min fraction of saturated pixels", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-bg_min", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-bg_max", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-alt_min", 0, "define min", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-alt_max", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-az_min", 0, "define min", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-az_max", 0, "define max", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-ccd_temp_min",0, "define min ccd tempature", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-ccd_temp_max",0, "define max ccd tempature", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(revertprocessedimfileArgs, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddStr(revertprocessedimfileArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF32(revertprocessedimfileArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + + psMetadataAddBool(revertprocessedimfileArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + psMetadataAddS16(revertprocessedimfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + + // -advanceexp + psMetadata *advanceexpArgs = psMetadataAlloc(); + psMetadataAddS64(advanceexpArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake ID", 0); + psMetadataAddStr(advanceexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label ", NULL); + psMetadataAddU64(advanceexpArgs, PS_LIST_TAIL, "-limit", 0, "limit exposures to advance to N items", 0); + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to mask out (required)", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddStr(maskedArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(maskedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -unmasked + psMetadata *unmaskedArgs = psMetadataAlloc(); + psMetadataAddStr(unmaskedArgs, PS_LIST_TAIL, "-label", 0, "restrict to specified label", NULL); + psMetadataAddBool(unmaskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(unmaskedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to unmask (required)", NULL); + + // -pendingcleanuprun + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupimfile + psMetadata *pendingcleanupimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "search by chip ID", 0); + psMetadataAddBool(pendingcleanupimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + // -tocleanedimfile + psMetadata *tocleanedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "fake ID to update", 0); + psMetadataAddStr(tocleanedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -tofullimfile + psMetadata *tofullimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tofullimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "fake ID to update", 0); + psMetadataAddStr(tofullimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -topurgedimfile + psMetadata *topurgedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(topurgedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "fake ID to update", 0); + psMetadataAddStr(topurgedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -toscrubbedimfile + psMetadata *toscrubbedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(toscrubbedimfileArgs, PS_LIST_TAIL, "-fake_id", 0, "fake ID to update", 0); + psMetadataAddStr(toscrubbedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-fake_id", 0, "export this fake ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from raw exposures", FAKETOOL_MODE_DEFINEBYQUERY, queueArgs); + PXOPT_ADD_MODE("-updaterun", "change fake run properties", FAKETOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-pendingexp", "show pending exposures", FAKETOOL_MODE_PENDINGEXP, pendingexpArgs); + PXOPT_ADD_MODE("-pendingimfile", "show pending imfiles", FAKETOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXOPT_ADD_MODE("-addprocessedimfile", "add a processed imfile", FAKETOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileArgs); + PXOPT_ADD_MODE("-processedimfile", "show processed imfiles", FAKETOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXOPT_ADD_MODE("-updateprocessedimfile","change procesed imfile properties", FAKETOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileArgs); + PXOPT_ADD_MODE("-revertprocessedimfile", "undo a processed imfile", FAKETOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileArgs); + PXOPT_ADD_MODE("-advanceexp", "advance completed exposoures", FAKETOOL_MODE_ADVANCEEXP, advanceexpArgs); + PXOPT_ADD_MODE("-block", "set a label block", FAKETOOL_MODE_BLOCK, blockArgs); + PXOPT_ADD_MODE("-masked", "show blocked labels", FAKETOOL_MODE_MASKED, maskedArgs); + PXOPT_ADD_MODE("-unmasked", "", FAKETOOL_MODE_UNMASKED, unmaskedArgs); + PXOPT_ADD_MODE("-unblock", "remove a label block", FAKETOOL_MODE_UNBLOCK, unblockArgs); + PXOPT_ADD_MODE("-pendingcleanuprun", "show runs that need to be cleaned up", FAKETOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunArgs); + PXOPT_ADD_MODE("-pendingcleanupimfile", "show runs that need to be cleaned up", FAKETOOL_MODE_PENDINGCLEANUPIMFILE, pendingcleanupimfileArgs); + PXOPT_ADD_MODE("-donecleanup", "show runs that have been cleaned", FAKETOOL_MODE_DONECLEANUP, donecleanupArgs); + PXOPT_ADD_MODE("-tocleanedimfile", "set imfile state to cleaned", FAKETOOL_MODE_TOCLEANEDIMFILE, tocleanedimfileArgs); + PXOPT_ADD_MODE("-tofullimfile", "set imfile state to full", FAKETOOL_MODE_TOFULLIMFILE, tofullimfileArgs); + PXOPT_ADD_MODE("-topurgedimfile", "set imfile state to purged", FAKETOOL_MODE_TOPURGEDIMFILE, topurgedimfileArgs); + PXOPT_ADD_MODE("-toscrubbedimfile", "set imfile state to scrubbed", FAKETOOL_MODE_TOSCRUBBEDIMFILE, toscrubbedimfileArgs); + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", FAKETOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", FAKETOOL_MODE_IMPORTRUN, importrunArgs); + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/fftool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/fftool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/fftool.c (revision 41104) @@ -0,0 +1,1106 @@ +/* + * fftool.c + * + * Copyright (C) 2013 Institute for Astronomy, University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxspace.h" +#include "fftool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool todoMode(pxConfig *config); +static bool addresultMode(pxConfig *config); +static bool resultMode(pxConfig *config); +static bool revertMode(pxConfig *config); +static bool updateresultMode(pxConfig *config); +static bool toadvanceMode(pxConfig *config); +static bool addsummaryMode(pxConfig *config); +static bool revertsummaryMode(pxConfig *config); +static bool updatesummaryMode(pxConfig *config); +static bool summaryMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +static bool setfullForceRunState(pxConfig *config, psS64 sky_id, const char *state); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = fftoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(FFTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(FFTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(FFTOOL_MODE_TODO, todoMode); + MODECASE(FFTOOL_MODE_ADDRESULT, addresultMode); + MODECASE(FFTOOL_MODE_RESULT, resultMode); + MODECASE(FFTOOL_MODE_REVERT, revertMode); + MODECASE(FFTOOL_MODE_UPDATERESULT, updateresultMode); + MODECASE(FFTOOL_MODE_TOADVANCE, toadvanceMode); + MODECASE(FFTOOL_MODE_ADDSUMMARY, addsummaryMode); + MODECASE(FFTOOL_MODE_REVERTSUMMARY, revertsummaryMode); + MODECASE(FFTOOL_MODE_UPDATESUMMARY, updatesummaryMode); + MODECASE(FFTOOL_MODE_SUMMARY, summaryMode); + MODECASE(FFTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(FFTOOL_MODE_IMPORTRUN, importrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + + // optional + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + + PXOPT_LOOKUP_STR(sources_path_base, config->args, "-set_sources_path_base", false, false); + + psMetadata *skycalWhereMD = psMetadataAlloc(); + pxAddLabelSearchArgs(config, skycalWhereMD, "-select_skycal_label", "skycalRun.label", "LIKE"); + pxAddLabelSearchArgs(config, skycalWhereMD, "-select_skycal_data_group", "skycalRun.data_group", "LIKE"); + PXOPT_COPY_S64(config->args, skycalWhereMD, "-select_skycal_id", "skycalRun.skycal_id", "=="); + + if (!psListLength(skycalWhereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "skycal search parameters are required"); + psFree(skycalWhereMD); + return false; + } + + PXOPT_COPY_STR(config->args, skycalWhereMD, "-select_skycell_id", "stackRun.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, skycalWhereMD, "-select_tess_id", "stackRun.tess_id", "=="); + pxAddLabelSearchArgs(config, skycalWhereMD, "-select_filter", "stackRun.filter", "LIKE"); + if (!pxskycellAddWhere(config, skycalWhereMD)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + psFree(skycalWhereMD); + return false; + } + + psMetadata *warpWhereMD = psMetadataAlloc(); + pxAddLabelSearchArgs(config, warpWhereMD, "-select_warp_label", "warpRun.label", "LIKE"); + pxAddLabelSearchArgs(config, warpWhereMD, "-select_warp_data_group","warpRun.data_group", "LIKE"); + PXOPT_COPY_S64(config->args, warpWhereMD, "-select_warp_id", "warpRun.warp_id", "=="); + if (!psListLength(warpWhereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "warp search parameters are required"); + psFree(warpWhereMD); + psFree(skycalWhereMD); + return false; + } + PXOPT_COPY_F32(config->args, warpWhereMD, "-select_good_frac_min", "warpSkyfile.good_frac", ">="); + PXOPT_COPY_STR(config->args, warpWhereMD, "-select_tess_id", "warpRun.tess_id", "=="); + pxAddLabelSearchArgs(config, warpWhereMD, "-select_filter", "rawExp.filter", "LIKE"); + + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + psString select = pxDataGet("fftool_definebyquery.sql"); + if (!select) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(skycalWhereMD); + psFree(warpWhereMD); + return false; + } + + psString where = NULL; + psString whereClause = psDBGenerateWhereConditionSQL(skycalWhereMD, NULL); + psStringAppend(&where, "\nAND %s", whereClause); + psStringAppend(&select, " %s", where); + psFree(whereClause); + psFree(skycalWhereMD); + + psString joinHook = NULL; + if (!rerun) { + psStringAppend(&joinHook, "\nLEFT JOIN fullForceRun ON fullForceRun.skycal_id = skycalRun.skycal_id"); + psStringAppend(&joinHook, "\n %s\nAND fullForceRun.label = '%s'", where, label); + psStringAppend(&select, "\nAND ff_id IS NULL"); + } + + if (!p_psDBRunQueryF(config->dbh, select, joinHook)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(select); + return false; + } + psFree(select); + psFree(joinHook); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + psFree(where); + return false; + } + if (!psArrayLength(output)) { + psWarning("fftool: no rows found"); + psFree(output); + psFree(where); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "toFullForce", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + psFree(where); + return false; + } + psFree(output); + psFree(where); + return true; + } + + psString warpQueryTemplate = pxDataGet("fftool_definebyquery_select_warps.sql"); + + whereClause = psDBGenerateWhereConditionSQL(warpWhereMD, NULL); + psStringAppend(&warpQueryTemplate, "\nAND %s", whereClause); + + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + bool status; + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // psS64 warp_id = psMetadataLookupS64(&status, row, "warp_id"); + psS64 skycal_id = psMetadataLookupS64(&status, row, "skycal_id"); + + psString path_base = NULL; + if (sources_path_base) { + path_base = sources_path_base; + } else { + path_base = psMetadataLookupStr(&status, row, "path_base"); + psAssert(status, "failed to find skycal path_base?"); + } + + psString skycal_data_group = psMetadataLookupStr(&status, row, "data_group"); + psString tess_id = psMetadataLookupStr(&status, row, "tess_id"); + psString skycell_id = psMetadataLookupStr(&status, row, "skycell_id"); + psString filter = psMetadataLookupStr(&status, row, "filter"); + + psString query = NULL; + psStringAppend(&query, warpQueryTemplate, tess_id, skycell_id, filter); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + // Find the warps for this skycell and filter combination + psArray *warpOutput = p_psDBFetchResult(config->dbh); + if (!warpOutput) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + return false; + } + if (!psArrayLength(warpOutput)) { + // no warps for this skycal. Suprise? + psFree(warpOutput); + psFree(query); + continue; + } + + // create a staticskyRun + if (!fullForceRunInsert(config->dbh, + 0x0, // ff_id + skycal_id, + path_base, + "new", // state + workdir, + label, + data_group ? data_group : (skycal_data_group ? skycal_data_group : label), + dist_group, + note, + reduction, + NULL // registered + ) + ) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error failed to rollback transaction"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + psS64 ff_id = psDBLastInsertID(config->dbh); + + for (int j = 0; j < warpOutput->n; j++) { + psMetadata *warpRow = warpOutput->data[j]; + psS64 warp_id = psMetadataLookupS64(&status, warpRow, "warp_id"); + + if (!fullForceInputInsert(config->dbh, + ff_id, + warp_id) + ) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error failed to rollback transaction"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warpOutput); + psFree(output); + return false; + } + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warpOutput); + psFree(output); + return false; + } + psFree(warpOutput); + } + psFree(output); + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "ff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "fullForceRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "fullForceRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "fullForceRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "=="); + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE fullForceRun JOIN skycalRun USING(skycal_id) JOIN stackRun USING(stack_id) JOIN skycell USING(tess_id, skycell_id)"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "fullForceRun", "ff_id", "fullForceResult", true, false); + psFree(query); + psFree(where); + + return result; +} + + +static bool todoMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *whereMD = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, whereMD, "-ff_id", "ff_id", "=="); + pxAddLabelSearchArgs (config, whereMD, "-label", "fullForceRun.label", "=="); + pxskycellAddWhere(config, whereMD); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("fftool_todo.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!psListLength(whereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(whereMD); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(whereMD, NULL); + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + psFree(whereMD); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("fftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fullForceRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return true; +} + +static bool addresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + // required + PXOPT_LOOKUP_S64(ff_id, config->args, "-ff_id", true, false); + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + + // optional + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", true, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(software_ver, config->args, "-software_ver", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + // XXX not sure we need a transaction here... + if (!fullForceResultInsert(config->dbh, + ff_id, + warp_id, + path_base, + dtime_script, + quality, + hostname, + software_ver, + fault + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool resultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "fullForceResult.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "fullForceRun.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "fullForceRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "fullForceRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "stackRun.filter", "LIKE"); + PXOPT_COPY_S16(config->args, where, "-fault", "fullForceResult.fault", "=="); + PXOPT_COPY_S16(config->args, where, "-quality", "fullForceResult.quality", "=="); + pxskycellAddWhere(config, where); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("fftool_result.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("fftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "fullForceResult", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool revertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "fullForceResult.warp_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "fullForceRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fullForceResult.fault", "=="); + + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // Delete product + psString delete = pxDataGet("fftool_revert.sql"); + if (!delete) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&delete, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, delete)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete); + psFree(where); + return false; + } + psFree(delete); + + int numRows = psDBAffectedRows(config->dbh); // Number of row affected + psLogMsg("fftool", PS_LOG_INFO, "Deleted %d rows", numRows); + + psFree(where); + return true; +} + +static bool updateresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "ff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + + if (!pxSetFaultCode(config->dbh, "fullForceResult", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + return true; +} + +static bool setfullForceRunState(pxConfig *config, psS64 ff_id, const char *state) +{ + psString query = "UPDATE fullForceRun SET state = 'full' WHERE ff_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, ff_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool toadvanceMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *whereMD = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, whereMD, "-ff_id", "ff_id", "=="); + pxAddLabelSearchArgs (config, whereMD, "-label", "fullForceRun.label", "=="); + pxskycellAddWhere(config, whereMD); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("fftool_toadvance.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!psListLength(whereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(whereMD); + return false; + } + + psString whereClause = NULL; + psString temp = psDBGenerateWhereConditionSQL(whereMD, NULL); + psStringAppend(&whereClause, "\n AND %s", temp); + psFree(temp); + psFree(whereMD); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + // the where clause is required and is added to the query by the "WHERE hook" format string + if (!p_psDBRunQueryF(config->dbh, query, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereClause); + psFree(query); + return false; + } + psFree(whereClause); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("fftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "fullForceRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return true; +} + +static bool addsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + // required + PXOPT_LOOKUP_S64(ff_id, config->args, "-ff_id", true, false); + + // optional + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", true, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_STR(software_ver, config->args, "-software_ver", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!fullForceSummaryInsert(config->dbh, + ff_id, + path_base, + dtime_script, + quality, + hostname, + software_ver, + fault + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!fault) { + if (!setfullForceRunState(config, ff_id, "full")) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to change staticskyRun state"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool revertsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "fullForceRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fullForceSummary.fault", "=="); + + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // Delete product + psString delete = pxDataGet("fftool_revertsummary.sql"); + if (!delete) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&delete, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, delete)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete); + psFree(where); + return false; + } + psFree(delete); + + int numRows = psDBAffectedRows(config->dbh); // Number of row affected + psLogMsg("fftool", PS_LOG_INFO, "Deleted %d rows", numRows); + + psFree(where); + return true; +} +static bool updatesummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "ff_id", "=="); + + + if (!pxSetFaultCode(config->dbh, "fullForceSummary", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + return true; +} +static bool summaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "fullForceRun.ff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "fullForceInput.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "fullForceRun.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "fullForceRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "fullForceRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "stackRun.filter", "LIKE"); + PXOPT_COPY_S16(config->args, where, "-fault", "fullForceSummary.fault", "=="); + pxskycellAddWhere(config, where); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("fftool_summary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("fftool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "fullForceSummary", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); +// PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ff_id", "ff_id", "=="); + + ExportTable tables [] = { + {"fullForceRun", "fftool_export_run.sql"}, + {"fullForceInput", "fftool_export_input.sql"}, + {"fullForceResult", "fftool_export_result.sql"}, + {"fullForceSummary", "fftool_export_summary.sql"}, + }; + + int numTables = sizeof(tables)/sizeof(tables[0]); + + for (int i=0; i < numTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + int numImportTables = 3; + + char tables[3] [80] = {"fullForceInput", "fullForceResult", "fullForceSummary"}; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + +#ifdef notdef + fprintf (stderr, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); +#endif + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + + psMetadataItem *item = psMetadataLookup (input, "fullForceRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + fullForceRunRow *fullForceRun = fullForceRunObjectFromMetadata (entry->data.md); + fullForceRunInsertObject (config->dbh, fullForceRun); + + // fprintf (stdout, "---- warp run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + for (int i = 0; i < numImportTables; i++) { + item = psMetadataLookup (input, tables[i]); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + switch (i) { + case 0: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + fullForceInputRow *fullForceInput = fullForceInputObjectFromMetadata (entry->data.md); + fullForceInputInsertObject (config->dbh, fullForceInput); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 1: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + fullForceResultRow *fullForceResult = fullForceResultObjectFromMetadata (entry->data.md); + fullForceResultInsertObject (config->dbh, fullForceResult); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 2: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + fullForceSummaryRow *fullForceSummary = fullForceSummaryObjectFromMetadata (entry->data.md); + fullForceSummaryInsertObject (config->dbh, fullForceSummary); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + } + } + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/fftool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/fftool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/fftool.h (revision 41104) @@ -0,0 +1,45 @@ +/* + * fftool.h + * + * Copyright (C) 2013 IfA Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef FFTOOL_H +#define FFTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + FFTOOL_MODE_NONE = 0x0, + FFTOOL_MODE_DEFINEBYQUERY, + FFTOOL_MODE_UPDATERUN, + FFTOOL_MODE_TODO, + FFTOOL_MODE_ADDRESULT, + FFTOOL_MODE_RESULT, + FFTOOL_MODE_REVERT, + FFTOOL_MODE_UPDATERESULT, + FFTOOL_MODE_TOADVANCE, + FFTOOL_MODE_ADDSUMMARY, + FFTOOL_MODE_REVERTSUMMARY, + FFTOOL_MODE_UPDATESUMMARY, + FFTOOL_MODE_SUMMARY, + FFTOOL_MODE_EXPORTRUN, + FFTOOL_MODE_IMPORTRUN, +} fftoolMode; + +pxConfig *fftoolConfig(pxConfig *config, int argc, char **argv); + +#endif // FFTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/fftoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/fftoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/fftoolConfig.c (revision 41104) @@ -0,0 +1,249 @@ +/* + * fftool.c + * + * Copyright (C) 2013 Institute for Astronomy, University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "fftool.h" + +pxConfig *fftoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + (void) now; + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_warp_label", PS_META_DUPLICATE_OK, "search by warp label (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_skycal_label", PS_META_DUPLICATE_OK, "search by skycal label (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_warp_data_group", PS_META_DUPLICATE_OK, "search by warp data_group (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_skycal_data_group", PS_META_DUPLICATE_OK, "search by skycal data_group (LIKE comparison, multiple OK)", NULL); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-select_warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-select_skycal_id", 0, "search by skycal ID", 0); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_skycell_id", 0, "search for skycell_id (LIKE comparision)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_tess_id", 0, "search for tess_id", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_filter", PS_META_DUPLICATE_OK, "search by filter (LIKE comparison, multiple OK)", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_good_frac_min", 0, "mimimum good_frac in warp", 0.0); + + pxskycellAddArguments(definebyqueryArgs); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_sources_path_base", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-rerun", 0, "queue new run even if one already exists for inputs", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + pxskycellAddArguments(updaterunArgs); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + + // -todo + psMetadata *todoArgs = psMetadataAlloc(); + psMetadataAddS64(todoArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddStr(todoArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + pxskycellAddArguments(todoArgs); + psMetadataAddU64(todoArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(todoArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addresult + psMetadata *addresultArgs = psMetadataAlloc(); + psMetadataAddS64(addresultArgs, PS_LIST_TAIL, "-ff_id", 0, "define full force ID (required)", 0); + psMetadataAddS64(addresultArgs, PS_LIST_TAIL, "-warp_id", 0, "define warp ID (required)", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location(required)", 0); + psMetadataAddF32(addresultArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddS16(addresultArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-software_ver", 0, "define software version", 0); + psMetadataAddS16(addresultArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -result + psMetadata *resultArgs= psMetadataAlloc(); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", 0); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal ID", 0); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-filter", 0, "search by filter (LIKE comparison)", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID (LIKE comparison)", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID (LIKE comparison)", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-label", 0, "search by label", NULL); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddS16(resultArgs, PS_LIST_TAIL, "-quality", 0, "search by quality value", 0); + pxskycellAddArguments(resultArgs); + psMetadataAddS16(resultArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddU64(resultArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(resultArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revert + psMetadata *revertArgs= psMetadataAlloc(); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddS16(revertArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + pxskycellAddArguments(revertArgs); + + // -updateresult + psMetadata *updateresultArgs = psMetadataAlloc(); + psMetadataAddS64(updateresultArgs, PS_LIST_TAIL, "-ff_id", 0, "define full force ID (required)", 0); + psMetadataAddS64(updateresultArgs, PS_LIST_TAIL, "-warp_id", 0, "define warp ID (required)", 0); + psMetadataAddS16(updateresultArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + psMetadataAddS16(updateresultArgs, PS_LIST_TAIL, "-quality", 0, "set quality code", 0); + + // -toadvance + psMetadata *toadvanceArgs = psMetadataAlloc(); + psMetadataAddS64(toadvanceArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + pxskycellAddArguments(toadvanceArgs); + psMetadataAddU64(toadvanceArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toadvanceArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addsummary + psMetadata *addsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(addsummaryArgs, PS_LIST_TAIL, "-ff_id", 0, "define full force ID (required)", 0); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location(required)", 0); + psMetadataAddF32(addsummaryArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddS16(addsummaryArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-software_ver", 0, "define software version", 0); + psMetadataAddS16(addsummaryArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -updatesummary + psMetadata *updatesummaryArgs = psMetadataAlloc(); + psMetadataAddS64(updatesummaryArgs, PS_LIST_TAIL, "-ff_id", 0, "define full force ID (required)", 0); + psMetadataAddStr(updatesummaryArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", 0); + psMetadataAddF32(updatesummaryArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddS16(updatesummaryArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddStr(updatesummaryArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddStr(updatesummaryArgs, PS_LIST_TAIL, "-software_ver", 0, "define software version", 0); + psMetadataAddS16(updatesummaryArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -summary + psMetadata *summaryArgs= psMetadataAlloc(); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal ID", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-filter", 0, "search by filter (LIKE comparison)", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID (LIKE comparison)", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID (LIKE comparison)", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-label", 0, "search by label", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + pxskycellAddArguments(summaryArgs); + psMetadataAddS16(summaryArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddU64(summaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(summaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertsummary + psMetadata *revertsummaryArgs= psMetadataAlloc(); + psMetadataAddS64(revertsummaryArgs, PS_LIST_TAIL, "-ff_id", 0, "search by full force ID", 0); + psMetadataAddStr(revertsummaryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddS16(revertsummaryArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + pxskycellAddArguments(revertsummaryArgs); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-ff_id", 0, "export this full force ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); +// psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "export run in cleaned state", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "Define a new full force run", FFTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "Update a run", FFTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-todo", "Get runs to do", FFTOOL_MODE_TODO, todoArgs); + PXOPT_ADD_MODE("-addresult", "Add result for fullforce run on a warp", FFTOOL_MODE_ADDRESULT, addresultArgs); + PXOPT_ADD_MODE("-result", "Get result fullforce run on a warp", FFTOOL_MODE_RESULT, resultArgs); + PXOPT_ADD_MODE("-revert", "Revert failed fullforce run on a warp", FFTOOL_MODE_REVERT, revertArgs); + PXOPT_ADD_MODE("-updateresult", "Update result for fullforce run on a warp", FFTOOL_MODE_UPDATERESULT, updateresultArgs); + PXOPT_ADD_MODE("-toadvance", "list completed runs to summarize", FFTOOL_MODE_TOADVANCE, toadvanceArgs); + PXOPT_ADD_MODE("-addsummary", "insert summary result", FFTOOL_MODE_ADDSUMMARY, addsummaryArgs); + PXOPT_ADD_MODE("-updatesummary", "update summary result", FFTOOL_MODE_UPDATESUMMARY, updatesummaryArgs); + PXOPT_ADD_MODE("-revertsummary", "revert faulted summary", FFTOOL_MODE_REVERTSUMMARY, revertsummaryArgs); + PXOPT_ADD_MODE("-summary", "list summary results", FFTOOL_MODE_SUMMARY, summaryArgs); + PXOPT_ADD_MODE("-exportrun", "list summary results", FFTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "list summary results", FFTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorr.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorr.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorr.c (revision 41104) @@ -0,0 +1,933 @@ +/* + * flatcorr.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * Copyright (C) 2008 Eugene Magnier + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include + +#include "pxtools.h" +#include "pxchip.h" +#include "pxcam.h" +#include "pxadd.h" +#include "flatcorr.h" + +static bool definebyqueryMode(pxConfig *config); +static bool definerunMode(pxConfig *config); +static bool addchipMode(pxConfig *config); +static bool dropchipMode(pxConfig *config); +static bool addcameraMode(pxConfig *config); +static bool dropcameraMode(pxConfig *config); +static bool advancecameraMode(pxConfig *config); +static bool advanceaddstarMode(pxConfig *config); +static bool pendingprocessMode(pxConfig *config); +static bool addprocessMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool inputexpMode(pxConfig *config); +static bool inputimfileMode(pxConfig *config); + +static bool setflatcorrRunState(pxConfig *config, psS64 corr_id, const char *state); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = flatcorrConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(FLATCORR_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(FLATCORR_MODE_DEFINERUN, definerunMode); + MODECASE(FLATCORR_MODE_ADDCHIP, addchipMode); + MODECASE(FLATCORR_MODE_DROPCHIP, dropchipMode); + MODECASE(FLATCORR_MODE_ADDCAMERA, addcameraMode); + MODECASE(FLATCORR_MODE_DROPCAMERA, dropcameraMode); + MODECASE(FLATCORR_MODE_ADVANCECAMERA, advancecameraMode); + MODECASE(FLATCORR_MODE_ADVANCEADDSTAR, advanceaddstarMode); + MODECASE(FLATCORR_MODE_PENDINGPROCESS, pendingprocessMode); + MODECASE(FLATCORR_MODE_ADDPROCESS, addprocessMode); + MODECASE(FLATCORR_MODE_UPDATERUN, updaterunMode); + MODECASE(FLATCORR_MODE_INPUTEXP, inputexpMode); + MODECASE(FLATCORR_MODE_INPUTIMFILE, inputimfileMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // require the camera to be defined: this analysis does not make sense + // across multiple cameras + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(det_type, config->args, "-det_type", true, false); + + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-set_filter", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(region, config->args, "-set_region", false, false); + PXOPT_LOOKUP_BOOL(make_correction, config->args, "-make_correction", false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("chiptool_find_rawexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("flatcorr", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + for (long i = 0; i < psArrayLength(output); i++) { + // negative simple so the default is true + if (!ippdbPrintMetadataRaw(stdout, output->data[i], !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + psFree(output); + return true; + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // create a new flatcorrRun + if (!flatcorrRunInsert( + config->dbh, + 0, // corr_id + det_type, + dvodb, + camera, + telescope, + NULL, + filter, + "reg", // state + make_correction, + workdir, + label, + reduction, + region, + NULL, + 0 + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // figure out the ID of the flatcorrRun we just created + psS64 corr_id = psDBLastInsertID(config->dbh); + + // loop over our list of exp_ids + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status; + psS64 exp_id = psMetadataLookupS64(&status, md, "exp_id"); + if (!status) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for exp_id"); + psFree(output); + return false; + } + + // queue the exp : force this to stop at the chip stage + psS64 chip_id = pxchipQueueByExpTag(config, exp_id, workdir, label, label, NULL, reduction, expgroup, dvodb, tess_id, "chip", note); + if (!chip_id) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue exp_id: %" PRId64, exp_id); + psFree(output); + return false; + } + + // add a flatcorrChipLink to the flatcorr Run we just created (initial state has include = TRUE) + if (!flatcorrChipLinkInsert(config->dbh, corr_id, chip_id, 1)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + } + psFree(output); + + // set the flatcorrRun to a state of 'new' + if (!setflatcorrRunState(config, corr_id, "new")) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to set run state"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // require the camera to be defined: this analysis does not make sense across multiple + // cameras + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(det_type, config->args, "-det_type", true, false); + + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-set_filter", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(region, config->args, "-set_region", false, false); + PXOPT_LOOKUP_BOOL(make_correction, config->args, "-make_correction", false); + // XXX probably should make the region in -set_region match ra_min, ra_max, etc + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // start a transaction so we don't end up with an exp without any associted + // imfiles + // if (!psDBTransaction(config->dbh)) { + // psError(PS_ERR_UNKNOWN, false, "database error"); + // return false; + // } + + // create a new flatcorrRun + flatcorrRunRow *row = flatcorrRunRowAlloc( + 0, // corr_id + det_type, + dvodb, + camera, + telescope, + NULL, + filter, + "reg", // state + make_correction, + workdir, + label, + reduction, + region, + NULL, // hostname + 0 // fault + ); + + // create a new flatcorrRun + if (!flatcorrRunInsertObject(config->dbh,row)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // figure out the ID of the flatcorrRun we just created + psS64 corr_id = psDBLastInsertID(config->dbh); + row->corr_id = corr_id; + + // if (!psDBCommit(config->dbh)) { + // psError(PS_ERR_UNKNOWN, false, "database error"); + // return false; + // } + + flatcorrRunPrintObject (stdout, row, !simple); + return true; +} + +static bool addchipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + + // add a flatcorrChipLink (initial state has include = TRUE) + if (!flatcorrChipLinkInsert(config->dbh, corr_id, chip_id, 1)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool dropchipMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + + // UPDATE flatcorrChipLink set include = 0 where corr_id = %lld AND chip_id = %lld + psString query = pxDataGet("flatcorr_dropchip.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, (long long) corr_id, (long long) chip_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} + +static bool addcameraMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", true, false); + + // add a flatcorrCamLink (initial state has include = TRUE) + // XXX should add checks that the chip_id and corr_id are in ChipLink + if (!flatcorrCamLinkInsert(config->dbh, corr_id, chip_id, cam_id, 1)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool dropcameraMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", true, false); + + // UPDATE flatcorrCamLink set include = 0 where corr_id = %lld AND cam_id = %lld + psString query = pxDataGet("flatcorr_dropcamera.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, (long long) corr_id, (long long) cam_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} + +// select the flatcorr chip runs that have completed and for which there is no camera entry +// queue a new camera run for them +static bool advancecameraMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(limit, config->args, "-limit", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "flatcorrRun.label", "=="); + + psString query = pxDataGet("flatcorr_chiprundone.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("flatcorr", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "flatcorr_addcamera", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // loop over our list of chipRun rows + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status; + psS64 corr_id = psMetadataLookupS64(&status, md, "corr_id"); + if (!status) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for corr_id"); + psFree(output); + return false; + } + + chipRunRow *row = chipRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into chipRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxcamQueueByChipID( + config, + row->chip_id, + row->workdir, + row->label, + row->data_group, + row->dist_group, + row->reduction, + row->expgroup, + row->dvodb, + row->tess_id, + "camera", + row->magicked, + NULL // note does not propragate + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue chip_id: %" PRId64, row->chip_id); + psFree(row); + psFree(output); + return false; + } + + // figure out the ID of the flatcorrRun we just created + psS64 cam_id = psDBLastInsertID(config->dbh); + + // add the camRun entry to the flatcorrCamLink table (include is TRUE) + if (!flatcorrCamLinkInsert(config->dbh, corr_id, row->chip_id, cam_id, 1)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return false; +} + +// Select the flatcorr camera runs that have completed and for which there is no addstar +// entry. Queue a new addstar run for them +static bool advanceaddstarMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(limit, config->args, "-limit", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "flatcorrRun.label", "=="); + + psString query = pxDataGet("flatcorr_camerarundone.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("flatcorr", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "flatcorr_addcamera", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // loop over our list of chipRun rows + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status; + psS64 corr_id = psMetadataLookupS64(&status, md, "corr_id"); + if (!status) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for corr_id"); + psFree(output); + return false; + } + + camRunRow *row = camRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into chipRun"); + psFree(output); + return false; + } + + // queue the exp : force image_only to be false (flatcorr is meaningless with just image info) + if (!pxaddQueueByCamID( + config, + "cam", + row->cam_id, + 0, + row->workdir, + row->reduction, + row->label, + row->data_group, + row->dvodb, + NULL, // note is not propagated + 0, + 0, //The minidvodb stuff is off + NULL, + NULL)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue cam_id: %" PRId64, row->cam_id); + psFree(row); + psFree(output); + return false; + } + + // figure out the ID of the flatcorrRun we just created + psS64 add_id = psDBLastInsertID(config->dbh); + + // add the addRun entry to the flatcorrAddstarLink table (include is TRUE) + if (!flatcorrAddstarLinkInsert(config->dbh, corr_id, row->cam_id, add_id, 1)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return false; +} + +// select the flatcorr chip runs that have completed and for which there is no camera entry +// queue a new camera run for them +static bool pendingprocessMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(limit, config->args, "-limit", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + psString query = pxDataGet("flatcorr_pendingprocess.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("flatcorr", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "flatcorrPending", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + return true; +} + +// XXX need a fault state +static bool addprocessMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_S16(code, config->args, "-code", false, false); + + char *query = "UPDATE flatcorrRun SET state = 'full', hostname = '%s', fault = '%hd' WHERE corr_id = %" PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, hostname, code, corr_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for corr_id %" PRId64, corr_id); + return false; + } + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + + if (!setflatcorrRunState(config, corr_id, state)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to set run state"); + return false; + } + + return true; +} + +// select the flatcorr chip runs that have completed and for which there is no camera entry +// queue a new camera run for them +static bool inputexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(corr_id, config->args, "-corr_id", true, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(limit, config->args, "-limit", false); + + char *query = psStringCopy ("SELECT * FROM flatcorrChipLink WHERE corr_id = %" PRId64); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, corr_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("flatcorr", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "flatcorrPending", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + return true; +} + +// select the flatcorr chip runs that have completed and for which there is no camera entry +// queue a new camera run for them +static bool inputimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(limit, config->args, "-limit", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipProcessedImfile.chip_id", "=="); + + psString query = pxDataGet("flatcorr_inputimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, chip_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("flatcorr", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "flatcorrPending", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + return true; +} + +static bool setflatcorrRunState(pxConfig *config, psS64 corr_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!strcmp(state, "reg") && + !strcmp(state, "new") && + !strcmp(state, "full")) + { + psError(PS_ERR_UNKNOWN, false, "invalid state: %s", state); + return false; + } + + char *query = "UPDATE flatcorrRun SET state = '%s' WHERE corr_id = %" PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, state, corr_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for corr_id %" PRId64, corr_id); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorr.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorr.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorr.h (revision 41104) @@ -0,0 +1,44 @@ +/* + * flatcorr.h + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef FLATCORR_H +#define FLATCORR_H 1 + +#include "pxtools.h" + +typedef enum { + FLATCORR_MODE_NONE = PXTOOL_MODE_NONE, + FLATCORR_MODE_DEFINEBYQUERY, + FLATCORR_MODE_DEFINERUN, + FLATCORR_MODE_ADDCHIP, + FLATCORR_MODE_DROPCHIP, + FLATCORR_MODE_ADDCAMERA, + FLATCORR_MODE_DROPCAMERA, + FLATCORR_MODE_ADVANCECAMERA, + FLATCORR_MODE_ADVANCEADDSTAR, + FLATCORR_MODE_PENDINGPROCESS, + FLATCORR_MODE_ADDPROCESS, + FLATCORR_MODE_UPDATERUN, + FLATCORR_MODE_INPUTEXP, + FLATCORR_MODE_INPUTIMFILE, +} flatcorrMode; + +pxConfig *flatcorrConfig(pxConfig *config, int argc, char **argv); + +#endif // FLATCORR_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorrConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorrConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/flatcorrConfig.c (revision 41104) @@ -0,0 +1,199 @@ +/* + * flatcorrConfig.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "flatcorr.h" + +pxConfig *flatcorrConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + // XXX need to allow multiple exp_ids + pxchipSetSearchArgs (definebyqueryArgs); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-det_type", 0, "define detrend type to be generated", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction",0, "define reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_expgroup", 0, "define exposure group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_filter", 0, "define filter", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tessalation", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_region", 0, "define region", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-make_correction", 0, "generate a correction image and add to detrend database", false); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "print the exposures that would be included in the detrend run and exit", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-inst", 0, "define camera", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-det_type", 0, "define detrend type to be generated", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_expgroup", 0, "define exposure group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_filter", 0, "define filter", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tessalation", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_region", 0, "define region", NULL); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-make_correction", 0, "generate a correction image and add to detrend database", false); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addchip + psMetadata *addchipArgs = psMetadataAlloc(); + psMetadataAddS64(addchipArgs, PS_LIST_TAIL, "-corr_id", 0, "define Flat Correction ID (required)", 0); + psMetadataAddS64(addchipArgs, PS_LIST_TAIL, "-chip_id", 0, "define Chip ID (required)", 0); + + // -dropchip + psMetadata *dropchipArgs = psMetadataAlloc(); + psMetadataAddS64(dropchipArgs, PS_LIST_TAIL, "-corr_id", 0, "define Flat Correction ID (required)", 0); + psMetadataAddS64(dropchipArgs, PS_LIST_TAIL, "-chip_id", 0, "define Chip ID (required)", 0); + + // -addcamera + psMetadata *addcameraArgs = psMetadataAlloc(); + psMetadataAddS64(addcameraArgs, PS_LIST_TAIL, "-corr_id", 0, "define Flat Correction ID (required)", 0); + psMetadataAddS64(addcameraArgs, PS_LIST_TAIL, "-chip_id", 0, "define Chip ID (required)", 0); + psMetadataAddS64(addcameraArgs, PS_LIST_TAIL, "-cam_id", 0, "define Camera ID (required)", 0); + + // -dropcamera + psMetadata *dropcameraArgs = psMetadataAlloc(); + psMetadataAddS64(dropcameraArgs, PS_LIST_TAIL, "-corr_id", 0, "define Flat Correction ID (required)", 0); + psMetadataAddS64(dropcameraArgs, PS_LIST_TAIL, "-cam_id", 0, "define Camera ID (required)", 0); + + // -advancecamera + psMetadata *advancecameraArgs = psMetadataAlloc(); + psMetadataAddU64 (advancecameraArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(advancecameraArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(advancecameraArgs, PS_LIST_TAIL, "-pretend", 0, "use the simple output format", false); + psMetadataAddStr(advancecameraArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + + // -advanceaddstar + psMetadata *advanceaddstarArgs = psMetadataAlloc(); + psMetadataAddU64 (advanceaddstarArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(advanceaddstarArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(advanceaddstarArgs, PS_LIST_TAIL, "-pretend", 0, "use the simple output format", false); + psMetadataAddStr(advanceaddstarArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + + // -pendingprocess + psMetadata *pendingprocessArgs = psMetadataAlloc(); + psMetadataAddU64 (pendingprocessArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingprocessArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddStr(pendingprocessArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + + // -addprocess (XXX need to add fault and stats) + psMetadata *addprocessArgs = psMetadataAlloc(); + psMetadataAddS64 (addprocessArgs, PS_LIST_TAIL, "-corr_id", 0, "add complete run for specified corr_id (required)", 0); + psMetadataAddStr (addprocessArgs, PS_LIST_TAIL, "-hostname", 0, "set hostname", NULL); + psMetadataAddS16 (addprocessArgs, PS_LIST_TAIL, "-code", 0, "set fault code", 0); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-corr_id", 0, "define correction id (required)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "set state (required)", NULL); + // XXX add mechanism to change this value: psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-set_correction", 0, "generate a correction image and add to detrend database", false); + + // -inputexp + psMetadata *inputexpArgs = psMetadataAlloc(); + psMetadataAddS64(inputexpArgs, PS_LIST_TAIL, "-corr_id", 0, "define correction id (required)", 0); + psMetadataAddBool(inputexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(inputexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -inputimfile + psMetadata *inputimfileArgs = psMetadataAlloc(); + psMetadataAddS64(inputimfileArgs, PS_LIST_TAIL, "-chip_id", 0, "define chip id (required)", 0); + psMetadataAddBool(inputimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(inputimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-corr_id", 0, "export this correction ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create a new, populated flat correction run", FLATCORR_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definerun", "create a new, empty flat correction run", FLATCORR_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-addchip", "add an existing chip run to a flat correction run", FLATCORR_MODE_ADDCHIP, addchipArgs); + PXOPT_ADD_MODE("-addcamera", "add an existing camera run to a flat correction run", FLATCORR_MODE_ADDCAMERA, addcameraArgs); + PXOPT_ADD_MODE("-advancecamera", "migrate completed chips to camera stage analysis", FLATCORR_MODE_ADVANCECAMERA, advancecameraArgs); + PXOPT_ADD_MODE("-advanceaddstar", "migrate completed exposures to addstar processing", FLATCORR_MODE_ADVANCEADDSTAR, advanceaddstarArgs); + PXOPT_ADD_MODE("-dropchip", "drop a chip from a flat correction run", FLATCORR_MODE_DROPCHIP, dropchipArgs); + PXOPT_ADD_MODE("-dropcamera", "drop an exposure (camera stage analysis)", FLATCORR_MODE_DROPCAMERA, dropcameraArgs); + PXOPT_ADD_MODE("-pendingprocess", "show flat correction runs needing to be processed", FLATCORR_MODE_PENDINGPROCESS, pendingprocessArgs); + PXOPT_ADD_MODE("-addprocess", "report completed flat correction analysis", FLATCORR_MODE_ADDPROCESS, addprocessArgs); + PXOPT_ADD_MODE("-updaterun", "change a flat calibration run's state", FLATCORR_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-inputexp", "list exposures for a correction run", FLATCORR_MODE_INPUTEXP, inputexpArgs); + PXOPT_ADD_MODE("-inputimfile", "list imfiles for a chip run", FLATCORR_MODE_INPUTIMFILE, inputimfileArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltool.c (revision 41104) @@ -0,0 +1,290 @@ +/* + * magictool.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "labeltool.h" + +static bool definelabelMode(pxConfig *config); +static bool updatelabelMode(pxConfig *config); +static bool deletelabelMode(pxConfig *config); +static bool listlabelMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = labeltoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(LABELTOOL_MODE_DEFINELABEL, definelabelMode); + MODECASE(LABELTOOL_MODE_UPDATELABEL, updatelabelMode); + MODECASE(LABELTOOL_MODE_DELETELABEL, deletelabelMode); + MODECASE(LABELTOOL_MODE_LISTLABEL, listlabelMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool definelabelMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + // XXX: perhaps we should have a default priority? + PXOPT_LOOKUP_S32(priority, config->args, "-set_priority", true, false); + + // optional + PXOPT_LOOKUP_BOOL(inactive, config->args, "-set_inactive", false); + PXOPT_LOOKUP_STR(comment, config->args, "-set_comment", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + LabelRow *row = LabelRowAlloc( + label, + priority, + inactive ? false : true, + comment + ); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to allocate Label object"); + return false; + } + if (!LabelInsertObject(config->dbh, row)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + return false; + } + + if (!LabelPrintObject(stdout, row, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(row); + return false; + } + + psFree(row); + + return true; +} + + +static bool updatelabelMode(pxConfig *config) +{ + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + + // optional (at least one is required) + PXOPT_LOOKUP_S32(priority, config->args, "-set_priority", false, false); + PXOPT_LOOKUP_BOOL(inactive, config->args, "-set_inactive", false); + PXOPT_LOOKUP_BOOL(active, config->args, "-set_active", false); + PXOPT_LOOKUP_STR(comment, config->args, "-set_comment", false, false); + + if (! (priority || active || inactive || comment) ) { + psError(PS_ERR_UNKNOWN, true, "at least one -set option is required\n"); + return false; + } + if (active && inactive) { + psError(PS_ERR_UNKNOWN, true, "only one of -active and -inactive may be supplied"); + return false; + } + + psString query = psStringCopy("UPDATE Label SET"); + char sep = ' '; + + if (priority) { + psStringAppend(&query, "%c priority = %d", sep, priority); + sep = ','; + } + if (active) { + psStringAppend(&query, "%c active = %d", sep, 1); + sep = ','; + } + if (inactive) { + psStringAppend(&query, "%c active = %d", sep, 0); + sep = ','; + } + if (comment) { + psStringAppend(&query, "%c comment = '%s'", sep, comment); + sep = ','; + } + + psStringAppend(&query, " WHERE label = '%s'", label); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool deletelabelMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + psString query = NULL; + + psStringAppend(&query, "DELETE FROM Label WHERE label = '%s'", label); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool listlabelMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "LIKE"); + + PXOPT_LOOKUP_BOOL(active, config->args, "-active", false); + PXOPT_LOOKUP_BOOL(inactive, config->args, "-inactive", false); + if (active && inactive) { + psError(PS_ERR_UNKNOWN, true, "only one of -active and inactive may be supplied"); + return false; + } + PXOPT_LOOKUP_BOOL(lowtohigh, config->args, "-lowtohigh", false); + PXOPT_LOOKUP_BOOL(hightolow, config->args, "-hightolow", false); + if (lowtohigh && hightolow) { + psError(PS_ERR_UNKNOWN, true, "only one of -lowtohigh and -hightolow may be supplied"); + return false; + } + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = psStringCopy("SELECT * FROM Label\n"); + + char *sep = " WHERE "; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + sep = " AND "; + } + psFree(where); + + if (active) { + psStringAppend(&query, "%s active\n", sep); + sep = " AND "; + } else if (inactive) { + psStringAppend(&query, "%s NOT active\n", sep); + sep = " AND "; + } + + if (lowtohigh || hightolow) { + char *order = lowtohigh ? "" : "DESC"; + psStringAppend(&query, "\nORDER BY priority %s\n", order); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("labeltool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "Label", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltool.h (revision 41104) @@ -0,0 +1,35 @@ +/* + * labeltool.h + * + * Copyright (C) 2010 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef LABELTOOL_H +#define LABELTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + LABELTOOL_MODE_NONE = 0x0, + LABELTOOL_MODE_DEFINELABEL, + LABELTOOL_MODE_UPDATELABEL, + LABELTOOL_MODE_DELETELABEL, + LABELTOOL_MODE_LISTLABEL, +} labeltoolMode; + +pxConfig *labeltoolConfig(pxConfig *config, int argc, char **argv); + +#endif // LABELTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/labeltoolConfig.c (revision 41104) @@ -0,0 +1,113 @@ +/* + * labeltoolConfig.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "labeltool.h" + +pxConfig *labeltoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definelabel + psMetadata *definelabelArgs = psMetadataAlloc(); + psMetadataAddStr(definelabelArgs, PS_LIST_TAIL, "-set_label", 0, "define label (required)", NULL); + psMetadataAddS64(definelabelArgs, PS_LIST_TAIL, "-set_priority", 0, "define priority (required)", 0); + psMetadataAddBool(definelabelArgs, PS_LIST_TAIL, "-set_inactive", 0, "set label inactive", false); + psMetadataAddStr(definelabelArgs, PS_LIST_TAIL, "-set_comment", 0, "define label comment", NULL); + psMetadataAddBool(definelabelArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updatelabel + psMetadata *updatelabelArgs = psMetadataAlloc(); + psMetadataAddStr(updatelabelArgs, PS_LIST_TAIL, "-label", 0, "search by label (LIKE comparison) (required)", NULL); + psMetadataAddS64(updatelabelArgs, PS_LIST_TAIL, "-set_priority", 0, "define new priority", 0); + psMetadataAddBool(updatelabelArgs, PS_LIST_TAIL, "-set_active", 0, "set label active", false); + psMetadataAddBool(updatelabelArgs, PS_LIST_TAIL, "-set_inactive", 0, "set label inactive", false); + psMetadataAddStr(updatelabelArgs, PS_LIST_TAIL, "-set_comment", 0, "define new label comment", NULL); + + // -deletelabel + psMetadata *deletelabelArgs = psMetadataAlloc(); + psMetadataAddStr(deletelabelArgs, PS_LIST_TAIL, "-label", 0, "label to delete (required)", NULL); + + // -listlabel + psMetadata *listlabelArgs = psMetadataAlloc(); + psMetadataAddStr(listlabelArgs, PS_LIST_TAIL, "-label", 0, "search by label (LIKE comparison)", NULL); + psMetadataAddBool(listlabelArgs, PS_LIST_TAIL, "-hightolow", 0, "order by priority high to low", false); + psMetadataAddBool(listlabelArgs, PS_LIST_TAIL, "-lowtohigh", 0, "order by priority low to high", false); + psMetadataAddBool(listlabelArgs, PS_LIST_TAIL, "-active", 0, "list active labels", false); + psMetadataAddBool(listlabelArgs, PS_LIST_TAIL, "-inactive", 0, "list inactive labels", false); + psMetadataAddU64(listlabelArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listlabelArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definelabel", "", LABELTOOL_MODE_DEFINELABEL, definelabelArgs); + PXOPT_ADD_MODE("-updatelabel", "", LABELTOOL_MODE_UPDATELABEL, updatelabelArgs); + PXOPT_ADD_MODE("-deletelabel", "", LABELTOOL_MODE_DELETELABEL, deletelabelArgs); + PXOPT_ADD_MODE("-listlabel", "", LABELTOOL_MODE_LISTLABEL, listlabelArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/laptool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/laptool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/laptool.c (revision 41104) @@ -0,0 +1,1588 @@ +/* + * laptool.c + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "laptool.h" + + +// Sequence level +static bool definesequenceMode(pxConfig *config); +static bool listsequenceMode(pxConfig *config); + +// Run level +static bool definerunMode(pxConfig *config); +static bool definerunbyreleaseMode(pxConfig *config); +static bool pendingrunMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool revertsassMode(pxConfig *config); + +// Exposure level +static bool pendingexpMode(pxConfig *config); +static bool exposuresMode(pxConfig *config); +static bool stacksMode(pxConfig *config); +static bool updateexpMode(pxConfig *config); + +static bool diffcheckMode(pxConfig *config); + +static bool inactiveexpMode(pxConfig *config); + +// Groups +static bool definegroupMode(pxConfig *config); +static bool pendinggroupMode(pxConfig *config); +static bool filtersforgroupMode(pxConfig *config); +static bool updategroupMode(pxConfig *config); +static bool revertgroupMode(pxConfig *config); +static bool listgroupMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = laptoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(LAPTOOL_MODE_DEFINESEQUENCE, definesequenceMode); + MODECASE(LAPTOOL_MODE_LISTSEQUENCE, listsequenceMode); + + MODECASE(LAPTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(LAPTOOL_MODE_DEFINERUNBYRELEASE, definerunbyreleaseMode); + MODECASE(LAPTOOL_MODE_PENDINGRUN, pendingrunMode); + MODECASE(LAPTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(LAPTOOL_MODE_REVERTSASS, revertsassMode); + + MODECASE(LAPTOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(LAPTOOL_MODE_EXPOSURES, exposuresMode); + MODECASE(LAPTOOL_MODE_STACKS, stacksMode); + MODECASE(LAPTOOL_MODE_UPDATEEXP, updateexpMode); + + MODECASE(LAPTOOL_MODE_DIFFCHECK, diffcheckMode); + + MODECASE(LAPTOOL_MODE_INACTIVEEXP, inactiveexpMode); + + MODECASE(LAPTOOL_MODE_DEFINEGROUP, definegroupMode); + MODECASE(LAPTOOL_MODE_PENDINGGROUP, pendinggroupMode); + MODECASE(LAPTOOL_MODE_FILTERSFORGROUP, filtersforgroupMode); + MODECASE(LAPTOOL_MODE_UPDATEGROUP, updategroupMode); + MODECASE(LAPTOOL_MODE_REVERTGROUP, revertgroupMode); + MODECASE(LAPTOOL_MODE_LISTGROUP, listgroupMode); + + default: + psAbort("invalid option (this should not happen)"); + } + psTrace("laptool", 9, "Attempting to free config\n"); + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_SUCCESS); + + FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +// Sequence level + +static bool definesequenceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(name, config->args, "-name", true, false); + PXOPT_LOOKUP_STR(description, config->args, "-description", true, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + lapSequenceRow *run = lapSequenceRowAlloc(0, // seq_id + name, + description); + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc lapSequence object"); + return(true); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!lapSequenceInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return(true); + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!lapSequencePrintObject(stdout, run, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + psFree(run); + + return true; +} + +static bool listsequenceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_STR(config->args, where, "-seq_name", "name", "LIKE"); + + psString query = pxDataGet("laptool_listsequence.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapSequence", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +// Run level +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(seq_id, config->args, "-seq_id", true, false); + PXOPT_LOOKUP_STR(projection_cell, config->args, "-projection_cell", true, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); + PXOPT_LOOKUP_F64(ra, config->args, "-ra", true, false); + PXOPT_LOOKUP_F64(decl, config->args, "-decl", true, false); + PXOPT_LOOKUP_F32(radius, config->args, "-radius", true, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", false, false); + PXOPT_LOOKUP_BOOL(all_obsmode, config->args, "-all_obsmode", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + lapRunRow *run = lapRunRowAlloc(0, // lap_id + seq_id, + tess_id, + projection_cell, + filter, + "new", // state + label, + dist_group, + NULL, // registered + 0, // fault + INT64_MAX, // quick_sass_id + INT64_MAX // final_sass_id + ); + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc lapRun object"); + return(true); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!lapRunInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return(true); + } + + + if (!lapRunPrintObject(stdout, run, !simple)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + psS64 lap_id = psDBLastInsertID(config->dbh); + + // Find the input exposures + psString query = pxDataGet("laptool_definerun.sql"); + if (!query) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + + // Add constraints to the exposure search: + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + + psMetadata *chipWhere = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, chipWhere, "-seq_id", "lapRun.seq_id", "=="); + + // This seems unnecessarily clunky. + if (!all_obsmode) { + // fprintf(stderr, "Not doing all obsmodes!\n"); + if (!psMetadataLookupStr(NULL,config->args,"-obsmode")) { + // fprintf(stderr, "Not doing all obsmodes and none specified!\n"); + psMetadataAddStr(where,PS_LIST_TAIL,"rawExp.obs_mode",0,"==","3PI"); + } + pxAddLabelSearchArgs (config, where, "-obsmode", "rawExp.obs_mode", "=="); + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + if (!pxspaceAddWhere(config, &whereClause, "rawExp")) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return(false); + } + + psString chipWhereClause = psDBGenerateWhereConditionSQL(chipWhere,NULL); + if (whereClause) { + psStringSubstitute(&query,whereClause,"@WHERE@"); + } + if (chipWhereClause) { + psStringSubstitute(&query,chipWhereClause,"@CHIPWHERE@"); + } + psFree(where); + psFree(chipWhere); + + // Fetch exposures + if (!p_psDBRunQuery(config->dbh, query)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return(true); + } + + // Insert the exposure data + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + // Add default values from this run: + psMetadataAddS64(row, PS_LIST_TAIL, "lap_id", 0, "", lap_id); + psMetadataAddS64(row, PS_LIST_TAIL, "pair_id", 0, "", INT64_MAX); + psMetadataAddStr(row, PS_LIST_TAIL, "data_state", 0, "", "new"); + lapExpRow *lapExp = lapExpObjectFromMetadata(row); + lapExp->lap_id = lap_id; + + if (!lapExpInsertObject(config->dbh,lapExp)) { + if (!lapExpPrintObject(stdout, lapExp, !simple)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(lapExp); + return(false); + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + psFree(output); + return(true); +} + +static bool definerunbyreleaseMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(seq_id, config->args, "-seq_id", true, false); + PXOPT_LOOKUP_STR(projection_cell, config->args, "-projection_cell", true, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", false, false); + PXOPT_LOOKUP_S64(rel_id, config->args, "-rel_id", false, false); + PXOPT_LOOKUP_STR(rel_name, config->args, "-release_name", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // Validate config + if ((!rel_name)&&(!rel_id)) { + // Die here. + } + + lapRunRow *run = lapRunRowAlloc(0, // lap_id + seq_id, + tess_id, + projection_cell, + filter, + "registered", // state + label, + dist_group, + NULL, // registered + 0, // fault + INT64_MAX, // quick_sass_id + INT64_MAX // final_sass_id + ); + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc lapRun object"); + return(true); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + if (!lapRunInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return(false); + } + + if (!lapRunPrintObject(stdout, run, !simple)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return(false); + } + + psS64 lap_id = psDBLastInsertID(config->dbh); + + psString query = pxDataGet("laptool_definerunbyrelease.sql"); + if (!query) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + + // Add constraints + psMetadata *relWhere = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, relWhere, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_STR(config->args, relWhere, "-release_name", "ippRelease.release_name", "=="); + PXOPT_COPY_S64(config->args, relWhere, "-rel_id", "ippRelease.rel_id", "=="); + PXOPT_COPY_TIME(config->args, relWhere, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, relWhere, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_RADEC(config->args, relWhere, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, relWhere, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, relWhere, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, relWhere, "-decl_max", "rawExp.decl", "<"); + + psMetadata *lapWhere = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, lapWhere, "-seq_id", "lapRun.seq_id", "=="); + PXOPT_COPY_STR(config->args, lapWhere, "-filter", "lapRun.filter", "=="); + + psString relWhereClause = psDBGenerateWhereConditionSQL(relWhere,NULL); + psString lapWhereClause = psDBGenerateWhereConditionSQL(lapWhere,NULL); + + if (relWhereClause) { + psStringSubstitute(&query,relWhereClause,"@RELWHERE@"); + } + if (lapWhereClause) { + psStringSubstitute(&query,lapWhereClause,"@LAPWHERE@"); + } + psFree(relWhere); + psFree(lapWhere); + + // Fetch exposures + if (!p_psDBRunQuery(config->dbh, query)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return(true); + } + + // Insert the exposure data + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + // Add default values from this run: + psMetadataAddS64(row, PS_LIST_TAIL, "lap_id", 0, "", lap_id); + psMetadataAddS64(row, PS_LIST_TAIL, "pair_id", 0, "", INT64_MAX); + psMetadataAddStr(row, PS_LIST_TAIL, "data_state", 0, "", "new"); + lapExpRow *lapExp = lapExpObjectFromMetadata(row); + lapExp->lap_id = lap_id; + + if (!lapExpInsertObject(config->dbh,lapExp)) { + if (!lapExpPrintObject(stdout, lapExp, !simple)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(lapExp); + return(false); + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + psFree(output); + return(true); +} + + + + + +static bool pendingrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "projection_cell", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + // PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-fault", "fault", "=="); + + pxAddLabelSearchArgs(config, where, "-label", "lapRun.label", "=="); + + psString query = pxDataGet("laptool_pendingrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + // require? PXOPT_LOOKUP_S64(lap_id, config->args, "-lap_id", true, false); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + + psMetadata *values = psMetadataAlloc(); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_COPY_STR(config->args, values, "-set_state", "state", "=="); + PXOPT_COPY_S16(config->args, values, "-fault", "fault", "=="); + PXOPT_COPY_STR(config->args, values, "-set_label", "label", "=="); + PXOPT_COPY_S64(config->args, values, "-set_quick_sass_id", "quick_sass_id", "=="); + PXOPT_COPY_S64(config->args, values, "-set_final_sass_id", "final_sass_id", "=="); + + long rows = psDBUpdateRows(config->dbh, "lapRun", where, values); + psFree(values); + + if (rows) { + // We're done with these exposures now, so mark them as inactive. + if (state) { + if ((strcmp(state,"drop") == 0)|| + (strcmp(state,"full") == 0)) { + values = psMetadataAlloc(); + psMetadataAddBool(values, PS_LIST_TAIL, "active", 0, "", false); + long exps = psDBUpdateRows(config->dbh, "lapExp", where, values); + + if (exps) { + return(true); + } + else { + return(true); // We shouldn't really fail if we didn't change anything. Maybe there's nothing to change. + } + } + } + } + + return(true); +} + +static bool revertsassMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sass_id", "final_sass_id", "=="); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("laptool_revertsass.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n WHERE %s", whereClause); + psFree(whereClause); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + long numUpdated = psDBAffectedRows(config->dbh); + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have updated only one row. updated %ld", numUpdated); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +// Exposure level + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + + psString query = pxDataGet("laptool_pendingexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + } + psStringAppend(&query, " ORDER BY rawExp.dateobs "); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n %s", limitString); + psFree(limitString); + } + + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool exposuresMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "lapExp.exp_id", "=="); + + psString query = pxDataGet("laptool_exposures.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + if (whereClause) { + psStringSubstitute(&query,whereClause,"@WHERE@"); + } + + psString limitString = NULL; + if (limit) { + limitString = psDBGenerateLimitSQL(limit); + psStringPrepend(&limitString, "\n"); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause, limitString ? limitString : "")) { + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(limitString); + psFree(query); + psFree(whereClause); + return(false); + } + psFree(limitString); + psFree(query); + psFree(whereClause); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return(true); + } + + if (!ippdbPrintMetadatas(stdout, output, "lapExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return(false); + } + + psFree(output); + return(true); +} + +static bool stacksMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // require? PXOPT_LOOKUP_S64(lap_id, config->args, "-lap_id", true, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lapRun.lap_id", "=="); + + psString query = pxDataGet("laptool_stacks.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringPrepend(&whereClause, "\n AND "); + psStringSubstitute(&query,whereClause,"@WHERE@"); + psFree(whereClause); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error %d",err); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapRunStacks", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return(true); +} + +static bool updateexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + // test for required opts? + // PXOPT_LOOKUP_S64(lap_id, config->args, "-lap_id", true, false); + // PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + // PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", false, false); + + PXOPT_LOOKUP_S64(set_chip_id, config->args, "-set_chip_id", false, false); + PXOPT_LOOKUP_S64(set_pair_id, config->args, "-set_pair_id", false, false); + PXOPT_LOOKUP_STR(set_data_state, config->args, "-set_data_state", false, false); + PXOPT_LOOKUP_BOOL(private, config->args, "-private", false); + PXOPT_LOOKUP_BOOL(public, config->args, "-public", false); + PXOPT_LOOKUP_BOOL(pairwise, config->args, "-pairwise", false); + PXOPT_LOOKUP_BOOL(nopairwise, config->args, "-nopairwise", false); + PXOPT_LOOKUP_BOOL(active, config->args, "-active", false); + PXOPT_LOOKUP_BOOL(inactive, config->args, "-inactive", false); + + + if (private && public) { + psError(PS_ERR_UNKNOWN, false, "only one of -private and -public may be selected"); + } + if (active && inactive) { + psError(PS_ERR_UNKNOWN, false, "only one of -active and -inactive may be selected"); + } + if (pairwise && nopairwise) { + psError(PS_ERR_UNKNOWN, false, "only one of -pairwise and -nopairwise may be selected"); + } + + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + + + psMetadata *values = psMetadataAlloc(); + if (set_chip_id) { + PXOPT_COPY_S64(config->args, values, "-set_chip_id", "chip_id", "=="); + } + if (set_pair_id) { + PXOPT_COPY_S64(config->args, values, "-set_pair_id", "pair_id", "=="); + } + if (private) { + psMetadataAddBool(values, PS_LIST_TAIL, "private", 0, "==", true); + } + else if (public) { + psMetadataAddBool(values, PS_LIST_TAIL, "private", 0, "==", false); + } + if (active) { + psMetadataAddBool(values, PS_LIST_TAIL, "active", 0, "==", true); + } + else if (inactive) { + psMetadataAddBool(values, PS_LIST_TAIL, "active", 0, "==", false); + } + if (pairwise) { + psMetadataAddBool(values, PS_LIST_TAIL, "pairwise", 0, "==", true); + } + else if (nopairwise) { + psMetadataAddBool(values, PS_LIST_TAIL, "pairwise", 0, "==", false); + } + if (set_data_state) { + PXOPT_COPY_STR(config->args, values, "-set_data_state", "data_state", "=="); + } + long rows = psDBUpdateRows(config->dbh,"lapExp",where,values); + if (rows) { + return(true); + } +/* else { */ +/* return(false); */ +/* } */ + return(true); +} + +static bool diffcheckMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lapRun.lap_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + + psString query = pxDataGet("laptool_WSdiff_check.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringPrepend(&whereClause, "\n AND "); + psStringSubstitute(&query,whereClause,"@WHERE@"); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error %d",err); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapRunWarpStackSkycells", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return(true); +} + +static bool inactiveexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // PXOPT_LOOKUP_S64(lap_id, config->args, "-lap_id", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-lap_id", "lap_id", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("laptool_inactiveexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return(false); + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + if (whereClause) { + psStringSubstitute(&query,whereClause,"@WHERE@"); + } + + psString limitString = NULL; + if (limit) { + limitString = psDBGenerateLimitSQL(limit); + psStringPrepend(&limitString, "\n"); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause, limitString ? limitString : "")) { + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(limitString); + psFree(query); + psFree(whereClause); + return(false); + } + psFree(limitString); + psFree(query); + psFree(whereClause); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return(true); + } + + if (!ippdbPrintMetadatas(stdout, output, "lapExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return(false); + } + + psFree(output); + return(true); +} +// --------------------------- +// Group level (a collection of completed lapRuns for a given lapSequence projection cell and collection of filters + +static bool definegroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(seq_id, config->args, "-seq_id", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // the following insures that the -filter argument has been set up properly (adapted from pxAddLabelSearchArgs) + psMetadataItem *item = psMetadataLookup(config->args, "-filter"); + psAssert (item, "-filter argument not found in config->args"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "%s should be a multi container", "filter"); + psAssert (item->data.list->n, "%s should at least have a place-holder", "filter"); + psMetadataItem *entry = (psMetadataItem *)item->data.list->head->data; + psAssert (entry, "%s should at least have a place-holder", "filter"); + // end of checking + + // Now if the ony entry is the place-holder then the user supplied no -filter arguments + // which are required + if (!entry->data.str) { + psError(PXTOOLS_ERR_ARGUMENTS, true, "at least one -filter is required"); + return false; + } + + int nFilters = item->data.list->n; + + psString query = psStringCopy("SELECT seq_id, tess_id, projection_cell,\n"); + + // We construct a query joining completed lap runs with the same lapSequence.seq_id, + // tess_id, and projection_cell + + // select the lap_ids for the various filters. These are only used for debugging. + psStringAppend(&query, "lap_id_%d", 0); + for (int i = 1; i < nFilters; i++) { + psStringAppend(&query, ", lap_id_%d", i); + } + + + // sub query for each supplied filter + char * lapRunForFilter = "(\nSELECT seq_id, tess_id, projection_cell, lap_id as 'lap_id_%d'\n" + "FROM lapRun\n" + "WHERE filter LIKE '%s'\n" + " AND lapRun.seq_id = %"PRId64"\n" + " AND (lapRun.state = 'done' or lapRun.state = 'full')\n" + " ) as lap_%d\n"; + + // loop over supplied filters and flesh out the query using the format above + psListIterator *iter = psListIteratorAlloc (item->data.list, PS_LIST_HEAD, true); + psMetadataItem *filterItem = NULL; + int i = -1; + while ((filterItem = psListGetAndIncrement(iter))) { + ++i; + if (i == 0) { + psStringAppend(&query, "\nFROM\n"); + } else { + psStringAppend(&query, "\nJOIN\n"); + } + + psString filter = filterItem->data.str; + psStringAppend(&query, lapRunForFilter, i, filter, seq_id, i); + + if (i != 0) { + psStringAppend(&query, "USING (seq_id, tess_id, projection_cell)\n"); + } + } + // now join to lapGroup + psStringAppend(&query, "\nLEFT JOIN lapGroup USING(seq_id, tess_id, projection_cell)\n"); + + // we only want projection cells which do not already of an entry + psStringAppend(&query, "\nWHERE lapGroup.projection_cell IS NULL\n"); + // Projection cell's don't have a convienient order. Queue runs in the order of lap_id + // psStringAppend(&query, "\nORDER by projection_cell\n"); + psStringAppend(&query, "\nORDER by lap_id_0\n"); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + if (!ippdbPrintMetadatas(stdout, output, "new_lapGroups", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + psString last_projection_cell = NULL; + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + psString tess_id = psMetadataLookupStr(NULL, row, "tess_id"); + psString projection_cell = psMetadataLookupStr(NULL, row, "projection_cell"); + if (last_projection_cell && !strcmp(last_projection_cell, projection_cell)) { + // duplicate lap runs for a filter will generate multiple rows + // Since we care about projection_cells not lapRuns per se this is not problem. + // Skip any duplicates. + continue; + } + last_projection_cell = projection_cell; + + lapGroupRow *group = lapGroupRowAlloc( + seq_id, + tess_id, + projection_cell, + "new", // state + label, + now, // registered + 0 // fault + ); + if (!group) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc lapGroup object"); + psFree(output); + psFree(now); + return(false); + } + + + if (!lapGroupInsertObject(config->dbh, group)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return(true); + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(now); + psFree(output); + + return(true); +} + +static bool pendinggroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "lapRun.label", "=="); + + psString query = pxDataGet("laptool_pendinggroup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psStringAppend(&query, "\nGROUP by projection_cell ORDER by MIN(lap_id)"); + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapGroup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +static bool filtersforgroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "projection_cell", "=="); + + psString query = pxDataGet("laptool_filtersforgroup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "filters", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool updategroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // check that all of the keys that define a lapGroup are supplied + PXOPT_LOOKUP_S64(seq_id, config->args, "-seq_id", true, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); + PXOPT_LOOKUP_STR(projection_cell, config->args, "-projection_cell", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "projection_cell", "=="); + + psMetadata *values = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, values, "-set_state", "state", "=="); + PXOPT_COPY_STR(config->args, values, "-set_label", "label", "=="); + PXOPT_COPY_S16(config->args, values, "-set_fault", "fault", "=="); + + if (!psListLength(values->list)) { + psFree(values); + psFree(where); + psError(PXTOOLS_ERR_ARGUMENTS, true, "must set at least one column"); + return false; + } + + long rows = psDBUpdateRows(config->dbh, "lapGroup", where, values); + if (rows < 1) { + psFree(values); + psError(PXTOOLS_ERR_SYS, true, "failed to update lapGroup for %" PRId64 " %s %s", seq_id, tess_id, projection_cell); + return false; + } + psFree(values); + + return(true); +} + +static bool revertgroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "projection_cell", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search arguments are required"); + return false; + } + + psString query = pxDataGet("laptool_revertgroup.sql"); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + psFree(where); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("laptool", PS_LOG_INFO, "Updated %" PRIu64 " lapGroups", affected); + + return true; +} + +static bool listgroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psError(PXTOOLS_ERR_SYS, true, "not yet implemented"); + return false; + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-seq_id", "seq_id", "=="); + PXOPT_COPY_STR(config->args, where, "-seq_name", "name", "LIKE"); + + psString query = pxDataGet("laptool_listsequence.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("laptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "lapSequence", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/laptool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/laptool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/laptool.h (revision 41104) @@ -0,0 +1,37 @@ +/* + * laptool.h + */ + +#ifndef LAPTOOL_H +#define LAPTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + LAPTOOL_MODE_NONE = 0x0, + LAPTOOL_MODE_DEFINESEQUENCE, + LAPTOOL_MODE_LISTSEQUENCE, + LAPTOOL_MODE_DEFINERUN, + LAPTOOL_MODE_DEFINERUNBYRELEASE, + LAPTOOL_MODE_PENDINGRUN, + LAPTOOL_MODE_UPDATERUN, + LAPTOOL_MODE_PENDINGEXP, + LAPTOOL_MODE_EXPOSURES, + LAPTOOL_MODE_STACKS, + LAPTOOL_MODE_REVERTSASS, + LAPTOOL_MODE_UPDATEEXP, + LAPTOOL_MODE_DIFFCHECK, + LAPTOOL_MODE_INACTIVEEXP, + LAPTOOL_MODE_DEFINEGROUP, + LAPTOOL_MODE_PENDINGGROUP, + LAPTOOL_MODE_FILTERSFORGROUP, + LAPTOOL_MODE_UPDATEGROUP, + LAPTOOL_MODE_REVERTGROUP, + LAPTOOL_MODE_LISTGROUP, +} laptoolMode; + +pxConfig *laptoolConfig(pxConfig *config, int argc, char **argv); + +#endif // LAPTOOL_H + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/laptoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/laptoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/laptoolConfig.c (revision 41104) @@ -0,0 +1,270 @@ +/* + * laptoolConfig.c + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include +#include "pxtools.h" +#include "laptool.h" + +#define ADD_OPT(TYPE,TARG,NAME,COMMENT,DEFAULT) psMetadataAdd##TYPE(TARG, PS_LIST_TAIL, NAME, 0, COMMENT, DEFAULT) + +pxConfig *laptoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definesequence + psMetadata *definesequenceArgs = psMetadataAlloc(); + ADD_OPT(Str, definesequenceArgs, "-name", "short name for this LAP sequence (required)", NULL); + ADD_OPT(Str, definesequenceArgs, "-description", "define the description for this LAP sequence (required)", NULL); + ADD_OPT(Bool,definesequenceArgs, "-simple", "use the simple output format", false); + + // -listsequence + psMetadata *listsequenceArgs = psMetadataAlloc(); + ADD_OPT(S64, listsequenceArgs, "-seq_id", "search by LAP sequence ID", 0); + ADD_OPT(Str, listsequenceArgs, "-seq_name", "search by LAP sequence name", 0); + ADD_OPT(Bool,listsequenceArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, listsequenceArgs, "-limit", "limit result set to N items", 0); + + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + ADD_OPT(S64, definerunArgs, "-seq_id", "define the LAP sequence for this run (required)", 0); + ADD_OPT(Str, definerunArgs, "-projection_cell", "define the projection cell for this run (required)", NULL); + ADD_OPT(Str, definerunArgs, "-tess_id", "define the tessellation used (required)", NULL); + ADD_OPT(F64, definerunArgs, "-ra", "define RA center (required)", NAN); + ADD_OPT(F64, definerunArgs, "-decl", "define DEC center (required)", NAN); + ADD_OPT(F32, definerunArgs, "-radius", "define radius from center to consider (required)", NAN); + ADD_OPT(Str, definerunArgs, "-filter", "define the filter used (required)", NULL); + ADD_OPT(Str, definerunArgs, "-label", "define the label used", NULL); + ADD_OPT(Str, definerunArgs, "-dist_group", "define the distribution group for this data", NULL); + + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-obsmode", PS_META_DUPLICATE_OK, "search by obsmode", NULL); + ADD_OPT(Bool,definerunArgs, "-all_obsmode", "use all science obsmodes", false); + + ADD_OPT(Bool,definerunArgs, "-simple", "use the simple output format", false); + + // -definerunbyrelease + psMetadata *definerunbyreleaseArgs = psMetadataAlloc(); + ADD_OPT(S64, definerunbyreleaseArgs, "-seq_id", "define the LAP sequence for this run (required)", 0); + ADD_OPT(Str, definerunbyreleaseArgs, "-projection_cell", "define the projection cell for this run (required)", NULL); + ADD_OPT(Str, definerunbyreleaseArgs, "-tess_id", "define the tessellation used (required)", NULL); + ADD_OPT(Str, definerunbyreleaseArgs, "-filter", "define the filter used (required)", NULL); + ADD_OPT(Str, definerunbyreleaseArgs, "-label", "define the label used", NULL); + ADD_OPT(Str, definerunbyreleaseArgs, "-dist_group", "define the distribution group for this data", NULL); + ADD_OPT(S64, definerunbyreleaseArgs, "-rel_id", "define the release to copy", 0); + ADD_OPT(Str, definerunbyreleaseArgs, "-release_name", "define the release to copy", NULL); + + ADD_OPT(Bool, definerunbyreleaseArgs, "-simple", "use the simple output format", false); + psMetadataAddTime(definerunbyreleaseArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definerunbyreleaseArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddF64(definerunbyreleaseArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(definerunbyreleaseArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(definerunbyreleaseArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(definerunbyreleaseArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + + // -pendingrun + psMetadata *pendingrunArgs = psMetadataAlloc(); + ADD_OPT(S64, pendingrunArgs, "-seq_id", "search by LAP sequence ID", 0); + ADD_OPT(S64, pendingrunArgs, "-lap_id", "search by LAP run ID", 0); + ADD_OPT(Str, pendingrunArgs, "-projection_cell", "search by projection cell", NULL); + ADD_OPT(Str, pendingrunArgs, "-filter", "search by filter", NULL); + // ADD_OPT(Str, pendingrunArgs, "-label", "search by LAP run label", NULL); + psMetadataAddStr(pendingrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by LAP run label", NULL); + ADD_OPT(Str, pendingrunArgs, "-state", "search by LAP run state", NULL); + ADD_OPT(Str, pendingrunArgs, "-fault", "search by LAP run fault", NULL); + ADD_OPT(Bool,pendingrunArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, pendingrunArgs, "-limit", "limit result set to N items", 0); + + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + ADD_OPT(S64, listrunArgs, "-seq_id", "search by LAP sequence ID", 0); + ADD_OPT(S64, listrunArgs, "-lap_id", "search by LAP run ID", 0); + ADD_OPT(Str, listrunArgs, "-projection_cell", "search by projection cell", NULL); + ADD_OPT(Str, listrunArgs, "-filter", "search by filter", NULL); + // ADD_OPT(Str, listrunArgs, "-label", "search by LAP run label", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by LAP run label", NULL); + ADD_OPT(Str, listrunArgs, "-state", "search by LAP run state", NULL); + ADD_OPT(Str, listrunArgs, "-fault", "search by LAP run fault", NULL); + ADD_OPT(Bool,listrunArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, listrunArgs, "-limit", "limit result set to N items", 0); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + ADD_OPT(S64, updaterunArgs, "-lap_id", "search by lap run ID", 0); + ADD_OPT(Str, updaterunArgs, "-set_state", "set state", NULL); + ADD_OPT(S16, updaterunArgs, "-fault", "set fault code", INT16_MAX); + ADD_OPT(Str, updaterunArgs, "-set_label", "set label", NULL); + ADD_OPT(S64, updaterunArgs, "-set_quick_sass_id", "set quick stack sass_id", 0); + ADD_OPT(S64, updaterunArgs, "-set_final_sass_id", "set final stack sass_id", 0); + + // -revertsass + psMetadata *revertsassArgs = psMetadataAlloc(); + ADD_OPT(S64, revertsassArgs, "-lap_id", "search by lap run id", 0); + ADD_OPT(S64, revertsassArgs, "-sass_id", "sass id that is being removed", 0); + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + ADD_OPT(S64, pendingexpArgs, "-lap_id", "lap run ID", 0); + ADD_OPT(Str, pendingexpArgs, "-projection_cell", "projection cell to consider", NULL); + ADD_OPT(U64, pendingexpArgs, "-limit", "limit result set to N items", 0); + ADD_OPT(Bool,pendingexpArgs, "-simple", "use the simple output format", false); + + // -exposures + psMetadata *exposuresArgs = psMetadataAlloc(); + ADD_OPT(S64, exposuresArgs, "-lap_id", "search by lap run ID", 0); + ADD_OPT(S64, exposuresArgs, "-exp_id", "search by exp_id", 0); + ADD_OPT(Bool,exposuresArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, exposuresArgs, "-limit", "limit result set to N items", 0); + + // -stacks + psMetadata *stacksArgs = psMetadataAlloc(); + ADD_OPT(S64, stacksArgs, "-lap_id", "search by lap run ID", 0); + ADD_OPT(Bool,stacksArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, stacksArgs, "-limit", "limit result set to N items", 0); + + // -updateexp + psMetadata *updateexpArgs = psMetadataAlloc(); + ADD_OPT(S64, updateexpArgs, "-lap_id", "search by lap run ID", 0); + ADD_OPT(S64, updateexpArgs, "-exp_id", "search by exposure ID", 0); + ADD_OPT(S64, updateexpArgs, "-chip_id", "search by chip ID", 0); + ADD_OPT(S64, updateexpArgs, "-set_chip_id", "set the chip ID", 0); + ADD_OPT(S64, updateexpArgs, "-set_pair_id", "set the pair ID", 0); + ADD_OPT(Str, updateexpArgs, "-set_data_state", "set the lapExp data_state", NULL); + ADD_OPT(Bool,updateexpArgs, "-private", "set this exposure as private", 0); + ADD_OPT(Bool,updateexpArgs, "-public", "set this exposure as public", 0); + ADD_OPT(Bool,updateexpArgs, "-pairwise", "set this exposure to be pairwise", 0); + ADD_OPT(Bool,updateexpArgs, "-nopairwise", "set this exposure to not be pairwise", 0); + ADD_OPT(Bool,updateexpArgs, "-active", "set this exposure to active", 0); + ADD_OPT(Bool,updateexpArgs, "-inactive", "set this exposure to active", 0); + + // -diffcheck + psMetadata *diffcheckArgs = psMetadataAlloc(); + ADD_OPT(S64, diffcheckArgs, "-lap_id", "search by lap run ID", 0); + ADD_OPT(S64, diffcheckArgs, "-exp_id", "search by exposure ID", 0); + ADD_OPT(S64, diffcheckArgs, "-chip_id", "search by chip ID", 0); + ADD_OPT(S64, diffcheckArgs, "-warp_id", "search by warp ID", 0); + ADD_OPT(S64, diffcheckArgs, "-seq_id", "search by lap sequence ID", 0); + ADD_OPT(Str, diffcheckArgs, "-skycell_id", "search by skycell ID", 0); + ADD_OPT(Bool, diffcheckArgs, "-simple", "use simple output format", 0); + + // -inactiveexp + psMetadata *inactiveexpArgs = psMetadataAlloc(); + ADD_OPT(S64, inactiveexpArgs, "-lap_id", "search by lap run ID", 0); + ADD_OPT(Bool,inactiveexpArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, inactiveexpArgs, "-limit", "limit result set to N items", 0); + + // -definegroup + psMetadata *definegroupArgs = psMetadataAlloc(); + ADD_OPT(S64, definegroupArgs, "-seq_id", "search by lap sequence ID (required)", 0); + ADD_OPT(Str, definegroupArgs, "-set_label", "set label (required)", 0); + + psMetadataAddStr(definegroupArgs, PS_LIST_TAIL, + "-filter", PS_META_DUPLICATE_OK, + "search by filter (LIKE comparison, multiple OK)", NULL); + ADD_OPT(Bool,definegroupArgs, "-pretend", "do not actuallym modify the database", false); + ADD_OPT(Bool,definegroupArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, definegroupArgs, "-limit", "limit result set to N items", 0); + + // -pendinggroup + psMetadata *pendinggroupArgs = psMetadataAlloc(); + psMetadataAddStr(pendinggroupArgs, PS_LIST_TAIL, + "-label", PS_META_DUPLICATE_OK, + "search by label (multiple OK)", NULL); + ADD_OPT(S64, pendinggroupArgs, "-seq_id", "search by lap sequence ID (required)", 0); + + ADD_OPT(Bool,pendinggroupArgs, "-simple", "use the simple output format", false); + ADD_OPT(U64, pendinggroupArgs, "-limit", "limit result set to N items", 0); + + // -updategroup + psMetadata *updategroupArgs = psMetadataAlloc(); + ADD_OPT(S64, updategroupArgs, "-seq_id", "search by lap sequence ID (required)", 0); + ADD_OPT(Str, updategroupArgs, "-tess_id", "search by tess_id (required)", 0); + ADD_OPT(Str, updategroupArgs, "-projection_cell", "search by projection_cell (required)", 0); + ADD_OPT(Str, updategroupArgs, "-set_state", "set state", 0); + ADD_OPT(S16, updategroupArgs, "-set_fault", "set fault code", INT16_MAX); + ADD_OPT(Str, updategroupArgs, "-set_label", "set label", 0); + + // -revertgroup + psMetadata *revertgroupArgs = psMetadataAlloc(); + psMetadataAddStr(revertgroupArgs, PS_LIST_TAIL, + "-label", PS_META_DUPLICATE_OK, + "search by label (multiple OK)", NULL); + ADD_OPT(S64, revertgroupArgs, "-seq_id", "search by lap sequence ID", 0); + ADD_OPT(Str, revertgroupArgs, "-tess_id", "search by tess_id", 0); + ADD_OPT(Str, revertgroupArgs, "-projection_cell", "search by projection_cell", 0); + ADD_OPT(S16, revertgroupArgs, "-fault", "fault code", INT16_MAX); + + // -filtersforgroup + psMetadata *filtersforgroupArgs = psMetadataAlloc(); + ADD_OPT(S64, filtersforgroupArgs, "-seq_id", "search by lap sequence ID (required)", 0); + ADD_OPT(Str, filtersforgroupArgs, "-tess_id", "search by tess_id (required)", 0); + ADD_OPT(Str, filtersforgroupArgs, "-projection_cell", "search by projection_cell (required)", 0); + ADD_OPT(Bool,filtersforgroupArgs, "-simple", "use the simple output format", false); + + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definesequence", "", LAPTOOL_MODE_DEFINESEQUENCE, definesequenceArgs); + PXOPT_ADD_MODE("-listsequence", "", LAPTOOL_MODE_LISTSEQUENCE, listsequenceArgs); + PXOPT_ADD_MODE("-definerun", "", LAPTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-definerunbyrelease", "", LAPTOOL_MODE_DEFINERUNBYRELEASE, definerunbyreleaseArgs); + PXOPT_ADD_MODE("-pendingrun", "", LAPTOOL_MODE_PENDINGRUN, pendingrunArgs); + PXOPT_ADD_MODE("-listrun", "", LAPTOOL_MODE_PENDINGRUN, listrunArgs); + PXOPT_ADD_MODE("-updaterun", "", LAPTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-revertsass", "mode to remove the sass_id", LAPTOOL_MODE_REVERTSASS, revertsassArgs); + PXOPT_ADD_MODE("-pendingexp", "", LAPTOOL_MODE_PENDINGEXP, pendingexpArgs); + PXOPT_ADD_MODE("-exposures", "", LAPTOOL_MODE_EXPOSURES, exposuresArgs); + PXOPT_ADD_MODE("-stacks", "", LAPTOOL_MODE_STACKS, stacksArgs); + PXOPT_ADD_MODE("-updateexp", "", LAPTOOL_MODE_UPDATEEXP, updateexpArgs); + PXOPT_ADD_MODE("-diffcheck", "", LAPTOOL_MODE_DIFFCHECK, diffcheckArgs); + PXOPT_ADD_MODE("-inactiveexp", "", LAPTOOL_MODE_INACTIVEEXP, inactiveexpArgs); + PXOPT_ADD_MODE("-definegroup", "", LAPTOOL_MODE_DEFINEGROUP, definegroupArgs); + PXOPT_ADD_MODE("-pendinggroup", "", LAPTOOL_MODE_PENDINGGROUP, pendinggroupArgs); + PXOPT_ADD_MODE("-filtersforgroup", "", LAPTOOL_MODE_FILTERSFORGROUP, filtersforgroupArgs); + PXOPT_ADD_MODE("-updategroup", "", LAPTOOL_MODE_UPDATEGROUP, updategroupArgs); + PXOPT_ADD_MODE("-revertgroup", "", LAPTOOL_MODE_REVERTGROUP, revertgroupArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PXTOOLS_ERR_SYS, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstool.c (revision 41104) @@ -0,0 +1,1986 @@ +/* + * magicdstool.c + * + * Copyright (C) 2006-2009 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "magicdstool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool definecopyMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool todestreakMode(pxConfig *config); +static bool adddestreakedfileMode(pxConfig *config); +static bool advancerunMode(pxConfig *config); +static bool revertdestreakedfileMode(pxConfig *config); +static bool updatedestreakedfileMode(pxConfig *config); +static bool clearstatefaultsMode(pxConfig *config); +static bool getskycellsMode(pxConfig *config); +static bool toremoveMode(pxConfig *config); +static bool torevertMode(pxConfig *config); +static bool completedrevertMode(pxConfig *config); +static bool tocleanupMode(pxConfig *config); +static bool tofullfileMode(pxConfig *config); +static bool tocleanedfileMode(pxConfig *config); +static bool setfiletoupdateMode(pxConfig *config); +static bool destreakedfileMode(pxConfig *config); +static bool listrunMode(pxConfig *config); + +static bool setmagicDSRunState(pxConfig *config, psS64 magic_id, psString extraSetString, psMetadata *where, const char *state); +static bool validDSRunState(const char *state); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = magicdstoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(MAGICDSTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(MAGICDSTOOL_MODE_DEFINECOPY, definecopyMode); + MODECASE(MAGICDSTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(MAGICDSTOOL_MODE_TODESTREAK, todestreakMode); + MODECASE(MAGICDSTOOL_MODE_ADDDESTREAKEDFILE, adddestreakedfileMode); + MODECASE(MAGICDSTOOL_MODE_ADVANCERUN, advancerunMode); + MODECASE(MAGICDSTOOL_MODE_REVERTDESTREAKEDFILE,revertdestreakedfileMode); + MODECASE(MAGICDSTOOL_MODE_UPDATEDESTREAKEDFILE,updatedestreakedfileMode); + MODECASE(MAGICDSTOOL_MODE_CLEARSTATEFAULTS, clearstatefaultsMode); + MODECASE(MAGICDSTOOL_MODE_GETSKYCELLS, getskycellsMode); + MODECASE(MAGICDSTOOL_MODE_TOREMOVE, toremoveMode); + MODECASE(MAGICDSTOOL_MODE_TOREVERT, torevertMode); + MODECASE(MAGICDSTOOL_MODE_COMPLETEDREVERT, completedrevertMode); + MODECASE(MAGICDSTOOL_MODE_TOCLEANUP, tocleanupMode); + MODECASE(MAGICDSTOOL_MODE_TOFULLFILE, tofullfileMode); + MODECASE(MAGICDSTOOL_MODE_TOCLEANEDFILE, tocleanedfileMode); + MODECASE(MAGICDSTOOL_MODE_SETFILETOUPDATE, setfiletoupdateMode); + MODECASE(MAGICDSTOOL_MODE_DESTREAKEDFILE, destreakedfileMode); + MODECASE(MAGICDSTOOL_MODE_LISTRUN, listrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool definebyqueryMode(pxConfig *config) +{ + + // Required + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + + // Optional + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", false, false); + PXOPT_LOOKUP_STR(recoveryroot, config->args, "-recoveryroot", false, false); + PXOPT_LOOKUP_BOOL(noreplace, config->args, "-noreplace", false); + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(set_data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // search args + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_bg_id", "chip_bg_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_bg_id", "warp_bg_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id","magicRun.magic_id", "=="); + PXOPT_COPY_S32(config->args, where, "-streaks_max","streaks", "<="); + + pxAddLabelSearchArgs (config, where, "-label", "magicRun.label", "=="); // define using magicRun label + + ippStage stageNum = ippStringToStage(stage); + + psString query = NULL; + switch (stageNum) { + case IPP_STAGE_RAW: + query = pxDataGet("magicdstool_definebyquery_raw.sql"); + break; + case IPP_STAGE_CHIP: + query = pxDataGet("magicdstool_definebyquery_chip.sql"); + break; + case IPP_STAGE_CHIP_BG: + query = pxDataGet("magicdstool_definebyquery_chip_bg.sql"); + break; + case IPP_STAGE_CAMERA: + query = pxDataGet("magicdstool_definebyquery_camera.sql"); + break; + case IPP_STAGE_WARP: + query = pxDataGet("magicdstool_definebyquery_warp.sql"); + break; + case IPP_STAGE_WARP_BG: + query = pxDataGet("magicdstool_definebyquery_warp_bg.sql"); + break; + case IPP_STAGE_DIFF: + query = pxDataGet("magicdstool_definebyquery_diff.sql"); + break; + case IPP_STAGE_FAKE: + case IPP_STAGE_STACK: + psError(PXTOOLS_ERR_CONFIG, true, "%sRuns do not need to be destreaked", stage); + return false; + case IPP_STAGE_NONE: + psError(PXTOOLS_ERR_CONFIG, true, "%s is not a valid stage", stage); + return false; + default: + psError(PXTOOLS_ERR_PROG, true, "ippStageToString returned %d for invalid stage %s", + stageNum, stage); + return false; + } + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + char *rerun_flag = rerun ? "\n1 " : "\n0" ; + + if (stageNum != IPP_STAGE_DIFF) { + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQueryF(config->dbh, query, rerun_flag)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + } else { + // diff stage query has two types bothways and !bothways + // so we need to send the rerun flag and the where data twice + psString whereString = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereString, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, rerun_flag, whereString, rerun_flag, whereString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString); + psFree(query); + return false; + } + psFree(whereString); + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // Parse the list of runs ready to be destreaked + + if (!pretend && !psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; // Row of interest + psS64 stage_id = psMetadataLookupS64(NULL, row, "stage_id"); + psS64 exp_id = psMetadataLookupS64(NULL, row, "exp_id"); + psS64 magic_id = psMetadataLookupS64(NULL, row, "magic_id"); + psS64 inv_magic_id = psMetadataLookupS64(NULL, row, "inv_magic_id"); + psS64 cam_id = psMetadataLookupS64(NULL, row, "cam_id"); + psString magicRunLabel = psMetadataLookupStr(NULL, row, "label"); + psString magicRunDataGroup = psMetadataLookupStr(NULL, row, "data_group"); + psString magicRunWorkdir = psMetadataLookupStr(NULL, row, "workdir"); + + // if workdir is not supplied use the magicRun's + if (!workdir) { + workdir = magicRunWorkdir; + } + + psString outroot = NULL; + // set outroot to workdir/exp_id/stage for example /somewhere/424242/chip + psStringAppend(&outroot, "%s/%" PRId64 "/%s", workdir, exp_id, stage); + + // create a new magicRun for this group + magicDSRunRow *run = magicDSRunRowAlloc( + 0, // magic_ds_id + magic_id, + inv_magic_id, + "new", + stage, + stage_id, + cam_id, + set_label ? set_label : magicRunLabel, + set_data_group ? set_data_group : magicRunDataGroup, + outroot, + recoveryroot, + noreplace ? 0 :1, // re_place + 0, // remove + 0, // fault + note); // remove + + psFree(outroot); + if (!run) { + psAbort("failed to alloc magicDSRun object"); + } + + if (!pretend) { + if (!magicDSRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + psFree(output); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 magic_ds_id = psDBLastInsertID(config->dbh); // Assigned identifier + run->magic_ds_id = magic_ds_id; + } + + psArrayAdd(list, list->n, run); + psFree(run); + } + + if (!pretend && !psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(output); + + if (!magicDSRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + + psFree(list); + + return true; +} + + +// XXX This currently allows multiple destreak runs to be queued on the same exposure if there are multiple +// magicRuns selected! +static bool definecopyMode(pxConfig *config) +{ + // Required + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", true, false); + + // Optional + PXOPT_LOOKUP_STR(recoveryroot, config->args, "-recoveryroot", false, false); + PXOPT_LOOKUP_BOOL(noreplace, config->args, "-noreplace", false); + PXOPT_LOOKUP_STR(set_data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // search args + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "chipRun.exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diffRun.diff_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + PXOPT_COPY_S32(config->args, where, "-streaks_max", "magicMask.streaks", "<="); + + pxAddLabelSearchArgs (config, where, "-magic_label", "magicRun.label", "=="); // define magic label + psString labelName = NULL; // Name of label + if (strcmp(stage, "camera")) { + psStringAppend(&labelName, "%sRun.label", stage); + } else { + psStringAppend(&labelName, "camRun.label"); + } + pxAddLabelSearchArgs (config, where, "-stage_label", labelName, "=="); // define stageRun label + psFree(labelName); + + ippStage stageNum = ippStringToStage(stage); + + psString query = NULL; + switch (stageNum) { + case IPP_STAGE_RAW: + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "Raw stage is not appropriate for a copied destreak"); + return false; + case IPP_STAGE_CHIP: + query = pxDataGet("magicdstool_definecopy_chip.sql"); + break; + case IPP_STAGE_WARP: + query = pxDataGet("magicdstool_definecopy_warp.sql"); + break; + case IPP_STAGE_CAMERA: + query = pxDataGet("magicdstool_definecopy_camera.sql"); + break; + case IPP_STAGE_DIFF: + case IPP_STAGE_FAKE: + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "%s has not been coded.", stage); + return false; + case IPP_STAGE_STACK: + psError(PXTOOLS_ERR_CONFIG, true, "Stacks do not need to be destreaked"); + return false; + case IPP_STAGE_NONE: + psError(PXTOOLS_ERR_CONFIG, true, "%s is not a valid stage", stage); + return false; + default: + psError(PXTOOLS_ERR_PROG, true, "ippStageToString returned %d for invalid stage %s", + stageNum, stage); + return false; + } + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + const char *rerun_flag = rerun ? "" : "\n"; // String to give query to activate (or not) rerun + + if (stageNum != IPP_STAGE_DIFF) { + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQueryF(config->dbh, query, set_label, rerun_flag)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + } else { + // diff stage query has two types bothways and !bothways + // so we need to send the rerun flag and the where data twice + psString whereString = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereString, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, rerun_flag, whereString, rerun_flag, whereString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString); + psFree(query); + return false; + } + psFree(whereString); + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // Parse the list of runs ready to be destreaked + + if (!pretend && !psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; // Row of interest + psS64 stage_id = psMetadataLookupS64(NULL, row, "stage_id"); + psS64 exp_id = psMetadataLookupS64(NULL, row, "exp_id"); + psS64 magic_id = psMetadataLookupS64(NULL, row, "magic_id"); + psS64 inv_magic_id = psMetadataLookupS64(NULL, row, "inv_magic_id"); + psS64 cam_id = psMetadataLookupS64(NULL, row, "cam_id"); + psString magicRunLabel = psMetadataLookupStr(NULL, row, "label"); + psString magicRunDataGroup = psMetadataLookupStr(NULL, row, "data_group"); + + psString outroot = NULL; + // set outroot to workdir/exp_id/stage for example /somewhere/424242/chip + psStringAppend(&outroot, "%s/%" PRId64 "/%s", workdir, exp_id, stage); + + // create a new magicRun for this group + magicDSRunRow *run = magicDSRunRowAlloc( + 0, // magic_ds_id + magic_id, + inv_magic_id, + "new", + stage, + stage_id, + cam_id, + set_label ? set_label : magicRunLabel, + set_data_group ? set_data_group : magicRunDataGroup, + outroot, + recoveryroot, + noreplace ? 0 :1, // re_place + 0, // remove + 0, // fault + note); // remove + + psFree(outroot); + if (!run) { + psAbort("failed to alloc magicDSRun object"); + } + + if (!pretend) { + if (!magicDSRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + psFree(output); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS64 magic_ds_id = psDBLastInsertID(config->dbh); // Assigned identifier + run->magic_ds_id = magic_ds_id; + } + + psArrayAdd(list, list->n, run); + psFree(run); + } + + if (!pretend && !psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(output); + + if (!magicDSRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + + psFree(list); + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + + if (!strcmp(state, "update")) { + fprintf(stderr, "'-updaterun -set_state update' is not supported. Use -setfiletoupdate"); + return false; + } + + // optional + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(set_data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(set_note, config->args, "-set_note", false, false); + psString setString = NULL; + if (set_label) { + psStringAppend(&setString, ", label = '%s'", set_label); + } + if (set_data_group) { + psStringAppend(&setString, ", data_group = '%s'", set_data_group); + } + if (set_note) { + psStringAppend(&setString, ", note = '%s'", set_note); + } + + PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", false, false); + if (magic_ds_id) { + + return setmagicDSRunState(config, magic_ds_id, setString, NULL, state); + + } else if (!strcmp(state, "full")) { + psError(PS_ERR_UNKNOWN, true, "magic_ds_id is required to update run state to full"); + return false; + } + // we can transition by query as well + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "data_group", "LIKE"); + + if (psListLength(where->list) < 2) { + psError(PS_ERR_UNKNOWN, true, "at least 2 search arguments are required"); + return false; + } + + + PXOPT_LOOKUP_BOOL(noreplace, config->args, "-noreplace", false); + if (!noreplace) { + psMetadataAddS32(where, PS_LIST_TAIL, "re_place", 0, ">", 0); + } + bool result = setmagicDSRunState(config, magic_ds_id, setString, where, state); + psFree(where); + + return result; +} + + +static bool todestreakMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magicDSRun.magic_ds_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicDSRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString sql_file = NULL; + psStringAppend(&sql_file, "magicdstool_todestreak_%s.sql", stage); + + psString query = pxDataGet(sql_file); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement from %s", sql_file); + psFree(sql_file); + return false; + } + psFree(sql_file); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, magic_ds_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "todestreak", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +// update the magicked column for the underlying component +// Note: Must be called inside a transaction + +static bool +setMagicked(pxConfig *config, psS64 magic_ds_id, psString component, bool clearMagicked) +{ + // first query the magicDSRun to find the stage and the stage_id + psString query = "SELECT stage, stage_id, magic_id from magicDSRun where magic_ds_id = %" PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, magic_ds_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "magic_ds_id: %" PRId64 " not found", magic_ds_id); + psFree(output); + return true; + } + if (psArrayLength(output) > 1) { + psError(PS_ERR_UNKNOWN, true, "unexpected number of rows found %ld for magic_ds_id %" PRId64, + psArrayLength(output), magic_ds_id); + psFree(output); + return false; + } + psMetadata *row = output->data[0]; + + psString stage= psMetadataLookupStr(NULL, row, "stage"); + psS64 stage_id = psMetadataLookupS64(NULL, row, "stage_id"); + psS64 magic_id = psMetadataLookupS64(NULL, row, "magic_id"); + psFree(output); + + psS64 newMagickedValue; + if (clearMagicked) { + newMagickedValue = 0; + } else { + newMagickedValue = magic_id; + } + + ippStage stageNum = ippStringToStage(stage); + + // chose the appropriate query based on the stage + char *clearRunQuery = NULL; + switch (stageNum) { + case IPP_STAGE_RAW: + query = "UPDATE rawImfile SET magicked = %" PRId64 " where exp_id = %" PRId64 " AND class_id = '%s'"; + clearRunQuery = "UPDATE rawExp SET magicked = 0 where exp_id = %" PRId64; + break; + case IPP_STAGE_CHIP: + query = "UPDATE chipProcessedImfile SET magicked = %" PRId64 " where chip_id = %" PRId64 " AND class_id = '%s'"; + clearRunQuery = "UPDATE chipRun set magicked = 0 where chip_id = %" PRId64; + break; + case IPP_STAGE_CHIP_BG: + query = "UPDATE chipBackgroundImfile SET magicked = %" PRId64 " where chip_bg_id = %" PRId64 " AND class_id = '%s'"; + clearRunQuery = "UPDATE chipBackgroundRun SET magicked = 0 where chip_bg_id = %" PRId64; + break; + case IPP_STAGE_CAMERA: + query = NULL; + clearRunQuery = "UPDATE camRun SET magicked = 0 where cam_id = %" PRId64; + break; + case IPP_STAGE_WARP: + query = "UPDATE warpSkyfile SET magicked = %" PRId64 " where warp_id = %" PRId64 " AND skycell_id = '%s'"; + clearRunQuery = "UPDATE warpRun SET magicked = 0 where warp_id = %" PRId64; + break; + case IPP_STAGE_WARP_BG: + query = "UPDATE warpBackgroundSkyfile SET magicked = %" PRId64 " where warp_bg_id = %" PRId64 " AND skycell_id = '%s'"; + clearRunQuery = "UPDATE warpBackgroundRun SET magicked = 0 where warp_bg_id = %" PRId64; + break; + case IPP_STAGE_DIFF: + query = "UPDATE diffSkyfile SET magicked = %" PRId64 " where diff_id = %" PRId64 " AND skycell_id = '%s'"; + clearRunQuery = "UPDATE diffRun SET magicked = 0 where diff_id = %" PRId64; + break; + default: + psError(PS_ERR_UNKNOWN, true, "unexpected value for stage: %s found", stage); + return false; + } + + if (query) { + if (!p_psDBRunQueryF(config->dbh, query, newMagickedValue, stage_id, component)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + if (clearMagicked && clearRunQuery) { + if (!p_psDBRunQueryF(config->dbh, clearRunQuery, stage_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + return true; +} + +static bool +setRunMagicked(pxConfig *config, psS64 magic_ds_id) +{ + // first query the magicDSRun to find the stage and the stage_id + psString query = "SELECT stage, stage_id, magic_id from magicDSRun where magic_ds_id = %" PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, magic_ds_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "magicDSRun not found for magic_ds_id %" PRId64, magic_ds_id); + psFree(output); + return false; + } + if (psArrayLength(output) > 1) { + psError(PS_ERR_UNKNOWN, true, "unexpected number of rows found %ld for magic_ds_id %" PRId64, + psArrayLength(output), magic_ds_id); + psFree(output); + return false; + } + psMetadata *row = output->data[0]; + + psString stage = psMetadataLookupStr(NULL, row, "stage"); + psS64 stage_id = psMetadataLookupS64(NULL, row, "stage_id"); + psS64 magic_id = psMetadataLookupS64(NULL, row, "magic_id"); + + ippStage stageNum = ippStringToStage(stage); + + // chose the appropriate query based on the stage + switch (stageNum) { + case IPP_STAGE_RAW: + query = "UPDATE rawExp SET magicked = %" PRId64 " where exp_id = %" PRId64; + break; + case IPP_STAGE_CHIP: + query = "UPDATE chipRun SET magicked = %" PRId64 " where chip_id = %" PRId64; + break; + case IPP_STAGE_CHIP_BG: + query = "UPDATE chipBackgroundRun SET magicked = %" PRId64 " where chip_bg_id = %" PRId64; + break; + case IPP_STAGE_CAMERA: + query = "UPDATE camRun SET magicked = %" PRId64 " where cam_id = %" PRId64; + break; + case IPP_STAGE_WARP: + query = "UPDATE warpRun SET magicked = %" PRId64 " where warp_id = %" PRId64; + break; + case IPP_STAGE_WARP_BG: + query = "UPDATE warpBackgroundRun SET magicked = %" PRId64 " where warp_bg_id = %" PRId64; + break; + case IPP_STAGE_DIFF: + query = "UPDATE diffRun SET magicked = %" PRId64 " where diff_id = %" PRId64; + break; + default: + psError(PS_ERR_UNKNOWN, true, "unexpected value for stage: %s found", stage); + psFree(output); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, magic_id, stage_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(output); + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + return false; + } + + return true; +} + +static bool adddestreakedfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required values + PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(backup_path_base, config->args, "-backup_path_base", false, false); + PXOPT_LOOKUP_STR(recovery_path_base, config->args, "-recovery_path_base", false, false); + PXOPT_LOOKUP_BOOL(setmagicked, config->args, "-setmagicked", false); + PXOPT_LOOKUP_F32(streak_frac, config->args, "-streak_frac", false, false); + PXOPT_LOOKUP_F32(nondiff_frac, config->args, "-nondiff_frac", false, false); + PXOPT_LOOKUP_F32(run_time, config->args, "-run_time", false, false); + + if (setmagicked && (fault != 0)) { + psError(PS_ERR_UNKNOWN, true, " cannot setmagicked for faulted file"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (setmagicked) { + // set the image file's magicked flag + if (!setMagicked(config, magic_ds_id, component, false)) { + psError(PS_ERR_UNKNOWN, false, "setMagicked failed"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } + + if (!magicDSFileInsert(config->dbh, + magic_ds_id, + component, + backup_path_base, + recovery_path_base, + streak_frac, + nondiff_frac, + run_time, + fault, + "full" // data_state + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool updatedestreakedfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required values + PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(backup_path_base, config->args, "-backup_path_base", false, false); + PXOPT_LOOKUP_STR(recovery_path_base, config->args, "-recovery_path_base", false, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", false, false); + PXOPT_LOOKUP_F32(streak_frac, config->args, "-streak_frac", false, false); + PXOPT_LOOKUP_F32(nondiff_frac, config->args, "-nondiff_frac", false, false); + PXOPT_LOOKUP_F32(run_time, config->args, "-run_time", false, false); + + psString query = psStringCopy("UPDATE magicDSFile"); + + + char *initial_separator = " SET"; + char *sep = initial_separator; + if (fault) { + psStringAppend(&query, "%s fault = %d", sep, fault); + sep = ", "; + } + if (backup_path_base) { + psStringAppend(&query, "%s backup_path_base = '%s'", sep, backup_path_base); + sep = ", "; + } + if (recovery_path_base) { + psStringAppend(&query, "%s recovery_path_base = '%s'", sep, recovery_path_base); + sep = ", "; + } + if (data_state) { + psStringAppend(&query, "%s data_state = '%s'", sep, data_state); + sep = ", "; + } + if (streak_frac) { + psStringAppend(&query, "%s streak_frac = '%f'", sep, streak_frac); + sep = ", "; + } + if (nondiff_frac) { + psStringAppend(&query, "%s nondiff_frac = '%f'", sep, nondiff_frac); + sep = ", "; + } + if (run_time) { + psStringAppend(&query, "%s run_time = '%f'", sep, run_time); + sep = ", "; + } + if (sep == initial_separator) { + psFree(query); + psError(PS_ERR_UNKNOWN, true, "must set at least one value"); + return false; + } + + psStringAppend(&query, "\nWHERE magic_ds_id = %" PRId64 " AND component = '%s'\n", magic_ds_id, component); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + + +static bool advancerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magicDSRun.magic_ds_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicDSRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // look for completed magicDSRuns + psString query = pxDataGet("magicdstool_completed_runs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + psS64 magic_ds_id = psMetadataLookupS64(NULL, row, "magic_ds_id"); + psS64 magicked = psMetadataLookupS64(NULL, row, "magicked"); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // if re_place, set stageRun.magicked + bool setmagicked = psMetadataLookupBool(NULL, row, "re_place"); + if (setmagicked) { + if (magicked <= 0) { + if (!setRunMagicked(config, magic_ds_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change stageRun.magicked for magic_ds_id: %" PRId64, + magic_ds_id); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } else { + fprintf(stderr, "run is already marked as destreaked for magic_ds_id %" PRId64 "\n", magic_ds_id); + } + } + + // set magicDSRun.state to 'full' + if (!setmagicDSRunState(config, magic_ds_id, NULL, NULL, "full")) { + psError(PS_ERR_UNKNOWN, false, "failed to change magicDSRun.state for magic_ds_id: %" PRId64, + magic_ds_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + return true; +} + + +static bool revertdestreakedfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(i_am_sure, config->args, "-i_am_sure", true); + if (!i_am_sure) { + psError(PS_ERR_UNKNOWN, true, "Reverting destreaked files must be done carefully. -i_am_sure is required."); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magicDSRun.magic_ds_id", "=="); + PXOPT_COPY_STR(config->args, where, "-component", "component", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + psString queryFile = NULL; + bool stateIsUpdate = false; + bool toRestored = !strcmp(state, "goto_restored"); + if (state) { + if (! strcmp(state, "new") || toRestored) { + queryFile = "magicdstool_revertdestreakedfile.sql"; + } else if (!strcmp(state, "update")) { + queryFile = "magicdstool_revertupdated.sql"; + stateIsUpdate = true; + } else { + psError(PXTOOLS_ERR_SYS, true, "%s is not a valid value for state", state); + return false; + } + } else { + queryFile = "magicdstool_revertdestreakedfile.sql"; + } + psString query = pxDataGet(queryFile); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search arguments are required"); + return false; + } + psFree(where); + + if (stateIsUpdate && !fault) { + // If fault has not been supplied, don't revert update faults with + // the magic "do not update" value + // We don't do this for new runs because then they would never complete + // quality should be used to drop bad components + psStringAppend(&query, " AND magicDSFile.fault != %d", PXTOOL_DO_NOT_REVERT_FAULT); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "failed to revert"); + psFree(query); + return false; + } + psFree(query); + if (toRestored) { + // clear the underlying component's magicked value + if (!setMagicked(config, magic_ds_id, component, true)) { + psError(PS_ERR_UNKNOWN, false, "setMagicked failed"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; +} +static bool clearstatefaultsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + // new state + PXOPT_LOOKUP_STR(new_state, config->args, "-set_state", false, false); + // old state (required) + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magicDSRun.magic_ds_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + psString query = pxDataGet("magicdstool_clearstatefaults.sql"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search arguments are required"); + return false; + } + psFree(where); + + if (!new_state) { + if (!strcmp(state, "failed_revert")) { + new_state = "new"; + } else if (!strcmp(state, "failed_revert_ud")) { + new_state = "update"; + } else if (!strcmp(state, "failed_cleanup")) { + new_state = "goto_cleaned"; + } else if (!strcmp(state, "failed_restore")) { + new_state = "goto_restored"; + } else { + psError(PS_ERR_UNKNOWN, true, "unexpected value for state: %s", state); + return false; + } + } else { + if (!validDSRunState(new_state)) { + psError(PS_ERR_UNKNOWN, true, "unexpected value for new state: %s", new_state); + return false; + } + } + if (!p_psDBRunQueryF(config->dbh, query, new_state)) { + psError(PS_ERR_UNKNOWN, false, "failed to clear state faults"); + psFree(query); + return false; + } + psFree(query); + return true; +} + +static bool completedrevertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magicDSRun.magic_ds_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("magicdstool_completedrevert.sql"); + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + psString whereString = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereString, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, whereString ? whereString : "")) { + psFree(whereString); + psError(PS_ERR_UNKNOWN, false, "failed to revert"); + return false; + } + psFree(whereString); + psFree(query); + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + for (int i=0; idata[i]; + psS64 magic_ds_id = psMetadataLookupS64(NULL, row, "magic_ds_id"); + psString old_state = psMetadataLookupStr(NULL, row, "state"); + psString new_state; + if (!strcmp(old_state, "goto_censored")) { + new_state = "censored"; + } else if (!strcmp(old_state, "goto_restored")) { + new_state = "restored"; + } else { + psError(PXTOOLS_ERR_PROG, true, "unexpected state found: %s", old_state); + psFree(output); + return false; + } + char *query2 = "UPDATE magicDSRun SET state = '%s' WHERE magic_ds_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query2, new_state, magic_ds_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to set run magicDSRun.state to %s", new_state); + return false; + } + } + psFree(output); + + return true; +} + +static bool getskycellsMode(pxConfig *config) +{ + // required + // PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magicDSRun.magic_ds_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "warpSkyCellMap.class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyCellMap.skycell_id", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("magicdstool_getskycells.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = NULL; + if (psListLength(where->list)) { + whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringPrepend(&whereClause, "\n AND "); + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, whereClause, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicDiffSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool validDSRunState(const char *state) +{ + if (!((strcmp(state, "new") == 0) || + (strcmp(state, "full") == 0) || + (strcmp(state, "drop") == 0) || + (strcmp(state, "wait") == 0) || + (strcmp(state, "failed_restore") == 0) || + (strcmp(state, "failed_revert") == 0) || + (strcmp(state, "failed_revert_ud") == 0) || + (strcmp(state, "error_cleaned") == 0) || + (strcmp(state, "restored") == 0) || + (strcmp(state, "censored") == 0) || + (strcmp(state, "cleaned") == 0) || + (strcmp(state, "update") == 0) || + (strcmp(state, "goto_restored") == 0) || + (strcmp(state, "goto_censored") == 0) || + (strcmp(state, "goto_cleaned") == 0)) + ) { + return false; + } else { + return true; + } +} + +static bool setmagicDSRunState(pxConfig *config, psS64 magic_ds_id, psString extraSetStr, psMetadata *where, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + if (!validDSRunState(state)) { + psError(PS_ERR_UNKNOWN, false, + "invalid magicDSRun state: %s", state); + return false; + } + + psString query = NULL; + psStringAppend(&query, "UPDATE magicDSRun SET state = '%s' %s\n", state, extraSetStr ? extraSetStr : ""); + if (magic_ds_id) { + psStringAppend(&query, " WHERE magic_ds_id = %" PRId64, magic_ds_id); + } else if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search arguments are required"); + return false; + } + + if (!strcmp(state, "goto_cleaned")) { + // Don't set state back to goto_cleaned if it is already cleaned + psStringAppend(&query, " AND (magicDSRun.state != 'cleaned')"); + + // don't clean up magicDSRun's where stage is camera + psStringAppend(&query, " AND (magicDSRun.stage != 'camera')"); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for magic_id %" PRId64, magic_ds_id); + return false; + } + + return true; +} + +static bool toremoveMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magic_ds_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("magicdstool_toremove.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "toremove", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool torevertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magic_ds_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicDSRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString sql_file = NULL; + psStringAppend(&sql_file, "magicdstool_torevert_%s.sql", stage); + + psString query = pxDataGet(sql_file); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement from %s", sql_file); + psFree(sql_file); + return false; + } + psFree(sql_file); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "torevert", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool tocleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magic_ds_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicDSRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("magicdstool_tocleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "tocleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +// update magicDSFile.data_state to given value. +// afterwards, if all files in the magicDSRun have the new state, +// update the state for it as well +// shared code for the modes -tocleanedfile -tofullfile + +static bool change_file_data_state(pxConfig *config, psString data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // magic_id, component + PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + + psString query = pxDataGet("magicdstool_change_file_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!strcmp(data_state, "full")) { + // set the component's magicked flag + + if (!setMagicked(config, magic_ds_id, component, false)) { + psError(PS_ERR_UNKNOWN, false, "setMagicked failed"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state, magic_ds_id, component)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + query = pxDataGet("magicdstool_change_run_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, magic_ds_id, data_state)) { + psFree(query); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool tocleanedfileMode(pxConfig *config) +{ + return change_file_data_state(config, "cleaned"); +} +static bool tofullfileMode(pxConfig *config) +{ + return change_file_data_state(config, "full"); +} + +// a very specfic function to queue a cleaned magicDSFile to be updated +static bool setfiletoupdateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(magic_ds_id, config->args, "-magic_ds_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(recoveryroot, config->args, "-set_recoveryroot", false, false); + + psString query = pxDataGet("magicdstool_setfiletoupdate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString setHook = psStringCopy(""); + if (label) { + psStringAppend(&setHook, "\n , magicDSRun.label = '%s'", label); + } + if (recoveryroot) { + psStringAppend(&setHook, "\n , magicDSRun.recoveryroot = '%s'", recoveryroot); + } + + if (component) { + psStringAppend(&query, " AND (magicDSFile.component = '%s')", component); + } + + // we do not update components with the magic fault value. They are non-updateable + // (But can be recovered with "magicdstool -revertdestreakedfile -fault 26" (PXTOOL_DO_NOT_REVERT_FAULT) + psStringAppend(&query, " AND (magicDSFile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + + if (!p_psDBRunQueryF(config->dbh, query, setHook, magic_ds_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(setHook); + psFree(query); + + return true; +} + + +static bool destreakedfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magic_ds_id", "=="); + PXOPT_COPY_STR(config->args, where, "-component", "component", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicDSRun.label", "=="); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + + psString query = pxDataGet("magicdstool_destreakedfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search arguments are required"); + return false; + } + psFree(where); + + if (faulted) { + psStringAppend(&query, " AND magicDSFile.fault > 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicDSFile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +static bool listrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-magic_ds_id", "magic_ds_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicDSRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + + psString query = pxDataGet("magicdstool_listrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search arguments are required"); + return false; + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magicdstool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicDSRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstool.h (revision 41104) @@ -0,0 +1,50 @@ +/* + * magicdstool.h + * + * Copyright (C) 2007 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef MAGICDSTOOL_H +#define MAGICDSTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + MAGICDSTOOL_MODE_NONE = 0x0, + MAGICDSTOOL_MODE_DEFINEBYQUERY, + MAGICDSTOOL_MODE_DEFINECOPY, + MAGICDSTOOL_MODE_UPDATERUN, + MAGICDSTOOL_MODE_TODESTREAK, + MAGICDSTOOL_MODE_ADDDESTREAKEDFILE, + MAGICDSTOOL_MODE_ADVANCERUN, + MAGICDSTOOL_MODE_REVERTDESTREAKEDFILE, + MAGICDSTOOL_MODE_CLEARSTATEFAULTS, + MAGICDSTOOL_MODE_GETSKYCELLS, + MAGICDSTOOL_MODE_TOREMOVE, + MAGICDSTOOL_MODE_TOREVERT, + MAGICDSTOOL_MODE_COMPLETEDREVERT, + MAGICDSTOOL_MODE_TOCLEANUP, + MAGICDSTOOL_MODE_TOCLEANEDFILE, + MAGICDSTOOL_MODE_TOFULLFILE, + MAGICDSTOOL_MODE_SETFILETOUPDATE, + MAGICDSTOOL_MODE_UPDATEDESTREAKEDFILE, + MAGICDSTOOL_MODE_LISTRUN, + MAGICDSTOOL_MODE_DESTREAKEDFILE, +} MAGICDStoolMode; + +pxConfig *magicdstoolConfig(pxConfig *config, int argc, char **argv); + +#endif // MAGICDSTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/magicdstoolConfig.c (revision 41104) @@ -0,0 +1,331 @@ +/* + * magicdstoolConfig.c + * + * Copyright (C) 2007 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "magicdstool.h" + +pxConfig *magicdstoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-stage", 0, "define stage (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-recoveryroot", 0, "define recovery directory", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-noreplace", 0, "do not replace input files with the destreaked versions", false); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by magicRun.label", NULL); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-chip_bg_id", 0, "search by chip_bg_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-warp_bg_id", 0, "search by warp_bg_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic_id", 0); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-streaks_max", 0, "maximum number of streaks", 0); + + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-rerun", 0, "queue a new run even if one already exists", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "don't queue runs just display what would be queued", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(definebyqueryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -definecopy + psMetadata *definecopyArgs = psMetadataAlloc(); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-stage", 0, "define stage (required)", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-recoveryroot", 0, "define recovery directory", NULL); + psMetadataAddBool(definecopyArgs, PS_LIST_TAIL, "-noreplace", 0, "do not replace input files with the destreaked versions", false); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_label", 0, "define label (required)", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data_group", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-stage_label", PS_META_DUPLICATE_OK, "search by stageRun.label", NULL); + psMetadataAddStr(definecopyArgs, PS_LIST_TAIL, "-magic_label", PS_META_DUPLICATE_OK, "search by magicRun.label", NULL); + psMetadataAddS64(definecopyArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddS64(definecopyArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddS64(definecopyArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(definecopyArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp_id", 0); + psMetadataAddS64(definecopyArgs, PS_LIST_TAIL, "-diff_id", 0, "search by diff_id", 0); + psMetadataAddS64(definecopyArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic_id", 0); + psMetadataAddS32(definecopyArgs, PS_LIST_TAIL, "-streaks_max", 0, "maximum number of streaks", 0); + psMetadataAddBool(definecopyArgs, PS_LIST_TAIL, "-rerun", 0, "queue a new run even if one already exists", false); + psMetadataAddBool(definecopyArgs, PS_LIST_TAIL, "-pretend", 0, "don't queue runs just display what would be queued", false); + psMetadataAddBool(definecopyArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(definecopyArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "set data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "set note", NULL); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magictool ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "define state", NULL); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-stage_id", 0, "define stage_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "define label (LIKE) comparison", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "data_group (LIKE) comparision", NULL); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-noreplace", 0, "only update runs with replace not set", false); + + // -addinputskyfile + psMetadata *addinputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID (required)", 0); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "define difftool ID (required)", 0); + psMetadataAddStr(addinputskyfileArgs, PS_LIST_TAIL, "-node", 0, "define symbolic node name (required)", NULL); + + // -todestreak + psMetadata *todestreakArgs = psMetadataAlloc(); + psMetadataAddStr(todestreakArgs, PS_LIST_TAIL, "-stage", 0, "limit query to stage (required)", NULL); + psMetadataAddS64(todestreakArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic Destreak ID", 0); + psMetadataAddS64(todestreakArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddS64(todestreakArgs, PS_LIST_TAIL, "-stage_id", 0, "search by stage ID", 0); + psMetadataAddStr(todestreakArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddU64(todestreakArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(todestreakArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -adddestreakedfile + psMetadata *adddestreakedfileArgs = psMetadataAlloc(); + psMetadataAddS64(adddestreakedfileArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(adddestreakedfileArgs, PS_LIST_TAIL, "-component", 0, "define component name (required)", NULL); + psMetadataAddStr(adddestreakedfileArgs, PS_LIST_TAIL, "-backup_path_base", 0, "define backup URI", NULL); + psMetadataAddStr(adddestreakedfileArgs, PS_LIST_TAIL, "-recovery_path_base", 0, "define recovery pixels URI", NULL); + psMetadataAddBool(adddestreakedfileArgs, PS_LIST_TAIL, "-setmagicked", 0, "update the magicked state of the file", false); + psMetadataAddF32(adddestreakedfileArgs, PS_LIST_TAIL, "-streak_frac", 0, "set fraction of pixels masked by streaks", 0); + psMetadataAddF32(adddestreakedfileArgs, PS_LIST_TAIL, "-nondiff_frac", 0, "set fraction of pixels masked because nondiffed", 0); + psMetadataAddF32(adddestreakedfileArgs, PS_LIST_TAIL, "-run_time", 0, "set the streaksremove run time for component ", 0); + psMetadataAddS16(adddestreakedfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -updatedestreakedfile + psMetadata *updatedestreakedfileArgs = psMetadataAlloc(); + psMetadataAddS64(updatedestreakedfileArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(updatedestreakedfileArgs, PS_LIST_TAIL, "-component", 0, "define component name (required)", NULL); + psMetadataAddStr(updatedestreakedfileArgs, PS_LIST_TAIL, "-state", 0, "current value for magicDSRun.state", NULL); + psMetadataAddStr(updatedestreakedfileArgs, PS_LIST_TAIL, "-backup_path_base", 0, "define backup URI", NULL); + psMetadataAddStr(updatedestreakedfileArgs, PS_LIST_TAIL, "-recovery_path_base", 0, "define recovery pixels URI", NULL); + psMetadataAddStr(updatedestreakedfileArgs, PS_LIST_TAIL, "-data_state", 0, "change data_state", NULL); + psMetadataAddBool(updatedestreakedfileArgs, PS_LIST_TAIL, "-setmagicked", 0, "update the magicked state of the file", false); + psMetadataAddF32(updatedestreakedfileArgs, PS_LIST_TAIL, "-streak_frac", 0, "set fraction of pixels masked by streaks", 0); + psMetadataAddF32(updatedestreakedfileArgs, PS_LIST_TAIL, "-nondiff_frac", 0, "set fraction of pixels masked because nondiffed", 0); + psMetadataAddF32(updatedestreakedfileArgs, PS_LIST_TAIL, "-run_time", 0, "set the streaksremove run time for component ", 0); + psMetadataAddS16(updatedestreakedfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -revertdestreakedfile + psMetadata *revertdestreakedfileArgs = psMetadataAlloc(); + psMetadataAddS64(revertdestreakedfileArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magictool de-streak ID", 0); + psMetadataAddStr(revertdestreakedfileArgs, PS_LIST_TAIL, "-state", 0, "current value for magicDSRun.state", NULL); + psMetadataAddStr(revertdestreakedfileArgs, PS_LIST_TAIL, "-component", 0, "search by component", NULL); + psMetadataAddS16(revertdestreakedfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertdestreakedfileArgs, PS_LIST_TAIL, "-i_am_sure", 0, "confirm that you know what you are doing", false); + psMetadataAddStr(revertdestreakedfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + + // -clearstatefaults + psMetadata *clearstatefaultsArgs = psMetadataAlloc(); + psMetadataAddStr(clearstatefaultsArgs, PS_LIST_TAIL, "-set_state", 0, "new value for state", NULL); + psMetadataAddStr(clearstatefaultsArgs, PS_LIST_TAIL, "-state", 0, "search by state (required)", NULL); + psMetadataAddS64(clearstatefaultsArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magictool de-streak ID", 0); + psMetadataAddS16(clearstatefaultsArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(clearstatefaultsArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + + // -getskycells + psMetadata *getskycellsArgs = psMetadataAlloc(); + psMetadataAddS64(getskycellsArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magic de-streak ID", 0); + psMetadataAddStr(getskycellsArgs, PS_LIST_TAIL, "-class_id", 0, "define class identifier", NULL); + psMetadataAddStr(getskycellsArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell identifier", NULL); + psMetadataAddBool(getskycellsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -toremove + psMetadata *toremoveArgs = psMetadataAlloc(); + psMetadataAddS64(toremoveArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic Destreak ID", 0); + psMetadataAddS64(toremoveArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddStr(toremoveArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddU64(toremoveArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(toremoveArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -torevert + psMetadata *torevertArgs = psMetadataAlloc(); + psMetadataAddS64(torevertArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic Destreak ID", 0); + psMetadataAddS64(torevertArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddStr(torevertArgs, PS_LIST_TAIL, "-stage", 0, "define output directory (required)", NULL); + psMetadataAddStr(torevertArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddU64(torevertArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(torevertArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -completedrevert + psMetadata *completedrevertArgs = psMetadataAlloc(); + psMetadataAddS64(completedrevertArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic Destreak ID", 0); + psMetadataAddStr(completedrevertArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddU64(completedrevertArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -advancerun + psMetadata *advancerunArgs = psMetadataAlloc(); + psMetadataAddS64(advancerunArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic Destreak ID", 0); + psMetadataAddStr(advancerunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddU64(advancerunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tocleanup + psMetadata *tocleanupArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanupArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic Destreak ID", 0); + psMetadataAddS64(tocleanupArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddStr(tocleanupArgs, PS_LIST_TAIL, "-stage", 0, "define output directory", NULL); + psMetadataAddStr(tocleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddU64(tocleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tocleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -tocleanedfile + psMetadata *tocleanedfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedfileArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magic_ds_id", 0); + psMetadataAddStr(tocleanedfileArgs, PS_LIST_TAIL, "-component", 0, "define component", NULL); + + // -tofullfile + psMetadata *tofullfileArgs = psMetadataAlloc(); + psMetadataAddS64(tofullfileArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magic_ds_id", 0); + psMetadataAddStr(tofullfileArgs, PS_LIST_TAIL, "-component", 0, "define component", NULL); + psMetadataAddBool(tofullfileArgs, PS_LIST_TAIL, "-setmagicked", 0, "update the magicked state of the file", false); + + // -setfiletoupdate + psMetadata *setfiletoupdateArgs = psMetadataAlloc(); + psMetadataAddS64(setfiletoupdateArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "define magic_ds_id", 0); + psMetadataAddStr(setfiletoupdateArgs, PS_LIST_TAIL, "-component", 0, "define component", NULL); + psMetadataAddStr(setfiletoupdateArgs, PS_LIST_TAIL, "-set_label", 0, "set new label", NULL); + psMetadataAddStr(setfiletoupdateArgs, PS_LIST_TAIL, "-set_recoveryroot", 0, "define new recovery directory", NULL); + + // -destreakedfile + psMetadata *destreakedfileArgs = psMetadataAlloc(); + psMetadataAddS64(destreakedfileArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic_ds_id", 0); + psMetadataAddStr(destreakedfileArgs, PS_LIST_TAIL, "-component", 0, "search by component", NULL); + psMetadataAddStr(destreakedfileArgs, PS_LIST_TAIL, "-stage", 0, "search by stage", NULL); + psMetadataAddS64(destreakedfileArgs, PS_LIST_TAIL, "-stage_id", 0, "search by magic_id", 0); + + psMetadataAddStr(destreakedfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by magicRun.label", NULL); + psMetadataAddS64(destreakedfileArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic_id", 0); + + psMetadataAddBool(destreakedfileArgs, PS_LIST_TAIL, "-faulted", 0, "list only faulted components", false); + psMetadataAddBool(destreakedfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(destreakedfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-magic_ds_id", 0, "search by magic_ds_id", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-stage", 0, "define stage", NULL); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-stage_id", 0, "search by magic_id", 0); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic_id", 0); + + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by magicRun.label", NULL); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-faulted", 0, "list faulted components", false); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(listrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create magic de-streak runs from magic runs", + MAGICDSTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definecopy", "create magic de-streak runs from magic runs for different path", + MAGICDSTOOL_MODE_DEFINECOPY, definecopyArgs); + PXOPT_ADD_MODE("-updaterun", "update state of magic de-streak run", + MAGICDSTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-todestreak", "show pending files", + MAGICDSTOOL_MODE_TODESTREAK, todestreakArgs); + PXOPT_ADD_MODE("-adddestreakedfile", "add a de-streaked file", + MAGICDSTOOL_MODE_ADDDESTREAKEDFILE, adddestreakedfileArgs); + PXOPT_ADD_MODE("-advancerun", "change state for runs that have finished destrreaking", + MAGICDSTOOL_MODE_ADVANCERUN, advancerunArgs); + PXOPT_ADD_MODE("-revertdestreakedfile", " revert a faulted de-streaked file", + MAGICDSTOOL_MODE_REVERTDESTREAKEDFILE, revertdestreakedfileArgs); + PXOPT_ADD_MODE("-clearstatefaults", " clear magicDSRuns in fault state", + MAGICDSTOOL_MODE_CLEARSTATEFAULTS, clearstatefaultsArgs); + PXOPT_ADD_MODE("-getskycells", "get skycell files ", + MAGICDSTOOL_MODE_GETSKYCELLS, getskycellsArgs); + PXOPT_ADD_MODE("-toremove", "backup images pending removal", + MAGICDSTOOL_MODE_TOREMOVE, toremoveArgs); + PXOPT_ADD_MODE("-torevert", "images to restore or revert", + MAGICDSTOOL_MODE_TOREVERT, torevertArgs); + + PXOPT_ADD_MODE("-completedrevert", "change state for runs that have finished reverting", + MAGICDSTOOL_MODE_COMPLETEDREVERT, completedrevertArgs); + PXOPT_ADD_MODE("-tocleanup", "destreak runs to clean up", + MAGICDSTOOL_MODE_TOCLEANUP, tocleanupArgs); + + PXOPT_ADD_MODE("-tofullfile", "set component's data_state to full", + MAGICDSTOOL_MODE_TOFULLFILE, tofullfileArgs); + PXOPT_ADD_MODE("-tocleanedfile", "set component's data_state to cleaned", + MAGICDSTOOL_MODE_TOCLEANEDFILE, tocleanedfileArgs); + PXOPT_ADD_MODE("-setfiletoupdate", "set component's data_state to update", + MAGICDSTOOL_MODE_SETFILETOUPDATE, setfiletoupdateArgs); + PXOPT_ADD_MODE("-updatedestreakedfile", "update parameters of destreaked file", + MAGICDSTOOL_MODE_UPDATEDESTREAKEDFILE, updatedestreakedfileArgs); + PXOPT_ADD_MODE("-destreakedfile", "list destreaked file", + MAGICDSTOOL_MODE_DESTREAKEDFILE, destreakedfileArgs); + PXOPT_ADD_MODE("-listrun", "list parameters of destreak run", + MAGICDSTOOL_MODE_LISTRUN, listrunArgs); + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/magictool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/magictool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/magictool.c (revision 41104) @@ -0,0 +1,1794 @@ +/* + * magictool.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "magictool.h" + +static bool definebyqueryMode(pxConfig *config); +static psS64 definerunMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool addinputskyfileMode(pxConfig *config); +static bool inputskyfileMode(pxConfig *config); +static bool totreeMode(pxConfig *config); +static bool inputtreeMode(pxConfig *config); +static bool reverttreeMode(pxConfig *config); +static bool toprocessMode(pxConfig *config); +static bool addresultMode(pxConfig *config); +static bool revertnodeMode(pxConfig *config); +static bool inputsMode(pxConfig *config); +static bool tomaskMode(pxConfig *config); +static bool addmaskMode(pxConfig *config); +static bool revertmaskMode(pxConfig *config); +static bool maskMode(pxConfig *config); +static bool censorrunMode(pxConfig *config); +static bool exposureMode(pxConfig *config); +static bool setgotocleanedMode(pxConfig *config); +static bool tocleanupMode(pxConfig *config); +static bool setworkdirstateMode(pxConfig *config); + +static bool setmagicRunState(pxConfig *config, psS64 magic_id, const char *state, psString setString); +static bool parseAndInsertNodeDeps(pxConfig *config, psS64 magic_id, const char *filename); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = magictoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(MAGICTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(MAGICTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(MAGICTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(MAGICTOOL_MODE_ADDINPUTSKYFILE, addinputskyfileMode); + MODECASE(MAGICTOOL_MODE_INPUTSKYFILE, inputskyfileMode); + MODECASE(MAGICTOOL_MODE_TOTREE, totreeMode); + MODECASE(MAGICTOOL_MODE_INPUTTREE, inputtreeMode); + MODECASE(MAGICTOOL_MODE_REVERTTREE, reverttreeMode); + MODECASE(MAGICTOOL_MODE_TOPROCESS, toprocessMode); + MODECASE(MAGICTOOL_MODE_ADDRESULT, addresultMode); + MODECASE(MAGICTOOL_MODE_REVERTNODE, revertnodeMode); + MODECASE(MAGICTOOL_MODE_INPUTS, inputsMode); + MODECASE(MAGICTOOL_MODE_TOMASK, tomaskMode); + MODECASE(MAGICTOOL_MODE_ADDMASK, addmaskMode); + MODECASE(MAGICTOOL_MODE_REVERTMASK, revertmaskMode); + MODECASE(MAGICTOOL_MODE_MASK, maskMode); + MODECASE(MAGICTOOL_MODE_CENSORRUN, censorrunMode); + MODECASE(MAGICTOOL_MODE_EXPOSURE, exposureMode); + MODECASE(MAGICTOOL_MODE_SETGOTOCLEANED, setgotocleanedMode); + MODECASE(MAGICTOOL_MODE_TOCLEANUP, tocleanupMode); + MODECASE(MAGICTOOL_MODE_SETWORKDIRSTATE, setworkdirstateMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // Required + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + // Optional + PXOPT_LOOKUP_STR(data_group, config->args, "-data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-note", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(multidiff, config->args, "-multidiff", false); + psMetadata *diffWhere = psMetadataAlloc(); // WHERE conditions for diffRuns + PXOPT_COPY_STR(config->args, diffWhere, "-diff_label", "diffRun.label", "=="); + PXOPT_COPY_S64(config->args, diffWhere, "-diff_id", "diff_id", "=="); + + psMetadata *queryWhere = psMetadataAlloc(); // WHERE conditions for everything else + PXOPT_COPY_S64(config->args, queryWhere, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, queryWhere, "-select_filter", "rawExp.filter", "=="); + + // Get list of exposures ready to magic + { + psString query; + if (multidiff) { + query = pxDataGet("magictool_definebyquery_select_multidiff.sql"); + } + else { + query = pxDataGet("magictool_definebyquery_select.sql"); + } + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // "available" means queue magic run even though the diffRun has skycells that did not complete + // "rerun" means we want a new run even if there's already a magic run defined for the exposure + PXOPT_LOOKUP_BOOL(available, config->args, "-available", false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + + psString diffWhereStr = NULL; // WHERE conditions for diffRuns + psString queryWhereStr = NULL; // WHERE conditions for entire query + if (!available) { + psStringAppend(&diffWhereStr, "\nAND diffRun.state = 'full'"); + } + // what if no skycells for the diff run completed? + + psStringAppend(&queryWhereStr, "\n%s magic_id IS NULL", queryWhereStr ? "AND" : "WHERE"); + psString rerunWhereStr = NULL; + if (rerun) { + psStringAppend(&rerunWhereStr, "\n WHERE magicRun.label = '%s'", label); + } + + // now add the user specified qualifiers + if (psListLength(diffWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(diffWhere, NULL); + psStringAppend(&diffWhereStr, "\nAND %s", whereClause); + psFree(whereClause); + } + if (psListLength(queryWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(queryWhere, NULL); + psStringAppend(&queryWhereStr, "\n%s %s", queryWhereStr ? "AND" : "WHERE", whereClause); + psFree(whereClause); + } + psFree(diffWhere); + psFree(queryWhere); + + // Ensure the WHERE strings have something + if (!diffWhereStr) { + diffWhereStr = psStringCopy(""); + } + if (!queryWhereStr) { + queryWhereStr = psStringCopy(""); + } + if (!rerunWhereStr) { + rerunWhereStr = psStringCopy(""); + } + + if (!p_psDBRunQueryF(config->dbh, query, diffWhereStr, diffWhereStr, rerunWhereStr, queryWhereStr)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(diffWhereStr); + psFree(queryWhereStr); + psFree(rerunWhereStr); + psFree(query); + return false; + } + psFree(diffWhereStr); + psFree(queryWhereStr); + psFree(rerunWhereStr); + psFree(query); + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "diffRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + + // Parse the list of exposures ready to magic + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString insert = pxDataGet("magictool_definebyquery_insert.sql"); // Insert query + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; // Row of interest + psS64 exp_id = psMetadataLookupS64(NULL, row, "exp_id"); // Exposure identifier + psS64 diff_id = psMetadataLookupS64(NULL, row, "diff_id"); // difference identifier + bool inverse = psMetadataLookupU64(NULL, row, "inverse"); // Inverse subtraction? Note types! + psString diff_data_group = psMetadataLookupStr(NULL, row, "diff_data_group"); + + // create a new magicRun for this group + magicRunRow *run = magicRunRowAlloc(0, + exp_id, + diff_id, + inverse, + "new", // state + workdir, + "dirty", // workdir_state + label, + data_group ? data_group : (diff_data_group ? diff_data_group : label), + dvodb, + registered, + 0, // fault + note); + if (!run) { + psAbort("failed to alloc magicRun object"); + } + + if (!magicRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + psFree(insert); + psFree(output); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psS64 magic_id = psDBLastInsertID(config->dbh); // Assigned identifier + run->magic_id = magic_id; + + psArrayAdd(list, list->n, run); + psFree(run); + + // Create a suitable insertion query for this run + psString thisInsert = psStringCopy(insert); + { + psString idString = NULL; + psStringAppend(&idString, "%" PRId64, magic_id); + psStringSubstitute(&thisInsert, idString, "@MAGIC_ID@"); + psFree(idString); + } + { + psString idString = NULL; + psStringAppend(&idString, "%" PRId64, diff_id); + psStringSubstitute(&thisInsert, idString, "@DIFF_ID@"); + psFree(idString); + } + + if (!p_psDBRunQueryF(config->dbh, thisInsert, magic_id, exp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(thisInsert); + psFree(insert); + psFree(output); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(thisInsert); + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(output); + + if (!magicRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + + psFree(list); + + return true; +} + +static psS64 definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_S64(diff_id, config->args, "-diff_id", false, false); + + // optional + PXOPT_LOOKUP_BOOL(inverse, config->args, "-inverse", false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + magicRunRow *run = magicRunRowAlloc( + 0, // ID + exp_id, + diff_id ? diff_id : PS_MAX_S64, + inverse, + "reg", // state + workdir, + "dirty", // workdir_state + label, + NULL, // data_group + dvodb, + registered, + 0, // fault + NULL + ); + + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc magicRun object"); + return false; + } + if (!magicRunInsertObject(config->dbh, run)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return false; + } + + // get the assigned magic_id + psS64 magic_id = psDBLastInsertID(config->dbh); + run->magic_id = magic_id; + + if (!magicRunPrintObject(stdout, run, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + psFree(run); + + return magic_id; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_BOOL(clearfault, config->args, "-clearfault", false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + + psString setString = NULL; + if (fault || clearfault) { + psStringAppend(&setString, ", fault = %d", fault); + } + if (note) { + psStringAppend(&setString, ", note = '%s'", note); + } + if (label) { + psStringAppend(&setString, ", label = '%s'", label); + } + + if (state) { + // set detRun.state to state + return setmagicRunState(config, magic_id, state, setString); + } + + return true; +} + + +static bool addinputskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", true, false); + PXOPT_LOOKUP_STR(node, config->args, "-node", true, false); + + magicInputSkyfileInsert( + config->dbh, + magic_id, + node + ); + + return true; +} + + +static bool inputskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + PXOPT_COPY_S64(config->args, where, "-diff_id", "diff_id", "=="); + PXOPT_COPY_STR(config->args, where, "-node", "node", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("magictool_inputskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "magicInputSkyfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicInputSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool totreeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("magictool_totree.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, magic_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "totree", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool inputtreeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", true, false); + + // Optional values + PXOPT_LOOKUP_STR(dep_file, config->args, "-dep_file", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (fault > 0) { + char *query = "UPDATE magicRun SET fault = %d WHERE magic_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, fault, magic_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to set fault for magic_id %" PRId64, magic_id); + return false; + } + return true; + } + + if (!parseAndInsertNodeDeps(config, magic_id, dep_file)) { + psError(PS_ERR_UNKNOWN, false, "failed to parse file"); + return false; + } + + return true; +} + +static bool reverttreeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicRun.label", "=="); + + psString query = psStringCopy("UPDATE magicRun SET fault = 0 WHERE state = 'new' AND fault != 0"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "failed to revert"); + return false; + } + psS32 numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("magictool", PS_LOG_INFO, "Reverted %d magic runs", numUpdated); + + return true; +} + + +static bool inputsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // Regrettably, there are multiple WHERE hooks which call the same things different names + psMetadata *templatesWhere = psMetadataAlloc(); // WHERE for selecting template + psMetadata *magicWhere = psMetadataAlloc(); // WHERE for selecting magic runs + + PXOPT_COPY_S64(config->args, templatesWhere, "-magic_id", "magicRun.magic_id", "=="); + PXOPT_COPY_STR(config->args, templatesWhere, "-node", "diffInputSkyfile.skycell_id", "=="); + + PXOPT_COPY_S64(config->args, magicWhere, "-magic_id", "magicRun.magic_id", "=="); + PXOPT_COPY_STR(config->args, magicWhere, "-node", "magicTree.node", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("magictool_inputs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString templatesWhereStr = psStringCopy(""); // WHERE for selecting template + psString magicWhereStr = psStringCopy(""); // WHERE for selecting magic runs + + if (psListLength(templatesWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(templatesWhere, NULL); + psStringAppend(&templatesWhereStr, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(templatesWhere); + + if (psListLength(magicWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(magicWhere, NULL); + psStringAppend(&magicWhereStr, "\nWHERE %s", whereClause); + psFree(whereClause); + } + psFree(magicWhere); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, + templatesWhereStr, templatesWhereStr, + magicWhereStr, magicWhereStr)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(templatesWhereStr); + psFree(magicWhereStr); + psFree(query); + return false; + } + psFree(templatesWhereStr); + psFree(magicWhereStr); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicNode", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +bool findBaseNodes(void *arg, pxNode *node) +{ + bool status = false; + psS64 done = psMetadataLookupS64(&status, node->data, "done"); + if (!status) { + psAbort("failed to lookup value for done column"); + } + + if ((!pxNodeHasChildren(node)) && (!done)) { + // if this node has no child and it's not 'done', then push it's data + // onto the void *array + psArrayAdd((psArray *)arg, 0, node->data); + return false; + } + + return true; +} + + +bool findReadyNodes(void *arg, pxNode *node) +{ + if (!node) { + // It's not there --- must have failed + return false; + } + + if (!node->data) { + // It's a leaf node, not done + return false; + } + + if (psMetadataLookupBool(NULL, node->data, "done")) { + // It's already done + return true; + } + + if (pxNodeHasChildren(node)) { + psListIterator *iter = psListIteratorAlloc(node->children, 0, false); + psMetadata *work = psMetadataCopy(NULL, node->data); + psMetadataRemoveKey(work, "dep"); + psMetadataRemoveKey(work, "done"); + pxNode *child = NULL; + while ((child = psListGetAndIncrement(iter))) { + psMetadata *data = child->data; + if (!data) { + // Child is a leaf node, not done + psFree(iter); + psFree(work); + return false; + } + + bool status = false; + psS64 done = psMetadataLookupS64(&status, data, "done"); + if (!status) { + psAbort("failed to lookup value for done column"); + } + psS64 bad = psMetadataLookupS64(&status, data, "bad"); + if (!status) { + psAbort("failed to lookup value for bad column"); + } + + if (!done || bad) { + // if a child isn't "done", give up on this node and continue + // to crawl the tree + psFree(iter); + psFree(work); + return true; + } + } + psFree(iter); + // if all this nodes children are done, then push it's data onto the + // void *array + psArrayAdd((psArray *)arg, 0, work); + psFree(work); + return false; + } + + return true; +} + + +static bool toprocessMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicRun.label", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString whereString = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereString, "\nAND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // First look for branch nodes that need to be processed. + // These get priority over skycells because they are from runs + // that are already in progress and there are fewer of them. + // When we looked for skycells first we got starved. + + // first find incomplete magicRuns + psString query = pxDataGet("magictool_toprocess_runs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // Find outstanding magicRuns in new state. + // We limit the query, but this is problematic. In practice do + // we need to? + // XXX: If the first 1000 magicRuns have no branch nodes ready + // but higher runs do we they won't be noticed. + // Perhaps have this limit be an argument. + { + psString limitString = psDBGenerateLimitSQL( 1000 ); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereString ? whereString : "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString); + psFree(query); + return false; + } + psFree(query); + + psArray *magicRuns = p_psDBFetchResult(config->dbh); + if (!magicRuns) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + + if (!psArrayLength(magicRuns)) { + // no pending magicRuns nothing to do + psTrace("magictool", PS_LOG_INFO, "no rows found"); + return true; + } + psArray *output = psArrayAllocEmpty(100); + + { + query = pxDataGet("magictool_toprocess_tree.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + for (psS64 index = 0; index < psArrayLength(magicRuns); index++) { + if (limit && (psArrayLength(output) >= limit)) { + break; + } + bool status; + psS64 magic_id = psMetadataLookupS64(&status, magicRuns->data[index], "magic_id"); + if (!status) { + psAbort("failed to lookup value for magic_id column"); + } + + psString whereString2 = NULL; + psStringAppend(&whereString2, "\nAND (magic_id = %" PRId64 ")", magic_id); + if (!p_psDBRunQueryF(config->dbh, query, whereString2 )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString2); + psFree(query); + return false; + } + psFree(whereString2); + psArray *magicTree = p_psDBFetchResult(config->dbh); + if (!magicTree) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + psS64 length = psArrayLength(magicTree); + if (!length) { + psTrace("magictool", PS_LOG_INFO, "no rows found for magic_id %" PRId64, magic_id); + psFree(magicTree); + continue; + } + + psHash *forest = psHashAlloc(length); + + // convert the array of metadata into a pxTree structure + for (long i = 0; i < length; i++) { + bool status; + psString node = psMetadataLookupStr(&status, magicTree->data[i], "node"); + if (!status) { + psAbort("failed to lookup value for node column"); + } + + psString dep = psMetadataLookupStr(&status, magicTree->data[i], "dep"); + if (!status) { + psAbort("failed to lookup value for dep column"); + } + + pxTreeBuilder(forest, node, dep, magicTree->data[i]); + + } + + // find the root of the tree + pxNode *root = psMemIncrRefCounter(psHashLookup(forest, "root")); + psFree(forest); + // pxTreePrint(stdout, root); + + // crawl through the tree and looking for nodes with children that are all + // "done" + pxTreeCrawl(root, findReadyNodes, output); + psFree(root); + psFree(magicTree); + } + + int len = psArrayLength(output); + if (len) { + if (limit) { + if (limit < psArrayLength(output)) { + // truncate the array + long arrayLength = psArrayLength(output); + for (long i = arrayLength - 1; i >= limit; i--) { + psArrayRemoveIndex(output, i); + } + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicMe", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } else { + limit -= len; + } + } + } + } + + // look for "inputs" (skycells) that need to processed + query = pxDataGet("magictool_toprocess_inputs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psStringAppend(&query, "\nORDER BY priority DESC, magic_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereString, whereString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereString); + psFree(query); + return false; + } + psFree(whereString); + psFree(query); + + psArray *skycellOutput = p_psDBFetchResult(config->dbh); + if (!skycellOutput) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + // merge the arrays so that we can print them all at once + if (psArrayLength(skycellOutput)) { + int len = psArrayLength(skycellOutput); + for (int i=0; i < len; i++) { + psArrayAdd(output, 0, skycellOutput->data[i]); + } + } + psFree(skycellOutput); + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "magicMe", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } else { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + } + psFree(output); + return true; +} + + +static bool addresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", true, false); + PXOPT_LOOKUP_STR(node, config->args, "-node", true, false); + + // optional + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!magicNodeResultInsert(config->dbh, + magic_id, + node, + path_base, + fault + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (fault == 0 && !strcmp(node, "root")) { + // Set the magicRun state + psString query = pxDataGet("magictool_setfull.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + // manually add constraint + psStringAppend(&query, " AND magic_id = %" PRId64, magic_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool revertnodeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + PXOPT_COPY_STR(config->args, where, "-node", "node", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "magicNodeResult.fault", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicRun.label", "=="); + + if (!psListLength(where->list)) { + psError(PS_ERR_UNKNOWN, false, "search parameters are required"); + psFree(where); + return false; + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + + // don't need a transaction because it is ok if the first query succeeds + // but the second query does not + psString query = pxDataGet("magictool_deletemask.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psStringAppend(&query, " AND %s", whereClause); + if (!p_psDBRunQuery(config->dbh, query)) { + psFree(whereClause); + psFree(query); + psError(PS_ERR_UNKNOWN, false, "failed to delete faulted masks"); + return false; + } + psFree(query); + + query = pxDataGet("magictool_revertnode.sql"); + if (!query) { + psFree(whereClause); + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "failed to revert"); + return false; + } + psFree(query); + + psS32 numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("magictool", PS_LOG_INFO, "Reverted %d magic nodes", numUpdated); + + return true; +} + + +static bool tomaskMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // look for "inputs" that need to processed + psString query = pxDataGet("magictool_tomask.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "tomask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool addmaskMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + // optional + PXOPT_LOOKUP_S32(streaks, config->args, "-streaks", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!magicMaskInsert(config->dbh, + magic_id, + NULL, + path_base, + streaks, + fault + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + return true; +} + +static bool revertmaskMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // Set to "new" + { + psString query = psStringCopy("UPDATE magicRun JOIN magicMask USING(magic_id) " + "SET magicRun.state = 'new' WHERE magicMask.fault != 0"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "magicMask"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "failed to revert"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psFree(query); + } + + // Delete failed attempt at mask + { + psString query = psStringCopy("DELETE FROM magicMask WHERE fault != 0"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "magicMask"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "failed to revert"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psFree(query); + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(where); + + return true; +} + + +static bool maskMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("magictool_mask.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "magicMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool setmagicRunState(pxConfig *config, psS64 magic_id, const char *state, psString setString) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!( + (strncmp(state, "new", 3) == 0) + || (strncmp(state, "full", 4) == 0) + || (strncmp(state, "drop", 4) == 0) + || (strncmp(state, "reg", 3) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid magicRun state: %s", state); + return false; + } + psString query = NULL; + psStringAppend(&query, "UPDATE magicRun SET state = '%s'", state); + if (setString) { + psStringAppend(&query, "%s", setString); + } + psStringAppend(&query, " WHERE magic_id = %" PRId64, magic_id);; + +// char *query = "UPDATE magicRun SET state = '%s' WHERE magic_id = %" PRId64; + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for magic_id %" PRId64, magic_id); + return false; + } + + return true; +} + +static bool parseAndInsertNodeDeps(pxConfig *config, psS64 magic_id, const char *filename) +{ + unsigned int nFail = 0; + psMetadata *deps = psMetadataConfigRead(NULL, &nFail, filename, false); + if (!deps) { + psError(PS_ERR_UNKNOWN, false, "failed to parse file: %s", filename); + return false; + } + if (nFail) { + psError(PS_ERR_UNKNOWN, false, "there were %d errors parsing file: %s", nFail, filename); + psFree(deps); + return false; + } + + psMetadataItem *item = NULL; + psMetadataIterator *iter = psMetadataIteratorAlloc(deps, 0, NULL); + while ((item = psMetadataGetAndIncrement(iter))) { + if (item->type != PS_DATA_STRING) { + psError(PS_ERR_UNKNOWN, false, "file: %s is in the wrong format", filename); + psFree(iter); + psFree(deps); + return false; + } + + char *name = item->name; + char *dependsOn = item->data.str; + + if (!magicTreeInsert( + config->dbh, + magic_id, + name, + dependsOn + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(iter); + psFree(deps); + return false; + } + } + psFree(iter); + psFree(deps); + + return true; +} + +static bool censorrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", false, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + + if (!magic_id) { + if (!exp_id && !label) { + psError(PS_ERR_UNKNOWN, true, "either -magic_id or exp_id and label is required"); + return false; + } + } + + // at least one of these required + PXOPT_COPY_S64(config->args, where, "-magic_id", "magic_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (!psListLength(where->list)) { + psError(PS_ERR_UNKNOWN, true, "either -exp_id or -magic_id is required"); + psFree(where); + return false; + } + + psString query = psStringCopy("UPDATE magicRun SET state = 'censored'"); + + psString whereClause = psDBGenerateWhereConditionSQL(where, "magicRun"); + psFree(where); + psStringAppend(&query, " WHERE %s", whereClause); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereClause); + psFree(query); + return false; + } + psFree(query); + + psS32 numUpdated = psDBAffectedRows(config->dbh); + if (numUpdated == 0) { + psError(PS_ERR_UNKNOWN, false, "failed to censor magicRun"); + psFree(whereClause); + return false; + } + + // Now queue any destreaked files to be re-verted + + // note: on failure pxmagicRestoreStage issues the rollback + if (!pxmagicRestoreStage(config, "raw", whereClause, "goto_censored")) { + psFree(whereClause); + return false; + } + if (!pxmagicRestoreStage(config, "chip", whereClause, "goto_censored")) { + psFree(whereClause); + return false; + } + if (!pxmagicRestoreStage(config, "camera", whereClause, "goto_censored")) { + psFree(whereClause); + return false; + } + if (!pxmagicRestoreStage(config, "warp", whereClause, "goto_censored")) { + psFree(whereClause); + return false; + } + if (!pxmagicRestoreStage(config, "diff", whereClause, "goto_censored")) { + psFree(whereClause); + return false; + } + + psFree(whereClause); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool exposureMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + PXOPT_COPY_BOOL(config->args, where, "-inverse", "magicRun.inverse", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("magictool_exposure.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psStringCopy(""); // WHERE restrictions + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereClause, "\nWHERE %s", clause); + psFree(clause); + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, whereClause, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + break; + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + break; + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + break; + } + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "magicMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} +static bool setgotocleanedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "magicRun.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "magicRun.label", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-data_group", "magicRun.data_group", "LIKE"); + + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + + psString query = psStringCopy("UPDATE magicRun SET workdir_state = 'goto_cleaned'\n"); + if (set_label) { + psStringAppend(&query, ", label = '%s'", set_label); + } + // This mode doubles as a revert function for cleanup errors + PXOPT_LOOKUP_BOOL(clearfault, config->args, "-clearfault", false); + if (!clearfault) { + psStringAppend(&query, "WHERE workdir_state = 'dirty'"); + } else { + psStringAppend(&query, "WHERE workdir_state = 'error_cleaned'"); + } + psStringAppend(&query, "\nAND (magicRun.state = 'full' OR magicRun.state = 'drop')"); + + // Require search parameters unless we're just clearing faults + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", clause); + psFree(clause); + psFree(where); + } else if ( !clearfault) { + psError(PS_ERR_UNKNOWN, false, "search parameters are required"); + psFree(where); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool setworkdirstateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(magic_id, config->args, "-magic_id", true, false); + PXOPT_LOOKUP_STR(workdir_state, config->args, "-set_workdir_state", true, false); + + if (strcmp(workdir_state, "cleaned") && strcmp(workdir_state, "error_cleaned")) { + psError(PS_ERR_UNKNOWN, true, "%s is not a valid value for workdir_state", workdir_state); + return false; + } + + psString query = NULL; + psStringAppend(&query, "UPDATE magicRun SET workdir_state = '%s' WHERE magic_id = %" PRId64, workdir_state, magic_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} +static bool tocleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-magic_id", "magicRun.magic_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "magicRun.label", "=="); + pxAddLabelSearchArgs (config, where, "-data_group", "magicRun.data_group", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("magictool_tocleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, magic_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("magictool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "tocleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/magictool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/magictool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/magictool.h (revision 41104) @@ -0,0 +1,52 @@ +/* + * magictool.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef MAGICTOOL_H +#define MAGICTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + MAGICTOOL_MODE_NONE = 0x0, + MAGICTOOL_MODE_DEFINEBYQUERY, + MAGICTOOL_MODE_DEFINERUN, + MAGICTOOL_MODE_UPDATERUN, + MAGICTOOL_MODE_ADDINPUTSKYFILE, + MAGICTOOL_MODE_INPUTSKYFILE, + MAGICTOOL_MODE_TOTREE, + MAGICTOOL_MODE_INPUTTREE, + MAGICTOOL_MODE_REVERTTREE, + MAGICTOOL_MODE_TOPROCESS, + MAGICTOOL_MODE_INPUTS, + MAGICTOOL_MODE_ADDRESULT, + MAGICTOOL_MODE_REVERTNODE, + MAGICTOOL_MODE_TOMASK, + MAGICTOOL_MODE_ADDMASK, + MAGICTOOL_MODE_REVERTMASK, + MAGICTOOL_MODE_MASK, + MAGICTOOL_MODE_CENSORRUN, + MAGICTOOL_MODE_EXPOSURE, + MAGICTOOL_MODE_SETGOTOCLEANED, + MAGICTOOL_MODE_TOCLEANUP, + MAGICTOOL_MODE_SETWORKDIRSTATE, +} MAGICtoolMode; + +pxConfig *magictoolConfig(pxConfig *config, int argc, char **argv); + +#endif // MAGICTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/magictoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/magictoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/magictoolConfig.c (revision 41104) @@ -0,0 +1,254 @@ +/* + * magictoolConfig.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "magictool.h" + +pxConfig *magictoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", 0, "define label (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-dvodb", 0, "define dvodb", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-note", 0, "define note", NULL); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-diff_id", 0, "search diff_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-exp_id", 0, "search exp_id", 0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-diff_label", 0, "select diff label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_filter", 0, "select filter", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-available", 0, "process what's immediately available?", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-rerun", 0, "generate new run even if existing?", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-multidiff", 0, "allow multiple diffs to be magicked with the same label", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "list results but do not queue", false); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddS64(definerunArgs, PS_LIST_TAIL, "-exp_id", 0, "define exp_id (required)", 0); + // following argument is for compatability + psMetadataAddS64(definerunArgs, PS_LIST_TAIL, "-diff_id", 0, "define diff_id", 0); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-inverse", 0, "use the inverse subtraction?", false); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-label", 0, "define label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-dvodb", 0, "define dvodb", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state (required)", NULL); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-clearfault", 0, "set fault to zero", NULL); + + // -addinputskyfile + psMetadata *addinputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(addinputskyfileArgs, PS_LIST_TAIL, "-node", 0, "define symbolic node name (required)", NULL); + + // -inputskyfile + psMetadata *inputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(inputskyfileArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magictool ID", 0); + psMetadataAddS64(inputskyfileArgs, PS_LIST_TAIL, "-diff_id", 0, "search by difftool ID", 0); + psMetadataAddStr(inputskyfileArgs, PS_LIST_TAIL, "-node", 0, "search by symbolic node name", NULL); + psMetadataAddU64(inputskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(inputskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -totree + psMetadata *totreeArgs = psMetadataAlloc(); + psMetadataAddS64(totreeArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddU64(totreeArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddStr(totreeArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddBool(totreeArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -inputtree + psMetadata *inputtreeArgs = psMetadataAlloc(); + psMetadataAddS64(inputtreeArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(inputtreeArgs, PS_LIST_TAIL, "-dep_file", 0, "order of operations dep. file", NULL); + psMetadataAddS16(inputtreeArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -reverttree + psMetadata *reverttreeArgs = psMetadataAlloc(); + psMetadataAddS64(reverttreeArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magictool ID", 0); + psMetadataAddS16(reverttreeArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(reverttreeArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + + // -inputs + psMetadata *inputsArgs = psMetadataAlloc(); + psMetadataAddS64(inputsArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magictool ID", 0); + psMetadataAddStr(inputsArgs, PS_LIST_TAIL, "-node", 0, "search by symbolic node name", NULL); + psMetadataAddU64(inputsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(inputsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -tooprocess + psMetadata *toprocessArgs = psMetadataAlloc(); + psMetadataAddS64(toprocessArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddU64(toprocessArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddStr(toprocessArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label", NULL); + psMetadataAddBool(toprocessArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addresult + psMetadata *addresultArgs = psMetadataAlloc(); + psMetadataAddS64(addresultArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-node", 0, "define symbolic node name (required)", NULL); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-path_base", 0, "define path_base", NULL); + psMetadataAddS16(addresultArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -revertnode + psMetadata *revertnodeArgs = psMetadataAlloc(); + psMetadataAddS64(revertnodeArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magictool ID", 0); + psMetadataAddStr(revertnodeArgs, PS_LIST_TAIL, "-node", 0, "search by node name", NULL); + psMetadataAddStr(revertnodeArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS16(revertnodeArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -tomask + psMetadata *tomaskArgs = psMetadataAlloc(); + psMetadataAddU64(tomaskArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tomaskArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addmask + psMetadata *addmaskArgs = psMetadataAlloc(); + psMetadataAddS64(addmaskArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID (required)", 0); + psMetadataAddStr(addmaskArgs, PS_LIST_TAIL, "-path_base", 0, "define path_base (required)", NULL); + psMetadataAddS32(addmaskArgs, PS_LIST_TAIL, "-streaks", 0, "define number of streaks", 0); + psMetadataAddS16(addmaskArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -revertmask + psMetadata *revertmaskArgs = psMetadataAlloc(); + psMetadataAddS64(revertmaskArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magictool ID", 0); + psMetadataAddS16(revertmaskArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -mask + psMetadata *maskArgs = psMetadataAlloc(); + psMetadataAddS64(maskArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID", 0); + psMetadataAddU64(maskArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(maskArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -censorun + psMetadata *censorrunArgs = psMetadataAlloc(); + psMetadataAddS64(censorrunArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID", 0); + psMetadataAddS64(censorrunArgs, PS_LIST_TAIL, "-exp_id", 0, "define exposure ID", 0); + psMetadataAddStr(censorrunArgs, PS_LIST_TAIL, "-label", 0, "define label", NULL); + + // -exposure + psMetadata *exposureArgs = psMetadataAlloc(); + psMetadataAddS64(exposureArgs, PS_LIST_TAIL, "-magic_id", 0, "define magictool ID", 0); + psMetadataAddBool(exposureArgs, PS_LIST_TAIL, "-inverse", 0, "select the inverse subtraction?", false); + psMetadataAddBool(exposureArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -setgotocleaned + psMetadata *setgotocleanedArgs = psMetadataAlloc(); + psMetadataAddS64(setgotocleanedArgs, PS_LIST_TAIL, "-magic_id", 0, "select by magictool ID", 0); + psMetadataAddS64(setgotocleanedArgs, PS_LIST_TAIL, "-exp_id", 0, "select by exposure ID)", 0); + psMetadataAddStr(setgotocleanedArgs, PS_LIST_TAIL, "-label", 0, "select by label", NULL); + psMetadataAddStr(setgotocleanedArgs, PS_LIST_TAIL, "-data_group", 0, "select by data_group", NULL); + psMetadataAddStr(setgotocleanedArgs, PS_LIST_TAIL, "-set_label", 0, "set new label", NULL); + psMetadataAddBool(setgotocleanedArgs, PS_LIST_TAIL, "-clearfault", 0, "clear cleanup errors", NULL); + + // -tocleanup + psMetadata *tocleanupArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanupArgs, PS_LIST_TAIL, "-magic_id", 0, "search by magic ID", 0); + psMetadataAddStr(tocleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddStr(tocleanupArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddU64(tocleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tocleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -setworkdirstate + psMetadata *setworkdirstateArgs = psMetadataAlloc(); + psMetadataAddS64(setworkdirstateArgs, PS_LIST_TAIL, "-magic_id", 0, "select by magictool ID (required)", 0); + psMetadataAddStr(setworkdirstateArgs, PS_LIST_TAIL, "-set_workdir_state", 0, "new workdir_state (required)", 0); + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "", MAGICTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definerun", "", MAGICTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-updaterun", "", MAGICTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-addinputskyfile", "", MAGICTOOL_MODE_ADDINPUTSKYFILE, addinputskyfileArgs); + PXOPT_ADD_MODE("-inputskyfile", "", MAGICTOOL_MODE_INPUTSKYFILE, inputskyfileArgs); + PXOPT_ADD_MODE("-totree", "", MAGICTOOL_MODE_TOTREE, totreeArgs); + PXOPT_ADD_MODE("-inputtree", "", MAGICTOOL_MODE_INPUTTREE, inputtreeArgs); + PXOPT_ADD_MODE("-reverttree", "", MAGICTOOL_MODE_REVERTTREE, reverttreeArgs); + PXOPT_ADD_MODE("-toprocess", "", MAGICTOOL_MODE_TOPROCESS, toprocessArgs); + PXOPT_ADD_MODE("-inputs", "", MAGICTOOL_MODE_INPUTS, inputsArgs); + PXOPT_ADD_MODE("-addresult", "", MAGICTOOL_MODE_ADDRESULT, addresultArgs); + PXOPT_ADD_MODE("-revertnode", "", MAGICTOOL_MODE_REVERTNODE, revertnodeArgs); + PXOPT_ADD_MODE("-tomask", "", MAGICTOOL_MODE_TOMASK, tomaskArgs); + PXOPT_ADD_MODE("-addmask", "", MAGICTOOL_MODE_ADDMASK, addmaskArgs); + PXOPT_ADD_MODE("-revertmask", "", MAGICTOOL_MODE_REVERTMASK, revertmaskArgs); + PXOPT_ADD_MODE("-mask", "", MAGICTOOL_MODE_MASK, maskArgs); + PXOPT_ADD_MODE("-censorrun", "", MAGICTOOL_MODE_CENSORRUN, censorrunArgs); + PXOPT_ADD_MODE("-exposure", "", MAGICTOOL_MODE_EXPOSURE, exposureArgs); + PXOPT_ADD_MODE("-setgotocleaned", "", MAGICTOOL_MODE_SETGOTOCLEANED, setgotocleanedArgs); + PXOPT_ADD_MODE("-tocleanup", "", MAGICTOOL_MODE_TOCLEANUP, tocleanupArgs); + PXOPT_ADD_MODE("-setworkdirstate", "", MAGICTOOL_MODE_SETWORKDIRSTATE, setworkdirstateArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetool.c (revision 41104) @@ -0,0 +1,1228 @@ +/* + * addtool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxmerge.h" +#include "mergetool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool pendingmergeMode(pxConfig *config); +static bool addmergedMode(pxConfig *config); +static bool listmergedMode(pxConfig *config); +static bool revertmergedMode(pxConfig *config); +static bool updatemergedMode(pxConfig *config); +static bool definebyquerymergecopyMode(pxConfig *config); +static bool listmergedvodbcopyMode(pxConfig *config); +static bool revertmergedvodbcopyMode(pxConfig *config); +static bool updatemergedvodbcopyMode(pxConfig *config); + + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = mergetoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(MERGETOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(MERGETOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(MERGETOOL_MODE_PENDINGMERGE, pendingmergeMode); + MODECASE(MERGETOOL_MODE_ADDMERGED, addmergedMode); + MODECASE(MERGETOOL_MODE_LISTMERGED, listmergedMode); + MODECASE(MERGETOOL_MODE_REVERTMERGED, revertmergedMode); + MODECASE(MERGETOOL_MODE_UPDATEMERGED, updatemergedMode); + MODECASE(MERGETOOL_MODE_DEFINEBYQUERYMERGECOPY, definebyquerymergecopyMode); + MODECASE(MERGETOOL_MODE_LISTMERGEDVODBCOPY, listmergedvodbcopyMode); + MODECASE(MERGETOOL_MODE_REVERTMERGEDVODBCOPY, revertmergedvodbcopyMode); + MODECASE(MERGETOOL_MODE_UPDATEMERGEDVODBCOPY, updatemergedvodbcopyMode); + + + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbRun.minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodbRun.minidvodb_group", "=="); + // PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(mergedvodb_path, config->args, "-set_mergedvodb_path", false, false); + PXOPT_LOOKUP_STR(mergedvodb, config->args, "-mergedvodb", true, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + //find the items to queue (uniq minidvodb_id) + psString dvodb_string = NULL; + psString bare_query = NULL; + +bare_query = pxDataGet("mergetool_find_minidvodb_id.sql"); + psStringAppend(&dvodb_string, "mergedvodbRun.mergedvodb = '%s'", mergedvodb); + +if (!bare_query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + psFree(where); + return false; + } + // Take the bare query and add the dvodb selector + psString query = NULL; + psStringAppend(&query, bare_query, dvodb_string); + psFree(dvodb_string); + psFree(bare_query); + + // use psDBGenerateWhereConditionSQL because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + psFree(where); + + + psStringAppend(&query, " GROUP BY minidvodb_id"); + + + psTrace("mergetool.c", PS_LOG_INFO,"query: \n\n%s\n\n",query); + +if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error, \n%s\n", query); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "addRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + +for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + minidvodbRunRow *row = minidvodbRunObjectFromMetadata(md); + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into minidvodbRun"); + psFree(output); + return false; + } + + if (!mergedvodb) { + psError(PS_ERR_UNKNOWN, false, "cannot queue merge run without a defined mergedvodb: minidvodb_id %" PRId64,row->minidvodb_id); + psFree(output); + return false; + } + if (!mergedvodb_path) { + psError(PS_ERR_UNKNOWN, false, "cannot queue merge run without a defined mergedvodb_path: minidvodb_id %" PRId64,row->minidvodb_id); + psFree(output); + return false; + } + + psFree(row); + +} + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } +for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 minidvodb_id =0; + + minidvodbRunRow *row = minidvodbRunObjectFromMetadata(md); + minidvodb_id = row->minidvodb_id; + + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into minidvodbRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxmergeQueueByMinidvodbID(config, + minidvodb_id, + mergedvodb, + mergedvodb_path + + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue mergedvodb %" PRId64, minidvodb_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + + } + +psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + + + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbRun.merge_id", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(mergedvodb_path, config->args, "-set_mergedvodb_path", false, false); + psString query = NULL; + + query = psStringCopy("UPDATE mergedvodbRun SET"); + + int cnt = 0; + psString comma = ","; + if (state) { + psStringAppend(&query, " state = '%s'", state); + cnt++; + } + + if (mergedvodb_path) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " mergedvodb_path = '%s'", mergedvodb_path); + cnt++; + } + + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psFree(query); + psFree(where); + + + return true; +} + +static bool pendingmergeMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "mergedvodbRun.minidvodb_id", "=="); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbRun.merge_id", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + + + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = NULL; + query = pxDataGet("mergetool_find_pending_merge.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "mergedvodbPendingMerge", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addmergedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_LOOKUP_U64(merge_id, config->args, "-merge_id", true, false); + PXOPT_LOOKUP_STR(mergedvodb, config->args, "-mergedvodb", true, false); + + // optional + +//PXOPT_LOOKUP_U64(merge_order, config->args, "-merge_order", false, false); + PXOPT_LOOKUP_F32(dtime_merge, config->args, "-dtime_merge", false, false); + PXOPT_LOOKUP_F32(dtime_verify, config->args, "-dtime_verify", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + + PXOPT_LOOKUP_TIME(epoch, config->args, "-epoch", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbRun.merge_id", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + + + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psString query = pxDataGet("mergetool_find_pendingmergeprocess.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClaus = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClaus); + psFree(whereClaus); + } + psFree(where); + psTrace("mergetool.c", PS_LOG_INFO, "query is:\n%s", query); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + +psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("mergetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + mergedvodbRunRow *pendingRow = mergedvodbRunObjectFromMetadata(output->data[0]); + psFree(output); + mergedvodbProcessedRow *row = mergedvodbProcessedRowAlloc( + pendingRow->merge_id, + 0, + dtime_verify ? dtime_verify : 0, + dtime_merge ? dtime_merge : 0, + dtime_script ? dtime_script : 0, + epoch, + fault + ); + + if (!mergedvodbProcessedInsertObject(config->dbh, row)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + psFree(pendingRow); + return false; + } + +//this finds the # of merged things (for the merge order) + psString query3 = NULL; + psStringAppend(&query3, "select count(*) from mergedvodbRun join mergedvodbProcessed using (merge_id) where state = 'full' and mergedvodb = '%s';", mergedvodb); + + if (!p_psDBRunQuery(config->dbh, query3)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query3); + return false; + } + psArray *output2 = p_psDBFetchResult(config->dbh); + if (!output2) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output2)) { + psTrace("mergetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + bool status; + psS64 m_order = psMetadataLookupS64(&status, output2->data[0], "count(*)"); + if (!status) { + + psAbort("failed to lookup value for count column"); + return false; + } + psString final = NULL; + psStringAppend(&final, "%" PRIu64, m_order); + //return false; + psFree(query3); + psFree(output2); + + + + //update the merge_order + + psString query4 = NULL; + psStringAppend(&query4, "update mergedvodbProcessed set merge_order = %"PRIu64,m_order); + psStringAppend(&query4," where merge_id = %" PRIu64, merge_id); + //printf("%s", query4); + if (!p_psDBRunQuery(config->dbh, query4)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query4); + return false; + } + // +psFree(query4); + +// since there is only one exp per 'new' set mergedvodbRun.state = 'full' + + psString query2 = NULL ; + + if (!fault) { + psStringAppend(&query2, "UPDATE mergedvodbRun SET state = 'full' WHERE merge_id = %'" PRIu64, row->merge_id); + + if (!p_psDBRunQuery(config->dbh, query2)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query2); + return false; + } + + } + + + + + + + psFree(row); + psFree(pendingRow); + + + + + + //commit the changes + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + + + //print the merge_order (why not!) + printf("%s", final); + psFree(final); + + + + + + + + + + + + return true; +} + + +static bool listmergedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "mergedvodbRun.minidvodb_id", "=="); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbRun.merge_id", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + psString query = NULL; + query = pxDataGet("mergetool_find_processed.sql"); + +if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + + + + // we either add AND (condition) or WHERE (condition): + if (psListLength(where->list) && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " AND fault != 0"); + } + if (psListLength(where->list) && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND fault = 0"); + } + if (!psListLength(where->list) && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " WHERE fault != 0"); + } + if (!psListLength(where->list) && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " WHERE fault = 0"); + } + +psFree(where); + + // order by merge_id + psStringAppend(&query, " ORDER BY merge_id"); + + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "addProcessedExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + + + return true; +} + + +static bool revertmergedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbRun.merge_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "mergedvodbRun.fault", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + { + psString query = NULL; + query = pxDataGet("mergetool_revertmergedvodbprocessed.sql"); + +if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(where); + return false; + } + psFree(query); + } + psFree(where); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + + return true; +} + +static bool updatemergedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + psMetadata *where = psMetadataAlloc(); + // PXOPT_LOOKUP_U64(minidvodb_id, config->args, "-merge_id", true, false); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbProcessed.merge_id", "=="); + PXOPT_LOOKUP_F32(dtime_merge, config->args, "-set_dtime_merge", false, false); + PXOPT_LOOKUP_F32(dtime_verify, config->args, "-set_dtime_verify", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-set_dtime_script", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psString query = psStringCopy("UPDATE mergedvodbProcessed JOIN mergedvodbRun USING (merge_id) SET "); +int cnt = 0; + psString comma = ","; + if (fault) { + psStringAppend(&query, " fault = %d", fault); + cnt++; + } + + + + if (dtime_verify) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " dtime_verify = %f", dtime_verify); + cnt++; + } + + if (dtime_merge) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " dtime_merge = %f", dtime_merge); + cnt++; + } + +if (dtime_script) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " dtime_script = %f", dtime_script); + cnt++; + } +if (state) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " state = '%s'", state); + cnt++; + } + + + + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error \n%s\n",query ); + psFree(query); + return false; + } + + psFree(query); + psFree(where); + + return true; +} + +static bool definebyquerymergecopyMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbRun.merge_id", "=="); + pxAddLabelSearchArgs (config, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); // define using camRun label + + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(destination_host, config->args, "-set_destination_host", false, false); + PXOPT_LOOKUP_STR(mergedvodb_rsync_path, config->args, "-set_mergedvodb_rsync_path", false, false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(last_merged, config->args, "-last_merged", false); + // prevent queueing an addRun if a given exposure has already been added to + // the given dvo database + psString dvodb_string = NULL; + psString bare_query = NULL; + if (destination_host) { + psTrace("mergedvodbtool.c", PS_LOG_INFO, "destination_host argument found (%s) using mergedvodbtool_find_merge_id_dvo.sql\n", destination_host); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("mergedvodbtool_find_merge_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "mergedvodbCopy.destination_host = '%s'", destination_host); + } else { + psError(PS_ERR_UNKNOWN, false, "cannot queue mergedvodbcopy run without a defined destination_host"); + + return false; + } + + if (!bare_query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + psFree(where); + return false; + } + // Take the bare query and add the dvodb selector + psString query = NULL; + psStringAppend(&query, bare_query, dvodb_string); + psFree(dvodb_string); + psFree(bare_query); + + // use psDBGenerateWhereConditionSQL because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + if (last_merged) { + psStringAppend(&query, " ORDER BY merge_order DESC LIMIT 1"); + } + + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "mergedvodbCopy", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + // loop over our list of camRun rows to check the supplied and selected dvodb and workdir values: + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + mergedvodbRunRow *row = mergedvodbRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into mergedvodbRun"); + psFree(output); + return false; + } + + if (!mergedvodb_rsync_path) { + psError(PS_ERR_UNKNOWN, false, "cannot queue mergedvodbcopy run without a defined mergedvodb_rsync_path: merge_id %" PRId64, row->merge_id); + psFree(output); + return false; + } + if (!destination_host) { + psError(PS_ERR_UNKNOWN, false, "cannot queue mergedvodbcopy run without a defined destination_host: merge_id %" PRId64, row->merge_id); + psFree(output); + return false; + } + + psFree(row); + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + mergedvodbRunRow *row = mergedvodbRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByMergeID(config, + row->merge_id, + destination_host ? destination_host : "NULL", + mergedvodb_rsync_path ?mergedvodb_rsync_path : "NULL" + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue merge_id: %" PRId64, row->merge_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool listmergedvodbcopyMode(pxConfig *config) { + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbCopy.merge_id", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodbcopy_id", "mergedvodbCopy.mergedvodbcopy_id", "=="); + PXOPT_COPY_STR(config->args, where, "-destination_host", "mergedvodbCopy.destination_host", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb", "mergedvodbRun.mergedvodb", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pending, config->args, "-pending", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("mergedvodbtool_find_mergedvodbcopy.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + +if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + +// we either add AND (condition) or WHERE (condition): + if (where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " AND mergedvodbCopy.fault != 0"); + } + if (where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND mergedvodbCopy.fault = 0"); + } + if (!where->list && faulted) { + + // list only faulted rows + psStringAppend(&query, " %s", " AND mergedvodbCopy.fault != 0"); + } + if (where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND mergedvodbCopy.fault = 0"); + } + if (!where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " WHERE mergedvodbCopy.fault != 0"); + } + if (!where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " WHERE mergedvodbCopy.fault = 0"); + } + psFree(where); + + if (pending) { + //add the cuts for pending (state new, no faults) + psStringAppend(&query, " %s", " AND mergedvodbCopy.state = 'new' AND mergedvodbCopy.fault = 0 AND mergedvodbCopy.destination_host IS NOT NULL AND mergedvodbCopy.mergedvodb_rsync_path IS NOT NULL"); + } + + + // order by epoch + psStringAppend(&query, " ORDER BY mergedvodbcopy_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error %s ", query); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "mergedvodbCopy", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + +return true; +} + +static bool revertmergedvodbcopyMode(pxConfig *config) { + psMetadata *where = psMetadataAlloc(); + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_COPY_S64(config->args, where, "-mergedvodbcopy_id", "mergedvodbCopy.mergedvodbcopy_id", "=="); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbCopy.merge_id", "=="); + PXOPT_COPY_STR(config->args, where, "-destination_host", "destination_host", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "mergedvodbCopy.fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + { + psString query = pxDataGet("mergedvodbtool_revertmergedvodbcopy.sql"); + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + } + psError(PS_ERR_UNKNOWN, false, "database error %s", query ); + psFree(query); + psFree(where); + return false; + } + psFree(query); + } + psFree(where); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updatemergedvodbcopyMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + // require? PXOPT_LOOKUP_U64(mergedvodbcopy_id, config->args, "-mergedvodbcopy_id", false, false); + // require? PXOPT_LOOKUP_U64(merge_id, config->args, "-merge_id", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + PXOPT_LOOKUP_STR(host, config->args, "-host", false, false); + PXOPT_LOOKUP_STR(rsync_path, config->args, "-mergedvodb_rsync_path", false, false); + // PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + PXOPT_LOOKUP_S16(set_fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_STR(set_rsync_path, config->args, "-set_mergedvodb_rsync_path", false, false); + PXOPT_LOOKUP_STR(set_host, config->args, "-set_destination_host", false, false); + PXOPT_LOOKUP_STR(set_state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_F32(dtime, config->args, "-set_dtime", false, false); + PXOPT_COPY_S64(config->args, where, "-mergedvodbcopy_id", "mergedvodbCopy.mergedvodbcopy_id", "=="); + PXOPT_COPY_S64(config->args, where, "-merge_id", "mergedvodbCopy.merge_id", "=="); + + PXOPT_COPY_STR(config->args, where, "-state", "mergedvodbCopy.state", "=="); + PXOPT_COPY_STR(config->args, where, "-host", "mergedvodbCopy.destination_host", "=="); + PXOPT_COPY_STR(config->args, where, "-mergedvodb_rsync_path", "mergedvodbCopy.mergedvodb_rsync_path", "=="); + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE mergedvodbCopy JOIN mergedvodbRun USING (merge_id) SET "); + int cnt = 0; + psString comma = ","; + if (set_fault) { + psStringAppend(&query, " mergedvodbCopy.fault = %d", set_fault); + cnt++; + } + if (set_rsync_path) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " mergedvodbCopy.mergedvodb_rsync_path = '%s'" , set_rsync_path); + cnt++; + } + + if (set_host) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " mergedvodbCopy.destination_host = '%s'" , set_host); + cnt++; + } + + + if (set_state) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " mergedvodbCopy.state = '%s'" , set_state); + cnt++; + } + + if (dtime) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " mergedvodbCopy.dtime = %f", dtime); + cnt++; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + psFree(query); + return false; + } + + psFree(query); + psFree(where); + + return true; +} + + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetool.h (revision 41104) @@ -0,0 +1,41 @@ +/* + * mergetool.h + * + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef MERGETOOL_H +#define MERGETOOL_H 1 + +#include "pxtools.h" + +typedef enum { + MERGETOOL_MODE_NONE = 0x0, + MERGETOOL_MODE_DEFINEBYQUERY, + MERGETOOL_MODE_UPDATERUN, + MERGETOOL_MODE_PENDINGMERGE, + MERGETOOL_MODE_ADDMERGED, + MERGETOOL_MODE_LISTMERGED, + MERGETOOL_MODE_REVERTMERGED, + MERGETOOL_MODE_UPDATEMERGED, + MERGETOOL_MODE_DEFINEBYQUERYMERGECOPY, + MERGETOOL_MODE_LISTMERGEDVODBCOPY, + MERGETOOL_MODE_REVERTMERGEDVODBCOPY, + MERGETOOL_MODE_UPDATEMERGEDVODBCOPY +} mergetoolMode; + +pxConfig *mergetoolConfig(pxConfig *config, int argc, char **argv); + +#endif // MERGETOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/mergetoolConfig.c (revision 41104) @@ -0,0 +1,196 @@ +/* + * addtoolConfig.c + * + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include + +#include "pxtools.h" +#include "pxmerge.h" +#include "mergetool.h" + +pxConfig *mergetoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + // pxcamSetSearchArgs(definebyqueryArgs); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-mergedvodb", PS_META_DUPLICATE_OK, "search by mergedvodbRun.mergedvodb and use as mergedvodb", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-minidvodb_group", PS_META_DUPLICATE_OK, "search by minidvodbRun.minidvodb_group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_mergedvodb_path", 0, "define workdir", NULL); + // psMetadataAddStr(definebyqueryArge, PS_LIST_TAIL, "-set_mergedvodb", 0, "define mergedvodb"); + + // psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *updaterunArgs = psMetadataAlloc(); + + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-mergedvodb", 0, "search by mergedvodb", 0); + psMetadataAddU64(updaterunArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "change the state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_mergedvodb_path", 0, "define workdir", NULL); + + + psMetadata *pendingmergeArgs = psMetadataAlloc(); + psMetadataAddU64(pendingmergeArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(pendingmergeArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", NULL); + psMetadataAddStr(pendingmergeArgs, PS_LIST_TAIL, "-mergedvodb", 0, "search by mergedvodb", NULL); + + psMetadataAddU64(pendingmergeArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(pendingmergeArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + + + psMetadata *addmergedArgs = psMetadataAlloc(); + psMetadataAddU64(addmergedArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id (required)", 0); + psMetadataAddStr(addmergedArgs, PS_LIST_TAIL, "-mergedvodb", 0, "search by mergedvodb (required)", 0); + psMetadataAddF32(addmergedArgs, PS_LIST_TAIL, "-dtime_verify", 0, "define elapsed time for DVO verify (seconds)", NAN); + psMetadataAddF32(addmergedArgs, PS_LIST_TAIL, "-dtime_merge", 0, "define elapsed time for DVO merge (seconds)", NAN); + psMetadataAddF32(addmergedArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time for script (seconds)", NAN); + psMetadataAddTime(addmergedArgs, PS_LIST_TAIL, "-epoch", 0, "time merge is finished", NULL); + psMetadataAddS16(addmergedArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + + + + psMetadata *listmergedArgs = psMetadataAlloc(); + psMetadataAddU64(listmergedArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(listmergedArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", NULL); + psMetadataAddStr(listmergedArgs, PS_LIST_TAIL, "-mergedvodb", 0, "search by minidvodbRun.minidvodb_group", NULL); + psMetadataAddU64(listmergedArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(listmergedArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + psMetadataAddBool(listmergedArgs, PS_LIST_TAIL, "-faulted", 0, "limit to faulted state", false); + + + psMetadata *revertmergedArgs = psMetadataAlloc(); + psMetadataAddU64(revertmergedArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", 0); + psMetadataAddS16(revertmergedArgs, PS_LIST_TAIL, "-fault", 0, "search by minidvodbCopyfault", 0); + psMetadataAddStr(revertmergedArgs, PS_LIST_TAIL, "-mergedvodb", 0, "search by mergedvodb", NULL); + + psMetadata *updatemergedArgs = psMetadataAlloc(); + + psMetadataAddU64(updatemergedArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", 0); + psMetadataAddS16(updatemergedArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + psMetadataAddF32(updatemergedArgs, PS_LIST_TAIL, "-set_dtime_verify", 0, "define elapsed time for DVO verify (seconds)", 0); + psMetadataAddF32(updatemergedArgs, PS_LIST_TAIL, "-set_dtime_merge", 0, "define elapsed time for DVO merge (seconds)", 0); + psMetadataAddF32(updatemergedArgs, PS_LIST_TAIL, "-set_dtime_script", 0, "define elapsed time for script (seconds)", 0); + psMetadataAddStr(updatemergedArgs, PS_LIST_TAIL, "-set_state", 0, "change the state", NULL); + // -definebyquerymergecopy + psMetadata *definebyquerymergecopyArgs = psMetadataAlloc(); + psMetadataAddS64(definebyquerymergecopyArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", 0); + psMetadataAddStr(definebyquerymergecopyArgs, PS_LIST_TAIL, "-mergedvodb", PS_META_DUPLICATE_OK, "search by mergedvodb", NULL); + psMetadataAddStr(definebyquerymergecopyArgs, PS_LIST_TAIL, "-set_mergedvodb_rsync_path", 0, "define workdir", NULL); + psMetadataAddStr(definebyquerymergecopyArgs, PS_LIST_TAIL, "-set_destination_host", 0, "set destination host", NULL); + psMetadataAddBool(definebyquerymergecopyArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyquerymergecopyArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definebyquerymergecopyArgs, PS_LIST_TAIL, "-last_merged", 0, "if multiple minidvodbs have been dvomerged between rsyncs, then only queue the most recent merged dvodb for copying", false); + + psMetadata *listmergedvodbcopyArgs = psMetadataAlloc(); + psMetadataAddU64(listmergedvodbcopyArgs, PS_LIST_TAIL, "-merge_id", 0, "search by mergedvodb_id", 0); + psMetadataAddStr(listmergedvodbcopyArgs, PS_LIST_TAIL, "-mergedvodbcopy_id", 0, "search by mergedvodbcopy_id", NULL); + psMetadataAddStr(listmergedvodbcopyArgs, PS_LIST_TAIL, "-mergedvodb", 0, "search by mergedvodbRun.mergedvodb", NULL); + psMetadataAddStr(listmergedvodbcopyArgs, PS_LIST_TAIL, "-destination_host", 0, "search by mergedvodbCopy.destination_host", NULL); + psMetadataAddBool(listmergedvodbcopyArgs, PS_LIST_TAIL, "-pending", 0, "limit to pending items", false); + psMetadataAddU64(listmergedvodbcopyArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(listmergedvodbcopyArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + psMetadataAddBool(listmergedvodbcopyArgs, PS_LIST_TAIL, "-faulted", 0, "limit to faulted state", false); + + psMetadata *revertmergedvodbcopyArgs = psMetadataAlloc(); + psMetadataAddU64(revertmergedvodbcopyArgs, PS_LIST_TAIL, "-mergedvodbcopy_id", 0, "search by mergedvodbcopy_id", 0); + psMetadataAddU64(revertmergedvodbcopyArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", 0); + psMetadataAddStr(revertmergedvodbcopyArgs, PS_LIST_TAIL, "-destination_host", 0, "search by destination_host", NULL); + psMetadataAddS16(revertmergedvodbcopyArgs, PS_LIST_TAIL, "-fault", 0, "search by mergedvodbCopyfault", 0); + psMetadata *updatemergedvodbcopyArgs = psMetadataAlloc(); + psMetadataAddU64(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-mergedvodbcopy_id", 0, "search by mergedvodbcopy_id", 0); + psMetadataAddU64(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-merge_id", 0, "search by merge_id", 0); + psMetadataAddStr(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-host", 0, "search by host", NULL); + psMetadataAddStr(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-mergedvodb_rsync_path", 0, "search by rsync", NULL); + psMetadataAddS16(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + + psMetadataAddS16(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + + psMetadataAddF32(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-set_dtime", 0, "set elapsed time for transfer", 0); + psMetadataAddStr(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-set_mergedvodb_rsync_path", 0, "change the mergedvodb_rsync_path", NULL); + psMetadataAddStr(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-set_destination_host", 0, "change the host", NULL); + psMetadataAddStr(updatemergedvodbcopyArgs, PS_LIST_TAIL, "-set_state", 0, "change the state", NULL); + + + + + + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from minindvodbRun", MERGETOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "update existing mergedvodbRuns", MERGETOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-pendingmerge", "list pending mergedvodbRuns", MERGETOOL_MODE_PENDINGMERGE, pendingmergeArgs); + PXOPT_ADD_MODE("-addmerged", "add processed mergedvodbRun", MERGETOOL_MODE_ADDMERGED, addmergedArgs); + PXOPT_ADD_MODE("-listmerged","list merged dvodbs", MERGETOOL_MODE_LISTMERGED, listmergedArgs); + PXOPT_ADD_MODE("-revertmerged","revert merged dvodbs", MERGETOOL_MODE_REVERTMERGED, revertmergedArgs); + PXOPT_ADD_MODE("-updatemerged","change merged dvodb properties",MERGETOOL_MODE_UPDATEMERGED, updatemergedArgs); + PXOPT_ADD_MODE("-definebyquerymergecopy", "create runs from mergedvodbRun", MERGETOOL_MODE_DEFINEBYQUERYMERGECOPY, definebyquerymergecopyArgs); + PXOPT_ADD_MODE("-listmergedvodbcopy","list copy mergedvodbs", MERGETOOL_MODE_LISTMERGEDVODBCOPY, listmergedvodbcopyArgs); + PXOPT_ADD_MODE("-revertmergedvodbcopy","revert copy mergedvobs",MERGETOOL_MODE_REVERTMERGEDVODBCOPY, revertmergedvodbcopyArgs); + PXOPT_ADD_MODE("-updatemergedvodbcopy","change mergedvodb copy properties",MERGETOOL_MODE_UPDATEMERGEDVODBCOPY, updatemergedvodbcopyArgs); + + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtool.c (revision 41104) @@ -0,0 +1,500 @@ +/* + * addtool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxadd.h" +#include "pxminidvodb.h" +#include "minidvodbtool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool listminidvodbcopyMode(pxConfig *config); +static bool revertminidvodbcopyMode(pxConfig *config); +static bool updateminidvodbcopyMode(pxConfig *config); + + + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = addtoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(ADDTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(ADDTOOL_MODE_LISTMINIDVODBCOPY, listminidvodbcopyMode); + MODECASE(ADDTOOL_MODE_REVERTMINIDVODBCOPY, revertminidvodbcopyMode); + MODECASE(ADDTOOL_MODE_UPDATEMINIDVODBCOPY, updateminidvodbcopyMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + pxcamGetSearchArgs (config, where); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbRun.minidvodb_id", "=="); + pxAddLabelSearchArgs (config, where, "-minidvodb_group", "minidvodbRun.minidvodb_group", "=="); // define using camRun label + + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(destination_host, config->args, "-set_destination_host", false, false); + PXOPT_LOOKUP_STR(minidvodb_rsync_path, config->args, "-set_minidvodb_rsync_path", false, false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // prevent queueing an addRun if a given exposure has already been added to + // the given dvo database + psString dvodb_string = NULL; + psString bare_query = NULL; + if (destination_host) { + psTrace("minidvodbtool.c", PS_LOG_INFO, "destination_host argument found (%s) using minidvodbtool_find_minidvodb_id_dvo.sql\n", destination_host); + // find the cam_id of all the exposures that we want to queue up. + bare_query = pxDataGet("minidvodbtool_find_minidvodb_id_dvo.sql"); + // user supplied dvodb + psStringAppend(&dvodb_string, "minidvodbCopy.destination_host = '%s'", destination_host); + } else { + psError(PS_ERR_UNKNOWN, false, "cannot queue minidvodbcopy run without a defined destination_host"); + + return false; + } + if (!bare_query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + psFree(where); + return false; + } + // Take the bare query and add the dvodb selector + psString query = NULL; + psStringAppend(&query, bare_query, dvodb_string); + psFree(dvodb_string); + psFree(bare_query); + + // use psDBGenerateWhereConditionSQL because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "minidvodbCopy", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + // loop over our list of camRun rows to check the supplied and selected dvodb and workdir values: + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + minidvodbRunRow *row = minidvodbRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into minidvodbRun"); + psFree(output); + return false; + } + + if (!minidvodb_rsync_path) { + psError(PS_ERR_UNKNOWN, false, "cannot queue minidvodbcopy run without a defined minidvodb_rsync_path: minidvodb_id %" PRId64, row->minidvodb_id); + psFree(output); + return false; + } + if (!destination_host) { + psError(PS_ERR_UNKNOWN, false, "cannot queue minidvodbcopy run without a defined destination_host: minidvodb_id %" PRId64, row->minidvodb_id); + psFree(output); + return false; + } + + psFree(row); + } + + // start a transaction so we don't end up with an exp without any associted + // imfiles + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // would could do this "all in the database" if we didn't want the option + // of changing the label/reduction/expgroup/dvodb/etc. So we're pulling the + // data out so we have the option of changing these values or leaving the + // old values in place (i.e., passing the values through). + + // loop over our list of camRun rows + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + minidvodbRunRow *row = minidvodbRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into camRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxaddQueueByMinidvodbID(config, + row->minidvodb_id, + destination_host ? destination_host : "NULL", + minidvodb_rsync_path ?minidvodb_rsync_path : "NULL" + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error sfg"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue minidvodb_id: %" PRId64, row->minidvodb_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + + + +static bool listminidvodbcopyMode(pxConfig *config) { + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbCopy.minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodbcopy_id", "minidvodbCopy.minidvodbcopy_id", "=="); + PXOPT_COPY_STR(config->args, where, "-destination_host", "minidvodbCopy.destination_host", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_group", "minidvodbRun.minidvodb_group", "=="); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pending, config->args, "-pending", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("minidvodbtool_find_minidvodbcopy.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + +if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + +// we either add AND (condition) or WHERE (condition): + if (where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " AND minidvodbCopy.fault != 0"); + } + if (where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " AND minidvodbCopy.fault = 0"); + } + if (!where->list && faulted) { + // list only faulted rows + psStringAppend(&query, " %s", " WHERE minidvodbCopy.fault != 0"); + } + if (!where->list && !faulted) { + // don't list faulted rows + psStringAppend(&query, " %s", " WHERE minidvodbCopy.fault = 0"); + } + psFree(where); + + if (pending) { + //add the cuts for pending (state new, no faults) + psStringAppend(&query, " %s", " AND minidvodbCopy.state = 'new' AND minidvodbCopy.fault = 0 AND minidvodbCopy.destination_host IS NOT NULL AND minidvodbCopy.minidvodb_rsync_path IS NOT NULL"); + } + + + // order by epoch + psStringAppend(&query, " ORDER BY minidvodbcopy_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error %s ", query); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("addtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "minidvodbCopy", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + +return true; +} + + + +static bool revertminidvodbcopyMode(pxConfig *config) { + psMetadata *where = psMetadataAlloc(); + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_COPY_S64(config->args, where, "-minidvodbcopy_id", "minidvodbCopy.minidvodbcopy_id", "=="); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbCopy.minidvodb_id", "=="); + PXOPT_COPY_STR(config->args, where, "-destination_host", "destination_host", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "minidvodbCopy.fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + { + psString query = pxDataGet("minidvodbtool_revertminidvodbcopy.sql"); + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereConditionalSQL with AND ... because the SQL ends in a WHERE + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + } + psError(PS_ERR_UNKNOWN, false, "database error %s", query ); + psFree(query); + psFree(where); + return false; + } + psFree(query); + } + psFree(where); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + + +static bool updateminidvodbcopyMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + // PXOPT_LOOKUP_U64(minidvodbcopy_id, config->args, "-minidvodbcopy_id", false, false); + // PXOPT_LOOKUP_U64(minidvodb_id, config->args, "-minidvodb_id", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + PXOPT_LOOKUP_STR(host, config->args, "-host", false, false); + PXOPT_LOOKUP_STR(rsync_path, config->args, "-minidvodb_rsync_path", false, false); + // PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + PXOPT_LOOKUP_S16(set_fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_STR(set_rsync_path, config->args, "-set_minidvodb_rsync_path", false, false); + PXOPT_LOOKUP_STR(set_host, config->args, "-set_destination_host", false, false); + PXOPT_LOOKUP_STR(set_state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_F32(dtime, config->args, "-set_dtime", false, false); + PXOPT_COPY_S64(config->args, where, "-minidvodbcopy_id", "minidvodbCopy.minidvodbcopy_id", "=="); + PXOPT_COPY_S64(config->args, where, "-minidvodb_id", "minidvodbCopy.minidvodb_id", "=="); + + PXOPT_COPY_STR(config->args, where, "-state", "minidvodbCopy.state", "=="); + PXOPT_COPY_STR(config->args, where, "-host", "minidvodbCopy.destination_host", "=="); + PXOPT_COPY_STR(config->args, where, "-minidvodb_rsync_path", "minidvodbCopy.minidvodb_rsync_path", "=="); + + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE minidvodbCopy JOIN minidvodbRun USING (minidvodb_id) SET "); + int cnt = 0; + psString comma = ","; + if (set_fault) { + psStringAppend(&query, " minidvodbCopy.fault = %d", set_fault); + cnt++; + } + + if (set_rsync_path) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " minidvodbCopy.minidvodb_rsync_path = '%s'" , set_rsync_path); + cnt++; + } + + if (set_host) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " minidvodbCopy.destination_host = '%s'" , set_host); + cnt++; + } + + + if (set_state) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + + psStringAppend(&query, " minidvodbCopy.state = '%s'" , set_state); + cnt++; + } + + if (dtime) { + if (cnt) { + psStringAppend(&query, "%s", comma); + } + psStringAppend(&query, " minidvodbCopy.dtime = %f", dtime); + cnt++; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + psFree(query); + return false; + } + + psFree(query); + psFree(where); + + return true; +} + + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtool.h (revision 41104) @@ -0,0 +1,35 @@ +/* + * addtool.h + * + * Copyright (C) 2006 Joshua Hoblitt, Christopher Waters + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef ADDTOOL_H +#define ADDTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + ADDTOOL_MODE_NONE = 0x0, + ADDTOOL_MODE_DEFINEBYQUERY, + ADDTOOL_MODE_LISTMINIDVODBCOPY, + ADDTOOL_MODE_REVERTMINIDVODBCOPY, + ADDTOOL_MODE_UPDATEMINIDVODBCOPY +} addtoolMode; + +pxConfig *addtoolConfig(pxConfig *config, int argc, char **argv); + +#endif // ADDTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/minidvodbtoolConfig.c (revision 41104) @@ -0,0 +1,124 @@ +/* + * addtoolConfig.c + * + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include + +#include "pxtools.h" +#include "pxadd.h" +#include "pxminidvodb.h" +#include "minidvodbtool.h" + +pxConfig *addtoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + pxcamSetSearchArgs(definebyqueryArgs); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-minidvodb_group", PS_META_DUPLICATE_OK, "search by minidvodbRun minidvodb_group", NULL); + + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_minidvodb_rsync_path", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_destination_host", 0, "define label", NULL); + // psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *listminidvodbcopyArgs = psMetadataAlloc(); + psMetadataAddU64(listminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(listminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodbcopy_id", 0, "search by minidvodbcopy_id", NULL); + psMetadataAddStr(listminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodb_group", 0, "search by minidvodbRun.minidvodb_group", NULL); + psMetadataAddStr(listminidvodbcopyArgs, PS_LIST_TAIL, "-destination_host", 0, "search by minidvodbCopy.destination_host", NULL); + psMetadataAddBool(listminidvodbcopyArgs, PS_LIST_TAIL, "-pending", 0, "limit to pending items", false); + psMetadataAddU64(listminidvodbcopyArgs, PS_LIST_TAIL, "-limit", 0, "limit to N items", 0); + psMetadataAddBool(listminidvodbcopyArgs, PS_LIST_TAIL, "-simple", 0, "simple output", false); + psMetadataAddBool(listminidvodbcopyArgs, PS_LIST_TAIL, "-faulted", 0, "limit to faulted state", false); + + psMetadata *revertminidvodbcopyArgs = psMetadataAlloc(); + psMetadataAddU64(revertminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodbcopy_id", 0, "search by minidvodbcopy_id", 0); + psMetadataAddU64(revertminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(revertminidvodbcopyArgs, PS_LIST_TAIL, "-destination_host", 0, "search by destination_host", NULL); + psMetadataAddS16(revertminidvodbcopyArgs, PS_LIST_TAIL, "-fault", 0, "search by minidvodbCopyfault", 0); + psMetadata *updateminidvodbcopyArgs = psMetadataAlloc(); + psMetadataAddU64(updateminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodbcopy_id", 0, "search by minidvodbcopy_id", 0); + psMetadataAddU64(updateminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodb_id", 0, "search by minidvodb_id", 0); + psMetadataAddStr(updateminidvodbcopyArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updateminidvodbcopyArgs, PS_LIST_TAIL, "-host", 0, "search by host", NULL); + psMetadataAddStr(updateminidvodbcopyArgs, PS_LIST_TAIL, "-minidvodb_rsync_path", 0, "search by rsync", NULL); + psMetadataAddS16(updateminidvodbcopyArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + + psMetadataAddS16(updateminidvodbcopyArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + + psMetadataAddF32(updateminidvodbcopyArgs, PS_LIST_TAIL, "-set_dtime", 0, "set elapsed time for transfer", 0); + psMetadataAddStr(updateminidvodbcopyArgs, PS_LIST_TAIL, "-set_minidvodb_rsync_path", 0, "change the minidvodb_rsync_path", NULL); + psMetadataAddStr(updateminidvodbcopyArgs, PS_LIST_TAIL, "-set_destination_host", 0, "change the host", NULL); + psMetadataAddStr(updateminidvodbcopyArgs, PS_LIST_TAIL, "-set_state", 0, "change the state", NULL); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from cam stage", ADDTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-listminidvodbcopy","list copy minidvodbs", ADDTOOL_MODE_LISTMINIDVODBCOPY, listminidvodbcopyArgs); + PXOPT_ADD_MODE("-revertminidvodbcopy","revert copy minidvobs", ADDTOOL_MODE_REVERTMINIDVODBCOPY, revertminidvodbcopyArgs); + PXOPT_ADD_MODE("-updateminidvodbcopy","change minidvodb copy properties",ADDTOOL_MODE_UPDATEMINIDVODBCOPY, updateminidvodbcopyArgs); + + + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptool.c (revision 41104) @@ -0,0 +1,2101 @@ +/* + * pstamptool.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "pstamptool.h" +#include "pstamp.h" + +static bool adddatastoreMode(pxConfig *config); +static bool datastoreMode(pxConfig *config); +static bool moddatastoreMode(pxConfig *config); +static bool addreqMode(pxConfig *config); +static bool completedreqMode(pxConfig *config); +static bool listreqMode(pxConfig *config); +static bool pendingreqMode(pxConfig *config); +static bool updatereqMode(pxConfig *config); +static bool revertreqMode(pxConfig *config); +static bool pendingcleanupMode(pxConfig *config); +static bool addjobMode(pxConfig *config); +static bool listjobMode(pxConfig *config); +static bool pendingjobMode(pxConfig *config); +static bool updatejobMode(pxConfig *config); +static bool stopdependentjobMode(pxConfig *config); +static bool revertjobMode(pxConfig *config); + +static bool addprojectMode(pxConfig *config); +static bool projectMode(pxConfig *config); +static bool modprojectMode(pxConfig *config); +static bool getdependentMode(pxConfig *config); +static bool pendingdependentMode(pxConfig *config); +static bool updatedependentMode(pxConfig *config); +static bool revertdependentMode(pxConfig *config); +static bool getwebrequestnumMode(pxConfig *config); +static bool addfileMode(pxConfig *config); +static bool listfileMode(pxConfig *config); +static bool deletefileMode(pxConfig *config); + +static bool adddomainMode(pxConfig * config); +static bool updatedomainMode(pxConfig * config); +static bool listdomainMode(pxConfig *config); +static bool adduserMode(pxConfig * config); +static bool updateuserMode(pxConfig * config); +static bool listuserMode(pxConfig * config); +static bool addaccesslevelMode(pxConfig * config); +static bool updateaccesslevelMode(pxConfig * config); +static bool listaccesslevelMode(pxConfig * config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +// XXX make this a configurable parameter +#define PSTAMP_MAX_JOB_FAULTS 5 +#define PSTAMP_MAX_DEP_FAULTS 5 + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pstamptoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(PSTAMPTOOL_MODE_ADDDATASTORE, adddatastoreMode); + MODECASE(PSTAMPTOOL_MODE_DATASTORE, datastoreMode); + MODECASE(PSTAMPTOOL_MODE_MODDATASTORE, moddatastoreMode); + MODECASE(PSTAMPTOOL_MODE_ADDREQ, addreqMode); + MODECASE(PSTAMPTOOL_MODE_COMPLETEDREQ, completedreqMode); + MODECASE(PSTAMPTOOL_MODE_LISTREQ, listreqMode); + MODECASE(PSTAMPTOOL_MODE_PENDINGREQ, pendingreqMode); + MODECASE(PSTAMPTOOL_MODE_UPDATEREQ, updatereqMode); + MODECASE(PSTAMPTOOL_MODE_REVERTREQ, revertreqMode); + MODECASE(PSTAMPTOOL_MODE_PENDINGCLEANUP, pendingcleanupMode); + MODECASE(PSTAMPTOOL_MODE_ADDJOB, addjobMode); + MODECASE(PSTAMPTOOL_MODE_LISTJOB, listjobMode); + MODECASE(PSTAMPTOOL_MODE_PENDINGJOB, pendingjobMode); + MODECASE(PSTAMPTOOL_MODE_UPDATEJOB, updatejobMode); + MODECASE(PSTAMPTOOL_MODE_STOPDEPENDENTJOB, stopdependentjobMode); + MODECASE(PSTAMPTOOL_MODE_REVERTJOB, revertjobMode); + MODECASE(PSTAMPTOOL_MODE_ADDPROJECT, addprojectMode); + MODECASE(PSTAMPTOOL_MODE_MODPROJECT, modprojectMode); + MODECASE(PSTAMPTOOL_MODE_PROJECT, projectMode); + MODECASE(PSTAMPTOOL_MODE_GETDEPENDENT, getdependentMode); + MODECASE(PSTAMPTOOL_MODE_PENDINGDEPENDENT, pendingdependentMode); + MODECASE(PSTAMPTOOL_MODE_UPDATEDEPENDENT, updatedependentMode); + MODECASE(PSTAMPTOOL_MODE_REVERTDEPENDENT, revertdependentMode); + MODECASE(PSTAMPTOOL_MODE_GETWEBREQUESTNUM, getwebrequestnumMode); + MODECASE(PSTAMPTOOL_MODE_ADDFILE, addfileMode); + MODECASE(PSTAMPTOOL_MODE_LISTFILE, listfileMode); + MODECASE(PSTAMPTOOL_MODE_DELETEFILE, deletefileMode); + + MODECASE(PSTAMPTOOL_MODE_ADDDOMAIN, adddomainMode); + MODECASE(PSTAMPTOOL_MODE_UPDATEDOMAIN, updatedomainMode); + MODECASE(PSTAMPTOOL_MODE_LISTDOMAIN, listdomainMode); + MODECASE(PSTAMPTOOL_MODE_ADDUSER, adduserMode); + MODECASE(PSTAMPTOOL_MODE_UPDATEUSER, updateuserMode); + MODECASE(PSTAMPTOOL_MODE_LISTUSER, listuserMode); + MODECASE(PSTAMPTOOL_MODE_ADDACCESSLEVEL, addaccesslevelMode); + MODECASE(PSTAMPTOOL_MODE_UPDATEACCESSLEVEL, updateaccesslevelMode); + MODECASE(PSTAMPTOOL_MODE_LISTACCESSLEVEL, listaccesslevelMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool adddatastoreMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(uri, config->args, "-set_uri", true, false); + PXOPT_LOOKUP_STR(outProduct, config->args, "-set_out_product", true, false); + PXOPT_LOOKUP_STR(lastFileset, config->args, "-set_last_fileset", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_S32(pollInterval, config->args, "-set_poll_interval",false, false); + + if (!pstampDataStoreInsert(config->dbh, + 0, + state, + lastFileset, + NULL, // timestamp + label, // label + outProduct, + uri, + pollInterval, + false + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool datastoreMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-ds_id", "ds_id", "=="); + + PXOPT_LOOKUP_BOOL(ready, config->args, "-ready", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pstamptool_datastore.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + if (ready) { + psStringAppend(&query, " %s", "\nAND TIMESTAMPDIFF(SECOND, timestamp, now()) > pollInterval"); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampDataStore", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool moddatastoreMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(ds_id, config->args, "-ds_id", true, false); + PXOPT_LOOKUP_STR(lastFileset, config->args, "-set_last_fileset", false, false); + PXOPT_LOOKUP_STR(uri, config->args, "-set_uri", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_S32(pollInterval, config->args, "-set_poll_interval", false, false); + PXOPT_LOOKUP_BOOL(update_timestamp, config->args, "-update_timestamp", false); + + if (!state && !lastFileset && !pollInterval && !update_timestamp && !label &&!uri) { + psError(PS_ERR_UNKNOWN, true, "at least one of -last_fileset or -set_state is required"); + return false; + } + + char *query = psStringCopy ("UPDATE pstampDataStore SET timestamp = CURRENT_TIMESTAMP() "); + + if (lastFileset) { + psStringAppend(&query, " , lastFileset = '%s'", lastFileset); + } + + if (uri) { + psStringAppend(&query, ", uri = '%s'", uri); + } + if (state) { + psStringAppend(&query, ", state = '%s'", state); + } + + if (label) { + psStringAppend(&query, ", label = '%s'", label); + } + + if (pollInterval) { + psStringAppend(&query, ", pollInterval = %d", pollInterval); + } + + psStringAppend(&query, " WHERE ds_id = %" PRId64, ds_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected one row but %" + PRIu64 " rows were modified", affected); + return false; + } + + return true; +} + +static bool addreqMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + PXOPT_LOOKUP_STR(name, config->args, "-name", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_S64(ds_id, config->args, "-ds_id", false, false); + + PXOPT_LOOKUP_STR(username, config->args, "-username", false, false); + PXOPT_LOOKUP_S64(proj_id, config->args, "-proj_id", false, false); + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + if (!pstampRequestInsert(config->dbh, + 0, // req_id + ds_id, + "new", //state + name, + NULL, // reqType + label, + NULL, // outProduct + uri, + NULL, // outdir + username, + proj_id, + now, // registered + now, // timestamp + 0 // fault + )) { + psError(PS_ERR_UNKNOWN, false, "failed to insert request"); + return false; + } + + psS64 req_id = psDBLastInsertID(config->dbh); + + printf("%" PRId64 "\n", req_id); + + return true; +} + +static bool pendingreqMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "label", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pstamptool_pendingreq.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "pstampRequest"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, " ORDER BY priority DESC, req_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampRequest", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingcleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "label", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pstamptool_pendingcleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "pstampRequest"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, " ORDER BY priority DESC, req_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampRequest", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool listreqMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-not_req_id", "req_id", "!="); + PXOPT_COPY_STR(config->args, where, "-name", "name", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-username", "username", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!psListLength(where->list)) { + psError(PS_ERR_UNKNOWN, true, "search paramters are required"); + return false; + } + + psString query = psStringCopy("SELECT * from pstampRequest"); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "request not found"); + // This causes main to exit with PS_EXIT_DATA_ERROR which the pstamp scripts are looking for + psError(PXTOOLS_ERR_CONFIG, true, "request not found"); + psFree(output); + // we return false so that the caller can easily determine that a request does not exist + return false; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampRequest", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool completedreqMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs(config, where, "-label", "label", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pstamptool_completedreq.sql"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, " ORDER BY priority DESC, req_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampRequest", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool updatereqMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // PXOPT_LOOKUP_S64(req_id, config->args, "-req_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(outProduct, config->args, "-set_outProduct", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_STR(uri, config->args, "-set_uri", false, false); + PXOPT_LOOKUP_STR(outdir, config->args, "-set_outdir", false, false); + PXOPT_LOOKUP_STR(name, config->args, "-set_name", false, false); + PXOPT_LOOKUP_STR(username, config->args, "-set_username", false, false); + PXOPT_LOOKUP_STR(reqType, config->args, "-set_reqType", false, false); + PXOPT_LOOKUP_BOOL(clearfault,config->args, "-clearfault", false); + + if (!state && !label && !outProduct && !fault && !uri && !outdir && !name && !username && !reqType && !clearfault) { + psError(PS_ERR_UNKNOWN, true, "at least one set option is required"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id_max", "req_id", "<="); + PXOPT_COPY_S32(config->args, where, "-fault", "fault", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-reqType", "reqType", "=="); + PXOPT_COPY_STR(config->args, where, "-name", "name", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-username", "username", "=="); + PXOPT_COPY_TIME(config->args, where, "-timestamp_begin", "timestamp", ">="); + PXOPT_COPY_TIME(config->args, where, "-timestamp_end", "timestamp", "<="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "LIKE"); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE pstampRequest SET timestamp = UTC_TIMESTAMP()"); + + psString stateCheck = NULL; + if (state) { + psStringAppend(&query, ", state = '%s'", state); + if (!strcmp(state, "goto_cleaned")) { + psStringAppend(&stateCheck, " AND (state != 'cleaned' AND state != 'goto_cleaned')"); + } + } + if (label) { + psStringAppend(&query, ", label = '%s'", label); + } + if (outProduct) { + psStringAppend(&query, ", outProduct = '%s'", outProduct); + } + if (outdir) { + psStringAppend(&query, ", outdir = '%s'", outdir); + } + if (username) { + psStringAppend(&query, ", username = '%s'", username); + } + if (clearfault) { + if (fault) { + psError(PXTOOLS_ERR_CONFIG, true, "only one of -fault and -clearfault is allowed"); + return false; + } + psStringAppend(&query, ", fault = 0"); + } else if (fault) { + psStringAppend(&query, ", fault = %d", fault); + } + if (uri) { + psStringAppend(&query, ", uri = '%s'", uri); + } + if (name) { + psStringAppend(&query, ", name = '%s'", name); + } + if (reqType) { + psStringAppend(&query, ", reqType = '%s'", reqType); + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (stateCheck) { + psStringAppend(&query, "%s", stateCheck); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + // psU64 affected = psDBAffectedRows(config->dbh); + // psLogMsg("pstamptool", PS_LOG_INFO, "Updated %" PRIu64 " pstampRequests", affected); + + return true; +} + +static bool revertreqMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fault", "pstampRequest.fault", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "pstampRequest.state", "=="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "LIKE"); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + + // delete any jobs that were queued by requests that didn't complete parsing (pstampRequest.state = 'new' + // If state = 'run' was supplied this will be a no-op + psString query = pxDataGet("pstamptool_revertreq_deletejobs.sql"); + psStringAppend(&query, " AND %s", whereClause); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + // clear fault for requests + query = pxDataGet("pstamptool_revertreq.sql"); + psStringAppend(&query, " AND %s", whereClause); + + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + return true; +} + +static bool addjobMode(pxConfig *config) +{ + bool stampJob = false; + + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(req_id, config->args, "-req_id", true, false); + PXOPT_LOOKUP_STR(rownum, config->args, "-rownum", true, false); + PXOPT_LOOKUP_STR(job_type, config->args, "-job_type", false, false); + PXOPT_LOOKUP_STR(outputBase, config->args, "-outputBase", false, false); + PXOPT_LOOKUP_STR(stateString, config->args, "-state", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + PXOPT_LOOKUP_S64(options, config->args, "-options", false, false); + PXOPT_LOOKUP_S64(dep_id, config->args, "-dep_id", false, false); + PXOPT_LOOKUP_S64(parent_id, config->args, "-parent_id", false, false); + PXOPT_LOOKUP_BOOL(is_parent, config->args, "-is_parent", false); + + // unless the job is being inserted with stop state require outputBase + if (strcmp(stateString, "stop") && !outputBase) { + psError(PS_ERR_UNKNOWN, true, "-outputBase is required"); + return false; + } + + // default value for job_type is defined in pstamptoolConfig.c + if (!strcmp(job_type, "get_image") || !strcmp(job_type, "detect_query") || !strcmp(job_type, "none")) { + stampJob = false; + } else if (!strcmp(job_type, "child")) { + // job_type child's only action is to resolve a dependent + // XXX: IS this necessary? + if (!dep_id) { + psError(PS_ERR_UNKNOWN, true, "dep_id required for child job\n"); + return false; + } + if (is_parent) { + psError(PS_ERR_UNKNOWN, true, "job type child can not be a parent job\n"); + return false; + } + stampJob = false; + } else if (!strcmp(job_type, "mosaic")) { + stampJob = false; + is_parent = true; + } else if (!strcmp(job_type, "stamp")) { + stampJob = true; + } else { + psError(PS_ERR_UNKNOWN, false, "unknown value for -job_type: %s", job_type); + return false; + } + if (stampJob) { /* do something?? */ } + + if (!pstampJobInsert(config->dbh, + 0, // job_id + req_id, + rownum, + stateString, + job_type, + fault, + exp_id, + outputBase, + options, + dep_id, + 0, // fault_count + parent_id, + is_parent + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, + "should have affected one row but %" PRIu64 " rows were modified", + affected); + return false; + } + + psS64 job_id = psDBLastInsertID(config->dbh); + printf("%" PRId64 "\n", job_id); + + return true; +} + +static bool listjobMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dep_id", "dep_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-jobType", "jobType", "=="); + PXOPT_COPY_S64(config->args, where, "-fault", "fault", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!psListLength(where->list)) { + fprintf(stderr, "search arguments are required\n"); + exit (1); + } + + psString query = pxDataGet("pstamptool_listjob.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "pstampJob"); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampJob", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingjobMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pstamptool_pendingjob.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + psString whereStr = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereStr, "\n AND %s", whereClause); + psFree(whereClause); + } else { + psStringAppend(&whereStr, "%s", ""); + } + psFree(where); + + psStringAppend(&query, " ORDER BY priority DESC, req_id, job_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereStr, whereStr)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereStr); + psFree(query); + return false; + } + psFree(whereStr); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampJob", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool updatejobMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(job_id, config->args, "-job_id", false, false); + PXOPT_LOOKUP_S64(req_id, config->args, "-req_id", false, false); + PXOPT_LOOKUP_S64(dep_id, config->args, "-dep_id", false, false); + PXOPT_LOOKUP_S32(fault_count, config->args, "-fault_count", false, false); + + if (!job_id && !req_id && !dep_id && !fault_count) { + psError(PS_ERR_UNKNOWN, true, "at least one of -job_id -req_id -dep_id or -fault_count is required"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_S32(fault, config->args, "-set_fault", false, false); + + if (!state && !fault) { + psError(PS_ERR_UNKNOWN, true, "at least one of -set_state and -set_fault is required"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dep_id", "dep_id", "=="); + PXOPT_COPY_S32(config->args, where, "-fault", "pstampJob.fault", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "pstampJob.state", "=="); + PXOPT_COPY_S32(config->args, where, "-fault_count", "pstampJob.fault_count", ">="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "LIKE"); + + psString query = pxDataGet("pstamptool_updatejob.sql"); + + char c = ' '; + if (state) { + psStringAppend(&query, "\n %c pstampJob.state = '%s'", c, state); + c = ','; + } + if (fault) { + psStringAppend(&query, "\n %c pstampJob.fault = %d", c, fault); + c = ','; + psStringAppend(&query, ", pstampJob.fault_count = pstampJob.fault_count+ 1"); + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("pstamptool", PS_LOG_INFO, "Updated %" PRIu64 " pstampJobs", affected); + + return true; +} +// Terminate jobs which have dependents setting both the pstampDependent and pstampJob.fault +static bool stopdependentjobMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S32(fault, config->args, "-set_fault", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dep_id", "dep_id", "=="); + + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-component", "component", "=="); + + PXOPT_COPY_S32(config->args, where, "-fault", "pstampDependent.fault", "=="); + + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "LIKE"); + PXOPT_COPY_S32(config->args, where, "-fault_count", "pstampDependent.fault_count", ">="); + + // XXX: How about selecting by pstampRequest.label? No. That is too dangerous by itself. + + psString query = pxDataGet("pstamptool_stopdependentjob.sql"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, fault, fault, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("pstamptool", PS_LOG_INFO, "Updated %" PRIu64 " pstampJobs and pstampDependents", affected); + + return true; +} + +static bool revertjobMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fault", "pstampJob.fault", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id_min", "req_id", ">="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "LIKE"); + + PXOPT_LOOKUP_BOOL(clear_fault_count, config->args, "-clear_fault_count", false); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + // XXX: we don't actually use -limit. It doesn't work for UPDATE + // it's an allowed arg because add_poll_args adds it + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + if (limit) { /* do something?? */ } + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // By default only revert faults < PSTAMP_FIRST_ERROR_CODE which are our "ipp exit codes" + // codes larger than that are the pstamp request interface. + // Don't fault those unless -fault was explicitly provided + psString faultClause = psStringCopy(""); + if (!fault) { + psStringAppend(&faultClause, " \nAND (pstampJob.fault < %d)", PSTAMP_FIRST_ERROR_CODE); + } + + psString query = pxDataGet("pstamptool_revertjob.sql"); + if (!psListLength(where->list) && !all) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters or -all are required"); + return false; + } + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // don't keep reverting once the number of faults reaches some value unless + // a parameter asking us to clear that count is provided + psString faultCountClause = NULL; + if (clear_fault_count) { + psStringAppend(&faultCountClause, "\n, pstampJob.fault_count = 0"); + } else { + psStringAppend(&query, " AND pstampJob.fault_count < %d", PSTAMP_MAX_JOB_FAULTS); + } + + if (!p_psDBRunQueryF(config->dbh, query, faultCountClause, faultClause)) { + psFree(faultCountClause); + psFree(faultClause); + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(faultClause); + psFree(faultCountClause); + psFree(query); + + return true; +} + +# if (1) +// these are unused functions +// XXX: yes they are +static bool addprojectMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(name, config->args, "-name", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + PXOPT_LOOKUP_STR(imagedb, config->args, "-imagedb", true, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_BOOL(need_magic, config->args, "-need_magic", false); + + if (!pstampProjectInsert(config->dbh, + 0, + name, + state, + imagedb, + dvodb, + camera, + telescope, + need_magic + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool projectMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-name", "name", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pstamptool_project.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampProject", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool modprojectMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(proj_id, config->args, "-proj_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + + char *query = psStringCopy ("UPDATE pstampProject SET"); + + psStringAppend(&query, " state = '%s'", state); + + psStringAppend(&query, " WHERE proj_id = %" PRId64, proj_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected one row but %" + PRIu64 " rows were modified", affected); + return false; + } + + return true; +} + +static bool getdependentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + PXOPT_LOOKUP_S64(stage_id, config->args, "-stage_id", true, false); + PXOPT_LOOKUP_STR(component, config->args, "-component", true, false); + PXOPT_LOOKUP_STR(imagedb, config->args, "-imagedb", true, false); + PXOPT_LOOKUP_STR(rlabel, config->args, "-rlabel", true, false); + PXOPT_LOOKUP_STR(outdir, config->args, "-outdir", true, false); + PXOPT_LOOKUP_BOOL(need_magic, config->args, "-need_magic", false); + PXOPT_LOOKUP_BOOL(no_create, config->args, "-no_create", false); + PXOPT_LOOKUP_BOOL(hold, config->args, "-hold", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-imagedb", "imagedb", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-component", "component", "=="); + + // start a transaction early so it will contain any row level locks + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString query = pxDataGet("pstamptool_getdependent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + if (!no_create) { + // This will lock the row until the transaction is committed + psStringAppend(&query, " AND %s FOR UPDATE", whereClause); + } else { + psStringAppend(&query, " AND %s", whereClause); + } + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (psArrayLength(output)) { + psMetadata *dep = output->data[0]; + psS64 dep_id = psMetadataLookupS64(NULL, dep, "dep_id"); + if (!dep_id) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psS32 fault = psMetadataLookupS64(NULL, dep, "fault"); + if (fault > 0) { + fprintf(stderr, "existing dependent has fault %d\n", fault); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + // return the fault to the client. This is used by the postage stamp parser + exit (fault); + } + bool commit = false; + // Check the state of the exisiting dependent. By query it is either + // new or hold + psString state = psMetadataLookupStr(NULL, dep, "state"); + if (!hold && !strcmp(state, "hold")) { + // There is a dependent for this component but it's state is hold. + // This client needs one that will run. + // Update the state + psString updateQuery = NULL; + psStringAppend(&updateQuery, "UPDATE pstampDependent SET state = 'new' " + "\nWHERE stage = '%s' AND imagedb = '%s' AND stage_id = %"PRId64 " AND component = '%s'", + stage, imagedb, stage_id, component); + + if (!p_psDBRunQuery(config->dbh, updateQuery)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(updateQuery); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + // set flag to commit this change + commit = true; + psFree(updateQuery); + } + // print the dep_id for the user + printf("%" PRId64 "\n", dep_id); + psFree(output); + // now either commit the change or rollback the transaction which releases the lock + if (commit) { + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } else if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + if (no_create) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + // no existing dependent that matches, insert one + // Since we have multiple processes running jobs we have a + // race condition here so that's why we need to lock the table + + if (!pstampDependentInsert( + config->dbh, + 0, // dep_id + hold ? "hold" : "new", // state + stage, + stage_id, + component, + imagedb, + rlabel, + need_magic, + outdir, + 0, // fault + 0 // fault_count + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to insert pstampDependent"); + return false; + } + + // if we try and get this after commit zero is returned + psS64 dep_id = psDBLastInsertID(config->dbh); + if (!dep_id) { + psError(PS_ERR_UNKNOWN, false, "psDBLastInsertID returned NULL"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + printf("%" PRId64 "\n", dep_id); + + return true; +} + +static bool pendingdependentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-imagedb", "imagedb", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_S64(config->args, where, "-dep_id", "dep_id", "=="); + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_STR(config->args, where, "-rlabel", "rlabel", "=="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(includeFaulted, config->args, "-includefaulted", false); + + psString query = pxDataGet("pstamptool_pendingdependent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!includeFaulted) { + psStringAppend(&query, " AND (pstampDependent.fault = 0)\n"); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nGROUP BY dep_id ORDER BY priority DESC, MIN(req_id), dep_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampDependent", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool updatedependentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(dep_id, config->args, "-dep_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + + if (!fault && !state) { + psError(PS_ERR_UNKNOWN, true, "at least one of -set_state or fault is required"); + return false; + } + psString query = psStringCopy("UPDATE pstampDependent SET"); + bool needComma = false; + if (state) { + psStringAppend(&query, " state = '%s'", state); + needComma = true; + } + if (fault) { + psStringAppend(&query, "%s fault = %d", needComma ? ", " : "", fault); + needComma = true; + psStringAppend(&query, ", pstampDependent.fault_count = pstampDependent.fault_count + 1"); + } + psStringAppend(&query, " WHERE dep_id = %" PRId64, dep_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected one row but %" + PRIu64 " rows were modified", affected); + return false; + } + + return true; +} +static bool revertdependentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fault", "pstampDependent.fault", "=="); + PXOPT_COPY_S64(config->args, where, "-dep_id", "dep_id", "=="); + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "pstampRequest.label", "=="); + PXOPT_LOOKUP_BOOL(clear_fault_count, config->args, "-clear_fault_count", false); + + // XXX: we don't actually use -limit. It doesn't work for UPDATE + // it's an allowed arg because add_poll_args adds it + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + if (limit) { /* do something?? */ } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + psString query = pxDataGet("pstamptool_revertdependent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + psFree(where); + + if (!fault) { + // unless asked to clear a specific fault value, restrict reverts to fault codes + // that are less than the minimum fault in the API + psStringAppend(&query, " AND (pstampDependent.fault < %d)", PSTAMP_FIRST_ERROR_CODE); + } + + // don't keep reverting once the number of faults reaches some value unless + // a parameter asking us to clear that count is provided + psString faultCountClause = NULL; + if (clear_fault_count) { + psStringAppend(&faultCountClause, "\n, pstampDependent.fault_count = 0"); + } else { + psStringAppend(&faultCountClause, " "); + psStringAppend(&query, " AND pstampDependent.fault_count < %d", PSTAMP_MAX_DEP_FAULTS); + } + + if (!p_psDBRunQueryF(config->dbh, query, faultCountClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + return true; +} + +static bool getwebrequestnumMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + if (!pstampWebRequestInsert(config->dbh, 0 )) { + psError(PS_ERR_UNKNOWN, false, "failed to insert pstampWebRequest"); + return false; + } + + psS64 req_id = psDBLastInsertID(config->dbh); + + printf("%" PRId64 "\n", req_id); + + return true; +} + +static bool addfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(job_id, config->args, "-job_id", true, false); + PXOPT_LOOKUP_STR(path, config->args, "-path", true, false); + + if (!pstampFileInsert(config->dbh, + 0, // file_id + job_id, + path + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + if (affected != 1) { + psError(PS_ERR_UNKNOWN, false, + "should have affected one row but %" PRIu64 " rows were modified", + affected); + return false; + } + + psS64 file_id = psDBLastInsertID(config->dbh); + printf("%" PRId64 "\n", file_id); + + return true; +} + +static bool listfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_S64(config->args, where, "-req_id", "req_id", "=="); + PXOPT_COPY_S64(config->args, where, "-file_id", "file_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!psListLength(where->list)) { + fprintf(stderr, "search arguments are required\n"); + exit (1); + } + + psString query = pxDataGet("pstamptool_listfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pstamptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampFile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool deletefileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(job_id, config->args, "-job_id", true, false); + + psString query = NULL; + psStringAppend(&query, "DELETE FROM pstampFile WHERE job_id = %" PRId64, job_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("pstamptool", PS_LOG_INFO, "Deleted %" PRIu64 " rows from pstampFile", affected); + + return true; +} + +static bool adddomainMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(domainName, config->args, "-set_domain", true, false); + PXOPT_LOOKUP_S32(accessLevel, config->args, "-set_accessLevel", true, false); + PXOPT_LOOKUP_STR(defaultProduct, config->args, "-set_defaultProduct", false, false); + PXOPT_LOOKUP_STR(defaultLabel, config->args, "-set_defaultLabel", false, false); + + if (!pstampUserDomainInsert(config->dbh, + domainName, + accessLevel, + defaultProduct, + defaultLabel + )) { + psError(PS_ERR_UNKNOWN, false, "failed to insert domain"); + return false; + } + + return true; +} +static bool updatedomainMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(domainName, config->args, "-domain", true, false); + PXOPT_LOOKUP_S32(accessLevel, config->args, "-set_accessLevel", false, false); + PXOPT_LOOKUP_STR(defaultProduct, config->args, "-set_defaultProduct", false, false); + PXOPT_LOOKUP_STR(defaultLabel, config->args, "-set_defaultLabel", false, false); + + if (!accessLevel && !defaultProduct && !defaultLabel) { + psError(PS_ERR_UNKNOWN, true, "at least one set option is required"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-domain", "domainName", "=="); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE pstampUserDomain SET"); + + char * sep = ""; + if (accessLevel) { + psStringAppend(&query, "%s accessLevel = '%d'", sep, accessLevel); + sep = ", "; + } + if (defaultProduct) { + psStringAppend(&query, "%s defaultProduct = '%s'", sep, defaultProduct); + sep = ", "; + } + if (defaultLabel) { + psStringAppend(&query, "%s defaultLabel = '%s'", sep, defaultLabel); + sep = ", "; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + // psU64 affected = psDBAffectedRows(config->dbh); + // psLogMsg("pstamptool", PS_LOG_INFO, "Updated %" PRIu64 " pstampRequests", affected); + + return true; +} + +static bool listdomainMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-domain", "domainName", "="); + PXOPT_COPY_S32(config->args, where, "-accessLevel", "accessLevel", "="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = psStringCopy("SELECT * from pstampUserDomain"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampUserDomain", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool adduserMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(userName, config->args, "-set_user", true, false); + PXOPT_LOOKUP_STR(domainName, config->args, "-set_domain", true, false); + PXOPT_LOOKUP_S32(accessLevel, config->args, "-set_accessLevel", false, false); + PXOPT_LOOKUP_STR(defaultProduct, config->args, "-set_defaultProduct", false, false); + PXOPT_LOOKUP_STR(defaultLabel, config->args, "-set_defaultLabel", false, false); + + if (!pstampUserInsert(config->dbh, + userName, + domainName, + accessLevel, + defaultProduct, + defaultLabel + )) { + psError(PS_ERR_UNKNOWN, false, "failed to insert user"); + return false; + } + + return true; +} +static bool updateuserMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(userName, config->args, "-user", true, false); + PXOPT_LOOKUP_STR(domainName, config->args, "-domain", true, false); + PXOPT_LOOKUP_S32(accessLevel, config->args, "-set_accessLevel", false, false); + PXOPT_LOOKUP_STR(defaultProduct, config->args, "-set_defaultProduct", false, false); + PXOPT_LOOKUP_STR(defaultLabel, config->args, "-set_defaultLabel", false, false); + + if (!accessLevel && !defaultProduct && !defaultLabel) { + psError(PS_ERR_UNKNOWN, true, "at least one set option is required"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-user", "userName", "=="); + PXOPT_COPY_STR(config->args, where, "-domain", "domainName", "=="); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE pstampUser SET"); + + char * sep = ""; + if (accessLevel) { + psStringAppend(&query, "%s accessLevel = '%d'", sep, accessLevel); + sep = ", "; + } + if (defaultProduct) { + psStringAppend(&query, "%s defaultProduct = '%s'", sep, defaultProduct); + sep = ", "; + } + if (defaultLabel) { + psStringAppend(&query, "%s defaultLabel = '%s'", sep, defaultLabel); + sep = ", "; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + // psU64 affected = psDBAffectedRows(config->dbh); + // psLogMsg("pstamptool", PS_LOG_INFO, "Updated %" PRIu64 " pstampRequests", affected); + + return true; +} +static bool listuserMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-user", "userName", "="); + PXOPT_COPY_STR(config->args, where, "-domain", "domainName", "="); + PXOPT_COPY_S32(config->args, where, "-accessLevel", "pstampUser.accessLevel", "="); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + if (!psListLength(where->list)) { + psError(PS_ERR_UNKNOWN, true, "search paramters are required"); + return false; + } + + psString query = pxDataGet("pstamptool_listuser.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement from pstamptool_listuser.sql"); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampUser", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool addaccesslevelMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(proj_id, config->args, "-set_proj_id", false, false); + PXOPT_LOOKUP_STR(project_name, config->args, "-set_project_name", false, false); + PXOPT_LOOKUP_S32(accessLevel, config->args, "-set_accessLevel", true, false); + PXOPT_LOOKUP_F32(mjd_min, config->args, "-set_mjd_min", false, false); + PXOPT_LOOKUP_F32(mjd_max, config->args, "-set_mjd_max", false, false); + + if (proj_id) { + if (!pstampAccessLevelInsert(config->dbh, + proj_id, + accessLevel, + mjd_min, + mjd_max + )) { + psError(PS_ERR_UNKNOWN, false, "failed to insert accessLevel"); + return false; + } + } else if (project_name) { + // use select insert getting proj_id from pstampProject + psString query = pxDataGet("pstamptool_addaccesslevel.sql"); + if (!p_psDBRunQueryF(config->dbh, query, accessLevel, mjd_min, mjd_max, project_name)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + } else { + psError(PS_ERR_UNKNOWN, true, "proj_id or project_name is required"); + return false; + } + + return true; +} +static bool updateaccesslevelMode(pxConfig *config) +{ + return false; +} +static bool listaccesslevelMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-project_name", "pstampProject.name", "="); + PXOPT_COPY_S64(config->args, where, "-proj_id", "proj_id", "="); + PXOPT_COPY_S32(config->args, where, "-accessLevel", "accessLevel", "="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = psStringCopy("SELECT * from pstampAccessLevel join pstampProject USING(proj_id)"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pstampAccessLevel", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +# endif Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptool.h (revision 41104) @@ -0,0 +1,72 @@ +/* + * pstamptool.h + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PSTAMPTOOL_H +#define PSTAMPTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + PSTAMPTOOL_MODE_NONE = 0x0, + PSTAMPTOOL_MODE_ADDDATASTORE, + PSTAMPTOOL_MODE_DATASTORE, + PSTAMPTOOL_MODE_MODDATASTORE, + PSTAMPTOOL_MODE_ADDREQ, + PSTAMPTOOL_MODE_LISTREQ, + PSTAMPTOOL_MODE_COMPLETEDREQ, + PSTAMPTOOL_MODE_PENDINGREQ, + PSTAMPTOOL_MODE_UPDATEREQ, + PSTAMPTOOL_MODE_REVERTREQ, + PSTAMPTOOL_MODE_PENDINGCLEANUP, + PSTAMPTOOL_MODE_ADDJOB, + PSTAMPTOOL_MODE_LISTJOB, + PSTAMPTOOL_MODE_PENDINGJOB, + PSTAMPTOOL_MODE_STOPDEPENDENTJOB, + PSTAMPTOOL_MODE_JOBRESULT, + PSTAMPTOOL_MODE_UPDATEJOB, + PSTAMPTOOL_MODE_REVERTJOB, + PSTAMPTOOL_MODE_ADDPROJECT, + PSTAMPTOOL_MODE_MODPROJECT, + PSTAMPTOOL_MODE_PROJECT, + PSTAMPTOOL_MODE_GETDEPENDENT, + PSTAMPTOOL_MODE_PENDINGDEPENDENT, + PSTAMPTOOL_MODE_UPDATEDEPENDENT, + PSTAMPTOOL_MODE_REVERTDEPENDENT, + PSTAMPTOOL_MODE_GETWEBREQUESTNUM, + PSTAMPTOOL_MODE_ADDFILE, + PSTAMPTOOL_MODE_LISTFILE, + PSTAMPTOOL_MODE_DELETEFILE, + + PSTAMPTOOL_MODE_ADDUSER, + PSTAMPTOOL_MODE_UPDATEUSER, + PSTAMPTOOL_MODE_LISTUSER, +// PSTAMPTOOL_MODE_GETUSERINFO, + + PSTAMPTOOL_MODE_ADDDOMAIN, + PSTAMPTOOL_MODE_UPDATEDOMAIN, + PSTAMPTOOL_MODE_LISTDOMAIN, + + PSTAMPTOOL_MODE_ADDACCESSLEVEL, + PSTAMPTOOL_MODE_UPDATEACCESSLEVEL, + PSTAMPTOOL_MODE_LISTACCESSLEVEL, +} pstamptoolMode; + +pxConfig *pstamptoolConfig(pxConfig *config, int argc, char **argv); + +#endif // PSTAMPTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pstamptoolConfig.c (revision 41104) @@ -0,0 +1,442 @@ +/* + * pstamptoolConfig.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pstamptool.h" + +pxConfig *pstamptoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration!\n"); + psFree(config); + return NULL; + } + + // -adddatastore + psMetadata *adddatastoreArgs = psMetadataAlloc(); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-set_uri", 0, "define storage uri (required)", NULL); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-set_out_product", 0, "define output product name (required)", NULL); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-set_last_fileset", 0, "define last fileset seen", NULL); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-set_state", 0, "define datastore state", "enabled"); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-set_label", 0, "define datastore label", NULL); + psMetadataAddS32(adddatastoreArgs, PS_LIST_TAIL, "-set_poll_interval", 0, "define datastore poll interval (seconds)", 60); + + // -datastore + psMetadata *datastoreArgs = psMetadataAlloc(); + psMetadataAddS64(datastoreArgs, PS_LIST_TAIL, "-ds_id", 0, "define ds_id", 0); + psMetadataAddBool(datastoreArgs, PS_LIST_TAIL, "-ready", 0, "list data stores ready to be polled", false); + psMetadataAddBool(datastoreArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -moddatastore + psMetadata *moddatastoreArgs = psMetadataAlloc(); + psMetadataAddS64(moddatastoreArgs, PS_LIST_TAIL, "-ds_id", 0, "define ds_id", 0); + psMetadataAddStr(moddatastoreArgs, PS_LIST_TAIL, "-set_last_fileset", 0, "set last_fileset seen", NULL); + psMetadataAddStr(moddatastoreArgs, PS_LIST_TAIL, "-set_uri", 0, "set uri for data store", NULL); + psMetadataAddStr(moddatastoreArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(moddatastoreArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddS32(moddatastoreArgs, PS_LIST_TAIL, "-set_poll_interval", 0, "define datastore poll interval (seconds)", 0); + psMetadataAddBool(moddatastoreArgs, PS_LIST_TAIL, "-update_timestamp", 0, "update the timestamp", false); + psMetadataAddStr(moddatastoreArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + + // -addreq + psMetadata *addreqArgs = psMetadataAlloc(); + psMetadataAddStr(addreqArgs, PS_LIST_TAIL, "-uri", 0, "define request file uri (required)", NULL); + psMetadataAddS64(addreqArgs, PS_LIST_TAIL, "-ds_id", 0, "define request ds_id", 0); + psMetadataAddS64(addreqArgs, PS_LIST_TAIL, "-proj_id", 0, "define request proj_id", 0); + psMetadataAddStr(addreqArgs, PS_LIST_TAIL, "-name", 0, "define request name", NULL); + psMetadataAddStr(addreqArgs, PS_LIST_TAIL, "-username", 0, "define user name", NULL); + psMetadataAddStr(addreqArgs, PS_LIST_TAIL, "-label", 0, "define request label", NULL); + + // -pendingreq + psMetadata *pendingreqArgs = psMetadataAlloc(); + psMetadataAddS64(pendingreqArgs, PS_LIST_TAIL, "-req_id", 0, "define req_id", 0); + psMetadataAddStr(pendingreqArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampRequest label (LIKE comparision)", NULL); + psMetadataAddU64(pendingreqArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingreqArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -listreq + psMetadata *listreqArgs = psMetadataAlloc(); + psMetadataAddS64(listreqArgs, PS_LIST_TAIL, "-req_id", 0, "list by req_id", 0); + psMetadataAddStr(listreqArgs, PS_LIST_TAIL, "-name", 0, "list by name", NULL); + psMetadataAddStr(listreqArgs, PS_LIST_TAIL, "-username", 0, "list by user name", NULL); + psMetadataAddStr(listreqArgs, PS_LIST_TAIL, "-state", 0, "list by state", NULL); + psMetadataAddStr(listreqArgs, PS_LIST_TAIL, "-label", 0, "list by label", NULL); + psMetadataAddS64(listreqArgs, PS_LIST_TAIL, "-not_req_id", 0, "req_id to not list", 0); + psMetadataAddU64(listreqArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listreqArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -completedreq + psMetadata *completedreqArgs = psMetadataAlloc(); + psMetadataAddStr(completedreqArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampJob label (LIKE comparision)", NULL); + psMetadataAddU64(completedreqArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(completedreqArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updatereq + psMetadata *updatereqArgs = psMetadataAlloc(); + psMetadataAddS64(updatereqArgs, PS_LIST_TAIL, "-req_id", 0, "req_id for which to update", 0); + psMetadataAddS64(updatereqArgs, PS_LIST_TAIL, "-req_id_max", 0, "maximum req_id for which to update", 0); + psMetadataAddS16(updatereqArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-reqType", 0, "search by reqType", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-name", 0, "search by reqType (LIKE comparsion)", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-username", 0, "search by username", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampJob label (LIKE comparision)", NULL); + psMetadataAddTime(updatereqArgs, PS_LIST_TAIL, "-timestamp_begin", 0, "search by timestamp (>=)", NULL); + psMetadataAddTime(updatereqArgs, PS_LIST_TAIL, "-timestamp_end", 0, "search by timestamp (<=)", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_state", 0, "new state", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_label", 0, "new label", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_outProduct", 0, "new outProduct", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_outdir", 0, "new outdir", NULL); + psMetadataAddS16(updatereqArgs, PS_LIST_TAIL, "-set_fault", 0, "new fault code", 0); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_uri", 0, "new uri", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_name", 0, "new name", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_username", 0, "new username", NULL); + psMetadataAddStr(updatereqArgs, PS_LIST_TAIL, "-set_reqType", 0, "new reqType", NULL); + psMetadataAddBool(updatereqArgs, PS_LIST_TAIL, "-clearfault", 0, "set fault to zero", false); + + // -revertreq + psMetadata *revertreqArgs = psMetadataAlloc(); + psMetadataAddS64(revertreqArgs, PS_LIST_TAIL, "-req_id", 0, "req_id to revert", 0); + psMetadataAddS16(revertreqArgs, PS_LIST_TAIL, "-fault", 0, "fault to revert", 0); + psMetadataAddStr(revertreqArgs, PS_LIST_TAIL, "-state", 0, "state to revert", NULL); + psMetadataAddStr(revertreqArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampRequest label (LIKE comparision)", NULL); + // this argument is supplied by pantasks so we accept it but don't use it + psMetadataAddU64(revertreqArgs, PS_LIST_TAIL, "-limit", 0, "not used", 0); + + // -pendingcleanup + psMetadata *pendingcleanupArgs = psMetadataAlloc(); + psMetadataAddS64(pendingcleanupArgs, PS_LIST_TAIL, "-req_id", 0, "define req_id", 0); + psMetadataAddS64(pendingcleanupArgs, PS_LIST_TAIL, "-ds_id", 0, "define ds_id", 0); + psMetadataAddStr(pendingcleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampRequest label (LIKE comparision)", NULL); + psMetadataAddU64(pendingcleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingcleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addjob + psMetadata *addjobArgs = psMetadataAlloc(); + psMetadataAddS64(addjobArgs, PS_LIST_TAIL, "-req_id", 0, "define job req_id", 0); + psMetadataAddStr(addjobArgs, PS_LIST_TAIL, "-rownum", 0, "define job rownum", NULL); + psMetadataAddStr(addjobArgs, PS_LIST_TAIL, "-job_type", 0, "define job job_type", "stamp"); + psMetadataAddStr(addjobArgs, PS_LIST_TAIL, "-outputBase", 0, "define job outputBase", NULL); + psMetadataAddStr(addjobArgs, PS_LIST_TAIL, "-state", 0, "define state", "run"); + psMetadataAddS64(addjobArgs, PS_LIST_TAIL, "-exp_id", 0, "define exposure id", 0); + psMetadataAddS64(addjobArgs, PS_LIST_TAIL, "-options", 0, "define options", 0); + psMetadataAddS64(addjobArgs, PS_LIST_TAIL, "-dep_id", 0, "define job dep_id", 0); + psMetadataAddS64(addjobArgs, PS_LIST_TAIL, "-parent_id", 0, "define parent's job_id", 0); + psMetadataAddBool(addjobArgs, PS_LIST_TAIL, "-is_parent", 0, "define whether job has children", false); + psMetadataAddS16(addjobArgs, PS_LIST_TAIL, "-fault", 0, "define job result", 0); + + // -listjob + psMetadata *listjobArgs = psMetadataAlloc(); + psMetadataAddS64(listjobArgs, PS_LIST_TAIL, "-req_id", 0, "select by request ID", 0); + psMetadataAddS64(listjobArgs, PS_LIST_TAIL, "-job_id", 0, "select by job ID", 0); + psMetadataAddS64(listjobArgs, PS_LIST_TAIL, "-dep_id", 0, "select by dependent ID", 0); + psMetadataAddS16(listjobArgs, PS_LIST_TAIL, "-fault", 0, "select by fault", 0); + psMetadataAddStr(listjobArgs, PS_LIST_TAIL, "-jobType", 0, "select by jobType", 0); + psMetadataAddStr(listjobArgs, PS_LIST_TAIL, "-state", 0, "select by job state", 0); + psMetadataAddU64(listjobArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listjobArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingjob + psMetadata *pendingjobArgs = psMetadataAlloc(); + psMetadataAddS64(pendingjobArgs, PS_LIST_TAIL, "-job_id", 0, "define job", 0); + psMetadataAddS64(pendingjobArgs, PS_LIST_TAIL, "-req_id", 0, "define request", 0); + psMetadataAddStr(pendingjobArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampJob label (LIKE comparision)", NULL); + psMetadataAddU64(pendingjobArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingjobArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updatejob + psMetadata *updatejobArgs = psMetadataAlloc(); + psMetadataAddS64(updatejobArgs, PS_LIST_TAIL, "-req_id", 0, "req_id of jobs to update", 0); + psMetadataAddS64(updatejobArgs, PS_LIST_TAIL, "-job_id", 0, "job_id of jobs to update", 0); + psMetadataAddS64(updatejobArgs, PS_LIST_TAIL, "-dep_id", 0, "dep_id of jobs to update", 0); + psMetadataAddStr(updatejobArgs, PS_LIST_TAIL, "-state", 0, "current state of jobs to update", 0); + psMetadataAddS16(updatejobArgs, PS_LIST_TAIL, "-fault", 0, "current value for job fault", 0); + psMetadataAddStr(updatejobArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampJob label (LIKE comparision)", NULL); + psMetadataAddS32(updatejobArgs, PS_LIST_TAIL, "-fault_count", 0, "select by fault_count (>=)", 0); + psMetadataAddStr(updatejobArgs, PS_LIST_TAIL, "-set_state", 0, "new state", NULL); + psMetadataAddS16(updatejobArgs, PS_LIST_TAIL, "-set_fault", 0, "new result", 0); + psMetadataAddU64(updatejobArgs, PS_LIST_TAIL, "-limit", 0, "not used", 0); + + // -stopdependentjob + psMetadata *stopdependentjobArgs = psMetadataAlloc(); + psMetadataAddS64(stopdependentjobArgs, PS_LIST_TAIL, "-req_id", 0, "req_id of jobs to update", 0); + psMetadataAddS64(stopdependentjobArgs, PS_LIST_TAIL, "-job_id", 0, "job_id of jobs to update", 0); + psMetadataAddS64(stopdependentjobArgs, PS_LIST_TAIL, "-dep_id", 0, "dep_id of jobs to update", 0); + psMetadataAddS64(stopdependentjobArgs, PS_LIST_TAIL, "-stage_id", 0, "stage_id of jobs to update", 0); + psMetadataAddStr(stopdependentjobArgs, PS_LIST_TAIL, "-component", 0, "component of jobs to update", NULL); + psMetadataAddS16(stopdependentjobArgs, PS_LIST_TAIL, "-fault", 0, "current value for dependent fault", 0); + psMetadataAddS32(stopdependentjobArgs, PS_LIST_TAIL, "-fault_count", 0, "select by fault_count (>=)", 0); + psMetadataAddS16(stopdependentjobArgs, PS_LIST_TAIL, "-set_fault", 0, "new fault value for job and dependent (required)", 0); + psMetadataAddStr(stopdependentjobArgs, PS_LIST_TAIL, "-set_state", 0, "new pstampDependent.state", "new"); + psMetadataAddStr(stopdependentjobArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampJob label (LIKE comparision)", NULL); + psMetadataAddU64(stopdependentjobArgs, PS_LIST_TAIL, "-limit", 0, "not used", 0); + + // -revertjob + psMetadata *revertjobArgs = psMetadataAlloc(); + psMetadataAddS64(revertjobArgs, PS_LIST_TAIL, "-req_id", 0, "req_id to revert", 0); + psMetadataAddS64(revertjobArgs, PS_LIST_TAIL, "-req_id_min", 0, "minimum req_id to revert", 0); + psMetadataAddS64(revertjobArgs, PS_LIST_TAIL, "-job_id", 0, "job_id to revert", 0); + psMetadataAddS16(revertjobArgs, PS_LIST_TAIL, "-fault", 0, "fault to revert", 0); + psMetadataAddStr(revertjobArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by pstampRequest label (LIKE comparision)", NULL); + psMetadataAddBool(revertjobArgs, PS_LIST_TAIL, "-clear_fault_count", 0, "clear job fault count", false); + psMetadataAddBool(revertjobArgs, PS_LIST_TAIL, "-all", 0, "revert all faulted jobs", false); + psMetadataAddU64(revertjobArgs, PS_LIST_TAIL, "-limit", 0, "not used", 0); + + // -getdependent + psMetadata *getdependentArgs = psMetadataAlloc(); + // arguabely all of these could be -set_ arguments, but since this mode is both a query and + // a create if doesn't exist I like it this way + psMetadataAddS64(getdependentArgs, PS_LIST_TAIL, "-stage_id", 0, "define stage id for dependent (required)", 0); + psMetadataAddStr(getdependentArgs, PS_LIST_TAIL, "-stage", 0, "define stage for dependent (required)", NULL); + psMetadataAddStr(getdependentArgs, PS_LIST_TAIL, "-component", 0, "define component for depenent (required)", NULL); + psMetadataAddStr(getdependentArgs, PS_LIST_TAIL, "-imagedb", 0, "define imagedb for depenent (required)", NULL); + psMetadataAddStr(getdependentArgs, PS_LIST_TAIL, "-rlabel", 0, "define label for dependent ", NULL); + psMetadataAddBool(getdependentArgs,PS_LIST_TAIL, "-need_magic", 0, "define need_magic", false); + psMetadataAddStr(getdependentArgs, PS_LIST_TAIL, "-outdir", 0, "define output directory for dependent (required)", NULL); + psMetadataAddBool(getdependentArgs,PS_LIST_TAIL, "-hold", 0, "if creating new dependent set it's state to hold", false); + psMetadataAddBool(getdependentArgs,PS_LIST_TAIL, "-no_create", 0, "if no matching dependent do not create one", false); + + // -updatedependent + psMetadata *updatedependentArgs = psMetadataAlloc(); + psMetadataAddS64(updatedependentArgs, PS_LIST_TAIL, "-dep_id", 0, "define id for dependent (required)", 0); + psMetadataAddStr(updatedependentArgs, PS_LIST_TAIL, "-set_state", 0, "new value for state", NULL); + psMetadataAddS16(updatedependentArgs, PS_LIST_TAIL, "-set_fault", 0, "new value for fault", 0); + + // -revertdependent + psMetadata *revertdependentArgs = psMetadataAlloc(); + psMetadataAddS64(revertdependentArgs, PS_LIST_TAIL, "-dep_id", 0, "search by dep_id for dependent", 0); + psMetadataAddS64(revertdependentArgs, PS_LIST_TAIL, "-job_id", 0, "search by job_ idfor dependent", 0); + psMetadataAddS64(revertdependentArgs, PS_LIST_TAIL, "-req_id", 0, "search by req_id for dependent", 0); + psMetadataAddStr(revertdependentArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label for dependent ", NULL); + psMetadataAddS16(revertdependentArgs, PS_LIST_TAIL, "-fault", 0, "search by dependent fault", 0); + psMetadataAddBool(revertdependentArgs, PS_LIST_TAIL, "-clear_fault_count", 0, "clear job fault count", false); + psMetadataAddU64(revertdependentArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingdependent + psMetadata *pendingdependentArgs = psMetadataAlloc(); + psMetadataAddS64(pendingdependentArgs, PS_LIST_TAIL, "-stage_id", 0, "define id for dependent", 0); + psMetadataAddStr(pendingdependentArgs, PS_LIST_TAIL, "-stage", 0, "define stage for dependent", NULL); + psMetadataAddStr(pendingdependentArgs, PS_LIST_TAIL, "-component", 0, "define component for dependent", NULL); + psMetadataAddS64(pendingdependentArgs, PS_LIST_TAIL, "-dep_id", 0, "define dep_id for dependent", 0); + psMetadataAddS64(pendingdependentArgs, PS_LIST_TAIL, "-job_id", 0, "define job_id for dependent", 0); + psMetadataAddS64(pendingdependentArgs, PS_LIST_TAIL, "-req_id", 0, "define eqp_id for dependent", 0); + psMetadataAddStr(pendingdependentArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "define label for pstampRequest", NULL); + psMetadataAddStr(pendingdependentArgs, PS_LIST_TAIL, "-rlabel", 0, "define label for dependent", NULL); + psMetadataAddStr(pendingdependentArgs, PS_LIST_TAIL, "-imagedb", 0, "define imagedb for depenent", NULL); + psMetadataAddBool(pendingdependentArgs, PS_LIST_TAIL, "-includefaulted", 0, "include faulted dependents in the output", false); + psMetadataAddU64(pendingdependentArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingdependentArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addproject + psMetadata *addprojectArgs = psMetadataAlloc(); + psMetadataAddStr(addprojectArgs, PS_LIST_TAIL, "-name", 0, "define project name (required)", NULL); + psMetadataAddStr(addprojectArgs, PS_LIST_TAIL, "-state", 0, "define state for project (enabled, disabled)", "enabled"); + psMetadataAddStr(addprojectArgs, PS_LIST_TAIL, "-imagedb", 0, "define name of database for project (required)", NULL); + psMetadataAddStr(addprojectArgs, PS_LIST_TAIL, "-dvodb", 0, "define name of dvo database for project", NULL); + psMetadataAddStr(addprojectArgs, PS_LIST_TAIL, "-inst", 0, "define name of camera for project (required)", NULL); + psMetadataAddStr(addprojectArgs, PS_LIST_TAIL, "-telescope", 0, "define name of telescope for project (required)", NULL); + psMetadataAddBool(addprojectArgs, PS_LIST_TAIL, "-need_magic", 0, "define need_magic for project", false); + + // -modproject + psMetadata *modprojectArgs = psMetadataAlloc(); + psMetadataAddS64(modprojectArgs, PS_LIST_TAIL, "-proj_id", 0, "define project ID to modify (required)", 0); + psMetadataAddStr(modprojectArgs, PS_LIST_TAIL, "-imagedb", 0, "define name of database for project", NULL); + psMetadataAddStr(modprojectArgs, PS_LIST_TAIL, "-state", 0, "define state for project (enabled, disabled)", NULL); + psMetadataAddStr(modprojectArgs, PS_LIST_TAIL, "-dvodb", 0, "define name of dvo database for project", NULL); + psMetadataAddStr(modprojectArgs, PS_LIST_TAIL, "-camera", 0, "define name of camera for project", NULL); + psMetadataAddStr(modprojectArgs, PS_LIST_TAIL, "-telescope", 0, "define name of telescope for project", NULL); + psMetadataAddBool(modprojectArgs, PS_LIST_TAIL, "-need_magic", 0, "define need_magic for project", false); + + // -project + psMetadata *projectArgs = psMetadataAlloc(); + psMetadataAddStr(projectArgs, PS_LIST_TAIL, "-name", 0, "define project name to list", NULL); + psMetadataAddBool(projectArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -getwebrequestnum + psMetadata *getwebrequestnumArgs = psMetadataAlloc(); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + // -addfile + psMetadata *addfileArgs = psMetadataAlloc(); + psMetadataAddS64(addfileArgs, PS_LIST_TAIL, "-job_id", 0, "define job ID for file (required)", 0); + psMetadataAddStr(addfileArgs, PS_LIST_TAIL, "-path", 0, "define path for file (required)", NULL); + + // -listfile + psMetadata *listfileArgs = psMetadataAlloc(); + psMetadataAddS64(listfileArgs, PS_LIST_TAIL, "-file_id", 0, "select by file ID", 0); + psMetadataAddS64(listfileArgs, PS_LIST_TAIL, "-job_id", 0, "select by job ID", 0); + psMetadataAddS64(listfileArgs, PS_LIST_TAIL, "-req_id", 0, "select by request ID", 0); + psMetadataAddU64(listfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -deletefile + psMetadata *deletefileArgs = psMetadataAlloc(); + psMetadataAddS64(deletefileArgs, PS_LIST_TAIL, "-job_id", 0, "select by job ID (required)", 0); + + // Access control tables + + // -adddomain + psMetadata *adddomainArgs = psMetadataAlloc(); + psMetadataAddStr(adddomainArgs, PS_LIST_TAIL, "-set_domain", 0, "define domain name (required)", NULL); + psMetadataAddS32(adddomainArgs, PS_LIST_TAIL, "-set_accessLevel", 0, "define access level for this domain (required)", 0); + psMetadataAddStr(adddomainArgs, PS_LIST_TAIL, "-set_defaultProduct", 0, "define default product", NULL); + psMetadataAddStr(adddomainArgs, PS_LIST_TAIL, "-set_defaultLabel", 0, "define default Label", NULL); + + // -updatedomain + psMetadata *updatedomainArgs = psMetadataAlloc(); + psMetadataAddStr(updatedomainArgs, PS_LIST_TAIL, "-domain", 0, "define domain name (required)", NULL); + psMetadataAddS32(updatedomainArgs, PS_LIST_TAIL, "-set_accessLevel", 0, "define access level for this domain", 0); + psMetadataAddStr(updatedomainArgs, PS_LIST_TAIL, "-set_defaultProduct", 0, "define default product", NULL); + psMetadataAddStr(updatedomainArgs, PS_LIST_TAIL, "-set_defaultLabel", 0, "define default Label", NULL); + + // -listdomain + psMetadata *listdomainArgs = psMetadataAlloc(); + psMetadataAddStr(listdomainArgs, PS_LIST_TAIL, "-domain", 0, "define domain name", NULL); + psMetadataAddS32(listdomainArgs, PS_LIST_TAIL, "-accessLevel", 0, "select by access level", 0); + psMetadataAddU64(listdomainArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listdomainArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -adduser + psMetadata *adduserArgs = psMetadataAlloc(); + psMetadataAddStr(adduserArgs, PS_LIST_TAIL, "-set_user", 0, "define user name (required)", NULL); + psMetadataAddStr(adduserArgs, PS_LIST_TAIL, "-set_domain", 0, "define user's domain name (required)", NULL); + psMetadataAddS32(adduserArgs, PS_LIST_TAIL, "-set_accessLevel", 0, "define maximum access level for this user", 0); + psMetadataAddStr(adduserArgs, PS_LIST_TAIL, "-set_defaultProduct", 0, "define default data store product for this user", NULL); + psMetadataAddStr(adduserArgs, PS_LIST_TAIL, "-set_defaultLabel", 0, "define default label for requests from this user", NULL); + + // -updateuser + psMetadata *updateuserArgs = psMetadataAlloc(); + psMetadataAddStr(updateuserArgs, PS_LIST_TAIL, "-user", 0, "select user to update (required)", NULL); + psMetadataAddStr(updateuserArgs, PS_LIST_TAIL, "-domain", 0, "select domain of user to update (required)", NULL); + psMetadataAddS32(updateuserArgs, PS_LIST_TAIL, "-set_accessLevel", 0, "define access level for this domain", 0); + psMetadataAddStr(updateuserArgs, PS_LIST_TAIL, "-set_defaultProduct", 0, "define default product", NULL); + psMetadataAddStr(updateuserArgs, PS_LIST_TAIL, "-set_defaultLabel", 0, "define default Label", NULL); + + // -listuser + psMetadata *listuserArgs = psMetadataAlloc(); + psMetadataAddStr(listuserArgs, PS_LIST_TAIL, "-user", 0, "select by user name", NULL); + psMetadataAddStr(listuserArgs, PS_LIST_TAIL, "-domain", 0, "select by user's domain name", NULL); + psMetadataAddS32(listuserArgs, PS_LIST_TAIL, "-accessLevel", 0, "select by maximum access level", 0); + psMetadataAddU64(listuserArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listuserArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addaccesslevel + psMetadata *addaccesslevelArgs = psMetadataAlloc(); + psMetadataAddStr(addaccesslevelArgs, PS_LIST_TAIL, "-set_project_name", 0, "define project name", NULL); + psMetadataAddS64(addaccesslevelArgs, PS_LIST_TAIL, "-set_proj_id", 0, "define project ID", 0); + psMetadataAddS32(addaccesslevelArgs, PS_LIST_TAIL, "-set_accessLevel", 0, "define access level (required)", 0); + psMetadataAddF32(addaccesslevelArgs, PS_LIST_TAIL, "-set_mjd_min", 0, "define mjd_min", 0.0); + psMetadataAddF32(addaccesslevelArgs, PS_LIST_TAIL, "-set_mjd_max", 0, "define mjd_max", 0.0); + + // -listaccesslevel + psMetadata *listaccesslevelArgs = psMetadataAlloc(); + psMetadataAddStr(listaccesslevelArgs, PS_LIST_TAIL, "-project_name", 0, "search by project name", NULL); + psMetadataAddS64(listaccesslevelArgs, PS_LIST_TAIL, "-proj_id", 0, "select by project ID", 0); + psMetadataAddS32(listaccesslevelArgs, PS_LIST_TAIL, "-accessLevel", 0, "select by access level", 0); + psMetadataAddU64(listaccesslevelArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listaccesslevelArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updateaccesslevel + psMetadata *updateaccesslevelArgs = psMetadataAlloc(); + psMetadataAddStr(updateaccesslevelArgs, PS_LIST_TAIL, "-project_name", 0, "select by project name", NULL); + psMetadataAddS64(updateaccesslevelArgs, PS_LIST_TAIL, "-proj_id", 0, "select by project ID", 0); + psMetadataAddS32(updateaccesslevelArgs, PS_LIST_TAIL, "-accessLevel", 0, "select by access level (required)", 0); + psMetadataAddF32(updateaccesslevelArgs, PS_LIST_TAIL, "-set_mjd_min", 0, "define new mjd_min", NAN); + psMetadataAddF32(updateaccesslevelArgs, PS_LIST_TAIL, "-set_mjd_max", 0, "define new mjd_max", NAN); + + PXOPT_ADD_MODE("-addreq", "", PSTAMPTOOL_MODE_ADDREQ, addreqArgs); + PXOPT_ADD_MODE("-pendingreq", "", PSTAMPTOOL_MODE_PENDINGREQ, pendingreqArgs); + PXOPT_ADD_MODE("-updatereq", "", PSTAMPTOOL_MODE_UPDATEREQ, updatereqArgs); + PXOPT_ADD_MODE("-listreq", "", PSTAMPTOOL_MODE_LISTREQ, listreqArgs); + PXOPT_ADD_MODE("-completedreq", "", PSTAMPTOOL_MODE_COMPLETEDREQ, completedreqArgs); + PXOPT_ADD_MODE("-revertreq", "", PSTAMPTOOL_MODE_REVERTREQ, revertreqArgs); + PXOPT_ADD_MODE("-pendingcleanup", "", PSTAMPTOOL_MODE_PENDINGCLEANUP, pendingcleanupArgs); + + PXOPT_ADD_MODE("-addjob", "", PSTAMPTOOL_MODE_ADDJOB, addjobArgs); + PXOPT_ADD_MODE("-listjob", "", PSTAMPTOOL_MODE_LISTJOB, listjobArgs); + PXOPT_ADD_MODE("-pendingjob", "", PSTAMPTOOL_MODE_PENDINGJOB, pendingjobArgs); + PXOPT_ADD_MODE("-updatejob", "", PSTAMPTOOL_MODE_UPDATEJOB, updatejobArgs); + PXOPT_ADD_MODE("-stopdependentjob", "", PSTAMPTOOL_MODE_STOPDEPENDENTJOB, stopdependentjobArgs); + PXOPT_ADD_MODE("-revertjob", "", PSTAMPTOOL_MODE_REVERTJOB, revertjobArgs); + + PXOPT_ADD_MODE("-adddatastore", "", PSTAMPTOOL_MODE_ADDDATASTORE, adddatastoreArgs); + PXOPT_ADD_MODE("-datastore", "", PSTAMPTOOL_MODE_DATASTORE, datastoreArgs); + PXOPT_ADD_MODE("-moddatastore", "", PSTAMPTOOL_MODE_MODDATASTORE, moddatastoreArgs); + + PXOPT_ADD_MODE("-getdependent", "", PSTAMPTOOL_MODE_GETDEPENDENT, getdependentArgs); + PXOPT_ADD_MODE("-updatedependent", "", PSTAMPTOOL_MODE_UPDATEDEPENDENT, updatedependentArgs); + PXOPT_ADD_MODE("-pendingdependent","", PSTAMPTOOL_MODE_PENDINGDEPENDENT, pendingdependentArgs); + PXOPT_ADD_MODE("-revertdependent","", PSTAMPTOOL_MODE_REVERTDEPENDENT, revertdependentArgs); + + PXOPT_ADD_MODE("-addproject", "", PSTAMPTOOL_MODE_ADDPROJECT, addprojectArgs); + PXOPT_ADD_MODE("-modproject", "", PSTAMPTOOL_MODE_MODPROJECT, modprojectArgs); + PXOPT_ADD_MODE("-project", "", PSTAMPTOOL_MODE_PROJECT, projectArgs); + PXOPT_ADD_MODE("-getwebrequestnum","", PSTAMPTOOL_MODE_GETWEBREQUESTNUM, getwebrequestnumArgs); + PXOPT_ADD_MODE("-addfile", "", PSTAMPTOOL_MODE_ADDFILE, addfileArgs); + PXOPT_ADD_MODE("-listfile", "", PSTAMPTOOL_MODE_LISTFILE, listfileArgs); + PXOPT_ADD_MODE("-deletefile", "", PSTAMPTOOL_MODE_DELETEFILE, deletefileArgs); + + PXOPT_ADD_MODE("-adddomain", "", PSTAMPTOOL_MODE_ADDDOMAIN, adddomainArgs); + PXOPT_ADD_MODE("-updatedomain", "", PSTAMPTOOL_MODE_UPDATEDOMAIN, updatedomainArgs); + PXOPT_ADD_MODE("-listdomain", "", PSTAMPTOOL_MODE_LISTDOMAIN, listdomainArgs); + PXOPT_ADD_MODE("-adduser", "", PSTAMPTOOL_MODE_ADDUSER, adduserArgs); + PXOPT_ADD_MODE("-updateuser", "", PSTAMPTOOL_MODE_UPDATEUSER, updateuserArgs); + PXOPT_ADD_MODE("-listuser", "", PSTAMPTOOL_MODE_LISTUSER, listuserArgs); + PXOPT_ADD_MODE("-addaccesslevel", "", PSTAMPTOOL_MODE_ADDACCESSLEVEL, addaccesslevelArgs); + PXOPT_ADD_MODE("-listaccesslevel", "", PSTAMPTOOL_MODE_LISTACCESSLEVEL, listaccesslevelArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtool.c (revision 41104) @@ -0,0 +1,500 @@ +/* + * pubtool.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "pubtool.h" + +static bool defineclientMode(pxConfig *config); +static bool updateclientMode(pxConfig *config); +static bool definerunMode(pxConfig *config); +static bool pendingMode(pxConfig *config); +static bool addMode(pxConfig *config); +static bool revertMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pubtoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(PUBTOOL_MODE_DEFINECLIENT, defineclientMode); + MODECASE(PUBTOOL_MODE_UPDATECLIENT, updateclientMode); + MODECASE(PUBTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(PUBTOOL_MODE_PENDING, pendingMode); + MODECASE(PUBTOOL_MODE_ADD, addMode); + MODECASE(PUBTOOL_MODE_REVERT, revertMode); + MODECASE(PUBTOOL_MODE_UPDATERUN, updaterunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool defineclientMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(product, config->args, "-product", true, false); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + + // optional + PXOPT_LOOKUP_STR(comment, config->args, "-comment", false, false); + PXOPT_LOOKUP_BOOL(unmagicked, config->args, "-unmagicked", false); + PXOPT_LOOKUP_STR(name, config->args, "-name", false, false); + PXOPT_LOOKUP_S16(output_format, config->args, "-output_format", false, false); + + if (!publishClientInsert(config->dbh, 0, 0, product, stage, !unmagicked, workdir, comment, name, output_format)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + +static bool updateclientMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + PXOPT_COPY_S64(config->args, where, "-client_id", "client_id", "=="); + PXOPT_COPY_STR(config->args, where, "-product", "product", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "comment", "LIKE"); + + PXOPT_LOOKUP_BOOL(active, config->args, "-active", false); + PXOPT_LOOKUP_BOOL(inactive, config->args, "-inactive", false); + + if ((!active && !inactive) || (active && inactive)) { + psError(PS_ERR_UNKNOWN, false, "Must specify one, and only one, of -active and -inactive."); + psFree(where); + return false; + } + + psString query = NULL; // Query to run + psStringAppend(&query, "UPDATE publishClient SET active = %d", active ? 1 : 0); + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n WHERE %s", clause); + psFree(clause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + long numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("pubtool", PS_LOG_INFO, "%ld rows updated.", numUpdated); + + return true; +} + +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *diffWhere = psMetadataAlloc(); // WHERE conditions for diffs + psMetadata *camWhere = psMetadataAlloc(); // WHERE conditions for cams + psMetadata *diffphotWhere = psMetadataAlloc(); // WHERE conditions for diffphots + + // required + + // optional + PXOPT_COPY_S64(config->args, diffWhere, "-client_id", "client_id", "=="); + pxAddLabelSearchArgs(config, diffWhere, "-label", "diffRun.label", "=="); + pxAddLabelSearchArgs(config, diffWhere, "-data_group", "diffRun.data_group", "LIKE"); + PXOPT_COPY_TIME(config->args, diffWhere, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, diffWhere, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, diffWhere, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_STR(config->args, diffWhere, "-obs_mode", "rawExp.obs_mode", "LIKE"); + PXOPT_COPY_STR(config->args, diffWhere, "-comment", "rawExp.comment", "LIKE"); + + PXOPT_COPY_S64(config->args, camWhere, "-client_id", "client_id", "=="); + pxAddLabelSearchArgs(config, camWhere, "-label", "camRun.label", "=="); + pxAddLabelSearchArgs(config, camWhere, "-data_group", "camRun.data_group", "LIKE"); + PXOPT_COPY_TIME(config->args, camWhere, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, camWhere, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, camWhere, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_STR(config->args, camWhere, "-obs_mode", "rawExp.obs_mode", "LIKE"); + + PXOPT_COPY_S64(config->args, diffphotWhere, "-client_id", "client_id", "=="); + pxAddLabelSearchArgs(config, diffphotWhere, "-label", "diffPhotRun.label", "=="); + pxAddLabelSearchArgs(config, diffphotWhere, "-data_group", "diffPhotRun.data_group", "LIKE"); + PXOPT_COPY_TIME(config->args, diffphotWhere, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, diffphotWhere, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, diffphotWhere, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_STR(config->args, diffphotWhere, "-obs_mode", "rawExp.obs_mode", "LIKE"); + + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pubtool_definerun.sql"); // Query to run + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(diffWhere); + psFree(camWhere); + return false; + } + + if (!rerun) { + psStringAppend(&query, "\nWHERE publishRun.client_id IS NULL"); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(diffWhere); + psFree(camWhere); + psFree(diffphotWhere); + return false; + } + + psString whereDiff = psStringCopy(""); // Additional constraints to add to query + if (psListLength(diffWhere->list)) { + psString clause = psDBGenerateWhereConditionSQL(diffWhere, NULL); + psStringAppend(&whereDiff, "\n AND %s", clause); + psFree(clause); + } + psFree(diffWhere); + + psString whereCam = psStringCopy(""); // Additional constraints to add to query + if (psListLength(camWhere->list)) { + psString clause = psDBGenerateWhereConditionSQL(camWhere, NULL); + psStringAppend(&whereCam, "\n AND %s", clause); + psFree(clause); + } + psFree(camWhere); + + psString whereDiffphot = psStringCopy(""); // Additional constraints to add to query + if (psListLength(diffphotWhere->list)) { + psString clause = psDBGenerateWhereConditionSQL(diffphotWhere, NULL); + psStringAppend(&whereDiffphot, "\n AND %s", clause); + psFree(clause); + } + psFree(diffphotWhere); + + if (!p_psDBRunQueryF(config->dbh, query, whereDiff, whereCam, whereDiffphot)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + psFree(whereDiff); + psFree(whereCam); + psFree(whereDiffphot); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + psFree(query); + psFree(whereDiff); + psFree(whereCam); + psFree(whereDiffphot); + + psArray *output = p_psDBFetchResult(config->dbh); // Output of SELECT statement + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + if (!psArrayLength(output)) { + psTrace("pubtool", PS_LOG_INFO, "No rows found"); + psFree(output); + return true; + } + + if (pretend) { + if (!ippdbPrintMetadatas(stdout, output, "publishRun", !simple)) { + psError(psErrorCodeLast(), false, "Failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + for (int i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row of interest + psS64 client = psMetadataLookupS64(NULL, row, "client_id"); // Client identifier + psS64 stage = psMetadataLookupS64(NULL, row, "stage_id"); // Stage identifier + const char *label = psMetadataLookupStr(NULL, row, "src_label"); // label from correct source + + if (!publishRunInsert(config->dbh, 0, client, stage, set_label ? set_label : label, "new")) { + psError(PS_ERR_UNKNOWN, false, "Unable to add fileset"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + +static bool pendingMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + + // required + PXOPT_COPY_STR(config->args, where, "-client_id", "publishClient.client_id", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "publishClient.stage", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "publishClient.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-name", "publishClient.name", "LIKE"); + pxAddLabelSearchArgs(config, where, "-label", "publishRun.label", "=="); + + // optional + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("pubtool_pending.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString whereClause = psStringCopy(""); // WHERE conditions to add + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereClause, "\nAND %s", clause); + psFree(clause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause, whereClause, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(whereClause); + psFree(query); + return false; + } + psFree(whereClause); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pubtool", PS_LOG_INFO, "No rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "publishRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "Failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool addMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(pub_id, config->args, "-pub_id", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + // optional + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_S32(fault, config->args, "-fault", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + if (!publishDoneInsert(config->dbh, pub_id, path_base, hostname, dtime_script, fault)) { + psError(PS_ERR_UNKNOWN, false, "Unable to add file"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + + if (fault == 0) { + if (!p_psDBRunQueryF(config->dbh, + "UPDATE publishRun SET state = 'full' WHERE pub_id = %" PRId64, + pub_id)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + + + + +static bool revertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + PXOPT_COPY_S64(config->args, where, "-pub_id", "publishRun.pub_id", "=="); + PXOPT_COPY_S32(config->args, where, "-fault", "publishDone.fault", "=="); + PXOPT_COPY_STR(config->args, where, "-client_id", "publishClient.client_id", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "publishClient.comment", "LIKE"); + pxAddLabelSearchArgs(config, where, "-label", "publishRun.label", "=="); + + psString query = pxDataGet("pubtool_revert.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + psLogMsg("pubtool", PS_LOG_INFO, "Deleted %" PRIu64 " rows", psDBAffectedRows(config->dbh)); + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + PXOPT_COPY_S64(config->args, where, "-pub_id", "pub_id", "=="); + PXOPT_COPY_S64(config->args, where, "-client_id", "client_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-fault", "fault", "=="); + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + + psString query = NULL; // Query to run + psStringAppend(&query, "UPDATE publishRun LEFT JOIN publishDone using(pub_id) SET state = '%s'", state); + + if (psListLength(where->list)) { + psString clause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n WHERE %s", clause); + psFree(clause); + } else { + psError(PS_ERR_UNKNOWN, false, "select arguments are required"); + return false; + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + long numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("pubtool", PS_LOG_INFO, "%ld rows updated.", numUpdated); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtool.h (revision 41104) @@ -0,0 +1,38 @@ +/* + * pubtool.h + * + * Copyright (C) 2008 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PUBTOOL_H +#define PUBTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + PUBTOOL_MODE_NONE = 0x0, + PUBTOOL_MODE_DEFINECLIENT, + PUBTOOL_MODE_UPDATECLIENT, + PUBTOOL_MODE_DEFINERUN, + PUBTOOL_MODE_PENDING, + PUBTOOL_MODE_ADD, + PUBTOOL_MODE_REVERT, + PUBTOOL_MODE_UPDATERUN, +} pubtoolMode; + +pxConfig *pubtoolConfig(pxConfig *config, int argc, char **argv); + +#endif // PUBTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pubtoolConfig.c (revision 41104) @@ -0,0 +1,148 @@ +/* + * pubtoolConfig.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pubtool.h" + +pxConfig *pubtoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration!"); + psFree(config); + return NULL; + } + + // -defineclient + psMetadata *defineclientArgs = psMetadataAlloc(); + psMetadataAddStr(defineclientArgs, PS_LIST_TAIL, "-stage", 0, "define stage (required)", NULL); + psMetadataAddStr(defineclientArgs, PS_LIST_TAIL, "-product", 0, "define product (required)", NULL); + psMetadataAddStr(defineclientArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddS16(defineclientArgs, PS_LIST_TAIL, "-output_format", 0, "define output format", 2); + psMetadataAddStr(defineclientArgs, PS_LIST_TAIL, "-comment", 0, "define comment", NULL); + psMetadataAddStr(defineclientArgs, PS_LIST_TAIL, "-name", 0, "define name", NULL); + psMetadataAddBool(defineclientArgs, PS_LIST_TAIL, "-unmagicked", 0, "allow unmagicked data?", false); + + // -updateclient + psMetadata *updateclientArgs = psMetadataAlloc(); + psMetadataAddS64(updateclientArgs, PS_LIST_TAIL, "-client_id", 0, "search by client_id", 0); + psMetadataAddStr(updateclientArgs, PS_LIST_TAIL, "-stage", 0, "search by stage", NULL); + psMetadataAddStr(updateclientArgs, PS_LIST_TAIL, "-product", 0, "search by product", NULL); + psMetadataAddStr(updateclientArgs, PS_LIST_TAIL, "-comment", 0, "search by comment (LIKE)", NULL); + psMetadataAddBool(updateclientArgs, PS_LIST_TAIL, "-active", 0, "set to active state", NULL); + psMetadataAddBool(updateclientArgs, PS_LIST_TAIL, "-inactive", 0, "set to inactive state", NULL); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddS64(definerunArgs, PS_LIST_TAIL, "-client_id", 0, "search by client_id", 0); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "set and search by label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by data_group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-comment", 0, "search by comment", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by observation mode", NULL); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-rerun", 0, "Re-run publish?", false); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-pretend", 0, "Pretend to define?", false); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use simple output format?", false); + psMetadataAddU64(definerunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set", 0); + + // -pending + psMetadata *pendingArgs = psMetadataAlloc(); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-client_id", 0, "search on client_id", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-stage", 0, "search on source", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-comment", 0, "search on comment (LIKE)", NULL); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search on label", NULL); + psMetadataAddBool(pendingArgs, PS_LIST_TAIL, "-simple", 0, "use simple output format?", false); + psMetadataAddU64(pendingArgs, PS_LIST_TAIL, "-limit", 0, "limit result set", 0); + psMetadataAddStr(pendingArgs, PS_LIST_TAIL, "-name", 0, "search on client name", NULL); + + // -add + psMetadata *addArgs = psMetadataAlloc(); + psMetadataAddS64(addArgs, PS_LIST_TAIL, "-pub_id", 0, "define pub_id (required)", 0); + psMetadataAddStr(addArgs, PS_LIST_TAIL, "-path_base", 0, "define path_base (required)", NULL); + psMetadataAddStr(addArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + psMetadataAddF32(addArgs, PS_LIST_TAIL, "-dtime_script", 0, "define time for script", NAN); + psMetadataAddS32(addArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -revert + psMetadata *revertArgs = psMetadataAlloc(); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-pub_id", 0, "search on pub_id", 0); + psMetadataAddS32(revertArgs, PS_LIST_TAIL, "-fault", 0, "search on fault code", 0); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-client_id", 0, "search on client_id", NULL); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-comment", 0, "search on comment (LIKE)", NULL); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search on label", NULL); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "new value for state (required)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-pub_id", 0, "search on pub_id", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-client_id", 0, "search on client_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-fault", 0, "search by fault", NULL); + + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-defineclient", "", PUBTOOL_MODE_DEFINECLIENT, defineclientArgs); + PXOPT_ADD_MODE("-updateclient", "", PUBTOOL_MODE_UPDATECLIENT, updateclientArgs); + PXOPT_ADD_MODE("-definerun", "", PUBTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-pending", "", PUBTOOL_MODE_PENDING, pendingArgs); + PXOPT_ADD_MODE("-add", "", PUBTOOL_MODE_ADD, addArgs); + PXOPT_ADD_MODE("-revert", "", PUBTOOL_MODE_REVERT, revertArgs); + PXOPT_ADD_MODE("-updaterun", "", PUBTOOL_MODE_UPDATERUN, updaterunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define database handle, if used + // do this last so we don't setup a connection before CLI options are validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadd.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadd.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadd.c (revision 41104) @@ -0,0 +1,224 @@ +/* + * pxadd.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxadd.h" + +bool pxaddRunSetState(pxConfig *config, psS64 add_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, + "invalid stageRun state: %s", state); + return false; + } + + char *query = "UPDATE addRun SET state = '%s' WHERE add_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, add_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for add_id %" PRId64, add_id); + return false; + } + + return true; +} + + +bool pxaddRunSetStateByQuery(pxConfig *config, psMetadata *where, const char *stage, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, + "invalid stageRun state: %s", state); + return false; + } + psString query = NULL; + if (strcmp(stage, "cam") == 0) { + query = psStringCopy("UPDATE addRun JOIN camRun USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) SET addRun.state = '%s'"); + } + if (strcmp(stage, "stack") == 0) { + ///xxx this needs to be fixed + + /// psString query = psStringCopy("UPDATE addRun JOIN camRun USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) SET addRun.state = '%s'"); + } + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQueryF(config->dbh, query, state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(query); + + return true; +} + + +bool pxaddRunSetLabel(pxConfig *config, psS64 add_id, const char *label) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + // note label == NULL should be explicitly allowed + + char *query = "UPDATE addRun SET addRun.label = '%s' WHERE add_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, label, add_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for add_id %" PRId64, add_id); + return false; + } + + return true; +} + +bool pxaddRunSetLabelByQuery(pxConfig *config, psMetadata *where, const char *stage, const char *label) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + // note label == NULL should be explicitly allowed + //xxx fix for stack + psString query = psStringCopy("UPDATE addRun JOIN camRun USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) SET addRun.label = '%s'"); + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(query); + + return true; +} + +bool pxaddQueueByCamID(pxConfig *config, + char *stage, + psS64 stage_id, + psS32 stage_extra1, + char *workdir, + char *reduction, + char *label, + char *data_group, + char *dvodb, + char *note, + bool image_only, + bool minidvodb, + char *minidvodb_group, + char *minidvodb_name) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // load the SQL to enqueue our exp_ids from disk once + static psString query = NULL; + if (!query) { + if (strcmp( stage , "cam") == 0) { + query = pxDataGet("addtool_queue_cam_id.sql"); + psMemSetPersistent(query, true); + } + else if (strcmp(stage,"stack") == 0) { + query = pxDataGet("addtool_queue_stack_id.sql"); + psMemSetPersistent(query, true); + } + else if (strcmp(stage,"staticsky") == 0) { + query = pxDataGet("addtool_queue_sky_id_multi.sql"); + psMemSetPersistent(query, true); + } + else if (strcmp(stage,"skycal") == 0) { + query = pxDataGet("addtool_queue_skycal_id.sql"); + psMemSetPersistent(query, true); + } + else if (strcmp(stage,"diff") == 0) { + query = pxDataGet("addtool_queue_diff_id.sql"); + psMemSetPersistent(query, true); + } + else if (strcmp(stage,"fullforce") == 0) { + query = pxDataGet("addtool_queue_ff_id.sql"); + psMemSetPersistent(query, true); + } + else if (strcmp(stage,"fullforce_summary") == 0) { + query = pxDataGet("addtool_queue_ffsummary.sql"); + psMemSetPersistent(query,true); + } + + + else + { + psError(PXTOOLS_ERR_SYS, false, "uknown stage %s for pxadd.c", stage); + return false; + } + + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + } + + psTrace("addtool.c", PS_LOG_INFO, "pxadd query \n%s\n",query); + // queue the exp + // Note: cam_id is being cast here work around psS64 have a different type different + // on 32/64 + if (!p_psDBRunQueryF(config->dbh, query, + stage_extra1 ? stage_extra1 : 0, + "new", // state + workdir ? workdir : "NULL", + "dirty", //workdir_state + reduction? reduction : "NULL", + label ? label : "NULL", + data_group ? data_group : "NULL", + dvodb ? dvodb : "NULL", + note ? note : "NULL", + image_only, + minidvodb, + minidvodb_group, + minidvodb_name, + (long long) stage_id + )) { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + return false; + } + + // just to be safe, we should have changed at least one row + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, + "no rows affected - should have changed at least one row"); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadd.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadd.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadd.h (revision 41104) @@ -0,0 +1,47 @@ +/* + * pxadd.h + * + * Copyright (C) 2009 Joshua Hoblitt, Chris Waters + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXADD_H +#define PXADD_H 1 + +#include + +#include "pxtools.h" + +bool pxaddRunSetState(pxConfig *config, psS64 add_id, const char *state); +bool pxaddRunSetStateByQuery(pxConfig *config, psMetadata *where, const char *stage, const char *state); +bool pxaddRunSetLabel(pxConfig *config, psS64 add_id, const char *label); +bool pxaddRunSetLabelByQuery(pxConfig *config, psMetadata *where, const char *stage, const char *label); + +bool pxaddQueueByCamID(pxConfig *config, + char *stage, + psS64 stage_id, + psS32 stage_extra1, + char *workdir, + char *reduction, + char *label, + char *data_group, + char *dvodb, + char *note, + bool image_only, + bool minidvodb, + char *minidvodb_group, + char *minidvodb_name); + +#endif // PXADD_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadmin.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadmin.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadmin.c (revision 41104) @@ -0,0 +1,307 @@ +/* + * pxadmin.c + * + * Copyright (C) 2006-2009 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include "pxtools.h" +#include "pxadmin.h" + +bool createMode(pxConfig *config); +bool createMirrorMode(pxConfig *config); +bool deleteMode(pxConfig *config); +static bool insert_dbversion(pxConfig * config, const char *versionString); +static bool runMultipleStatments(pxConfig *config, const char *query); + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pxAdminConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + case PXADMIN_MODE_RECREATE: + if (!deleteMode(config)) { + goto FAIL; + } + // fall through + case PXADMIN_MODE_CREATE: + if (!createMode(config)) { + goto FAIL; + } + break; + case PXADMIN_MODE_CREATE_MIRROR: + if (!createMirrorMode(config)) { + goto FAIL; + } + break; + case PXADMIN_MODE_DELETE: + if (!deleteMode(config)) { + goto FAIL; + } + break; + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +bool createMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = pxDataGet("pxadmin_create_tables.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // BEGIN + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!runMultipleStatments(config, query)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + if (!insert_dbversion(config, IPPDB_VERSION)) { + psError(PS_ERR_UNKNOWN, false, "failed to set database version"); + return false; + } + + // COMMIT + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +bool createMirrorMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = pxDataGet("pxadmin_create_mirror_tables.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // BEGIN + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!runMultipleStatments(config, query)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + if (!insert_dbversion(config, IPPDB_VERSION)) { + psError(PS_ERR_UNKNOWN, false, "failed to set database version"); + return false; + } + + // COMMIT + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +bool deleteMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + { + char line[128], answer[128]; + + psMetadataItem *name = pmConfigUserSite(config->modules, "DBNAME", PS_DATA_STRING); + if (!name) { + psError(PS_ERR_UNKNOWN, false, "Unable to determine database name."); + return false; + } + psString dbName = name->data.str; + + fprintf(stdout, "*** delete the tables from database %s? ***\n", dbName); + fprintf(stdout, "*** to delete the tables, answer YES, and give password ***\n"); + fprintf(stdout, "*** WARNING: this action is permanent ***\n\n"); + + fprintf(stdout, "delete the tables (YES/[n]): "); + if (!fgets(line, 128, stdin)) { + psError(PS_ERR_IO, true, "Unable to read response."); + return false; + } + sscanf(line, "%s", answer); + if (strcmp (answer, "YES")) { + psError(PS_ERR_UNKNOWN, true, "tables NOT deleleted"); + return false; + } + + fprintf(stdout, "enter dbh connection password: "); + if (!fgets(line, 128, stdin)) { + psError(PS_ERR_IO, true, "Unable to read response."); + return false; + } + sscanf(line, "%s", answer); + + psMetadataItem *pass = pmConfigUserSite(config->modules, "DBPASSWORD", PS_DATA_STRING); + if (!pass) { + psError(PS_ERR_UNKNOWN, false, "Unable to determine database password."); + return false; + } + psString dbPassword = pass->data.str; + if (strcmp (answer, dbPassword)) { + psError(PS_ERR_UNKNOWN, true, "invalid passwd - tables NOT deleleted"); + return false; + } + } + + psString query = pxDataGet("pxadmin_drop_tables.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // BEGIN + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!runMultipleStatments(config, query)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + // COMMIT + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool runMultipleStatments(pxConfig *config, const char *query) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(query, false); + + // loop over all statements in string + psList *statements = psStringSplit(query, ";", false); + + psString q = NULL; + psListIterator *iter = psListIteratorAlloc(statements, PS_LIST_HEAD, false); + while ((q = psListGetAndIncrement(iter))) { + if (!p_psDBRunQuery(config->dbh, q)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(iter); + psFree(statements); + return false; + } + } + psFree(iter); + psFree(statements); + + return true; +} + +#ifdef notdef +static bool update_dbversion(pxConfig * config, const char *versionString) +{ + psString query = pxDataGet("pxadmin_update_version.sql"); + if (!query) { + psError(PS_ERR_UNKNOWN, false, "failed to retrieve SQL statement"); + psFree(query); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, versionString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + return true; +} +#endif +static bool insert_dbversion(pxConfig * config, const char *versionString) +{ + psString query = "INSERT INTO dbversion VALUES('%s', CURRENT_TIMESTAMP())"; + if (!query) { + psError(PS_ERR_UNKNOWN, false, "failed to retrieve SQL statement"); + psFree(query); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, versionString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadmin.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadmin.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadmin.h (revision 41104) @@ -0,0 +1,35 @@ +/* + * pxadmin.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXADMIN_H +#define PXADMIN_H 1 + +#include "pxtools.h" + +typedef enum { + PXADMIN_MODE_NONE = 0x0, + PXADMIN_MODE_CREATE, + PXADMIN_MODE_CREATE_MIRROR, + PXADMIN_MODE_DELETE, + PXADMIN_MODE_RECREATE +} pxAdminMode; + +pxConfig *pxAdminConfig (pxConfig *config, int argc, char **argv); + +#endif // PXADMIN_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadminConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadminConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxadminConfig.c (revision 41104) @@ -0,0 +1,162 @@ +/* + * pxadminConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxadmin.h" + +static void pxAdminShowDB (const pxConfig *config, const char *program) +{ + fprintf (stderr, "\nPan-STARRS DataBase Admin Tool\n\n"); + fprintf (stderr, "Usage: %s [mode]\n", program); + fprintf (stderr, " [mode] : -create | -create-mirror | -delete\n\n"); + + psMetadataItem *server = pmConfigUserSite(config->modules, "DBSERVER", PS_DATA_STRING); + psMetadataItem *user = pmConfigUserSite(config->modules, "DBUSER", PS_DATA_STRING); + psMetadataItem *name = pmConfigUserSite(config->modules, "DBNAME", PS_DATA_STRING); + psMetadataItem *port = pmConfigUserSite(config->modules, "DBPORT", PS_TYPE_S32); + + if (!server || !user || !name) { + psErrorClear(); + psWarning("Unable to determine database connection details."); + return; + } + if (!port) { + psErrorClear(); + } + + fprintf (stderr, "connecting to %s as %s (port %d)\n", server->data.str, user->data.str, + port ? port->data.S32 : 0); + fprintf (stderr, "using database %s\n\n", name->data.str); + + return; +} + +pxConfig *pxAdminConfig(pxConfig *config, int argc, char **argv) +{ + if (!psArgumentGet(argc, argv, "-dbname")) { + psError(PS_ERR_UNEXPECTED_NULL, true, + "Command-line argument '-dbname XXX' is required, for database safety"); + return NULL; + } + + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // Parse other command-line arguments + psMetadata *arguments = psMetadataAlloc(); // The arguments, with default values + + int N; + config->mode = PXADMIN_MODE_NONE; + if ((N = psArgumentGet(argc, argv, "-create"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort("only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_CREATE; + } + if ((N = psArgumentGet(argc, argv, "-create-mirror"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort("only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_CREATE_MIRROR; + } + if ((N = psArgumentGet(argc, argv, "-delete"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort("only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_DELETE; + } + if ((N = psArgumentGet(argc, argv, "-recreate"))) { + psArgumentRemove(N, &argc, argv); + if (config->mode) { + psAbort("only one mode selection is allowed"); + } + config->mode = PXADMIN_MODE_RECREATE; + } + + // paul's argument parsing convention requires: -key value + psMetadataAddStr(arguments, PS_LIST_TAIL, "-create", 0, + "create all IPP tables", ""); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-create-mirror", 0, + "mirror all IPP tables", ""); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-delete", 0, + "delete all IPP tables", ""); + psMetadataAddStr(arguments, PS_LIST_TAIL, "-recreate", 0, + "delete and recreate all tables", ""); + + if (config->mode == PXADMIN_MODE_NONE) { + pxAdminShowDB (config, argv[0]); + fprintf (stderr, "admin mode not specified\n"); + + psArgumentHelp(arguments); + psFree(arguments); + psFree(config); + return NULL; + } + + if ((N = psArgumentGet (argc, argv, "-help"))) { + pxAdminShowDB (config, argv[0]); + psArgumentHelp(arguments); + psFree(arguments); + psFree(config); + return NULL; + } + + if (! psArgumentParse(arguments, &argc, argv) || argc != 1) { + pxAdminShowDB (config, argv[0]); + psArgumentHelp(arguments); + psFree(arguments); + psFree(config); + return NULL; + } + + psFree(arguments); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxcam.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxcam.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxcam.c (revision 41104) @@ -0,0 +1,213 @@ +/* + * pxcam.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxcam.h" + +bool pxcamSetSearchArgs (psMetadata *md) { + + psMetadataAddS64(md, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddS64(md, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(md, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(md, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-comment", 0, "search by comment", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF64(md, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_min", 0, "define min RA (degrees) ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_max", 0, "define max RA (degrees) ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl_min", 0, "define min DEC (degrees)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl_max", 0, "define max DEC (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_min", 0, "define min background", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_max", 0, "define max background", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_stdev_min", 0, "define min background standard deviation", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_stdev_max", 0, "define max background standard deviation", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "define min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "define max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-alt_min", 0, "define min altitude", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-alt_max", 0, "define max altitude", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-az_min", 0, "define min azimuth ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-az_max", 0, "define max azimuth ", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddStr(md, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddF32(md, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + + return true; +} + +bool pxcamGetSearchArgs (pxConfig *config, psMetadata *where) { + + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "=="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + + return true; +} + +bool pxcamRunSetState(pxConfig *config, psS64 cam_id, const char *state, psS64 magicked) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, + "invalid camRun state: %s", state); + return false; + } + + char *query = "UPDATE camRun SET state = '%s', magicked = %" PRId64 " WHERE cam_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, magicked, cam_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for cam_id %" PRId64, cam_id); + return false; + } + + return true; +} + +bool pxcamQueueByChipID(pxConfig *config, + psS64 chip_id, + char *workdir, + char *label, + char *data_group, + char *dist_group, + char *recipe, + char *expgroup, + char *dvodb, + char *tess_id, + char *end_stage, + psS64 magicked, + char *note) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // load the SQL to enqueue our exp_ids from disk once + static psString query = NULL; + if (!query) { + query = pxDataGet("camtool_queue_chip_id.sql"); + psMemSetPersistent(query, true); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + } + + // queue the exp + // XXX chip_id and magicked are being cast here work around psS64 have a different type + // different on 32/64 + if (!p_psDBRunQueryF(config->dbh, query, + "new", // state + workdir ? workdir : "NULL", + "dirty", //workdir_state + label ? label : "NULL", + data_group ? data_group : "NULL", + dist_group ? dist_group : "NULL", + recipe ? recipe : "NULL", + expgroup ? expgroup : "NULL", + dvodb ? dvodb : "NULL", + tess_id ? tess_id : "NULL", + end_stage ? end_stage : "NULL", + (long long) magicked, + // Here follows the software version and mask statistics which shouldn't be propagated + "NULL", + "0", + "0", + "0", + "0", + "0", + "0", + "0", + "0", + "0", + "0", + note ? note : "NULL", + (long long)chip_id + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // just to be safe, we should have changed at least one row + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, + "no rows affected - should have changed at least one row"); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxcam.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxcam.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxcam.h (revision 41104) @@ -0,0 +1,45 @@ +/* + * pxcam.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXCAM_H +#define PXCAM_H 1 + +#include + +#include "pxtools.h" + +bool pxcamRunSetState(pxConfig *config, psS64 cam_id, const char *state, psS64 magicked); +bool pxcamSetSearchArgs (psMetadata *md); +bool pxcamGetSearchArgs (pxConfig *config, psMetadata *where); + +bool pxcamQueueByChipID(pxConfig *config, + psS64 chip_id, + char *workdir, + char *label, + char *data_group, + char *dist_group, + char *recipe, + char *expgroup, + char *dvodb, + char *tess_id, + char *end_stage, + psS64 magicked, + char *note); + +#endif // PXCAM_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxchip.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxchip.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxchip.c (revision 41104) @@ -0,0 +1,323 @@ +/* + * pxchip.c + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxchip.h" + +bool pxchipSetSearchArgs (psMetadata *md) { + + // psMetadataAddStr(md, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + // psMetadataAddStr(md, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + + // XXX need to allow multiple exp_ids + psMetadataAddS64(md, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(md, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(md, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-data_state", 0, "search by data_state", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(md, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sun_angle_min", 0, "search by min solar angle", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sun_angle_max", 0, "search by max solar angle", NAN); + psMetadataAddStr(md, PS_LIST_TAIL, "-object", 0, "search by exposure object (LIKE comparison)", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-comment", 0, "search by comment field (LIKE comparison)", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-comment-skip", 0, "exclude by comment field (NOT LIKE comparison)", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-obs_mode", 0, "search by observation mode", NULL); + return true; +} + +// XXX explicit refs to rawExp and chipRun? +// XXX careful with entries that could match rawExp rawImfile, chipRun, or chipProcessedImfile (eg, bg) +bool pxchipGetSearchArgs (pxConfig *config, psMetadata *where) { + + // definebyquery : rawExp only + // updaterun : rawExp, chipRun + // pendingimfile : rawExp, chipRun + // processedimfile : rawExp, chipRun, chipProcessedImfile + // revertprocessedimfile : rawExp, chipProcessedImfile + // updateprocessedimfile : chipProcessedImfile + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-data_state", "rawExp.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-comment-skip", "rawExp.comment", "NOTLKE"); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "rawExp.obs_mode", "LIKE"); + return true; +} + +bool pxchipRunSetState(pxConfig *config, psS64 chip_id, const char *state, const psS64 magicked) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid chipRun state: %s", state); + return false; + } + + char *query = "UPDATE chipRun SET state = '%s', magicked = %" PRId64 " WHERE chip_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, magicked, chip_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for chip_id %" PRId64, chip_id); + return false; + } + + return true; +} + + +bool pxchipRunSetStateByQuery(pxConfig *config, psMetadata *where, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid chipRun state: %s", state); + return false; + } + + if (!strcmp(state, "full")) { + // There are states that need to be met for a run to be set to full that we don't + // check here. + // for example all of the run's Imfiles must have chipProcessedImfile.data_state == "full" + // chipRun.magicked = (SUM(!chipProcessedImfile.magicked) = 0) + // so don't do allow setting the state to full + psError(PS_ERR_UNKNOWN, true, "cannot use -updaterun so set chipRun state to full"); + return false; + } + + + psString query = psStringCopy("UPDATE chipRun JOIN rawExp USING(exp_id) SET state = '%s'"); + + if (!strcmp(state, "cleaned") || !strcmp(state, "purged")) { + // if magicked is non-zero set it to -1 + psStringAppend(&query, ", chipRun.magicked = IF(chipRun.magicked = 0, 0, -1)"); + } + + if (where && psListLength(where->list) > 0) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQueryF(config->dbh, query, state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(query); + + return true; +} + +bool pxchipProcessedImfileSetStateByQuery(pxConfig *config, psMetadata *where, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid chipProcessedImfile state: %s", state); + return false; + } + +/* if (!strcmp(state, "full")) { */ +/* // There are states that need to be met for a run to be set to full that we don't */ +/* // check here. */ +/* // for example all of the run's Imfiles must have chipProcessedImfile.data_state == "full" */ +/* // chipRun.magicked = (SUM(!chipProcessedImfile.magicked) = 0) */ +/* // so don't do allow setting the state to full */ +/* psError(PS_ERR_UNKNOWN, true, "cannot use -updaterun so set chipRun state to full"); */ +/* return false; */ +/* } */ + + psString query = psStringCopy("UPDATE chipProcessedImfile JOIN chipRun USING(chip_id) JOIN rawExp ON chipRun.exp_id = rawExp.exp_id SET data_state = '%s'"); + + if (where && psListLength(where->list) > 0) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } else { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, state)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(query); + + return true; +} + +psS64 pxchipQueueByExpTag(pxConfig *config, + psS64 exp_id, + const char *workdir, + const char *label, + const char *data_group, + const char *dist_group, + const char *reduction, + const char *expgroup, + const char *dvodb, + const char *tess_id, + const char *end_stage, + const char *note) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + // create a chipRun + if (!chipRunInsert(config->dbh, + 0x0, // chip_id + exp_id, + "new", // state + workdir, + "dirty", // workdir_state + label, + data_group, + dist_group, + reduction, + expgroup, + dvodb, + tess_id, + end_stage, + 0, // magicked + NULL, + 0, + NAN, + NAN, + NAN, + NAN, + 0, // update_mode + note + ) + ) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error failed to rollback transaction"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return 0; + } + + psS64 chip_id = psDBLastInsertID(config->dbh); + + // Create rows in chipImfile table for each input exposure. + // This creates the chip_image_id values + psString query = "INSERT INTO chipImfile " + "SELECT %" PRId64 ", class_id, 0 FROM rawImfile WHERE exp_id = %" PRId64; + + if (!p_psDBRunQueryF(config->dbh, query, chip_id, exp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return 0; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return 0; + } + + return chip_id; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxchip.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxchip.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxchip.h (revision 41104) @@ -0,0 +1,49 @@ +/* + * pxchip.h + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXCHIP_H +#define PXCHIP_H 1 + +#include + +#include "pxtools.h" + + + +bool pxchipRunSetState(pxConfig *config, psS64 chip_id, const char *state, const psS64 magicked); +bool pxchipProcessedImfileSetStateByQuery(pxConfig *config, psMetadata *where, const char *state); + +psS64 pxchipQueueByExpTag(pxConfig *config, + psS64 exp_id, + const char *workdir, + const char *label, + const char *data_group, + const char *dist_group, + const char *reduction, + const char *expgroup, + const char *dvodb, + const char *tess_id, + const char *end_stage, + const char *note); + + +bool pxchipSetSearchArgs (psMetadata *md); +bool pxchipGetSearchArgs (pxConfig *config, psMetadata *where); + +#endif // PXCHIP_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxconfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxconfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxconfig.c (revision 41104) @@ -0,0 +1,227 @@ +/* + * pxconfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include +#include + +#include "pxtools.h" + +static void pxConfigFree(pxConfig *ptr); + +pxConfig *pxConfigAlloc(void) +{ + pxConfig *config; + + config = psAlloc(sizeof(pxConfig)); + psMemSetDeallocator(config, (psFreeFunc)pxConfigFree); + + config->modeName = NULL; + config->mode = 0; + config->dbh = NULL; + config->modules = NULL; + // XXX config->where = NULL; + config->args = NULL; + + return config; +} + +static void pxConfigFree(pxConfig *config) +{ + psFree(config->modeName); + psFree(config->dbh); + psFree(config->modules); + // XXX psFree(config->where); + psFree(config->args); +} + +void pxUsage(FILE *stream, int argc, char **argv, const char *modeName, psMetadata *argSet) +{ + fprintf(stream, "Usage: %s %s []\n\n", argv[0], modeName); + fprintf(stream, "%s:\n", modeName); + + psArgumentHelpSimple(stream, argSet); +} + +bool pxGetOptions(FILE *stream, int argc, char **argv, pxConfig *config, psMetadata *modes, psMetadata *argSets) +{ + // figure out what mode we're running in + psMetadataIterator *iter = psMetadataIteratorAlloc(modes, PS_LIST_HEAD, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + int N = 0; + if ((N = psArgumentGet(argc, argv, item->name))) { + psArgumentRemove(N, &argc, argv); + // check for duplicate mode specification + if (config->mode) { + psError(PS_ERR_UNKNOWN, true, "only one mode selection is allowed"); + fprintf(stream, "only one mode selection is allowed\n"); + pxUsage(stream, argc, argv, "", modes); + psFree(iter); + return NULL; + } + + config->mode = item->data.U32; + config->modeName = psStringCopy(item->name); + + bool status = false; + psMetadata *argset = psMetadataLookupMetadata(&status, argSets, item->name); + // make sure we can find the argSet for the sepcified mode + if (!status) { + psError(PS_ERR_UNKNOWN, true, "can not find arguments for mode"); + fprintf(stderr, "can not find arugments for mode"); + psFree(iter); + return NULL; + } + + config->args = psMemIncrRefCounter(argset); + } + + } + + psFree(iter); + + // make sure we found a mode + if (config->mode == PXTOOL_MODE_NONE) { + psError(PS_ERR_UNKNOWN, true, "mode argument is required"); + fprintf(stderr, "mode argument is required\n"); + pxUsage(stream, argc, argv, "", modes); + return NULL; + } + + // actually parse the command line + if (!psArgumentParse(config->args, &argc, argv)) { + psError(PS_ERR_UNKNOWN, false, "error parsing arguments"); + fprintf(stderr, "error parsing arguments\n"); + pxUsage(stream, argc, argv, config->modeName, config->args); + return NULL; + } + + // look for left overs on the command line + if (argc != 1) { + psError(PS_ERR_UNKNOWN, false, "extra arguments: "); + fprintf(stderr, "extra arguments: "); + for (int i = 1; i < argc; i++) { + fprintf (stderr, "%s ", argv[i]); + } + fprintf(stderr, "\n"); + pxUsage(stream, argc, argv, config->modeName, config->args); + return NULL; + } + + // make sure that all required parameters have been specified + iter = psMetadataIteratorAlloc(config->args, PS_LIST_HEAD, NULL); + item = NULL; // Item from iterator + while ((item = psMetadataGetAndIncrement(iter))) { + if (strstr(item->comment, "require") == NULL) { + continue; + } + if (strstr(item->comment, "(found)") != NULL) { + continue; + } + + switch (item->type) { + case PS_DATA_BOOL: + psError(PS_ERR_UNKNOWN, false, "boolean type can not be required"); + fprintf(stderr, "boolean type can not be required\n"); + psFree(iter); + return NULL; + break; + case PS_DATA_S8: + if (item->data.S8 != INT8_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_S16: + if (item->data.S16 != INT16_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_S32: + if (item->data.S32 != INT32_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_S64: + if (item->data.S64 != INT64_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_U8: + if (item->data.U8 != UINT8_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_U16: + if (item->data.U16 != UINT16_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_U32: + if (item->data.U32 != UINT32_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_U64: + if (item->data.U64 != UINT64_MAX) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_F32: + case PS_DATA_F64: + if (!isnan(item->data.F64)) { + break; + } + goto ARG_REQUIRED; + case PS_DATA_STRING: + case PS_DATA_TIME: + case PS_DATA_METADATA: + if (item->data.V) { + break; + } + goto ARG_REQUIRED; + default: + psError(PS_ERR_UNKNOWN, false, "unknown argument type"); + fprintf(stream, "unknown argument type\n"); + psFree(iter); + return NULL; + ARG_REQUIRED: + psError(PS_ERR_UNKNOWN, false, "argument %s is required", item->name); + fprintf(stream, "argument %s is required\n", item->name); + pxUsage(stream, argc, argv, config->modeName, config->args); + psFree(iter); + return NULL; + } + + } + + psFree(iter); + + // save argv/argc + config->argv = argv; + config->argc = argc; + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxconfig.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxconfig.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxconfig.h (revision 41104) @@ -0,0 +1,39 @@ +/* + * pxconfig.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXCONFIG_H +#define PXCONFIG_H 1 + +#include "pslib.h" +#include "psmodules.h" + +typedef struct { + char *modeName; + int mode; + pmConfig *modules; + psDB *dbh; + psMetadata *args; + // XXX deprecate psMetadata *where; + int argc; + char **argv; +} pxConfig; + +pxConfig *pxConfigAlloc(void); + +#endif // PXCONFIG_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdata.c.template =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdata.c.template (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdata.c.template (revision 41104) @@ -0,0 +1,145 @@ +/* + * pxdata.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include // getenv +#include +#include +#include + +#include + +#include "pxtools.h" +#include "pxdata.h" + +static psHash *cachedData = NULL; + +static int test_f(const char *path); + +psString pxDataGet(const char *filename) +{ + PS_ASSERT_PTR_NON_NULL(filename, NULL); + + { + psString filepath = NULL; + + // look to see if PXDATA is set + char *PXDATA = getenv("PXDATA"); + if (PXDATA) { + // if it is, then prepend filename to PXDATA + psStringAppend(&filepath, "%s/%s", PXDATA, filename); + } + + // see if we have a valid filename + if (test_f(filepath)) { + // if we do, slurp the file + psString data = psSlurpFilename(filepath); + if (!data) { + psError(PS_ERR_IO, false, "failed to slurp %s", filepath); + psFree(filepath); + return NULL; + } + psTrace("pxdata", PS_LOG_INFO, "slupred %s", filepath); + psFree(filepath); + + return data; + } + + psFree(filepath); + } + + { + psString filepath = NULL; + + // look under our share path + psStringAppend(&filepath, "%s/%s", "@PKGDATADIR@", filename); + + // see if we have a valid filename + if (test_f(filepath)) { + // if we do, slurp the file + psString data = psSlurpFilename(filepath); + if (!data) { + psError(PS_ERR_IO, false, "failed to slurp %s", filepath); + psFree(filepath); + return NULL; + } + psTrace("pxdata", PS_LOG_INFO, "slupred %s", filepath); + psFree(filepath); + + return data; + } + + psFree(filepath); + } + + // couldn't find a matching filename + psError(PS_ERR_IO, true, "can't find a file in search path(s) to match : %s", filename); + + return NULL; +} + +static int test_f(const char *path) +{ + // copied from coreutils 6.4 test.c licenced under GPL v2 + struct stat stat_buf; + return (stat (path, &stat_buf) == 0 + && S_ISREG (stat_buf.st_mode)); +} + +psString pxDataGetCached(const char *filename) +{ + bool persistence = p_psMemAllocatePersistent(true); + + // make sure the cache is initalized + if (!cachedData) { + cachedData = psHashAlloc(10); + } + + // look to see if we've cached this file + { + psString data = psHashLookup(cachedData, filename); + if (data) { + p_psMemAllocatePersistent(persistence); + return psStringCopy(data); + } + } + + // we didn't have a cached copy so we need to try to read the file + psString data = pxDataGet(filename); + if (!data) { + p_psMemAllocatePersistent(persistence); + psError(PS_ERR_IO, false, "pxDataGet(%s) failed", filename); + return NULL; + } + + // store a copy of the file + if (!psHashAdd(cachedData, filename, data)) { + psFree(data); + p_psMemAllocatePersistent(persistence); + psError(PS_ERR_UNKNOWN, false, "psHashAdd() failed"); + return NULL; + } + + p_psMemAllocatePersistent(persistence); + + return psStringCopy(data); +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdata.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdata.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdata.h (revision 41104) @@ -0,0 +1,28 @@ +/* + * pxdata.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXDATA_H +#define PXDATA_H 1 + +#include + +psString pxDataGet(const char *filename); +psString pxDataGetCached(const char *filename); + +#endif // PXDATA_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdqstats.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdqstats.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdqstats.c (revision 41104) @@ -0,0 +1,103 @@ +/* pxdqstats.c + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxdqstats.h" + +bool pxdqstatsSetSearchArgs (psMetadata *md) { + + psMetadataAddS64(md, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(md, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(md, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(md, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-comment", 0, "search by comment", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(md, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF64(md, PS_LIST_TAIL, "-airmass_min", 0, "define min airmass", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-airmass_max", 0, "define max airmass", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_min", 0, "define min RA (degrees) ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_max", 0, "define max RA (degrees) ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl_min", 0, "define min DEC (degrees)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl_max", 0, "define max DEC (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_min", 0, "define min background", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_max", 0, "define max background", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_stdev_min", 0, "define min background standard deviation", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_stdev_max", 0, "define max background standard deviation", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "define min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "define max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-alt_min", 0, "define min altitude", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-alt_max", 0, "define max altitude", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-az_min", 0, "define min azimuth ", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-az_max", 0, "define max azimuth ", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddStr(md, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddF32(md, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + + return true; +} + +bool pxdqstatsGetSearchArgs (pxConfig *config, psMetadata *where) { + + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "=="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdqstats.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdqstats.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxdqstats.h (revision 41104) @@ -0,0 +1,15 @@ +/* pxdqstats.h + */ + +#ifndef PXDQSTATS_H +#define PXDQSTATS_H 1 + +#include + +#include "pxtools.h" + +bool pxdqstatsSetSearchArgs (psMetadata *md); +bool pxdqstatsGetSearchArgs (pxConfig *config, psMetadata *where); + + +#endif // PXDQSTATS_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxerrors.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxerrors.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxerrors.c (revision 41104) @@ -0,0 +1,43 @@ +/* + * pxerrors.c + * + * Copyright (C) 2006 Eugene Magnier + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include "pxtools.h" + +psExit pxerrorGetExitStatus(void) { + + // psErrorCodeLast returns a psErrorCode, but we are being a bit sleezy and defining a set + // of enum values that have non-overlapping-ranges (not enforced by the compiler) + + int err = psErrorCodeLast (); + switch (err) { + case PXTOOLS_ERR_SYS: + return PS_EXIT_SYS_ERROR; + case PXTOOLS_ERR_CONFIG: + return PS_EXIT_CONFIG_ERROR; + case PXTOOLS_ERR_PROG: + return PS_EXIT_PROG_ERROR; + default: + return PS_EXIT_UNKNOWN_ERROR; + } + return PS_EXIT_UNKNOWN_ERROR; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfake.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfake.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfake.c (revision 41104) @@ -0,0 +1,110 @@ +/* + * pxfake.c + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxcam.h" + +bool pxfakeRunSetState(pxConfig *config, psS64 fake_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, + "invalid fakeRun state: %s", state); + return false; + } + + char *query = "UPDATE fakeRun SET state = '%s' WHERE fake_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, fake_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for fake_id %" PRId64, fake_id); + return false; + } + + return true; +} + +psS64 pxfakeQueueByCamID(pxConfig *config, + psS64 cam_id, + char *workdir, + char *label, + char *data_group, + char *dist_group, + char *reduction, + char *expgroup, + char *dvodb, + char *tess_id, + char *end_stage, + char *note) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psTrace("pxtool", PS_LOG_INFO, "attemping to queue cam_id: %"PRId64, cam_id); + + // load the SQL to enqueue our exp_ids from disk once + static psString query = NULL; + if (!query) { + query = pxDataGet("faketool_queue_cam_id.sql"); + psMemSetPersistent(query, true); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + } + + // queue the exp + // XXX chip_id is being cast here work around psS64 have a different type + // different on 32/64 + if (!p_psDBRunQueryF(config->dbh, query, + "new", // state + workdir ? workdir : "NULL", + label ? label : "NULL", + data_group ? data_group : "NULL", + dist_group ? dist_group : "NULL", + reduction? reduction: "NULL", + expgroup ? expgroup : "NULL", + dvodb ? dvodb : "NULL", + tess_id ? tess_id : "NULL", + end_stage ? end_stage : "NULL", + note ? note : "NULL", + (long long)cam_id + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // just to be safe, we should have changed at least one row + if (psDBAffectedRows(config->dbh) != 1) { + psError(PS_ERR_UNKNOWN, false, + "should have changed just one row"); + return false; + } + + return psDBLastInsertID(config->dbh); +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfake.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfake.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfake.h (revision 41104) @@ -0,0 +1,43 @@ +/* + * pxfake.h + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXFAKE_H +#define PXFAKE_H 1 + +#include + +#include "pxtools.h" + +bool pxfakeRunSetState(pxConfig *config, psS64 fake_id, const char *state); + +psS64 pxfakeQueueByCamID(pxConfig *config, + psS64 cam_id, + char *workdir, + char *label, + char *data_group, + char *dist_group, + char *reduction, + char *expgroup, + char *dvodb, + char *tess_id, + char *end_stage, + char *note); + + +#endif // PXFAKE_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfault.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfault.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxfault.c (revision 41104) @@ -0,0 +1,72 @@ +/* + * pxfault.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" + +bool pxSetFaultCode(psDB *dbh, const char *tableName, psMetadata *where, psS16 code, psS16 quality) +{ + PS_ASSERT_PTR_NON_NULL(dbh, false); + PS_ASSERT_PTR_NON_NULL(tableName, false); + PS_ASSERT_PTR_NON_NULL(where, false); + +#if 0 + // map code string to numeric fault code + psU32 code = mapCodeStrToInt(codeStr); + if (code == (psU32)-1) { + psError(PS_ERR_UNKNOWN, false, "error resolving error code"); + return false; + } +#endif + + // update the database + psMetadata *values = psMetadataAlloc(); + if (!psMetadataAddS16(values, PS_LIST_HEAD, "fault", 0, NULL, code)) { + psError(PS_ERR_UNKNOWN, false, "failed to add metadata item fault"); + psFree(values); + return false; + } + if (quality) { + if (!psMetadataAddS16(values, PS_LIST_HEAD, "quality", 0, NULL, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to add metadata item quality"); + psFree(values); + return false; + } + } + + long rowsAffected = psDBUpdateRows(dbh, tableName, where, values); + psFree(values); + if (rowsAffected < 0) { + // database error + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (rowsAffected < 1) { + // we didn't do anything + psError(PS_ERR_UNKNOWN, false, "zero rows were affected - either the search criteria didn't match any rows or the field already had the value being set."); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinject.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinject.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinject.c (revision 41104) @@ -0,0 +1,166 @@ +/* + * pxinject.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxinject.h" + +static bool newExpMode(pxConfig *config); +static bool newImfileMode(pxConfig *config); +static bool updatenewExpMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pxinjectConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(PXINJECT_MODE_NEWEXP, newExpMode); + MODECASE(PXINJECT_MODE_NEWIMFILE, newImfileMode); + MODECASE(PXINJECT_MODE_UPDATENEWEXP, updatenewExpMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint (stderr, "failure\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool newExpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(tmp_exp_name, config->args, "-tmp_exp_name", true, false); + PXOPT_LOOKUP_STR(tmp_camera, config->args, "-tmp_inst", true, false); + PXOPT_LOOKUP_STR(tmp_telescope, config->args, "-tmp_telescope", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-end_stage", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!newExpInsert(config->dbh, + 0, // exp_id + 0, // summit_id + tmp_exp_name, + tmp_camera, + tmp_telescope, + "reg", // state + workdir, + "dirty", + reduction, + dvodb, + tess_id, + end_stage, + label, + NULL // epoch + ) + ) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psS64 exp_id = psDBLastInsertID(config->dbh); + + psMetadata *md = psMetadataAlloc(); + if (!psMetadataAddS64(md, PS_LIST_TAIL, "exp_id", 0, NULL, exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(md); + } + + // negate simple so the default is true + if (!ippdbPrintMetadata(stdout, md, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(md); + return false; + } + + psFree(md); + + return true; +} + +static bool newImfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(tmp_class_id, config->args, "-tmp_class_id", true, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + PXOPT_LOOKUP_S32(bytes, config->args, "-bytes", false, false); + PXOPT_LOOKUP_STR(md5sum, config->args, "-md5sum", false, false); + + // insert with error flag state set to 0 (no errors) + if (!newImfileInsert(config->dbh, exp_id, tmp_class_id, uri, NULL, bytes, md5sum)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool updatenewExpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + + if (state) { + // set detRun.state to state + return pxnewExpSetState(config, exp_id, state); + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinject.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinject.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinject.h (revision 41104) @@ -0,0 +1,34 @@ +/* + * pxinject.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXINJECT_H +#define PXINJECT_H 1 + +#include "pxtools.h" + +typedef enum { + PXINJECT_MODE_NONE = 0x0, + PXINJECT_MODE_NEWEXP, + PXINJECT_MODE_NEWIMFILE, + PXINJECT_MODE_UPDATENEWEXP +} pxinjectMode; + +pxConfig *pxinjectConfig(pxConfig *config, int argc, char **argv); + +#endif // PXINJECT_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinjectConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinjectConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxinjectConfig.c (revision 41104) @@ -0,0 +1,100 @@ +/* + * pxinjectConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxinject.h" + +pxConfig *pxinjectConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -newExp + psMetadata *newExpArgs = psMetadataAlloc(); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-tmp_exp_name", 0, "define the exp_name (required)", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-tmp_inst", 0, "define the camera name (required)", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-tmp_telescope", 0, "define the telescope name (required)", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", 0); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-reduction", 0, "define reduction class", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-dvodb", 0, "define the dvodb for the next processing step", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-tess_id", 0, "define the tess_id for the next processing step", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-end_stage", 0, "define the end goal processing step", NULL); + psMetadataAddStr(newExpArgs, PS_LIST_TAIL, "-label", 0, "define a label (carried to chip stage)", NULL); + psMetadataAddBool(newExpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -newImfile + psMetadata *newImfileArgs = psMetadataAlloc(); + psMetadataAddS64(newImfileArgs, PS_LIST_TAIL, "-exp_id", 0, "define the exp_id (required)", 0); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-tmp_class_id", 0, "define the class ID (required)", NULL); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-uri", 0, "define the URI (required)", NULL); + psMetadataAddS32(newImfileArgs, PS_LIST_TAIL, "-bytes", 0, "define the size of the file", 0); + psMetadataAddStr(newImfileArgs, PS_LIST_TAIL, "-md5sum", 0, "define the size of the file", NULL); + + // -updatenewExp + psMetadata *updatenewExpArgs = psMetadataAlloc(); + psMetadataAddS64(updatenewExpArgs, PS_LIST_TAIL, "-exp_id", 0, "define the exp_id (required)", 0); + psMetadataAddStr(updatenewExpArgs, PS_LIST_TAIL, "-state", 0, "set state (required)", NULL); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-newExp", "", PXINJECT_MODE_NEWEXP, newExpArgs); + PXOPT_ADD_MODE("-newImfile", "", PXINJECT_MODE_NEWIMFILE, newImfileArgs); + PXOPT_ADD_MODE("-updatenewExp", "", PXINJECT_MODE_UPDATENEWEXP, updatenewExpArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmagic.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmagic.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmagic.c (revision 41104) @@ -0,0 +1,98 @@ +/* + * pxmagic.c + * + * Copyright (C) 2009 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxmagic.h" + +bool pxmagicRestoreStage(pxConfig *config, psString stage, psString whereClause, psString newState) +{ + psString queryFile = NULL; + psStringAppend(&queryFile, "magictool_restore_%s.sql", stage); + psString query_temp = pxDataGet(queryFile); + if (!query_temp) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement from %s", queryFile); + psFree(queryFile); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(queryFile); + + // psStringSubstittute fails on strings created by pxDataGet() + psString query = psStringCopy(query_temp); + // change the @NEW_STATE@ in the sql file to our state + if (!psStringSubstitute(&query, newState, "@NEW_STATE@")) { + psError(PS_ERR_UNKNOWN, false, "failed to substitute state string"); + return false; + } + + psStringAppend(&query, " AND %s", whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + return true; +} + +bool pxmagicAddWhere(pxConfig *config, psString *out, psString table) +{ + PXOPT_LOOKUP_U64(magicked, config->args, "-magicked", false, false); + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + PXOPT_LOOKUP_BOOL(not_destreaked, config->args, "-not_destreaked", false); + + if (not_destreaked) { + if (destreaked) { + psError(PXTOOLS_ERR_CONFIG, true, "providing -not_destreaked and -destreaked makes no sense"); + return false; + } + if (magicked) { + psError(PXTOOLS_ERR_CONFIG, true, "providing -not_destreaked and -magicked makes no sense"); + return false; + } + psStringAppend(out, " AND %s.magicked = 0", table); + } + if (destreaked) { + psStringAppend(out, " AND %s.magicked != 0", table); + } + // Note -magicked is handled by the caller. XXX: Why? + return true; +} + +void pxmagicAddArguments(psMetadata *md) +{ + psMetadataAddBool(md, PS_LIST_TAIL, "-destreaked", 0, "search for destreaked images", false); + psMetadataAddBool(md, PS_LIST_TAIL, "-not_destreaked", 0, "search for images that have not been destreaked", false); + psMetadataAddS64(md, PS_LIST_TAIL, "-magicked", 0, "search by magicked value", 0); +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmagic.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmagic.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmagic.h (revision 41104) @@ -0,0 +1,32 @@ +/* + * pxmagic.h + * + * Copyright (C) 2009 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXMAGIC_H +#define PXMAGIC_H 1 + +#include + +#include "pxtools.h" + +extern bool pxmagicRestoreStage(pxConfig *config, psString stage, psString whereClause, psString newState); + +extern bool pxmagicAddWhere(pxConfig *config, psString *string, psString table); +extern void pxmagicAddArguments(psMetadata *md); + +#endif // PXMAGIC_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmerge.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmerge.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmerge.c (revision 41104) @@ -0,0 +1,97 @@ +/* + * pxadd.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxmerge.h" + +bool pxmergeQueueByMinidvodbID(pxConfig *config, + psS64 minidvodb_id, + char *mergedvodb, + char *mergedvodb_path + ) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // load the SQL to enqueue our exp_ids from disk once + static psString query = NULL; + if (!query) { + query = pxDataGet("mergetool_queue_merge_id.sql"); + psMemSetPersistent(query, true); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + } + if (!p_psDBRunQueryF(config->dbh,"INSERT INTO mergedvodbRun SELECT 0, minidvodb_id, '%s', '%s', 'new', NULL FROM minidvodbRun WHERE minidvodbRun.minidvodb_id = %lld and minidvodbRun.state = 'merged'", mergedvodb, mergedvodb_path, (long long) minidvodb_id)) + { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + return false; + } + // just to be safe, we should have changed at least one row + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, + "no rows affected - should have changed at least one row \n%s", query); + return false; + } + return true; +} + +bool pxaddQueueByMergeID(pxConfig *config, + psS64 merge_id, + char *destination_host, + char *mergedvodb_rsync_path + ) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // load the SQL to enqueue our exp_ids from disk once + static psString query = NULL; + if (!query) { + query = pxDataGet("mergetool_queue_mergecopy_id.sql"); + psMemSetPersistent(query, true); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + } + + if (!p_psDBRunQueryF(config->dbh,"INSERT INTO mergedvodbCopy SELECT 0, merge_id, '%s', '%s', 0, 'new', NULL, 0 FROM mergedvodbRun WHERE mergedvodbRun.merge_id = %lld and (mergedvodbRun.state = 'merged' or mergedvodbRun.state = 'full')", mergedvodb_rsync_path, destination_host, (long long) merge_id)) + + { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + return false; + } + + // just to be safe, we should have changed at least one row + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, + "no rows affected - should have changed at least one row \n%s", query); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmerge.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmerge.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxmerge.h (revision 41104) @@ -0,0 +1,41 @@ +/* + * pxadd.h + * + * Copyright (C) 2009 Joshua Hoblitt, Chris Waters + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXMERGE_H +#define PXMERGE_H 1 + +#include + +#include "pxtools.h" + + +bool pxmergeQueueByMinidvodbID(pxConfig *config, + psS64 minidvodb_id, + char *mergedvodbb, + char *mergedvodb_path + ); + +bool pxaddQueueByMergeID(pxConfig *config, + psS64 merge_id, + char *destination_host, + char *mergedvodb_rsync_path + ); + + +#endif // PXMERGE_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxminidvodb.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxminidvodb.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxminidvodb.c (revision 41104) @@ -0,0 +1,78 @@ +/* + * pxadd.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxminidvodb.h" + +bool pxaddQueueByMinidvodbID(pxConfig *config, + psS64 minidvodb_id, + char *destination_host, + char *minidvodb_rsync_path + ) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // load the SQL to enqueue our exp_ids from disk once + static psString query = NULL; + if (!query) { + query = pxDataGet("minidvodbtool_queue_minidvodb_id.sql"); + psMemSetPersistent(query, true); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + } + + // queue the exp + // Note: cam_id is being cast here work around psS64 have a different type different + // on 32/64 + //if (!p_psDBRunQueryF(config->dbh, query, + // minidvodb_rsync_path ? minidvodb_rsync_path : "NULL", + // destination_host ? destination_host : "NULL", + // 0, //fault + // "new", // state + // "NULL", + // 0, //dtime + + // (long long) minidvodb_id + // )) + if (!p_psDBRunQueryF(config->dbh,"INSERT INTO minidvodbCopy SELECT 0, minidvodb_id, '%s', '%s', 0, 'new', NULL, 0 FROM minidvodbRun WHERE minidvodbRun.minidvodb_id = %lld and minidvodbRun.state = 'merged'", minidvodb_rsync_path, destination_host, (long long) minidvodb_id)) + + { + psError(PS_ERR_UNKNOWN, false, "database error %s", query); + return false; + } + + // just to be safe, we should have changed at least one row + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, + "no rows affected - should have changed at least one row \n%s", query); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxminidvodb.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxminidvodb.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxminidvodb.h (revision 41104) @@ -0,0 +1,34 @@ +/* + * pxadd.h + * + * Copyright (C) 2009 Joshua Hoblitt, Chris Waters + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXMINIDVODB_H +#define PXMINIDVODB_H 1 + +#include + +#include "pxtools.h" + + +bool pxaddQueueByMinidvodbID(pxConfig *config, + psS64 minidvodb_id, + char *destination_host, + char *minidvodb_rsync_path + ); + +#endif // PXMINIDVODB_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxregister.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxregister.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxregister.c (revision 41104) @@ -0,0 +1,124 @@ +/* + * register.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxregister.h" + +bool pxnewExpSetState(pxConfig *config, psS64 exp_id, const char *state) +{ + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, "0 is not a valid exp_id"); + return false; + } + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!( + (strncmp(state, "run", 4) == 0) + || (strncmp(state, "stop", 5) == 0) + || (strncmp(state, "reg", 4) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid newExp state: %s", state); + return false; + } + + char *query = "UPDATE newExp SET state = '%s' WHERE exp_id = %"PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, exp_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for exp_id %"PRId64, exp_id); + return false; + } + + return true; +} + +bool pxrawExpSetState(pxConfig *config, psS64 exp_id, const char *state) +{ + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, "0 is not a valid exp_id"); + return false; + } + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!( + (strncmp(state, "full", 5) == 0) + || (strncmp(state, "new", 4) == 0) + || (strncmp(state, "goto_compressed", 14) == 0) + || (strncmp(state, "compressed", 11) == 0) + || (strncmp(state, "goto_lossy", 11) == 0) + || (strncmp(state, "lossy", 6) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid rawExp state: %s", state); + return false; + } + + char *query = "UPDATE rawExp SET state = '%s' WHERE exp_id = %"PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, exp_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for exp_id %"PRId64, exp_id); + return false; + } + + return true; +} +bool pxrawImfileSetState(pxConfig *config, psS64 exp_id, const char *class_id, const char *state) +{ + if (!exp_id) { + psError(PS_ERR_UNKNOWN, true, "0 is not a valid exp_id"); + return false; + } + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!( + (strncmp(state, "full", 5) == 0) + || (strncmp(state, "new", 4) == 0) + || (strncmp(state, "goto_compressed", 14) == 0) + || (strncmp(state, "compressed", 11) == 0) + || (strncmp(state, "goto_lossy", 11) == 0) + || (strncmp(state, "lossy", 6) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid rawImfile data_state: %s", state); + return false; + } + + char *query = "UPDATE rawImfile SET data_state = '%s' WHERE class_id = %s AND exp_id = %"PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, class_id, exp_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for exp_id %"PRId64, exp_id); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxregister.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxregister.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxregister.h (revision 41104) @@ -0,0 +1,31 @@ +/* + * pxregister.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXREGISTER_H +#define PXREGISTER_H 1 + +#include + +#include "pxtools.h" + +bool pxnewExpSetState(pxConfig *config, psS64 exp_id, const char *state); +bool pxrawExpSetState(pxConfig *config, psS64 exp_id, const char *state); +bool pxrawImfileSetState(pxConfig *config, psS64 exp_id, const char *class_id, const char *state); + +#endif // PXREGISTER_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxspace.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxspace.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxspace.c (revision 41104) @@ -0,0 +1,122 @@ +/* + * pxmagic.c + * + * Copyright (C) 2009 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +// #include +// #include +// #include + +#include "pxtools.h" + +bool pxspaceAddWhere(pxConfig *config, psString *pQuery, psString table) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(table, NULL); + + PXOPT_LOOKUP_F64(radius, config->args, "-radius", false, false); + + if (radius > 0) { + PXOPT_LOOKUP_F64(ra, config->args, "-ra", false, false); + PXOPT_LOOKUP_F64(decl, config->args, "-decl", false, false); + + ra *= PS_RAD_DEG; + decl *= PS_RAD_DEG; + radius *= PS_RAD_DEG; + + psStringAppend(pQuery, " AND (ACOS((cos(%s.decl) * cos(%lf) * cos(%s.ra - %lf)) +(sin(%s.decl) * sin(%lf))) < %lf)", + table, decl, table, ra, table, decl, radius); + } + + return true; +} + +void pxspaceAddArguments(psMetadata *md) +{ + psMetadataAddF64(md, PS_LIST_TAIL, "-radius", 0, "search for exposures within radius RA DEC (degrees)", false); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra", 0, "RA value for radius search (degrees)", false); + psMetadataAddF64(md, PS_LIST_TAIL, "-decl", 0, "DEC value for radius search (degrees)", false); +} + +bool pxspaceBoxAddWhere(pxConfig *config, psMetadata *where) +{ + PXOPT_LOOKUP_F64(ra_min, config->args, "-ra_min", false, NAN); + PXOPT_LOOKUP_F64(ra_max, config->args, "-ra_max", false, NAN); + PXOPT_LOOKUP_F64(dec_min, config->args, "-dec_min", false, NAN); + PXOPT_LOOKUP_F64(dec_max, config->args, "-dec_max", false, NAN); + + if (isfinite(ra_min)) { + ra_min *= PS_RAD_DEG; + psMetadataAddF32(where,PS_LIST_TAIL,"rawExp.ra",PS_META_DUPLICATE_OK, ">=", ra_min); + } + if (isfinite(ra_max)) { + ra_max *= PS_RAD_DEG; + psMetadataAddF32(where,PS_LIST_TAIL,"rawExp.ra",PS_META_DUPLICATE_OK, "<=", ra_max); + } + if (isfinite(dec_min)) { + dec_min *= PS_RAD_DEG; + psMetadataAddF32(where,PS_LIST_TAIL,"rawExp.decl",PS_META_DUPLICATE_OK, ">=", dec_min); + } + if (isfinite(dec_max)) { + dec_max *= PS_RAD_DEG; + psMetadataAddF32(where,PS_LIST_TAIL,"rawExp.decl",PS_META_DUPLICATE_OK, "<=", dec_max); + } + + return true; +} + +void pxspaceBoxAddArguments(psMetadata *md) +{ + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_min", 0, "search by right ascension (degrees)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-dec_min", 0, "search by declination (degrees)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-ra_max", 0, "search by right ascension (degrees)", NAN); + psMetadataAddF64(md, PS_LIST_TAIL, "-dec_max", 0, "search by declination (degrees)", NAN); +} + + +void pxskycellAddArguments(psMetadata *md) +{ + psMetadataAddF32(md, PS_LIST_TAIL, "-ra_min", 0, "search by right ascension (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-dec_min", 0, "search by declination (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-ra_max", 0, "search by right ascension (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-dec_max", 0, "search by declination (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-glong_min", 0, "search by galactic longitude (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-glat_min", 0, "search by galactic latitude (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-glong_max", 0, "search by galactic longitude (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-glat_max", 0, "search by galactic latitude (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-abs_glat_min", 0, "search by absolute value(galactic latitude) (degrees)", NAN); + psMetadataAddF32(md, PS_LIST_TAIL, "-abs_glat_max", 0, "search by absolute value(galactic latitude) (degrees)", NAN); +} + +bool pxskycellAddWhere(pxConfig *config, psMetadata *where) +{ + PXOPT_COPY_F32(config->args, where, "-ra_min", "skycell.radeg", ">="); + PXOPT_COPY_F32(config->args, where, "-dec_min", "skycell.decdeg", ">="); + PXOPT_COPY_F32(config->args, where, "-ra_max", "skycell.radeg", "<"); + PXOPT_COPY_F32(config->args, where, "-dec_max", "skycell.decdeg", "<"); + PXOPT_COPY_F32(config->args, where, "-glong_min", "skycell.glong", ">="); + PXOPT_COPY_F32(config->args, where, "-glat_min", "skycell.glat", ">="); + PXOPT_COPY_F32(config->args, where, "-glong_max", "skycell.glong", "<"); + PXOPT_COPY_F32(config->args, where, "-glat_max", "skycell.glat", "<"); + PXOPT_COPY_F32(config->args, where, "-abs_glat_min", "abs(skycell.glat)", ">="); + PXOPT_COPY_F32(config->args, where, "-abs_glat_max", "abs(skycell.glat)", "<"); + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxspace.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxspace.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxspace.h (revision 41104) @@ -0,0 +1,32 @@ +/* + * pxspace.h + * + * Copyright (C) 2009 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXSPACE_H +#define PXSPACE_H 1 + +extern bool pxspaceAddWhere(pxConfig *config, psString *pQuery, psString table); +extern void pxspaceAddArguments(psMetadata *md); + +extern bool pxspaceBoxAddWhere(pxConfig *config, psMetadata *where); +extern void pxspaceBoxAddArguments(psMetadata *md); + +extern bool pxskycellAddWhere(pxConfig *config, psMetadata *where); +extern void pxskycellAddArguments(psMetadata *md); + +#endif // PXSPACE_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtag.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtag.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtag.c (revision 41104) @@ -0,0 +1,102 @@ +/* + * pxtag.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxtag.h" + +psString pxGenExpTag(pxConfig *config, const char *exp_name) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // start a transaction so we don't increment the expTag counter unless we + // can successfully retreive it's value + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + + if (!p_psDBRunQuery(config->dbh, + "UPDATE expTagCounter SET counter = LAST_INSERT_ID(counter + 1)")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + + // psDBLastInsertID() can't be used here as it called mysql_insert_id() + // which doesn't work with this trick. See: + // http://dev.mysql.com/doc/refman/4.1/en/information-functions.html + if (!p_psDBRunQuery(config->dbh, "SELECT LAST_INSERT_ID() as counter")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + // sanity check that we only got one row + if (psArrayLength(output) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have gotten 1 row but %lu rows were returned", psArrayLength(output)); + psFree(output); + return NULL; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return NULL; + } + + psMetadata *row = output->data[0]; + bool status = false; + psU64 counter = psMetadataLookupU64(&status, row, "counter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for counter"); + psFree(output); + return NULL; + } + psString exp_id = NULL; + psStringAppend(&exp_id, "%s.%" PRIu64, exp_name, counter); + psFree(output); + + return exp_id; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtag.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtag.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtag.h (revision 41104) @@ -0,0 +1,29 @@ +/* + * pxtag.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXTAG_H +#define PXTAG_H 1 + +#include + +#include "pxconfig.h" + +psString pxGenExpTag(pxConfig *config, const char *exp_name); + +#endif // PXTAG_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtools.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtools.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtools.c (revision 41104) @@ -0,0 +1,501 @@ +/* + * pxtool.c + * + * Copyright (C) 2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" + +bool pxIsValidState(const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + if (!strcmp(state, "new")) return true; + if (!strcmp(state, "reg")) return true; + if (!strcmp(state, "full")) return true; + if (!strcmp(state, "drop")) return true; + if (!strcmp(state, "wait")) return true; + if (!strcmp(state, "keep")) return true; + if (!strcmp(state, "goto_cleaned")) return true; + if (!strcmp(state, "error_cleaned")) return true; + if (!strcmp(state, "goto_purged")) return true; + if (!strcmp(state, "error_purged")) return true; + if (!strcmp(state, "goto_scrubbed")) return true; + if (!strcmp(state, "error_scrubbed")) return true; + if (!strcmp(state, "cleaned")) return true; + if (!strcmp(state, "update")) return true; + if (!strcmp(state, "purged")) return true; + if (!strcmp(state, "scrubbed")) return true; + return false; +} + +bool pxIsValidCleanedState(const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + if (!strcmp(state, "cleaned")) return true; + if (!strcmp(state, "purged")) return true; + if (!strcmp(state, "scrubbed")) return true; + return false; +} + +psString pxMergeCodeVersions(psString version1, psString version2) +{ + if (!version1 && !version2) { + return NULL; + } + + if (!version1) { + return psStringCopy(version2); + } + if (!version2) { + return psStringCopy(version1); + } + + bool mod1 = false, mod2 = false; // Modified versions? + if (strchr(version1, 'M')) { + psStringSubstitute(&version1, "M", ""); + mod1 = true; + } + if (strchr(version2, 'M')) { + psStringSubstitute(&version2, "M", ""); + mod2 = true; + } + + int num1 = strtol(version1, NULL, 10); + int num2 = strtol(version2, NULL, 10); + int numO = PS_MAX(num1, num2); + + psString out = NULL; + psStringAppend(&out, "%" PRId32, numO); + if (mod1 || mod2) { + psStringAppend(&out, "M"); + } + return out; +} + +bool pxCoalesceRunStatus(pxConfig *config, const psString dbQFile, psS64 stage_id, psString *software_ver, + psS64 *maskfrac_npix, psF32 *maskfrac_static, psF32 *maskfrac_dynamic, + psF32 *maskfrac_magic, psF32 *maskfrac_advisory) +{ + psString query = pxDataGet(dbQFile); + if (!query) { + psError(psErrorCodeLast(), false, "Unable to read query"); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, stage_id)) { + psError(psErrorCodeLast(), false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + *maskfrac_npix = 0; + *maskfrac_static = 0.0; + *maskfrac_dynamic = 0.0; + *maskfrac_magic = 0.0; + *maskfrac_advisory = 0.0; + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + psS32 this_npix = psMetadataLookupS32(NULL, row, "maskfrac_npix"); + psF32 this_static = psMetadataLookupF32(NULL, row, "maskfrac_static"); + psF32 this_dynamic = psMetadataLookupF32(NULL, row, "maskfrac_dynamic"); + psF32 this_magic = psMetadataLookupF32(NULL, row, "maskfrac_magic"); + psF32 this_advisory = psMetadataLookupF32(NULL, row, "maskfrac_advisory"); + + psTrace("pxtools", 3, "Mask stats: %d %f %f %f %f\n", + this_npix, this_static, this_dynamic, this_magic, this_advisory); + + if (this_npix == 0 || this_npix == PS_MAX_S32) { + continue; + } + if (!isfinite(this_static) || !isfinite(this_dynamic) || + !isfinite(this_magic) || !isfinite(this_advisory)) { + continue; + } + + psString this_version = psMetadataLookupStr(NULL, row, "software_ver"); + *software_ver = pxMergeCodeVersions(*software_ver,this_version); + + *maskfrac_npix += this_npix; + *maskfrac_static += this_npix * this_static; + *maskfrac_dynamic += this_npix * this_dynamic; + *maskfrac_magic += this_npix * this_magic; + *maskfrac_advisory += this_npix * this_advisory; + } + psFree(output); + + if (*maskfrac_npix > 0) { + *maskfrac_static /= *maskfrac_npix; + *maskfrac_dynamic /= *maskfrac_npix; + *maskfrac_magic /= *maskfrac_npix; + *maskfrac_advisory /= *maskfrac_npix; + } + + return true; +} + +bool pxSetRunSoftware(pxConfig *config, const psString tableName, const psString stage_id_name, + const psS64 stage_id, psString software_ver) +{ + char *query = "UPDATE %s SET software_ver = '%s' WHERE %s = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, tableName, software_ver, stage_id_name, stage_id)) { + psError(psErrorCodeLast(), false, + "failed to set software version for %s %" PRId64, stage_id_name, stage_id); + return false; + } + + return true; +} + +bool pxSetRunMaskfrac(pxConfig *config, const psString tableName, const psString stage_id_name, + const psS64 stage_id, psS64 maskfrac_npix, psF32 maskfrac_static, + psF32 maskfrac_dynamic, psF32 maskfrac_magic, psF32 maskfrac_advisory) +{ + char *query = "UPDATE %s SET maskfrac_npix = %f, maskfrac_static = %f, maskfrac_dynamic = %f, " + "maskfrac_magic = %f, maskfrac_advisory = %f WHERE %s = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, tableName, (float)maskfrac_npix, maskfrac_static, + maskfrac_dynamic, maskfrac_magic, maskfrac_advisory, stage_id_name, stage_id)) { + psError(psErrorCodeLast(), false, + "failed to set maskfrac stats for %s %" PRId64,stage_id_name,stage_id); + return false; + } + + + + return true; +} + +bool pxCamSetRunMaskfrac(pxConfig *config, const psString tableName, + const psString stage_id_name, const psS64 stage_id, + psS64 maskfrac_ref_npix, psF32 maskfrac_ref_static, psF32 maskfrac_ref_dynamic, + psF32 maskfrac_ref_magic, psF32 maskfrac_ref_advisory, + psS64 maskfrac_max_npix, psF32 maskfrac_max_static, psF32 maskfrac_max_dynamic, + psF32 maskfrac_max_magic, psF32 maskfrac_max_advisory) { + char *query = "UPDATE %s SET maskfrac_ref_npix = %f, maskfrac_ref_static = %f, maskfrac_ref_dynamic = %f, " + "maskfrac_ref_magic = %f, maskfrac_ref_advisory = %f, maskfrac_max_npix = %f, maskfrac_max_static = %f, " + "maskfrac_max_dynamic = %f, maskfrac_max_magic = %f, maskfrac_max_advisory = %f WHERE %s = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, tableName, (float)maskfrac_ref_npix, maskfrac_ref_static, + maskfrac_ref_dynamic, maskfrac_ref_magic, maskfrac_ref_advisory, + (float)maskfrac_max_npix, maskfrac_max_static, + maskfrac_max_dynamic, maskfrac_max_magic, maskfrac_max_advisory, + stage_id_name, stage_id)) { + psError(psErrorCodeLast(), false, + "failed to set maskfrac stats for %s %" PRId64,stage_id_name,stage_id); + return false; + } + + return true; +} + +// 'scrubbed' is no longer a virtual state equivalent to cleaned, but allows files to be removed +// even if the config files is missing. This change was prompted as files that are cleaned can +// be regenerated, but that is not certain after being scrubbed. + + +// change the value for tableName.columName from 'full' to 'cleaned' if necessary +bool pxSetStateCleaned(const char *tableName, const char *columnName, psArray *rows) +{ + for (long i = 0; i < psArrayLength(rows); i++) { + psMetadata *row = rows->data[i]; + const char *state = psMetadataLookupStr(NULL, row, columnName); + if (!state) { + psError(PS_ERR_PROGRAMMING, false, "%s not found in row %ld of table %s", + columnName, i, tableName); + return false; + } + if (!strcmp("full", state)) { + // change full to cleaned + psMetadataAddStr(row, PS_LIST_TAIL, columnName, PS_META_REPLACE, "", "cleaned"); + } else if (strcmp("cleaned", state)) { + // if state isn't cleaned or full we can't set it to cleaned + psError(PS_ERR_PROGRAMMING, true, "%s with state %s may not be exported cleaned", + tableName, state); + return false; + } + } + return true; +} + +// XXX verify data type? +bool pxAddLabelSearchArgs (pxConfig *config, psMetadata *where, char *name, char *field, char *op) { + + psMetadataItem *item = psMetadataLookup(config->args, name); + if (!item) { + psError(psErrorCodeLast(), false, "failed to lookup value for %s", name); + return false; + } + psAssert (item->type == PS_DATA_METADATA_MULTI, "%s should be a multi container", name); + psAssert (item->data.list->n, "%s should at least have a place-holder", name); + psMetadataItem *entry = (psMetadataItem *)item->data.list->head->data; + psAssert (entry, "%s should at least have a place-holder", name); + if (entry->data.str) { + psListIterator *iter = psListIteratorAlloc (item->data.list, PS_LIST_HEAD, true); + psMetadataItem *item = NULL; + while ((item = psListGetAndIncrement(iter))) { + psMetadataItem *new = psMetadataItemCopy(item); + // need to change the name and comment + psFree (new->name); + new->name = psStringCopy (field); + psFree (new->comment); + new->comment = psStringCopy (op); + if (!psMetadataAddItem(where, new, PS_LIST_TAIL, PS_META_DUPLICATE_OK)) { + psError(psErrorCodeLast(), false, "failed to add item %s", field); + psFree(where); + return false; + } + } + psFree(iter); + } + return true; +} + +// shared code for updating the various strings for a Run +bool pxUpdateRun(pxConfig *config, psMetadata *where, psString *pQuery, psString runTable, psString idColumn, psString fileTable, bool has_dist_group, bool has_magicked) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(where, false); + PS_ASSERT_PTR_NON_NULL(pQuery, false); + PS_ASSERT_PTR_NON_NULL(*pQuery, false); + PS_ASSERT_PTR_NON_NULL(runTable, false); + PS_ASSERT_PTR_NON_NULL(idColumn, false); + PS_ASSERT_PTR_NON_NULL(fileTable, false); + + // make sure that -state is not the only selection parameter + PXOPT_LOOKUP_STR(where_state, config->args, "-state", false, false); + if (where_state && (psListLength(where->list) < 2)) { + psError(PXTOOLS_ERR_CONFIG, true, "selection by -state alone is not allowed"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + +#ifdef DISALLOW_CHANGE_TO_UPDATE + // Back in the days of magic we didn't allow state changes of Run's + // to update because the interaction with destreaking was problematic + // With the death of magic we can allow this now. + if ((state)&&(!strcmp(state, "update"))) { + fprintf(stderr, "'-updaterun -set_state update' is not supported."); + if (!strcmp(runTable, "chipRun")) { + fprintf(stderr, " Use -setimfiletoupdate.\n"); + } else { + fprintf(stderr, " Use -setskyfiletoupdate.\n"); + } + exit(1); + } +#endif + psString dist_group = NULL; + if (has_dist_group) { + PXOPT_LOOKUP_STR(tmp_dist_group, config->args, "-set_dist_group", false, false); + dist_group = tmp_dist_group; + } + + if ((!state) && (!label) && (!data_group) && (has_dist_group && !dist_group) && !(note)) { + psError(PXTOOLS_ERR_CONFIG, false, "parameters are required"); + return false; + } + + if (state && ! pxIsValidState(state)) { + psError(PXTOOLS_ERR_CONFIG, false, "pxIsValidState failed"); + return false; + } + + // first paramter is added with "SET param = 'value'" + // subseqent ones with ", param = 'value'" + char *separator = " SET "; + char *comma = ","; + +# define addColumn(_tab, _val) \ + do { \ + if (_val) { \ + psStringAppend(pQuery, "%s %s.%s = '%s'", separator, _tab, #_val, _val); \ + separator = comma; \ + } \ + } while (0) + + addColumn(runTable, state); + addColumn(runTable, data_group); + if (has_dist_group) { + addColumn(runTable, dist_group); + } + addColumn(runTable, note); + addColumn(runTable, label); + + psString joinHook = psStringCopy(""); + psString fileWhere = NULL; + +#ifdef DISALLOW_CHANGE_TO_UPDATE + if (state && !strcmp(state, "update")) { + psStringAppend(&joinHook, "\n JOIN %s USING(%s)", fileTable, idColumn); + psStringAppend(pQuery, ", %s.data_state = 'update'", fileTable); + psStringAppend(&fileWhere, "AND %s.data_state = 'cleaned'", fileTable); + } +#endif + + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(pQuery, " %s", whereClause); + psFree(whereClause); + if (fileWhere) { + psStringAppend(pQuery, "%s", fileWhere); + } + if (has_magicked) { + pxmagicAddWhere(config, pQuery, runTable); + } + + bool mdok; // Status of MD lookup + if (psMetadataLookupBool(&mdok, config->args, "-pretend")) { + psLogMsg("pxtools", PS_LOG_INFO, "Query to run: %s\n", *pQuery); + return true; + } + + if (!p_psDBRunQueryF(config->dbh, *pQuery, joinHook)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + return true; +} + +bool pxLookupVersion(pxConfig *config, psArray **pArray) +{ + const char *query = "SELECT * FROM dbversion"; + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(psErrorCodeLast(), false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psFree(output); + psError(PXTOOLS_ERR_PROG, true, "no rows in dbversion"); + return false; + } + if (psArrayLength(output) > 1) { + psError(PXTOOLS_ERR_PROG, true, "unexpected number of rows found in dbversion: %ld", + psArrayLength(output)); + return false; + } + *pArray = output; + + return true; +} + +psString pxGetDBVersion(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psArray *array = NULL; + if (!pxLookupVersion(config, &array)) { + psError(psErrorCodeLast(), false, "pxLookupVersion failed"); + return NULL; + } + psMetadata *md = array->data[0]; + if (!md) { + psError(PXTOOLS_ERR_PROG, true, "output of pxLookupVersion is null"); + return NULL; + } + + psString version = psMetadataLookupStr(NULL, md, "schema_version"); + + return version; +} + +bool pxExportVersion(pxConfig *config, FILE *file) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(file, NULL); + + psArray *array = NULL; + if (!pxLookupVersion(config, &array) || !array) { + psError(psErrorCodeLast(), false, "pxLookupVersion failed"); + return false; + } + if (!ippdbPrintMetadatas(file, array, "dbversion", true)) { + psError(psErrorCodeLast(), false, "failed to print array"); + psFree(array); + return false; + } + return true; +} + +bool pxCheckImportVersion(pxConfig *config, psMetadata *input) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(input, NULL); + + // This code was adapted from the way camtool parses the structures. + // Is this really the way to do it? + psMetadataItem *multi_item = psMetadataLookup(input, "dbversion"); + if (!multi_item || (multi_item->type != PS_DATA_METADATA_MULTI)) { + psError(PXTOOLS_ERR_PROG, true, "dbversion multi not found in input"); + return false; + } + + psMetadataItem *dbversion = psListGet(multi_item->data.list, 0); + if (!dbversion) { + psError(PXTOOLS_ERR_PROG, true, "dbversion not found in input"); + return false; + } + + if (!strcmp(dbversion->name, "dbversion")) { + // horray + psMetadata *md = dbversion->data.md; + psString schema_version = pxGetDBVersion(config); + if (!schema_version) { + psError(psErrorCodeLast(), false, "pxGetDBVersion failed"); + return false; + } + + psString import_version = psMetadataLookupStr(NULL, md, "schema_version"); + if (import_version && strcmp(import_version, schema_version)) { + psError(PXTOOLS_ERR_PROG, true, "input file schema_version: %s does not match data base: %s", + import_version, schema_version); + return false; + } else if (!import_version) { + psError(PXTOOLS_ERR_PROG, true, "input file schema_version is NULL"); + return false; + } else { + // YIPPEE this file is the same version + } + } else { + psError(PXTOOLS_ERR_PROG, true, "Unexpected config dump format"); + return false; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtools.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtools.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtools.h (revision 41104) @@ -0,0 +1,610 @@ +/* + * pxtools.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXTOOLS_H +#define PXTOOLS_H 1 + +#include +#include // for strcmp and strncmp +#include // for strcasecmp +#include // for unlink + +// #include +// #include +// #include + +#include +#include +#include + +#include "pxconfig.h" +#include "pxtoolsErrorCodes.h" + +#include "pxadd.h" +#include "pxcam.h" +#include "pxchip.h" +#include "pxdata.h" +#include "pxfake.h" +#include "pxwarp.h" +#include "pxregister.h" +#include "pxtag.h" +#include "pxtree.h" +#include "pxmagic.h" +#include "pxspace.h" + +# define MAX_ROWS 10e9 +# define PXTOOL_MODE_NONE 0x0 +// we do not revert or update components with this fault value +// This is the same value as PSTAMP_GONE +# define PXTOOL_DO_NOT_REVERT_FAULT 26 + +bool pxIsValidState(const char *state); +bool pxIsValidCleanedState(const char *state); +psString pxMergeCodeVersions(psString version1, psString version2); +bool pxCoalesceRunStatus(pxConfig *config, const psString dbQFile, psS64 stage_id, psString *software_ver, + psS64 *maskfrac_npix, psF32 *maskfrac_static, psF32 *maskfrac_dynamic, + psF32 *maskfrac_magic, psF32 *maskfrac_advisory); +bool pxSetRunSoftware(pxConfig *config, const psString tableName, const psString stage_id_name, const psS64 stage_id, + psString software_ver); +bool pxSetRunMaskfrac(pxConfig *config, const psString tableName, const psString stage_id_name, const psS64 stage_id, + psS64 maskfrac_npix, psF32 maskfrac_static, psF32 maskfrac_dynamic, + psF32 maskfrac_magic, psF32 maskfrac_advisory); +bool pxCamSetRunMaskfrac(pxConfig *config, const psString tableName, const psString stage_id_name, const psS64 stage_id, + psS64 maskfrac_ref_npix, psF32 maskfrac_ref_static, psF32 maskfrac_ref_dynamic, + psF32 maskfrac_ref_magic, psF32 maskfrac_ref_advisory, + psS64 maskfrac_max_npix, psF32 maskfrac_max_static, psF32 maskfrac_max_dynamic, + psF32 maskfrac_max_magic, psF32 maskfrac_max_advisory); + +bool pxSetStateCleaned(const char *tableName, const char *columnName, psArray *rows); +bool pxAddLabelSearchArgs (pxConfig *config, psMetadata *where, char *field, char *name, char *op); + +bool pxSetFaultCode(psDB *dbh, const char *tableName, psMetadata *where, psS16 code, psS16 quality); +bool pxExportVersion(pxConfig *config, FILE *f); +bool pxCheckImportVersion(pxConfig *config, psMetadata *md); + +psExit pxerrorGetExitStatus(void); + +void pxUsage(FILE *stream, int argc, char **argv, const char *modeName, psMetadata *argSet); +bool pxGetOptions(FILE *stream, int argc, char **argv, pxConfig *config, psMetadata *modes, psMetadata *argSets); + +bool pxUpdateRun(pxConfig *config, psMetadata *where, psString *pQuery, psString runTable, psString idColumn, psString fileTable, bool has_dist_group, bool has_magicked); + +#define PXOPT_ADD_MODE(option, comment, modeval, argset) \ +{ \ + if (!psMetadataAddMetadata(argSets, PS_LIST_TAIL, option, 0, comment, argset)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option); \ + } \ + psFree(argset); \ +\ + if (!psMetadataAddU32(modes, PS_LIST_TAIL, option, 0, comment, modeval)) {;\ + psError(PS_ERR_UNKNOWN, false, "failed to add argset for %s", option); \ + } \ +} + +#define PXOPT_LOOKUP_STR(var, md, key, required, ret) \ +psString var; \ +{ \ + bool status; \ + \ + var = psMetadataLookupStr(&status, md, key); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", key); \ + return ret; \ + } \ + \ + if (required && (!var)) { \ + psError(PS_ERR_UNKNOWN, true, "%s is required", key); \ + return ret; \ + } \ +} + +#define PXOPT_LOOKUP_F(var, md, key, type, required, ret) \ +ps##type var; \ +{ \ + bool status; \ + \ + var = psMetadataLookup##type(&status, md, key); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", key); \ + return ret; \ + } \ + \ + if (required && isnan(var)) { \ + psError(PS_ERR_UNKNOWN, true, "%s is required", key); \ + return ret; \ + } \ +} + +#define PXOPT_LOOKUP_F32(var, md, key, required, ret) \ + PXOPT_LOOKUP_F(var, md, key, F64, required, ret) + +#define PXOPT_LOOKUP_F64(var, md, key, required, ret) \ + PXOPT_LOOKUP_F(var, md, key, F64, required, ret) + +#define PXOPT_LOOKUP_PRIMITIVE(var, md, key, type, suffix, max, required, ret) \ +type var; \ +{ \ + psMetadataItem *item = psMetadataLookup(md, key); \ + if (!item) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", key); \ + return ret; \ + } \ + psAssert(item->comment, "metadata item comment should be defined"); \ + if (required && (!psStrcasestr(item->comment, "(found)"))) { \ + psError(PS_ERR_UNKNOWN, true, "%s is required", key); \ + return ret; \ + } \ + \ + var = item->data.suffix; \ + \ +} + +#define PXOPT_LOOKUP_S16(var, md, key, required, ret) \ + PXOPT_LOOKUP_PRIMITIVE(var, md, key, psS16, S16, INT16_MAX, required, ret) + +#define PXOPT_LOOKUP_S32(var, md, key, required, ret) \ + PXOPT_LOOKUP_PRIMITIVE(var, md, key, psS32, S32, INT32_MAX, required, ret) + +#define PXOPT_LOOKUP_S64(var, md, key, required, ret) \ + PXOPT_LOOKUP_PRIMITIVE(var, md, key, psS64, S64, INT64_MAX, required, ret) + +#define PXOPT_LOOKUP_U16(var, md, key, required, ret) \ + PXOPT_LOOKUP_PRIMITIVE(var, md, key, psU16, U16, UINT16_MAX, required, ret) + +#define PXOPT_LOOKUP_U32(var, md, key, required, ret) \ + PXOPT_LOOKUP_PRIMITIVE(var, md, key, psU32, U32, UINT32_MAX, required, ret) + +#define PXOPT_LOOKUP_U64(var, md, key, required, ret) \ + PXOPT_LOOKUP_PRIMITIVE(var, md, key, psU64, U64, UINT64_MAX, required, ret) + +#define PXOPT_LOOKUP_TIME(var, md, key, required, ret) \ +psTime *var; \ +{ \ + bool status; \ + \ + var = psMetadataLookupTime(&status, md, key); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", key); \ + return ret; \ + } \ + \ + if (required && (!var)) { \ + psError(PS_ERR_UNKNOWN, true, "%s is required", key); \ + return ret; \ + } \ +} + +#define PXOPT_LOOKUP_BOOL(var, md, key, ret) \ +bool var; \ +{ \ + bool status; \ + \ + var = psMetadataLookupBool(&status, md, key); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", key); \ + return ret; \ + } \ +} + +// XXX the PXOPT_COPY_* macros free 'to' on error + +#define PXOPT_COPY_PRIMITIVE(from, to, type, suffix, oldname, newname, newcomment) \ + { \ + psMetadataItem *item = psMetadataLookup(from, oldname); \ + if (!item) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", oldname); \ + return false; \ + } \ + psAssert(item->comment, "metadata item comment should be defined"); \ + \ + if (psStrcasestr(item->comment, "(found)")) { \ + if (!psMetadataAdd##suffix(to, PS_LIST_TAIL, newname, PS_META_DUPLICATE_OK, newcomment, item->data.suffix)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " newname); \ + psFree(to); \ + return false; \ + } \ + } \ + } + +#define PXOPT_COPY_V(from, to, type, suffix, oldname, newname, comment) \ +{ \ + bool status = false; \ + type var = psMetadataLookup##suffix(&status, from, oldname); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for " oldname); \ + return false; \ + } \ + if (var) { \ + if (!psMetadataAdd##suffix(to, PS_LIST_TAIL, newname, PS_META_DUPLICATE_OK, comment, var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " newname); \ + psFree(to); \ + return false; \ + } \ + }\ +} + +#define PXOPT_COPY_F(from, to, type, oldname, newname, comment) \ +{ \ + bool status = false; \ + ps##type var = psMetadataLookup##type(&status, from, oldname); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for " oldname); \ + return false; \ + } \ + if (!isnan(var)) { \ + if (!psMetadataAdd##type(to, PS_LIST_TAIL, newname, PS_META_DUPLICATE_OK, comment, var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " newname); \ + psFree(to); \ + return false; \ + } \ + } \ +} + +// convert the supplied value from degrees (external) to radians (internal) +#define PXOPT_COPY_RADEC(from, to, oldname, newname, comment) \ +{ \ + bool status = false; \ + psF64 var = psMetadataLookupF64(&status, from, oldname); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for " oldname); \ + return false; \ + } \ + if (!isnan(var)) { \ + if (!psMetadataAddF64(to, PS_LIST_TAIL, newname, PS_META_DUPLICATE_OK, comment, PS_RAD_DEG*var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " newname); \ + psFree(to); \ + return false; \ + } \ + } \ +} + +#define PXOPT_COPY_F32(from, to, oldname, newname, comment) \ + PXOPT_COPY_F(from, to, F32, oldname, newname, comment) + +#define PXOPT_COPY_F64(from, to, oldname, newname, comment) \ + PXOPT_COPY_F(from, to, F64, oldname, newname, comment) + +#define PXOPT_COPY_TIME(from, to, oldname, newname, comment) \ + PXOPT_COPY_V(from, to, psTime *, Time, oldname, newname, comment) + +#define PXOPT_COPY_STR(from, to, oldname, newname, comment) \ + PXOPT_COPY_V(from, to, psString, Str, oldname, newname, comment) + +#define PXOPT_COPY_S16(from, to, oldname, newname, comment) \ + PXOPT_COPY_PRIMITIVE(from, to, psS16, S16, oldname, newname, comment) + +#define PXOPT_COPY_S32(from, to, oldname, newname, comment) \ + PXOPT_COPY_PRIMITIVE(from, to, psS32, S32, oldname, newname, comment) + +#define PXOPT_COPY_S64(from, to, oldname, newname, comment) \ + PXOPT_COPY_PRIMITIVE(from, to, psS64, S64, oldname, newname, comment) + +#define PXOPT_COPY_U16(from, to, oldname, newname, comment) \ + PXOPT_COPY_PRIMITIVE(from, to, psU16, U16, oldname, newname, comment) + +#define PXOPT_COPY_U32(from, to, oldname, newname, comment) \ + PXOPT_COPY_PRIMITIVE(from, to, psU32, U32, oldname, newname, comment) + +#define PXOPT_COPY_U64(from, to, oldname, newname, comment) \ + PXOPT_COPY_PRIMITIVE(from, to, psU64, U64, oldname, newname, comment) + +#define PXOPT_COPY_BOOL(from, to, oldname, newname, comment) \ + PXOPT_COPY_V(from, to, psBool, Bool, oldname, newname, comment) + +/*** these PXOPT_ADD_WHERE macros are used to construct the default sql elements ***/ + +#define PXOPT_ADD_WHERE_STR(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, #name, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_STR_ALIAS(name,realname) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, name))) { \ + if (!psMetadataAddStr(config->where, PS_LIST_TAIL, realname, 0, "==", str)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", name); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_S16(name) \ +{ \ + psS16 s16 = 0; \ + bool status = false; \ + if ((s16= psMetadataLookupS16(&status, config->args, "-" #name))) { \ + if (!psMetadataAddS16(config->where, PS_LIST_TAIL, #name, 0, "==", s16)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_BOOL(name) \ +{ \ + bool value = false; \ + bool status = false; \ + if ((value = psMetadataLookupBool(&status, config->args, "-" #name))) { \ + if (!psMetadataAddBool(config->where, PS_LIST_TAIL, #name, 0, "==", value)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_S32(name) \ +{ \ + psS32 s32 = 0; \ + bool status = false; \ + if ((s32= psMetadataLookupS32(&status, config->args, "-" #name))) { \ + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, #name, 0, "==", s32)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_S64(name) \ +{ \ + psS64 s64 = 0; \ + bool status = false; \ + if ((s64= psMetadataLookupS64(&status, config->args, "-" #name))) { \ + if (!psMetadataAddS64(config->where, PS_LIST_TAIL, #name, 0, "==", s64)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_F32(name) \ +{ \ + psF32 var = 0; \ + bool status = false; \ + if ((var = psMetadataLookupF32(&status, config->args, "-" #name))) { \ + if (!isnan(var)) { \ + if (!psMetadataAddF32(config->where, PS_LIST_TAIL, #name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_F64(name) \ +{ \ + psF64 var = 0; \ + bool status = false; \ + if ((var = psMetadataLookupF64(&status, config->args, "-" #name))) { \ + if (!isnan(var)) { \ + if (!psMetadataAddF64(config->where, PS_LIST_TAIL, #name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_BOOL_ALIAS(flag,name) \ +{ \ + bool value = 0; \ + bool status = false; \ + if ((value = psMetadataLookupBool(&status, config->args, flag))) { \ + if (!psMetadataAddBOOL(config->where, PS_LIST_TAIL, name, 0, "==", value)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", flag); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_S16_ALIAS(flag,name) \ +{ \ + psS16 s16 = 0; \ + bool status = false; \ + if ((s16= psMetadataLookupS16(&status, config->args, flag))) { \ + if (!psMetadataAddS16(config->where, PS_LIST_TAIL, name, 0, "==", s16)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", flag); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_S32_ALIAS(flag,name) \ +{ \ + psS32 s32 = 0; \ + bool status = false; \ + if ((s32= psMetadataLookupS32(&status, config->args, flag))) { \ + if (!psMetadataAddS32(config->where, PS_LIST_TAIL, name, 0, "==", s32)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", flag); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_S64_ALIAS(flag,name) \ +{ \ + psS64 s64 = 0; \ + bool status = false; \ + if ((s64= psMetadataLookupS64(&status, config->args, flag))) { \ + if (!psMetadataAddS64(config->where, PS_LIST_TAIL, name, 0, "==", s64)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", flag); \ + psFree(config); \ + return NULL; \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_F32_ALIAS(flag,name) \ +{ \ + psF32 var = 0; \ + bool status = false; \ + if ((var = psMetadataLookupF32(&status, config->args, flag))) { \ + if (!isnan(var)) { \ + if (!psMetadataAddF32(config->where, PS_LIST_TAIL, name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", flag); \ + psFree(config); \ + return NULL; \ + } \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_F64_ALIAS(flag,name) \ +{ \ + psF64 var = 0; \ + bool status = false; \ + if ((var = psMetadataLookupF64(&status, config->args, flag))) { \ + if (!isnan(var)) { \ + if (!psMetadataAddF64(config->where, PS_LIST_TAIL, name, 0, "==", var)) { \ + psError(PS_ERR_UNKNOWN, false, "failed to add item %s", flag); \ + psFree(config); \ + return NULL; \ + } \ + } \ + } \ +} + +#define PXOPT_ADD_WHERE_TIME_STR(name) \ +{ \ + psString str = NULL; \ + bool status = false; \ + if ((str = psMetadataLookupStr(&status, config->args, "-" #name))) { \ + psTime *time = psTimeFromISO(str, PS_TIME_UTC); \ + if (!time) { \ + psError(PS_ERR_UNKNOWN, false, "failed to convert " #name " into a psTime object"); \ + psFree(config); \ + return NULL; \ + } \ + psMetadataItem *item = psMetadataLookup(config->args, "-" #name); \ + if (item) { \ + str = item->comment; \ + } else { \ + str = NULL; \ + } \ + if (!psMetadataAddTime(config->where, PS_LIST_TAIL, #name, 0, str, time)) {\ + psError(PS_ERR_UNKNOWN, false, "failed to add item " #name); \ + psFree(config); \ + return NULL; \ + } \ + psFree(time); \ + } \ +} + + +/// Add a primary item to the mirror database +/// +/// ITEM: Item to add (if it is of the correct type) +/// NAME: Name for table, to match item name +/// ROWTYPE: Type of the row (type returned by PARSEFUNC) +/// PARSEFUNC: Function to parse ITEM and return a ROWTYPE +/// IDENTIFIERS: Vector (of type U64) to store identifiers +/// IDNAME: Element of the row with U64 identifier +/// INSERTFUNC: Function to insert a row into the database +/// DATABASE: Database handle +/// CLEANUP: Operation(s) to perform to cleanup in case of an error +#define PXMIRROR_PRIMARY(ITEM, NAME, ROWTYPE, PARSEFUNC, IDENTIFIERS, IDNAME, INSERTFUNC, DATABASE, CLEANUP) \ + if (strcmp((ITEM)->name, NAME) == 0) { \ + if ((ITEM)->type != PS_DATA_METADATA) { \ + psError(PS_ERR_BAD_PARAMETER_TYPE, true, "Entry %s is not of type METADATA", (ITEM)->name); \ + CLEANUP; \ + return false; \ + } \ + ROWTYPE *row = PARSEFUNC((ITEM)->data.md); /* Row to insert */ \ + if (!row) { \ + psError(PS_ERR_UNEXPECTED_NULL, false, "Unable to generate %s row from metadata", (ITEM)->name); \ + CLEANUP; \ + return false; \ + } \ + if (!INSERTFUNC(DATABASE, row)) { \ + psError(PS_ERR_UNKNOWN, false, "Unable to add %s %" PRIu64, (ITEM)->name, row->IDNAME); \ + psFree(row); \ + CLEANUP; \ + return false; \ + } \ + psVectorAppend((IDENTIFIERS), row->IDNAME); \ + psFree(row); \ + } + +/// Add a dependent item to the mirror database +/// +/// ITEM: Item to add (if it is of the correct type) +/// NAME: Name for table, to match item name +/// ROWTYPE: Type of the row (type returned by PARSEFUNC) +/// PARSEFUNC: Function to parse ITEM and return a ROWTYPE +/// IDENTIFIERS: Vector (of type U64) to check identifiers +/// IDNAME: Element of the row with U64 identifier +/// INSERTFUNC: Function to insert a row into the database +/// DATABASE: Database handle +/// CLEANUP: Operation(s) to perform to cleanup in case of an error +#define PXMIRROR_OTHER(ITEM, NAME, ROWTYPE, PARSEFUNC, IDENTIFIERS, IDNAME, INSERTFUNC, DATABASE, CLEANUP) \ + if (strcmp((ITEM)->name, NAME) == 0) { \ + if ((ITEM)->type != PS_DATA_METADATA) { \ + psError(PS_ERR_BAD_PARAMETER_TYPE, true, "Entry %s is not of type METADATA", (ITEM)->name); \ + CLEANUP; \ + return false; \ + } \ + ROWTYPE *row = PARSEFUNC(item->data.md); /* Row to insert */ \ + if (!row) { \ + psError(PS_ERR_UNEXPECTED_NULL, false, "Unable to generate %s row from metadata", (ITEM)->name); \ + CLEANUP; \ + return false; \ + } \ + bool found = false; /* Found the identifier? */ \ + for (int i = 0; i < (IDENTIFIERS)->n && !found; i++) { \ + if (row->IDNAME == (IDENTIFIERS)->data.U64[i]) { \ + found = true; \ + break; \ + } \ + } \ + if (!found) { \ + psError(PS_ERR_BAD_PARAMETER_VALUE, true, "Identifier not found for %s %" PRIu64, \ + (ITEM)->name, row->IDNAME); \ + psFree(row); \ + CLEANUP; \ + return false; \ + } \ + if (!INSERTFUNC(DATABASE, row)) { \ + psError(PS_ERR_UNKNOWN, false, "Unable to add %s %" PRIu64, (ITEM)->name, row->IDNAME); \ + psFree(row); \ + CLEANUP; \ + return false; \ + } \ + psFree(row); \ + } + +#endif // PXTOOLS_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.c.in =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.c.in (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.c.in (revision 41104) @@ -0,0 +1,25 @@ +/* + * The line + { PXTOOLS_ERR_$X{ErrorCode}, "$X{ErrorDescription}"}, + * (without the Xs) + * will be replaced by values from errorCodes.dat + */ +#include "pslib.h" +#include "pxtoolsErrorCodes.h" + +void pxtoolsErrorRegister(void) +{ + static psErrorDescription errors[] = { + { PXTOOLS_ERR_BASE, "First value we use; lower values belong to psLib" }, + { PXTOOLS_ERR_${ErrorCode}, "${ErrorDescription}"}, + }; + static int nerror = PXTOOLS_ERR_NERROR - PXTOOLS_ERR_BASE; // number of values in enum + + for (int i = 0; i < nerror; i++) { + psErrorDescription *tmp = psAlloc(sizeof(psErrorDescription)); + *tmp = errors[i]; + psErrorRegister(tmp, 1); + psFree(tmp); /* it's on the internal list */ + } + nerror = 0; // don't register more than once +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.dat =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.dat (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.dat (revision 41104) @@ -0,0 +1,9 @@ +# +# This file is used to generate pxtoolsErrorCodes.h +# +BASE = 9000 First value we use; lower values belong to psLib +UNKNOWN Unknown ippTools error code +ARGUMENTS Incorrect arguments +SYS System error +CONFIG Problem in configure files +PROG Programming error Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.h.in =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.h.in (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtoolsErrorCodes.h.in (revision 41104) @@ -0,0 +1,18 @@ +#if !defined(PXTOOLS_ERROR_CODES_H) +#define PXTOOLS_ERROR_CODES_H +/* + * The line + * PXTOOLS_ERR_$X{ErrorCode}, + * (without the X) + * + * will be replaced by values from errorCodes.dat + */ +typedef enum { + PXTOOLS_ERR_BASE = 9000, + PXTOOLS_ERR_${ErrorCode}, + PXTOOLS_ERR_NERROR +} pxtoolsErrorCode; + +void pxtoolsErrorRegister(void); + +#endif Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtree.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtree.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtree.c (revision 41104) @@ -0,0 +1,239 @@ +/* + * pxio.c + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pxtree.h" + +static void pxNodeFree(pxNode *node) +{ + psFree(node->name); + psFree(node->parent); + psFree(node->children); + psFree(node->data); +} + +pxNode *pxNodeAlloc(const char *name) +{ + PS_ASSERT_PTR_NON_NULL(name, NULL); + + pxNode *node = psAlloc(sizeof(pxNode)); + + node->name = psStringCopy(name); + node->parent = NULL; + node->children = psListAlloc(NULL); + node->data = NULL; + + psMemSetDeallocator(node, (psFreeFunc) pxNodeFree); + + return node; +} + +pxNode *pxNodeAddParent(pxNode *node, pxNode *parent) +{ + // add this node to it's parent's children list + psListAdd(parent->children, PS_LIST_TAIL, node); + // and set the parent pointer + node->parent = psMemIncrRefCounter(parent); + + return node; +} + +pxNode *pxNodeAddChild(pxNode *node, pxNode *child) +{ + // add the child node to this nodes list of children + psListAdd(node->children, PS_LIST_TAIL, child); + // if the child already has a parent, release the ref count + if (child->parent) { + psFree(child->parent); + } + // [re]parent the child + child->parent = psMemIncrRefCounter(node); + + return node; +} + +pxNode *pxNodeAddData(pxNode *node, psPtr data) +{ + node->data = psMemIncrRefCounter(data); + + return node; +} + +void pxNodePrint(FILE *stream, pxNode *node) +{ + fprintf(stream, "node name: %s\n", node->name); + fprintf(stream, "node parent's name: %s\n", node->parent + ? node->parent->name + : NULL); + psListIterator *iter = psListIteratorAlloc(node->children, 0, false); + pxNode *child = NULL; + while ((child = psListGetAndIncrement(iter))) { + fprintf(stream, "node child's name: %s\n", child->name); + } + psFree(iter); +} + + +static bool printNodes(void *arg, pxNode *node) +{ + pxNodePrint((FILE *)arg, node); + fprintf((FILE *)arg, "\n\n"); + + return true; +} + + +void pxTreePrint(FILE *stream, pxNode *root) +{ + pxTreeCrawl(root, printNodes, (void *)stream); +} + +// func() returning false means decend no futher along this branch + +bool pxTreeCrawl(pxNode *node, pxNodeFunc func, void *arg) +{ + // if func() returns false, stop + if (!func(arg, node)) { + return false; + } + + psListIterator *iter = psListIteratorAlloc(node->children, 0, false); + pxNode *child = NULL; + while ((child = psListGetAndIncrement(iter))) { + pxTreeCrawl(child, func, arg); + } + psFree(iter); + + return true; +} + +bool pxNodeFuncCountDependants(void *arg, pxNode *node) +{ + (*(int *)arg)++; + return true; +} + +bool pxNodeHasChildren(pxNode *node) +{ + int children = psListLength(node->children); + + return children ? true : false; +} + +bool pxNodeHasGrandChildren(pxNode *node) +{ + // find out how many nodes there are lower in the tree + // subtract the parent node from the count so we have just a tally of + // decendants + int nodes = -1; + pxTreeCrawl(node, pxNodeFuncCountDependants, &nodes); + psTrace("pxtree", PS_LOG_INFO, "node %s has %d dependants", node->name, nodes); + + // find out how many children this node has + int children = psListLength(node->children); + psTrace("pxtree", PS_LOG_INFO, "node %s has %d children", node->name, children); + + if (!children) { + // no children + return false; + } + + if (nodes < children) { + psAbort("you can't have fewer children than decendants!"); + } + + if (nodes == children) { + // no grandchildren + return false; + } + + return true; +} + +pxNode *pxTreeFromMetadata(psMetadata *md) +{ + psHash *forest = psHashAlloc(10); + + psMetadataIterator *iter = psMetadataIteratorAlloc(md, 0, NULL); + psMetadataItem *item = NULL; + while ((item = psMetadataGetAndIncrement(iter))) { + if (!(item->type == PS_DATA_STRING)) { + continue; + } + + // add this node to the forest and hopefully it'll get attached to the + // root tree + pxTreeBuilder(forest, item->name, item->data.str, NULL); + } + psFree(iter); + + pxNode *root = psHashLookup(forest, "root"); + psFree(forest); + + return psMemIncrRefCounter(root); +} + +psHash *pxTreeBuilder(psHash *forest, + const char *nodeName, + const char *childName, + psPtr data) +{ + // try to find a node with this name + pxNode *node = psHashLookup(forest, nodeName); + if (!node) { + // create a new node with this name + node = pxNodeAlloc(nodeName); + if (data) { + pxNodeAddData(node, data); + } + // add it to the hash of nodes + psHashAdd(forest, nodeName, node); + // node may be used below because it will still have a ref count of + // 1 from being on the hash, this is equivalent to the "view" we + // get from a hashlookup + psFree(node); + } else if (!node->data) { + pxNodeAddData(node, data); + } + + // does this node declare a child? + if (childName) { + // try to find a node with this name + pxNode *child = psHashLookup(forest, childName); + if (!child ) { + // create a new node with this name + child = pxNodeAlloc(childName); + // add it to the hash of nodes + psHashAdd(forest, childName, child); + // child may be used below because it will still have a ref + // count of 1 from being on the hash, this is equivalent to the + // "view" we get from a hashlookup + psFree(child); + } + pxNodeAddChild(node, child); + } + + return forest; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtree.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtree.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxtree.h (revision 41104) @@ -0,0 +1,72 @@ +/* + * pxtree.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXTREE_H +#define PXTREE_H 1 + +#include +#include + +#include + + +typedef struct pxNode { + char *name; + struct pxNode *parent; + psList *children; + void *data; +} pxNode; + +typedef bool (*pxNodeFunc)(void *arg, pxNode *node); + +pxNode *pxNodeAlloc( + const char *name +) PS_ATTR_MALLOC; + +pxNode *pxNodeAddParent(pxNode *node, pxNode *parent); +pxNode *pxNodeAddChild(pxNode *node, pxNode *child); +pxNode *pxNodeAddData(pxNode *node, psPtr data); + +void pxTreePrint(FILE *stream, pxNode *root); + +void pxNodePrint( + FILE *stream, + pxNode *node +); + +bool pxTreeCrawl( + pxNode *node, + pxNodeFunc func, + void *arg +); + +bool pxNodeFuncCountDependants(void *arg, pxNode *node); + +bool pxNodeHasChildren(pxNode *node); + +bool pxNodeHasGrandChildren(pxNode *node); + +pxNode *pxTreeFromMetadata(psMetadata *md); + +psHash *pxTreeBuilder(psHash *forest, + const char *nodeName, + const char *childName, + psPtr data); + +#endif // PXTREE_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxwarp.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxwarp.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxwarp.c (revision 41104) @@ -0,0 +1,75 @@ +/* + * pxwarp.c + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxwarp.h" + +bool pxwarpQueueByFakeID(pxConfig *config, + psS64 fake_id, + const char *workdir, + const char *label, + const char *data_group, + const char *dist_group, + const char *dvodb, + const char *tess_id, + const char *reduction, + const char *end_stage, + const char *note) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // depend on the f-keys to make sure we have a valid cam_id + if (!warpRunInsert(config->dbh, + 0, // ID + fake_id, + "warp", // mode + "new", // state + workdir, + "dirty", // workdir_state + label, + data_group, + dist_group, + dvodb, + tess_id, + reduction, + end_stage, + NULL, // registered + 0, // magicked set to zero when created may get updated when warpRun goes to 'full' + NULL, // version + 0, // maskfrac_npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + NULL // note + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return true; + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pxwarp.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pxwarp.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pxwarp.h (revision 41104) @@ -0,0 +1,39 @@ +/* + * pxwarp.h + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PXWARP_H +#define PXWARP_H 1 + +#include + +#include "pxtools.h" + +bool pxwarpQueueByFakeID(pxConfig *config, + psS64 fake_id, + const char *workdir, + const char *label, + const char *data_group, + const char *dist_group, + const char *dvodb, + const char *tess_id, + const char *reduction, + const char *end_stage, + const char *note); + +#endif // PXWARP_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexp.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexp.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexp.c (revision 41104) @@ -0,0 +1,369 @@ +/* + * pzgetexp.c + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pzgetexp.h" + +#define PRODUCT_LS_CMD "dsproductls" + +static bool go (pxConfig *config); +static psArray *parseFileSets(pxConfig *config, const char *str); + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pzgetexpConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + if (!go(config)) { + goto FAIL; + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool go(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // required + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + + //optional + PXOPT_LOOKUP_S32(timeout, config->args, "-timeout", false, false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + // find last fileset/exp_name (if we have one) + bool haveLastFileSet = false; + psString lastFileSet = NULL; + + // -all means "request all known filesets" + if (!all) { + char *query = "SELECT * from summitExp WHERE camera = \"%s\" and TELESCOPE = \"%s\" ORDER BY dateobs DESC LIMIT 1"; + if (!p_psDBRunQueryF(config->dbh, query, camera, telescope)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no summitExp rows found"); + haveLastFileSet = false; + psFree(output); + } else { + haveLastFileSet = true; + bool status = false; + lastFileSet = psStringCopy(psMetadataLookupStr(&status, output->data[0], "exp_name")); + psFree(output); + } + } + + // invoke dsproductls + // dsproductls --uri <> --last_fileset <> + psString cmd = NULL; + if (haveLastFileSet) { + psStringAppend(&cmd, "%s --uri %s --last_fileset %s", + PRODUCT_LS_CMD, uri, lastFileSet); + psFree(lastFileSet); + } else { + psStringAppend(&cmd, "%s --uri %s", PRODUCT_LS_CMD, uri); + } + if (timeout) { + psStringAppend(&cmd, " --timeout %d", timeout); + } + + psTrace("pzgetexp", PS_LOG_INFO, "cmd is: %s\n", cmd); + + FILE *output = popen(cmd, "r"); + psFree(cmd); + + if (!output) { + psError(PS_ERR_UNKNOWN, true, "popen() failed"); + return false; + } + psString cmdOutput = psSlurpFile(output); + pclose(output); + + psArray *newSummitExps = parseFileSets(config, cmdOutput); + psFree(cmdOutput); + if (!newSummitExps) { + // XXX not nessicarily an error + psError(PS_ERR_UNKNOWN, true, "no new fileSet/exp IDs"); + return false; + } + if (!psArrayLength(newSummitExps)) { + psTrace("pzgetexp", PS_LOG_INFO, "no new fileSet/exp IDs"); + psFree(newSummitExps); + return true; + } + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newSummitExps); + return false; + } + + // increase memory table size limits; deafult is 16MB + { + // 512MB + char *query = "SET max_heap_table_size = 1024*1024*512"; + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newSummitExps); + return false; + } + } + + // create a temporry table + { + char *query = "CREATE TEMPORARY TABLE incoming" + " (exp_name VARCHAR(64), camera VARCHAR(64), telescope VARCHAR(64), dateobs DATETIME, exp_type VARCHAR(64), uri VARCHAR(255), PRIMARY KEY(exp_name, camera, telescope))" + " ENGINE=MEMORY"; + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newSummitExps); + return false; + } + } + + { + char *query = "INSERT INTO incoming (exp_name, camera, telescope, dateobs, exp_type, uri) VALUES (?, ?, ?, ?, ?, ?)"; + + long inserted = p_psDBRunQueryPrepared(config->dbh, newSummitExps, query); + if (inserted < 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newSummitExps); + return false; + } + // sanity check that we actually inserted something + if (inserted == 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error -- we should have inserted at least one row"); + psFree(newSummitExps); + return false; + } + } + + psFree(newSummitExps); + + // add new exps to summitExp + { + char *query = + "INSERT INTO summitExp" + " SELECT" + " NULL," // summit_id + " incoming.*," + " NULL," // imfiles + " 0," // fault + " NULL" // epoch + " FROM incoming" + " LEFT JOIN summitExp" + " USING(exp_name, camera, telescope)" + " WHERE" + " summitExp.exp_name is NULL" + " AND summitExp.camera is NULL" + " AND summitExp.telescope is NULL"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static psArray *parseFileSets(pxConfig *config, const char *str) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(str, NULL); + + // these are constants for all records parsed -- look them up before we do + // any work + // required + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + + // split the string into lines + psList *doc = psStringSplit(str, "\n", false); + + psListIterator *lineCursor = psListIteratorAlloc(doc, 0, false); + + psArray *summitExps = psArrayAllocEmpty(psListLength(doc)); + psString line; + while ((line = psListGetAndIncrement(lineCursor))) { + psTrace("pzgetimfile", PS_LOG_INFO, "parsing line: %s\n", line); + + // split line into tokens + psList *tokens = psStringSplit(line, " ", false); + + // check to see if this line is a comment (or if the first token is + // NULL) + if (!psListGet(tokens, 0) || *((char *)psListGet(tokens, 0)) == '#') { + psFree(tokens); + continue; + } + + // check that we have the right number of tokens + // print "# uri fileset datetime type\n"; + if (psListLength(tokens) != 4) { + // error + return false; + } + + // find the values of interest + psListIterator *tokenCursor = psListIteratorAlloc(tokens, 0, false); + char *uri = psListGetAndIncrement(tokenCursor); + char *exp_name = psListGetAndIncrement(tokenCursor); // fileset + char *dateobsStr= psListGetAndIncrement(tokenCursor); // datetime + char *exp_type = psListGetAndIncrement(tokenCursor); // type + + // create a new metadata to represent this line and it's values + psMetadata *md = psMetadataAlloc(); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_name", 0, NULL, exp_name)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_name"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "camera", 0, NULL, camera)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "telescope", 0, NULL, telescope)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + psTime *dateobs = psTimeFromISO(dateobsStr, PS_TIME_UTC); + if (!psMetadataAddTime(md, PS_LIST_TAIL, "dateobs", 0, NULL, dateobs)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(dateobs); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + psFree(dateobs); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_type", 0, NULL, exp_type)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_type"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "uri", 0, NULL, uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item uri"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + + // must be freed after the new metadata is built -- holds the strings + psFree(tokenCursor); + psFree(tokens); + + // add the new metadata to the result set + psArrayAdd(summitExps, 0, md); + + // debugging + if (psTraceGetLevel("pzgetexp") == PS_LOG_INFO) { + psString doc = psMetadataConfigFormat(md); + psTrace("pzgetexp", PS_LOG_INFO, "parsed line as:\n %s\n", doc); + psFree(doc); + } + + psFree(md); + + } + + psFree(lineCursor); + psFree(doc); + + return summitExps; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexp.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexp.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexp.h (revision 41104) @@ -0,0 +1,27 @@ +/* + * pzgetexp.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PZGETEXP_H +#define PZGETEXP_H 1 + +#include "pxtools.h" + +pxConfig *pzgetexpConfig(pxConfig *config, int argc, char **argv); + +#endif // PZGETEXP_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexpConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexpConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetexpConfig.c (revision 41104) @@ -0,0 +1,88 @@ +/* + * pzgetexpConfig.c + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" + +pxConfig *pzgetexpConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration!\n"); + psFree(config); + return NULL; + } + + psMetadata *args = psMetadataAlloc(); + psMetadataAddStr(args , PS_LIST_TAIL, "-uri", 0, + "DataStore product URI (required)", ""); + psMetadataAddStr(args , PS_LIST_TAIL, "-inst", 0, + "camera name (required)", ""); + psMetadataAddStr(args , PS_LIST_TAIL, "-telescope", 0, + "telescope name (required)", ""); + psMetadataAddS32(args, PS_LIST_TAIL, "-timeout", 0, + "HTTP timeout", 0); + psMetadataAddBool(args, PS_LIST_TAIL, "-all", 0, + "download ALL filesets", 0); + + + bool status = false; + if (!psArgumentParse(args, &argc, argv) + || argc != 1 + || strcmp(psMetadataLookupStr(&status, args, "-uri"), "") == 0 + || strcmp(psMetadataLookupStr(&status, args, "-inst"), "") == 0 + || strcmp(psMetadataLookupStr(&status, args, "-telescope"), "") == 0 + ) { + fprintf(stderr, "error parsing arguments\n"); + printf("\nPan-STARRS Phase Z Get Exposures Tool\n"); + printf("Usage: %s -uri -inst -telescope [-timeout ]\n\n", + argv[0]); + psArgumentHelp(args); + psFree(config); + return NULL; + } + + config->args = args; + // don't free args here as it's silly to increment the ref count then + // "free" it + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if(!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't connect to db\n"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfiles.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfiles.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfiles.c (revision 41104) @@ -0,0 +1,505 @@ +/* + * pzgetimfiles.c + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtag.h" +#include "pxtools.h" +#include "pzgetimfiles.h" + +#define FILESET_LS_CMD "dsfilesetls" + +static bool go (pxConfig *config); +static psArray *parseFiles(pxConfig *config, const char *str); + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pzgetimfilesConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + if (!go(config)) { + goto FAIL; + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool go(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // required + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + PXOPT_LOOKUP_STR(filesetid, config->args, "-filesetid", true, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + + // optional + PXOPT_LOOKUP_S32(timeout, config->args, "-timeout", false, false); + + // invoke dsfilesetls + psString cmd = NULL; + psStringAppend(&cmd, "%s --uri %s", FILESET_LS_CMD, uri); + if (timeout) { + psStringAppend(&cmd, " --timeout %d", timeout); + } + + psTrace("pzgetimfiles", PS_LOG_INFO, "cmd is: %s\n", cmd); + + FILE *output = popen(cmd, "r"); + psFree(cmd); + + if (!output) { + psError(PS_ERR_UNKNOWN, true, "popen() failed"); + return false; + } + + psString cmdOutput = psSlurpFile(output); + int status = pclose(output); + + // We have an id value now, let's get it and use it + char *id_query = "SELECT summit_id FROM summitExp WHERE exp_name = '%s' AND camera = '%s' AND telescope = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, id_query, filesetid, camera, telescope)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *id_output = p_psDBFetchResult(config->dbh); + if (!id_output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (psArrayLength(id_output) != 1) { + psError(PS_ERR_UNKNOWN, false, "database error: incorrect number of results"); + return false; + } + + psMetadata *id_result = id_output->data[0]; + + psS64 summit_id = psMetadataLookupS64(NULL, id_result, "summit_id"); + psFree(id_output); + // End of summit_id block. + + if (status != 0) { + // mark the summitExp row as faulted + if (!p_psDBRunQueryF(config->dbh, "UPDATE summitExp SET fault = %d WHERE summit_id = %lld", WEXITSTATUS(status), (long long) summit_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + + psError(PS_ERR_UNKNOWN, true, "%s failed with exit status %d", + FILESET_LS_CMD, WEXITSTATUS(status)); + psFree(cmdOutput); + return false; + } + + // prase output of dsfilesetls + psArray *newImfiles = parseFiles(config, cmdOutput); + if (!newImfiles) { + // XXX not nessicarily an error but we don't want to keep trying to + // download an "empty" fileset. + // mark the summitExp row as faulted + if (!p_psDBRunQueryF(config->dbh, "UPDATE summitExp SET fault = %d WHERE summit_id = %lld", 250, (long long) summit_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, true, "no new files/imfiles"); + psFree(cmdOutput); + return false; + } + psFree(cmdOutput); + + // save the number of new Imfiles; + long imfiles = psArrayLength(newImfiles); + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // if the fileset was empty (no files) then we can bail out early + if (imfiles == 0) { + psFree(newImfiles); + + char *query = + "UPDATE summitExp" + " SET imfiles = %d" + " WHERE summit_id = %ld"; + if (!p_psDBRunQueryF(config->dbh, query, imfiles, summit_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // remove the pzDownloadExp entry for this exp (fileset) + { + char *query = + "DELETE FROM pzDownloadExp" + " WHERE" + " summit_id = %ld"; + if (!p_psDBRunQueryF(config->dbh, query, summit_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have removed only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; + } + + // create a temp table + { + char *query = + "CREATE TEMPORARY TABLE incoming (exp_name VARCHAR(64), camera VARCHAR(64), telescope VARCHAR(64), file_id VARCHAR(64), bytes INT, md5sum VARCHAR(32), class VARCHAR(64), class_id VARCHAR(64), uri VARCHAR(255), PRIMARY KEY(exp_name, camera, telescope, class, class_id)) ENGINE=MEMORY"; + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newImfiles); + return false; + } + } + + // load the imfiles (files) into the temp table + { + char *query = "INSERT INTO incoming (exp_name, camera, telescope, file_id, bytes, md5sum, class, class_id, uri) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"; + + long inserted = p_psDBRunQueryPrepared(config->dbh, newImfiles, query); + if (inserted < 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newImfiles); + return false; + } + // sanity check that we actually inserted something + if (inserted == 0) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error -- we should have inserted at least one row"); + psFree(newImfiles); + return false; + } + } + + psFree(newImfiles); + + // copy imfiles (files) from the temp table into summitImfiles + { + char *query = + "INSERT IGNORE INTO summitImfile" + " SELECT" + " %ld," // summit_id + " incoming.exp_name," + " incoming.camera," + " incoming.telescope," + " incoming.file_id," + " incoming.bytes," + " incoming.md5sum," + " incoming.class," + " incoming.class_id," + " incoming.uri," + " NULL" // epoch + " FROM incoming"; + if (!p_psDBRunQueryF(config->dbh, query,summit_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // update summitExp.imfiles (should have been NULL) + { + char *query = + "UPDATE summitExp" + " SET imfiles = (SELECT COUNT(*) FROM summitImfile" + " WHERE" + " summit_id = %ld" + ")" + " WHERE" + " summit_id = %ld" + " AND imfiles IS NULL"; + if (!p_psDBRunQueryF(config->dbh, query, summit_id, summit_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have updated only one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // add new exps to pzDownloadExp -- must be done before the new exps are + // added to summitExp because of the SQL logic + { + char *query = + "INSERT IGNORE INTO pzDownloadExp" + " SELECT" + " %ld," // summit_id + " incoming.exp_name," + " incoming.camera," + " incoming.telescope," + " \"run\"," // state + " NULL" + " FROM incoming" + " GROUP BY" + " incoming.exp_name," + " incoming.camera," + " incoming.telescope"; + + if (!p_psDBRunQueryF(config->dbh, query,summit_id)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static psArray *parseFiles(pxConfig *config, const char *str) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + PS_ASSERT_PTR_NON_NULL(str, NULL); + + // these are constants for all records parsed -- look them up before we do + // any work + PXOPT_LOOKUP_STR(exp_name, config->args, "-filesetid", true, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + + // split the string into lines + psList *doc = psStringSplit(str, "\n", false); + + psListIterator *lineCursor = psListIteratorAlloc(doc, 0, false); + + psArray *pzPendingImfiles = psArrayAllocEmpty(psListLength(doc)); + psString line; + while ((line = psListGetAndIncrement(lineCursor))) { + psTrace("pzgetimfiles", PS_LOG_INFO, "parsing line: %s\n", line); + + // split line into tokens + psList *tokens = psStringSplit(line, " ", false); + + // check to see if this line is a comment (or if the first token is + // NULL) + if (!psListGet(tokens, 0) || *((char *)psListGet(tokens, 0)) == '#') { + psFree(tokens); + continue; + } + + // check that we have the right number of tokens + // print "# uri fileid bytes md5sum type \n"; + if (psListLength(tokens) < 6) { + psError(PS_ERR_UNKNOWN, true, "invalid line format: %s", line); + psFree(tokens); + psFree(pzPendingImfiles); + psFree(lineCursor); + psFree(doc); + return false; + } + + // find the values of interest + psListIterator *tokenCursor = psListIteratorAlloc(tokens, 0, false); + char *uri = psListGetAndIncrement(tokenCursor); + char *file_id = psListGetAndIncrement(tokenCursor); // fileid + char *bytes = psListGetAndIncrement(tokenCursor); // bytes + char *md5sum = psListGetAndIncrement(tokenCursor); // md5sum + char *class = psListGetAndIncrement(tokenCursor); // type + char *class_id = psListGetAndIncrement(tokenCursor); // chipname + + // create a new metadata to represent this line and it's values + psMetadata *md = psMetadataAlloc(); + if (!psMetadataAddStr(md, PS_LIST_TAIL, "exp_name", 0, NULL, exp_name)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_name"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "camera", 0, NULL, camera)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item camera"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "telescope", 0, NULL, telescope)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item telescope"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "file_id", 0, NULL, file_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bytes"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddS64(md, PS_LIST_TAIL, "bytes", 0, NULL, (psS64)atoll(bytes))) { + psError(PS_ERR_UNKNOWN, false, "failed to add item bytes"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "md5sum", 0, NULL, md5sum)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item md5sum"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "class", 0, NULL, class)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "class_id", 0, NULL, class_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item class_id"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + if (!psMetadataAddStr(md, PS_LIST_TAIL, "uri", 0, NULL, uri)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item uri"); + psFree(md); + psFree(tokenCursor); + psFree(tokens); + return NULL; + } + + // must be freed after the new metadata is built -- holds the strings + psFree(tokenCursor); + psFree(tokens); + + // debugging + if (psTraceGetLevel("pzgetimfiles") >= PS_LOG_INFO) { + psString doc = psMetadataConfigFormat(md); + psTrace("pzgetimfiles", PS_LOG_INFO, "parsed line as:\n %s\n", doc); + psFree(doc); + } + + psArrayAdd(pzPendingImfiles, 0, md); + + psFree(md); + } + + psFree(lineCursor); + psFree(doc); + + if (!psArrayLength(pzPendingImfiles)) { + psError(PS_ERR_UNKNOWN, false, "string contained no valid rows"); + psFree(pzPendingImfiles); + return false; + } + + return pzPendingImfiles; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfiles.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfiles.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfiles.h (revision 41104) @@ -0,0 +1,27 @@ +/* + * pzgetimfiles.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PZGETIMFILES_H +#define PZGETIMFILES 1 + +#include "pxtools.h" + +pxConfig *pzgetimfilesConfig(pxConfig *config, int argc, char **argv); + +#endif // PZGETIMFILES_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfilesConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfilesConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pzgetimfilesConfig.c (revision 41104) @@ -0,0 +1,89 @@ +/* + * pzgetimfilesConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" + +pxConfig *pzgetimfilesConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(PS_ERR_UNKNOWN, false, "Can't find site configuration!\n"); + psFree(config); + return NULL; + } + + psMetadata *args = psMetadataAlloc(); + psMetadataAddStr(args , PS_LIST_TAIL, "-uri", 0, + "DataStore FileSet URI (required)", NULL); + psMetadataAddStr(args , PS_LIST_TAIL, "-filesetid", 0, + "FileSet ID (required)", NULL); + psMetadataAddStr(args, PS_LIST_TAIL, "-inst", 0, + "Camera ID (required)", NULL); + psMetadataAddStr(args, PS_LIST_TAIL, "-telescope", 0, + "Telescope ID (required)", NULL); + psMetadataAddS32(args, PS_LIST_TAIL, "-timeout", 0, + "HTTP timeout", 0); + + + bool status = false; + if (!psArgumentParse(args, &argc, argv) + || argc != 1 + || (psMetadataLookupStr(&status, args, "-uri") == NULL) + || (psMetadataLookupStr(&status, args, "-filesetid") == NULL) + || (psMetadataLookupStr(&status, args, "-inst") == NULL) + || (psMetadataLookupStr(&status, args, "-telescope") == NULL) + ) { + fprintf(stderr, "error parsing arguments\n"); + printf("\nPan-STARRS Phase Z Get Imfiles Tool\n"); + printf("Usage: %s -uri -filesetid -inst -telescope ]\n\n", argv[0]); + psArgumentHelp(args); + psFree(args); + psFree(config); + return NULL; + } + + config->args = args; + // don't free args here as it's silly to increment the ref count then + // "free" it + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if(!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't connect to db\n"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pztool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pztool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pztool.c (revision 41104) @@ -0,0 +1,1002 @@ +/* + * pztool.c + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pslib.h" +#include "pxtools.h" +#include "pxdata.h" +#include "pztool.h" + +static bool adddatastoreMode(pxConfig *config); +static bool datastoreMode(pxConfig *config); +static bool seenMode(pxConfig *config); +static bool pendingExpMode(pxConfig *config); +static bool pendingImfileMode(pxConfig *config); + +static bool copydoneMode(pxConfig *config); +static bool copiedMode(pxConfig *config); +static bool updatecopiedMode(pxConfig *config); +static bool revertcopiedMode(pxConfig *config); + +static bool clearcommonfaultsMode(pxConfig *config); +static bool toadvanceMode(pxConfig *config); +static bool advanceMode(pxConfig *config); + +static bool updatepzexpMode(pxConfig *config); +static bool updatenewexpMode(pxConfig *config); + +// static bool copydoneCompleteExp(pxConfig *config); +static psArray *pzGetPendingCameras(pxConfig *config); +static psArray *pzArrayZip(psArray *arraySet, psS64 limit); +static bool pzDownloadExpSetState(pxConfig *config, const psS64 summit_id, const char *state); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = pztoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(PZTOOL_MODE_ADDDATASTORE, adddatastoreMode); + MODECASE(PZTOOL_MODE_DATASTORE, datastoreMode); + MODECASE(PZTOOL_MODE_SEEN, seenMode); + MODECASE(PZTOOL_MODE_PENDINGEXP, pendingExpMode); + MODECASE(PZTOOL_MODE_PENDINGIMFILE, pendingImfileMode); + MODECASE(PZTOOL_MODE_COPYDONE, copydoneMode); + MODECASE(PZTOOL_MODE_COPIED, copiedMode); + MODECASE(PZTOOL_MODE_UPDATECOPIED, updatecopiedMode); + MODECASE(PZTOOL_MODE_REVERTCOPIED, revertcopiedMode); + MODECASE(PZTOOL_MODE_CLEARCOMMONFAULTS, clearcommonfaultsMode); + MODECASE(PZTOOL_MODE_TOADVANCE, toadvanceMode); + MODECASE(PZTOOL_MODE_ADVANCE, advanceMode); + MODECASE(PZTOOL_MODE_UPDATEPZEXP, updatepzexpMode); + MODECASE(PZTOOL_MODE_UPDATENEWEXP, updatenewexpMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool adddatastoreMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + + if (!pzDataStoreInsert(config->dbh, + camera, + telescope, + uri, + NULL, // epoch + 0 // use_compress + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool datastoreMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + if (!p_psDBRunQuery(config->dbh, "SELECT * FROM pzDataStore")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pzDataStore", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool seenMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM summitExp"); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "summitExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "summitExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingExpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + psMetadata *where2 = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-summit_id", "summit_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + + PXOPT_COPY_TIME(config->args, where2, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where2, "-dateobs_end", "dateobs", "<="); + + PXOPT_LOOKUP_BOOL(desc, config->args, "-desc", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // XXX leave this query here ? + psString query = psStringCopy( + "SELECT" + " summitExp.*" + " FROM summitExp" + " LEFT JOIN pzDownloadExp" + " USING(summit_id)" + " WHERE" + " pzDownloadExp.summit_id IS NULL" + " AND summitExp.fault = 0" + ); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "pzDownloadExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (psListLength(where2->list)) { + psString where2Clause = psDBGenerateWhereConditionSQL(where2, "summitExp"); + psStringAppend(&query, " AND %s", where2Clause); + psFree(where2Clause); + } + psFree(where2); + + psStringAppend(&query, " ORDER BY summitExp.dateobs"); + if (desc) { + psStringAppend(&query, " DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pzDownloadExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingImfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-summit_id", "summit_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "dateobs", "<="); + + PXOPT_LOOKUP_BOOL(desc, config->args, "-desc", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psArray *cameras = pzGetPendingCameras(config); + if (!cameras) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to find any cameras"); + return false; + } + + // array to hold the aggregate results + psArray *cameraImfiles = psArrayAlloc(0); + + for (long i = 0; i < psArrayLength(cameras); i++) { + psString query = pxDataGet("pztool_pendingimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(cameraImfiles); + return false; + } + + bool status; + psString camera = psMetadataLookupStr(&status, cameras->data[i], "camera"); + psStringAppend(&query, " WHERE camera = \"%s\"", camera); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (0) { + psStringAppend(&query, " ORDER BY dateobs"); + if (desc) { + psStringAppend(&query, " DESC"); + } + } + + // request the full "limit" from each known camera and throw away any + // "extra" rows that we may have after merging the results. This is + // a lot simplier than a complicated scheme (tried that) to attempt to + // request on the right number of rows for each camera + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(cameraImfiles); + return false; + } + psFree(query); + + psArray *result = p_psDBFetchResult(config->dbh); + if (!result) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(cameraImfiles); + return false; + } + if (!psArrayLength(result)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(result); + continue; + } + + // add this query into the array of result set + psArrayAdd(cameraImfiles, 0, result); + psFree(result); + } + psFree(where); + + // stitch the arrays of imfiles together + psArray *output = pzArrayZip(cameraImfiles, limit); + psFree(cameraImfiles); + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pzDownloadImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool copydoneMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(summit_id, config->args, "-summit_id", true, false); + PXOPT_LOOKUP_STR(exp_name, config->args, "-exp_name", true, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_STR(class, config->args, "-class", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_BOOL(row_lock, config->args, "-row_lock", false); + PXOPT_LOOKUP_S32(bytes, config->args, "-bytes", false, false); + PXOPT_LOOKUP_STR(md5sum, config->args, "-md5sum", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + // start a transaction early so it will contain any row level locks + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // query to get an excluse lock on this exposure in + // pzDownloadExp + psString lock_query = NULL; + if (row_lock) { + lock_query = psStringCopy("SELECT * FROM pzDownloadExp"); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-summit_id", "summit_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&lock_query, " %s FOR UPDATE", whereClause); + psFree(whereClause); + } + psFree(where); + + // aquire a lock on the pzDownloadExp record + // lock persists until the transaction is committed + if (!p_psDBRunQuery(config->dbh, lock_query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(lock_query); + return false; + } + psFree(lock_query); + + // we must fetch the result set from aquiring the row level lock or + // MySQL will barf all over us. + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(output); + } + + if (!pzDownloadImfileInsert(config->dbh, + summit_id, + exp_name, + camera, + telescope, + class, + class_id, + uri, + fault, + NULL, // epoch + hostname, + bytes, + md5sum + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool toadvanceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // optional args + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // find all exposures that have had all of their imfiles downloaded but do + // not appear in newExp + psString query = pxDataGet("pztool_find_completed_exp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psMetadata *where = psMetadataAlloc(); +/* PXOPT_COPY_STR(config->args, where, "-summit_id", "summit_id", "=="); */ + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + // find completed exps + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "toadvance", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool advanceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(summit_id, config->args, "-summit_id", true, false); + PXOPT_LOOKUP_STR(exp_name, config->args, "-exp_name", true, false); + PXOPT_LOOKUP_STR(inst, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); + + // optional + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-end_stage", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + + // start a transaction so it's all rows or nothing + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!newExpInsert(config->dbh, + 0x0, // exp_id + summit_id, // summit_id + exp_name, // tmp_exp_name + inst, // tmp_camera + telescope, // tmp_telescope + "run", // state + workdir, // workdir + "dirty", // workdir state + NULL, // reduction class + dvodb, // dvodb + tess_id, // tess_id + end_stage, // end_stage + label, // label + NULL // epoch + ) + ) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psS64 exp_id = psDBLastInsertID(config->dbh); + + // insert newImfiles + { + char *query = + "INSERT INTO newImfile" + " SELECT" + " %" PRId64 "," // exp_id + " pzDownloadImfile.class_id," // tmp_class_id + " pzDownloadImfile.uri," // uri + " NULL," // epoch + " pzDownloadImfile.bytes," // bytes + " pzDownloadImfile.md5sum" // md5sum + " FROM pzDownloadImfile" + " WHERE" + " pzDownloadImfile.exp_name = '%s'" + " AND pzDownloadImfile.camera = '%s'" + " AND pzDownloadImfile.telescope = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, exp_id, exp_name, inst, telescope)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // sanity check: we should have inserted at least one row + psU64 affected = psDBAffectedRows(config->dbh); + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected at least 1 row but %" PRIu64 " rows were modified", affected); + return false; + } + } + + // set pzDownloadExp.state to 'stop' + if (!pzDownloadExpSetState(config, summit_id, "stop")) { + psError(PS_ERR_UNKNOWN, false, "failed to change pzDownloadExp.state for %lld", (long long) summit_id); + return false; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + return true; +} + +static psArray *pzGetPendingCameras(pxConfig *config) +{ + // get a list of cameras we've seen exps for + if (!p_psDBRunQuery(config->dbh, "SELECT DISTINCT camera FROM pzDownloadExp")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psArray *cameras = p_psDBFetchResult(config->dbh); + if (!cameras) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return NULL; + } + if (!psArrayLength(cameras)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(cameras); + return psArrayAlloc(0); + } + + return cameras; +} + +static psArray *pzArrayZip(psArray *arraySet, psS64 limit) +{ + // figure out the combined size of all arrays in the set + long setSize = 0; + for (long i = 0; i < psArrayLength(arraySet); i++) { + setSize += psArrayLength(arraySet->data[i]); + } + + // treat 0 as "no limit" + if (limit == 0) { + limit = setSize; + } + + psArray *output = psArrayAllocEmpty(limit); + // loop over each array in the set forever + for ( + // init + long counter = 0, // the total number of elements zipped so far + i = 0, // which array in the set + index = 0; // the depth into each array + // test + (counter < setSize) + && (counter < limit) + && (i < psArrayLength(arraySet)); + // incr + counter++, ++i, + i = i % psArrayLength(arraySet), + i % psArrayLength(arraySet) ? : ++index + ) { + + psArray *array = arraySet->data[i]; + // make sure that this array has not run out of elements + if (!(index < psArrayLength(array))) { + continue; + } + + psArrayAdd(output, 0, array->data[index]); + } + + return output; +} + + +static bool copiedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "inst", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-class", "class", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + psString query = psStringCopy("SELECT * FROM pzDownloadImfile"); + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "WHERE pzDownloadImfile.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "WHERE pzDownloadImfile.fault = 0"); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "pzDownloadImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("pztool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipPendingImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool updatecopiedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-class", "class", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + + if (!pxSetFaultCode(config->dbh, "pzDownloadImfile", where, fault, 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree(where); + + return true; +} + + +static bool revertcopiedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-class", "class", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString query = pxDataGet("pztool_revertcopied.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "pzDownloadImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + return true; +} + + +static bool clearcommonfaultsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + +{ + psString query = pxDataGet("pztool_revert_downloadimfile_faults.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); +} + +{ + psString query = pxDataGet("pztool_revert_fileset_faults.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); +} + + return true; +} + + +static bool pzDownloadExpSetState(pxConfig *config, const psS64 summit_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!( + (strncmp(state, "run", 4) == 0) + || (strncmp(state, "stop", 5) == 0) + || (strncmp(state, "reg", 4) == 0) + || (strncmp(state, "drop", 5) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid pzDownloadExp state: %s", state); + return false; + } + + char *query = "UPDATE pzDownloadExp SET state = '%s' WHERE summit_id = %ld"; + if (!p_psDBRunQueryF(config->dbh, query, state, summit_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for %lld", (long long) summit_id); + return false; + } + + return true; +} + +static bool updatepzexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_S64(summit_id, config->args, "-summit_id", true, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state",true, false); + + if (!pzDownloadExpSetState(config,summit_id, state)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for %lld", (long long) summit_id); + return false; + } + return true; +} + + +static bool updatenewexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "tmp_exp_name", "=="); + + PXOPT_LOOKUP_STR(new_state, config->args, "-set_state", true, false); + + if (strcmp(new_state, "drop") && strcmp(new_state, "run") && strcmp(new_state, "wait")) { + psError(PXTOOLS_ERR_ARGUMENTS, true, "%s is not a valid value for -set_state", new_state); + psFree(where); + return false; + } + + if (psListLength(where->list) < 1) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "-exp_name or -exp_id is required"); + return false; + } + + psString query = NULL; + psStringAppend(&query, "UPDATE newExp SET state = '%s'", new_state); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + + + + +#if 0 +static psArray *pzArrayAddArray(psArray *array, psArray *input) +{ + for (long i = 0; i < psArrayLength(input); i++) { + psArrayAdd(array, psArrayLength(input), input->data[i]); + } + + return array; +} +#endif Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pztool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pztool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pztool.h (revision 41104) @@ -0,0 +1,45 @@ +/* + * pztool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef PZTOOL_H +#define PZTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + PZTOOL_MODE_NONE = 0x0, + PZTOOL_MODE_ADDDATASTORE, + PZTOOL_MODE_DATASTORE, + PZTOOL_MODE_SEEN, + PZTOOL_MODE_PENDINGEXP, + PZTOOL_MODE_PENDINGIMFILE, + PZTOOL_MODE_COPYDONE, + PZTOOL_MODE_COPIED, + PZTOOL_MODE_UPDATECOPIED, + PZTOOL_MODE_REVERTCOPIED, + PZTOOL_MODE_CLEARCOMMONFAULTS, + PZTOOL_MODE_TOADVANCE, + PZTOOL_MODE_ADVANCE, + PZTOOL_MODE_UPDATEPZEXP, + PZTOOL_MODE_UPDATENEWEXP, +} pztoolMode; + +pxConfig *pztoolConfig(pxConfig *config, int argc, char **argv); + +#endif // PZTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/pztoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/pztoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/pztoolConfig.c (revision 41104) @@ -0,0 +1,220 @@ +/* + * pztoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "pztool.h" + +pxConfig *pztoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration!\n"); + psFree(config); + return NULL; + } + + // -adddatastore + psMetadata *adddatastoreArgs = psMetadataAlloc(); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID", NULL); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID", NULL); + psMetadataAddStr(adddatastoreArgs, PS_LIST_TAIL, "-uri", 0, "define storage uri", NULL); + + // -datastore + psMetadata *datastoreArgs = psMetadataAlloc(); + psMetadataAddBool(datastoreArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -seen + psMetadata *seenArgs = psMetadataAlloc(); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID", NULL); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID", NULL); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID", NULL); + psMetadataAddStr(seenArgs, PS_LIST_TAIL, "-exp_type", 0, "define exposure type", NULL); + psMetadataAddBool(seenArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddS64(pendingexpArgs, PS_LIST_TAIL, "-summit_id", 0, "define summit_id", 0); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID", NULL); + psMetadataAddTime(pendingexpArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(pendingexpArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(pendingexpArgs, PS_LIST_TAIL, "-exp_type", 0, "define exposure type", NULL); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-desc", 0, "sort ouput in descending format", false); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingimfileArgs, PS_LIST_TAIL, "-summit_id", 0, "define summit_id", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID", NULL); + psMetadataAddTime(pendingimfileArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(pendingimfileArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-exp_type", 0, "define exposure type", NULL); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-desc", 0, "sort ouput in descending format", false); + psMetadataAddU64(pendingimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -copydone + psMetadata *copydoneArgs = psMetadataAlloc(); + psMetadataAddS64(copydoneArgs, PS_LIST_TAIL, "-summit_id", 0, "define summit_id (required)", 0); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID (required)", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID (required)", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID (required)", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-class", 0, "define class", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-class_id", 0, "define class_id", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-uri", 0, "define storage uri", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-workdir", 0, "define the \"default\" workdir for this exposure", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-dvodb", 0, "define the dvodb for the next processing step", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-tess_id", 0, "define the tess_id for the next processing step", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-end_stage", 0, "define the end goal processing step", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-label", 0, "define the label for the chip stage", NULL); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-hostname", 0, "define the host that copied the image", NULL); + psMetadataAddS32(copydoneArgs, PS_LIST_TAIL, "-bytes", 0, "define the size in bytes for the copied image", 0); + psMetadataAddStr(copydoneArgs, PS_LIST_TAIL, "-md5sum", 0, "define the md5 sum for the copied image", NULL); + psMetadataAddS16(copydoneArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddBool(copydoneArgs, PS_LIST_TAIL, "-row_lock", 0, "lock pzDownImfile rows while advancing an exposure", false); + // XXX: remove this once advance is fixed + psMetadataAddU64(copydoneArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -copied + psMetadata *copiedArgs = psMetadataAlloc(); + psMetadataAddStr(copiedArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID", NULL); + psMetadataAddStr(copiedArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID", NULL); + psMetadataAddStr(copiedArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID", NULL); + psMetadataAddStr(copiedArgs, PS_LIST_TAIL, "-class", 0, "define class", NULL); + psMetadataAddStr(copiedArgs, PS_LIST_TAIL, "-class_id", 0, "define class_id", NULL); + psMetadataAddBool(copiedArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddU64(copiedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(copiedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updatecopied + psMetadata *updatecopiedArgs = psMetadataAlloc(); + psMetadataAddStr(updatecopiedArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID (required)", NULL); + psMetadataAddStr(updatecopiedArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID (required)", NULL); + psMetadataAddStr(updatecopiedArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID (required)", NULL); + psMetadataAddStr(updatecopiedArgs, PS_LIST_TAIL, "-class", 0, "define class", NULL); + psMetadataAddStr(updatecopiedArgs, PS_LIST_TAIL, "-class_id", 0, "define class_id", NULL); + psMetadataAddS16(updatecopiedArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -revertcopied + psMetadata *revertcopiedArgs = psMetadataAlloc(); + psMetadataAddStr(revertcopiedArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID (required)", NULL); + psMetadataAddStr(revertcopiedArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID (required)", NULL); + psMetadataAddStr(revertcopiedArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID (required)", NULL); + psMetadataAddStr(revertcopiedArgs, PS_LIST_TAIL, "-class", 0, "define class", NULL); + psMetadataAddStr(revertcopiedArgs, PS_LIST_TAIL, "-class_id", 0, "define class_id", NULL); + psMetadataAddS16(revertcopiedArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -clearcommonfaults + psMetadata *clearcommonfaultsArgs = psMetadataAlloc(); + // + // -toadvance + psMetadata *toadvanceArgs = psMetadataAlloc(); + psMetadataAddS64(toadvanceArgs, PS_LIST_TAIL, "-summit_id", 0, "define summit_id", 0); + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID", NULL); + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID", NULL); + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID", NULL); + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-label", 0, "define the label for the chip stage", NULL); + psMetadataAddBool(toadvanceArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(toadvanceArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -advance + psMetadata *advanceArgs = psMetadataAlloc(); + psMetadataAddS64(advanceArgs, PS_LIST_TAIL, "-summit_id", 0, "define summit_id", 0); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID (required)", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-inst", 0, "define camera ID (required)", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-telescope", 0, "define telescope ID (required)", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-workdir", 0, "define the \"default\" workdir for this exposure (required)", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-dvodb", 0, "define the dvodb for the next processing step", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-tess_id", 0, "define the tess_id for the next processing step", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-end_stage", 0, "define the end goal processing step", NULL); + psMetadataAddStr(advanceArgs, PS_LIST_TAIL, "-label", 0, "define the label for the chip stage", NULL); + + // -updatepzexp + psMetadata *updatepzexpArgs = psMetadataAlloc(); + psMetadataAddS64(updatepzexpArgs, PS_LIST_TAIL, "-summit_id", 0, "define summit_id", 0); + psMetadataAddStr(updatepzexpArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure name (required)", NULL); + psMetadataAddStr(updatepzexpArgs, PS_LIST_TAIL, "-inst", 0, "search by camera (required)", NULL); + psMetadataAddStr(updatepzexpArgs, PS_LIST_TAIL, "-telescope", 0, "search by telescope (required)", NULL); + psMetadataAddStr(updatepzexpArgs, PS_LIST_TAIL, "-set_state", 0, "define new state (required)", NULL); + + // -updatenewexp + psMetadata *updatenewexpArgs = psMetadataAlloc(); + psMetadataAddS64(updatenewexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + + psMetadataAddStr(updatenewexpArgs, PS_LIST_TAIL, "-exp_name", 0, "define exposure ID", NULL); + psMetadataAddStr(updatenewexpArgs, PS_LIST_TAIL, "-set_state", 0, "define new state (required)", NULL); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-adddatastore", "", PZTOOL_MODE_ADDDATASTORE, adddatastoreArgs); + PXOPT_ADD_MODE("-datastore", "", PZTOOL_MODE_DATASTORE, datastoreArgs); + PXOPT_ADD_MODE("-seen", "", PZTOOL_MODE_SEEN, seenArgs); + PXOPT_ADD_MODE("-pendingexp", "", PZTOOL_MODE_PENDINGEXP, pendingexpArgs); + PXOPT_ADD_MODE("-pendingimfile", "", PZTOOL_MODE_PENDINGIMFILE,pendingimfileArgs); + PXOPT_ADD_MODE("-copydone", "", PZTOOL_MODE_COPYDONE, copydoneArgs); + PXOPT_ADD_MODE("-copied", "", PZTOOL_MODE_COPIED, copiedArgs); + PXOPT_ADD_MODE("-updatecopied", "", PZTOOL_MODE_UPDATECOPIED,updatecopiedArgs); + PXOPT_ADD_MODE("-revertcopied", "", PZTOOL_MODE_REVERTCOPIED,revertcopiedArgs); + PXOPT_ADD_MODE("-clearcommonfaults","", PZTOOL_MODE_CLEARCOMMONFAULTS,clearcommonfaultsArgs); + PXOPT_ADD_MODE("-toadvance", "", PZTOOL_MODE_TOADVANCE, toadvanceArgs); + PXOPT_ADD_MODE("-advance", "", PZTOOL_MODE_ADVANCE, advanceArgs); + PXOPT_ADD_MODE("-updatepzexp", "", PZTOOL_MODE_UPDATEPZEXP, updatepzexpArgs); + PXOPT_ADD_MODE("-updatenewexp", "", PZTOOL_MODE_UPDATENEWEXP,updatenewexpArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetool.c (revision 41104) @@ -0,0 +1,644 @@ +/* + * receivetool.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "receivetool.h" + +static bool definesourceMode(pxConfig *config); +static bool listMode(pxConfig *config); +static bool addfilesetMode(pxConfig *config); +static bool updatelastMode(pxConfig *config); +static bool pendingfilesetMode(pxConfig *config); +static bool updatefilesetMode(pxConfig *config); +static bool addfileMode(pxConfig *config); +static bool pendingfileMode(pxConfig *config); +static bool addresultMode(pxConfig *config); +static bool toadvanceMode(pxConfig *config); +static bool revertMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = receivetoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(RECEIVETOOL_MODE_DEFINESOURCE, definesourceMode); + MODECASE(RECEIVETOOL_MODE_LIST, listMode); + MODECASE(RECEIVETOOL_MODE_ADDFILESET, addfilesetMode); + MODECASE(RECEIVETOOL_MODE_UPDATELAST, updatelastMode); + MODECASE(RECEIVETOOL_MODE_PENDINGFILESET, pendingfilesetMode); + MODECASE(RECEIVETOOL_MODE_UPDATEFILESET, updatefilesetMode); + MODECASE(RECEIVETOOL_MODE_TOADVANCE, toadvanceMode); + MODECASE(RECEIVETOOL_MODE_ADDFILE, addfileMode); + MODECASE(RECEIVETOOL_MODE_PENDINGFILE, pendingfileMode); + MODECASE(RECEIVETOOL_MODE_ADDRESULT, addresultMode); + MODECASE(RECEIVETOOL_MODE_REVERT, revertMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool definesourceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_STR(source, config->args, "-source", true, false); + PXOPT_LOOKUP_STR(product, config->args, "-product", true, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", false, false); + + // optional + PXOPT_LOOKUP_STR(comment, config->args, "-comment", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + PXOPT_LOOKUP_STR(last, config->args, "-last", false, false); + PXOPT_LOOKUP_STR(status_product, config->args, "-status_product", false, false); + PXOPT_LOOKUP_STR(ds_dbname, config->args, "-ds_dbname", false, false); + PXOPT_LOOKUP_STR(ds_dbhost, config->args, "-ds_dbhost", false, false); + + if (!receiveSourceInsert(config->dbh, 0, source, product, workdir, state, comment, last, status_product, ds_dbname, ds_dbhost)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + +static bool listMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + + // required + PXOPT_COPY_STR(config->args, where, "-source", "receiveSource.source", "=="); + PXOPT_COPY_STR(config->args, where, "-product", "receiveSource.product", "=="); + PXOPT_COPY_S64(config->args, where, "-comment", "receiveSource.comment", "LIKE"); + + // optional + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("receivetool_list.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("receivetool", PS_LOG_INFO, "No rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "receiveSource", !simple)) { + psError(PS_ERR_UNKNOWN, false, "Failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool addfilesetMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(source_id, config->args, "-src_id", true, false); + psMetadataItem *filesets = psMetadataLookup(config->args, "-fileset"); + if (!filesets) { + psError(PS_ERR_UNKNOWN, true, "-fileset is required"); + return false; + } + + psString query = pxDataGet("receivetool_addfileset.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + psString source_id_str = NULL; // source_id as a string + psStringAppend(&source_id_str, "%" PRId64, source_id); + + psListIterator *iter = psListIteratorAlloc(filesets->data.list, PS_LIST_HEAD, false); // Iterator + psMetadataItem *item = NULL; // Item from iteration + while ((item = psListGetAndIncrement(iter))) { + psAssert(item && item->data.V && item->type == PS_DATA_STRING, "Argument is bad"); + const char *fileset = item->data.str; // Fileset name + + if (!p_psDBRunQueryF(config->dbh, query, source_id_str, fileset)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(source_id_str); + psFree(query); + psFree(iter); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + + psArray *output = p_psDBFetchResult(config->dbh); // Output of query + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(source_id_str); + psFree(query); + psFree(iter); + return false; + } + if (psArrayLength(output) > 0) { + psTrace("receivetool", PS_LOG_INFO, "Fileset %s is already present", fileset); + psFree(output); + continue; + } + psFree(output); + + if (!receiveFilesetInsert(config->dbh, 0, source_id, fileset, "reg", NULL, NULL, 0)) { + psError(PS_ERR_UNKNOWN, false, "Unable to add fileset"); + psFree(source_id_str); + psFree(query); + psFree(iter); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + } + return false; + } + } + psFree(iter); + psFree(query); + psFree(source_id_str); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + +static bool updatelastMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(source_id, config->args, "-src_id", true, false); + PXOPT_LOOKUP_STR(fileset, config->args, "-fileset", true, false); + + psString query = NULL; // Query to execute + psStringAppend(&query, "UPDATE receiveSource SET fileset_last = \'%s\' WHERE source_id = %" PRId64, + fileset, source_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + + +static bool pendingfilesetMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + + // required + PXOPT_COPY_STR(config->args, where, "-source", "receiveSource.source", "=="); + PXOPT_COPY_STR(config->args, where, "-product", "receiveSource.product", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "receiveSource.comment", "LIKE"); + + // optional + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("receivetool_pendingfileset.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // make sure that the data is processed in the order that the filesets + // were posted + psStringAppend(&query, "\nORDER BY fileset_id"); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("receivetool", PS_LOG_INFO, "No rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "receiveFileset", !simple)) { + psError(PS_ERR_UNKNOWN, false, "Failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool addfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(fileset_id, config->args, "-fileset_id", true, false); + PXOPT_LOOKUP_STR(file_list, config->args, "-file_list", true, false); + + unsigned int numBad; + psMetadata *files = psMetadataConfigRead(NULL, &numBad, file_list, false); + if (!files) { + psError(PS_ERR_BAD_PARAMETER_VALUE, false, "Unable to cleanly read MDC file with file list."); + return false; + } + + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + psMetadataIterator *iter = psMetadataIteratorAlloc(files, PS_LIST_HEAD, NULL); // Iterator + psMetadataItem *item; + while ((item = psMetadataGetAndIncrement(iter))) { + psMetadata *md = item->data.md; + psString file = psMetadataLookupStr(NULL, md, "file"); + psS64 bytes = psMetadataLookupS64(NULL, md, "bytes"); + psString md5sum = psMetadataLookupStr(NULL, md, "md5sum"); + psString file_type = psMetadataLookupStr(NULL, md, "file_type"); + psString component = psMetadataLookupStr(NULL, md, "component"); + + if (!file) { + psError(PS_ERR_UNKNOWN, false, "failed to find value for file"); + return false; + } + if (!component) { + psError(PS_ERR_UNKNOWN, false, "failed to find value for component"); + return false; + } + + if (!receiveFileInsert(config->dbh, 0, fileset_id, file, bytes, md5sum, file_type, component)) { + psError(PS_ERR_UNKNOWN, false, "Unable to add file"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + psFree(iter); + return false; + } + } + + psFree(iter); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + +static bool pendingfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + + // required + PXOPT_COPY_STR(config->args, where, "-source", "receiveSource.source", "=="); + PXOPT_COPY_STR(config->args, where, "-product", "receiveSource.product", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "receiveSource.comment", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-fileset_id", "receiveFile.fileset_id", "=="); + PXOPT_COPY_S64(config->args, where, "-file_id", "receiveFile.file_id", "=="); + + // optional + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("receivetool_pendingfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // make sure that the data is processed in the order that the filesets + // were posted + psStringAppend(&query, "\nORDER BY fileset_id"); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("receivetool", PS_LOG_INFO, "No rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "receiveFile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "Failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool addresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(file_id, config->args, "-file_id", true, false); + + // optional + PXOPT_LOOKUP_F32(dtime_copy, config->args, "-dtime_copy", false, false); + PXOPT_LOOKUP_F32(dtime_extract, config->args, "-dtime_extract", false, false); + PXOPT_LOOKUP_S32(fault, config->args, "-fault", false, false); + + if (!receiveResultInsert(config->dbh, file_id, dtime_copy, dtime_extract, fault)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + + return true; +} + +static bool revertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + + PXOPT_COPY_S64(config->args, where, "-fileset_id", "receiveResult.fileset_id", "=="); + PXOPT_COPY_S32(config->args, where, "-fault", "receiveResult.fault", "=="); + PXOPT_COPY_STR(config->args, where, "-source", "receiveSource.source", "=="); + PXOPT_COPY_STR(config->args, where, "-product", "receiveSource.product", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "receiveSource.comment", "LIKE"); + + psString query = pxDataGet("receivetool_revert.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} +static bool toadvanceMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); // WHERE conditions + + // optional + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("receivetool_toadvance.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("receivetool", PS_LOG_INFO, "No rows found"); + psFree(output); + return true; + } + if (!ippdbPrintMetadatas(stdout, output, "toadvanceFilesets", !simple)) { + psError(PS_ERR_UNKNOWN, false, "Failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool updatefilesetMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(fileset_id, config->args, "-fileset_id", true, false); + + // to chanage + PXOPT_LOOKUP_S32(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_STR(destdir, config->args, "-destdir", false, false); + PXOPT_LOOKUP_STR(dirinfo, config->args, "-dirinfo", false, false); + PXOPT_LOOKUP_STR(dbinfo, config->args, "-dbinfo", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + + if (!fault && !dirinfo &&!dbinfo && !state) { + psError(PS_ERR_UNKNOWN, true, "at least one of -fault, -dirinfo, -dbinfo, -set_state are required"); + return false; + } + + psString query = NULL; // Query to execute + psStringAppend(&query, "UPDATE receiveFileset SET "); + + psString sep = ""; + if (fault) { + psStringAppend(&query, "%s fault = %d", sep, fault); + sep = ","; + } + if (dirinfo) { + psStringAppend(&query, "%s dirinfo = '%s'", sep, dirinfo); + sep = ","; + } + if (dbinfo) { + psStringAppend(&query, "%s dbinfo = '%s'", sep, dbinfo); + sep = ","; + } + if (state) { + psStringAppend(&query, "%s state = '%s'", sep, state); + sep = ","; + } + + psStringAppend(&query, " WHERE fileset_id = %" PRId64, fileset_id); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetool.h (revision 41104) @@ -0,0 +1,42 @@ +/* + * receivetool.h + * + * Copyright (C) 2008 IfA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef RECEIVETOOL_H +#define RECEIVETOOL_H 1 + +#include "pxtools.h" + +typedef enum { + RECEIVETOOL_MODE_NONE = 0x0, + RECEIVETOOL_MODE_DEFINESOURCE, + RECEIVETOOL_MODE_LIST, + RECEIVETOOL_MODE_ADDFILESET, + RECEIVETOOL_MODE_UPDATELAST, + RECEIVETOOL_MODE_PENDINGFILESET, + RECEIVETOOL_MODE_UPDATEFILESET, + RECEIVETOOL_MODE_TOADVANCE, + RECEIVETOOL_MODE_ADDFILE, + RECEIVETOOL_MODE_PENDINGFILE, + RECEIVETOOL_MODE_ADDRESULT, + RECEIVETOOL_MODE_REVERT, +} receivetoolMode; + +pxConfig *receivetoolConfig(pxConfig *config, int argc, char **argv); + +#endif // RECEIVETOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/receivetoolConfig.c (revision 41104) @@ -0,0 +1,171 @@ +/* + * receivetoolConfig.c + * + * Copyright (C) 2008 + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "receivetool.h" + +pxConfig *receivetoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration!"); + psFree(config); + return NULL; + } + + // -definesource + psMetadata *definesourceArgs = psMetadataAlloc(); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-source", 0, "define source (required)", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-product", 0, "define product (required)", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-comment", 0, "define comment", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-state", 0, "define state", "enabled"); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-last", 0, "define last fileset", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-status_product", 0, "define status_product", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-ds_dbname", 0, "define ds_dbname", NULL); + psMetadataAddStr(definesourceArgs, PS_LIST_TAIL, "-ds_dbhost", 0, "define ds_dbhost", NULL); + + // -list + psMetadata *listArgs = psMetadataAlloc(); + psMetadataAddStr(listArgs, PS_LIST_TAIL, "-source", 0, "search on source", NULL); + psMetadataAddStr(listArgs, PS_LIST_TAIL, "-product", 0, "search on product", NULL); + psMetadataAddStr(listArgs, PS_LIST_TAIL, "-comment", 0, "search on comment (LIKE)", NULL); + psMetadataAddU64(listArgs, PS_LIST_TAIL, "-limit", 0, "limit result set", 0); + psMetadataAddBool(listArgs, PS_LIST_TAIL, "-simple", 0, "use simple output format?", false); + + // -addfileset + psMetadata *addfilesetArgs = psMetadataAlloc(); + psMetadataAddS64(addfilesetArgs, PS_LIST_TAIL, "-src_id", 0, "define source_id (required)", 0); + psMetadataAddStr(addfilesetArgs, PS_LIST_TAIL, "-fileset", PS_META_DUPLICATE_OK, "define fileset (multiple OK, required)", NULL); + + // -updatelast + psMetadata *updatelastArgs = psMetadataAlloc(); + psMetadataAddS64(updatelastArgs, PS_LIST_TAIL, "-src_id", 0, "define source_id (required)", 0); + psMetadataAddStr(updatelastArgs, PS_LIST_TAIL, "-fileset", 0, "define last fileset (required)", NULL); + + // -pendingfileset + psMetadata *pendingfilesetArgs = psMetadataAlloc(); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-source", 0, "search on source", NULL); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-product", 0, "search on product", NULL); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-comment", 0, "search on comment (LIKE)", NULL); + psMetadataAddBool(pendingfilesetArgs, PS_LIST_TAIL, "-simple", 0, "use simple output format?", false); + psMetadataAddU64(pendingfilesetArgs, PS_LIST_TAIL, "-limit", 0, "limit result set", 0); + psMetadataAddStr(pendingfilesetArgs, PS_LIST_TAIL, "-label", 0, "search on label", NULL); + + // -updatefileset + psMetadata *updatefilesetArgs = psMetadataAlloc(); + psMetadataAddS64(updatefilesetArgs, PS_LIST_TAIL, "-fileset_id", 0, "define fileset_id (required)", 0); + psMetadataAddStr(updatefilesetArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + psMetadataAddStr(updatefilesetArgs, PS_LIST_TAIL, "-destdir", 0, "define destdir", NULL); + psMetadataAddStr(updatefilesetArgs, PS_LIST_TAIL, "-dirinfo", 0, "define dirinfo", NULL); + psMetadataAddStr(updatefilesetArgs, PS_LIST_TAIL, "-dbinfo", 0, "define dbinfo", NULL); + psMetadataAddS32(updatefilesetArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -addfile + psMetadata *addfileArgs = psMetadataAlloc(); + psMetadataAddS64(addfileArgs, PS_LIST_TAIL, "-fileset_id", 0, "define fileset_id (required)", 0); + psMetadataAddStr(addfileArgs, PS_LIST_TAIL, "-file_list", 0, "define file list (required)", NULL); + + // -pendingfile + psMetadata *pendingfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingfileArgs, PS_LIST_TAIL, "-source", 0, "search on source", NULL); + psMetadataAddStr(pendingfileArgs, PS_LIST_TAIL, "-product", 0, "search on product", NULL); + psMetadataAddStr(pendingfileArgs, PS_LIST_TAIL, "-comment", 0, "search on comment (LIKE)", NULL); + psMetadataAddS64(pendingfileArgs, PS_LIST_TAIL, "-fileset_id", 0, "search on fileset_id", 0); + psMetadataAddS64(pendingfileArgs, PS_LIST_TAIL, "-file_id", 0, "search on file_id", 0); + psMetadataAddU64(pendingfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set", 0); + psMetadataAddBool(pendingfileArgs, PS_LIST_TAIL, "-simple", 0, "use simple output format?", false); + psMetadataAddStr(pendingfileArgs, PS_LIST_TAIL, "-label", 0, "search on label", NULL); + + // -addresult + psMetadata *addresultArgs = psMetadataAlloc(); + psMetadataAddS64(addresultArgs, PS_LIST_TAIL, "-file_id", 0, "define receive_id (required)", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-path_base", 0, "path_base for component", NULL); + psMetadataAddF32(addresultArgs, PS_LIST_TAIL, "-dtime_copy", 0, "define time to copy", NAN); + psMetadataAddF32(addresultArgs, PS_LIST_TAIL, "-dtime_extract", 0, "define time to extract", NAN); + psMetadataAddS32(addresultArgs, PS_LIST_TAIL, "-fault", 0, "define fault code", 0); + + // -revert + psMetadata *revertArgs = psMetadataAlloc(); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-file_id", 0, "search on file_id", 0); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-fileset_id", 0, "search on fileset_id", 0); + psMetadataAddS32(revertArgs, PS_LIST_TAIL, "-fault", 0, "search on fault code", 0); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-source", 0, "search on source", NULL); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-product", 0, "search on product", NULL); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-comment", 0, "search on comment (LIKE)", NULL); + + // -toadvance + psMetadata *toadvanceArgs = psMetadataAlloc(); + psMetadataAddS64(toadvanceArgs, PS_LIST_TAIL, "-fileset_id", 0, "search on fileset_id", 0); + psMetadataAddS64(toadvanceArgs, PS_LIST_TAIL, "-source_id", 0, "search on source_id", 0); + psMetadataAddU64(toadvanceArgs, PS_LIST_TAIL, "-limit", 0, "limit result set", 0); + psMetadataAddBool(toadvanceArgs, PS_LIST_TAIL, "-simple", 0, "use simple output format?", false); + // label is not used but pantasks requires it + psMetadataAddStr(toadvanceArgs, PS_LIST_TAIL, "-label", 0, "search on label", NULL); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definesource", "", RECEIVETOOL_MODE_DEFINESOURCE, definesourceArgs); + PXOPT_ADD_MODE("-list", "", RECEIVETOOL_MODE_LIST, listArgs); + PXOPT_ADD_MODE("-addfileset", "", RECEIVETOOL_MODE_ADDFILESET, addfilesetArgs); + PXOPT_ADD_MODE("-updatelast", "", RECEIVETOOL_MODE_UPDATELAST, updatelastArgs); + PXOPT_ADD_MODE("-updatefileset", "", RECEIVETOOL_MODE_UPDATEFILESET, updatefilesetArgs); + PXOPT_ADD_MODE("-pendingfileset", "", RECEIVETOOL_MODE_PENDINGFILESET, pendingfilesetArgs); + PXOPT_ADD_MODE("-toadvance", "", RECEIVETOOL_MODE_TOADVANCE, toadvanceArgs); + PXOPT_ADD_MODE("-addfile", "", RECEIVETOOL_MODE_ADDFILE, addfileArgs); + PXOPT_ADD_MODE("-pendingfile", "", RECEIVETOOL_MODE_PENDINGFILE, pendingfileArgs); + PXOPT_ADD_MODE("-addresult", "", RECEIVETOOL_MODE_ADDRESULT, addresultArgs); + PXOPT_ADD_MODE("-revert", "", RECEIVETOOL_MODE_REVERT, revertArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define database handle, if used + // do this last so we don't setup a connection before CLI options are validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/regtool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/regtool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/regtool.c (revision 41104) @@ -0,0 +1,2164 @@ +/* + * regtool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "regtool.h" +#include "chiptool.h" + +// imfile +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedimfileMode(pxConfig *config); +static bool checkburntoolimfileMode(pxConfig *config); +static bool pendingburntoolimfileMode(pxConfig *config); +static bool processedimfileMode(pxConfig *config); +static bool revertprocessedimfileMode(pxConfig *config); +static bool updateprocessedimfileMode(pxConfig *config); +static bool pendingcompressimfileMode(pxConfig *config); +// exp +static bool pendingexpMode(pxConfig *config); +static bool addprocessedexpMode(pxConfig *config); +static bool processedexpMode(pxConfig *config); +static bool revertprocessedexpMode(pxConfig *config); +static bool updateprocessedexpMode(pxConfig *config); +static bool cleardupexpMode(pxConfig *config); +static bool finishcompressexpMode(pxConfig *config); + +static bool updatebyqueryMode(pxConfig *config); + +static bool checkstatusMode(pxConfig *config); + +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = regtoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + // imfile + MODECASE(REGTOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(REGTOOL_MODE_CHECKBURNTOOLIMFILE, checkburntoolimfileMode); + MODECASE(REGTOOL_MODE_PENDINGBURNTOOLIMFILE, pendingburntoolimfileMode); + MODECASE(REGTOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileMode); + MODECASE(REGTOOL_MODE_PROCESSEDIMFILE, processedimfileMode); + MODECASE(REGTOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileMode); + MODECASE(REGTOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileMode); + // exp + MODECASE(REGTOOL_MODE_PENDINGEXP, pendingexpMode); + MODECASE(REGTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpMode); + MODECASE(REGTOOL_MODE_PROCESSEDEXP, processedexpMode); + MODECASE(REGTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpMode); + MODECASE(REGTOOL_MODE_UPDATEPROCESSEDEXP, updateprocessedexpMode); + MODECASE(REGTOOL_MODE_UPDATEBYQUERY, updatebyqueryMode); + MODECASE(REGTOOL_MODE_PENDINGCOMPRESSIMFILE, pendingcompressimfileMode); + MODECASE(REGTOOL_MODE_FINISHCOMPRESSEXP, finishcompressexpMode); + MODECASE(REGTOOL_MODE_CLEARDUPEXP, cleardupexpMode); + MODECASE(REGTOOL_MODE_CHECKSTATUS, checkstatusMode); + MODECASE(REGTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(REGTOOL_MODE_IMPORTRUN, importrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + psTrace("regtool",9,"Attempting to free config\n"); + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // select newImfiles that: + // exp_id is in newExp + // don't have their exp_id in rawExp + // XXX having the same exp_id in newExp and raw*Exp is probably an error + // that should be checked for + + psString query = pxDataGet("regtool_pendingimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "regPendingImfile", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + + + +static bool checkburntoolimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config,false); + + // required + PXOPT_LOOKUP_STR(exp_name, config->args, "-exp_name", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(dateobs_begin, config->args, "-dateobs_begin", true, false); + PXOPT_LOOKUP_STR(dateobs_end, config->args, "-dateobs_end", true, false); + PXOPT_LOOKUP_S32(valid_burntool, config->args, "-valid_burntool", true, false); + // optional + PXOPT_LOOKUP_STR(camera, config->args, "-inst", false, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + psString query = pxDataGet("regtool_checkburntoolimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + psString rep = psStringCopy(query); + psFree(query); + query = rep; + + // convert regular class_id format to summitImfile.class_id format + rep = psStringCopy(class_id); + + class_id = rep; + + psStringSubstitute(&class_id,"ota","XY"); + + psStringSubstitute(&query,exp_name,"@EXP_NAME@"); + psStringSubstitute(&query,class_id,"@CLASS_ID@"); + psStringSubstitute(&query,dateobs_begin,"@DATEOBS_BEGIN@"); + psStringSubstitute(&query,dateobs_end,"@DATEOBS_END@"); + + // fprintf(stderr,"%s",query); + + if (!p_psDBRunQuery(config->dbh, query)) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + return true; + } + + psString previous_uri = NULL; + bool ok_to_burn = true; + bool already_burned = true; + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; + + if (strcasecmp(psMetadataLookupStr(NULL,row,"download_state"),"drop") == 0) { + continue; + } + char *is_ccim = strchr(psMetadataLookupStr(NULL,row,"exp_name"),'c'); + if (is_ccim) { // Is a camera commanded exposure + // fprintf(stderr,"IN camera commanded!\n"); + if ((strcasecmp(psMetadataLookupStr(NULL,row,"exp_type"),"DOMEFLAT") == 0)|| + (strcasecmp(psMetadataLookupStr(NULL,row,"exp_type"),"DARK") == 0) || + (strcasecmp(psMetadataLookupStr(NULL,row,"exp_type"),"BIAS") == 0)) { + continue; + } + } + char *is_stareim = strchr(psMetadataLookupStr(NULL,row,"exp_name"),'a'); + if (is_stareim) { + continue; + } + + if ((psMetadataLookupS32(NULL,row,"is_downloaded") != 1)|| + (psMetadataLookupS32(NULL,row,"is_registered") != 1)) { + ok_to_burn = false; + } + if (already_burned == false) { + ok_to_burn = false; + } + if (abs(psMetadataLookupS32(NULL,row,"burntool_state")) == valid_burntool) { + already_burned = true; + } + else { + already_burned = false; + } + + if (previous_uri) { + psMetadataAddStr(row,PS_LIST_TAIL,"previous_uri",PS_META_REPLACE,"",previous_uri); + psFree(previous_uri); + } + psMetadataAddBool(row,PS_LIST_TAIL,"burnable",PS_META_REPLACE,"",ok_to_burn); + psMetadataAddBool(row,PS_LIST_TAIL,"already_burned",PS_META_REPLACE,"",already_burned); + previous_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "regBurntoolImfile", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + + return(true); + +} + + +// CZW: 2013-12-11 A bad exposure caused this code to fail to return any rows. +// I don't clearly see any logic flaws, and so I fixed the issue by +// retricting dateobs_begin to exclude this exposure. Problem for +// another day. +static bool pendingburntoolimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config,false); + + // required + PXOPT_LOOKUP_STR(dateobs_begin, config->args, "-dateobs_begin", true, false); + PXOPT_LOOKUP_STR(dateobs_end, config->args, "-dateobs_end", true, false); + PXOPT_LOOKUP_S32(valid_burntool, config->args, "-valid_burntool", true, false); + // optional + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(ignore_state, config->args, "-ignore_state", false); + psString query = pxDataGet("regtool_pendingburntoolimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + psString rep = psStringCopy(query); + psFree(query); + query = rep; + + // This matches against summitExp.dateobs + psStringSubstitute(&query,dateobs_begin,"@DATEOBS_BEGIN@"); + psStringSubstitute(&query,dateobs_end,"@DATEOBS_END@"); + + // fprintf(stderr,"%s",query); + + if (!p_psDBRunQuery(config->dbh, query)) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + + // fprintf (stderr, "found %ld rows\n", output->n); + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + psString previous_uri = NULL; + psString this_uri = NULL; + bool ok_to_burn = true; + bool already_burned = true; + psString previous_class_id = NULL; + psString this_class_id = NULL; + psArray *results = psArrayAllocEmpty(60); // List of suggested imfiles to burntool. + + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; + + if (strcasecmp(psMetadataLookupStr(NULL,row,"download_state"),"drop") == 0) { + continue; + } + + char *is_ccim = strchr(psMetadataLookupStr(NULL,row,"exp_name"),'c'); + if (is_ccim) { // Is a camera commanded exposure + // fprintf(stderr,"IN camera commanded!\n"); + if ((strcasecmp(psMetadataLookupStr(NULL,row,"exp_type"),"DOMEFLAT") == 0)|| + (strcasecmp(psMetadataLookupStr(NULL,row,"exp_type"),"DARK") == 0) || + (strcasecmp(psMetadataLookupStr(NULL,row,"exp_type"),"BIAS") == 0)) { + continue; + } + } + char *is_stareim = strchr(psMetadataLookupStr(NULL,row,"exp_name"),'a'); + if (is_stareim) { + continue; + } + + + bool status = false; + char *tmp_id = psMetadataLookupStr(&status,row,"summit_class_id"); + if (!status) { + fprintf (stderr, "incomplete on %s\n", psMetadataLookupStr(NULL,row,"exp_name")); + continue; + } + + // Add the information about this row and the previous, if it exists. + // Write the class_id stuff for debugging. + // this_class_id = psStringCopy(psMetadataLookupStr(NULL,row,"tmp_class_id")); + this_class_id = psStringCopy(tmp_id); + + psMetadataAddStr(row,PS_LIST_TAIL,"this_class_id",PS_META_REPLACE,"",this_class_id); + + if (previous_class_id) { + psMetadataAddStr(row,PS_LIST_TAIL,"previous_class_id",PS_META_REPLACE,"",previous_class_id); + } + // class_id = NULL sorts to the top of the list, so skip those (incomplete downloads) + if (!this_class_id) { + continue; + } + + + if (0 && !strcmp(this_class_id, "ota44")) { + printf("STAT 1: %s (%d %d) %d %d %d\n", + this_class_id, + ok_to_burn, already_burned, + psMetadataLookupS32(NULL,row,"burntool_state"), + psMetadataLookupS32(NULL,row,"is_registered"), + psMetadataLookupS32(NULL,row,"is_downloaded")); + } + + // Determine if we've crossed a class_id boundary, as this resets the bits. + if (previous_class_id) { + if (strcmp(this_class_id,previous_class_id) != 0) { + ok_to_burn = true; + already_burned = true; + previous_uri = NULL; + // fprintf (stderr, "crossed boundary: %s : %s to %s\n", psMetadataLookupStr(NULL,row,"exp_name"), previous_class_id, this_class_id); + } + } + + // Write the URIs as well. + this_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); // Duplicate, but helpful for my debugging. + psMetadataAddStr(row,PS_LIST_TAIL,"this_uri",PS_META_REPLACE,"",this_uri); + if (previous_uri) { + psMetadataAddStr(row,PS_LIST_TAIL,"previous_uri",PS_META_REPLACE,"",previous_uri); + } + + // Check the two status variables + // Convert bits of the SQL query into booleans describing the data state + if ((psMetadataLookupS32(NULL,row,"is_downloaded") != 1)|| + (psMetadataLookupS32(NULL,row,"is_registered") != 1)) { + // printf("I claim this isn't downloaded or registered? %s %s\n",this_uri,this_class_id); + ok_to_burn = false; + } + if (already_burned == false) { + // printf("already_burned looks false %s %s\n",this_uri,this_class_id); + ok_to_burn = false; + } + if (abs(psMetadataLookupS32(NULL,row,"burntool_state")) == valid_burntool) { + already_burned = true; + } + else { + already_burned = false; + } + psMetadataAddBool(row,PS_LIST_TAIL,"burnable",PS_META_REPLACE,"",ok_to_burn); + psMetadataAddBool(row,PS_LIST_TAIL,"already_burned",PS_META_REPLACE,"",already_burned); + + // Check the uri for this exposure + if (!this_uri) { + ok_to_burn = false; + already_burned = false; + + fprintf (stderr, "missing uri: %s %s\n", psMetadataLookupStr(NULL,row,"exp_name"), this_class_id); + + // Save this round for next round. + psFree(previous_class_id); + psFree(previous_uri); + previous_class_id = psStringCopy(psMetadataLookupStr(NULL,row,"summit_class_id")); + psStringSubstitute(&previous_class_id,"ota","XY"); + previous_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); // Save for next round. + continue; + } + + if (0 && !strcmp(this_class_id, "ota44")) { + printf("STATUS: %s %s %s %s (%d %d) %d %d %d\n",this_uri,previous_uri,this_class_id,previous_class_id,ok_to_burn,already_burned,psMetadataLookupS32(NULL,row,"burntool_state"),psMetadataLookupS32(NULL,row,"is_registered"),psMetadataLookupS32(NULL,row,"is_downloaded")); + } + // If the state of this imfile is not "pending_burntool" then we can't burn it. + if (!ignore_state) { + psString imfile_state = psMetadataLookupStr(NULL,row,"imfile_state"); + + if (!imfile_state) { // imfile state is NULL, so we probably aren't registered. + ok_to_burn = false; + + // fprintf (stderr, "missing imfile_state: %s %s\n", psMetadataLookupStr(NULL,row,"exp_name"), this_class_id); + + // Save this round for next round. + psFree(previous_class_id); + psFree(previous_uri); + previous_class_id = psStringCopy(psMetadataLookupStr(NULL,row,"summit_class_id")); + psStringSubstitute(&previous_class_id,"ota","XY"); + previous_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); // Save for next round. + continue; + } + if (strcmp("pending_burntool",imfile_state) != 0) { // Probably the state is full, do not twiddle states + + if (0 && strcmp(imfile_state, "full")) { + fprintf (stderr, "not pending nor full: %s %s %s\n", psMetadataLookupStr(NULL,row,"exp_name"), this_class_id, imfile_state); + } + + // Save this round for next round. + psFree(previous_class_id); + psFree(previous_uri); + previous_class_id = psStringCopy(psMetadataLookupStr(NULL,row,"summit_class_id")); + psStringSubstitute(&previous_class_id,"ota","XY"); + previous_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); // Save for next round. + continue; + } + } + + // Determine if we've already suggested an entry for this ota, and if not, copy this + // suggestion into our output result list. + if (!ok_to_burn || already_burned) { + // Save this round for next round. + psFree(previous_class_id); + psFree(previous_uri); + previous_class_id = psStringCopy(psMetadataLookupStr(NULL,row,"summit_class_id")); + psStringSubstitute(&previous_class_id,"ota","XY"); + previous_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); // Save for next round. + continue; + } + // If we're here, then we think we could potentially burntool this file. + psArrayAdd(results,results->n,row); + + // Save this round for next round. + psFree(previous_class_id); + psFree(previous_uri); + previous_class_id = psStringCopy(psMetadataLookupStr(NULL,row,"summit_class_id")); + psStringSubstitute(&previous_class_id,"ota","XY"); + previous_uri = psStringCopy(psMetadataLookupStr(NULL,row,"uri")); // Save for next round. + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, results, "regPendingBurntoolImfile", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + psFree(results); + return false; + } + + psFree(output); + psFree(results); + + + return(true); + +} + + + +static bool addprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(exp_name, config->args, "-exp_name", true, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_STR(tmp_class_id, config->args, "-tmp_class_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(uri, config->args, "-uri", true, false); + + // optional + PXOPT_LOOKUP_STR(exp_type, config->args, "-exp_type", false, false); + PXOPT_LOOKUP_STR(filelevel, config->args, "-filelevel", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", false, false); + PXOPT_LOOKUP_STR(comment, config->args, "-comment", false, false); + PXOPT_LOOKUP_STR(obs_mode, config->args, "-obs_mode", false, false); + PXOPT_LOOKUP_STR(obs_group, config->args, "-obs_group", false, false); + PXOPT_LOOKUP_STR(data_state, config->args, "-data_state", false, false); + + PXOPT_LOOKUP_F32(airmass, config->args, "-airmass", false, false); + PXOPT_LOOKUP_F64(ra, config->args, "-ra", false, false); + PXOPT_LOOKUP_F64(decl, config->args, "-decl", false, false); + PXOPT_LOOKUP_F32(exp_time, config->args, "-exp_time", false, false); + PXOPT_LOOKUP_F32(sat_pixel_frac, config->args, "-sat_pixel_frac", false, false); + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(alt, config->args, "-alt", false, false); + PXOPT_LOOKUP_F64(az, config->args, "-az", false, false); + PXOPT_LOOKUP_F32(ccd_temp, config->args, "-ccd_temp", false, false); + PXOPT_LOOKUP_F64(posang, config->args, "-posang", false, false); + PXOPT_LOOKUP_F32(m1_x, config->args, "-m1_x", false, false); + PXOPT_LOOKUP_F32(m1_y, config->args, "-m1_y", false, false); + PXOPT_LOOKUP_F32(m1_z, config->args, "-m1_z", false, false); + PXOPT_LOOKUP_F32(m1_tip, config->args, "-m1_tip", false, false); + PXOPT_LOOKUP_F32(m1_tilt, config->args, "-m1_tilt", false, false); + PXOPT_LOOKUP_F32(m2_x, config->args, "-m2_x", false, false); + PXOPT_LOOKUP_F32(m2_y, config->args, "-m2_y", false, false); + PXOPT_LOOKUP_F32(m2_z, config->args, "-m2_z", false, false); + PXOPT_LOOKUP_F32(m2_tip, config->args, "-m2_tip", false, false); + PXOPT_LOOKUP_F32(m2_tilt, config->args, "-m2_tilt", false, false); + + PXOPT_LOOKUP_F32(env_temp, config->args, "-env_temperature", false, false); + PXOPT_LOOKUP_F32(env_humid, config->args, "-env_humidity", false, false); + PXOPT_LOOKUP_F32(env_wind, config->args, "-env_wind_speed", false, false); + PXOPT_LOOKUP_F32(env_dir, config->args, "-env_wind_dir", false, false); + PXOPT_LOOKUP_F32(teltemp_m1, config->args, "-teltemp_m1", false, false); + PXOPT_LOOKUP_F32(teltemp_m1cell, config->args, "-teltemp_m1cell", false, false); + PXOPT_LOOKUP_F32(teltemp_m2, config->args, "-teltemp_m2", false, false); + PXOPT_LOOKUP_F32(teltemp_spider, config->args, "-teltemp_spider", false, false); + PXOPT_LOOKUP_F32(teltemp_truss, config->args, "-teltemp_truss", false, false); + PXOPT_LOOKUP_F32(teltemp_extra, config->args, "-teltemp_extra", false, false); + PXOPT_LOOKUP_F32(pon_time, config->args, "-pon_time", false, false); + // PXOPT_LOOKUP_S16(burntool_state, config->args, "-burntool_state", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(object, config->args, "-object", false, false); + PXOPT_LOOKUP_F32(sun_angle, config->args, "-sun_angle", false, false); + PXOPT_LOOKUP_F32(sun_alt, config->args, "-sun_alt", false, false); + PXOPT_LOOKUP_F32(moon_angle, config->args, "-moon_angle", false, false); + PXOPT_LOOKUP_F32(moon_alt, config->args, "-moon_alt", false, false); + PXOPT_LOOKUP_F32(moon_phase, config->args, "-moon_phase", false, false); + PXOPT_LOOKUP_BOOL(ignored, config->args, "-ignore", false); + PXOPT_LOOKUP_TIME(dateobs, config->args, "-dateobs", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_S32(bytes, config->args, "-bytes", false, false); + PXOPT_LOOKUP_STR(md5sum, config->args, "-md5sum", false, false); + PXOPT_LOOKUP_BOOL(video_cells, config->args, "-video_cells", false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + if (!rawImfileInsert( + config->dbh, + exp_id, + exp_name, + camera, + telescope, + dateobs, + tmp_class_id, + class_id, + uri, + data_state, + exp_type, + filelevel, + filter, + comment, + obs_mode, + obs_group, + airmass, + ra, + decl, + exp_time, + sat_pixel_frac, + bg, + bg_stdev, + bg_mean_stdev, + alt, + az, + ccd_temp, + posang, + m1_x, + m1_y, + m1_z, + m1_tip, + m1_tilt, + m2_x, + m2_y, + m2_z, + m2_tip, + m2_tilt, + env_temp, + env_humid, + env_wind, + env_dir, + teltemp_m1, + teltemp_m1cell, + teltemp_m2, + teltemp_spider, + teltemp_truss, + teltemp_extra, + pon_time, + user_1, + user_2, + user_3, + user_4, + user_5, + object, + sun_angle, + sun_alt, + moon_angle, + moon_alt, + moon_phase, + ignored, + hostname, + fault, + quality, + NULL, + 0, + bytes, + md5sum, + 0, // burntool_state + video_cells + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool processedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "obs_mode", "=="); + PXOPT_COPY_STR(config->args, where, "-data_state", "data_state", "=="); + PXOPT_LOOKUP_BOOL(video_cells, config->args, "-video_cells", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + PXOPT_LOOKUP_BOOL(allfiles, config->args, "-allfiles", false); + if (allfiles) { + faulted = false; + } + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_BOOL(ordered_by_date, config->args, "-ordered_by_date", false); + + // build where string for some parameters that don't fit PXOPT_COPY* + psString where2 = NULL; + if (!pxmagicAddWhere(config, &where2, "rawImfile")) { + psError(psErrorCodeLast(), false, "pxmagicAddWhere failed"); + return false; + } + if (!pxspaceAddWhere(config, &where2, "rawImfile")) { + psError(psErrorCodeLast(), false, "pxspaceAddWhere failed"); + return false; + } + + psString query = pxDataGet("regtool_processedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else if (!all && !where2) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psFree(where); + if (where2) { + psStringAppend(&query, " %s", where2); + psFree(where2); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND rawImfile.fault != 0"); + } else if (!allfiles) { + // don't list faulted rows + psStringAppend(&query, " %s", "AND rawImfile.fault = 0"); + } + if (video_cells) { + psStringAppend(&query, " %s", "AND rawImfile.video_cells"); + } + + // add the ORDER BY statement if desired + if (ordered_by_date) { + psStringAppend(&query, " ORDER BY dateobs"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool revertprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tmp_class_id", "tmp_class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id_begin", "exp_id", ">="); + PXOPT_COPY_S64(config->args, where, "-exp_id_end", "exp_id", "<="); + + psString query = pxDataGet("regtool_revertprocessedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawImfile"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("regtool", PS_LOG_INFO, "Updated %d rawImfile", numUpdated); + + return true; +} + + +static bool updateprocessedimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(burntool_state, config->args, "-burntool_state", false, false); + PXOPT_LOOKUP_S32(set_bytes, config->args, "-set_bytes", false, false); + PXOPT_LOOKUP_STR(set_md5sum, config->args, "-set_md5sum", false, false); + PXOPT_LOOKUP_STR(set_state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_BOOL(set_ignored, config->args, "-set_ignored", false); + PXOPT_LOOKUP_BOOL(clear_ignored, config->args, "-clear_ignored", false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + + if ((fault == INT16_MAX) && (burntool_state == INT16_MAX) && !set_state && !set_ignored && !clear_ignored) { + psError(PS_ERR_UNKNOWN, false, "at least one of -fault or -burntool_state -set_ignored -clear_ignored or -set_state must be selected"); + return false; + } + if ((fault != INT16_MAX) && (burntool_state != INT16_MAX)) { + psError(PS_ERR_UNKNOWN, false, "only one of -fault or -burntool_state must be selected"); + return false; + } + if (set_ignored && clear_ignored) { + psError(PS_ERR_UNKNOWN, true, "only one of -set_ignored or -clear_ignored may be selected"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + if (fault != INT16_MAX) { + // this is fairly dangerous : can set all if the where is not set... but since exp_id and class_id + // were required above this is safe + if (!pxSetFaultCode(config->dbh, "rawImfile", where, fault, 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + return(true); + } + psFree (where); + + psString setvalues = NULL; + if (burntool_state != INT16_MAX) { + psStringAppend(&setvalues,"rawImfile.burntool_state = %d",burntool_state); + } + if (set_bytes != INT32_MAX) { + if (setvalues) { + psStringAppend(&setvalues,","); + } + psStringAppend(&setvalues,"rawImfile.bytes = %d",set_bytes); + } + if (set_md5sum) { + if (setvalues) { + psStringAppend(&setvalues,","); + } + psStringAppend(&setvalues,"rawImfile.md5sum = '%s'",set_md5sum); + } + if (set_state) { + if (setvalues) { + psStringAppend(&setvalues,","); + } + psStringAppend(&setvalues,"rawImfile.data_state = '%s'",set_state); + } + if (set_ignored) { + if (setvalues) { + psStringAppend(&setvalues,","); + } + psStringAppend(&setvalues,"rawImfile.ignored = 1"); + } + if (clear_ignored) { + if (setvalues) { + psStringAppend(&setvalues,","); + } + psStringAppend(&setvalues,"rawImfile.ignored = 0"); + } + if (hostname) { + if (setvalues) { + psStringAppend(&setvalues,","); + } + psStringAppend(&setvalues,"rawImfile.hostname = '%s'", hostname); + } + psString query = pxDataGet("regtool_updateprocessedimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // printf(query,setvalues,exp_id,class_id); + if (!p_psDBRunQueryF(config->dbh, query, setvalues, exp_id,class_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(setvalues); + return false; + } + psFree(setvalues); + psFree(query); + + return true; +} + +static bool pendingcompressimfileMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + if (whereClause) { + psStringPrepend(&whereClause, "\n AND "); + } + PXOPT_LOOKUP_BOOL(compress, config->args, "-compress", false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + if ((compress && clean) || (!compress && !clean)) { + psStringAppend(&whereClause, "\n AND ((data_state = 'goto_compressed' AND state = 'goto_compressed')\n OR (data_state = 'goto_lossy' AND state = 'goto_lossy')) "); + } + else if (compress) { + psStringAppend(&whereClause, "\n AND ((data_state = 'goto_compressed' AND state = 'goto_compressed')) "); + } + else if (clean) { + psStringAppend(&whereClause, "\n AND ((data_state = 'goto_lossy' AND state = 'goto_lossy')) "); + } + else { + psStringAppend(&whereClause, "\n AND ((data_state = 'goto_compressed' AND state = 'goto_compressed')\n OR (data_state = 'goto_lossy' AND state = 'goto_lossy')) "); + } + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("regtool_pendingcompressimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + psString limitString = NULL; + if (limit) { + limitString = psDBGenerateLimitSQL(limit); + psStringPrepend(&limitString, "\n"); + } + // printf(query,whereClause,limitString); + if (!p_psDBRunQueryF(config->dbh, query, whereClause, limitString ? limitString : "")) { + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(limitString); + psFree(query); + psFree(whereClause); + psFree(where); + return(false); + } + psFree(limitString); + psFree(query); + psFree(whereClause); + psFree(where); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "regPendingCompressImfile", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool pendingexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // return only exps that: + // are not in rawExp + // have ALL of their imfiles in rawImfile (by count) + // and have no associated imfiles left in newImfile + psString query = pxDataGet("regtool_pendingexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // treat limit == 0 as "no limit" + psString limitString = NULL; + if (limit) { + limitString = psDBGenerateLimitSQL(limit); + // skip past the "hook" comment + psStringPrepend(&limitString, "\n"); + } + + // 1st arg: where hook, 2nd arg: limit hook + if (!p_psDBRunQueryF(config->dbh, query, "", limitString ? limitString : "")) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(limitString); + psFree(query); + return false; + } + psFree(limitString); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negate simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "regPendingExp", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // make sure that the exp_id(s) are ready to be updated based on: + // exp_id is not in rawExp + // exp_id is not in newImfile + // that the correct count of imfiles is in rawImfile + + // required + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(exp_name, config->args, "-exp_name", true, false); + PXOPT_LOOKUP_STR(camera, config->args, "-inst", true, false); + PXOPT_LOOKUP_STR(telescope, config->args, "-telescope", true, false); + PXOPT_LOOKUP_STR(exp_tag, config->args, "-exp_tag", true, false); + PXOPT_LOOKUP_STR(filelevel, config->args, "-filelevel", true, false); + + // optional + PXOPT_LOOKUP_TIME(dateobs, config->args, "-dateobs", false, false); + PXOPT_LOOKUP_STR(exp_type, config->args, "-exp_type", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-end_stage", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", false, false); + PXOPT_LOOKUP_STR(comment, config->args, "-comment", false, false); + PXOPT_LOOKUP_STR(obs_mode, config->args, "-obs_mode", false, false); + PXOPT_LOOKUP_STR(obs_group, config->args, "-obs_group", false, false); + PXOPT_LOOKUP_F32(airmass, config->args, "-airmass", false, false); + PXOPT_LOOKUP_F64(ra, config->args, "-ra", false, false); + PXOPT_LOOKUP_F64(decl, config->args, "-decl", false, false); + PXOPT_LOOKUP_F32(exp_time, config->args, "-exp_time", false, false); + PXOPT_LOOKUP_F32(sat_pixel_frac, config->args, "-sat_pixel_frac", false, false); + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F64(bg_mean_stdev, config->args, "-bg_mean_stdev", false, false); + PXOPT_LOOKUP_F64(alt, config->args, "-alt", false, false); + PXOPT_LOOKUP_F64(az, config->args, "-az", false, false); + PXOPT_LOOKUP_F32(ccd_temp, config->args, "-ccd_temp", false, false); + PXOPT_LOOKUP_F64(posang, config->args, "-posang", false, false); + PXOPT_LOOKUP_F32(m1_x, config->args, "-m1_x", false, false); + PXOPT_LOOKUP_F32(m1_y, config->args, "-m1_y", false, false); + PXOPT_LOOKUP_F32(m1_z, config->args, "-m1_z", false, false); + PXOPT_LOOKUP_F32(m1_tip, config->args, "-m1_tip", false, false); + PXOPT_LOOKUP_F32(m1_tilt, config->args, "-m1_tilt", false, false); + PXOPT_LOOKUP_F32(m2_x, config->args, "-m2_x", false, false); + PXOPT_LOOKUP_F32(m2_y, config->args, "-m2_y", false, false); + PXOPT_LOOKUP_F32(m2_z, config->args, "-m2_z", false, false); + PXOPT_LOOKUP_F32(m2_tip, config->args, "-m2_tip", false, false); + PXOPT_LOOKUP_F32(m2_tilt, config->args, "-m2_tilt", false, false); + + PXOPT_LOOKUP_F32(env_temp, config->args, "-env_temperature", false, false); + PXOPT_LOOKUP_F32(env_humid, config->args, "-env_humidity", false, false); + PXOPT_LOOKUP_F32(env_wind, config->args, "-env_wind_speed", false, false); + PXOPT_LOOKUP_F32(env_dir, config->args, "-env_wind_dir", false, false); + PXOPT_LOOKUP_F32(teltemp_m1, config->args, "-teltemp_m1", false, false); + PXOPT_LOOKUP_F32(teltemp_m1cell, config->args, "-teltemp_m1cell", false, false); + PXOPT_LOOKUP_F32(teltemp_m2, config->args, "-teltemp_m2", false, false); + PXOPT_LOOKUP_F32(teltemp_spider, config->args, "-teltemp_spider", false, false); + PXOPT_LOOKUP_F32(teltemp_truss, config->args, "-teltemp_truss", false, false); + PXOPT_LOOKUP_F32(teltemp_extra, config->args, "-teltemp_extra", false, false); + PXOPT_LOOKUP_F32(pon_time, config->args, "-pon_time", false, false); + PXOPT_LOOKUP_F64(user_1, config->args, "-user_1", false, false); + PXOPT_LOOKUP_F64(user_2, config->args, "-user_2", false, false); + PXOPT_LOOKUP_F64(user_3, config->args, "-user_3", false, false); + PXOPT_LOOKUP_F64(user_4, config->args, "-user_4", false, false); + PXOPT_LOOKUP_F64(user_5, config->args, "-user_5", false, false); + PXOPT_LOOKUP_STR(object, config->args, "-object", false, false); + PXOPT_LOOKUP_F32(sun_angle, config->args, "-sun_angle", false, false); + PXOPT_LOOKUP_F32(sun_alt, config->args, "-sun_alt", false, false); + PXOPT_LOOKUP_F32(moon_angle, config->args, "-moon_angle", false, false); + PXOPT_LOOKUP_F32(moon_alt, config->args, "-moon_alt", false, false); + PXOPT_LOOKUP_F32(moon_phase, config->args, "-moon_phase", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", false, false); + PXOPT_LOOKUP_STR(chip_workdir,config->args, "-chip_workdir", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + + // default + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + psString query = pxDataGet("regtool_pendingexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = NULL; + { + // build a query to search by exp_id + psMetadata *where = psMetadataAlloc(); + if (!psMetadataAddS64(where, PS_LIST_TAIL, "newExp.exp_id", 0, "==", exp_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to add item exp_id"); + psFree(where); + psFree(query); + return false; + } + + whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + if (whereClause) { + // skip past comment "hook" + psStringPrepend(&whereClause, "\n AND "); + } + } + + // 1st arg: where hook, 2nd arg: limit hook + if (!p_psDBRunQueryF(config->dbh, query, whereClause ? whereClause : "", "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereClause); + psFree(query); + return false; + } + psFree(whereClause); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, false, "no pending newExp rows found"); + psFree(output); + return false; + } + // sanity check that we only got one row + if (psArrayLength(output) != 1) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "should have gotten 1 row but %lu rows were returned", psArrayLength(output)); + psFree(output); + return NULL; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + // insert the exp into rawExp + psMetadata *row = output->data[0]; + // convert metadata into a newExp object + psMetadataConfigPrint(stdout, row); + newExpRow *newExp = newExpObjectFromMetadata(row); + psFree(output); + if (!newExp) { + psError(PS_ERR_UNKNOWN, false, "this should not happen"); + return false; + } + + // carry through these values + // new CLI options overwrite existing values + workdir = workdir ? workdir : newExp->workdir; + reduction = reduction ? reduction : newExp->reduction; + dvodb = dvodb ? dvodb : newExp->dvodb; + tess_id = tess_id ? tess_id : newExp->tess_id; + end_stage = end_stage ? end_stage : newExp->end_stage; + label = label ? label : newExp->label; + // don't free newExp until just before we return, or these refs will break + + if (!rawExpInsert(config->dbh, + exp_id, + exp_name, + camera, + telescope, + dateobs, + exp_tag, + exp_type, + filelevel, + workdir, + state, + reduction, + dvodb, + tess_id, + end_stage, + filter, + comment, + obs_mode, + obs_group, + airmass, + ra, + decl, + exp_time, + sat_pixel_frac, + bg, + bg_stdev, + bg_mean_stdev, + alt, + az, + ccd_temp, + posang, + m1_x, + m1_y, + m1_z, + m1_tip, + m1_tilt, + m2_x, + m2_y, + m2_z, + m2_tip, + m2_tilt, + env_temp, + env_humid, + env_wind, + env_dir, + teltemp_m1, + teltemp_m1cell, + teltemp_m2, + teltemp_spider, + teltemp_truss, + teltemp_extra, + pon_time, + user_1, + user_2, + user_3, + user_4, + user_5, + object, + sun_angle, + sun_alt, + moon_angle, + moon_alt, + moon_phase, + hostname, + fault, + NULL, + 0 + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newExp); + return false; + } + + // set the state for the newExp to stop + // but only if we didn't encounter a fault + if (fault == 0) { + if (!pxnewExpSetState(config, exp_id, "stop")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to change newExp.state for exp_id: %"PRId64, exp_id); + psFree(newExp); + return false; + } + } + + // should we stop here and proceed on to the chip stage? + // NULL for end_stage means go as far as possible + if (end_stage && psStrcasestr(end_stage, "reg")) { + // then we are done here + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(newExp); + return true; + } + // else continue on... + + chip_workdir = chip_workdir ? chip_workdir : workdir; + // insert an entry into the chipPendingExp table + // this can only be run as the newExp's state has been set to stop + if (!pxchipQueueByExpTag(config, + exp_id, + chip_workdir, + label, + data_group, + dist_group, + reduction, + NULL, // expgroup + dvodb, + tess_id, + end_stage, + NULL // note + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to queue chipPendingExp"); + psFree(newExp); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(newExp); + return false; + } + + psFree(newExp); + return true; +} + + +static bool processedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = pxDataGet("regtool_processedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // XX test this out; need to make this consistent with the list in regtoolConfig.c + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-airmass_min", "airmass", ">="); + PXOPT_COPY_F32(config->args, where, "-airmass_max", "airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "az", "<"); + PXOPT_COPY_F64(config->args, where, "-ccd_temp_min", "ccd_temp", ">="); + PXOPT_COPY_F64(config->args, where, "-ccd_temp_max", "ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "object", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "obs_mode", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-comment", "comment", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "sun_angle", "<"); + + psString where2 = NULL; + if (!pxmagicAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxmagicAddWhere failed"); + return false; + } + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(faulted, config->args, "-faulted", false); + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (where2) { + psStringAppend(&query, " %s ", where2); + psFree(where2); + } + + if (faulted) { + // list only faulted rows + psStringAppend(&query, " %s", "AND rawExp.fault != 0"); + } else { + // don't list faulted rows + psStringAppend(&query, " %s", "AND rawExp.fault = 0"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool revertprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id_begin", "exp_id", ">="); + PXOPT_COPY_S64(config->args, where, "-exp_id_end", "exp_id", "<="); + + psString query = pxDataGet("regtool_revertprocessedexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "rawExp"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("regtool", PS_LOG_INFO, "Updated %d rawExp", numUpdated); + + return true; +} + + +static bool updateprocessedexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + + if (!pxSetFaultCode(config->dbh, "rawExp", where, fault, 0)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree(where); + return false; + } + psFree(where); + + return true; +} + + +static bool finishcompressexpMode(pxConfig *config) { + + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + // XXX unused PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("regtool_finishcompressexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " %s ", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s ", limitString); + psFree(limitString); + } + // printf(query); + // return(false); + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + if (!psArrayLength(output)) { + psTrace("chiptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return(true); + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + rawExpRow *row = rawExpObjectFromMetadata(md); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return(false); + } + + if (strcmp(row->state,"goto_compressed") == 0) { + if (!pxrawExpSetState(config,row->exp_id, "compressed")) { + psError(PS_ERR_UNKNOWN, false, "failed to set rawExp.state for exp_id: %" PRId64, row->exp_id); + psFree(row); + psFree(output); + return(false); + } + } + else if (strcmp(row->state,"goto_lossy") == 0) { + if (!pxrawExpSetState(config,row->exp_id, "lossy")) { + psError(PS_ERR_UNKNOWN, false, "failed to set rawExp.state for exp_id: %" PRId64, row->exp_id); + psFree(row); + psFree(output); + return(false); + } + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(row); + } + + psFree(output); + return(true); +} + +static bool updatebyqueryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(set_state, config->args, "-set_state", true, false); + + // XX test this out; need to make this consistent with the list in regtoolConfig.c + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "rawExp.state", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "rawExp.reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_F32(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F32(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F64(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F64(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "rawExp.obs_mode", "LIKE"); + + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringPrepend(&whereClause,"\n AND "); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + // PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString limitString = NULL; + if (limit) { + limitString = psDBGenerateLimitSQL(limit); + psStringPrepend(&limitString, "\n"); + } + // Update the imfiles first, because if you select by state, you'll clobber things + psString query = pxDataGet("regtool_updatebyqueryimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + + // printf(query,set_state,whereClause,limitString); + if (!p_psDBRunQueryF(config->dbh, query, set_state, whereClause, limitString ? limitString : "")) { + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(limitString); + psFree(query); + psFree(whereClause); + psFree(where); + return(false); + } + + psFree(query); + // Now up date the exposure. + query = pxDataGet("regtool_updatebyquery.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + if (!p_psDBRunQueryF(config->dbh, query, set_state,whereClause, limitString ? limitString : "")) { + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(limitString); + psFree(query); + psFree(whereClause); + psFree(where); + return(false); + } + + psFree(limitString); + psFree(query); + psFree(whereClause); + psFree(where); + + return true; +} + + + +static bool cleardupexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + +// create temp table of exp_ids to be removed +{ + psString query = pxDataGet("regtool_create_dup_table.sql"); + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); +} + +// populate that table +// XXX note that this query currently doesn't not correctly handle the case +// where there is more than one duplicate +{ + psString query = pxDataGet("regtool_populate_dup_table.sql"); + if (!query) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); +} + + if (!p_psDBRunQuery(config->dbh, "DELETE FROM rawImfile USING rawImfile, duplicate WHERE duplicate.exp_id = rawImfile.exp_id")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, "DELETE FROM rawExp USING rawExp, duplicate WHERE duplicate.exp_id = rawExp.exp_id")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, "DELETE FROM newImfile USING newImfile, duplicate WHERE duplicate.exp_id = newImfile.exp_id")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, "DELETE FROM newExp USING newExp, duplicate WHERE duplicate.exp_id = newExp.exp_id")) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool checkstatusMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config,false); + + // required + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + // Conditionally required + PXOPT_LOOKUP_STR(dateobs_begin, config->args, "-dateobs_begin", false, false); + PXOPT_LOOKUP_STR(dateobs_end, config->args, "-dateobs_end", false, false); + PXOPT_LOOKUP_STR(date, config->args, "-date", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("regtool_checkstatus.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + psString rep = psStringCopy(query); + psFree(query); + query = rep; + psStringSubstitute(&query,class_id,"@CLASS_ID@"); + psStringSubstitute(&query,"ota","XY"); + + if (!date) { + if (!dateobs_begin || !dateobs_end) { + psError(PXTOOLS_ERR_CONFIG, false, "Either -date or -dateobs_begin -dateobs_end is required"); + psFree(query); + return false; + } + psStringAppend(&query," AND summitExp.dateobs >= '%s' AND summitExp.dateobs <= '%s' ", + dateobs_begin,dateobs_end); + } + else { + psStringAppend(&query," AND summitExp.dateobs >= '%sT00:00:00' AND summitExp.dateobs <= '%sT23:59:59' ", + date,date); + } + + psStringAppend(&query," ORDER BY summitExp.dateobs "); + + if (!p_psDBRunQuery(config->dbh, query)) { + // XXX PS_EXIT_PROG_ERROR (incorrect SQL) or SYS_ERROR (database comms) + psError(PXTOOLS_ERR_PROG, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "regCheckStatus", !simple)) { + psError(PXTOOLS_ERR_PROG, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return(true); +} + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 2; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // unused and wrong! PXOPT_LOOKUP_S64(det_id, config->args, "-exp_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion to output file"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-exp_id", "exp_id", "=="); + + ExportTable tables [] = { + {"rawExp", "regtool_export_exp.sql"}, + {"rawImfile", "regtool_export_imfile.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("regtool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + + + fprintf (stdout, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); + + psMetadataItem *item = psMetadataLookup (input, "rawExp"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + rawExpRow *rawExp = rawExpObjectFromMetadata (entry->data.md); + rawExpInsertObject (config->dbh, rawExp); + + // fprintf (stdout, "---- raw exp ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + item = psMetadataLookup (input, "rawImfile"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + for (int i = 0; i < item->data.list->n; i++) { + psMetadataItem *entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + rawImfileRow *rawImfile = rawImfileObjectFromMetadata (entry->data.md); + rawImfileInsertObject (config->dbh, rawImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + + return true; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/regtool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/regtool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/regtool.h (revision 41104) @@ -0,0 +1,50 @@ +/* + * regtool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef REGTOOL_H +#define REGTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + REGTOOL_MODE_NONE = 0x0, + REGTOOL_MODE_PENDINGIMFILE, + REGTOOL_MODE_CHECKBURNTOOLIMFILE, + REGTOOL_MODE_PENDINGBURNTOOLIMFILE, + REGTOOL_MODE_ADDPROCESSEDIMFILE, + REGTOOL_MODE_PROCESSEDIMFILE, + REGTOOL_MODE_REVERTPROCESSEDIMFILE, + REGTOOL_MODE_UPDATEPROCESSEDIMFILE, + REGTOOL_MODE_PENDINGEXP, + REGTOOL_MODE_ADDPROCESSEDEXP, + REGTOOL_MODE_PROCESSEDEXP, + REGTOOL_MODE_REVERTPROCESSEDEXP, + REGTOOL_MODE_UPDATEPROCESSEDEXP, + REGTOOL_MODE_CLEARDUPEXP, + REGTOOL_MODE_UPDATEBYQUERY, + REGTOOL_MODE_PENDINGCOMPRESSIMFILE, + REGTOOL_MODE_FINISHCOMPRESSEXP, + REGTOOL_MODE_CHECKSTATUS, + REGTOOL_MODE_EXPORTRUN, + REGTOOL_MODE_IMPORTRUN +} regtoolMode; + +pxConfig *regtoolConfig(pxConfig *config, int argc, char **argv); + +#endif // REGTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/regtoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/regtoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/regtoolConfig.c (revision 41104) @@ -0,0 +1,477 @@ +/* + * regtoolConfig.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include + +#include + +#include "pxtools.h" +#include "regtool.h" + +#define ADD_OPT(TYPE,TARG,NAME,COMMENT,DEFAULT) psMetadataAdd##TYPE(TARG, PS_LIST_TAIL, NAME, 0, COMMENT, DEFAULT) + +pxConfig *regtoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + ADD_OPT(U64, pendingimfileArgs, "-limit", "limit result set to N items", 0); + ADD_OPT(Bool, pendingimfileArgs, "-simple", "use the simple output format", false); + + // -checkburntoolimfile + psMetadata *checkburntoolimfileArgs = psMetadataAlloc(); + ADD_OPT(Str, checkburntoolimfileArgs, "-exp_name", "define the exp_name (required)", NULL); + ADD_OPT(Str, checkburntoolimfileArgs, "-class_id", "define class ID (required)", NULL); + ADD_OPT(Str, checkburntoolimfileArgs, "-dateobs_begin", "set the earliest summit dateobs to consider (required)", NULL); + ADD_OPT(Str, checkburntoolimfileArgs, "-dateobs_end", "set the latest summit dateobs to consider (required)", NULL); + ADD_OPT(S32, checkburntoolimfileArgs, "-valid_burntool", "define the good burntool value (required)", 0); + ADD_OPT(Str, checkburntoolimfileArgs, "-inst", "define the camera name", NULL); + ADD_OPT(Str, checkburntoolimfileArgs, "-telescope", "define the telescope name", NULL); + ADD_OPT(Bool, checkburntoolimfileArgs, "-simple", "use the simple output format", false); + + // -pendingburntoolimfile + psMetadata *pendingburntoolimfileArgs = psMetadataAlloc(); + ADD_OPT(Str, pendingburntoolimfileArgs, "-dateobs_begin", "set the earliest summit dateobs to consider (required)", NULL); + ADD_OPT(Str, pendingburntoolimfileArgs, "-dateobs_end", "set the latest summit dateobs to consider (required)", NULL); + ADD_OPT(S32, pendingburntoolimfileArgs, "-valid_burntool", "define the good burntool value (required)", 0); + ADD_OPT(Bool, pendingburntoolimfileArgs, "-simple", "use the simple output format", false); + ADD_OPT(Bool, pendingburntoolimfileArgs, "-ignore_state", "ignore the data_state when deciding what to work on", false); + ADD_OPT(U64, pendingburntoolimfileArgs, "-limit", "limit result set to N items", 0); + + // -addprocessedimfile + psMetadata *addprocessedimfileArgs = psMetadataAlloc(); + ADD_OPT(S64, addprocessedimfileArgs, "-exp_id", "define exposure ID (required)", 0); + ADD_OPT(Str, addprocessedimfileArgs, "-exp_name", "define the exp_name (required)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-inst", "define the camera name (required)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-telescope", "define the telescope name (required)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-tmp_class_id", "define temp. class ID (required)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-class_id", "define class ID (required)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-uri", "define URI (required)", NULL); + + ADD_OPT(F32, addprocessedimfileArgs, "-longitude", "specify the observatory longitude (NOTE: not saved in db)", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-latitude", "specify the observatory latitude (NOTE: not saved in db)", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-elevation", "specify the elevation (NOTE: not saved in db)", NAN); + + ADD_OPT(Str, addprocessedimfileArgs, "-exp_type", "define exposure type", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-filelevel", "define filelevel", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-filter", "define filter ", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-comment", "define comment ", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-obs_mode", "define observing mode (data usage goal)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-obs_group", "define observing group (set of associated observations)", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-data_state", "define the data_state", "full"); + // Note: RA & DEC are supplied here in radians, but the query options use degrees. This + // can be justified by the fact that this option (addprocessedimfile) is a software + // interface, but the query mechanisms are human user interfaces. + ADD_OPT(F64, addprocessedimfileArgs, "-ra", "define RA (NOTE: radians)", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-decl", "define DEC (NOTE: radians)", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-airmass", "define airmass", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-exp_time", "define exposure time", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-sat_pixel_frac", "define fraction of saturated pixels", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-bg", "define exposue background", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-bg_stdev", "define exposue background stdev", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-bg_mean_stdev", "define exposue background mean stdev", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-alt", "define altitute", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-az", "define azimuth", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-ccd_temp", "define ccd tempature", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-posang", "define rotator position angle", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m1_x", "define M1 X position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m1_y", "define M1 Y position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m1_z", "define M1 Z position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m1_tip", "define M1 TIP position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m1_tilt", "define M1 TILT position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m2_x", "define M2 X position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m2_y", "define M2 Y position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m2_z", "define M2 Z position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m2_tip", "define M2 TIP position", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-m2_tilt", "define M2 TILT position", NAN); + + ADD_OPT(F32, addprocessedimfileArgs, "-env_temperature","define Environmental Temperature", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-env_humidity", "define Environmental Humidity", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-env_wind_speed", "define Environmental Wind Speed", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-env_wind_dir", "define Environmental Wind Direction", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-teltemp_m1", "define Telescope Temperature : M1", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-teltemp_m1cell", "define Telescope Temperature : M1 Cell", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-teltemp_m2", "define Telescope Temperature : M2", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-teltemp_spider", "define Telescope Temperature : Spider", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-teltemp_truss", "define Telescope Temperature : Truss", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-teltemp_extra", "define Telescope Temperature : Extra", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-pon_time", "define time to last Power On", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-user_1", "define user statistic (1)", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-user_2", "define user statistic (2)", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-user_3", "define user statistic (3)", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-user_4", "define user statistic (4)", NAN); + ADD_OPT(F64, addprocessedimfileArgs, "-user_5", "define user statistic (5)", NAN); + ADD_OPT(Str, addprocessedimfileArgs, "-object", "define exposure object", NULL); + ADD_OPT(F32, addprocessedimfileArgs, "-sun_angle", "define angle to sun", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-sun_alt", "define sun altitude (neg = below horizon)", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-moon_angle", "define angle to moon", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-moon_alt", "define moon altitude (neg = below horizon)", NAN); + ADD_OPT(F32, addprocessedimfileArgs, "-moon_phase", "define moon phase (0.0 = new)", NAN); + ADD_OPT(Bool, addprocessedimfileArgs, "-ignore", "ignore this imfile?", false); + ADD_OPT(Time, addprocessedimfileArgs, "-dateobs", "define observation time", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-hostname", "define host name", NULL); + ADD_OPT(Str, addprocessedimfileArgs, "-md5sum", "define md5sum", NULL); + ADD_OPT(S32, addprocessedimfileArgs, "-bytes", "define bytes", 0); + ADD_OPT(S16, addprocessedimfileArgs, "-burntool_state", "set burntool state", 0); + ADD_OPT(Bool, addprocessedimfileArgs, "-video_cells", "define if chip has video cells", false); + ADD_OPT(S16, addprocessedimfileArgs, "-fault", "set fault code", 0); + ADD_OPT(S16, addprocessedimfileArgs, "-quality", "set quality flag", 0); + + // -processedimfile + psMetadata *processedimfileArgs = psMetadataAlloc(); + ADD_OPT(S64, processedimfileArgs, "-exp_id", "search by exposure ID", 0); + ADD_OPT(Str, processedimfileArgs, "-exp_name", "search by exposure name", NULL); + ADD_OPT(Str, processedimfileArgs, "-class_id", "search by class ID", NULL); + ADD_OPT(Str, processedimfileArgs, "-filter", "search by filter", NULL); + ADD_OPT(Str, processedimfileArgs, "-obs_mode", "search by obs_mod", NULL); + ADD_OPT(Str, processedimfileArgs, "-exp_type", "search by exposure type", NULL); + ADD_OPT(Str, processedimfileArgs, "-data_state", "search by data_state", NULL); + ADD_OPT(Time, processedimfileArgs, "-dateobs_begin", "search for exposures by time (>=)", NULL); + ADD_OPT(Time, processedimfileArgs, "-dateobs_end", "search for exposures by time (<)", NULL); + ADD_OPT(U64, processedimfileArgs, "-limit", "limit result set to N items", 0); + ADD_OPT(Bool, processedimfileArgs, "-faulted", "only return imfiles with a fault status set", false); + ADD_OPT(Bool, processedimfileArgs, "-allfiles", "return imfiles regardless of fault status", false); + ADD_OPT(Bool, processedimfileArgs, "-all", "list without search arguments", false); + ADD_OPT(Bool, processedimfileArgs, "-simple", "use the simple output format", false); + ADD_OPT(Bool, processedimfileArgs, "-ordered_by_date", "force output to be in DATE order", false); + ADD_OPT(Bool, processedimfileArgs, "-video_cells", "limit result to chips that have video cells", false); + pxspaceAddArguments(processedimfileArgs); + pxmagicAddArguments(processedimfileArgs); + + // -revertprocessedimfile + psMetadata *revertprocessedimfileArgs = psMetadataAlloc(); + ADD_OPT(S64, revertprocessedimfileArgs, "-exp_id", "search by exposure ID", 0); + ADD_OPT(Str, revertprocessedimfileArgs, "-tmp_class_id", "searcy by temp. class ID", NULL); + ADD_OPT(Str, revertprocessedimfileArgs, "-class_id", "search by class ID", NULL); + ADD_OPT(S16, revertprocessedimfileArgs, "-fault", "search by fault code", 0); + ADD_OPT(S64, revertprocessedimfileArgs, "-exp_id_begin", "search by exposure ID", 0); + ADD_OPT(S64, revertprocessedimfileArgs, "-exp_id_end", "search by exposure ID", 0); + // This argument is not used but is needed because the task uses add.poll.args + ADD_OPT(U64, revertprocessedimfileArgs, "-limit", "for compatability not used", 0); + + // -updateprocessedimfile + psMetadata *updateprocessedimfileArgs = psMetadataAlloc(); + ADD_OPT(S64, updateprocessedimfileArgs, "-exp_id", "search by exposure ID", 0); + ADD_OPT(Str, updateprocessedimfileArgs, "-class_id", "search by class ID", NULL); + ADD_OPT(S16, updateprocessedimfileArgs, "-burntool_state", "set burntool state", INT16_MAX); + ADD_OPT(S16, updateprocessedimfileArgs, "-fault", "set fault code", INT16_MAX); + ADD_OPT(Str, updateprocessedimfileArgs, "-hostname", "set host name", NULL); + ADD_OPT(S32, updateprocessedimfileArgs, "-set_bytes", "set bytes", INT32_MAX); + ADD_OPT(Str, updateprocessedimfileArgs, "-set_md5sum", "set md5sum", NULL); + ADD_OPT(Str, updateprocessedimfileArgs, "-set_state", "set data state", NULL); + psMetadataAddBool(updateprocessedimfileArgs, PS_LIST_TAIL, "-set_ignored", 0, "set imfile to be ignored for processing", false); + psMetadataAddBool(updateprocessedimfileArgs, PS_LIST_TAIL, "-clear_ignored", 0, "set imfile to not be ignored for processing", false); + + + // -pendingexp + psMetadata *pendingexpArgs = psMetadataAlloc(); + psMetadataAddU64(pendingexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addprocessedexp + psMetadata *addprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "exp_id to operate on (required)", 0); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-exp_name", 0, "define the exp_name (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-inst", 0, "define the camera name (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-telescope", 0, "define the telescope name (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, "define the external exposure tag name (required)", NULL); + psMetadataAddStr(addprocessedexpArgs, PS_LIST_TAIL, "-filelevel", 0, "define the data partitioning level of this file (required)", NULL); + + ADD_OPT(F32, addprocessedexpArgs, "-longitude", "specify the observatory longitude (NOTE: not saved in db)", 0.0); + ADD_OPT(F32, addprocessedexpArgs, "-latitude", "specify the observatory latitude (NOTE: not saved in db)", 0.0); + + ADD_OPT(Time, addprocessedexpArgs, "-dateobs", "define observation time", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-exp_type", "define exposure type", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-workdir", "define the \"default\" workdir for this exposure", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-state", "define the state for this exposure", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-dvodb", "define the dvodb for the next processing step", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-tess_id", "define the tess_id for the next processing step", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-end_stage", "define the end goal processing step", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-reduction", "define the \"default\" reduction class for this exposure", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-filter", "define filter ", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-comment", "define comment ", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-obs_mode", "define observing mode (data usage goal)", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-obs_group", "define observing group (set of associated observations)", NULL); + + // Note: RA & DEC are supplied here in radians, but the query options use degrees. This + // can be justified by the fact that this option (addprocessedexp) is a software + // interface, but the query mechanisms are human user interfaces. + ADD_OPT(F64, addprocessedexpArgs, "-ra", "define RA (NOTE: radians)", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-decl", "define DEC (NOTE: radians)", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-airmass", "define airmass", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-exp_time", "define exposure time", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-sat_pixel_frac", "define fraction of saturated pixels", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-bg", "define exposue background", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-bg_stdev", "define exposue background stdev", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-bg_mean_stdev", "define exposue background mean stdev", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-alt", "define altitute", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-az", "define azimuth", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-ccd_temp", "define ccd tempature", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-posang", "define rotator position angle", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m1_x", "define M1 X position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m1_y", "define M1 Y position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m1_z", "define M1 Z position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m1_tip", "define M1 TIP position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m1_tilt", "define M1 TILT position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m2_x", "define M2 X position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m2_y", "define M2 Y position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m2_z", "define M2 Z position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m2_tip", "define M2 TIP position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-m2_tilt", "define M2 TILT position", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-env_temperature", "define Environmental Temperature", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-env_humidity", "define Environmental Humidity", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-env_wind_speed", "define Environmental Wind Speed", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-env_wind_dir", "define Environmental Wind Direction", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-teltemp_m1", "define Telescope Temperature : M1", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-teltemp_m1cell", "define Telescope Temperature : M1 Cell", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-teltemp_m2", "define Telescope Temperature : M2", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-teltemp_spider", "define Telescope Temperature : Spider", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-teltemp_truss", "define Telescope Temperature : Truss", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-teltemp_extra", "define Telescope Temperature : Extra", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-pon_time", "define time to last Power On", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-user_1", "define user statistic (1)", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-user_2", "define user statistic (2)", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-user_3", "define user statistic (3)", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-user_4", "define user statistic (4)", NAN); + ADD_OPT(F64, addprocessedexpArgs, "-user_5", "define user statistic (5)", NAN); + ADD_OPT(Str, addprocessedexpArgs, "-object", "define exposure object", NULL); + ADD_OPT(F32, addprocessedexpArgs, "-sun_angle", "define angle to sun", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-sun_alt", "define sun altitude (neg = below horizon)", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-moon_angle", "define angle to moon", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-moon_alt", "define moon altitude (neg = below horizon)", NAN); + ADD_OPT(F32, addprocessedexpArgs, "-moon_phase", "define moon phase (0.0 = new)", NAN); + ADD_OPT(Str, addprocessedexpArgs, "-label", "define label for chip stage (non-detrend data only)", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-data_group", "define data_group for chip stage (non-detrend data only)", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-dist_group", "define dist_group for chip stage (non-detrend data only)", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-chip_workdir", "define workdir for chip stage (non-detrend data only)", NULL); + ADD_OPT(Str, addprocessedexpArgs, "-hostname", "define host name", NULL); + ADD_OPT(S16, addprocessedexpArgs, "-fault", "set fault code", 0); + + // -processedexp + psMetadata *processedexpArgs = psMetadataAlloc(); + psMetadataAddS64(processedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(processedexpArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(processedexpArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-state", 0, "search by exposure state", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(processedexpArgs, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by exposure obs_mode", NULL); + psMetadataAddStr(processedexpArgs, PS_LIST_TAIL, "-comment", 0, "search by exposure comment", NULL); + + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF32(processedexpArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "define max solar angle", NAN); + pxspaceAddArguments(processedexpArgs); + pxmagicAddArguments(processedexpArgs); + + psMetadataAddU64(processedexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-faulted", 0, "only return imfiles with a fault status set", false); + psMetadataAddBool(processedexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertprocessedexp + psMetadata *revertprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddS16(revertprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-exp_id_begin", 0, "search by exposure ID", 0); + psMetadataAddS64(revertprocessedexpArgs, PS_LIST_TAIL, "-exp_id_end", 0, "search by exposure ID", 0); + + // -updatedprocessedexp + psMetadata *updatedprocessedexpArgs = psMetadataAlloc(); + psMetadataAddS64(updatedprocessedexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddS16(updatedprocessedexpArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", INT16_MAX); + psMetadataAddStr(updatedprocessedexpArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-exp_id", 0, "export this exposure ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + // -cleardupexp + psMetadata *cleardupexpArgs = psMetadataAlloc(); + + // -updatebyquery + psMetadata *updatebyqueryArgs = psMetadataAlloc(); + + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-set_state", 0, "set the state", NULL); + + psMetadataAddS64(updatebyqueryArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(updatebyqueryArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(updatebyqueryArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-state", 0, "search by rawExp.state", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(updatebyqueryArgs, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "search by min solar angle", NAN); + psMetadataAddF32(updatebyqueryArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "search by max solar angle", NAN); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-comment", 0, "search by comment", NULL); + psMetadataAddStr(updatebyqueryArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by obs_mode", NULL); + + psMetadataAddU64(updatebyqueryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(updatebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + + // -pendingcompressimfile + psMetadata *pendingcompressimfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingcompressimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(pendingcompressimfileArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddBool(pendingcompressimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcompressimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingcompressimfileArgs, PS_LIST_TAIL, "-compress", 0, "search only for files to compress", 0); + psMetadataAddBool(pendingcompressimfileArgs, PS_LIST_TAIL, "-clean", 0, "search only for files to clean originals", 0); + + // -finishcompressexp + psMetadata *finishcompressexpArgs = psMetadataAlloc(); + psMetadataAddS64(finishcompressexpArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddBool(finishcompressexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(finishcompressexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -checkstatus + psMetadata *checkstatusArgs = psMetadataAlloc(); + ADD_OPT(Str, checkstatusArgs, "-dateobs_begin", "set the earlist summit dateobs to consider", NULL); + ADD_OPT(Str, checkstatusArgs, "-dateobs_end", "set the latest summit dateobs to consider", NULL); + ADD_OPT(Str, checkstatusArgs, "-date", "use default observing extent and look over entire night", NULL); + ADD_OPT(Str, checkstatusArgs, "-class_id", "define class ID (required)", NULL); + ADD_OPT(Bool, checkstatusArgs, "-simple", "use the simple output format", false); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-pendingimfile", "", REGTOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + PXOPT_ADD_MODE("-checkburntoolimfile", "", REGTOOL_MODE_CHECKBURNTOOLIMFILE, checkburntoolimfileArgs); + PXOPT_ADD_MODE("-pendingburntoolimfile", "", REGTOOL_MODE_PENDINGBURNTOOLIMFILE, pendingburntoolimfileArgs); + PXOPT_ADD_MODE("-addprocessedimfile", "", REGTOOL_MODE_ADDPROCESSEDIMFILE, addprocessedimfileArgs); + PXOPT_ADD_MODE("-processedimfile", "", REGTOOL_MODE_PROCESSEDIMFILE, processedimfileArgs); + PXOPT_ADD_MODE("-revertprocessedimfile", "", REGTOOL_MODE_REVERTPROCESSEDIMFILE, revertprocessedimfileArgs); + PXOPT_ADD_MODE("-updateprocessedimfile", "", REGTOOL_MODE_UPDATEPROCESSEDIMFILE, updateprocessedimfileArgs); + PXOPT_ADD_MODE("-pendingexp", "", REGTOOL_MODE_PENDINGEXP,pendingexpArgs); + PXOPT_ADD_MODE("-addprocessedexp", "", REGTOOL_MODE_ADDPROCESSEDEXP, addprocessedexpArgs); + PXOPT_ADD_MODE("-processedexp", "", REGTOOL_MODE_PROCESSEDEXP, processedexpArgs); + PXOPT_ADD_MODE("-revertprocessedexp", "", REGTOOL_MODE_REVERTPROCESSEDEXP, revertprocessedexpArgs); + PXOPT_ADD_MODE("-updateprocessedexp", "", REGTOOL_MODE_UPDATEPROCESSEDEXP, updatedprocessedexpArgs); + PXOPT_ADD_MODE("-cleardupexp", "", REGTOOL_MODE_CLEARDUPEXP, cleardupexpArgs); + PXOPT_ADD_MODE("-updatebyquery", "", REGTOOL_MODE_UPDATEBYQUERY, updatebyqueryArgs); + PXOPT_ADD_MODE("-pendingcompressimfile", "", REGTOOL_MODE_PENDINGCOMPRESSIMFILE, pendingcompressimfileArgs); + PXOPT_ADD_MODE("-finishcompressexp", "", REGTOOL_MODE_FINISHCOMPRESSEXP, finishcompressexpArgs); + PXOPT_ADD_MODE("-checkstatus", "", REGTOOL_MODE_CHECKSTATUS, checkstatusArgs); + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", REGTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", REGTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PXTOOLS_ERR_SYS, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetool.c (revision 41104) @@ -0,0 +1,1934 @@ +/* + * releasetool.c + * + * Copyright (C) 2013 IfA University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "pxchip.h" +#include "pxspace.h" + +#include "releasetool.h" + +static bool definesurveyMode(pxConfig *config); +static bool listsurveyMode(pxConfig *config); +static bool definereleaseMode(pxConfig *config); +static bool updatereleaseMode(pxConfig *config); +static bool listreleaseMode(pxConfig *config); +static bool definerelexpMode(pxConfig *config); +static bool tocalibexpMode(pxConfig *config); +static bool updaterelexpMode(pxConfig *config); +static bool listrelexpMode(pxConfig *config); +static bool deleterelexpMode(pxConfig *config); +static bool definerelstackMode(pxConfig *config); +static bool updaterelstackMode(pxConfig *config); +static bool setrelstackcalibratedfromskycalMode(pxConfig *config); +static bool listrelstackMode(pxConfig *config); +static bool deleterelstackMode(pxConfig *config); +static bool summaryMode(pxConfig *config); +static bool definerelgroupMode(pxConfig *config); +static bool pendingrelgroupMode(pxConfig *config); +static bool updaterelgroupMode(pxConfig *config); +static bool listrelgroupMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = releasetoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(RELEASETOOL_MODE_DEFINESURVEY, definesurveyMode); + MODECASE(RELEASETOOL_MODE_LISTSURVEY, listsurveyMode); + + MODECASE(RELEASETOOL_MODE_DEFINERELEASE, definereleaseMode); + MODECASE(RELEASETOOL_MODE_UPDATERELEASE, updatereleaseMode); + MODECASE(RELEASETOOL_MODE_TOCALIBEXP, tocalibexpMode); + MODECASE(RELEASETOOL_MODE_LISTRELEASE, listreleaseMode); + + MODECASE(RELEASETOOL_MODE_DEFINERELEXP, definerelexpMode); + MODECASE(RELEASETOOL_MODE_UPDATERELEXP, updaterelexpMode); + MODECASE(RELEASETOOL_MODE_LISTRELEXP, listrelexpMode); + MODECASE(RELEASETOOL_MODE_DELETERELEXP, deleterelexpMode); + + MODECASE(RELEASETOOL_MODE_DEFINERELSTACK, definerelstackMode); + MODECASE(RELEASETOOL_MODE_UPDATERELSTACK, updaterelstackMode); + MODECASE(RELEASETOOL_MODE_SETRELSTACKCALIBRATEDFROMSKYCAL, setrelstackcalibratedfromskycalMode); + MODECASE(RELEASETOOL_MODE_LISTRELSTACK, listrelstackMode); + MODECASE(RELEASETOOL_MODE_DELETERELSTACK, deleterelstackMode); + MODECASE(RELEASETOOL_MODE_SUMMARY, summaryMode); + + MODECASE(RELEASETOOL_MODE_DEFINERELGROUP, definerelgroupMode); + MODECASE(RELEASETOOL_MODE_UPDATERELGROUP, updaterelgroupMode); + MODECASE(RELEASETOOL_MODE_PENDINGRELGROUP, pendingrelgroupMode); + MODECASE(RELEASETOOL_MODE_LISTRELGROUP, listrelgroupMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +static bool definesurveyMode(pxConfig *config) +{ + psError(PS_ERR_UNKNOWN, true, "not yet implemented"); + return false; +} + +static bool listsurveyMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "surveyName", "LIKE"); + + psString query = pxDataGet("releasetool_listsurvey.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + // carry on The list of surveys is short there is no need to require parameters + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "survey", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool definereleaseMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S32(surveyID, config->args, "-set_surveyID", true, false); + PXOPT_LOOKUP_STR(release_name, config->args, "-set_release_name", true, false); + PXOPT_LOOKUP_STR(release_state, config->args, "-set_release_state", true, false); + PXOPT_LOOKUP_S32(dataRelease, config->args, "-set_dataRelease", false, false); + PXOPT_LOOKUP_S32(priority, config->args, "-set_priority", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(ubercal_file, config->args, "-set_ubercal_file", false, false); + PXOPT_LOOKUP_S32(accessLevelMin, config->args, "-set_accessLevelMin", false, false); + + if (!ippReleaseInsert(config->dbh, + 0, // rel_id (auto_increment) + surveyID, + release_name, + release_state, + dataRelease, + priority, + dvodb, + ubercal_file, + accessLevelMin + )) { + psError(PS_ERR_UNKNOWN, false, "failed to insert ippRelease"); + return false; + } + + return true; +} + +static bool updatereleaseMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + PXOPT_LOOKUP_S32(rel_id, config->args, "-rel_id", false, false); + + if (!release_name && !rel_id) { + psError(PXTOOLS_ERR_CONFIG, true, "either -release_name or -rel_id is required\n"); + return false; + } + PXOPT_LOOKUP_STR(release_state, config->args, "-set_release_state", false, false); + PXOPT_LOOKUP_S32(priority, config->args, "-set_priority", false, false); + PXOPT_LOOKUP_S32(dataRelease, config->args, "-set_dataRelease", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(ubercal_file, config->args, "-set_ubercal_file", false, false); + PXOPT_LOOKUP_S32(accessLevelMin, config->args, "-set_accessLevelMin", false, false); + if (!release_state && priority == 0 && dataRelease == -1 && !dvodb && !ubercal_file && accessLevelMin < 0) { + psError(PXTOOLS_ERR_CONFIG, true, "at least one of -set_release_state -set_priority -set_dvodb -set_ubercal_file and -set_dataRelease is required\n"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S32(config->args, where, "-rel_id", "rel_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "release_name", "=="); + + psString sep = ""; + psString comma = ", "; + psString query = psStringCopy("UPDATE ippRelease SET"); + if (release_state) { + psStringAppend(&query, "%s release_state = '%s'", sep, release_state); + sep = comma; + } + if (priority != 0) { + psStringAppend(&query, "%s priority = %d", sep, priority); + sep = comma; + } + if (dataRelease != -1) { + psStringAppend(&query, "%s dataRelease = %d", sep, dataRelease); + sep = comma; + } + if (accessLevelMin != -1) { + psStringAppend(&query, "%s accessLevelMin = %d", sep, accessLevelMin); + sep = comma; + } + if (dvodb) { + psStringAppend(&query, "%s dvodb = '%s'", sep, dvodb); + sep = comma; + } + if (ubercal_file) { + psStringAppend(&query, "%s ubercal_file = '%s'", sep, ubercal_file); + sep = comma; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("releasetool", PS_LOG_INFO, "Updated %" PRIu64 " ippReleases", affected); + + + psFree(query); + + return true; +} + +static bool listreleaseMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "surveyName", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-release_name", "release_name", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-release_state", "release_state", "=="); + PXOPT_COPY_S32(config->args, where, "-rel_id", "rel_id", "=="); + + psString query = pxDataGet("releasetool_listrelease.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + // The list of releases is short there is no need to require parameters + #ifdef notyet + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + #endif + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "ippRelease", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; + } + +static bool definerelexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "camRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "camRun.data_group", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + + // Insure that at least one of the camRun selectors is supplied + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "at least one of -label, -data_group, or -cam_id is required"); + return false; + } + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + + // insure that at least one of these is supplied to select the release + PXOPT_LOOKUP_S64(rel_id, config->args, "-rel_id", false, false); + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + if (!rel_id && !release_name) { + psError(PXTOOLS_ERR_CONFIG, true, "either -rel_id or -release_name is required"); + return false; + } + + // limit query to the target release. (Note: this will select nothing if release_name and rel_id don't match) + // note: we add these after above to insure that search args are supplied + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "=="); + PXOPT_COPY_S32(config->args, where, "-rel_id", "ippRelease.rel_id", "=="); + + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + PXOPT_LOOKUP_U32(flags, config->args, "-set_flags", false, false); + PXOPT_LOOKUP_F32(user_zpt_obs, config->args, "-set_zpt_obs", false, false); + PXOPT_LOOKUP_F32(user_zpt_stdev, config->args, "-set_zpt_stdev", false, false); + PXOPT_LOOKUP_F32(mcal, config->args, "-set_mcal", false, false); + PXOPT_LOOKUP_S32(ubercal_dist, config->args, "-set_ubercal_dist", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-set_path_base", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find the parameters of all the exposures that we want to add to the release + psString query = pxDataGet("releasetool_definerelexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + // loop over our list of exposures + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status = true; + psS32 rel_id = psMetadataLookupS32(&status, md, "rel_id"); + psS64 exp_id = psMetadataLookupS64(&status, md, "exp_id"); + psS64 chip_id = psMetadataLookupS64(&status, md, "chip_id"); + psS64 cam_id = psMetadataLookupS64(&status, md, "cam_id"); + + // if zpt_obs or zpt_stdev were not supplied use the results from the camRun + psF32 zpt_obs, zpt_stdev; + if (!isfinite(user_zpt_obs)) { + zpt_obs = psMetadataLookupF32(&status, md, "zpt_obs"); + } else { + zpt_obs = user_zpt_obs; + } + if (!isfinite(user_zpt_stdev)) { + zpt_stdev = psMetadataLookupF32(&status, md, "zpt_stdev"); + } else { + zpt_stdev = user_zpt_stdev; + } + + if (!relExpInsert(config->dbh, + 0, // relexp_id (auto increment) + rel_id, + exp_id, + chip_id, + cam_id, + 0, // group_id + state, + flags, + zpt_obs, + zpt_stdev, + mcal, + ubercal_dist, + path_base, + fault, + now, // registered + now // time_stamp + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to insert relExp for cam_id %" PRId64, cam_id); + psFree(output); + psFree(now); + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updaterelexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + PXOPT_LOOKUP_S32(rel_id, config->args, "-rel_id", false, false); + PXOPT_LOOKUP_S32(relexp_id, config->args, "-relexp_id", false, false); + + if (!relexp_id && !release_name && !rel_id) { + psError(PXTOOLS_ERR_CONFIG, true, "at least one of -relexp_id, -release_name, and -rel_id is required\n"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_U32(flags, config->args, "-set_flags", false, false); + PXOPT_LOOKUP_STR(path_base,config->args, "-set_path_base", false, false); + PXOPT_LOOKUP_F32(zpt_obs, config->args, "-set_zpt_obs", false, false); + PXOPT_LOOKUP_F32(zpt_stdev,config->args, "-set_zpt_stdev", false, false); + PXOPT_LOOKUP_F32(mcal, config->args, "-set_mcal", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_S32(ubercal_dist, config->args, "-set_ubercal_dist", false, false); + PXOPT_LOOKUP_BOOL(clearfault, config->args, "-clearfault", false); + if (!state && !flags && !path_base && !isfinite(zpt_obs) && !isfinite(zpt_stdev) &&!isfinite(mcal) + && !fault && !clearfault && !ubercal_dist) { + psError(PXTOOLS_ERR_CONFIG, true, "must set at least one column\n"); + return false; + } + + if (fault && clearfault) { + psError(PXTOOLS_ERR_CONFIG, true, "cannot set and clear fault at same time\n"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-exp_id", "relExp.exp_id", "=="); + PXOPT_COPY_S32(config->args, where, "-relexp_id", "relExp.relexp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + // make sure that we have enough parameters to identify the relExp to change + if (!psListLength(where->list)){ + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + PXOPT_COPY_S32(config->args, where, "-rel_id", "rel_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "release_name", "=="); + + psString sep = ""; + psString comma = ", "; + psString query = psStringCopy("UPDATE relExp JOIN ippRelease USING(rel_id) JOIN rawExp USING(exp_id) SET"); + if (state) { + psStringAppend(&query, "%s relExp.state = '%s'", sep, state); + sep = comma; + } + if (path_base) { + psStringAppend(&query, "%s relExp.path_base = '%s'", sep, path_base); + sep = comma; + } + if (flags) { + psStringAppend(&query, "%s relExp.flags = %d", sep, flags); + sep = comma; + } + if (isfinite(zpt_obs)) { + psStringAppend(&query, "%s relExp.zpt_obs = %f", sep, zpt_obs); + sep = comma; + } + if (isfinite(zpt_stdev)) { + psStringAppend(&query, "%s relExp.zpt_stdev = %f", sep, zpt_stdev); + sep = comma; + } + if (fault) { + psStringAppend(&query, "%s relExp.fault = %d", sep, fault); + sep = comma; + } + if (clearfault) { + psStringAppend(&query, "%s relExp.fault = 0", sep); + sep = comma; + } + if (isfinite(mcal)) { + psStringAppend(&query, "%s relExp.mcal = %f", sep, mcal); + sep = comma; + } + if (ubercal_dist) { + psStringAppend(&query, "%s relExp.ubercal_dist = %d", sep, ubercal_dist); + sep = comma; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("releasetool", PS_LOG_INFO, "Updated %" PRIu64 " relExp rows", affected); + + + psFree(query); + + return true; +} + +static bool tocalibexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-relexp_id", "relExp.relexp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "LIKE"); +#ifdef notdef + PXOPT_COPY_S32(config->args, where, "-rel_id", "relExp.rel_id", "=="); + pxAddLabelSearchArgs(config, where, "-release_state","ippRelease.state", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "relExp.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin","rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_F32(config->args, where, "-fwhm_min", "camProcessedExp.fwhm_major", ">="); + PXOPT_COPY_F32(config->args, where, "-fwhm_max", "camProcessedExp.fwhm_major", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "relExp.exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "relExp.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "relExp.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-chip_data_group", "chipRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-cam_data_group", "camRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-warp_data_group", "warpRun.data_group", "LIKE"); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "survey.surveyName", "LIKE"); + + PXOPT_LOOKUP_BOOL(priority_order, config->args, "-priority_order", false); +#endif + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("releasetool_tocalibexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString where2 = NULL; +#ifdef notdef + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxspaceAddWhere failed"); + return false; + } +#endif + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); +#ifdef notdef + } else if (where2) { + psStringAppend(&query, "\nAND "); +#endif + } else { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + if (where2) { + psStringAppend(&query, "\n%s", where2); + psFree(where2); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "relExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool listrelexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-relexp_id", "relExp.relexp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "LIKE"); + pxAddLabelSearchArgs(config, where, "-release_state","ippRelease.state", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "relExp.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin","rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_F32(config->args, where, "-fwhm_min", "camProcessedExp.fwhm_major", ">="); + PXOPT_COPY_F32(config->args, where, "-fwhm_max", "camProcessedExp.fwhm_major", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "relExp.exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "relExp.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "relExp.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-chip_data_group", "chipRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-cam_data_group", "camRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-warp_data_group", "warpRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", false, false); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "survey.surveyName", "LIKE"); + PXOPT_COPY_S32(config->args, where, "-rel_id", "relExp.rel_id", "=="); + + PXOPT_LOOKUP_BOOL(priority_order, config->args, "-priority_order", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("releasetool_listrelexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (skycell_id) { + psStringAppend(&query, "\nJOIN warpSkyfile ON warpRun.warp_id = warpSkyfile.warp_id AND warpRun.tess_id = warpSkyfile.tess_id"); + } + + psString where2 = NULL; + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxspaceAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + if (where2) { + psStringAppend(&query, " %s", where2); + } + psStringAppend(&query, " AND warpRun.state != 'drop'"); + psFree(whereClause); + } else if (where2) { + psStringAppend(&query, "\nWHERE "); + } else { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + if (where2) { + psStringAppend(&query, "\n%s", where2); + psFree(where2); + } + + if (priority_order) { + psStringAppend(&query, "\nAND priority > 0 ORDER BY exp_id, priority DESC"); + } else { + psStringAppend(&query, "\nORDER BY exp_id, relexp_id DESC"); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "relExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool deleterelexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(relexp_id, config->args, "-relexp_id", false, false); + // If relexp_id is supplied that is all that we need + if (!relexp_id) { + // Otherwise we need to identify the release ... + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + PXOPT_LOOKUP_S32(rel_id, config->args, "-rel_id", false, false); + + if (!release_name && !rel_id) { + psError(PXTOOLS_ERR_CONFIG, true, "at least one of -relexp_id, -release_name, or -rel_id is required\n"); + return false; + } + // ... and the exposure + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_S64(cam_id, config->args, "-cam_id", false, false); + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + PXOPT_LOOKUP_STR(exp_name, config->args, "-exp_name", false, false); + if (!label && !cam_id && !exp_id && !exp_name) { + psError(PXTOOLS_ERR_CONFIG, true, "at least one of -label, -cam_id, -exp_id, -exp_name is required\n"); + return false; + } + } + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S32(config->args, where, "-relexp_id", "relExp.relexp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "camRun.label", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "relExp.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "relExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + + PXOPT_COPY_S32(config->args, where, "-rel_id", "relExp.rel_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "=="); + + psString query = pxDataGet("releasetool_deleterelexp.sql"); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("releasetool", PS_LOG_INFO, "Deleted %" PRIu64 " relExp rows", affected); + + + psFree(query); + + return true; +} +static bool definerelstackMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + bool includeSkycal = false; + + PXOPT_COPY_STR(config->args, where, "-skycal_label", "skycalRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-skycal_data_group", "skycalRun.data_group", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalRun.skycal_id", "=="); + if (psListLength(where->list)) { + includeSkycal = true; + } + + PXOPT_COPY_STR(config->args, where, "-label", "stackRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "stackRun.data_group", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + + // Insure that at least one of the skycalRun selectors is supplied + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "at least one of -label, -data_group, or -skycal_id is required"); + return false; + } + + // PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "=="); + // PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "=="); + + // insure that at least one of these is supplied to select the release + PXOPT_LOOKUP_S64(rel_id, config->args, "-rel_id", false, false); + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + if (!rel_id && !release_name) { + psError(PXTOOLS_ERR_CONFIG, true, "either -rel_id or -release_name is required"); + return false; + } + + // limit query to the target release. (Note: this will select nothing if release_name and rel_id don't match) + // note: we add these after above to insure that search args are supplied + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "=="); + PXOPT_COPY_S32(config->args, where, "-rel_id", "ippRelease.rel_id", "=="); + + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", true, false); + PXOPT_LOOKUP_STR(stack_type, config->args,"-set_stack_type", true, false); + PXOPT_LOOKUP_U32(flags, config->args, "-set_flags", false, false); + PXOPT_LOOKUP_F32(zpt_obs, config->args, "-set_zpt_obs", false, false); + PXOPT_LOOKUP_F32(zpt_stdev, config->args, "-set_zpt_stdev", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-set_path_base", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + // find the parameters of all the exposures that we want to add to the release + psString query; + + if (includeSkycal) { + query = pxDataGet("releasetool_definerelstack_with_skycal.sql"); + } else { + query = pxDataGet("releasetool_definerelstack.sql"); + } + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString joinConditions = psStringCopy(""); + // Finish up the selectors for the JOIN to previousRelStack + // join to rows with our stack_type + psStringAppend(&joinConditions, "\nAND previousRelStack.stack_type = '%s'", stack_type); + + if (!strcmp(stack_type, "nightly")) { + // detect nightly stack entries that already exist for this day + psStringAppend(&joinConditions, "\nAND previousRelStack.mjd_obs = floor(stackSumSkyfile.mjd_obs)"); + } + + // Add in the where conditions + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); + + if (!strcmp(stack_type, "nightly")) { + // avoid stacks with NAN mjd_obs. These are old skycells that were lost due to system failure + // prior to the time that the mjd_obs was extracted from the headers and added to stackSumSkyfile + psStringAppend(&query, "\nAND stackSumSkyfile.mjd_obs IS NOT NULL"); + } + + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, joinConditions)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(joinConditions); + return false; + } + psFree(query); + psFree(joinConditions); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "relstack", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + // loop over our list of exposures + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status = true; + psS32 rel_id = psMetadataLookupS32(&status, md, "rel_id"); + psS64 stack_id = psMetadataLookupS64(&status, md, "stack_id"); + psS64 skycal_id = psMetadataLookupS64(&status, md, "skycal_id"); + psString skycell_id = psMetadataLookupStr(&status, md, "skycell_id"); + psString tess_id = psMetadataLookupStr(&status, md, "tess_id"); + psString filter = psMetadataLookupStr(&status, md, "filter"); + + // use skycal zero point measurement if not supplied with arguments + psF32 sc_zpt_obs = psMetadataLookupF32(&status, md, "zpt_obs"); + psF32 sc_zpt_stdev = psMetadataLookupF32(&status, md, "zpt_stdev"); + psF32 stack_mjd_obs = psMetadataLookupF32(&status, md, "mjd_obs"); + + psU32 mjd_obs = 0; + if (!strcmp(stack_type, "nightly")) { + // only for nightly stacks do we set mjd_obs to a non zero value + // We use integer day because this is part of the key that is used + // to make sure entries are unique + // KEY (rel_id, tess_id, skycell_id, filter, mjd_obs), + // 1 entry per release, skycell, filter and mjd_obs. + // mjd_obs is zero for deep and reference stacks + mjd_obs = (psU32) stack_mjd_obs; + } + + if (!relStackInsert(config->dbh, + 0, // relstack_id (auto increment) + rel_id, + stack_id, + skycal_id, + skycell_id, + tess_id, + filter, + state, + flags, + stack_type, + isfinite(zpt_obs) ? zpt_obs : sc_zpt_obs, + isfinite(zpt_stdev) ? zpt_stdev : sc_zpt_stdev, + mjd_obs, + path_base, + fault, + now, // registered + now // time_stamp + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to insert relStack for stack_id %" PRId64, stack_id); + psFree(output); + psFree(now); + return false; + } + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updaterelstackMode(pxConfig *config) +{ + psError(PS_ERR_UNKNOWN, true, "not yet implemented"); + return false; +} +static bool setrelstackcalibratedfromskycalMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-skycal_label", "skycalRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-skycal_data_group", "skycalRun.data_group", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalRun.skycal_id", "=="); + if (!psListLength(where->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "at least one of -skycal_label, -skycal_data_group, or -skycal_id is required"); + return false; + } + + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "=="); + pxskycellAddWhere(config, where); + + // insure that at least one of these is supplied to select the release + PXOPT_LOOKUP_S64(rel_id, config->args, "-rel_id", false, false); + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + if (!rel_id && !release_name) { + psError(PXTOOLS_ERR_CONFIG, true, "either -rel_id or -release_name is required"); + return false; + } + + // limit query to the target release. (Note: this will select nothing if release_name and rel_id don't match) + // note: we add these after above to insure that search args are supplied + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "=="); + PXOPT_COPY_S32(config->args, where, "-rel_id", "ippRelease.rel_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(replace, config->args, "-replace", false); + + psString query = pxDataGet("releasetool_setrelstackcalibratedfromskycal.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (replace) { + // don't cut on relStack.state + psStringAppend(&query, "\nWHERE ");; + } else { + // normal path only update rows in state processed + psStringAppend(&query, "\nWHERE relStack.state = 'processed' and skycalResult.quality = 0\nAND ");; + } + + // Add in the where conditions + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "%s", whereClause); + psFree(whereClause); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("releasetool", PS_LOG_INFO, "Updated %" PRIu64 " relStack rows", affected); + + return true; +} + +static bool deleterelstackMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(relstack_id, config->args, "-relstack_id", false, false); + // If relexp_id is supplied that is all that we need + if (!relstack_id) { + // Otherwise we need to identify the release ... + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + PXOPT_LOOKUP_S32(rel_id, config->args, "-rel_id", false, false); + + if (!release_name && !rel_id) { + psError(PXTOOLS_ERR_CONFIG, true, "at least one of -relstack_id, -release_name, or -rel_id is required\n"); + return false; + } + // ... and the Stack + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_S64(stack_id, config->args, "-stack_id", false, false); + if (!label && !stack_id) { + psError(PXTOOLS_ERR_CONFIG, true, "at least one of -label and -stack_id is required\n"); + return false; + } + } + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S32(config->args, where, "-relstack_id", "relStack.relstack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "stackRun.label", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "relStack.stack_id", "=="); + + PXOPT_COPY_S32(config->args, where, "-rel_id", "relStack.rel_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "=="); + + psString query = pxDataGet("releasetool_deleterelstack.sql"); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("releasetool", PS_LOG_INFO, "Deleted %" PRIu64 " relStack rows", affected); + + + psFree(query); + + return true; +} + +static bool listrelstackMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-relstack_id", "relStack.relstack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "relStack.stack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "relStack.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-relstack_id", "relStack.relstack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "LIKE"); + pxAddLabelSearchArgs(config, where, "-release_state","ippRelease.state", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "relStack.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "relStack.filter", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-mjd_min", "stackSumSkyfile.mjd_obs", ">="); + PXOPT_COPY_F32(config->args, where, "-mjd_max", "stackSumSkyfile.mjd_obs", "<="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "relStack.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "relStack.skycell_id", "LIKE"); +// PXOPT_COPY_STR(config->args, where, "-stack_type", "relStack.stack_type", "=="); + PXOPT_COPY_STR(config->args, where, "-stack_data_group", "stackRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycal_data_group", "skycalRun.data_group", "LIKE"); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "survey.surveyName", "LIKE"); + PXOPT_COPY_S32(config->args, where, "-rel_id", "relStack.rel_id", "=="); + pxskycellAddWhere(config, where); + + PXOPT_COPY_F32(config->args, where, "-fwhm_min", "IFNULL(skycalResult.fwhm_major, 999)", ">="); + PXOPT_COPY_F32(config->args, where, "-fwhm_max", "IFNULL(skycalResult.fwhm_major, 0)", "<="); + + PXOPT_LOOKUP_BOOL(priority_order, config->args, "-priority_order", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + pxAddLabelSearchArgs (config, where, "-stack_type", "relStack.stack_type", "=="); + + psString query = pxDataGet("releasetool_listrelstack.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + if (priority_order) { + psStringAppend(&query, "\nAND priority > 0 order by priority DESC, stack_id"); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "relStack", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool summaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-sass_id", "stackAssociation.sass_id", "=="); + PXOPT_COPY_S64(config->args, where, "-relstack_id", "relStack.relstack_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "relStack.stack_id", "=="); +// PXOPT_COPY_S64(config->args, where, "-skycal_id", "relStack.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-relstack_id", "relStack.relstack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "LIKE"); + pxAddLabelSearchArgs(config, where, "-release_state","ippRelease.state", "=="); +// PXOPT_COPY_STR(config->args, where, "-state", "relStack.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "relStack.filter", "LIKE"); +// PXOPT_COPY_F32(config->args, where, "-mjd_min", "stackSumSkyfile.mjd_obs", ">="); +// PXOPT_COPY_F32(config->args, where, "-mjd_max", "stackSumSkyfile.mjd_obs", "<="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "relStack.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "relStack.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "stackAssociation.projection_cell", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-stack_data_group", "stackRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-data_group", "stackAssociation.data_group", "LIKE"); +// PXOPT_COPY_STR(config->args, where, "-skycal_data_group", "skycalRun.data_group", "LIKE"); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "survey.surveyName", "LIKE"); + PXOPT_COPY_S32(config->args, where, "-rel_id", "relExp.rel_id", "=="); + pxskycellAddWhere(config, where); + +// PXOPT_COPY_F32(config->args, where, "-fwhm_min", "IFNULL(skycalResult.fwhm_major, 999)", ">="); +// PXOPT_COPY_F32(config->args, where, "-fwhm_max", "IFNULL(skycalResult.fwhm_major, 0)", "<="); + + PXOPT_LOOKUP_BOOL(priority_order, config->args, "-priority_order", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + pxAddLabelSearchArgs (config, where, "-stack_type", "relStack.stack_type", "=="); + + psString query = pxDataGet("releasetool_summary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + if (priority_order) { + psStringAppend(&query, "\nAND priority > 0 order by priority DESC, stack_id"); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "stackSummary", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool definerelgroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(group_type, config->args, "-set_group_type", true, false); + PXOPT_LOOKUP_STR(group_name, config->args, "-set_group_name", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(exp_list_path, config->args, "-set_exp_list_path", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + + psString query = NULL; + + psMetadata *where = psMetadataAlloc(); + + // select sql and paramters based on group type + bool type_lap = false; + if (!strcmp(group_type, "lap")) { + type_lap = true; + if (group_name) { + psError(PXTOOLS_ERR_CONFIG, true, "group_name is not allowed with group_type lap"); + psFree(where); + return false; + } + PXOPT_COPY_S64(config->args, where, "-select_seq_id", "lapRun.seq_id", "=="); + PXOPT_COPY_S64(config->args, where, "-select_lap_id", "lapRun.lap_id", "=="); + query = pxDataGet("releasetool_definerelgroup_select_lap.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + } else { + if (!strcmp(group_type, "date")) { + // group_type ok + } else if (!strcmp(group_type, "data_group")) { + // group_type ok + } else { + psError(PXTOOLS_ERR_CONFIG, true, "%s is not valid group_type", group_type); + return false; + } + if (!group_name) { + psError(PXTOOLS_ERR_CONFIG, true, "group_name is required allowed with group_type %s", group_type); + psFree(where); + return false; + } + pxAddLabelSearchArgs (config, where, "-select_data_group", "camRun.data_group", "=="); + query = pxDataGet("releasetool_definerelgroup_select_data_group.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + } + + // Insure that at least one of the exposure selectors is supplied + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search arguments are required"); + return false; + } + + // insure that at least one of these is supplied to select the release + PXOPT_LOOKUP_S64(rel_id, config->args, "-rel_id", false, false); + PXOPT_LOOKUP_STR(release_name, config->args, "-release_name", false, false); + if (!rel_id && !release_name) { + psError(PXTOOLS_ERR_CONFIG, true, "either -rel_id or -release_name is required"); + return false; + } + + // limit query to the target release. (Note: this will select nothing if release_name and rel_id don't match) + // note: we add these after above to insure that search args are supplied + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "=="); + PXOPT_COPY_S32(config->args, where, "-rel_id", "ippRelease.rel_id", "=="); + + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + + psFree(where); + + if (!type_lap) { + psStringAppend(&query, "\nHAVING COUNT(exp_id) > 0"); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "relGroupToQueue", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + // loop over our list of exposures + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + bool status = true; + psS32 rel_id = psMetadataLookupS32(&status, md, "rel_id"); + psS64 lap_id = 0; + if (type_lap) { + lap_id = psMetadataLookupS64(&status, md, "lap_id"); + } + + if (!relGroupInsert(config->dbh, + 0, // group_id (auto increment) + rel_id, + group_type, + lap_id, + group_name, + "reg", // state + label, + exp_list_path, + fault, + now // registered + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to trying to insert relGroup"); + psFree(output); + psFree(now); + return false; + } + // Here's our new group_id + psS32 group_id = psDBLastInsertID(config->dbh); + + psString query; + if (type_lap) { + query = pxDataGet("releasetool_definerelgroup_select_exp_lap.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + psStringAppend(&query, " AND relExp.rel_id = %d AND lap_id = %" PRId64, rel_id, lap_id); + } else { + query = pxDataGet("releasetool_definerelgroup_select_exp_data_group.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + psStringAppend(&query, " AND relExp.rel_id = %d", rel_id); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-select_data_group", "camRun.data_group", "=="); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + psFree(where); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(output); + psFree(now); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + psArray *exposures = p_psDBFetchResult(config->dbh); + if (!exposures) { + psFree(now); + psFree(output); + psError(PS_ERR_UNKNOWN, false, "database error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + if (!psArrayLength(exposures)) { + fprintf(stderr, "no exposures found for lap_id %" PRId64 "\n", lap_id); +#ifdef notdef + continue; + psFree(now); + psFree(output); + psFree(exposures); + psError(PS_ERR_UNKNOWN, false, "no exposures found error"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; +#endif + } + + // assign the relExps to this new group + char *updateQuery = "UPDATE relExp SET group_id = %d WHERE relexp_id = %" PRId64; + for (int j = 0; j < psArrayLength(exposures); j++) { + psMetadata *md = exposures->data[j]; + + bool status = true; + psS32 relexp_id = psMetadataLookupS32(&status, md, "relexp_id"); + if (!p_psDBRunQueryF(config->dbh, updateQuery, group_id, relexp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(now); + psFree(output); + psFree(exposures); + + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } + psFree(exposures); + + // update the state of relGroup to 'new' + if (!p_psDBRunQueryF(config->dbh, "UPDATE relGroup set state = 'new' WHERE group_id = %d", group_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(now); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } + psFree(output); + psFree(now); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updaterelgroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S32(group_id, config->args, "-group_id", false, false); + PXOPT_LOOKUP_STR(group_name, config->args, "-group_name", false, false); + if (!group_id && !group_name) { + psError(PXTOOLS_ERR_CONFIG, true, "either group_id or group_name is required\n"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(exp_list_path, config->args, "-set_exp_list_path", false, false); + PXOPT_LOOKUP_STR(new_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_BOOL(clearfault, config->args, "-clearfault", false); + + if (!state && !exp_list_path && !new_label && !fault && !clearfault) { + psError(PXTOOLS_ERR_CONFIG, true, "must set at least one column\n"); + return false; + } + + if (fault && clearfault) { + psError(PXTOOLS_ERR_CONFIG, true, "cannot set and clear fault at same time\n"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-group_id", "relGroup.group_id", "=="); + PXOPT_COPY_STR(config->args, where, "-group_name", "relGroup.group_name", "=="); + // XXX TODO if label is enabled (for changing label or state for a block) we should + // disallow setting some parameters such as exp_list_path + // PXOPT_COPY_S64(config->args, where, "-label", "relGroup.label", "=="); + // make sure that we have enough parameters to identify the relGroup to change + if (!psListLength(where->list)){ + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + psString sep = ""; + psString comma = ", "; + psString query = psStringCopy("UPDATE relGroup SET"); + if (state) { + psStringAppend(&query, "%s relGroup.state = '%s'", sep, state); + sep = comma; + } + if (exp_list_path) { + psStringAppend(&query, "%s relGroup.exp_list_path = '%s'", sep, exp_list_path); + sep = comma; + } + if (new_label) { + psStringAppend(&query, "%s relGroup.label = '%s'", sep, new_label); + sep = comma; + } + if (fault) { + psStringAppend(&query, "%s relGroup.fault = %d", sep, fault); + sep = comma; + } + if (clearfault) { + psStringAppend(&query, "%s relGroup.fault = 0", sep); + sep = comma; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psU64 affected = psDBAffectedRows(config->dbh); + psLogMsg("releasetool", PS_LOG_INFO, "Updated %" PRIu64 " ippReleases", affected); + + + psFree(query); + + return true; +} + +static bool pendingrelgroupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + pxAddLabelSearchArgs (config, where, "-label", "relGroup.label", "=="); + PXOPT_COPY_S32(config->args, where, "-group_id", "relGroup.group_id", "=="); + PXOPT_COPY_S32(config->args, where, "-rel_id", "relExp.rel_id", "=="); + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("releasetool_pendingrelgroup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "pending_relGroup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool listrelgroupMode(pxConfig *config) +{ + return false; +#ifdef notdef + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-release_name", "ippRelease.release_name", "LIKE"); + pxAddLabelSearchArgs(config, where, "-release_state","ippRelease.state", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "relExp.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin","rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_F32(config->args, where, "-fwhm_min", "skycalResult.fwhm_major", ">="); + PXOPT_COPY_F32(config->args, where, "-fwhm_max", "skycalResult.fwhm_major", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_ame", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "relExp.exp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "relExp.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "relExp.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-chip_data_group", "chipRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-cam_data_group", "camRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-warp_data_group", "warpRun.data_group", "LIKE"); + + PXOPT_COPY_STR(config->args, where, "-surveyName", "survey.surveyName", "LIKE"); + PXOPT_COPY_S32(config->args, where, "-rel_id", "relExp.rel_id", "=="); + + PXOPT_LOOKUP_BOOL(priority_order, config->args, "-priority_order", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("releasetool_listrelexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString where2 = NULL; + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxspaceAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + } else if (where2) { + psStringAppend(&query, "\nWHERE "); + } else { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required\n"); + psFree(where); + return false; + } + + if (where2) { + psStringAppend(&query, "\n%s", where2); + psFree(where2); + } + + if (priority_order) { + psStringAppend(&query, "\nAND priority > 0 order by exp_id, priority"); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("releasetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "relExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +#endif //notdef +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetool.h (revision 41104) @@ -0,0 +1,52 @@ +/* + * releasetool.h + * + * Copyright (C) 2013 University of Hawaii IFA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef RELEASETOOL_H +#define RELEASETOOL_H 1 + +#include "pxtools.h" + +typedef enum { + RELEASETOOL_MODE_NONE = 0x0, + RELEASETOOL_MODE_DEFINESURVEY, + RELEASETOOL_MODE_LISTSURVEY, + RELEASETOOL_MODE_DEFINERELEASE, + RELEASETOOL_MODE_UPDATERELEASE, + RELEASETOOL_MODE_LISTRELEASE, + RELEASETOOL_MODE_DEFINERELEXP, + RELEASETOOL_MODE_UPDATERELEXP, + RELEASETOOL_MODE_TOCALIBEXP, + RELEASETOOL_MODE_LISTRELEXP, + RELEASETOOL_MODE_DELETERELEXP, + RELEASETOOL_MODE_DEFINERELSTACK, + RELEASETOOL_MODE_UPDATERELSTACK, + RELEASETOOL_MODE_SETRELSTACKCALIBRATEDFROMSKYCAL, + RELEASETOOL_MODE_DELETERELSTACK, + RELEASETOOL_MODE_TOCALIBSTACK, + RELEASETOOL_MODE_LISTRELSTACK, + RELEASETOOL_MODE_SUMMARY, + RELEASETOOL_MODE_DEFINERELGROUP, + RELEASETOOL_MODE_PENDINGRELGROUP, + RELEASETOOL_MODE_UPDATERELGROUP, + RELEASETOOL_MODE_LISTRELGROUP, +} releasetoolMode; + +pxConfig *releasetoolConfig(pxConfig *config, int argc, char **argv); + +#endif // RELEASETOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/releasetoolConfig.c (revision 41104) @@ -0,0 +1,430 @@ +/* + * releasetoolConfig.c + * + * Copyright (C) 2013 IfA University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "pxspace.h" +#include "releasetool.h" + +pxConfig *releasetoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definesurvey + psMetadata *definesurveyArgs = psMetadataAlloc(); + psMetadataAddS32(definesurveyArgs, PS_LIST_TAIL, "-set_surveyID", 0, "define survey ID (required)", 0); + psMetadataAddStr(definesurveyArgs, PS_LIST_TAIL, "-set_surveyName", 0, "define survey name (required)", NULL); + psMetadataAddStr(definesurveyArgs, PS_LIST_TAIL, "-set_description", 0, "define survey description", NULL); + + // -listsurvey + psMetadata *listsurveyArgs = psMetadataAlloc(); + psMetadataAddStr(listsurveyArgs, PS_LIST_TAIL, "-surveyName", 0, "select by survey name (LIKE comparision)", NULL); + psMetadataAddU64(listsurveyArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listsurveyArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -definerelease + psMetadata *definereleaseArgs = psMetadataAlloc(); + psMetadataAddS32(definereleaseArgs, PS_LIST_TAIL, "-set_surveyID", 0, "define survey ID (required)", 0); + psMetadataAddStr(definereleaseArgs, PS_LIST_TAIL, "-set_release_name", 0, "define release name (required)", NULL); + psMetadataAddS32(definereleaseArgs, PS_LIST_TAIL, "-set_dataRelease", 0, "define dataRelease", -1); + psMetadataAddS32(definereleaseArgs, PS_LIST_TAIL, "-set_accessLevelMin", 0, "define accessLevelMin", 0); + psMetadataAddStr(definereleaseArgs, PS_LIST_TAIL, "-set_release_state", 0, "define release state", NULL); + psMetadataAddS32(definereleaseArgs, PS_LIST_TAIL, "-set_priority", 0, "define release priority", 0); + psMetadataAddStr(definereleaseArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define dvo db name", NULL); + psMetadataAddStr(definereleaseArgs, PS_LIST_TAIL, "-set_ubercal_file", 0, "define ubercal file name", NULL); + + // -updaterelease + psMetadata *updatereleaseArgs = psMetadataAlloc(); + + // release_name and/or rel_id is required. This is handled in updatereleaseMode + psMetadataAddStr(updatereleaseArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name", NULL); + psMetadataAddS32(updatereleaseArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + + psMetadataAddStr(updatereleaseArgs, PS_LIST_TAIL, "-set_release_state", 0, "define new release state", NULL); + psMetadataAddS32(updatereleaseArgs, PS_LIST_TAIL, "-set_priority", 0, "define new release priority", 0); + psMetadataAddS32(updatereleaseArgs, PS_LIST_TAIL, "-set_dataRelease", 0, "define data release", -1); + psMetadataAddS32(updatereleaseArgs, PS_LIST_TAIL, "-set_accessLevelMin", 0, "define accessLevelMin", -1); + psMetadataAddStr(updatereleaseArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define new dvo db name", NULL); + psMetadataAddStr(updatereleaseArgs, PS_LIST_TAIL, "-set_ubercal_file", 0, "define new ubercal file name", NULL); + + + // -listrelease + psMetadata *listreleaseArgs = psMetadataAlloc(); + psMetadataAddStr(listreleaseArgs, PS_LIST_TAIL, "-surveyName", 0, "select by survey name (LIKE comparision)", NULL); + psMetadataAddStr(listreleaseArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + psMetadataAddStr(listreleaseArgs, PS_LIST_TAIL, "-release_state", 0, "select by release state", NULL); + psMetadataAddS32(listreleaseArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + + psMetadataAddU64(listreleaseArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listreleaseArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -definerelexp + psMetadata *definerelexpArgs = psMetadataAlloc(); + + // set the target release + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-release_name", 0, "define release name", NULL); + psMetadataAddS32(definerelexpArgs, PS_LIST_TAIL, "-rel_id", 0, "define release ID", 0); + + // select the processing + psMetadataAddS64(definerelexpArgs, PS_LIST_TAIL, "-cam_id", 0, "select by cam ID", 0); + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-label", 0, "select by camRun.label", NULL); + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-data_group", 0, "select by camRun.data_group", NULL); + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-filter", 0, "select by filter", NULL); + // select a specific exposure from the "processing" + psMetadataAddS64(definerelexpArgs, PS_LIST_TAIL, "-exp_id", 0, "select by exposure ID", 0); + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-exp_name", 0, "select by exposure name", NULL); + + // parameters of the relExp + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-set_state", 0, "define state (required)", NULL); + psMetadataAddU32(definerelexpArgs, PS_LIST_TAIL, "-set_flags", 0, "define flags", 0); + psMetadataAddF32(definerelexpArgs, PS_LIST_TAIL, "-set_zpt_obs", 0, "define zero point", NAN); + psMetadataAddF32(definerelexpArgs, PS_LIST_TAIL, "-set_zpt_stdev", 0, "define zero point stdev", NAN); + psMetadataAddF32(definerelexpArgs, PS_LIST_TAIL, "-set_mcal", 0, "define mcal", NAN); + psMetadataAddS32(definerelexpArgs, PS_LIST_TAIL, "-set_ubercal_dist", 0, "define ubercal dist", 0); + psMetadataAddStr(definerelexpArgs, PS_LIST_TAIL, "-set_path_base", 0, "define state", NULL); + psMetadataAddS16(definerelexpArgs, PS_LIST_TAIL, "-set_fault", 0, "define fault", 0); + + psMetadataAddBool(definerelexpArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definerelexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterelexp + psMetadata *updaterelexpArgs = psMetadataAlloc(); + psMetadataAddS64(updaterelexpArgs, PS_LIST_TAIL, "-relexp_id", 0, "select by released exposure ID", 0); + psMetadataAddS64(updaterelexpArgs, PS_LIST_TAIL, "-exp_id", 0, "select by exposure ID", 0); + psMetadataAddStr(updaterelexpArgs, PS_LIST_TAIL, "-exp_name", 0, "select by exposure name", NULL); + psMetadataAddS32(updaterelexpArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + psMetadataAddStr(updaterelexpArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + + psMetadataAddStr(updaterelexpArgs, PS_LIST_TAIL, "-set_state", 0, "define new state)", NULL); + psMetadataAddU32(updaterelexpArgs, PS_LIST_TAIL, "-set_flags", 0, "define new flags", 0); + psMetadataAddF32(updaterelexpArgs, PS_LIST_TAIL, "-set_zpt_obs", 0, "define new zero point", NAN); + psMetadataAddF32(updaterelexpArgs, PS_LIST_TAIL, "-set_zpt_stdev", 0, "define new zero point stdev", NAN); + psMetadataAddF32(updaterelexpArgs, PS_LIST_TAIL, "-set_mcal", 0, "define mcal", NAN); + psMetadataAddS32(updaterelexpArgs, PS_LIST_TAIL, "-set_ubercal_dist", 0, "define new ubercal dist", 0); + psMetadataAddStr(updaterelexpArgs, PS_LIST_TAIL, "-set_path_base", 0, "define path_base", NULL); + psMetadataAddS16(updaterelexpArgs, PS_LIST_TAIL, "-set_fault", 0, "define new fault", 0); + psMetadataAddBool(updaterelexpArgs, PS_LIST_TAIL, "-clearfault", 0, "set fault to zero", false); + + // -tocalibexp + psMetadata *tocalibexpArgs = psMetadataAlloc(); + psMetadataAddS64(tocalibexpArgs, PS_LIST_TAIL, "-relexp_id", 0, "select by released exposure ID", 0); + psMetadataAddStr(tocalibexpArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + + psMetadataAddU64(tocalibexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tocalibexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -listrelexp + psMetadata *listrelexpArgs = psMetadataAlloc(); + psMetadataAddS64(listrelexpArgs, PS_LIST_TAIL, "-relexp_id", 0, "select by released exposure ID", 0); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-release_state", PS_META_DUPLICATE_OK, "select by release state", NULL); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-state", 0, "select by released exposure state", NULL); + + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-filter", 0, "select by exposure name (LIKE comparison)", NULL); + psMetadataAddTime(listrelexpArgs, PS_LIST_TAIL, "-dateobs_begin", 0,"search for exposures by time (>=)", NULL); + psMetadataAddTime(listrelexpArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + + pxspaceAddArguments(listrelexpArgs); + + psMetadataAddF32(listrelexpArgs, PS_LIST_TAIL, "-fwhm_min", 0, "search by measured seeing (>=)", NAN); + psMetadataAddF32(listrelexpArgs, PS_LIST_TAIL, "-fwhm_max", 0, "search by seeing (<=)", NAN); + + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-exp_name", 0, "select by exposure name", NULL); + + psMetadataAddS64(listrelexpArgs, PS_LIST_TAIL, "-exp_id", 0, "select by exposure ID", 0); + psMetadataAddS64(listrelexpArgs, PS_LIST_TAIL, "-chip_id", 0, "select by chip ID", 0); + psMetadataAddS64(listrelexpArgs, PS_LIST_TAIL, "-cam_id", 0, "select by cam ID", 0); + psMetadataAddS64(listrelexpArgs, PS_LIST_TAIL, "-warp_id", 0, "select by warp ID", 0); + + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-chip_data_group", 0, "chipRun.data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-cam_data_group", 0, "camRun.data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-warp_data_group", 0, "warpRun.data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-skycell_id", 0, "select by skycell", NULL); + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-tess_id", 0, "warpRun.tess_id", NULL); + + psMetadataAddStr(listrelexpArgs, PS_LIST_TAIL, "-surveyName", 0, "select by survey name (LIKE comparision)", NULL); + psMetadataAddS64(listrelexpArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + + psMetadataAddBool(listrelexpArgs, PS_LIST_TAIL, "-priority_order", 0, "order by release priority", false); + + psMetadataAddU64(listrelexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listrelexpArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -deleterelexp + psMetadata *deleterelexpArgs = psMetadataAlloc(); + psMetadataAddS64(deleterelexpArgs, PS_LIST_TAIL, "-relexp_id", 0, "select by released exposure ID", 0); + psMetadataAddS64(deleterelexpArgs, PS_LIST_TAIL, "-exp_id", 0, "select by exposure ID", 0); + psMetadataAddStr(deleterelexpArgs, PS_LIST_TAIL, "-exp_name", 0, "select by exposure name", NULL); + psMetadataAddS32(deleterelexpArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + psMetadataAddS32(deleterelexpArgs, PS_LIST_TAIL, "-cam_id", 0, "select by cam run ID", 0); + psMetadataAddStr(deleterelexpArgs, PS_LIST_TAIL, "-label", 0, "select by camRun.label", NULL); + psMetadataAddStr(deleterelexpArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + + + // -definerelstack + psMetadata *definerelstackArgs = psMetadataAlloc(); + + // set the target release + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-release_name", 0, "define release name", NULL); + psMetadataAddS32(definerelstackArgs, PS_LIST_TAIL, "-rel_id", 0, "define release ID", 0); + + // select the processing + psMetadataAddS64(definerelstackArgs, PS_LIST_TAIL, "-stack_id", 0, "select by stack ID", 0); + psMetadataAddS64(definerelstackArgs, PS_LIST_TAIL, "-skycal_id", 0, "select by skycal ID", 0); + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-label", 0, "select by stackRun.label", NULL); + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-data_group", 0, "select by stackRun.data_group", NULL); + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-skycal_label", 0, "select by skycalRun.label", NULL); + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-skycal_data_group", 0, "select by skycalRun.data_group", NULL); + + // parameters of the relStack + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-set_state", 0, "define state (required)", NULL); + psMetadataAddU32(definerelstackArgs, PS_LIST_TAIL, "-set_flags", 0, "define flags", 0); + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-set_stack_type", 0, "define stack type (required)", NULL); + psMetadataAddF32(definerelstackArgs, PS_LIST_TAIL, "-set_zpt_obs", 0, "define zero point", NAN); + psMetadataAddF32(definerelstackArgs, PS_LIST_TAIL, "-set_zpt_stdev", 0, "define zero point stdev", NAN); + psMetadataAddF32(definerelstackArgs, PS_LIST_TAIL, "-set_fwhm_major", 0, "define fwhm_major", NAN); + psMetadataAddStr(definerelstackArgs, PS_LIST_TAIL, "-set_path_base", 0, "define state", NULL); + psMetadataAddS16(definerelstackArgs, PS_LIST_TAIL, "-set_fault", 0, "define fault", 0); + + psMetadataAddBool(definerelstackArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definerelstackArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(definerelstackArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -listrelstack + psMetadata *listrelstackArgs = psMetadataAlloc(); + psMetadataAddS64(listrelstackArgs, PS_LIST_TAIL, "-relstack_id", 0, "select by released exposure ID", 0); + psMetadataAddS64(listrelstackArgs, PS_LIST_TAIL, "-stack_id", 0, "select by stack ID", 0); + psMetadataAddS64(listrelstackArgs, PS_LIST_TAIL, "-skycal_id", 0, "select by skycal ID", 0); + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-release_state", PS_META_DUPLICATE_OK, "select by release state", NULL); + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-state", 0, "select by released stack state", NULL); + + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-filter", 0, "select by filter name (LIKE comparison)", NULL); + // psMetadataAddTime(listrelstackArgs, PS_LIST_TAIL, "-dateobs_begin", 0,"search for exposures by time (>=)", NULL); + // psMetadataAddTime(listrelstackArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + + pxskycellAddArguments(listrelstackArgs); + + psMetadataAddF32(listrelstackArgs, PS_LIST_TAIL, "-fwhm_min", 0, "search by measured seeing (>=)", NAN); + psMetadataAddF32(listrelstackArgs, PS_LIST_TAIL, "-fwhm_max", 0, "search by seeing (<=)", NAN); + + psMetadataAddF32(listrelstackArgs, PS_LIST_TAIL, "-mjd_min", 0, "search by MJD seeing (>=)", NAN); + psMetadataAddF32(listrelstackArgs, PS_LIST_TAIL, "-mjd_max", 0, "search by MJD (<=)", NAN); + + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-tess_id", 0, "select by tess_id", NULL); + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-skycell_id", 0, "select by skycell_id (LIKE comparision)", NULL); + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-stack_type", PS_META_DUPLICATE_OK, "select by stack_type", NULL); + + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-stack_data_group", 0, "select by stackRun.data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-skycal_data_group", 0, "select by skycalRun.data_group (LIKE comparison)", NULL); + + psMetadataAddStr(listrelstackArgs, PS_LIST_TAIL, "-surveyName", 0, "select by survey name (LIKE comparision)", NULL); + psMetadataAddS64(listrelstackArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + + psMetadataAddBool(listrelstackArgs, PS_LIST_TAIL, "-priority_order", 0, "order by release priority", false); + + psMetadataAddU64(listrelstackArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listrelstackArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *calibraterelstackArgs = psMetadataAlloc(); + + // set the target release and collection of relStacks (one of these is required) + psMetadataAddStr(calibraterelstackArgs, PS_LIST_TAIL, "-release_name", 0, "define release name", NULL); + psMetadataAddS32(calibraterelstackArgs, PS_LIST_TAIL, "-rel_id", 0, "define release ID", 0); + + // select the processing + psMetadataAddStr(calibraterelstackArgs, PS_LIST_TAIL, "-skycal_label", 0, "select by skycalRun.label", NULL); + psMetadataAddStr(calibraterelstackArgs, PS_LIST_TAIL, "-skycal_data_group", 0, "select by skycalRun.data_group", NULL); + psMetadataAddS64(calibraterelstackArgs, PS_LIST_TAIL, "-skycal_id", 0, "select by skycalRun.skycal_id", 0); + + psMetadataAddStr(calibraterelstackArgs, PS_LIST_TAIL, "-tess_id", 0, "select by tess_id", NULL); + psMetadataAddStr(calibraterelstackArgs, PS_LIST_TAIL, "-skycell_id", 0, "select by skycell_id", NULL); + pxskycellAddArguments(calibraterelstackArgs); + + psMetadataAddBool(calibraterelstackArgs, PS_LIST_TAIL, "-replace", 0, "replace any existing calibrations", false); + psMetadataAddU64(calibraterelstackArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -deleterelstack + psMetadata *deleterelstackArgs = psMetadataAlloc(); + psMetadataAddS64(deleterelstackArgs, PS_LIST_TAIL, "-relstack_id", 0, "select by released exposure ID", 0); + psMetadataAddS64(deleterelstackArgs, PS_LIST_TAIL, "-stack_id", 0, "select by stack ID", 0); +// psMetadataAddS64(deleterelstackArgs, PS_LIST_TAIL, "-skycal_id", 0, "select by skycal ID", 0); + psMetadataAddStr(deleterelstackArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + psMetadataAddS32(deleterelstackArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + + psMetadataAddStr(deleterelstackArgs, PS_LIST_TAIL, "-label", 0, "select by stackRun.label", NULL); +// psMetadataAddStr(deleterelstackArgs, PS_LIST_TAIL, "-data_group", 0, "select by stackRun.data_group", NULL); +// psMetadataAddStr(deleterelstackArgs, PS_LIST_TAIL, "-skycal_label", 0, "select by skycalRun.label", NULL); +// psMetadataAddStr(deleterelstackArgs, PS_LIST_TAIL, "-skycal_data_group", 0, "select by skycalRun.data_group", NULL); + + // -summary + psMetadata *summaryArgs = psMetadataAlloc(); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-sass_id", 0, "select by released SASS ID", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-relstack_id", 0, "select by released exposure ID", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-stack_id", 0, "select by stack ID", 0); +// psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-skycal_id", 0, "select by skycal ID", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name (LIKE comparision)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-release_state", PS_META_DUPLICATE_OK, "select by release state", NULL); +// psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-state", 0, "select by released stack state", NULL); + + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-filter", 0, "select by filter name (LIKE comparison)", NULL); + + pxskycellAddArguments(summaryArgs); + + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-tess_id", 0, "select by tess_id", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-skycell_id", 0, "select by skycell_id (LIKE comparision)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-projection_cell", 0, "select by projection_cell (LIKE comparision)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-stack_type", PS_META_DUPLICATE_OK, "select by stack_type", NULL); + + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-stack_data_group", 0, "select by stackRun.data_group (LIKE comparison)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-data_group", 0, "select by stackAsociation.data_group (LIKE comparison)", NULL); + + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-surveyName", 0, "select by survey name (LIKE comparision)", NULL); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + + psMetadataAddBool(summaryArgs, PS_LIST_TAIL, "-priority_order", 0, "order by release priority", false); + + psMetadataAddU64(summaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(summaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -definerelgroup + psMetadata *definerelgroupArgs = psMetadataAlloc(); + + // set the target release + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-release_name", 0, "define release name", NULL); + psMetadataAddS32(definerelgroupArgs, PS_LIST_TAIL, "-rel_id", 0, "define release ID", 0); + + // select the members of the group + psMetadataAddS64(definerelgroupArgs, PS_LIST_TAIL, "-select_seq_id", 0, "select by LAP sequence ID", 0); + psMetadataAddS64(definerelgroupArgs, PS_LIST_TAIL, "-select_lap_id", 0, "select by LAP ID", 0); + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-select_data_group", PS_META_DUPLICATE_OK, "select by camRun.data_group", NULL); + // do we want this? + // psMetadataAddS64(definerelgroupArgs, PS_LIST_TAIL, "-lap_id", 0, "select by LAP ID", NULL); + + // parameters of the relGroup + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-set_group_type", 0, "define group_type (required)", NULL); + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-set_group_name", 0, "define group_name", NULL); + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-set_label", 0, "define relgroup label (required)", NULL); + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + psMetadataAddStr(definerelgroupArgs, PS_LIST_TAIL, "-set_exp_list_path", 0, "define path in exposure list", NULL); + psMetadataAddS16(definerelgroupArgs, PS_LIST_TAIL, "-set_fault", 0, "define fault", 0); + + // options + psMetadataAddBool(definerelgroupArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definerelgroupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingrelgroup + psMetadata *pendingrelgroupArgs = psMetadataAlloc(); + + psMetadataAddStr(pendingrelgroupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by relGroup.label", NULL); + psMetadataAddS32(pendingrelgroupArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + psMetadataAddS32(pendingrelgroupArgs, PS_LIST_TAIL, "-group_id", 0, "select by relGroup ID", 0); + psMetadataAddStr(pendingrelgroupArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name", 0); + psMetadataAddBool(pendingrelgroupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingrelgroupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -updaterelgroup + psMetadata *updaterelgroupArgs = psMetadataAlloc(); + + psMetadataAddS32(updaterelgroupArgs, PS_LIST_TAIL, "-group_id", 0, "select by group ID", 0); + psMetadataAddStr(updaterelgroupArgs, PS_LIST_TAIL, "-group_name", 0, "select by group name", 0); + + // parameters of the relGroup + psMetadataAddStr(updaterelgroupArgs, PS_LIST_TAIL, "-set_state", 0, "define state", NULL); + psMetadataAddStr(updaterelgroupArgs, PS_LIST_TAIL, "-set_exp_list_path", 0, "define path in exposure list", NULL); + psMetadataAddStr(updaterelgroupArgs, PS_LIST_TAIL, "-set_label", 0, "define relgroup label", NULL); + psMetadataAddS16(updaterelgroupArgs, PS_LIST_TAIL, "-set_fault", 0, "define fault", 0); + psMetadataAddBool(updaterelgroupArgs, PS_LIST_TAIL, "-clearfault", 0, "set fault to zero", false); + + psMetadata *listrelgroupArgs = psMetadataAlloc(); + psMetadataAddS32(listrelgroupArgs, PS_LIST_TAIL, "-group_id", 0, "select by group ID", 0); + psMetadataAddS32(listrelgroupArgs, PS_LIST_TAIL, "-rel_id", 0, "select by release ID", 0); + psMetadataAddStr(listrelgroupArgs, PS_LIST_TAIL, "-release_name", 0, "select by release name", 0); + + // ****************************************************************** + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definesurvey", "define a survey", RELEASETOOL_MODE_DEFINESURVEY, definesurveyArgs); + PXOPT_ADD_MODE("-listsurvey", "list skycell parameters", RELEASETOOL_MODE_LISTSURVEY, listsurveyArgs); + + PXOPT_ADD_MODE("-definerelease", "define a release", RELEASETOOL_MODE_DEFINERELEASE, definereleaseArgs); + PXOPT_ADD_MODE("-updaterelease", "update a release", RELEASETOOL_MODE_UPDATERELEASE, updatereleaseArgs); + PXOPT_ADD_MODE("-listrelease", "list releases", RELEASETOOL_MODE_LISTRELEASE, listreleaseArgs); + + PXOPT_ADD_MODE("-definerelexp", "define a released exposure", RELEASETOOL_MODE_DEFINERELEXP, definerelexpArgs); + PXOPT_ADD_MODE("-updaterelexp", "update a released exposure", RELEASETOOL_MODE_UPDATERELEXP, updaterelexpArgs); + PXOPT_ADD_MODE("-tocalibexp", "list relExps pending calibration", RELEASETOOL_MODE_TOCALIBEXP, tocalibexpArgs); + PXOPT_ADD_MODE("-listrelexp", "list released exposures", RELEASETOOL_MODE_LISTRELEXP, listrelexpArgs); + PXOPT_ADD_MODE("-deleterelexp", "delete a released exposure", RELEASETOOL_MODE_DELETERELEXP, deleterelexpArgs); + + PXOPT_ADD_MODE("-definerelstack", "define a released stack", RELEASETOOL_MODE_DEFINERELSTACK, definerelstackArgs); + PXOPT_ADD_MODE("-listrelstack", "list released stacks", RELEASETOOL_MODE_LISTRELSTACK, listrelstackArgs); + PXOPT_ADD_MODE("-setrelstackcalibratedfromskycal", "update parameters of using skycalResults", RELEASETOOL_MODE_SETRELSTACKCALIBRATEDFROMSKYCAL, calibraterelstackArgs); + PXOPT_ADD_MODE("-deleterelstack", "update parameters of released stacks", RELEASETOOL_MODE_DELETERELSTACK, deleterelstackArgs); + + PXOPT_ADD_MODE("-summary", "list stackSummaryes for released stacks", RELEASETOOL_MODE_SUMMARY, summaryArgs); + + PXOPT_ADD_MODE("-definerelgroup", "define a group of exposures", RELEASETOOL_MODE_DEFINERELGROUP, definerelgroupArgs); + PXOPT_ADD_MODE("-pendingrelgroup", "list relGroups pending processing", RELEASETOOL_MODE_PENDINGRELGROUP, pendingrelgroupArgs); + PXOPT_ADD_MODE("-updaterelgroup", "update a relGroup", RELEASETOOL_MODE_UPDATERELGROUP, updaterelgroupArgs); + PXOPT_ADD_MODE("-listrelgroup", "define a group of exposures", RELEASETOOL_MODE_LISTRELGROUP, listrelgroupArgs); + + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetool.c (revision 41104) @@ -0,0 +1,669 @@ +/* + * remotetool.c + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "remotetool.h" + +// Create +static bool definebyqueryMode(pxConfig *config); + +// List +static bool listrunMode(pxConfig *config); +static bool listcomponentMode(pxConfig *config); + +// list remoteRuns with all remoteComponents in state 'prep_done' or 'prep_fail' +static bool doneprepMode(pxConfig *config); + +// Update +static bool dropcomponentMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool updatepollMode(pxConfig *config); +static bool updatecomponentMode(pxConfig *config); +static bool revertrunMode(pxConfig *config); +static bool revertauthMode(pxConfig *config); +static bool revertcomponentMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = remotetoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(REMOTETOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(REMOTETOOL_MODE_LISTRUN, listrunMode); + MODECASE(REMOTETOOL_MODE_DONEPREP, doneprepMode); + MODECASE(REMOTETOOL_MODE_LISTCOMPONENT, listcomponentMode); + MODECASE(REMOTETOOL_MODE_DROPCOMPONENT, dropcomponentMode); + MODECASE(REMOTETOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(REMOTETOOL_MODE_UPDATEPOLL, updatepollMode); + MODECASE(REMOTETOOL_MODE_UPDATECOMPONENT, updatecomponentMode); + MODECASE(REMOTETOOL_MODE_REVERTRUN, revertrunMode); + MODECASE(REMOTETOOL_MODE_REVERTAUTH, revertauthMode); + MODECASE(REMOTETOOL_MODE_REVERTCOMPONENT, revertcomponentMode); + + default: + psAbort("invalid option (this should not happen)"); + } + psTrace("remotetool", 9, "Attempting to free config\n"); + psFree(config); + pmConfigDone(); + psLibFinalize(); + exit(EXIT_SUCCESS); + + FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +// Create + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + // These are required + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + PXOPT_LOOKUP_STR(stage, config->args, "-stage", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + // These are not + PXOPT_LOOKUP_STR(set_label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_S16(limit, config->args, "-limit", false, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + set_label = set_label ? set_label : label; + + // Get a list of the things we can insert into a new component + // CZW: these labels no longer need to be linked to the table, as I've split stageRun and remoteRun into sub-queries. + psString query = NULL; + psString whereOption = psStringCopy(""); + if (!strcmp(stage,"chip")) { + query = pxDataGet("remotetool_definebyquery_chip.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else if (!strcmp(stage,"camera")) { + query = pxDataGet("remotetool_definebyquery_cam.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else if (!strcmp(stage,"warp")) { + query = pxDataGet("remotetool_definebyquery_warp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + // Because warp has to check to see if something other than the state is set correctly, I have to use this wonky way to pass the label in. + // psStringAppend(&query, " AND (warpRun.label = '%s')", label); + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else if (!strcmp(stage,"stack")) { + query = pxDataGet("remotetool_definebyquery_stack.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else if (!strcmp(stage,"staticsky")) { + query = pxDataGet("remotetool_definebyquery_staticsky.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else if (!strcmp(stage,"diff")) { + query = pxDataGet("remotetool_definebyquery_diff.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else if (!strcmp(stage,"ff")) { + query = pxDataGet("remotetool_definebyquery_ff.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (label) { + psStringAppend(&whereOption, "\n AND (label = '%s')", label); + } + } + else { + psError(PS_ERR_UNKNOWN, true, "unknown value for stage: %s", stage); + return false; + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + if (!p_psDBRunQueryF(config->dbh, query,whereOption,whereOption)) { // This needs to be here twice because we need to restrict on label twice. + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("remotetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + if (!ippdbPrintMetadatas(stdout, output, "newremoteComponents", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + + // Insert the top level run object + remoteRunRow *run = remoteRunRowAlloc(0, // remote_id + "new", // state + stage, + set_label, + path_base, + -1, // job_id + NULL, // last_poll + 0 // fault + ); + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc remoteRun object"); + return(true); + } + + if (!remoteRunInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return(true); + } + psS64 remote_id = psDBLastInsertID(config->dbh); + + psArray *list = psArrayAllocEmpty(limit); + for (long i=0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + psS64 stage_id = psMetadataLookupS64(NULL, md, "stage_id"); + + remoteComponentRow *comp = remoteComponentRowAlloc(remote_id, stage_id, 0, "new", NULL); + if (!comp) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc remoteComponent object"); + return(true); + } + + if (!remoteComponentInsertObject(config->dbh, comp)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return(true); + } + + psArrayAdd(list, list->n, comp); + psFree(comp); + + } + psFree(run); + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!remoteComponentPrintObjects(stdout, list, !simple)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + psFree(list); + psFree(output); + + return true; +} + +// List +static bool listrunMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-poll_begin", "last_poll", ">="); + PXOPT_COPY_TIME(config->args, where, "-poll_end", "last_poll", "<="); + + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + + PXOPT_LOOKUP_S16(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psArray *remRuns = NULL; + remRuns = psDBSelectRows(config->dbh, "remoteRun", where, limit); + + if (!ippdbPrintMetadatas(stdout, remRuns, "remoteRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(remRuns); + return false; + } + psFree(remRuns); + + return true; +} + +static bool listcomponentMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_S64(config->args, where, "-jobs", "jobs", "=="); + + PXOPT_COPY_STR(config->args, where, "-state", "remoteComponent.state", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "remoteRun.label", "=="); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_S16(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("remotetool_listcomponent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("remotetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "remoteComponent", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +// List +static bool doneprepMode(pxConfig *config) +{ + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_S64(config->args, where, "-job_id", "job_id", "=="); + + PXOPT_COPY_STR(config->args, where, "-stage", "stage", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "remoteRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + PXOPT_LOOKUP_S16(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("remotetool_doneprep.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + return(false); + } + + psString whereDone = NULL; + psString whereJobs = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereDone, " \n AND %s", whereClause); + psStringAppend(&whereJobs, " \n WHERE %s", whereClause); + psFree(whereClause); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + // fprintf (stderr, "query: %s\n", query); + // fprintf (stderr, "whereDone: %s\n", whereDone); + // fprintf (stderr, "whereJobs: %s\n", whereJobs); + + if (!p_psDBRunQueryF(config->dbh, query, whereDone, whereJobs)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereDone); + psFree(whereJobs); + psFree(query); + return false; + } + psFree(whereDone); + psFree(whereJobs); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("remotetool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "remoteRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +// Update + +static bool dropcomponentMode(pxConfig *config) +{ + fprintf (stderr, "do we really want to use -dropcomponent?\n"); + exit (1); + + // Assert that we have a unique component to operate on. + PXOPT_LOOKUP_S64(remote_id,config->args, "-remote_id",true, false); + psAssert(remote_id,"This must exist for this mode to be safe."); + PXOPT_LOOKUP_S64(stage_id, config->args, "-stage_id", true, false); + psAssert(stage_id,"This must exist for this mode to be safe."); + + psString query = pxDataGet("remotetool_dropcomponent.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "Failed to retreive SQL statement"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!p_psDBRunQueryF(config->dbh, query, remote_id, stage_id)) { + psError(PS_ERR_UNKNOWN, false, "Database error"); + psFree(query); + return false; + } + psFree(query); + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_S64(remote_id, config->args, "-remote_id", true, false); + psAssert(remote_id,"This should have failed before this point."); + + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + + // Wheres + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + + // Values to set + PXOPT_COPY_STR(config->args, values, "-set_label", "label", "=="); + PXOPT_COPY_STR(config->args, values, "-set_state", "state", "=="); + PXOPT_COPY_S16(config->args, values, "-fault", "fault", "=="); + PXOPT_COPY_S64(config->args, values, "-job_id", "job_id", "=="); + + long rows = psDBUpdateRows(config->dbh, "remoteRun", where, values); + psFree(values); + + if (rows < 1) fprintf (stderr, "no rows changed (run may already be in desired state)\n"); + return(true); +} + +static bool updatepollMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_S64(remote_id, config->args, "-remote_id", true, false); + psAssert(remote_id,"This should have failed before this point."); + + psMetadata *where = psMetadataAlloc(); + // Wheres + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + + psString query = NULL; + query = pxDataGet("remotetool_updatepoll.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS,false, "failed to retreive SQL statement"); + psFree(whereClause); + return(false); + } + + if (!p_psDBRunQueryF(config->dbh, query,whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + psFree(whereClause); + return false; + } + psFree(query); + psFree(whereClause); + return(true); +} + +static bool updatecomponentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_S64(remote_id, config->args, "-remote_id", true, false); + psAssert(remote_id,"This should have failed before this point."); + + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + + // Wheres + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + + // Values to set + PXOPT_COPY_S32(config->args, values, "-set_jobs", "jobs", "=="); + PXOPT_COPY_STR(config->args, values, "-set_path_base", "path_base", "=="); + PXOPT_COPY_STR(config->args, values, "-set_state", "state", "=="); + + long rows = psDBUpdateRows(config->dbh, "remoteComponent", where, values); + psFree(values); + + if (rows < 1) fprintf (stderr, "no rows changed (component may already be in desired state)\n"); + return true; +} + +static bool revertrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + + // required + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "fault", "=="); + // Value to set + psMetadataAddS16(values,PS_LIST_TAIL,"fault",0, "==", 0); + + long rows = psDBUpdateRows(config->dbh, "remoteRun", where, values); + psFree(values); + psFree(where); + if (rows < 1) { + psError(PS_ERR_UNKNOWN,false, "no rows affected"); + return(false); + } + + return(true); +} + +static bool revertcomponentMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config,false); + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + + // required + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stage_id", "stage_id", "=="); + psMetadataAddStr(where,PS_LIST_TAIL,"state",0, "LIKE", "%fail"); + + // value to set + psMetadataAddStr(values, PS_LIST_TAIL, "state",0, "==", "retry"); + + + long rows = psDBUpdateRows(config->dbh, "remoteComponent", where, values); + psFree(values); + psFree(where); + if (rows < 1) { + psError(PS_ERR_UNKNOWN,false, "no rows affected"); + return(false); + } + + return(true); +} + + + +static bool revertauthMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + + // required + PXOPT_COPY_S64(config->args, where, "-remote_id", "remote_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psMetadataAddStr(where, PS_LIST_TAIL, "state", 0, "==", "auth"); + + // Value to set + psMetadataAddStr(values, PS_LIST_TAIL, "state", 0, "==", "run"); + + long rows = psDBUpdateRows(config->dbh, "remoteRun", where, values); + psFree(values); + psFree(where); + if (rows < 1) { + psError(PS_ERR_UNKNOWN,false, "no rows affected"); + return(false); + } + + return(true); +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetool.h (revision 41104) @@ -0,0 +1,27 @@ +/* + * remotetool.h + */ + +#ifndef REMOTETOOL_H +#define REMOTETOOL_H 1 + +#include "pxtools.h" + +typedef enum { + REMOTETOOL_MODE_NONE = 0x0, + REMOTETOOL_MODE_DEFINEBYQUERY, + REMOTETOOL_MODE_LISTRUN, + REMOTETOOL_MODE_DONEPREP, + REMOTETOOL_MODE_LISTCOMPONENT, + REMOTETOOL_MODE_DROPCOMPONENT, + REMOTETOOL_MODE_UPDATERUN, + REMOTETOOL_MODE_UPDATEPOLL, + REMOTETOOL_MODE_UPDATECOMPONENT, + REMOTETOOL_MODE_REVERTRUN, + REMOTETOOL_MODE_REVERTAUTH, + REMOTETOOL_MODE_REVERTCOMPONENT, +} remotetoolMode; + +pxConfig *remotetoolConfig(pxConfig *config, int argc, char **argv); + +#endif // REMOTETOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/remotetoolConfig.c (revision 41104) @@ -0,0 +1,164 @@ +/* + * remotetoolConfig.c + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include "pxtools.h" +#include "remotetool.h" + +#define ADD_OPT(TYPE,TARG,NAME,COMMENT,DEFAULT) psMetadataAdd##TYPE(TARG, PS_LIST_TAIL, NAME, 0, COMMENT, DEFAULT) + +pxConfig *remotetoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + ADD_OPT(Str, definebyqueryArgs, "-label", "stage label to query", NULL); + ADD_OPT(Str, definebyqueryArgs, "-set_label", "remRun label to use", NULL); + ADD_OPT(Str, definebyqueryArgs, "-stage", "stage to query", NULL); + ADD_OPT(Str, definebyqueryArgs, "-path_base", "define base output location", NULL); + ADD_OPT(S16, definebyqueryArgs, "-limit", "limit to number of jobs to bundle",0); + ADD_OPT(Bool, definebyqueryArgs, "-pretend", "pretend to do this.", 0); + ADD_OPT(Bool, definebyqueryArgs,"-simple", "simple print format",0); + + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + ADD_OPT(S64, listrunArgs, "-remote_id", "run to list", 0); + ADD_OPT(Str, listrunArgs, "-state", "run state", NULL); + ADD_OPT(Str, listrunArgs, "-stage", "stage to return", NULL); + ADD_OPT(Str, listrunArgs, "-label", "remRun label to use", NULL); + + ADD_OPT(S64, listrunArgs, "-job_id", "job_id", 0); + // ADD_OPT(Str, listrunArgs, "-poll_begin", "last polled time begin", NULL); + // ADD_OPT(Str, listrunArgs, "-poll_end", "last polled time begin", NULL); + psMetadataAddTime(listrunArgs, PS_LIST_TAIL, "-poll_begin", 0, "last polled time begin", NULL); + psMetadataAddTime(listrunArgs, PS_LIST_TAIL, "-poll_end", 0, "last polled time end", NULL); + + ADD_OPT(S16, listrunArgs, "-fault", "fault code", 0); + + ADD_OPT(Bool, listrunArgs, "-simple", "simple print format", false); + ADD_OPT(S16, listrunArgs, "-limit", "limit to number of runs to display", 0); + + // -doneprep + psMetadata *doneprepArgs = psMetadataAlloc(); + ADD_OPT(S64, doneprepArgs, "-remote_id", "run to list", 0); + ADD_OPT(S64, doneprepArgs, "-job_id", "job_id", 0); + ADD_OPT(Str, doneprepArgs, "-stage", "stage to return", NULL); + ADD_OPT(Str, doneprepArgs, "-state", "state to return", NULL); + ADD_OPT(Str, doneprepArgs, "-label", "remRun label to use", NULL); + + ADD_OPT(Bool, doneprepArgs, "-simple", "simple print format", false); + ADD_OPT(S16, doneprepArgs, "-limit", "limit to number of runs to display", 0); + + // -revertcomponent + psMetadata *revertcomponentArgs = psMetadataAlloc(); + ADD_OPT(S64, revertcomponentArgs, "-remote_id", "run to revert components from", 0); + ADD_OPT(S64, revertcomponentArgs, "-stage_id", "component stage_id to revert", 0); + + + // -listcomponents + psMetadata *listcomponentArgs = psMetadataAlloc(); + ADD_OPT(S64, listcomponentArgs, "-remote_id", "run to list", 0); + ADD_OPT(S64, listcomponentArgs, "-stage_id", "run to list", 0); + ADD_OPT(S32, listcomponentArgs, "-jobs", "run to list", 0); + ADD_OPT(Str, listcomponentArgs, "-state", "run to list", NULL); + ADD_OPT(S16, listcomponentArgs, "-limit", "limit to number of runs to display", 0); + ADD_OPT(Bool, listcomponentArgs, "-simple", "simple print format", false); + ADD_OPT(Str, listcomponentArgs, "-label", "label to use", NULL); + + // -dropcomponent + psMetadata *dropcomponentArgs = psMetadataAlloc(); + ADD_OPT(S64, dropcomponentArgs, "-remote_id", "run to update", 0); + ADD_OPT(S64, dropcomponentArgs, "-stage_id", "stage_id to remote", 0); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + ADD_OPT(S64, updaterunArgs, "-remote_id", "run to update", 0); + // ADD_OPT(Str, updaterunArgs, "-label", "label to update", NULL); + ADD_OPT(Str, updaterunArgs, "-set_label", "remRun label to use", NULL); + ADD_OPT(Str, updaterunArgs, "-set_state", "remoteRun state to assign", NULL); + ADD_OPT(S64, updaterunArgs, "-job_id", "job_id to set", 0); + ADD_OPT(S16, updaterunArgs, "-fault", "fault code", 0); + // ADD_OPT(Bool, updaterunArgs, "-poll", "set poll date", NULL); + + // -updatepoll + psMetadata *updatepollArgs = psMetadataAlloc(); + ADD_OPT(S64, updatepollArgs, "-remote_id", "run to update", 0); + + // -updatecomponent + psMetadata *updatecomponentArgs = psMetadataAlloc(); + ADD_OPT(S64, updatecomponentArgs, "-remote_id", "remote_id to update", 0); + ADD_OPT(S64, updatecomponentArgs, "-stage_id", "stage_id to update", 0); + ADD_OPT(Str, updatecomponentArgs, "-state", "limit to this remoteComponent state", NULL); + ADD_OPT(Str, updatecomponentArgs, "-set_state", "remoteComponent state to assign", NULL); + ADD_OPT(Str, updatecomponentArgs, "-set_path_base", "remoteRun path_base to assign", NULL); + ADD_OPT(S32, updatecomponentArgs, "-set_jobs", "number of jobs for this stage_id", 0); + + // -revertrun + psMetadata *revertrunArgs = psMetadataAlloc(); + ADD_OPT(S64, revertrunArgs, "-remote_id", "run to revert", 0); + ADD_OPT(Str, revertrunArgs, "-label", "remRun label to use", NULL); + ADD_OPT(S16, revertrunArgs, "-fault", "fault code", 0); + + // -revertauth + psMetadata *revertauthArgs = psMetadataAlloc(); + ADD_OPT(S64, revertauthArgs, "-remote_id", "run to revert", 0); + ADD_OPT(Str, revertauthArgs, "-label", "remRun label to use", NULL); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "", REMOTETOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-listrun", "", REMOTETOOL_MODE_LISTRUN, listrunArgs); + PXOPT_ADD_MODE("-doneprep", "", REMOTETOOL_MODE_DONEPREP, doneprepArgs); + PXOPT_ADD_MODE("-listcomponent", "", REMOTETOOL_MODE_LISTCOMPONENT, listcomponentArgs); + PXOPT_ADD_MODE("-revertcomponent", "", REMOTETOOL_MODE_REVERTCOMPONENT, revertcomponentArgs); + PXOPT_ADD_MODE("-dropcomponent", "", REMOTETOOL_MODE_DROPCOMPONENT, dropcomponentArgs); + PXOPT_ADD_MODE("-updaterun", "", REMOTETOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-updatepoll", "", REMOTETOOL_MODE_UPDATEPOLL, updatepollArgs); + PXOPT_ADD_MODE("-updatecomponent", "", REMOTETOOL_MODE_UPDATECOMPONENT, updatecomponentArgs); + PXOPT_ADD_MODE("-revertrun", "", REMOTETOOL_MODE_REVERTRUN, revertrunArgs); + PXOPT_ADD_MODE("-revertauth", "", REMOTETOOL_MODE_REVERTAUTH, revertauthArgs); + + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PXTOOLS_ERR_SYS, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} + Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/sctool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/sctool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/sctool.c (revision 41104) @@ -0,0 +1,198 @@ +/* + * sctool.c + * + * Copyright (C) 2012 IfA University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "pxchip.h" +#include "pxspace.h" + +#include "sctool.h" + +static bool defineskycellsMode(pxConfig *config); +static bool listMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = sctoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(SCTOOL_MODE_DEFINESKYCELLS, defineskycellsMode); + MODECASE(SCTOOL_MODE_LIST, listMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool defineskycellsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(inputFile, config->args, "-input", true, false); + + unsigned int numBad = 0; + psMetadata *skycells = psMetadataConfigRead(NULL, &numBad, inputFile, false); + if (!skycells || numBad > 0) { + psError(PXTOOLS_ERR_UNKNOWN, false, "failed to read %s", inputFile); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psMetadataIterator *iter = psMetadataIteratorAlloc(skycells, PS_LIST_HEAD, NULL); + + psMetadataItem *item; + int numSkycells = 0; + while ((item = psMetadataGetAndIncrement(iter))) { + if (item->type != PS_DATA_METADATA) { + psError(PS_ERR_UNKNOWN, true, "unexpected item in metadata type: %d", item->type); + return false; + } + psMetadata *md = item->data.md; + psString tess_id = psMetadataLookupStr(NULL, md, "tess_id"); + psString skycell_id = psMetadataLookupStr(NULL, md, "skycell_id"); + + skycellRow *skycell = skycellObjectFromMetadata(md); + if (!skycell) { + psError(PS_ERR_UNKNOWN, false, "failed to create skycellRow from Metadata"); + return false; + } + + numSkycells++; + if (!skycellInsertObject(config->dbh, skycell)) { + psError(PS_ERR_UNKNOWN, false, "failed to insert skycell %s %s", tess_id, skycell_id); + return false; + } + } + psFree(iter); + psFree(skycells); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psLogMsg("sctool", PS_LOG_INFO, "%d skycells added", numSkycells); + + return true; +} +static bool listMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-tess_id", "skycell.tess_id", "LIKE"); + pxskycellAddWhere(config, where); + + psString query = pxDataGet("sctool_list.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!psListLength(where->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search paramters are required\n"); + psFree(where); + return false; + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psArrayLength(output)) { + psTrace("sctool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (!ippdbPrintMetadatas(stdout, output, "skycell", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/sctool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/sctool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/sctool.h (revision 41104) @@ -0,0 +1,33 @@ +/* + * sctool.h + * + * Copyright (C) 2012 University of Hawaii IFA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef SCTOOL_H +#define SCTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + SCTOOL_MODE_NONE = 0x0, + SCTOOL_MODE_DEFINESKYCELLS, + SCTOOL_MODE_LIST, +} sctoolMode; + +pxConfig *sctoolConfig(pxConfig *config, int argc, char **argv); + +#endif // SCTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/sctoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/sctoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/sctoolConfig.c (revision 41104) @@ -0,0 +1,86 @@ +/* + * sctoolConfig.c + * + * Copyright (C) 2012 IfA University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "pxspace.h" +#include "sctool.h" + +pxConfig *sctoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -defineskycells + psMetadata *defineskycellsArgs = psMetadataAlloc(); + psMetadataAddStr(defineskycellsArgs, PS_LIST_TAIL, "-input", 0, "file containing skycell data (required)", NULL); + + // -list + psMetadata *listArgs = psMetadataAlloc(); + psMetadataAddStr(listArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id (LIKE comparision)", NULL); + psMetadataAddStr(listArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess_id (LIKE comparision)", NULL); + // add skycell coordinate search arguments + pxskycellAddArguments(listArgs); + psMetadataAddU64(listArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-defineskycells", "insert skycells from mdc file", SCTOOL_MODE_DEFINESKYCELLS, defineskycellsArgs); + PXOPT_ADD_MODE("-list", "list skycell parameters", SCTOOL_MODE_LIST, listArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktool.c (revision 41104) @@ -0,0 +1,2139 @@ +/* + * stacktool.c + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "stacktool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool definerunMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool addinputskyfileMode(pxConfig *config); +static bool inputskyfileMode(pxConfig *config); +static bool tosumMode(pxConfig *config); +static bool tobkgMode(pxConfig *config); +static bool addsumskyfileMode(pxConfig *config); +static bool sumskyfileMode(pxConfig *config); +static bool sassskyfileMode(pxConfig *config); +static bool updatesassMode(pxConfig *config); +static bool revertsumskyfileMode(pxConfig *config); +static bool tosummaryMode(pxConfig *config); +static bool addsummaryMode(pxConfig *config); +static bool revertsummaryMode(pxConfig *config); +static bool summaryMode(pxConfig *config); +static bool pendingcleanuprunMode(pxConfig *config); +static bool pendingcleanupskyfileMode(pxConfig *config); +static bool donecleanupMode(pxConfig *config); +static bool updatesumskyfileMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +static bool setstackRunState(pxConfig *config, psS64 stack_id, const char *state); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = stacktoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(STACKTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(STACKTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(STACKTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(STACKTOOL_MODE_ADDINPUTSKYFILE, addinputskyfileMode); + MODECASE(STACKTOOL_MODE_INPUTSKYFILE, inputskyfileMode); + MODECASE(STACKTOOL_MODE_TOSUM, tosumMode); + MODECASE(STACKTOOL_MODE_TOBKG, tobkgMode); + MODECASE(STACKTOOL_MODE_ADDSUMSKYFILE, addsumskyfileMode); + MODECASE(STACKTOOL_MODE_SUMSKYFILE, sumskyfileMode); + MODECASE(STACKTOOL_MODE_SASSSKYFILE, sassskyfileMode); + MODECASE(STACKTOOL_MODE_UPDATESASS, updatesassMode); + MODECASE(STACKTOOL_MODE_REVERTSUMSKYFILE, revertsumskyfileMode); + MODECASE(STACKTOOL_MODE_TOSUMMARY, tosummaryMode); + MODECASE(STACKTOOL_MODE_ADDSUMMARY, addsummaryMode); + MODECASE(STACKTOOL_MODE_REVERTSUMMARY, revertsummaryMode); + MODECASE(STACKTOOL_MODE_SUMMARY, summaryMode); + MODECASE(STACKTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunMode); + MODECASE(STACKTOOL_MODE_PENDINGCLEANUPSKYFILE, pendingcleanupskyfileMode); + MODECASE(STACKTOOL_MODE_DONECLEANUP, donecleanupMode); + MODECASE(STACKTOOL_MODE_UPDATESUMSKYFILE, updatesumskyfileMode); + MODECASE(STACKTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(STACKTOOL_MODE_IMPORTRUN, importrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} +//stackAssociationRow *association = pxStackAssociationDefine(data_group,tess_id,filter,skycell_id); +stackAssociationRow *pxStackAssociationDefine(pxConfig *config, psS64 stack_id) { + psString select = pxDataGet("stacktool_associationdefine_select.sql"); + if (!select) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString idString = NULL; + psStringAppend(&idString, "%" PRId64, stack_id); + // Copy string to get around the issue with psStringSubstitute not believing + // that select is a psString. + psString rep = psStringCopy(select); + psFree(select); + select = rep; + psStringSubstitute(&select, idString, "@STACK_ID@"); + psFree(idString); + + if (!p_psDBRunQuery(config->dbh, select)) { + psError(PS_ERR_UNKNOWN,false, "database error"); + psFree(select); + return(NULL); + } + psFree(select); + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + return(NULL); + } + if (psArrayLength(output) != 1) { + psWarning("stacktool: incorrect number of rows found"); + psFree(output); + return(NULL); + } + psMetadata *outrow = psMetadataAlloc(); + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; + +/* printf("%" PRId64 " %s %s %s %s\n",psMetadataLookupS64(NULL,row,"sass_id"), */ +/* psMetadataLookupStr(NULL,row,"data_group"), */ +/* psMetadataLookupStr(NULL,row,"tess_id"), */ +/* psMetadataLookupStr(NULL,row,"filter"), */ +/* psMetadataLookupStr(NULL,row,"projection_cell")); */ + + if (psMetadataLookupS64(NULL,row,"sass_id") == PS_MAX_S64) { + psMetadataAddS64(outrow,PS_LIST_TAIL,"sass_id",PS_META_REPLACE,"",0); + } + else { + psMetadataAddS64(outrow,PS_LIST_TAIL,"sass_id",PS_META_REPLACE,"",psMetadataLookupS64(NULL,row,"sass_id")); + } + psMetadataAddStr(outrow,PS_LIST_TAIL,"data_group",PS_META_REPLACE,"",psMetadataLookupStr(NULL,row,"data_group")); + psMetadataAddStr(outrow,PS_LIST_TAIL,"tess_id",PS_META_REPLACE,"",psMetadataLookupStr(NULL,row,"tess_id")); + psMetadataAddStr(outrow,PS_LIST_TAIL,"filter",PS_META_REPLACE,"",psMetadataLookupStr(NULL,row,"filter")); + psMetadataAddStr(outrow,PS_LIST_TAIL,"projection_cell",PS_META_REPLACE,"", + psMetadataLookupStr(NULL,row,"projection_cell")); + } +/* printf("%" PRId64 " %s %s %s %s\n",psMetadataLookupS64(NULL,outrow,"sass_id"), */ +/* psMetadataLookupStr(NULL,outrow,"data_group"), */ +/* psMetadataLookupStr(NULL,outrow,"tess_id"), */ +/* psMetadataLookupStr(NULL,outrow,"filter"), */ +/* psMetadataLookupStr(NULL,outrow,"projection_cell")); */ + + + psFree(output); + stackAssociationRow *sassRow = stackAssociationObjectFromMetadata(outrow); + psFree(outrow); + return(sassRow); +} + + + + + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + + // optional + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + + psMetadata *where = psMetadataAlloc(); + psMetadata *having = psMetadataAlloc(); // HAVING clause + + // select based on properties of the raw exposures + PXOPT_COPY_STR(config->args, where, "-select_comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-select_inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-select_telescope", "rawExp.telescope", "=="); + PXOPT_COPY_STR(config->args, where, "-select_filter", "rawExp.filter", "=="); + PXOPT_COPY_STR(config->args, where, "-select_uri", "rawExp.uri", "=="); + PXOPT_COPY_TIME(config->args, where, "-select_dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-select_dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_F32(config->args, where, "-select_airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F32(config->args, where, "-select_airmass_max", "rawExp.airmass", "<="); + PXOPT_COPY_F32(config->args, where, "-select_sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<="); + PXOPT_COPY_F32(config->args, where, "-select_exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-select_exp_time_max", "rawExp.exp_time", "<="); + PXOPT_COPY_F32(config->args, where, "-select_ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-select_ccd_temp_max", "rawExp.ccd_temp", "<="); + PXOPT_COPY_F32(config->args, where, "-select_posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F32(config->args, where, "-select_posang_max", "rawExp.posang", "<="); + PXOPT_COPY_F32(config->args, where, "-select_sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-select_sun_angle_max", "rawExp.sun_angle", "<="); + PXOPT_COPY_F32(config->args, where, "-select_fwhm_major_min", "camProcessedExp.fwhm_major", ">="); + PXOPT_COPY_F32(config->args, where, "-select_fwhm_major_max", "camProcessedExp.fwhm_major", "<="); + PXOPT_COPY_F32(config->args, where, "-select_fwhm_minor_min", "camProcessedExp.fwhm_minor", ">="); + PXOPT_COPY_F32(config->args, where, "-select_fwhm_minor_max", "camProcessedExp.fwhm_minor", "<="); + PXOPT_COPY_F32(config->args, where, "-select_elong_max", "(camProcessedExp.fwhm_major / camProcessedExp.fwhm_minor)", "<="); + PXOPT_COPY_F32(config->args, where, "-select_iq_elong_max", "(camProcessedExp.iq_fwhm_major / camProcessedExp.iq_fwhm_minor)", "<="); + PXOPT_COPY_F32(config->args, where, "-select_iq_m2_max", "camProcessedExp.iq_m2", "<="); + PXOPT_COPY_F32(config->args, where, "-select_iq_m2_min", "camProcessedExp.iq_m2", ">="); + PXOPT_COPY_F32(config->args, where, "-select_iq_m3_max", "camProcessedExp.iq_m3", "<="); + PXOPT_COPY_F32(config->args, where, "-select_iq_m4_min", "camProcessedExp.iq_m4", ">="); + PXOPT_COPY_F32(config->args, where, "-select_iq_m4_max", "camProcessedExp.iq_m4", "<="); + PXOPT_COPY_F32(config->args, where, "-select_zpt_obs_min", "camProcessedExp.zpt_obs", ">="); + PXOPT_COPY_F32(config->args, where, "-select_zpt_obs_max", "camProcessedExp.zpt_obs", "<="); + PXOPT_COPY_F32(config->args, where, "-select_bg_max", "camProcessedExp.bg", "<="); + PXOPT_COPY_F32(config->args, where, "-select_bg_stdev_max", "camProcessedExp.bg_stdev", "<="); + PXOPT_COPY_F32(config->args, where, "-select_astrom", "sqrt(POWER(camProcessedExp.sigma_ra, 2) + POWER(camProcessedExp.sigma_dec, 2))", "<="); + + PXOPT_COPY_STR(config->args, where, "-select_exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_F32(config->args, where, "-select_good_frac_min", "warpSkyfile.good_frac", ">="); + pxAddLabelSearchArgs(config, where, "-select_skycell_id", "warpSkyfile.skycell_id", "LIKE"); + pxAddLabelSearchArgs(config, where, "-select_data_group", "warpRun.data_group", "LIKE"); + pxAddLabelSearchArgs (config, where, "-select_label", "warpRun.label", "LIKE"); // define using warp label + pxAddLabelSearchArgs (config, where, "-warp_id", "warpRun.warp_id", "=="); + + // these are used to build the HAVING restriction + PXOPT_COPY_S32(config->args, having, "-min_num", "num_warp", ">="); + PXOPT_COPY_S32(config->args, having, "-max_num", "num_warp", "<="); + + PXOPT_LOOKUP_S32(min_num, config->args, "-min_num", false, false); + if (min_num < 2) { + psError(PXTOOLS_ERR_CONFIG, true, "Require at least two inputs for a stack, but min_num = %d", + min_num); + psFree(where); + psFree(having); + return false; + } + + // other options applied outside of the WHERE + PXOPT_LOOKUP_S32(randomLimit, config->args, "-random", false, false); + PXOPT_LOOKUP_S32(min_new, config->args, "-min_new", false, false); + PXOPT_LOOKUP_F32(min_frac, config->args, "-min_frac", false, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + if (!psListLength(where->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(where); + psFree(having); + return false; + } + + psString select = pxDataGet("stacktool_definebyquery_select.sql"); + if (!select) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + psFree(having); + return false; + } + + psString where1 = psStringCopy(""); + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&where1, "\nAND %s", whereClause); + psFree(whereClause); + } + + psString where2 = psStringCopy(""); + if (label) { + psStringAppend(&where2, "\nWHERE stackRun.label = '%s'", label); + } + + // Restriction on aggregated quantities using HAVING + { + psString havingClause = NULL; // HAVING string + if (psListLength(having->list)) { + havingClause = psDBGenerateWhereConditionSQL(having, NULL); + } + + if (min_new > 0) { + if (havingClause) { + psStringAppend(&havingClause, " AND"); + } + psStringAppend(&havingClause, + " (num_warp - num_stack >= %d OR (num_warp >= %d AND num_stack IS NULL))", + min_new, min_new); + } + if (isfinite(min_frac)) { + if (havingClause) { + psStringAppend(&havingClause, " AND"); + } + // Avoiding division by zero + psStringAppend(&havingClause, " (num_warp >= %f * num_stack OR num_stack IS NULL)", + (double)min_frac); + } + if (havingClause) { + psStringAppend(&select, " HAVING %s", havingClause); + psFree(havingClause); + } + } + psFree(having); + + if (!p_psDBRunQueryF(config->dbh, select, where1, where2)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(select); + psFree(where1); + psFree(where2); + return false; + } + psFree(select); + psFree(where1); + psFree(where2); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + psFree(where); + return false; + } + if (!psArrayLength(output)) { + psWarning("stacktool: no rows found"); + psFree(output); + psFree(where); + return true; + } + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackSkycells", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + psFree(where); + return false; + } + psFree(output); + psFree(where); + return true; + } + + psString insert = NULL; // Insertion query + if (randomLimit > 0) { + insert = pxDataGet("stacktool_definebyquery_insert_random_part1.sql"); + } else { + insert = pxDataGet("stacktool_definebyquery_insert.sql"); + } + if (!insert) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&insert, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (randomLimit > 0) { + psString part2 = pxDataGet("stacktool_definebyquery_insert_random_part2.sql"); + if (!part2) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(insert); + return false; + } + psStringAppend(&insert, "%s", part2); + psFree(part2); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(where); + return false; + } + + psArray *list = psArrayAllocEmpty(16); // List of runs, to print + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + bool status; + + // pull out the skycell_id, tess_id, filter + psString skycell_id = psMetadataLookupStr(&status, row, "skycell_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup skycell_id"); + psFree(output); + psFree(insert); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString tess_id = psMetadataLookupStr(&status, row, "tess_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup tess_id"); + psFree(output); + psFree(insert); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString filter = psMetadataLookupStr(&status, row, "filter"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup filter"); + psFree(output); + psFree(insert); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + // create a new stackRun for this stack + stackRunRow *run = stackRunRowAlloc( + 0, // ID + "new", // state + workdir, + label, + data_group ? data_group : label, + dist_group, + reduction, + dvodb, + registered, + skycell_id, + tess_id, + filter, + NULL, // software_ver + note); + + if (!stackRunInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(run); + psFree(insert); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + // figure out the new stack_id + psS64 stack_id = psDBLastInsertID(config->dbh); + run->stack_id = stack_id; + + psArrayAdd(list, list->n, run); + psFree(run); + + //CZW Add an association entry here. + // Define the requested association, and insert it if it doesn't already exist + stackAssociationRow *association = pxStackAssociationDefine(config,stack_id); + psS64 sass_id=-1; + if (!association->sass_id) { + psTrace("stacktool.association",2,"No required Association found. Adding."); + + if (!stackAssociationInsertObject(config->dbh,association)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(insert); + psFree(list); + psFree(association); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + sass_id = psDBLastInsertID(config->dbh); + association->sass_id = sass_id; + } + else { + sass_id = association->sass_id; + } + // Insert the map entry for this row. + stackAssociationMapRow *maprow = stackAssociationMapRowAlloc(sass_id,stack_id); + if (!stackAssociationMapInsertObject(config->dbh,maprow)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(insert); + psFree(list); + psFree(association); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + + + // Create a suitable insertion query for this run + psString thisInsert = psStringCopy(insert); + psString idString = NULL; + psStringAppend(&idString, "%" PRId64, stack_id); + psStringSubstitute(&thisInsert, idString, "@STACK_ID@"); + psFree(idString); + + // replace @FILTER@, @SKYCELL_ID@, @RANDOM_LIMIT@ + psStringSubstitute(&thisInsert, filter, "@FILTER@"); + psStringSubstitute(&thisInsert, skycell_id, "@SKYCELL_ID@"); + + if (randomLimit > 0) { + psString limString = NULL; + psStringAppend(&limString, "%d", randomLimit); + psStringSubstitute(&thisInsert, limString, "@RANDOM_LIMIT@"); + psFree(limString); + } + + // XXX this insert uses a select to generate the list of warp_ids for the stack, + // we have applied a set of criteria above (WHERE) to select the relevant warps + // this insert needs to use exactly the same restrictions (race condition is probably not critical) + // the insert below seems to only restrict matches to the skycell, tess, and filter + if (!p_psDBRunQuery(config->dbh, thisInsert)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(thisInsert); + psFree(insert); + psFree(output); + psFree(list); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(thisInsert); + +# if (0) + { + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + psFree(where); + return false; + } + if (!psArrayLength(output)) { + psWarning("stacktool (definebyquery, insert): no rows found"); + psFree(output); + psFree(where); + return true; + } + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackSkycells", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + psFree(where); + return false; + } + psFree(output); + psFree(where); + return true; + } +# endif + + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(list); + return false; + } + + if (!stackRunPrintObjects(stdout, list, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(list); + return false; + } + psFree(list); + return true; +} + + +static bool definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); + PXOPT_LOOKUP_STR(filter, config->args, "-filter", true, false); + + // default + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + + // options + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + + // we have to support multipe exp_ids + psMetadataItem *warp_ids = psMetadataLookup(config->args, "-warp_id"); + if (!warp_ids) { + // this shouldn't actually happen when using psArgs + psError(PS_ERR_UNKNOWN, true, "-warp_id is required"); + return false; + } + + stackRunRow *run = stackRunRowAlloc( + 0, // ID + "new", // state + workdir, + label, + data_group ? data_group : label, + dist_group, + reduction, + dvodb, + registered, + skycell_id, + tess_id, + filter, + NULL, // software_ver + note); + + if (!run) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc stackRun object"); + return true; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!stackRunInsertObject(config->dbh, run)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(run); + return true; + } + + // get the assigned warp_id + run->stack_id = psDBLastInsertID(config->dbh); + + //CZW Add an association entry here. + + // insert the stackInputSkyfile rows + psListIterator *iter = psListIteratorAlloc(warp_ids->data.list, 0, false); + psMetadataItem *item = NULL; + while ((item = psListGetAndIncrement(iter))) { + // if the value is NULL this is probably the first pass through the + // loop and -warp_id was not specified at all + if (!item->data.V) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for -warp_id"); + return false; + } + if (!stackInputSkyfileInsert(config->dbh, run->stack_id, item->data.S64)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to insert stackInputSkyfile rows"); + return false; + } + } + psFree(iter); + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!stackRunPrintObject(stdout, run, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(run); + return false; + } + + psFree(run); + + return true; +} + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + +#ifdef notdef + PXOPT_LOOKUP_S64(stack_id, config->args, "-stack_id", false, false); + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); +#endif + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "stackRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "stackRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "stackRun.state", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S64(sass_id, config->args, "-sass_id", false, false); + + if (sass_id) { + PXOPT_COPY_S64(config->args, where, "-sass_id", "stackAssociationMap.sass_id", "=="); + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + //CZW join against stackAssociationMap + psString query = psStringCopy("UPDATE stackRun"); + + if (fault) { + psStringAppend(&query, " JOIN stackSumSkyfile USING(stack_id)"); + PXOPT_COPY_S16(config->args, where, "-fault", "stackSumSkyfile.fault", "=="); + } + if (sass_id) { + psStringAppend(&query, " JOIN stackAssociationMap USING(stack_id)"); + } + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "stackRun", "stack_id", "stackSumSkyfile", true, false); + + psFree(query); + psFree(where); + + return result; + +#ifdef notdef + // Hack-y work around to make stacktool more like the other tools, without breaking other stuff (hopefully). + + if ((state)&&(stack_id)) { + // set detRun.state to state + return setstackRunState(config, stack_id, state); + } + + if ((state)&&(label)) { + return setstackRunStateByLabel(config, label, state); + } + + psError(PS_ERR_UNKNOWN, false, "Required options not found."); + return false; +#endif +} + + +static bool addinputskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(stack_id, config->args, "-stack_id", true, false); + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + + // XXX need to validate the warp_id here + // XXX instead of validiting it here we should just use forgein key + // constrants + if (!stackInputSkyfileInsert(config->dbh, + stack_id, + warp_id + )) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool inputskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // XXX require at least a stack id (add better search options) + // PXOPT_LOOKUP_S64(stack_id, config->args, "-stack_id", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("stacktool_inputskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "stackInputSkyfile"); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackInputSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool tosumMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "stackRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("stacktool_tosum.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psStringCopy(""); // WHERE conditions to add + if (psListLength(where->list)) { + psString new = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereClause, "\nAND %s", new); + psFree(new); + } + psFree(where); + + psStringAppend(&query, "\nORDER by priority DESC, stack_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackSumSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool tobkgMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "stackRun.label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("stacktool_tobkg.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereClause = psStringCopy(""); // WHERE conditions to add + if (psListLength(where->list)) { + psString new = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereClause, "\nAND %s", new); + psFree(new); + } + psFree(where); + + psStringAppend(&query, "\nORDER by priority DESC, stack_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackBkgSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool addsumskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(stack_id, config->args, "-stack_id", true, false); + + // optional + PXOPT_LOOKUP_STR(uri, config->args, "-uri", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F32(dtime_stack, config->args, "-dtime_stack", false, false); + PXOPT_LOOKUP_F32(dtime_match_mean, config->args, "-dtime_match_mean", false, false); + PXOPT_LOOKUP_F32(dtime_match_stdev, config->args, "-dtime_match_stdev", false, false); + PXOPT_LOOKUP_F32(dtime_initial, config->args, "-dtime_initial", false, false); + PXOPT_LOOKUP_F32(dtime_reject, config->args, "-dtime_reject", false, false); + PXOPT_LOOKUP_F32(dtime_final, config->args, "-dtime_final", false, false); + PXOPT_LOOKUP_F32(dtime_convolve, config->args, "-dtime_convolve", false, false); + PXOPT_LOOKUP_F32(dtime_phot, config->args, "-dtime_phot", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_F32(match_mean, config->args, "-match_mean", false, false); + PXOPT_LOOKUP_F32(match_stdev, config->args, "-match_stdev", false, false); + PXOPT_LOOKUP_F32(match_rms, config->args, "-match_rms", false, false); + PXOPT_LOOKUP_F32(stamps_mean, config->args, "-stamps_mean", false, false); + PXOPT_LOOKUP_F32(stamps_stdev, config->args, "-stamps_stdev", false, false); + PXOPT_LOOKUP_S32(stamps_min, config->args, "-stamps_min", false, false); + PXOPT_LOOKUP_S32(reject_images, config->args, "-reject_images", false, false); + PXOPT_LOOKUP_F32(reject_pix_mean, config->args, "-reject_pix_mean", false, false); + PXOPT_LOOKUP_F32(reject_pix_stdev, config->args, "-reject_pix_stdev", false, false); + PXOPT_LOOKUP_S32(sources, config->args, "-sources", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_F32(good_frac, config->args, "-good_frac", false, false); + PXOPT_LOOKUP_F64(mjd_obs, config->args, "-mjd_obs", false, false); + + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_STR(ver_ppstack, config->args, "-ver_ppstack", false, false); + PXOPT_LOOKUP_STR(ver_streaks, config->args, "-ver_streaks", false, false); + + PXOPT_LOOKUP_S16(background_model, config->args, "-background_model", false, false); + + psTrace("czw.test",1,"Received versions: pslib %s psmodules %s psphot %s ppstats %s ppstack %s streaks %s\n", + ver_pslib,ver_psmodules,ver_psphot,ver_ppstats,ver_ppstack,ver_streaks); + psString software_ver = NULL; + if ((ver_pslib)&&(ver_psmodules)) { + software_ver = pxMergeCodeVersions(ver_pslib,ver_psmodules); + } + if (ver_psphot) { + software_ver = pxMergeCodeVersions(software_ver,ver_psphot); + } + if (ver_ppstats) { + software_ver = pxMergeCodeVersions(software_ver,ver_ppstats); + } + if (ver_ppstack) { + software_ver = pxMergeCodeVersions(software_ver,ver_ppstack); + } + if (ver_streaks) { + software_ver = pxMergeCodeVersions(software_ver,ver_streaks); + } + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // XXX need to validate the stack_id here + // XXX instead of validiting it here we should just use forgein key + // constrants + if (!stackSumSkyfileInsert(config->dbh, + stack_id, + uri, + path_base, + bg, + bg_stdev, + dtime_stack, + dtime_match_mean, + dtime_match_stdev, + dtime_convolve, + dtime_initial, + dtime_reject, + dtime_final, + dtime_phot, + dtime_script, + match_mean, + match_stdev, + match_rms, + stamps_mean, + stamps_stdev, + stamps_min, + reject_images, + reject_pix_mean, + reject_pix_stdev, + sources, + hostname, + good_frac, + mjd_obs, + fault, + software_ver, + background_model, + quality + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (fault == 0) { + // Set stackRun software if we are finished. + if (!pxSetRunSoftware(config, "stackRun", "stack_id", stack_id, software_ver)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to set stackRun.software_ver for stack_id: %" PRId64, + stack_id); + return(false); + } + + if (!setstackRunState(config, stack_id, "full")) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to change stackRun's state"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool sumskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackSumSkyfile.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "stackRun.filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "stackRun.state", "=="); + pxAddLabelSearchArgs(config, where, "-label", "stackRun.label", "LIKE"); + pxAddLabelSearchArgs(config, where, "-data_group", "stackRun.data_group", "LIKE"); + PXOPT_COPY_S16(config->args, where, "-fault", "stackSumSkyfile.fault", "=="); + PXOPT_COPY_F64(config->args, where, "-mjd_obs_begin", "stackSumSkyfile.mjd_obs", ">="); + PXOPT_COPY_F64(config->args, where, "-mjd_obs_end", "stackSumSkyfile.mjd_obs", "<="); + PXOPT_COPY_S16(config->args, where, "-background_model","stackSumSkyfile.background_model", "=="); + +// The following three selectors are incompatible with the sql so omit them +// PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); +// PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); +// PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("stacktool_sumskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "stackSumSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool sassskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sass_id", "stackAssociation.sass_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackAssociation.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "stackAssociation.projection_cell", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "stackAssociation.filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-data_group", "stackAssociation.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("stacktool_sassskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + psFree(where); + + // This needs to be sorted by skycell_id, or the ppSkycell calls to mosaic things + // don't work properly. + psStringAppend(&query, " ORDER BY stackRun.skycell_id "); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "stackSumSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool updatesassMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sass_id", "sass_id", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "projection_cell", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "filter", "=="); + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psMetadata *values = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, values, "-set_data_group", "data_group", "=="); + if (!psListLength(values->list)) { + psFree(values); + psError(PXTOOLS_ERR_CONFIG, false, "data_group to assign not found"); + return false; + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + long rows = psDBUpdateRows(config->dbh, "stackAssociation", where, values); + psFree(values); + psFree(where); + + if (rows != 1) { + psError(PS_ERR_UNKNOWN, false, "should have updated only one row. updated %ld", rows); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool revertsumskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackSumSkyfile.stack_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "stackRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "stackSumSkyfile.fault", "=="); + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // Delete product + psString delete = pxDataGet("stacktool_revertsumskyfile_delete.sql"); + if (!delete) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&delete, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, delete)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete); + psFree(where); + return false; + } + psFree(delete); + + int numRows = psDBAffectedRows(config->dbh); // Number of row affected + psLogMsg("stacktool", PS_LOG_INFO, "Deleted %d rows", numRows); + + psFree(where); + + return true; +} + + +static bool setstackRunState(pxConfig *config, psS64 stack_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid stackRun state: %s", state); + return false; + } + + char *query = "UPDATE stackRun SET state = '%s' WHERE stack_id = %"PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, stack_id)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for stack_id %"PRId64, stack_id); + return false; + } + + return true; +} + +#ifdef notdef +static bool setstackRunStateByLabel(pxConfig *config, const char *label, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid stackRun state: %s", state); + return false; + } + + char *query = "UPDATE stackRun SET state = '%s' WHERE label = '%s'"; + if (!p_psDBRunQueryF(config->dbh, query, state, label)) { + psError(PS_ERR_UNKNOWN, false, + "failed to change state for label %s", label); + return false; + } + + return true; +} +#endif +static bool tosummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackSumSkyfile.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sass_id", "stackAssociationMap.sass_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackSumSkyfile.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "stackRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "stackRun.filter", "LIKE"); + pxAddLabelSearchArgs (config, where, "-label", "stackRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "stackRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("stacktool_tosummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // generate where strings for arguments that require extra processing + // beyond PXOPT_COPY* + psString whereClause; + if (psListLength(where->list)) { + whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringPrepend(&whereClause, "\n AND "); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + psFree(whereClause); + return false; + } + + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, whereClause)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(whereClause); + psFree(query); + return false; + } + psFree(whereClause); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return(true); +} +static bool addsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(sass_id, config->args, "-sass_id", true, false); + PXOPT_LOOKUP_STR(projection_cell, config->args, "-projection_cell", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + psString query = pxDataGet("stacktool_addsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + if (!p_psDBRunQueryF(config->dbh, query, sass_id, projection_cell, path_base)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psS64 numUpdated = psDBAffectedRows(config->dbh); + + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + psFree(query); + return false; + } + + psFree(query); + + // Print anything here? + + return(true); +} + +static bool revertsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + PXOPT_LOOKUP_S64(sass_id, config->args, "-sass_id", true, false); + + psString query = pxDataGet("stacktool_revertsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + if (!p_psDBRunQueryF(config->dbh, query, sass_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psS64 numUpdated = psDBAffectedRows(config->dbh); + + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + psFree(query); + return(false); + } + psLogMsg("stacktool", PS_LOG_INFO, "Deleted %ld rows", numUpdated); + + psFree(query); + + return(true); +} + +static bool summaryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sass_id", "stackSummary.sass_id", "=="); + PXOPT_COPY_STR(config->args, where, "-projection_cell", "stackAssociation.projection_cell", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackAssociation.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "stackAssociation.filter", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackSumSkyfile.stack_id", "=="); + pxAddLabelSearchArgs(config, where, "-data_group", "stackAssociation.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "LIKE"); +// PXOPT_COPY_STR(config->args, where, "-state", "stackRun.state", "=="); + pxAddLabelSearchArgs(config, where, "-label", "stackRun.label", "LIKE"); +// PXOPT_COPY_S16(config->args, where, "-fault", "stackSumSkyfile.fault", "=="); +// PXOPT_COPY_F64(config->args, where, "-mjd_obs_begin", "stackSumSkyfile.mjd_obs", ">="); +// PXOPT_COPY_F64(config->args, where, "-mjd_obs_end", "stackSumSkyfile.mjd_obs", "<="); +// PXOPT_COPY_S16(config->args, where, "-background_model","stackSumSkyfile.background_model", "=="); + +// PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("stacktool_summary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters are required"); + return false; + } + + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "stackSummary", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); +/* PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); */ +/* PXOPT_COPY_S64(config->args, where, "-sass_id", "stackAssociationMap.sass_id", "=="); */ + + pxAddLabelSearchArgs (config, where, "-label", "stackRun.label", "=="); + + psString query = pxDataGet("stacktool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackPendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingcleanupskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(stack_id, config->args, "-stack_id", false, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + if (stack_id) { + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + } + pxAddLabelSearchArgs (config, where, "-label", "stackRun.label", "=="); + + psString query = pxDataGet("stacktool_pendingcleanupskyfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackPendingCleanupSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-sass_id", "sass_id", "=="); + + psString query = pxDataGet("stacktool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("stacktool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "stackDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} +static bool updatesumskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + PXOPT_LOOKUP_S16(background_model, config->args, "-set_background_model", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + + if (background_model) { + psMetadata *values = psMetadataAlloc(); + PXOPT_COPY_S16(config->args, values, "-set_background_model", "background_model", "=="); + long rows = psDBUpdateRows(config->dbh,"stackSumSkyfile", where, values); + psFree(values); + if (!rows) { + // This maybe should rollback and error if rows != 1 + psError(PS_ERR_UNKNOWN, true, "no rows changed"); + return false; + } + } + else { + if (!pxSetFaultCode(config->dbh, "stackSumSkyfile", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + } + psFree (where); + + return true; +} + +//CZW I have not added sass information to the export/import run modes yet. +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 3; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-stack_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + + ExportTable tables [] = { + {"stackRun", "stacktool_export_run.sql"}, + {"stackInputSkyfile", "stacktool_export_input_skyfile.sql"}, + {"stackSumSkyfile", "stacktool_export_sum_skyfile.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + if (!strcmp(tables[i].tableName, "stackRun")) { + if (!pxSetStateCleaned("stackRun", "state", output)) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + int numImportTables = 2; + + char tables[2] [80] = {"stackInputSkyfile", "stackSumSkyfile"}; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + +#ifdef notdef + fprintf (stderr, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); +#endif + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + psMetadataItem *item = psMetadataLookup (input, "stackRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + stackRunRow *stackRun = stackRunObjectFromMetadata (entry->data.md); + stackRunInsertObject (config->dbh, stackRun); + + // fprintf (stdout, "---- stack run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + for (int i = 0; i < numImportTables; i++) { + psMetadataItem *item = psMetadataLookup (input, tables[i]); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + switch (i) { + case 0: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + stackInputSkyfileRow *stackInputSkyfile = stackInputSkyfileObjectFromMetadata (entry->data.md); + stackInputSkyfileInsertObject (config->dbh, stackInputSkyfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 1: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + stackSumSkyfileRow *stackSumSkyfile = stackSumSkyfileObjectFromMetadata (entry->data.md); + stackSumSkyfileInsertObject (config->dbh, stackSumSkyfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + } + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktool.h (revision 41104) @@ -0,0 +1,53 @@ +/* + * stacktool.h + * + * Copyright (C) 2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef STACKTOOL_H +#define STACKTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + STACKTOOL_MODE_NONE = 0x0, + STACKTOOL_MODE_DEFINEBYQUERY, + STACKTOOL_MODE_DEFINERUN, + STACKTOOL_MODE_UPDATERUN, + STACKTOOL_MODE_ADDINPUTSKYFILE, + STACKTOOL_MODE_INPUTSKYFILE, + STACKTOOL_MODE_TOSUM, + STACKTOOL_MODE_TOBKG, + STACKTOOL_MODE_ADDSUMSKYFILE, + STACKTOOL_MODE_SUMSKYFILE, + STACKTOOL_MODE_SASSSKYFILE, + STACKTOOL_MODE_UPDATESASS, + STACKTOOL_MODE_REVERTSUMSKYFILE, + STACKTOOL_MODE_TOSUMMARY, + STACKTOOL_MODE_ADDSUMMARY, + STACKTOOL_MODE_REVERTSUMMARY, + STACKTOOL_MODE_SUMMARY, + STACKTOOL_MODE_PENDINGCLEANUPRUN, + STACKTOOL_MODE_PENDINGCLEANUPSKYFILE, + STACKTOOL_MODE_DONECLEANUP, + STACKTOOL_MODE_UPDATESUMSKYFILE, + STACKTOOL_MODE_EXPORTRUN, + STACKTOOL_MODE_IMPORTRUN +} stacktoolMode; + +pxConfig *stacktoolConfig(pxConfig *config, int argc, char **argv); + +#endif // STACKTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/stacktoolConfig.c (revision 41104) @@ -0,0 +1,392 @@ +/* + * stacktoolConfig.c + * + * Copyright (C) 2007-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "stacktool.h" + +pxConfig *stacktoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define dvodb", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_skycell_id", PS_META_DUPLICATE_OK, "search for skycell_id (LIKE comparison, multiple ok)", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_good_frac_min", 0, "define min good_frac", 0.0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_type", 0, "search for exp_type", "object"); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_label", PS_META_DUPLICATE_OK, "search by warpRun label (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_data_group", PS_META_DUPLICATE_OK , "search by warpRun data_group (LIKE comparison, multiple ok)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_comment", 0, "search for comment (LIKE)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_inst", 0, "search for camera", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_telescope", 0, "search for telescope", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_filter", 0, "search for filter", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_uri", 0, "search for uri", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-select_dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-select_dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_airmass_min", 0, "define min airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_airmass_max", 0, "define max airmass", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_sat_pixel_frac_max", 0, "define max fraction of saturated pixels", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_time_min", 0, "define min exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_exp_time_max", 0, "define max exposure time", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_ccd_temp_min", 0, "define min ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_ccd_temp_max", 0, "define max ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_posang_min", 0, "define min rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_posang_max", 0, "define max rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_sun_angle_min", 0, "define min solar angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_sun_angle_max", 0, "define max solar angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_fwhm_major_min", 0, "define min fwhm (major axis)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_fwhm_major_max", 0, "define max fwhm (major axis)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_fwhm_minor_min", 0, "define min fwhm (minor axis)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_fwhm_minor_max", 0, "define max fwhm (minor axis)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_elong_max", 0, "define max elongation from fwhm ", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_elong_max", 0, "define max elongation from iq_fwhm ", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_m2_min", 0, "define min iq_m2", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_m2_max", 0, "define max iq_m2", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_m3_min", 0, "define min iq_m3", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_m3_max", 0, "define max iq_m3", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_m4_min", 0, "define min iq_m4", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_iq_m4_max", 0, "define max iq_m4", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_zpt_obs_min", 0, "define min zero point", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_zpt_obs_max", 0, "define max zero point", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_bg_max", 0, "define max background", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_bg_stdev_max", 0, "define max background stdev", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-select_astrom", 0, "define max astrometry rms", NAN); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-warp_id", PS_META_DUPLICATE_OK, "include this warp ID (multiple OK)", 0); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-random", 0, "use this number of random elements", 0); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-min_num", 0, "minimum number of inputs", 0); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-max_num", 0, "maximum number of inputs", 0); + psMetadataAddS32(definebyqueryArgs, PS_LIST_TAIL, "-min_new", 0, "minimum number of new inputs", 0); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-min_frac", 0, "minumum fraction of new inputs", NAN); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define dvodb", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell ID (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-tess_id", 0, "define tessellation ID (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-filter", 0, "define filter (required)", NULL); + psMetadataAddS64(definerunArgs, PS_LIST_TAIL, "-warp_id", PS_META_DUPLICATE_OK, "include this warp ID (multiple OK, required)", 0); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group", 0); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-pretend", 0, "show query but do not actually modify the database", false); + + // -addinputskyfile + psMetadata *addinputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "define stack ID (required)", 0); + psMetadataAddS64(addinputskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "define warp ID (required)", 0); + + // -inputskyfile + // XXX add additional search terms + psMetadata *inputskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(inputskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddS64(inputskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddU64(inputskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(inputskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -tosum + psMetadata *tosumArgs = psMetadataAlloc(); + psMetadataAddS64(tosumArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(tosumArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddS64(tosumArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddU64(tosumArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tosumArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -tobkg + psMetadata *tobkgArgs = psMetadataAlloc(); + psMetadataAddS64(tobkgArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(tobkgArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddU64(tobkgArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tobkgArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addsumskyfile + psMetadata *addsumskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(addsumskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "define stack ID (required)", 0); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-uri", 0, "define URI of file", 0); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", 0); + psMetadataAddF64(addsumskyfileArgs, PS_LIST_TAIL, "-bg", 0, "define exposue background", NAN); + psMetadataAddF64(addsumskyfileArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposue background mean stdev", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_stack", 0, "define elapsed processing time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_match_mean", 0, "define mean matching time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_match_stdev", 0, "define stdev matching time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_initial", 0, "define initial stack time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_reject", 0, "define rejection time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_final", 0, "define final stack time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_convolve", 0, "define image convolution time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_phot", 0, "define photometry time", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-match_mean", 0, "define mean matching deviation", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-match_stdev", 0, "define stdev matching deviation", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-match_rms", 0, "define mean rms of deviation", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-stamps_mean", 0, "define mean number of stamps", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-stamps_stdev", 0, "define stdev of number of stamps", NAN); + psMetadataAddS32(addsumskyfileArgs, PS_LIST_TAIL, "-stamps_min", 0, "define minimum number of stamps", 0); + psMetadataAddS32(addsumskyfileArgs, PS_LIST_TAIL, "-reject_images", 0, "number of images rejected", 0); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-reject_pix_mean", 0, "mean number of pixels rejected", NAN); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-reject_pix_stdev", 0, "stdev number of pixels rejected", NAN); + psMetadataAddS32(addsumskyfileArgs, PS_LIST_TAIL, "-sources", 0, "number of sources", 0); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddF32(addsumskyfileArgs, PS_LIST_TAIL, "-good_frac", 0, "define %% of good pixels", NAN); + psMetadataAddF64(addsumskyfileArgs, PS_LIST_TAIL, "-mjd_obs", 0, "define mjd_obs (average mjd_obs of inputs)", NAN); + psMetadataAddS16(addsumskyfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(addsumskyfileArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-ver_ppstack", 0, "define ppStack version", NULL); + psMetadataAddStr(addsumskyfileArgs, PS_LIST_TAIL, "-ver_streaks", 0, "define streaksremove version", NULL); + + psMetadataAddS16(addsumskyfileArgs, PS_LIST_TAIL, "-background_model", 0, "define background model version", 0); + + // -sumskyfile + psMetadata *sumskyfileArgs= psMetadataAlloc(); + psMetadataAddS64(sumskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID (LIKE comparison)", 0); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", 0); +#ifdef notdef + // These don't work so omit (for now) We probably should create a different mode for this type of search. + psMetadataAddS64(sumskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddS64(sumskyfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure ID", 0); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure name", NULL); +#endif + psMetadataAddS16(sumskyfileArgs, PS_LIST_TAIL, "-background_model", 0, "search by background model version", 0); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by stackRun.label", NULL); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by stackRun.data_group (LIKE comparison)", NULL); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-filter", 0, "search by filter (LIKE comparison)", NULL); + psMetadataAddF64(sumskyfileArgs, PS_LIST_TAIL, "-mjd_obs_begin", 0, "search by mjd_obs (average mjd_obs of inputs <=)", NAN); + psMetadataAddF64(sumskyfileArgs, PS_LIST_TAIL, "-mjd_obs_end", 0, "search by mjd_obs (average mjd_obs of inputs>=)", NAN); + psMetadataAddStr(sumskyfileArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddS16(sumskyfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddU64(sumskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(sumskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(sumskyfileArgs, PS_LIST_TAIL, "-all", 0, "enable search without arguments", false); + + // -sassskyfile + psMetadata *sassskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(sassskyfileArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association ID", 0); + psMetadataAddStr(sassskyfileArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID (LIKE comparison)", 0); + psMetadataAddStr(sassskyfileArgs, PS_LIST_TAIL, "-projection_cell", 0, "search by projection cell", 0); + + psMetadataAddStr(sassskyfileArgs, PS_LIST_TAIL, "-data_group", 0, "search by stackAssociation.data_group (LIKE comparison)", NULL); + psMetadataAddStr(sassskyfileArgs, PS_LIST_TAIL, "-filter", 0, "search by filter (LIKE comparison)", NULL); + psMetadataAddU64(sassskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(sassskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(sassskyfileArgs, PS_LIST_TAIL, "-all", 0, "enable search without arguments", false); + + // -updatesass + psMetadata *updatesassArgs = psMetadataAlloc(); + psMetadataAddS64(updatesassArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association id", 0); + psMetadataAddStr(updatesassArgs, PS_LIST_TAIL, "-data_group", 0, "search by data group", 0); + psMetadataAddStr(updatesassArgs, PS_LIST_TAIL, "-projection_cell", 0, "search by projection cell", 0); + psMetadataAddStr(updatesassArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess_id", 0); + psMetadataAddStr(updatesassArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", 0); + + psMetadataAddStr(updatesassArgs, PS_LIST_TAIL, "-set_data_group", 0, "data_group to assign", 0); + + // -revertsumskyfile + psMetadata *revertsumskyfileArgs= psMetadataAlloc(); + psMetadataAddS64(revertsumskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(revertsumskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddS16(revertsumskyfileArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertsumskyfileArgs, PS_LIST_TAIL, "-all", 0, "allow no search terms", 0); + + // -tosummary + psMetadata *tosummaryArgs = psMetadataAlloc(); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association ID", 0); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by stackRun label (LIKE comparison)", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by stackRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by stackRun dist_group (LIKE comparison)", NULL); + + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(tosummaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addsummary + psMetadata *addsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(addsummaryArgs, PS_LIST_TAIL, "-sass_id", 0, "set stack Association ID", 0); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-projection_cell", 0, "set projection cell", NULL); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-path_base", 0, "set summary path base", NULL); + + // -revertsummary + psMetadata *revertsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(revertsummaryArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association ID", 0); + + // -summary + psMetadata *summaryArgs = psMetadataAlloc(); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association ID", 0); + psMetadataAddS64(summaryArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-projection_cell", 0, "search by projection cell ID", NULL); +// psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-filter", 0, "search by filter (LIKE comparison)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by stackRun label (LIKE comparison)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by stackAssociation data_group (LIKE comparison)", NULL); + psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); +// psMetadataAddStr(summaryArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by stackRun dist_group (LIKE comparison)", NULL); + +// psMetadataAddBool(summaryArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(summaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(summaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -pendingcleanuprun + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanuprunArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddS64(pendingcleanuprunArgs, PS_LIST_TAIL, "-sass_id", 0, "search by stack association ID", 0); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupskyfile + psMetadata *pendingcleanupskyfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddBool(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + // -updatesumskyfile + psMetadata *updatesumskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(updatesumskyfileArgs, PS_LIST_TAIL, "-stack_id", 0, "define stack ID (required)", 0); + psMetadataAddS16(updatesumskyfileArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + psMetadataAddS16(updatesumskyfileArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality", 0); + psMetadataAddS16(updatesumskyfileArgs, PS_LIST_TAIL, "-set_background_model", 0, "set background model", 0); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-stack_id", 0, "export this stack ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "Define a new stackRun by searching for warp IDs", STACKTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definerun", "", STACKTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-updaterun", "", STACKTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-addinputskyfile", "", STACKTOOL_MODE_ADDINPUTSKYFILE, addinputskyfileArgs); + PXOPT_ADD_MODE("-inputskyfile", "", STACKTOOL_MODE_INPUTSKYFILE, inputskyfileArgs); + PXOPT_ADD_MODE("-tosum", "", STACKTOOL_MODE_TOSUM, tosumArgs); + PXOPT_ADD_MODE("-tobkg", "", STACKTOOL_MODE_TOBKG, tobkgArgs); + PXOPT_ADD_MODE("-addsumskyfile", "", STACKTOOL_MODE_ADDSUMSKYFILE, addsumskyfileArgs); + PXOPT_ADD_MODE("-sumskyfile", "list results of stackRun", STACKTOOL_MODE_SUMSKYFILE, sumskyfileArgs); + + PXOPT_ADD_MODE("-revertsumskyfile","", STACKTOOL_MODE_REVERTSUMSKYFILE, revertsumskyfileArgs); + PXOPT_ADD_MODE("-updatesumskyfile", "update fault code for sumskyfile", STACKTOOL_MODE_UPDATESUMSKYFILE, updatesumskyfileArgs); + + PXOPT_ADD_MODE("-pendingcleanuprun", "show runs that need to be cleaned up", STACKTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunArgs); + PXOPT_ADD_MODE("-pendingcleanupskyfile", "show runs that need to be cleaned up", STACKTOOL_MODE_PENDINGCLEANUPSKYFILE, pendingcleanupskyfileArgs); + PXOPT_ADD_MODE("-donecleanup", "show runs that have been cleaned", STACKTOOL_MODE_DONECLEANUP, donecleanupArgs); + + PXOPT_ADD_MODE("-sassskyfile", "list results of stackAssociation", STACKTOOL_MODE_SASSSKYFILE, sassskyfileArgs); + PXOPT_ADD_MODE("-updatesass", "update data_group of stackAssociation", STACKTOOL_MODE_UPDATESASS, updatesassArgs); + PXOPT_ADD_MODE("-tosummary", "show runs that can be summarized", STACKTOOL_MODE_TOSUMMARY, tosummaryArgs); + PXOPT_ADD_MODE("-summary", "show runs that have been summarized", STACKTOOL_MODE_SUMMARY, summaryArgs); + PXOPT_ADD_MODE("-addsummary", "add entry to the summary table", STACKTOOL_MODE_ADDSUMMARY, addsummaryArgs); + PXOPT_ADD_MODE("-revertsummary", "revert entry in the summary table", STACKTOOL_MODE_REVERTSUMMARY, revertsummaryArgs); + + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", STACKTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", STACKTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytool.c (revision 41104) @@ -0,0 +1,1556 @@ +/* + * staticskytool.c + * + * Copyright (C) 2007-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxspace.h" +#include "staticskytool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool inputsMode(pxConfig *config); +static bool todoMode(pxConfig *config); +static bool addresultMode(pxConfig *config); +static bool resultMode(pxConfig *config); +static bool revertMode(pxConfig *config); +static bool updateresultMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); + +static bool defineskycalrunMode(pxConfig *config); +static bool updateskycalrunMode(pxConfig *config); +static bool pendingskycalrunMode(pxConfig *config); +static bool addskycalresultMode(pxConfig *config); +static bool skycalresultMode(pxConfig *config); +static bool revertskycalresultMode(pxConfig *config); +static bool updateskycalresultMode(pxConfig *config); +static bool exportskycalrunMode(pxConfig *config); +static bool importskycalrunMode(pxConfig *config); + +static bool setstaticskyRunState(pxConfig *config, psS64 sky_id, const char *state); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = staticskytoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(STATICSKYTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(STATICSKYTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(STATICSKYTOOL_MODE_INPUTS, inputsMode); + MODECASE(STATICSKYTOOL_MODE_TODO, todoMode); + MODECASE(STATICSKYTOOL_MODE_ADDRESULT, addresultMode); + MODECASE(STATICSKYTOOL_MODE_RESULT, resultMode); + MODECASE(STATICSKYTOOL_MODE_REVERT, revertMode); + MODECASE(STATICSKYTOOL_MODE_UPDATERESULT, updateresultMode); + MODECASE(STATICSKYTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(STATICSKYTOOL_MODE_IMPORTRUN, importrunMode); + MODECASE(STATICSKYTOOL_MODE_DEFINESKYCALRUN, defineskycalrunMode); + MODECASE(STATICSKYTOOL_MODE_UPDATESKYCALRUN, updateskycalrunMode); + MODECASE(STATICSKYTOOL_MODE_PENDINGSKYCALRUN, pendingskycalrunMode); + MODECASE(STATICSKYTOOL_MODE_ADDSKYCALRESULT, addskycalresultMode); + MODECASE(STATICSKYTOOL_MODE_UPDATESKYCALRESULT,updateskycalresultMode); + MODECASE(STATICSKYTOOL_MODE_REVERTSKYCALRESULT,revertskycalresultMode); + MODECASE(STATICSKYTOOL_MODE_SKYCALRESULT, skycalresultMode); + MODECASE(STATICSKYTOOL_MODE_EXPORTSKYCALRUN, exportskycalrunMode); + MODECASE(STATICSKYTOOL_MODE_IMPORTSKYCALRUN, importskycalrunMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + + // optional + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + + psMetadata *whereMD = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, whereMD, "-select_stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_STR(config->args, whereMD, "-select_skycell_id", "stackRun.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, whereMD, "-select_tess_id", "stackRun.tess_id", "=="); + PXOPT_COPY_F32(config->args, whereMD, "-select_good_frac_min", "stackSumSkyfile.good_frac", ">="); + pxAddLabelSearchArgs(config, whereMD, "-select_label", "stackRun.label", "LIKE"); + pxAddLabelSearchArgs(config, whereMD, "-select_data_group", "stackRun.data_group", "LIKE"); + pxAddLabelSearchArgs(config, whereMD, "-select_filter", "stackRun.filter", "LIKE"); + if (!pxskycellAddWhere(config, whereMD)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + // find the number of requested filters: + psMetadataItem *filters = psMetadataLookup(config->args, "-select_filter"); + psAssert (filters, "-select_filter must exist"); + psAssert (filters->type == PS_DATA_METADATA_MULTI, "-select_filter should be a multi container"); + psAssert (filters->data.list->n, "-select_filter should at least have a place-holder"); + int num_filter = filters->data.list->n; + + PXOPT_LOOKUP_BOOL(group_by_data_group, config->args, "-group_by_data_group", false); + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(check_inputs, config->args, "-check_inputs", false); + + psString select; + if (group_by_data_group) { + select = pxDataGet("staticskytool_definebyquery_select_by_dg.sql"); + } else { + select = pxDataGet("staticskytool_definebyquery_select.sql"); + } + if (!select) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(whereMD); + return false; + } + + if (!psListLength(whereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(whereMD); + return false; + } + + // this 'where' is used for both staticskytool_definebyquery_select.sql and staticskytool_definebyquery_inputs.sql + psString where = NULL; + psString whereClause = psDBGenerateWhereConditionSQL(whereMD, NULL); + psStringAppend(&where, "\nAND %s", whereClause); + psFree(whereClause); + psFree(whereMD); + + psString where2 = NULL; + psString make_unique = NULL; + if (!rerun) { + psStringAppend(&where2, "\n %s\nAND staticskyRun.label = '%s'", where, label); + psStringAppend(&make_unique, "\nAND sky_id IS NULL"); + } + + if (!p_psDBRunQueryF(config->dbh, select, where, where2 ? where2 : where, num_filter, make_unique ? make_unique : "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(select); + return false; + } + psFree(select); + psFree(where2); + psFree(make_unique); + + // we now have a list of (tess_id, skycell_id) that (potentially) meet out needs + // we now need to loop over all of these and for each pair, select the best set of + // inputs + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + psFree(where); + return false; + } + if (!psArrayLength(output)) { + psWarning("staticskytool: no rows found"); + psFree(output); + psFree(where); + return true; + } + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "staticskyInput", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + psFree(where); + return false; + } + psFree(output); + psFree(where); + return true; + } + + psString inputsSQL = pxDataGet("staticskytool_definebyquery_inputs.sql"); + if (!inputsSQL) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + bool status; + + // pull out the skycell_id, tess_id, filter + psString skycell_id = psMetadataLookupStr(&status, row, "skycell_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup skycell_id"); + psFree(output); + psFree(inputsSQL); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + psString tess_id = psMetadataLookupStr(&status, row, "tess_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup tess_id"); + psFree(output); + psFree(inputsSQL); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psString by_data_group = NULL; + psString stack_data_group = NULL; + if (group_by_data_group) { + stack_data_group = psMetadataLookupStr(&status, row, "data_group"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to lookup data_group"); + psFree(output); + psFree(inputsSQL); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psStringAppend(&by_data_group, "\nAND stackRun.data_group = '%s'", stack_data_group); + } + + // query for the inputs + if (!p_psDBRunQueryF(config->dbh, inputsSQL, tess_id, skycell_id, where, + by_data_group ? by_data_group : "")) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(inputsSQL); + return false; + } + + psArray *inputs = p_psDBFetchResult(config->dbh); + if (!inputs) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + psFree(inputs); + psFree(output); + psFree(inputsSQL); + return false; + } + if (!psArrayLength(inputs)) { + psWarning("staticskytool ERROR: no rows found for known tess_id, skycell_id?"); + continue; + } + + if (check_inputs) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, inputs, "staticskyInput", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + psFree(where); + return false; + } + psFree(inputs); + continue; + } + + // XXX if we are unable to guarantee that all selected inputs have all and only the + // requested filters, see the code at the bottom of this file (ifdef'ed out for now) + + // insert a new staticsky entry and find its new id + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(inputs); + psFree(output); + psFree(inputsSQL); + return false; + } + + // create a staticskyRun + if (!staticskyRunInsert(config->dbh, + 0x0, // sky_id + "new", // state + workdir, + label, + data_group ? data_group : (stack_data_group ? stack_data_group : label), + dist_group, + reduction, + registered, + note + ) + ) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error failed to rollback transaction"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(inputs); + psFree(output); + psFree(inputsSQL); + return false; + } + + psS64 sky_id = psDBLastInsertID(config->dbh); + + // loop over the possible inputs and record only the valid ones + for (int j = 0; j < inputs->n; j++) { + psMetadata *inputRow = inputs->data[j]; // Row from select + + // pull out the skycell_id, tess_id, filter + psS64 stack_id = psMetadataLookupS64(&status, inputRow, "stack_id"); + psAssert(status, "failed to find stack_id?"); + + // add a staticskyInput entry + if (!staticskyInputInsert(config->dbh, sky_id, stack_id)) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error failed to rollback transaction"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(inputs); + psFree(output); + psFree(inputsSQL); + return false; + } + } + psFree (inputs); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(inputsSQL); + return false; + } + } + psFree(inputsSQL); + psFree(output); + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sky_id", "sky_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "staticskyRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "staticskyRun.data_group", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "staticskyRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "=="); + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE staticskyRun JOIN staticskyInput USING(sky_id) JOIN stackRun using(stack_id) JOIN skycell USING(tess_id, skycell_id)"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "staticskyRun", "sky_id", "staticskyResult", true, false); + psFree(query); + psFree(where); + + return result; +} + + +static bool inputsMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // XXX require at least a sky id (add better search options) + // PXOPT_LOOKUP_S64(sky_id, config->args, "-sky_id", true, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sky_id", "sky_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("staticskytool_inputs.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "staticskyInput"); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("staticskytool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "staticskyInput", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool todoMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *whereMD = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, whereMD, "-sky_id", "sky_id", "=="); + pxAddLabelSearchArgs (config, whereMD, "-label", "staticskyRun.label", "=="); + pxskycellAddWhere(config, whereMD); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("staticskytool_todo.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!psListLength(whereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(whereMD); + return false; + } + + psString where = NULL; + psString whereClause = psDBGenerateWhereConditionSQL(whereMD, NULL); + psStringAppend(&where, "\n AND %s", whereClause); + psFree(whereClause); + psFree(whereMD); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + // the where clause is required and matches the WHERE hook format string + if (!p_psDBRunQueryF(config->dbh, query, where)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("staticskytool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "staticskyResult", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return true; +} + +static bool addresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(sky_id, config->args, "-sky_id", true, false); + + // optional + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_F32(dtime_phot, config->args, "-dtime_phot", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_S32(sources, config->args, "-sources", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_F32(good_frac, config->args, "-good_frac", false, false); + PXOPT_LOOKUP_S32(num_inputs, config->args, "-num_inputs", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + // XXX not sure we need a transaction here... + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!staticskyResultInsert(config->dbh, + sky_id, + path_base, + dtime_phot, + dtime_script, + sources, + num_inputs, + hostname, + good_frac, + fault, + quality + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!fault) { + if (!setstaticskyRunState(config, sky_id, "full")) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to change staticskyRun state"); + return false; + } + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +// XXX what is this used by? what filters are needed? +static bool resultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sky_id", "staticskyRun.sky_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "staticskyInput.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "staticskyRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "staticskyRun.data_group", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "LIKE"); + PXOPT_COPY_S16(config->args, where, "-fault", "staticskyResult.fault", "=="); + pxskycellAddWhere(config, where); + PXOPT_LOOKUP_S32(num_filters, config->args, "-num_filters", false, false); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("staticskytool_result.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + psFree(where); + + psStringAppend(&query, "\nGROUP BY sky_id"); + if (num_filters) { + psStringAppend(&query, "\nHAVING COUNT(filter) = %d", num_filters); + } + + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("staticskytool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "staticskyResult", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool revertMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sky_id", "staticskyResult.sky_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "staticskyRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "staticskyResult.fault", "=="); + + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString delete = pxDataGet("staticskytool_revert.sql"); + if (!delete) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString update = pxDataGet("staticskytool_revert_update.sql"); + if (!update) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&delete, " AND %s", whereClause); + psStringAppend(&update, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, delete)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete); + psFree(where); + return false; + } + psFree(delete); + + int numRows = psDBAffectedRows(config->dbh); + psLogMsg("staticskytool", PS_LOG_INFO, "Deleted %d rows", numRows); + + if (!p_psDBRunQuery(config->dbh, update)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete); + psFree(where); + return false; + } + psFree(update); + + numRows = psDBAffectedRows(config->dbh); + psLogMsg("staticskytool", PS_LOG_INFO, "Updated %d rows", numRows); + + psFree(where); + return true; +} + +static bool updateresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sky_id", "sky_id", "=="); + + if (!pxSetFaultCode(config->dbh, "staticskyResult", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + return true; +} + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 3; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // XXX unused PXOPT_LOOKUP_S64(det_id, config->args, "-sky_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-sky_id", "sky_id", "=="); + + ExportTable tables [] = { + {"staticskyRun", "staticskytool_export_run.sql"}, + {"staticskyInput", "staticskytool_export_input.sql"}, + {"staticskyResult", "staticskytool_export_result.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + if (!strcmp(tables[i].tableName, "staticskyRun")) { + if (!pxSetStateCleaned("staticskyRun", "state", output)) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + return false; +} + +bool exportskycalrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 2; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // XXX unused PXOPT_LOOKUP_S64(det_id, config->args, "-sky_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycal_id", "=="); + + ExportTable tables [] = { + {"skycalRun", "staticskytool_export_skycalrun.sql"}, + {"skycalResult", "staticskytool_export_skycalresult.sql"}, + }; + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + if (!strcmp(tables[i].tableName, "skycalRun")) { + if (!pxSetStateCleaned("skycalRun", "state", output)) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + } + + fclose (f); + + return true; +} + +bool importskycalrunMode(pxConfig *config) +{ + return false; +} + +static bool setstaticskyRunState(pxConfig *config, psS64 sky_id, const char *state) +{ + PS_ASSERT_PTR_NON_NULL(state, false); + + // check that state is a valid string value + if (!pxIsValidState(state)) { + psError(PS_ERR_UNKNOWN, false, "invalid staticskyRun state: %s", state); + return false; + } + + char *query = "UPDATE staticskyRun SET state = '%s' WHERE sky_id = %"PRId64; + if (!p_psDBRunQueryF(config->dbh, query, state, sky_id)) { + psError(PS_ERR_UNKNOWN, false, "failed to change state for sky_id %"PRId64, sky_id); + return false; + } + return true; + +} + +static bool defineskycalrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required options + // none required. We get workdir, etc from staticskyRun if not provided + + // optional + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-set_registered", false, false); + + psMetadata *whereMD = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, whereMD, "-select_sky_id", "staticskyRun.sky_id", "=="); + PXOPT_COPY_S64(config->args, whereMD, "-select_stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_STR(config->args, whereMD, "-select_skycell_id", "stackRun.skycell_id", "=="); + PXOPT_COPY_STR(config->args, whereMD, "-select_tess_id", "stackRun.tess_id", "=="); + pxAddLabelSearchArgs(config, whereMD, "-select_filter", "stackRun.filter", "LIKE"); + PXOPT_COPY_F32(config->args, whereMD, "-select_good_frac_min", "stackSumSkyfile.good_frac", ">="); + pxAddLabelSearchArgs(config, whereMD, "-select_label", "staticskyRun.label", "LIKE"); + pxAddLabelSearchArgs(config, whereMD, "-select_data_group", "staticskyRun.data_group", "LIKE"); + if (!pxskycellAddWhere(config, whereMD)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("staticskytool_defineskycalrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(whereMD); + return false; + } + + if (!psListLength(whereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(whereMD); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(whereMD, NULL); + psStringAppend(&query, "\nAND %s", whereClause); + psFree(whereClause); + psFree(whereMD); + + psString labelHook = psStringCopy(""); + if (!rerun) { + if (label) { + psStringAppend(&labelHook, "\nAND skycalRun.label = '%s'", label); + } + psStringAppend(&query, "\nAND skycal_id IS NULL"); + } + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQueryF(config->dbh, query, labelHook)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(labelHook); + psFree(query); + return false; + } + psFree(labelHook); + psFree(query); + + // we now have a list of (tess_id, skycell_id) that (potentially) meet out needs + // we now need to loop over all of these and for each pair, select the best set of + // inputs + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + return false; + } + if (!psArrayLength(output)) { + psWarning("staticskytool: no rows found"); + psFree(output); + return true; + } + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "skycalRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + for (long i = 0; i < output->n; i++) { + psMetadata *row = output->data[i]; // Row from select + bool status; + + psS64 sky_id = psMetadataLookupS64(&status, row, "sky_id"); + psS64 stack_id = psMetadataLookupS64(&status, row, "stack_id"); + psString sky_workdir = psMetadataLookupStr(&status, row, "workdir"); + psString sky_label = psMetadataLookupStr(&status, row, "label"); + psString sky_data_group = psMetadataLookupStr(&status, row, "data_group"); + psString sky_dist_group = psMetadataLookupStr(&status, row, "dist_group"); + + // create a staticskyRun + if (!skycalRunInsert(config->dbh, + 0x0, // skycal_id + sky_id, + stack_id, + "new", // state + workdir ? workdir : sky_workdir, + label ? label : sky_label, + data_group ? data_group : sky_data_group, + dist_group ? dist_group : sky_dist_group, + reduction, + registered, + note + ) + ) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + return false; + } + + } + psFree(output); + return true; +} + +static bool updateskycalrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sky_id", "sky_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "skycalRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "skycalRun.state", "=="); + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE skycalRun JOIN stackRun using(stack_id) join skycell using(tess_id, skycell_id)"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "skycalRun", "skycal_id", "skycalResult", true, false); + psFree(query); + psFree(where); + + return result; +} + +static bool pendingskycalrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *whereMD = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, whereMD, "-skycal_id", "skycal_id", "=="); + PXOPT_COPY_S64(config->args, whereMD, "-sky_id", "sky_id", "=="); + pxAddLabelSearchArgs (config, whereMD, "-label", "skycalRun.label", "=="); + PXOPT_COPY_STR(config->args, whereMD, "-filter", "stackRun.filter", "LIKE"); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("staticskytool_pendingskycalrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!psListLength(whereMD->list)) { + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + psFree(whereMD); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(whereMD, NULL); + psStringAppend(&query, "\n AND %s", whereClause); + psFree(whereClause); + psFree(whereMD); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, "\n%s", limitString); + psFree(limitString); + } + + // the where clause is required and matches the WHERE hook format string + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("staticskytool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "pendingskycalRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return true; +} + +static bool addskycalresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // required + PXOPT_LOOKUP_S64(skycal_id, config->args, "-skycal_id", true, false); + + // optional + PXOPT_LOOKUP_F32(sigma_ra, config->args, "-sigma_ra", false, false); + PXOPT_LOOKUP_F32(sigma_dec, config->args, "-sigma_dec", false, false); + + PXOPT_LOOKUP_F32(zpt_obs, config->args, "-zpt_obs", false, false); + PXOPT_LOOKUP_F32(zpt_stdev, config->args, "-zpt_stdev", false, false); + PXOPT_LOOKUP_F32(fwhm_major, config->args, "-fwhm_major", false, false); + PXOPT_LOOKUP_F32(fwhm_minor, config->args, "-fwhm_minor", false, false); + + + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_F32(dtime_astrom, config->args, "-dtime_astrom", false, false); + + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_S32(n_astrom, config->args, "-n_astrom", false, false); + PXOPT_LOOKUP_S32(n_detections, config->args,"-n_detections", false, false); + PXOPT_LOOKUP_S32(n_extended, config->args, "-n_extended", false, false); + PXOPT_LOOKUP_S32(n_forced, config->args, "-n_forced", false, false); + + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_STR(ver_psastro, config->args, "-ver_psastro", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + + psString software_ver = NULL; + if ((ver_pslib)&&(ver_psmodules)) { + software_ver = pxMergeCodeVersions(ver_pslib,ver_psmodules); + } + if (ver_psphot) { + software_ver = pxMergeCodeVersions(software_ver,ver_psphot); + } + if (ver_psastro) { + software_ver = pxMergeCodeVersions(software_ver,ver_psastro); + } + if (ver_ppstats) { + software_ver = pxMergeCodeVersions(software_ver,ver_ppstats); + } + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + skycalResultRow *row = skycalResultRowAlloc( + skycal_id, + path_base, + dtime_script, + dtime_astrom, + sigma_ra, + sigma_dec, + n_astrom, + n_detections, + n_extended, + n_forced, + zpt_obs, + zpt_stdev, + fwhm_major, + fwhm_minor, + quality, + software_ver, + hostname, + fault + ); + + if (!skycalResultInsertObject(config->dbh, row)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(row); + return false; + } + + psFree(row); + + if (fault) { + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + return true; + } + + psString query = "UPDATE skycalRun SET state = 'full' WHERE skycal_id = %" PRId64; + if (!p_psDBRunQueryF(config->dbh, query, skycal_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool updateskycalresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycal_id", "=="); + + if (!pxSetFaultCode(config->dbh, "skycalResult", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree (where); + return false; + } + psFree (where); + return true; +} +static bool skycalresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalRun.skycal_id", "=="); + PXOPT_COPY_S64(config->args, where, "-sky_id", "skycalRun.sky_id", "=="); + PXOPT_COPY_S64(config->args, where, "-stack_id", "stackRun.stack_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "stackRun.tess_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "stackRun.skycell_id", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-filter", "stackRun.filter", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-label", "skycalRun.label", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "skycalRun.data_group", "LIKE"); + PXOPT_COPY_S16(config->args, where, "-quality", "skycalResult.quality", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "skycalResult.fault", "=="); + + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("staticskytool_skycalresult.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (!psListLength(where->list)) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters are required"); + return false; + } + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("staticskytool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + if (!ippdbPrintMetadatas(stdout, output, "staticskyResult", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} +static bool revertskycalresultMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-skycal_id", "skycalResult.skycal_id", "=="); + pxAddLabelSearchArgs(config, where, "-label", "skycalRun.label", "=="); + pxAddLabelSearchArgs(config, where, "-data_group", "skycalRun.data_group", "=="); + pxAddLabelSearchArgs(config, where, "-filter", "stackRun.filter", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "skycalResult.fault", "=="); + if (!pxskycellAddWhere(config, where)) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to add skycell search arguments"); + return false; + } + + if (!psListLength(where->list) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // Delete product + psString delete = pxDataGet("staticskytool_revertskycal.sql"); + if (!delete) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&delete, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, delete)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(delete); + psFree(where); + return false; + } + psFree(delete); + + int numRows = psDBAffectedRows(config->dbh); // Number of row affected + psLogMsg("staticskytool", PS_LOG_INFO, "Deleted %d rows", numRows); + + psFree(where); + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytool.h (revision 41104) @@ -0,0 +1,51 @@ +/* + * staticskytool.h + * + * Copyright (C) 2007,2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef STATICSKYTOOL_H +#define STATICSKYTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + STATICSKYTOOL_MODE_NONE = 0x0, + STATICSKYTOOL_MODE_DEFINEBYQUERY, + STATICSKYTOOL_MODE_UPDATERUN, + STATICSKYTOOL_MODE_INPUTS, + STATICSKYTOOL_MODE_TODO, + STATICSKYTOOL_MODE_ADDRESULT, + STATICSKYTOOL_MODE_RESULT, + STATICSKYTOOL_MODE_REVERT, + STATICSKYTOOL_MODE_UPDATERESULT, + STATICSKYTOOL_MODE_EXPORTRUN, + STATICSKYTOOL_MODE_IMPORTRUN, + + STATICSKYTOOL_MODE_DEFINESKYCALRUN, + STATICSKYTOOL_MODE_PENDINGSKYCALRUN, + STATICSKYTOOL_MODE_UPDATESKYCALRUN, + STATICSKYTOOL_MODE_ADDSKYCALRESULT, + STATICSKYTOOL_MODE_UPDATESKYCALRESULT, + STATICSKYTOOL_MODE_SKYCALRESULT, + STATICSKYTOOL_MODE_REVERTSKYCALRESULT, + STATICSKYTOOL_MODE_EXPORTSKYCALRUN, + STATICSKYTOOL_MODE_IMPORTSKYCALRUN, +} staticskytoolMode; + +pxConfig *staticskytoolConfig(pxConfig *config, int argc, char **argv); + +#endif // STATICSKYTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/staticskytoolConfig.c (revision 41104) @@ -0,0 +1,334 @@ +/* + * staticskytoolConfig.c + * + * Copyright (C) 2007-2010 Joshua Hoblitt, Paul Price + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include "pxtools.h" +#include "staticskytool.h" + +pxConfig *staticskytoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definebyquery + // XXX need a 'ra_min,max', 'dec_min,max' : to do this, we need a table of skycell boundaries + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-select_stack_id", 0, "search for stack_id", 0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_skycell_id", 0, "search for skycell_id", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_tess_id", 0, "search for tess_id", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_good_frac_min", 0, "define min good_frac", NAN); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_label", PS_META_DUPLICATE_OK, "search by stackRun label (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_data_group", PS_META_DUPLICATE_OK, "search by stackRun data_group (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-select_filter", PS_META_DUPLICATE_OK, "search by filter (LIKE comparison, multiple OK)", NULL); + pxskycellAddArguments(definebyqueryArgs); +#ifdef notdef + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_glat_min", 0, "define min galactic latitude", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_glat_max", 0, "define max galactic latitude", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_rahours_min", 0, "define min RA in hours", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_rahours_max", 0, "define max RA in hours", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_radeg_min", 0, "define min RA in degrees", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_radeg_max", 0, "define max RA in degrees", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_decdeg_min", 0, "define min DEC in degrees", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-select_decdeg_max", 0, "define max DEC in degrees", NAN); +#endif + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-group_by_data_group", 0, "queue unique run for each data_group", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-rerun", 0, "queue new run even if one already exists for inputs", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-check_inputs", 0, "list inputs, do not modify database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-sky_id", 0, "search by stack ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess_id", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", NULL); + pxskycellAddArguments(updaterunArgs); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -inputs + psMetadata *inputsArgs = psMetadataAlloc(); + psMetadataAddS64(inputsArgs, PS_LIST_TAIL, "-sky_id", 0, "search by staticsky ID", 0); + psMetadataAddU64(inputsArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(inputsArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -todo + psMetadata *todoArgs = psMetadataAlloc(); + psMetadataAddS64(todoArgs, PS_LIST_TAIL, "-sky_id", 0, "search by staticsky ID", 0); + psMetadataAddStr(todoArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + pxskycellAddArguments(todoArgs); + psMetadataAddU64(todoArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(todoArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addresult + psMetadata *addresultArgs = psMetadataAlloc(); + psMetadataAddS64(addresultArgs, PS_LIST_TAIL, "-sky_id", 0, "define sky ID (required)", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", 0); + psMetadataAddF32(addresultArgs, PS_LIST_TAIL, "-dtime_phot", 0, "define photometry time", NAN); + psMetadataAddF32(addresultArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddS32(addresultArgs, PS_LIST_TAIL, "-sources", 0, "number of sources", 0); + psMetadataAddS32(addresultArgs, PS_LIST_TAIL, "-num_inputs", 0, "number of inputs", 0); + psMetadataAddStr(addresultArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddF32(addresultArgs, PS_LIST_TAIL, "-good_frac", 0, "define fraction of good pixels", NAN); + psMetadataAddS16(addresultArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + psMetadataAddS16(addresultArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -result + psMetadata *resultArgs= psMetadataAlloc(); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-sky_id", 0, "search by staticsky ID", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID (LIKE comparison)", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID (LIKE comparison)", 0); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-label", 0, "search by label", NULL); + psMetadataAddStr(resultArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddS16(resultArgs, PS_LIST_TAIL, "-num_filters", 0, "search by number of filters in the inputs", 0); + psMetadataAddS64(resultArgs, PS_LIST_TAIL, "-stack_id", 0, "search by input stack ID (if used num_filters will be wrong)", 0); + pxskycellAddArguments(resultArgs); + psMetadataAddS16(resultArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddU64(resultArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(resultArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(resultArgs, PS_LIST_TAIL, "-all", 0, "allow no search terms", false); + + // -revert + psMetadata *revertArgs= psMetadataAlloc(); + psMetadataAddS64(revertArgs, PS_LIST_TAIL, "-sky_id", 0, "search by staticsky ID", 0); + psMetadataAddStr(revertArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddS16(revertArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + pxskycellAddArguments(revertArgs); + psMetadataAddBool(revertArgs, PS_LIST_TAIL, "-all", 0, "allow no search terms", false); + + // -updateresult + psMetadata *updateresultArgs = psMetadataAlloc(); + psMetadataAddS64(updateresultArgs, PS_LIST_TAIL, "-sky_id", 0, "define staticksky ID (required)", 0); + psMetadataAddS16(updateresultArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + psMetadataAddS16(updateresultArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality", 0); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-sky_id", 0, "export this staticsky ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + // -defineskycalrun + psMetadata *defineskycalrunArgs = psMetadataAlloc(); + psMetadataAddS64(defineskycalrunArgs, PS_LIST_TAIL, "-select_sky_id", 0, "search for sky_id", 0); + psMetadataAddS64(defineskycalrunArgs, PS_LIST_TAIL, "-select_stack_id", 0, "search for stack_id", 0); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-select_skycell_id", 0, "search for skycell_id", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-select_tess_id", 0, "search for tess_id", NULL); + psMetadataAddF32(defineskycalrunArgs, PS_LIST_TAIL, "-select_good_frac_min", 0, "define min good_frac", NAN); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-select_label", PS_META_DUPLICATE_OK, "search by stackRun label (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-select_data_group", PS_META_DUPLICATE_OK, "search by stackRun data_group (LIKE comparison, multiple OK)", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-select_filter", PS_META_DUPLICATE_OK, "search by filter (LIKE comparison, multiple OK)", NULL); + pxskycellAddArguments(defineskycalrunArgs); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddStr(defineskycalrunArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction", NULL); + psMetadataAddTime(defineskycalrunArgs, PS_LIST_TAIL, "-set_registered", 0, "time detrend run was registered", now); + psMetadataAddBool(defineskycalrunArgs, PS_LIST_TAIL, "-rerun", 0, "queue new run even if one already exists for inputs", false); + psMetadataAddBool(defineskycalrunArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(defineskycalrunArgs, PS_LIST_TAIL, "-check_inputs", 0, "list inputs, do not modify database", false); + psMetadataAddBool(defineskycalrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(defineskycalrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -updateskycalrun + psMetadata *updateskycalrunArgs = psMetadataAlloc(); + psMetadataAddS64(updateskycalrunArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal ID", 0); + psMetadataAddS64(updateskycalrunArgs, PS_LIST_TAIL, "-sky_id", 0, "search by sky ID", 0); + psMetadataAddS64(updateskycalrunArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell_id", 0); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-label", 0, "search by label", 0); + pxskycellAddArguments(updateskycalrunArgs); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", NULL); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updateskycalrunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + + // -pendingskycalrun + psMetadata *pendingskycalrunArgs = psMetadataAlloc(); + psMetadataAddS64(pendingskycalrunArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal ID", 0); + psMetadataAddS64(pendingskycalrunArgs, PS_LIST_TAIL, "-sky_id", 0, "search by staticsky ID", 0); + psMetadataAddStr(pendingskycalrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddStr(pendingskycalrunArgs, PS_LIST_TAIL, "-filter", 0, "search by filter", NULL); + psMetadataAddU64(pendingskycalrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingskycalrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + + // -addskycalresult + psMetadata *addskycalresultArgs = psMetadataAlloc(); + psMetadataAddS64(addskycalresultArgs, PS_LIST_TAIL, "-skycal_id", 0, "define camtool ID (required)", 0); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-sigma_ra", 0, "define exposure E ra", NAN); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-sigma_dec", 0, "define exposure E dec", NAN); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-zpt_obs", 0, "define observed zero point", NAN); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-zpt_stdev", 0, "define observed zero point stdandard deviation", NAN); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-fwhm_major", 0, "define fwhm (major axis; pixels)", NAN); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-fwhm_minor", 0, "define fwhm (minor axis; pixels)", NAN); + + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddF32(addskycalresultArgs, PS_LIST_TAIL, "-dtime_astrom", 0, "define elapsed time for astrometry (seconds)", NAN); + + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + // psMetadataAddS32(addskycalresultArgs, PS_LIST_TAIL, "-n_stars", 0, "define number of stars", 0); + psMetadataAddS32(addskycalresultArgs, PS_LIST_TAIL, "-n_astrom", 0, "define number of astrometry reference objects", 0); + psMetadataAddS32(addskycalresultArgs, PS_LIST_TAIL, "-n_detections", 0, "define number of detections", 0); + psMetadataAddS32(addskycalresultArgs, PS_LIST_TAIL, "-n_extended", 0, "define number of extended detections", 0); + psMetadataAddS32(addskycalresultArgs, PS_LIST_TAIL, "-n_forced", 0, "define number of forced detections", 0); + + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", NULL); + psMetadataAddS16(addskycalresultArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(addskycalresultArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-ver_psastro", 0, "define psastro version", NULL); + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(addskycalresultArgs, PS_LIST_TAIL, "-ver_streaks", 0, "define streaksremove version", NULL); + + // -revertskycal + psMetadata *revertskycalArgs= psMetadataAlloc(); + psMetadataAddS64(revertskycalArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal ID", 0); + psMetadataAddStr(revertskycalArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + psMetadataAddStr(revertskycalArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by data_group", 0); + psMetadataAddStr(revertskycalArgs, PS_LIST_TAIL, "-filter", PS_META_DUPLICATE_OK, "search by filter", 0); + pxskycellAddArguments(revertskycalArgs); + psMetadataAddS16(revertskycalArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertskycalArgs, PS_LIST_TAIL, "-all", 0, "allow no search terms", false); + + // -updateskycalresult + psMetadata *updateskycalresultArgs = psMetadataAlloc(); + psMetadataAddS64(updateskycalresultArgs, PS_LIST_TAIL, "-skycal_id", 0, "define staticksky ID (required)", 0); + psMetadataAddS16(updateskycalresultArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code (required)", 0); + psMetadataAddS16(updateskycalresultArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality code", 0); + + // -skycalresult + psMetadata *skycalresultArgs= psMetadataAlloc(); + psMetadataAddS64(skycalresultArgs, PS_LIST_TAIL, "-skycal_id", 0, "search by skycal ID", 0); + psMetadataAddS64(skycalresultArgs, PS_LIST_TAIL, "-sky_id", 0, "search by staticsky ID", 0); + psMetadataAddS64(skycalresultArgs, PS_LIST_TAIL, "-stack_id", 0, "search by stack ID", 0); + psMetadataAddStr(skycalresultArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID (LIKE comparison)", 0); + psMetadataAddStr(skycalresultArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID (LIKE comparison)", 0); + psMetadataAddStr(skycalresultArgs, PS_LIST_TAIL, "-filter", 0, "search by filter (LIKE comparison)", 0); + psMetadataAddStr(skycalresultArgs, PS_LIST_TAIL, "-label", 0, "search by label (LIKE comparison)", NULL); + psMetadataAddStr(skycalresultArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddS16(skycalresultArgs, PS_LIST_TAIL, "-quality", 0, "search by quality", 0); + psMetadataAddS16(skycalresultArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + pxskycellAddArguments(skycalresultArgs); + psMetadataAddU64(skycalresultArgs, PS_LIST_TAIL, "-limit", 0, "limit skycalresult set to N items", 0); + psMetadataAddBool(skycalresultArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -exportskycalrun + psMetadata *exportskycalrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportskycalrunArgs, PS_LIST_TAIL, "-skycal_id", 0, "export this skycal ID (required)", 0); + psMetadataAddStr(exportskycalrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportskycalrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportskycalrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + + // -importrun + psMetadata *importskycalrunArgs = psMetadataAlloc(); + psMetadataAddStr(importskycalrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + + psFree(now); + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "Define a new run", STATICSKYTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-updaterun", "Update a run", STATICSKYTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-inputs", "Get inputs", STATICSKYTOOL_MODE_INPUTS, inputsArgs); + PXOPT_ADD_MODE("-todo", "Get runs to do", STATICSKYTOOL_MODE_TODO, todoArgs); + PXOPT_ADD_MODE("-addresult", "Add result of run", STATICSKYTOOL_MODE_ADDRESULT, addresultArgs); + PXOPT_ADD_MODE("-result", "Get result of run", STATICSKYTOOL_MODE_RESULT, resultArgs); + PXOPT_ADD_MODE("-revert", "Revert failed run", STATICSKYTOOL_MODE_REVERT, revertArgs); + PXOPT_ADD_MODE("-updateresult", "Update fault for run", STATICSKYTOOL_MODE_UPDATERESULT, updateresultArgs); + PXOPT_ADD_MODE("-exportrun", "Export run", STATICSKYTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "Import run", STATICSKYTOOL_MODE_IMPORTRUN, importrunArgs); + PXOPT_ADD_MODE("-defineskycalrun", "Define a new skycalrun", STATICSKYTOOL_MODE_DEFINESKYCALRUN, defineskycalrunArgs); + PXOPT_ADD_MODE("-updateskycalrun", "Update a skycalrun", STATICSKYTOOL_MODE_UPDATESKYCALRUN, updateskycalrunArgs); + PXOPT_ADD_MODE("-pendingskycalrun", "Get skcal runs to do", STATICSKYTOOL_MODE_PENDINGSKYCALRUN, pendingskycalrunArgs); + PXOPT_ADD_MODE("-addskycalresult", "add skycal result", STATICSKYTOOL_MODE_ADDSKYCALRESULT, addskycalresultArgs); + PXOPT_ADD_MODE("-revertskycal", "revert faulted skycal run", STATICSKYTOOL_MODE_REVERTSKYCALRESULT, revertskycalArgs); + PXOPT_ADD_MODE("-updateskycal", "revert faulted skycal run", STATICSKYTOOL_MODE_UPDATESKYCALRESULT, updateskycalresultArgs); + PXOPT_ADD_MODE("-skycalresult", "Get result of skycal run", STATICSKYTOOL_MODE_SKYCALRESULT, skycalresultArgs); + PXOPT_ADD_MODE("-exportskycalrun", "Export skycal run", STATICSKYTOOL_MODE_EXPORTSKYCALRUN, exportskycalrunArgs); + PXOPT_ADD_MODE("-importskycalrun", "Import skycal run", STATICSKYTOOL_MODE_IMPORTSKYCALRUN, importskycalrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/vptool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/vptool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/vptool.c (revision 41104) @@ -0,0 +1,1848 @@ +/* + * vptool.c + * + * Copyright (C) 2011 IfA University of Hawaii + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "pxdata.h" +#include "pxchip.h" + +#include "vptool.h" + +static bool definebyqueryMode(pxConfig *config); +static bool pendingrunMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool revertrunMode(pxConfig *config); +static bool listrunMode(pxConfig *config); +static bool pendingimfileMode(pxConfig *config); +static bool addprocessedcellMode(pxConfig *config); +static bool processedcellMode(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) { + psLibInit(NULL); + + pxConfig *config = vptoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(VPTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(VPTOOL_MODE_PENDINGRUN, pendingrunMode); + MODECASE(VPTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(VPTOOL_MODE_REVERTRUN, revertrunMode); + MODECASE(VPTOOL_MODE_LISTRUN, listrunMode); + MODECASE(VPTOOL_MODE_PENDINGIMFILE, pendingimfileMode); + MODECASE(VPTOOL_MODE_ADDPROCESSEDCELL, addprocessedcellMode); + MODECASE(VPTOOL_MODE_PROCESSEDCELL, processedcellMode); + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + +// Queue exposures for videophot processing +static bool queue_exposures(pxConfig *config, // Configuration + const psArray *exps, // Exposures information + const char *label, // Label, or NULL to inherit + const char *data_group, // Data group, or NULL to inherit + const char *workdir, // Working directory, or NULL to inherit + const char *note, // Note + const psS64 dest_id + ) +{ + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + +# define GET_VALUE(PTYPE,CTYPE,VALUE,NAME) \ + PTYPE VALUE; \ + { \ + bool status; \ + VALUE = psMetadataLookup##CTYPE(&status, md, NAME); \ + if (!status) { \ + psError(PS_ERR_UNKNOWN, false, "failed to lookup value for %s", NAME); \ + return false; \ + } \ + } + + // loop over our list of exp_ids + for (long i = 0; i < psArrayLength(exps); i++) { + psMetadata *md = exps->data[i]; + + rawExpRow *row = rawExpObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into chipRun"); + return false; + } + + GET_VALUE (psS64, S64, exp_id, "exp_id"); + GET_VALUE (psString, Str, raw_workdir, "workdir"); + GET_VALUE (psString, Str, raw_label, "label"); + // GET_VALUE (psString, Str, raw_reduction, "reduction"); + // GET_VALUE (psString, Str, raw_expgroup, "expgroup"); + // GET_VALUE (psString, Str, raw_dvodb, "dvodb"); + // GET_VALUE (psString, Str, raw_tess_id, "tess_id"); + // GET_VALUE (psString, Str, raw_end_stage, "end_stage"); + + if (!row->exp_id) { + psError(PS_ERR_UNKNOWN, false, "failed to find value for exp_id"); + return false; + } + + // queue the exp + if (!vpRunInsert(config->dbh, + 0, //vp_id + exp_id, + "new", + label ? label : raw_label, + data_group ? data_group : (label ? label : raw_label), + workdir ? workdir : raw_workdir, + note, + dest_id, + NULL, // outroot + 0, // dtime_script + NULL, // hostname + 0 // fault + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, + "failed to trying to queue exp_id: %" PRId64, exp_id); + return false; + } + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_S64(dest_id, config->args, "-set_dest_id", false, false); + + // default + PXOPT_LOOKUP_BOOL(rerun, config->args, "-rerun", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // rawExp only + pxAddLabelSearchArgs (config, where, "-label", "newExp.label", "="); // define using newExp label + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("vptool_find_rawexp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + + psFree(whereClause); + psFree(where); + + if (!rerun) { + psStringAppend(&query, "\n AND vp_id IS NULL\n AND (vpRun.label = '%s' OR vpRun.label IS NULL)", label); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!queue_exposures(config, output, label, data_group, workdir, note, dest_id)) { + psError(psErrorCodeLast(), false, "Unable to queue exposures for videophot processing."); + psFree(output); + return false; + } + psFree(output); + + return true; +} + +static bool pendingrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + // Do we need pxchipArgs? + pxchipGetSearchArgs (config, where); //chipRun, rawExp + PXOPT_COPY_S64(config->args, where, "-vp_id", "vpRun.vp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "vpRun.label", "=="); + + psString query = pxDataGet("vptool_pendingrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psStringAppend(&query, "\nORDER BY priority DESC, vp_id"); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "vpPendingRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +#ifdef notdef +static bool definecopyMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(exp_id, config->args, "-exp_id", false, false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // rawExp only + pxAddLabelSearchArgs (config, where, "-label", "chipOld.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipOld.data_group", "LIKE"); + + // psListLength(where->list) is at least 1 because exp_type defaults to "object" + // so we require a list longer than 1 entry + if ((psListLength(where->list) <= 1) && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", true, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", true, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(expgroup, config->args, "-set_expgroup", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-set_end_stage", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + // default + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("vptool_definecopy.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + + psFree(whereClause); + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + if (!queue_exposures(config, output, workdir, label, data_group, dist_group, + note)) { + psError(psErrorCodeLast(), false, "Unable to queue exposures for chip."); + psFree(output); + return false; + } + psFree(output); + + return true; +} +#endif + + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); +// pxchipGetSearchArgs (config, where); // rawExp, vpRun + PXOPT_COPY_S64(config->args, where, "-vp_id", "vpRun.vp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-label", "vpRun.label", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-state", "vpRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group", "vpRun.data_group", "LIKE"); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-set_hostname", false, false); + PXOPT_LOOKUP_STR(outroot, config->args, "-set_outroot", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-set_fault", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-set_dtime_script", false, false); + + if (!state && !label && !data_group && !note && !hostname && !fault) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, true, "must supply at least one set parameter"); + return false; + } + + psString query = psStringCopy("UPDATE vpRun"); + + char *separator = " SET "; + char * comma = ","; + +# define addColumn(_tab, _fmt, _val) \ + do { \ + if (_val) { \ + psStringAppend(&query, "%s %s.%s =" _fmt, separator, _tab, #_val, _val); \ + separator = comma; \ + } \ + } while (0) + + addColumn("vpRun", "'\%s'", state); + addColumn("vpRun", "'\%s'", label); + addColumn("vpRun", "'\%s'", data_group); + addColumn("vpRun", "'\%s'", note); + addColumn("vpRun", "'\%s'", hostname); + addColumn("vpRun", "'\%s'", outroot); + addColumn("vpRun", "%d", fault); + addColumn("vpRun", "%f", dtime_script); + + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psFree(where); + psStringAppend(&query, "\nWHERE %s", whereClause); + psFree(whereClause); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + + psFree(query); + + return true; +} + + +static bool pendingimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + // Do we need pxchipArgs? + pxchipGetSearchArgs (config, where); //chipRun, rawExp + PXOPT_COPY_S64(config->args, where, "-vp_id", "vpRun.vp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "vpRun.label", "=="); + + psString query = pxDataGet("vptool_pendingimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "vpPendingImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool addprocessedcellMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // vp_id, class_id, and cell_id are required + PXOPT_LOOKUP_S64(vp_id, config->args, "-vp_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + PXOPT_LOOKUP_STR(cell_id, config->args, "-cell_id", true, false); + + // optional + PXOPT_LOOKUP_F32(dtime_photom, config->args, "-dtime_photom", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!vpProcessedCellInsert(config->dbh, + vp_id, + class_id, + cell_id, + dtime_photom, + quality, + path_base, + fault + )) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool processedcellMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipRun, chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-vp_id", "vp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_COPY_STR(config->args, where, "-cell_id", "cell_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "vpRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "vpRun.data_group", "LIKE"); + + psString query = pxDataGet("vptool_processedcell.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + psFree(where); + + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "vpProcessedCell", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool revertrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-vp_id", "vpRun.vp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "vpRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "vpProcessedCell.fault", "=="); + + // PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + // find vpRuns with faulted cells + psString query = pxDataGet("vptool_revertrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + int numUpdated = psDBAffectedRows(config->dbh); + psLogMsg("vptool", PS_LOG_INFO, "Reverted %d vpRuns", numUpdated); + + return true; +} + +#ifdef NOTDEF +static bool revertcleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipRun.data_group", "LIKE"); + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + + char* newState = NULL; + if (!state) { + state = "error_cleaned"; + } + if (!strcmp(state, "error_cleaned")) { + newState = "goto_cleaned"; + } else if (!strcmp(state, "error_purged")) { + newState = "goto_purged"; + } else if (!strcmp(state, "error_scrubbed")) { + newState = "goto_scrubbed"; + } else { + psError(PXTOOLS_ERR_CONFIG, true, "-state must be either error_cleaned, error_purged, or error_scrubbed"); + return false; + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("vptool_revertcleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, newState, state, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + return true; +} + +static bool updateprocessedcellMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, NULL); + if (psListLength(where->list) == 0) { + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + return false; + } + + if (!state) { + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + if (!pxSetFaultCode(config->dbh, "chipProcessedImfile", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + return false; + } + psFree(where); + } + else { + if (!pxchipProcessedImfileSetStateByQuery(config,where,state)) { + psError(PS_ERR_UNKNOWN, false, "failed to set chipProcessedImfile state"); + return(false); + } + } + + + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + if (!chipMaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (where->list->n < 1) { + psFree(where); + where = NULL; + } + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM chipMask"); + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psFree(where); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +// return the list of labels which are NOT blocked +static bool unmaskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + if (where->list->n < 1) { + psFree(where); + where = NULL; + } + + psString query = pxDataGet("vptool_unmasked.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where) { + psString whereClause = psDBGenerateWhereSQL(where, "chipUnmask"); + psFree(where); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipUnmask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + char *query = "DELETE FROM chipMask WHERE label = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + psString query = pxDataGet("vptool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipPendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool pendingcleanupimfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", false, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + if (chip_id) { + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + } + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + psString query = pxDataGet("vptool_pendingcleanupimfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipPendingCleanupImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psString query = pxDataGet("vptool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool runMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(state, config->args, "-state", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // make sure that the state string is valid + if (!pxIsValidState(state)) { + psError(PXTOOLS_ERR_CONFIG, false, "%s is not a valid state", state); + return false; + } + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipRun, chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "state", "=="); + + psString query = pxDataGet("vptool_run.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool advanceexpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + // look for completed chipPendingExp + // migrate them to chipProccessedExp & camPendingExp + psString query = pxDataGet("vptool_completely_processed_exp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + psS64 chip_id = psMetadataLookupS64(NULL,row,"chip_id"); + + psString software_ver = NULL; + psS64 maskfrac_npix = 0; + psF32 maskfrac_static = 0; + psF32 maskfrac_dynamic = 0; + psF32 maskfrac_magic = 0; + psF32 maskfrac_advisory = 0; + + // Calculate run level masking and software state + if (!pxCoalesceRunStatus(config,"vptool_coalesce_run.sql",chip_id, + &software_ver,&maskfrac_npix, + &maskfrac_static,&maskfrac_dynamic, + &maskfrac_magic,&maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to generate run level statistics"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + psMetadataAddStr(row, PS_LIST_TAIL, "software_ver", 0, "Software version", software_ver); + psMetadataAddS32(row, PS_LIST_TAIL, "maskfrac_npix", 0, "Number of pixels used for mask stats", (psS32) maskfrac_npix); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_static", 0, "Fraction of static masked pixels", maskfrac_static); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_dynamic", 0, "Fraction of dynamic masked pixels", maskfrac_dynamic); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_magic", 0, "Fraction of magic masked pixels", maskfrac_magic); + psMetadataAddF32(row, PS_LIST_TAIL, "maskfrac_advisory", 0, "Fraction of advisory pixels", maskfrac_advisory); +/* psWarning("ADVANCE %ld %s %d %f %f %f %f\n",chip_id,software_ver,maskfrac_npix,maskfrac_static,maskfrac_dynamic,maskfrac_magic,maskfrac_advisory); */ + chipRunRow *chipRun = chipRunObjectFromMetadata(row); + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // Set chipRun.software_ver to the appropriate value + if (software_ver) { + if (!pxSetRunSoftware(config, "chipRun", "chip_id", chip_id, software_ver)) { + psError(PS_ERR_UNKNOWN, false, "failed to set chipRun.software_ver for chip_id: %" PRId64, + chip_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // Set chipRun.maskfrac* to the appropriate values. + if (maskfrac_npix) { + if (!pxSetRunMaskfrac(config, "chipRun", "chip_id",chip_id, maskfrac_npix, maskfrac_static, + maskfrac_dynamic, maskfrac_magic, maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to set chipRun.software_ver for chip_id: %" PRId64, + chip_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // set chipRun.state to 'stop' and update the magicked state + if (!pxchipRunSetState(config, chipRun->chip_id, "full", chipRun->magicked)) { + psError(PS_ERR_UNKNOWN, false, "failed to change chipRun.state for chip_id: %" PRId64, chipRun->chip_id); + psFree(chipRun); + psFree(output); + return false; + } + + // should we stop here or proceed on to the cam stage? + // NULL for end_stage means go as far as possible + if (chipRun->end_stage && psStrcasestr(chipRun->end_stage, "chip")) { + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(chipRun); + continue; + } + // else continue on... + + // camQueueChipID() can only be run after chipRun.state has been set to + // stop + if (!pxcamQueueByChipID(config, + chipRun->chip_id, + chipRun->workdir, + chipRun->label, + chipRun->data_group, + chipRun->dist_group, + chipRun->reduction, + chipRun->expgroup, + chipRun->dvodb, + chipRun->tess_id, + chipRun->end_stage, + chipRun->magicked, + NULL // note does not propagate + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "failed to queue camPendingExp"); + psFree(chipRun); + psFree(output); + return false; + } + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(chipRun); + } + + psFree(output); + + return true; +} + +// update chipProcessedImfile.data_state to given value. +// afterwards, if all imfiles in the exposure have the new state, update the state for the exposure as well +// shared code for the modes -tocleanedimfile -tofullimfile -topurgedimfile + +static bool change_imfile_data_state(pxConfig *config, psString data_state, psString run_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // chip_id, class_id are required + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", true, false); + + psString query = pxDataGet("vptool_change_imfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString set_magicked_imfile = psStringCopy(""); + psString set_magicked_run = psStringCopy(""); + if (!strcmp(data_state, "full")) { + // if (chipProcessedImfile.magicked < 0 and rawImfile.magicked = 0) leave magicked unchanged. This will + // block warp processing until destreaking has been done + // otherwise copy magicked from the rawImfile + // Same thing for chipRun/rawExp + psStringAppend(&set_magicked_imfile, "\n , chipProcessedImfile.magicked = IF((chipProcessedImfile.magicked < 0" + " AND rawImfile.magicked = 0), chipProcessedImfile.magicked, rawImfile.magicked)"); + psStringAppend(&set_magicked_run, "\n , chipRun.magicked = IF((chipRun.magicked < 0 AND rawExp.magicked = 0), " + " chipRun.magicked, rawExp.magicked)"); + + } else if (!strcmp(data_state, "cleaned") || !strcmp(data_state, "purged")) { + // if magicked is non-zero set it to -1 + // Once one imfile has been cleaned, the chipRun is no longer 'magicked' + psStringAppend(&set_magicked_imfile, "\n, chipProcessedImfile.magicked = IF(chipProcessedImfile.magicked = 0, 0, -1)," + " chipRun.magicked = IF(chipRun.magicked = 0, 0, -1)"); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_imfile, chip_id, class_id)) { + psFree(query); + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + psFree(set_magicked_imfile); + psFree(set_magicked_run); + if (psDBAffectedRows(config->dbh) < 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected atleast 1 row"); + return false; + } + + query = pxDataGet("vptool_change_exp_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_run, chip_id, data_state)) { + psFree(query); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool tocleanedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "cleaned", "goto_cleaned"); +} +static bool tofullimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "full", "update"); +} +static bool topurgedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "purged", "goto_purged"); +} +static bool toscrubbedimfileMode(pxConfig *config) +{ + return change_imfile_data_state(config, "scrubbed", "goto_scrubbed"); +} +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + int numExportTables = 3; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(dummy, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chip_id", "=="); + + ExportTable tables [] = { + {"chipRun", "vptool_export_run.sql"}, + {"chipImfile", "vptool_export_imfile.sql"}, + {"chipProcessedImfile", "vptool_export_processed_imfile.sql"}, + }; + + + for (int i=0; i < numExportTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + bool success = true; + if (!strcmp(tables[i].tableName, "chipRun")) { + success = pxSetStateCleaned("chipRun", "state", output); + } else if (!strcmp(tables[i].tableName, "chipProcessedImfile")) { + success = pxSetStateCleaned("chipProcessedImfile", "data_state", output); + } + if (!success) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + unsigned int nFail = 0; // Number of failed lines + psMetadata *input = psMetadataConfigRead(NULL, &nFail, infile, false); + if (nFail) { + psError(PS_ERR_IO, false, "%d failed lines in input", nFail); + psFree(input); + return false; + } + + psVector *identifiers = psVectorAllocEmpty(16, PS_TYPE_U64); // Identifiers inserted + + psMetadataIterator *iter = psMetadataIteratorAlloc(input, PS_LIST_HEAD, NULL); // Iterator + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + // first item is the dbversion, skip it + psMetadataItem *dbversion = psMetadataGetAndIncrement(iter); + (void) dbversion; + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psMetadataItem *item; // Item from iteration + while ((item = psMetadataGetAndIncrement(iter))) { + PXMIRROR_PRIMARY(item, "chipRun", chipRunRow, chipRunObjectFromMetadata, identifiers, chip_id, + chipRunInsertObject, config->dbh, + { psFree(iter); psFree(identifiers); psFree(input); }); + + PXMIRROR_OTHER(item, "chipImfile", chipImfileRow, chipImfileObjectFromMetadata, identifiers, chip_id, + chipImfileInsertObject, config->dbh, + { psFree(iter); psFree(identifiers); psFree(input); }); + + PXMIRROR_OTHER(item, "chipProcessedImfile", chipProcessedImfileRow, + chipProcessedImfileObjectFromMetadata, identifiers, chip_id, + chipProcessedImfileInsertObject, config->dbh, + { psFree(iter); psFree(identifiers); psFree(input); }); + } + psFree(iter); + psFree(input); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psLogMsg("vptool", PS_LOG_INFO, "%ld chipRuns added", identifiers->n); + psFree(identifiers); + + return true; +} + +static bool runstateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + +// PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + PXOPT_LOOKUP_BOOL(no_magic, config->args, "-no_magic", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("vptool_runstate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipRunState", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +// a very specfic function to queue a cleaned chipProcessedImfile to be updated +static bool setimfiletoupdateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(chip_id, config->args, "-chip_id", true, false); + PXOPT_LOOKUP_STR(class_id, config->args, "-class_id", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + + psString query = pxDataGet("vptool_setimfiletoupdate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString setHook = psStringCopy(""); + if (label) { + psStringAppend(&setHook, "\n , chipRun.label = '%s'", label); + } + + if (class_id) { + psStringAppend(&query, " AND (chipProcessedImfile.class_id = '%s')", class_id); + } + + // we do not update components with the magic fault value. They are non-updateable + psStringAppend(&query, " AND (chipProcessedImfile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + + if (!p_psDBRunQueryF(config->dbh, query, setHook, chip_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(setHook); + psFree(query); + + return true; +} +#endif // NOTDEF +static bool listrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + + psMetadata *where = psMetadataAlloc(); + pxchipGetSearchArgs (config, where); // chipRun, chipProcessedImfile, rawExp + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "chipRun.reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "chipRun.state", "=="); + pxAddLabelSearchArgs (config, where, "-label", "chipRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "chipRun.data_group", "LIKE"); + pxAddLabelSearchArgs (config, where, "-dist_group", "chipRun.dist_group", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "chipRun.magicked", "=="); + + psString where2 = NULL; + pxmagicAddWhere(config, &where2, "chipRun"); + // add cuts on ra and decl if supplied + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + psString query = pxDataGet("vptool_listrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s %s", whereClause, where2 ? where2 : ""); + psFree(whereClause); + } else if (psMetadataLookupBool(NULL, config->args, "-all") || where2) { + psStringAppend(&query, " WHERE chipRun.chip_id IS NOT NULL %s", where2 ? where2 : ""); + } else { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters (or -all) are required"); + return false; + } + psFree(where); + + if (pstamp_order) { + // put runs in order of exposure id with newest chip Runs first + // The postage stamp parser depends on this behavior + psStringAppend(&query, "\nORDER by exp_id, chip_id DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("vptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "chipRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/vptool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/vptool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/vptool.h (revision 41104) @@ -0,0 +1,39 @@ +/* + * vptool.h + * + * Copyright (C) 2011 University of Hawaii IFA + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef VPTOOL_H +#define VPTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + VPTOOL_MODE_NONE = 0x0, + VPTOOL_MODE_DEFINEBYQUERY, + VPTOOL_MODE_PENDINGRUN, + VPTOOL_MODE_UPDATERUN, + VPTOOL_MODE_REVERTRUN, + VPTOOL_MODE_LISTRUN, + VPTOOL_MODE_PENDINGIMFILE, + VPTOOL_MODE_ADDPROCESSEDCELL, + VPTOOL_MODE_PROCESSEDCELL, +} vptoolMode; + +pxConfig *vptoolConfig(pxConfig *config, int argc, char **argv); + +#endif // VPTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/vptoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/vptoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/vptoolConfig.c (revision 41104) @@ -0,0 +1,297 @@ +/* + * vptoolConfig.c + * + * Copyright (C) 2006-2008 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "vptool.h" + +pxConfig *vptoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + config->modules = pmConfigRead(&argc, argv, NULL); + if (! config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + // -definebyquery + psMetadata *definebyqueryArgs = psMetadataAlloc(); + // allow selecting by all of the args used to queue chipRuns + pxchipSetSearchArgs (definebyqueryArgs); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by newExp label (LIKE comparison)", NULL); + + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-set_dest_id", 0, "define destination", 0); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-rerun", 0, "queue exposures even if a previous vpRun exists", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + // -pendingrun + psMetadata *pendingrunArgs = psMetadataAlloc(); + psMetadataAddS64(pendingrunArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(pendingrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + pxchipSetSearchArgs(pendingrunArgs); + psMetadataAddU64(pendingrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -updaterun + psMetadata *updaterunArgs = psMetadataAlloc(); +// pxchipSetSearchArgs (updaterunArgs); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-vp_id", 0, "search by vpRun ID", 0); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", 0, "search by label (LIKE comparison)", NULL); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by data_group (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "set label", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "set data group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "set note", NULL); + psMetadataAddF32(updaterunArgs, PS_LIST_TAIL, "-set_dtime_script", 0, "define elapsed time for script (seconds)", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_hostname", 0, "set hostname", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_outroot", 0, "set outroot", NULL); + psMetadataAddS16(updaterunArgs, PS_LIST_TAIL, "-set_fault", 0, "set fault code", 0); + + // -revertrun + psMetadata *revertrunArgs = psMetadataAlloc(); + psMetadataAddS64(revertrunArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(revertrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by vpRun label (LIKE comparison)", NULL); + psMetadataAddStr(revertrunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by vpRun data_group (LIKE comparison)", NULL); + // pxchipSetSearchArgs(revertrunArgs); + psMetadataAddS16(revertrunArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -pendingimfile + psMetadata *pendingimfileArgs = psMetadataAlloc(); + psMetadataAddS64(pendingimfileArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(pendingimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", 0); + pxchipSetSearchArgs(pendingimfileArgs); + psMetadataAddU64(pendingimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(pendingimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + // -addprocessedcell + psMetadata *addprocessedcellArgs = psMetadataAlloc(); + psMetadataAddS64(addprocessedcellArgs, PS_LIST_TAIL, "-vp_id", 0, "define chip ID (required)", 0); + psMetadataAddStr(addprocessedcellArgs, PS_LIST_TAIL, "-class_id", 0, "define class ID (required)", NULL); + psMetadataAddStr(addprocessedcellArgs, PS_LIST_TAIL, "-cell_id", 0, "define class ID (required)", NULL); + + psMetadataAddStr(addprocessedcellArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location (required)", NULL); + psMetadataAddF32(addprocessedcellArgs, PS_LIST_TAIL, "-dtime_photom", 0, "define elapsed time for photometry (seconds)", NAN); + psMetadataAddStr(addprocessedcellArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", NULL); + + psMetadataAddS16(addprocessedcellArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(addprocessedcellArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + + + // -processedcell + psMetadata *processedcellArgs = psMetadataAlloc(); + pxchipSetSearchArgs(processedcellArgs); + psMetadataAddS64(processedcellArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(processedcellArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddStr(processedcellArgs, PS_LIST_TAIL, "-cell_id", 0, "search by class ID", NULL); + psMetadataAddStr(processedcellArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by vpRun label (LIKE comparison)", NULL); + psMetadataAddStr(processedcellArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by vpRun data_group (LIKE comparison)", NULL); + + psMetadataAddU64(processedcellArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(processedcellArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + pxspaceAddArguments(processedcellArgs); + +#ifdef notdef + // -updateprocessedcell + psMetadata *updateprocessedcellArgs = psMetadataAlloc(); + psMetadataAddS64(updateprocessedcellArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(updateprocessedcellArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddS16(updateprocessedcellArgs, PS_LIST_TAIL, "-fault", 0, "set fault code (required)", 0); + psMetadataAddStr(updateprocessedcellArgs, PS_LIST_TAIL, "-set_state", 0, "set state", NULL); + psMetadataAddS16(updateprocessedcellArgs, PS_LIST_TAIL, "-set_quality", 0, "set quality", 0); +#endif // notdef + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + pxchipSetSearchArgs(listrunArgs); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-vp_id", 0, "search by vpRun ID", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by vpRun label (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-state", 0, "search by vpRun state", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by vpRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by vpRun dist_group (LIKE comparison)", NULL); + // pxmagicAddArguments(listrunArgs); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + psMetadataAddU64(listrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-all", 0, "list everything without search terms", false); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + pxspaceAddArguments(listrunArgs); + +#ifdef notdef + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to mask out (required)", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddStr(maskedArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(maskedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -unmasked + psMetadata *unmaskedArgs = psMetadataAlloc(); + psMetadataAddStr(unmaskedArgs, PS_LIST_TAIL, "-label", 0, "restrict to specified label", NULL); + psMetadataAddBool(unmaskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(unmaskedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to unmask (required)", NULL); + + // -pendingcleanuprun + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupimfile + psMetadata *pendingcleanupimfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupimfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupimfileArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(pendingcleanupimfileArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", NULL); + psMetadataAddBool(pendingcleanupimfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupimfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -revertcleanup + psMetadata *revertcleanupArgs = psMetadataAlloc(); + psMetadataAddS64(revertcleanupArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by vpRun label (LIKE comparison)", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by vpRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-state", 0, "search by current state", NULL); + psMetadataAddS16(revertcleanupArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + + // -run + psMetadata *runArgs = psMetadataAlloc(); + psMetadataAddStr(runArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddBool(runArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(runArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddStr(runArgs, PS_LIST_TAIL, "-state", 0, "search by state (required)", NULL); + pxchipSetSearchArgs(runArgs); + psMetadataAddS64(runArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(runArgs, PS_LIST_TAIL, "-reduction", 0, "search by reduction class", NULL); + + // -advanceexp + psMetadata *advanceexpArgs = psMetadataAlloc(); + psMetadataAddS64(advanceexpArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddStr(advanceexpArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "advance exposures for specified label", NULL); + psMetadataAddU64(advanceexpArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tocleanedimfile + psMetadata *tocleanedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedimfileArgs, PS_LIST_TAIL, "-vp_id", 0, "chip ID to update", 0); + psMetadataAddStr(tocleanedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -tofullimfile + psMetadata *tofullimfileArgs = psMetadataAlloc(); + psMetadataAddS64(tofullimfileArgs, PS_LIST_TAIL, "-vp_id", 0, "chip ID to update", 0); + psMetadataAddStr(tofullimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -topurgedimfile + psMetadata *topurgedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(topurgedimfileArgs, PS_LIST_TAIL, "-vp_id", 0, "chip ID to update", 0); + psMetadataAddStr(topurgedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -toscrubbedimfile + psMetadata *toscrubbedimfileArgs = psMetadataAlloc(); + psMetadataAddS64(toscrubbedimfileArgs, PS_LIST_TAIL, "-vp_id", 0, "chip ID to update", 0); + psMetadataAddStr(toscrubbedimfileArgs, PS_LIST_TAIL, "-class_id", 0, "class ID to update", NULL); + + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + // pxchipSetSearchArgs (exportrunArgs); XXX include search terms? + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-vp_id", 0, "export this chip ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "mark tables as cleaned", false); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + // -runstate + psMetadata *runstateArgs = psMetadataAlloc(); + psMetadataAddS64(runstateArgs, PS_LIST_TAIL, "-vp_id", 0, "search by chip ID", 0); + psMetadataAddS64(runstateArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure tag", 0); + psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure tag", 0); + psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddBool(runstateArgs, PS_LIST_TAIL, "-no_magic", 0, "magic is not necessary for result", false); + + psMetadataAddU64(runstateArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(runstateArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); +#endif //notdef + + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "create runs from raw exposures", VPTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-pendingrun", "list runs ready to process", VPTOOL_MODE_PENDINGRUN, pendingrunArgs); + PXOPT_ADD_MODE("-updaterun", "change vpRun properties", VPTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-revertrun","clear a faulted vpRun", VPTOOL_MODE_REVERTRUN, revertrunArgs); + PXOPT_ADD_MODE("-pendingimfile", "list pending imfiles", VPTOOL_MODE_PENDINGIMFILE, pendingimfileArgs); + + PXOPT_ADD_MODE("-addprocessedcell", "add a processed cell", VPTOOL_MODE_ADDPROCESSEDCELL, addprocessedcellArgs); + PXOPT_ADD_MODE("-processedcell", "show processed cells", VPTOOL_MODE_PROCESSEDCELL, processedcellArgs); + PXOPT_ADD_MODE("-listrun", "list vpRuns", VPTOOL_MODE_LISTRUN, listrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, false, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/warptool.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/warptool.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/warptool.c (revision 41104) @@ -0,0 +1,2621 @@ +/* + * warptool.c + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVB_CONFIG_H +#include +#endif + +#include +#include +#include +#include +#include + +#include "pxtools.h" +#include "warptool.h" + +static psS64 definerunMode(pxConfig *config); +static bool definebyqueryMode(pxConfig *config); +static bool updaterunMode(pxConfig *config); +static bool expMode(pxConfig *config); +static bool imfileMode(pxConfig *config); +static bool tooverlapMode(pxConfig *config); +static bool addoverlapMode(pxConfig *config); +static bool revertoverlapMode(pxConfig *config); +static bool scmapMode(pxConfig *config); +static bool towarpedMode(pxConfig *config); +static bool addwarpedMode(pxConfig *config); +static bool advancerunMode(pxConfig *config); +static bool warpedMode(pxConfig *config); +static bool revertwarpedMode(pxConfig *config); +static bool blockMode(pxConfig *config); +static bool maskedMode(pxConfig *config); +static bool unblockMode(pxConfig *config); +static bool tosummaryMode(pxConfig *config); +static bool addsummaryMode(pxConfig *config); +static bool pendingcleanuprunMode(pxConfig *config); +static bool pendingcleanupwarpMode(pxConfig *config); +static bool revertcleanupMode(pxConfig *config); +static bool donecleanupMode(pxConfig *config); +static bool tocleanedskyfileMode(pxConfig *config); +static bool topurgedskyfileMode(pxConfig *config); +static bool toscrubbedskyfileMode(pxConfig *config); +static bool tofullskyfileMode(pxConfig *config); +static bool updateskyfileMode(pxConfig *config); +static bool exportrunMode(pxConfig *config); +static bool importrunMode(pxConfig *config); +static bool runstateMode(pxConfig *config); +static bool listrunMode(pxConfig *config); +static bool setskyfiletoupdateMode(pxConfig *config); + +static bool parseAndInsertSkyCellMap(pxConfig *config, const char *mapfile); +static bool isValidMode(pxConfig *config, const char *mode); +bool warpCompletedRuns(pxConfig *config); + +# define MODECASE(caseName, func) \ + case caseName: \ + if (!func(config)) { \ + goto FAIL; \ + } \ + break; + +int main(int argc, char **argv) +{ + psLibInit(NULL); + + pxConfig *config = warptoolConfig(NULL, argc, argv); + if (!config) { + psError(PXTOOLS_ERR_CONFIG, false, "failed to configure"); + goto FAIL; + } + + switch (config->mode) { + MODECASE(WARPTOOL_MODE_DEFINERUN, definerunMode); + MODECASE(WARPTOOL_MODE_DEFINEBYQUERY, definebyqueryMode); + MODECASE(WARPTOOL_MODE_UPDATERUN, updaterunMode); + MODECASE(WARPTOOL_MODE_EXP, expMode); + MODECASE(WARPTOOL_MODE_IMFILE, imfileMode); + MODECASE(WARPTOOL_MODE_TOOVERLAP, tooverlapMode); + MODECASE(WARPTOOL_MODE_ADDOVERLAP, addoverlapMode); + MODECASE(WARPTOOL_MODE_REVERTOVERLAP, revertoverlapMode); + MODECASE(WARPTOOL_MODE_SCMAP, scmapMode); + MODECASE(WARPTOOL_MODE_TOWARPED, towarpedMode); + MODECASE(WARPTOOL_MODE_ADDWARPED, addwarpedMode); + MODECASE(WARPTOOL_MODE_ADVANCERUN, advancerunMode); + MODECASE(WARPTOOL_MODE_WARPED, warpedMode); + MODECASE(WARPTOOL_MODE_REVERTWARPED, revertwarpedMode); + MODECASE(WARPTOOL_MODE_BLOCK, blockMode); + MODECASE(WARPTOOL_MODE_MASKED, maskedMode); + MODECASE(WARPTOOL_MODE_UNBLOCK, unblockMode); + MODECASE(WARPTOOL_MODE_TOSUMMARY, tosummaryMode); + MODECASE(WARPTOOL_MODE_ADDSUMMARY, addsummaryMode); + MODECASE(WARPTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunMode); + MODECASE(WARPTOOL_MODE_PENDINGCLEANUPSKYFILE, pendingcleanupwarpMode); + MODECASE(WARPTOOL_MODE_REVERTCLEANUP, revertcleanupMode); + MODECASE(WARPTOOL_MODE_DONECLEANUP, donecleanupMode); + MODECASE(WARPTOOL_MODE_TOCLEANEDSKYFILE, tocleanedskyfileMode); + MODECASE(WARPTOOL_MODE_TOPURGEDSKYFILE, topurgedskyfileMode); + MODECASE(WARPTOOL_MODE_TOSCRUBBEDSKYFILE, toscrubbedskyfileMode); + MODECASE(WARPTOOL_MODE_TOFULLSKYFILE, tofullskyfileMode); + MODECASE(WARPTOOL_MODE_UPDATESKYFILE, updateskyfileMode); + MODECASE(WARPTOOL_MODE_EXPORTRUN, exportrunMode); + MODECASE(WARPTOOL_MODE_IMPORTRUN, importrunMode); + MODECASE(WARPTOOL_MODE_RUNSTATE, runstateMode); + MODECASE(WARPTOOL_MODE_LISTRUN, listrunMode); + MODECASE(WARPTOOL_MODE_SETSKYFILETOUPDATE, setskyfiletoupdateMode); + + default: + psAbort("invalid option (this should not happen)"); + } + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(EXIT_SUCCESS); + +FAIL: + psErrorStackPrint(stderr, "\n"); + int exit_status = pxerrorGetExitStatus(); + + psFree(config); + pmConfigDone(); + psLibFinalize(); + + exit(exit_status); +} + + +static psS64 definerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(fake_id, config->args, "-fake_id", true, false); // required + PXOPT_LOOKUP_STR(mode, config->args, "-mode", true, false); // required + PXOPT_LOOKUP_STR(workdir, config->args, "-workdir", true, false); // required + PXOPT_LOOKUP_STR(label, config->args, "-label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-dist_group", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-note", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); // required (no default TESS) + PXOPT_LOOKUP_STR(reduction, config->args, "-reduction", false, false); // required (no default TESS) + PXOPT_LOOKUP_STR(end_stage, config->args, "-end_stage", false, false); + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // check mode + if (mode && !isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalid mode"); + return false; + } + + warpRunRow *warpRun = warpRunRowAlloc( + 0, // ID + fake_id, + mode, + "new", // state + workdir, + "dirty", // workdir_state + label, + data_group ? data_group : label, + dist_group, + dvodb, + tess_id, + reduction, + end_stage, + registered, + 0, // magicked + NULL, // software version + 0, // mask stat npix + NAN, // static + NAN, // dynamic + NAN, // magic + NAN, // advisory + note + ); + if (!warpRun) { + psError(PS_ERR_UNKNOWN, false, "failed to alloc warpRun object"); + return true; + } + if (!warpRunInsertObject(config->dbh, warpRun)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(warpRun); + return true; + } + + // get the assigned warp_id + psS64 warp_id = psDBLastInsertID(config->dbh); + warpRun->warp_id = warp_id; + + if (!warpRunPrintObject(stdout, warpRun, !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print object"); + psFree(warpRun); + return false; + } + + psFree(warpRun); + + return warp_id; +} + + +static bool definebyqueryMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fakeRun.fake_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_STR(config->args, where, "-inst", "rawExp.camera", "=="); + PXOPT_COPY_STR(config->args, where, "-telescope", "rawExp.telescope", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-exp_tag", "rawExp.exp_tag", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_type", "rawExp.exp_type", "=="); + PXOPT_COPY_STR(config->args, where, "-filelevel", "rawExp.filelevel", "=="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "=="); + PXOPT_COPY_F64(config->args, where, "-airmass_min", "rawExp.airmass", ">="); + PXOPT_COPY_F64(config->args, where, "-airmass_max", "rawExp.airmass", "<"); + PXOPT_COPY_RADEC(config->args, where, "-ra_min", "rawExp.ra", ">="); + PXOPT_COPY_RADEC(config->args, where, "-ra_max", "rawExp.ra", "<"); + PXOPT_COPY_RADEC(config->args, where, "-decl_min", "rawExp.decl", ">="); + PXOPT_COPY_RADEC(config->args, where, "-decl_max", "rawExp.decl", "<"); + PXOPT_COPY_F32(config->args, where, "-exp_time_min", "rawExp.exp_time", ">="); + PXOPT_COPY_F32(config->args, where, "-exp_time_max", "rawExp.exp_time", "<"); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_min", "rawExp.sat_pixel_frac", ">="); + PXOPT_COPY_F32(config->args, where, "-sat_pixel_frac_max", "rawExp.sat_pixel_frac", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_min", "rawExp.bg", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_max", "rawExp.bg", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_min", "rawExp.bg_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_stdev_max", "rawExp.bg_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_min", "rawExp.bg_mean_stdev", ">="); + PXOPT_COPY_F64(config->args, where, "-bg_mean_stdev_max", "rawExp.bg_mean_stdev", "<"); + PXOPT_COPY_F64(config->args, where, "-alt_min", "rawExp.alt", ">="); + PXOPT_COPY_F64(config->args, where, "-alt_max", "rawExp.alt", "<"); + PXOPT_COPY_F64(config->args, where, "-az_min", "rawExp.az", ">="); + PXOPT_COPY_F64(config->args, where, "-az_max", "rawExp.az", "<"); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_min", "rawExp.ccd_temp", ">="); + PXOPT_COPY_F32(config->args, where, "-ccd_temp_max", "rawExp.ccd_temp", "<"); + PXOPT_COPY_F64(config->args, where, "-posang_min", "rawExp.posang", ">="); + PXOPT_COPY_F64(config->args, where, "-posang_max", "rawExp.posang", "<"); + PXOPT_COPY_STR(config->args, where, "-object", "rawExp.object", "=="); + PXOPT_COPY_STR(config->args, where, "-comment", "rawExp.comment", "LIKE"); + PXOPT_COPY_STR(config->args, where, "-obs_mode", "rawExp.obs_mode", "LIKE"); + PXOPT_COPY_F32(config->args, where, "-sun_angle_min", "rawExp.sun_angle", ">="); + PXOPT_COPY_F32(config->args, where, "-sun_angle_max", "rawExp.sun_angle", "<"); + PXOPT_COPY_STR(config->args, where, "-reduction", "fakeRun.reduction", "=="); + pxAddLabelSearchArgs (config, where, "-label", "fakeRun.label", "=="); // define using fake label + + if (!psListLength(where->list) && + !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + PXOPT_LOOKUP_STR(mode, config->args, "-set_mode", true, false); // required + PXOPT_LOOKUP_STR(workdir, config->args, "-set_workdir", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + PXOPT_LOOKUP_STR(data_group, config->args, "-set_data_group", false, false); + PXOPT_LOOKUP_STR(dist_group, config->args, "-set_dist_group", false, false); + PXOPT_LOOKUP_STR(dvodb, config->args, "-set_dvodb", false, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-set_tess_id", false, false); + PXOPT_LOOKUP_STR(reduction, config->args, "-set_reduction", false, false); + PXOPT_LOOKUP_STR(end_stage, config->args, "-set_end_stage", false, false); + PXOPT_LOOKUP_STR(note, config->args, "-set_note", false, false); + + PXOPT_LOOKUP_TIME(registered, config->args, "-registered", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pretend, config->args, "-pretend", false); + + // check mode + // XXX mode is required but is not used?! + if (mode && !isValidMode(config, mode)) { + psError(PS_ERR_UNKNOWN, false, "invalid mode"); + return false; + } + + // find the exp_id of all the exposures that we want to queue up. + psString query = pxDataGet("warptool_definebyquery.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + psFree(where); + return false; + } + + // use psDBGenerateWhereSQL because the SQL yields an intermediate table + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, "AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (pretend) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "rawExp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + return true; + } + + // would could do this "all in the database" if we didn't want the option + // of changing the label/reduction/expgroup/dvodb/etc. So we're pulling the + // data out so we have the option of changing these values or leaving the + // old values in place (i.e., passing the values through). + + // loop over our list of fakeRun rows to check the supplied and selected tess_id values: + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + fakeRunRow *row = fakeRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into fakeRun"); + psFree(output); + return false; + } + + if (!tess_id && !row->tess_id) { + psError(PS_ERR_UNKNOWN, false, "cannot queue warp run without a defined tess id: label: %s, fake_id %" PRId64, row->label, row->fake_id); + psFree(output); + return false; + } + + psFree(row); + } + + // loop over our list of fakeRun rows + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *md = output->data[i]; + + fakeRunRow *row = fakeRunObjectFromMetadata(md); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert metadata into fakeRun"); + psFree(output); + return false; + } + + // queue the exp + if (!pxwarpQueueByFakeID(config, + row->fake_id, + workdir ? workdir : row->workdir, + label ? label : row->label, + data_group ? data_group: row->data_group, + dist_group ? dist_group: row->dist_group, + dvodb ? dvodb : row->dvodb, + tess_id ? tess_id : row->tess_id, + reduction ? reduction : row->reduction, + end_stage ? end_stage : row->end_stage, + note)) + { + psError(PS_ERR_UNKNOWN, false, "failed to trying to queue fake_id: %" PRId64, row->fake_id); + psFree(row); + psFree(output); + return false; + } + psFree(row); + } + psFree(output); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool updaterunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "warpRun.reduction", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "warpRun.state", "=="); + PXOPT_COPY_STR(config->args, where, "-data_group","warpRun.data_group","LIKE"); + PXOPT_COPY_STR(config->args, where, "-dist_group","warpRun.dist_group","LIKE"); + pxAddLabelSearchArgs(config, where, "-label", "warpRun.label", "LIKE"); + + PXOPT_COPY_TIME(config->args, where, "-registered_begin", "warpRun.registered", ">="); + PXOPT_COPY_TIME(config->args, where, "-registered_end", "warpRun.registered", "<"); + + PXOPT_LOOKUP_BOOL(destreaked, config->args, "-destreaked", false); + if (destreaked) { + psMetadataAddS64(where, PS_LIST_TAIL, "warpRun.magicked", PS_META_DUPLICATE_OK, ">", 0); + } + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = psStringCopy("UPDATE warpRun"); + + // pxUpdateRun gets parameters from config->args and updates + bool result = pxUpdateRun(config, where, &query, "warpRun", "warp_id", "warpSkyfile", true, true); + + psFree(query); + psFree(where); + + return result; +} + +static bool expMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_exp.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "warpRun"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool imfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fake_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_imfile.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "warpRun"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpInputImfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool tooverlapMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_tooverlap.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "warpRun"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool addoverlapMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(mapfile, config->args, "-mapfile", false, false); + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", false, false); + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + if (fault == 0) { + if (!parseAndInsertSkyCellMap(config, mapfile)) { + psError(PS_ERR_UNKNOWN, false, "failed to inject mapfile: %s into the database", mapfile); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + } else { + warpSkyCellMapInsert(config->dbh, + warp_id, + "faulted", // skycell_id + "faulted", // tess_id + "faulted", // class_id + fault // fault + ); + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool revertoverlapMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpSkyCellMap.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyCellMap.skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpSkyCellMap.tess_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "warpSkyCellMap.fault", "=="); + + if (!psListLength(where->list) + && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + int numDeleted; // Number deleted + { + psString query = pxDataGet("warptool_revertoverlap.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return false; + } + psFree(query); + + numDeleted = psDBAffectedRows(config->dbh); + } + + psLogMsg("warptool", PS_LOG_INFO, "Deleted %d warpSkycellMap", numDeleted); + + psFree(where); + + return true; +} + + + +static bool parseAndInsertSkyCellMap(pxConfig *config, const char *mapfile) +{ + unsigned int nFail = 0; + psMetadata *imfiles = psMetadataAlloc(); + psMetadata *skycells = psMetadataConfigRead(NULL, &nFail, mapfile, false); + if (!skycells) { + psError(PS_ERR_UNKNOWN, false, "failed to parse mapfile: %s", mapfile); + return false; + } + if (nFail) { + psError(PS_ERR_UNKNOWN, false, "there were %d errors parsing mapfile: %s", nFail, mapfile); + psFree(skycells); + psFree(imfiles); + return false; + } + + psMetadataItem *item = NULL; + psMetadataIterator *iter = psMetadataIteratorAlloc(skycells, 0, NULL); + while ((item = psMetadataGetAndIncrement(iter))) { + if (item->type != PS_DATA_METADATA) { + psError(PS_ERR_UNKNOWN, false, "mapfile: %s is in the wrong format", mapfile); + psFree(iter); + psFree(skycells); + psFree(imfiles); + return false; + } + + psMetadata *sc = item->data.md; + // this conversion isn't strictly nessicary but it's an easy way of + // validating the format + warpSkyCellMapRow *row = warpSkyCellMapObjectFromMetadata(sc); + if (!row) { + psError(PS_ERR_UNKNOWN, false, "failed to convert mapfile: %s metdata entry into a warpSkyCellMap object", mapfile); + psFree(iter); + psFree(skycells); + psFree(imfiles); + return false; + } + psMetadataAddS64(imfiles, PS_LIST_TAIL, row->skycell_id, PS_META_REPLACE, "", row->warp_id); + + if (!warpSkyCellMapInsertObject(config->dbh, row)) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + psFree(row); + psFree(iter); + psFree(skycells); + psFree(imfiles); + return false; + } + + psFree(row); + } + psFree(iter); + + // create warp_skyfile_ids for the output skyfiles + psString query = "INSERT INTO warpImfile VALUES(%" PRId64 ", '%s', 0)"; + iter = psMetadataIteratorAlloc(imfiles, 0, NULL); + while ((item = psMetadataGetAndIncrement(iter))) { + psString skycell_id = item->name; + psS64 warp_id = item->data.S64; + + if (!p_psDBRunQueryF(config->dbh, query, warp_id, skycell_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + psFree(iter); + psFree(skycells); + psFree(imfiles); + + return true; +} + + +static bool scmapMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-class_id", "class_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_scmap.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, "warpSkyCellMap"); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpSkyCellMap", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool towarpedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpSkyCellMap.warp_id", "=="); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // First find Label's with warpRuns in state new in priority order + psMetadata *labelWhere = psMetadataAlloc(); + pxAddLabelSearchArgs (config, labelWhere, "-label", "warpRun.label", "=="); + + psString labelQuery = pxDataGet("warptool_towarped_labels.sql"); + if (!labelQuery) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString labelWhereStr = psStringCopy(""); + if (psListLength(labelWhere->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(labelWhere, NULL); + psStringAppend(&labelWhereStr, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(labelWhere); + + if (!p_psDBRunQueryF(config->dbh, labelQuery, labelWhereStr)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(labelQuery); + return false; + } + psFree(labelQuery); + psFree(labelWhereStr); + + psArray *labelOutput = p_psDBFetchResult(config->dbh); + if (psArrayLength(labelOutput) == 0) { + psTrace("warptool", PS_LOG_INFO, "no labels with pending warps found."); + return true; + } + + psString originalQuery = pxDataGet("warptool_towarped.sql"); + if (!originalQuery) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString whereStr = NULL; + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&whereStr, "\n AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + psArray *allOutput = psArrayAllocEmpty(0); + // Now loop over the labels and query each individually + for (long i = 0; i < psArrayLength(labelOutput); i++) { + psMetadata *labelRow = labelOutput->data[i]; + + bool status; + psString label = psMetadataLookupStr(&status, labelRow, "label"); + + psString query = psStringCopy(originalQuery); + psStringAppend(&query,"\nORDER BY warp_id"); + // treat limit == 0 as "no limit" + psString limitString = psStringCopy(""); + if (limit) { + // We apply the limit to both sides of the UNION to avoid slow queries + // and to the query itself to satisfy the user's requested limit + psStringAppend(&limitString, "\n%s", psDBGenerateLimitSQL(limit)); + psStringAppend(&query, "%s", limitString); + } + + psString thisWhere = NULL; + if (whereStr) { + psStringAppend(&thisWhere, "\n%s", whereStr); + } + psStringAppend(&thisWhere, "\nAND warpRun.label = '%s'", label); + + if (!p_psDBRunQueryF(config->dbh, query, thisWhere, limitString, thisWhere, limitString)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(limitString); + psFree(thisWhere); + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + long outputLength = psArrayLength(output); + if (outputLength) { + for (int i = 0; i< outputLength; i++) { + psPtr ptr = output->data[i]; + allOutput = psArrayAdd(allOutput, outputLength - i + 10, ptr); + if (!allOutput) { + psError(PS_ERR_UNKNOWN, false, "failed to add element to array"); + return false; + } + } + } else { + psTrace("warptool", PS_LOG_INFO, "no rows found for %s", label); + psFree(output); + continue; + } + psFree(output); + + if (limit) { + limit -= outputLength; + if (limit <= 0) { + // All done + break; + } + } + } + long allLabelsLength = psArrayLength(allOutput); + if (allLabelsLength) { + if (!ippdbPrintMetadatas(stdout, allOutput, "warpPendingSkyCell", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(allOutput); + return false; + } + } + psFree(allOutput); + psFree(labelOutput); + psFree(whereStr); + psFree(originalQuery); + + return true; +} + + +static bool addwarpedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + PXOPT_LOOKUP_STR(tess_id, config->args, "-tess_id", true, false); + + // optional + PXOPT_LOOKUP_STR(uri, config->args, "-uri", false, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", false, false); + PXOPT_LOOKUP_F64(bg, config->args, "-bg", false, false); + PXOPT_LOOKUP_F64(bg_stdev, config->args, "-bg_stdev", false, false); + PXOPT_LOOKUP_F32(dtime_warp, config->args, "-dtime_warp", false, false); + PXOPT_LOOKUP_F32(dtime_script, config->args, "-dtime_script", false, false); + PXOPT_LOOKUP_S32(xmin, config->args, "-xmin", false, false); + PXOPT_LOOKUP_S32(xmax, config->args, "-xmax", false, false); + PXOPT_LOOKUP_S32(ymin, config->args, "-ymin", false, false); + PXOPT_LOOKUP_S32(ymax, config->args, "-ymax", false, false); + PXOPT_LOOKUP_STR(hostname, config->args, "-hostname", false, false); + PXOPT_LOOKUP_F32(good_frac, config->args, "-good_frac", false, false); + PXOPT_LOOKUP_S64(magicked, config->args, "-set_magicked", false, false); + + PXOPT_LOOKUP_STR(ver_pslib, config->args, "-ver_pslib", false, false); + PXOPT_LOOKUP_STR(ver_psmodules, config->args, "-ver_psmodules", false, false); + PXOPT_LOOKUP_STR(ver_psphot, config->args, "-ver_psphot", false, false); + PXOPT_LOOKUP_STR(ver_ppstats, config->args, "-ver_ppstats", false, false); + PXOPT_LOOKUP_STR(ver_pswarp, config->args, "-ver_pswarp", false, false); + PXOPT_LOOKUP_STR(ver_streaks, config->args, "-ver_streaks", false, false); + + PXOPT_LOOKUP_S32(maskfrac_npix, config->args, "-maskfrac_npix", false, false); + PXOPT_LOOKUP_F32(maskfrac_static, config->args, "-maskfrac_static", false, false); + PXOPT_LOOKUP_F32(maskfrac_dynamic, config->args, "-maskfrac_dynamic", false, false); + PXOPT_LOOKUP_F32(maskfrac_magic, config->args, "-maskfrac_magic", false, false); + PXOPT_LOOKUP_F32(maskfrac_advisory, config->args, "-maskfrac_advisory", false, false); + + PXOPT_LOOKUP_S16(background_model, config->args, "-background_model", false, false); + + psTrace("czw.test",1,"Received versions: pslib %s psmodules %s psphot %s ppstats %s pswarp %s streaks %s\n", + ver_pslib,ver_psmodules,ver_psphot,ver_ppstats,ver_pswarp,ver_streaks); + psString ver_code = NULL; + if ((ver_pslib)&&(ver_psmodules)) { + ver_code = pxMergeCodeVersions(ver_pslib,ver_psmodules); + } + if (ver_psphot) { + ver_code = pxMergeCodeVersions(ver_code,ver_psphot); + } + if (ver_ppstats) { + ver_code = pxMergeCodeVersions(ver_code,ver_ppstats); + } + if (ver_pswarp) { + ver_code = pxMergeCodeVersions(ver_code,ver_pswarp); + } + if (ver_streaks) { + ver_code = pxMergeCodeVersions(ver_code,ver_streaks); + } + + // default values + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-quality", false, false); + + + + // we don't want to insert the last skyfile in a run but then not mark the + // run as 'stop' + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // XXX need to validate that this coresponds to an warpInputImfile + if (!warpSkyfileInsert(config->dbh, + warp_id, + skycell_id, + tess_id, + uri, + path_base, + "full", // data_state + bg, + bg_stdev, + dtime_warp, + dtime_script, + hostname, + good_frac, + xmin, + xmax, + ymin, + ymax, + fault, + quality, + magicked, + ver_code, + maskfrac_npix, + maskfrac_static, + maskfrac_dynamic, + maskfrac_magic, + maskfrac_advisory, + background_model + )) { + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + // point of no return + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool advancerunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "label", "=="); + + // PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + + psString query = pxDataGet("warptool_finished_run_select.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + query = pxDataGet("warptool_finish_run.sql"); + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + bool status; + psS64 warp_id = psMetadataLookupS64(&status, row, "warp_id"); + + psString software_ver = NULL; + psS64 maskfrac_npix = 0; + psF32 maskfrac_static = 0; + psF32 maskfrac_dynamic = 0; + psF32 maskfrac_magic = 0; + psF32 maskfrac_advisory = 0; + + // Calculate run level masking and software state + if (!pxCoalesceRunStatus(config,"warptool_coalesce_run.sql",warp_id, + &software_ver,&maskfrac_npix, + &maskfrac_static,&maskfrac_dynamic, + &maskfrac_magic,&maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to generate run level statistics"); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + // Set warpRun.software_ver to the appropriate value if ( + if (software_ver) { + if (!pxSetRunSoftware(config, "warpRun", "warp_id", warp_id, software_ver)) { + psError(PS_ERR_UNKNOWN, false, "failed to set warpRun.software_ver for warp_id: %" PRId64, + warp_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + // Set warpRun.maskfrac* to the appropriate values. + if (maskfrac_npix) { + if (!pxSetRunMaskfrac(config, "warpRun", "warp_id", warp_id, maskfrac_npix, maskfrac_static, + maskfrac_dynamic, maskfrac_magic, maskfrac_advisory)) { + psError(PS_ERR_UNKNOWN, false, "failed to set warpRun.software_ver for warp_id: %" PRId64, + warp_id); + psFree(output); + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + return(false); + } + } + + + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to look up value for warp_id"); + psFree(output); + psFree(query); + return false; + } + psS64 magicked = psMetadataLookupS64(&status, row, "magicked"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to look up value for magicked"); + psFree(output); + psFree(query); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, magicked, warp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(query); + return false; + } + + psS64 numUpdated = psDBAffectedRows(config->dbh); + + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + psFree(query); + psFree(output); + return false; + } + } + psFree(output); + psFree(query); + + return true; +} + +bool warpCompletedRuns(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psString query = pxDataGet("warptool_finished_run_select.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retrieve SQL statement"); + return false; + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + query = pxDataGet("warptool_finish_run.sql"); + for (long i = 0; i < psArrayLength(output); i++) { + psMetadata *row = output->data[i]; + + bool status; + psS64 warp_id = psMetadataLookupS64(&status, row, "warp_id"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to look up value for warp_id"); + psFree(output); + psFree(query); + return false; + } + psS32 magicked = psMetadataLookupS64(&status, row, "magicked"); + if (!status) { + psError(PS_ERR_UNKNOWN, false, "failed to look up value for magicked"); + psFree(output); + psFree(query); + return false; + } + if (!p_psDBRunQueryF(config->dbh, query, magicked, warp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(output); + psFree(query); + return false; + } + + psS64 numUpdated = psDBAffectedRows(config->dbh); + + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + psFree(query); + psFree(output); + return false; + } + } + psFree(output); + psFree(query); + + return true; +} + +static bool warpedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpSkyfile.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + PXOPT_COPY_S64(config->args, where, "-warp_skyfile_id", "warpImfile.warp_skyfile_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpSkyfile.tess_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fakeRun.fake_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "warpSkyfile.magicked", "=="); + PXOPT_COPY_S16(config->args, where, "-background_model", "warpSkyfile.background_model", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "warpRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_warped.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // generate where strings for arguments that require extra processing + // beyond PXOPT_COPY* + psString where2 = NULL; + if (!pxmagicAddWhere(config, &where2, "warpSkyfile")) { + psError(psErrorCodeLast(), false, "pxMagicAddWhere failed"); + return false; + } + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all && !where2) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + if (where2) { + if (psListLength(where->list)) { + psStringAppend(&query, " %s", where2); + } else { + psStringAppend(&query, " WHERE 1 %s", where2); + } + } + psFree(where); + + if (pstamp_order) { + // put runs in order of exposure id with newest warp Runs first + // The postage stamp parser depends on this behavior + psStringAppend(&query, "\nORDER by exp_id, warp_id DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpSkyfile", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + + +static bool revertwarpedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpSkyfile.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "warpSkyfile.skycell_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpSkyfile.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-reduction", "rawExp.reduction", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "=="); + PXOPT_COPY_S16(config->args, where, "-fault", "warpSkyfile.fault", "=="); + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", false, false); + + if (!psListLength(where->list) + && !psMetadataLookupBool(NULL, config->args, "-all")) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + + psString query = pxDataGet("warptool_revertwarped_delete.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + psString query_updated = pxDataGet("warptool_revertwarped_updated.sql"); + if (!query_updated) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psStringAppend(&query_updated, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!fault) { + // If fault has not been supplied, don't revert update faults with the magic value + // We don't do this for new runs because then they would never complete + // quality should be used to drop bad components + psStringAppend(&query_updated, " AND warpSkyfile.fault != %d", PXTOOL_DO_NOT_REVERT_FAULT); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numDeleted = psDBAffectedRows(config->dbh); + + psLogMsg("warptool", PS_LOG_INFO, "Deleted %d warpSkyfiles", numDeleted); + + // fix any faulted warpSkyfiles in data_state 'update' + + if (!p_psDBRunQuery(config->dbh, query_updated)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query_updated); + return false; + } + psFree(query_updated); + + int numUpdated = psDBAffectedRows(config->dbh); + + psLogMsg("warptool", PS_LOG_INFO, "Updated %d warpSkyfiles", numUpdated); + + return true; +} + + +static bool blockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + if (!warpMaskInsert(config->dbh, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + + +static bool maskedMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = psStringCopy("SELECT * FROM warpMask"); + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warpool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpMask", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool unblockMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + PXOPT_LOOKUP_STR(label, config->args, "-label", true, false); + + char *query = "DELETE FROM warpMask WHERE label = '%s'"; + + if (!p_psDBRunQueryF(config->dbh, query, label)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} + +static bool tosummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpSkyfile.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpSkyfile.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "warpRun.state", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fakeRun.fake_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "warpSkyfile.magicked", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "warpRun.data_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_tosummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // generate where strings for arguments that require extra processing + // beyond PXOPT_COPY* + psString where2 = NULL; + if (!pxmagicAddWhere(config, &where2, "warpSkyfile")) { + psError(psErrorCodeLast(), false, "pxMagicAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } else if (!all && !where2) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + if (where2) { + psStringAppend(&query, " %s", where2); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + return(true); +} +static bool addsummaryMode(pxConfig *config) { + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_STR(projection_cell, config->args, "-projection_cell", true, false); + PXOPT_LOOKUP_STR(path_base, config->args, "-path_base", true, false); + + psString query = pxDataGet("warptool_addsummary.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return(false); + } + if (!p_psDBRunQueryF(config->dbh, query, warp_id, projection_cell, path_base)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return(false); + } + psS64 numUpdated = psDBAffectedRows(config->dbh); + + if (numUpdated != 1) { + psError(PS_ERR_UNKNOWN, false, "should have affected 1 row"); + psFree(query); + return false; + } + + psFree(query); + + // Print anything here? + + return(true); +} + +static bool pendingcleanuprunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "=="); + + psString query = pxDataGet("warptool_pendingcleanuprun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpPendingCleanupRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool pendingcleanupwarpMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "=="); + + char * sql_file = all ? "warptool_pendingcleanupskyfile_all.sql" : "warptool_pendingcleanupskyfile.sql"; + psString query = pxDataGet(sql_file); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpPendingCleanupWarp", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + + +static bool revertcleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpSkyfile.warp_id", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "warpRun.data_group", "LIKE"); + + PXOPT_LOOKUP_STR(state, config->args, "-state", false, false); + + if (!psListLength(where->list)) { + psFree(where); + psError(PXTOOLS_ERR_CONFIG, false, "search parameters are required"); + return false; + } + if (!state) { + state = "error_cleaned"; + } + char *newState = NULL; + if (!strcmp(state, "error_cleaned")) { + newState = "goto_cleaned"; + } else if (!strcmp(state, "error_purged")) { + newState = "goto_purged"; + } else if (!strcmp(state, "error_scrubbed")) { + newState = "goto_scrubbed"; + } else { + psError(PXTOOLS_ERR_CONFIG, true, "-state must be either error_cleaned, error_purged, or error_scrubbed"); + return false; + } + + psString query = pxDataGet("warptool_revertcleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + if (!p_psDBRunQueryF(config->dbh, query, newState, state, state)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + int numDeleted = psDBAffectedRows(config->dbh); + + psLogMsg("warptool", PS_LOG_INFO, "Reverted %d warpRuns and warpSkyfiles", numDeleted); + + return true; +} +static bool donecleanupMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_STR(config->args, where, "-label", "label", "=="); + + psString query = pxDataGet("warptool_donecleanup.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " AND %s", whereClause); + psFree(whereClause); + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpDoneCleanup", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + + psFree(output); + + return true; +} + +static bool isValidMode(pxConfig *config, const char *mode) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + PS_ASSERT_PTR_NON_NULL(mode, false); + + // check that state is a valid string value + if (!( + (strncmp(mode, "warp", 5) == 0) + || (strncmp(mode, "diff", 5) == 0) + || (strncmp(mode, "stack", 6) == 0) + || (strncmp(mode, "magic", 6) == 0) + ) + ) { + psError(PS_ERR_UNKNOWN, false, + "invalid warpRun mode: %s", mode); + return false; + } + + return true; +} + +// update warpSkyfile.data_state to given value. +// afterwards, if all skfyiles in the run have the new state, update the state for the run as well +// shared code for the modes -tocleanedskyfile -tofullskyfile -topurgedskyfile + +static bool change_skyfile_data_state(pxConfig *config, psString data_state) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // warp_id, skycell_id are required + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", true, false); + + psString query = pxDataGet("warptool_change_skyfile_data_state.sql"); + + if (!psDBTransaction(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psString set_magicked_skyfile = psStringCopy(""); + psString set_magicked_run = psStringCopy(""); + if (!strcmp(data_state, "full")) { + // magicked is only an argument for for -tofullskyfile + PXOPT_LOOKUP_S64(magicked, config->args, "-set_magicked", false, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + if (magicked) { + psStringAppend(&set_magicked_skyfile, "\n , warpSkyfile.magicked = %" PRId64, magicked); + psStringAppend(&set_magicked_run, "\n, warpRun.magicked = %" PRId64, magicked); + } + if (quality) { + psStringAppend(&set_magicked_skyfile, "\n , warpSkyfile.quality = %"PRId16, quality); + } + PXOPT_LOOKUP_S16(background_model, config->args, "-set_background_model", false, false); + if (background_model) { + psStringAppend(&set_magicked_skyfile, "\n , warpSkyfile.background_model = %"PRId16, background_model); + } + } else if (!strcmp(data_state, "cleaned") || !strcmp(data_state, "purged")) { + // if magicked is currently nonzero set it to -1 + // Set warpRun.magicked when the first skyfile is cleaned + psStringAppend(&set_magicked_skyfile, "\n, warpSkyfile.magicked = IF(warpSkyfile.magicked = 0, 0, -1), warpRun.magicked = IF(warpRun.magicked = 0, 0, -1)"); + } + + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_skyfile, warp_id, skycell_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(query); + psFree(set_magicked_skyfile); + + query = pxDataGet("warptool_change_run_state.sql"); + if (!p_psDBRunQueryF(config->dbh, query, data_state, set_magicked_run, warp_id, data_state)) { + // rollback + if (!psDBRollback(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + } + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + psFree(set_magicked_run); + + if (!psDBCommit(config->dbh)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + return true; +} +static bool tocleanedskyfileMode(pxConfig *config) +{ + return change_skyfile_data_state(config, "cleaned"); +} +static bool tofullskyfileMode(pxConfig *config) +{ + return change_skyfile_data_state(config, "full"); +} +static bool topurgedskyfileMode(pxConfig *config) +{ + return change_skyfile_data_state(config, "purged"); +} +static bool toscrubbedskyfileMode(pxConfig *config) +{ + return change_skyfile_data_state(config, "scrubbed"); +} + +static bool updateskyfileMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + // warp_id, skycell_id, fault are required + PXOPT_LOOKUP_STR(state, config->args, "-set_state", false, false); + PXOPT_LOOKUP_S16(background_model, config->args, "-set_background_model", false, false); + if (background_model) { + // CZW 2012-12-06: I'm unclear why we don't use this form for all updates? + psMetadata *where = psMetadataAlloc(); + psMetadata *values = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + PXOPT_COPY_S16(config->args, values, "-set_background_model", "background_model", "=="); + long rows = psDBUpdateRows(config->dbh,"warpSkyfile", where, values); + psFree(values); + psFree(where); + if (!rows) { + // This maybe should rollback and error if rows != 1 + psError(PS_ERR_UNKNOWN, true, "no rows changed"); + return false; + } + } + else if (!state) { + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-skycell_id", "skycell_id", "=="); + if (psListLength(where->list) == 0) { + // this won't happen because warptoolConfig requires these arguments + psError(PS_ERR_UNKNOWN, true, "search parameters are required"); + psFree(where); + return false; + } + + PXOPT_LOOKUP_S16(fault, config->args, "-fault", true, false); + PXOPT_LOOKUP_S16(quality, config->args, "-set_quality", false, false); + + if (!pxSetFaultCode(config->dbh, "warpSkyfile", where, fault, quality)) { + psError(PS_ERR_UNKNOWN, false, "failed to set set fault flag"); + psFree(where); + return false; + } + psFree(where); + } + else { + if (strcmp(state,"error_cleaned") == 0) { + change_skyfile_data_state(config,"error_cleaned"); + } + else if (strcmp(state, "error_scrubbed") == 0) { + change_skyfile_data_state(config,"error_scrubbed"); + } + else if (strcmp(state, "error_purged") == 0) { + change_skyfile_data_state(config,"error_purged"); + } + else { + psError(PS_ERR_UNKNOWN, false, "unhandled state given"); + return(false); + } + } + + return true; +} + +bool exportrunMode(pxConfig *config) +{ + typedef struct ExportTable { + char tableName[80]; + char sqlFilename[80]; + } ExportTable; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + // PXOPT_LOOKUP_S64(det_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_STR(outfile, config->args, "-outfile", true, false); + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(clean, config->args, "-clean", false); + + FILE *f = fopen (outfile, "w"); + if (f == NULL) { + psError(PS_ERR_UNKNOWN, false, "failed to open output file"); + return false; + } + + if (!pxExportVersion(config, f)) { + psError(PS_ERR_UNKNOWN, false, "failed to write dbversion output file"); + return false; + } + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warp_id", "=="); + + ExportTable tables [] = { + {"warpRun", "warptool_export_run.sql"}, + {"warpImfile", "warptool_export_imfile.sql"}, + {"warpSkyfile", "warptool_export_skyfile.sql"}, + {"warpSkyCellMap", "warptool_export_skycell_map.sql"}, + }; + + int numTables = sizeof(tables)/sizeof(tables[0]); + + for (int i=0; i < numTables; i++) { + psString query = pxDataGet(tables[i].sqlFilename); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (where && psListLength(where->list)) { + psString whereClause = psDBGenerateWhereSQL(where, NULL); + psStringAppend(&query, " %s", whereClause); + psFree(whereClause); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + if (!psArrayLength(output)) { + psError(PS_ERR_UNKNOWN, true, "no rows found"); + psFree(output); + return false; + } + + if (clean) { + bool success = true; + if (!strcmp(tables[i].tableName, "warpRun")) { + success = pxSetStateCleaned("warpRun", "state", output); + } else if (!strcmp(tables[i].tableName, "warpSkyfile")) { + success = pxSetStateCleaned("warpSkyfile", "data_state", output); + } + if (!success) { + psFree(output); + psError(PS_ERR_UNKNOWN, false, "pxSetStateClean failed for table %s", tables[i].tableName); + return false; + } + } + + // we must write the export table in non-simple (true) format + if (!ippdbPrintMetadatas(f, output, tables[i].tableName, true)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + psFree(output); + } + + fclose (f); + + return true; +} + +bool importrunMode(pxConfig *config) +{ + unsigned int nFail; + + int numImportTables = 3; + + char tables[3] [80] = {"warpImfile", "warpSkyfile", "warpSkyCellMap"}; + + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_STR(infile, config->args, "-infile", true, false); + + psMetadata *input = psMetadataConfigRead (NULL, &nFail, infile, false); + +#ifdef notdef + fprintf (stderr, "---- input ----\n"); + psMetadataPrint (stderr, input, 1); +#endif + + if (!pxCheckImportVersion(config, input)) { + psError(PS_ERR_UNKNOWN, false, "pxCheckImportVersion failed"); + return false; + } + + psMetadataItem *item = psMetadataLookup (input, "warpRun"); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + psMetadataItem *entry = psListGet (item->data.list, 0); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + warpRunRow *warpRun = warpRunObjectFromMetadata (entry->data.md); + warpRunInsertObject (config->dbh, warpRun); + + // fprintf (stdout, "---- warp run ----\n"); + // psMetadataPrint (stderr, entry->data.md, 1); + + for (int i = 0; i < numImportTables; i++) { + item = psMetadataLookup (input, tables[i]); + psAssert (item, "entry not in input?"); + psAssert (item->type == PS_DATA_METADATA_MULTI, "entry not multi?"); + + switch (i) { + case 0: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + warpImfileRow *warpImfile = warpImfileObjectFromMetadata (entry->data.md); + warpImfileInsertObject (config->dbh, warpImfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 1: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + warpSkyfileRow *warpSkyfile = warpSkyfileObjectFromMetadata (entry->data.md); + warpSkyfileInsertObject (config->dbh, warpSkyfile); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + + case 2: + for (int i = 0; i < item->data.list->n; i++) { + entry = psListGet (item->data.list, i); + assert (entry); + assert (entry->type == PS_DATA_METADATA); + warpSkyCellMapRow *warpSkyCellMap = warpSkyCellMapObjectFromMetadata (entry->data.md); + warpSkyCellMapInsertObject (config->dbh, warpSkyCellMap); + + // fprintf (stdout, "---- row %d ----\n", i); + // psMetadataPrint (stderr, entry->data.md, 1); + } + break; + } + } + return true; +} + +static bool runstateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "LIKE"); + + // PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + // PXOPT_LOOKUP_BOOL(no_magic, config->args, "-no_magic", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + + psString query = pxDataGet("warptool_runstate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + psFree(where); + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpRunState", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +static bool listrunMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, false); + + psMetadata *where = psMetadataAlloc(); + PXOPT_COPY_S64(config->args, where, "-warp_id", "warpRun.warp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-tess_id", "warpRun.tess_id", "=="); + PXOPT_COPY_STR(config->args, where, "-state", "warpRun.state", "=="); + PXOPT_COPY_S64(config->args, where, "-exp_id", "rawExp.exp_id", "=="); + PXOPT_COPY_STR(config->args, where, "-exp_name", "rawExp.exp_name", "=="); + PXOPT_COPY_S64(config->args, where, "-chip_id", "chipRun.chip_id", "=="); + PXOPT_COPY_S64(config->args, where, "-cam_id", "camRun.cam_id", "=="); + PXOPT_COPY_S64(config->args, where, "-fake_id", "fakeRun.fake_id", "=="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_begin", "rawExp.dateobs", ">="); + PXOPT_COPY_TIME(config->args, where, "-dateobs_end", "rawExp.dateobs", "<="); + PXOPT_COPY_STR(config->args, where, "-filter", "rawExp.filter", "LIKE"); + PXOPT_COPY_S64(config->args, where, "-magicked", "warpRun.magicked", "=="); + pxAddLabelSearchArgs (config, where, "-label", "warpRun.label", "LIKE"); + pxAddLabelSearchArgs (config, where, "-data_group", "warpRun.data_group", "LIKE"); + pxAddLabelSearchArgs (config, where, "-dist_group", "warpRun.dist_group", "LIKE"); + + PXOPT_LOOKUP_BOOL(all, config->args, "-all", false); + + PXOPT_LOOKUP_U64(limit, config->args, "-limit", false, false); + PXOPT_LOOKUP_BOOL(simple, config->args, "-simple", false); + PXOPT_LOOKUP_BOOL(pstamp_order, config->args, "-pstamp_order", false); + + // find all rawImfiles matching the default query + psString query = pxDataGet("warptool_listrun.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + // generate where strings for arguments that require extra processing + // beyond PXOPT_COPY* + psString where2 = NULL; + if (!pxmagicAddWhere(config, &where2, "warpRun")) { + psError(psErrorCodeLast(), false, "pxMagicAddWhere failed"); + return false; + } + if (!pxspaceAddWhere(config, &where2, "rawExp")) { + psError(psErrorCodeLast(), false, "pxSpaceAddWhere failed"); + return false; + } + + if (psListLength(where->list)) { + psString whereClause = psDBGenerateWhereConditionSQL(where, NULL); + psStringAppend(&query, " WHERE %s", whereClause); + psFree(whereClause); + } else if (!all && !where2) { + psError(PXTOOLS_ERR_CONFIG, true, "search parameters or -all are required"); + return false; + } + + if (where2) { + if (psListLength(where->list)) { + psStringAppend(&query, " %s", where2); + } else { + psStringAppend(&query, " WHERE 1 %s", where2); + } + } + psFree(where); + + if (pstamp_order) { + // put runs in order of exposure id with newest warp Runs first + // The postage stamp parser depends on this behavior + psStringAppend(&query, "\nORDER by exp_id, warp_id DESC"); + } + + // treat limit == 0 as "no limit" + if (limit) { + psString limitString = psDBGenerateLimitSQL(limit); + psStringAppend(&query, " %s", limitString); + psFree(limitString); + } + + if (!p_psDBRunQuery(config->dbh, query)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + psFree(query); + return false; + } + psFree(query); + + psArray *output = p_psDBFetchResult(config->dbh); + if (!output) { + psErrorCode err = psErrorCodeLast(); + switch (err) { + case PS_ERR_DB_CLIENT: + psError(PXTOOLS_ERR_SYS, false, "database error"); + case PS_ERR_DB_SERVER: + psError(PXTOOLS_ERR_PROG, false, "database error"); + default: + psError(PXTOOLS_ERR_PROG, false, "unknown error"); + } + + return false; + } + if (!psArrayLength(output)) { + psTrace("warptool", PS_LOG_INFO, "no rows found"); + psFree(output); + return true; + } + + if (psArrayLength(output)) { + // negative simple so the default is true + if (!ippdbPrintMetadatas(stdout, output, "warpRun", !simple)) { + psError(PS_ERR_UNKNOWN, false, "failed to print array"); + psFree(output); + return false; + } + } + + psFree(output); + + return true; +} + +// a very specfic function to queue a cleaned warpSkyfile to be updated +static bool setskyfiletoupdateMode(pxConfig *config) +{ + PS_ASSERT_PTR_NON_NULL(config, NULL); + + PXOPT_LOOKUP_S64(warp_id, config->args, "-warp_id", true, false); + PXOPT_LOOKUP_STR(skycell_id, config->args, "-skycell_id", false, false); + PXOPT_LOOKUP_STR(label, config->args, "-set_label", false, false); + + psString query = pxDataGet("warptool_setskyfiletoupdate.sql"); + if (!query) { + psError(PXTOOLS_ERR_SYS, false, "failed to retreive SQL statement"); + return false; + } + + psString setHook = psStringCopy(""); + if (label) { + psStringAppend(&setHook, "\n , warpRun.label = '%s'", label); + } + + if (skycell_id) { + psStringAppend(&query, " AND (warpSkyfile.skycell_id = '%s')", skycell_id); + } + // we do not update components with the magic fault value. They are non-updateable + // (But can be recovered with "warptool -revertwarped -fault 26" (PXTOOL_DO_NOT_REVERT_FAULT) + psStringAppend(&query, " AND (warpSkyfile.fault != %d)", PXTOOL_DO_NOT_REVERT_FAULT); + + if (!p_psDBRunQueryF(config->dbh, query, setHook, warp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + + psFree(setHook); + psFree(query); + + if (!skycell_id) { + // If we are updateing a whole warpRun set skycells with bad quality to 'full' + query = "UPDATE warpSkyfile SET data_state ='full', fault = 0 WHERE warp_id = %" PRId64 " AND quality != 0 AND (data_state != 'full')"; + if (!p_psDBRunQueryF(config->dbh, query, warp_id)) { + psError(PS_ERR_UNKNOWN, false, "database error"); + return false; + } + } + + return true; +} Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/warptool.h =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/warptool.h (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/warptool.h (revision 41104) @@ -0,0 +1,66 @@ +/* + * warptool.h + * + * Copyright (C) 2006 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifndef WARPTOOL_H +#define WARPTOOL_H 1 + +#include "pxtools.h" + +typedef enum { + WARPTOOL_MODE_NONE = 0x0, + WARPTOOL_MODE_DEFINEBYQUERY, + WARPTOOL_MODE_DEFINERUN, + WARPTOOL_MODE_RUNONE, + WARPTOOL_MODE_UPDATERUN, + WARPTOOL_MODE_ADDINPUTEXP, + WARPTOOL_MODE_EXP, + WARPTOOL_MODE_IMFILE, + WARPTOOL_MODE_TOOVERLAP, + WARPTOOL_MODE_ADDOVERLAP, + WARPTOOL_MODE_REVERTOVERLAP, + WARPTOOL_MODE_SCMAP, + WARPTOOL_MODE_TOWARPED, + WARPTOOL_MODE_ADDWARPED, + WARPTOOL_MODE_ADVANCERUN, + WARPTOOL_MODE_WARPED, + WARPTOOL_MODE_REVERTWARPED, + WARPTOOL_MODE_BLOCK, + WARPTOOL_MODE_MASKED, + WARPTOOL_MODE_UNBLOCK, + WARPTOOL_MODE_TOSUMMARY, + WARPTOOL_MODE_ADDSUMMARY, + WARPTOOL_MODE_PENDINGCLEANUPRUN, + WARPTOOL_MODE_PENDINGCLEANUPSKYFILE, + WARPTOOL_MODE_REVERTCLEANUP, + WARPTOOL_MODE_DONECLEANUP, + WARPTOOL_MODE_TOCLEANEDSKYFILE, + WARPTOOL_MODE_TOPURGEDSKYFILE, + WARPTOOL_MODE_TOSCRUBBEDSKYFILE, + WARPTOOL_MODE_TOFULLSKYFILE, + WARPTOOL_MODE_UPDATESKYFILE, + WARPTOOL_MODE_EXPORTRUN, + WARPTOOL_MODE_IMPORTRUN, + WARPTOOL_MODE_RUNSTATE, + WARPTOOL_MODE_LISTRUN, + WARPTOOL_MODE_SETSKYFILETOUPDATE, +} warptoolMode; + +pxConfig *warptoolConfig(pxConfig *config, int argc, char **argv); + +#endif // WARPTOOL_H Index: /branches/ccl_branches/ipponly-20191108/ippTools/src/warptoolConfig.c =================================================================== --- /branches/ccl_branches/ipponly-20191108/ippTools/src/warptoolConfig.c (revision 41104) +++ /branches/ccl_branches/ipponly-20191108/ippTools/src/warptoolConfig.c (revision 41104) @@ -0,0 +1,505 @@ +/* + * warptoolConfig.c + * + * Copyright (C) 2006-2007 Joshua Hoblitt + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the Free + * Software Foundation; version 2 of the License. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * You should have received a copy of the GNU General Public License along with + * program; see the file COPYING. If not, write to the Free Software + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + */ + +#ifdef HAVE_CONFIG_H +#include +#endif + +#include + +#include + +#include "pxtools.h" +#include "warptool.h" + +pxConfig *warptoolConfig(pxConfig *config, int argc, char **argv) +{ + if (!config) { + config = pxConfigAlloc(); + } + + pmConfigReadParamsSet(false); + + // setup site config + config->modules = pmConfigRead(&argc, argv, NULL); + if (!config->modules) { + psError(psErrorCodeLast(), false, "Can't find site configuration"); + psFree(config); + return NULL; + } + + psTime *now = psTimeGetNow(PS_TIME_TAI); + + // -definebyquery + // XXX need to allow multiple chip_ids + // XXX need to allow multiple exp_ids + psMetadata *definebyqueryArgs = psMetadataAlloc(); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-fake_id", 0, "search by fake_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip_id", 0); + psMetadataAddS64(definebyqueryArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exp_id", 0); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exp_name", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-inst", 0, "search for camera", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-telescope", 0, "search for telescope", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-exp_tag", 0, "search by exp_tag", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-exp_type", 0, "search by exp_type", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-filelevel", 0, "search by filelevel", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-filter", 0, "search for filter", NULL); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-airmass_min", 0, "search by min airmass", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-airmass_max", 0, "search by max airmass", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-ra_min", 0, "search by min RA (degrees) ", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-ra_max", 0, "search by max RA (degrees) ", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-decl_min", 0, "search by min DEC (degrees)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-decl_max", 0, "search by max DEC (degrees)", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-exp_time_min", 0, "search by min exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-exp_time_max", 0, "search by max exposure time", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-sat_pixel_frac_min", 0, "search by min fraction of saturated pixels", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-sat_pixel_frac_max", 0, "search by max fraction of saturated pixels", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-bg_min", 0, "search by min background", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-bg_max", 0, "search by max background", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-bg_stdev_min", 0, "search by min background standard deviation", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-bg_stdev_max", 0, "search by max background standard deviation", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-bg_mean_stdev_min", 0, "search by min background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-bg_mean_stdev_max", 0, "search by max background mean standard deviation (across imfiles)", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-alt_min", 0, "search by min altitude", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-alt_max", 0, "search by max altitude", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-az_min", 0, "search by min azimuth ", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-az_max", 0, "search by max azimuth ", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_min", 0, "search by min ccd tempature", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-ccd_temp_max", 0, "search by max ccd tempature", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-posang_min", 0, "search by min rotator position angle", NAN); + psMetadataAddF64(definebyqueryArgs, PS_LIST_TAIL, "-posang_max", 0, "search by max rotator position angle", NAN); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-object", 0, "search by exposure object", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-comment", 0, "search by comment field (LIKE comparison)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-obs_mode", 0, "search by observation mode", NULL); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-sun_angle_min", 0, "search by min solar angle", NAN); + psMetadataAddF32(definebyqueryArgs, PS_LIST_TAIL, "-sun_angle_max", 0, "search by max solar angle", NAN); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-reduction", 0, "search by fakeRun reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search on fakeRun label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_mode", 0, "define mode (warp, diff, stack, magic)", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_workdir", 0, "define workdir", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_label", 0, "define label", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dvodb", 0, "define DVO db", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_tess_id", 0, "define tess ID", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_end_stage", 0, "define end stage", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definebyqueryArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + + psMetadataAddTime(definebyqueryArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddBool(definebyqueryArgs, PS_LIST_TAIL, "-pretend", 0, "do not actually modify the database", false); + + // -definerun + psMetadata *definerunArgs = psMetadataAlloc(); + psMetadataAddS64(definerunArgs, PS_LIST_TAIL, "-fake_id", 0, "define camtool ID (required)", 0); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-mode", 0, "define mode (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-workdir", 0, "define workdir (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-label", 0, "define label", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-dvodb", 0, "define dvodb", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-tess_id", 0, "define tess_id (required)", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-reduction", 0, "define reduction class", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-end_stage", 0, "define end stage", NULL); + psMetadataAddTime(definerunArgs, PS_LIST_TAIL, "-registered", 0, "time detrend run was registered", now); + psMetadataAddBool(definerunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_end_stage", 0, "define end stage", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define data group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define dist group", NULL); + psMetadataAddStr(definerunArgs, PS_LIST_TAIL, "-set_note", 0, "define note", NULL); + + // -updaterun + // XXX need to allow multiple fake_ids + // XXX need to allow multiple fake_ids + // XXX need to allow multiple chip_ids + // XXX need to allow multiple exp_ids + psMetadata *updaterunArgs = psMetadataAlloc(); + psMetadataAddS64(updaterunArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-reduction", 0, "search by warpRun reduction class", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-state", 0, "search by warpRun state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-data_group", 0, "search by warpRun data_group (LIKE comparison)", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-dist_group", 0, "search by warpRun dist_group (LIKE comparison)", NULL); + psMetadataAddTime(updaterunArgs, PS_LIST_TAIL, "-registered_begin", 0, "search for runs by registration time (>=)", NULL); + psMetadataAddTime(updaterunArgs, PS_LIST_TAIL, "-registered_end", 0, "search for runs by registration time (<)", NULL); + psMetadataAddBool(updaterunArgs, PS_LIST_TAIL, "-pretend", 0, "only pretend to run the query", false); + + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_state", 0, "define new state", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_label", 0, "define new value for label", 0); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_data_group", 0, "define new data_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_dist_group", 0, "define new dist_group", NULL); + psMetadataAddStr(updaterunArgs, PS_LIST_TAIL, "-set_note", 0, "define new note", NULL); + pxmagicAddArguments(updaterunArgs); + + // -exp + psMetadata *expArgs = psMetadataAlloc(); + psMetadataAddS64(expArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddS64(expArgs, PS_LIST_TAIL, "-fake_id", 0, "search by camtool ID", 0); + psMetadataAddU64(expArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(expArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -imfile + psMetadata *imfileArgs = psMetadataAlloc(); + psMetadataAddS64(imfileArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddS64(imfileArgs, PS_LIST_TAIL, "-fake_id", 0, "search by camtool ID", 0); + psMetadataAddU64(imfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(imfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -tooverlap + psMetadata *tooverlapArgs = psMetadataAlloc(); + psMetadataAddS64(tooverlapArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddStr(tooverlapArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddU64(tooverlapArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tooverlapArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addoverlap + psMetadata *addoverlapArgs = psMetadataAlloc(); + psMetadataAddStr(addoverlapArgs, PS_LIST_TAIL, "-mapfile", 0, "path to skycell <-> imfile mapping file", NULL); + psMetadataAddS64(addoverlapArgs, PS_LIST_TAIL, "-warp_id", 0, "set warp ID", 0); + psMetadataAddS16(addoverlapArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + + // -revertoverlap + psMetadata *revertoverlapArgs = psMetadataAlloc(); + psMetadataAddS64(revertoverlapArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(revertoverlapArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(revertoverlapArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(revertoverlapArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddS16(revertoverlapArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertoverlapArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + + + // -scmap + psMetadata *scmapArgs = psMetadataAlloc(); + psMetadataAddS64(scmapArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(scmapArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(scmapArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tess ID", NULL); + psMetadataAddStr(scmapArgs, PS_LIST_TAIL, "-class_id", 0, "search by class ID", NULL); + psMetadataAddU64(scmapArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(scmapArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -towarped + psMetadata *towarpedArgs = psMetadataAlloc(); + psMetadataAddS64(towarpedArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(towarpedArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label", NULL); + psMetadataAddU64(towarpedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(towarpedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addwarped + psMetadata *addwarpedArgs = psMetadataAlloc(); + psMetadataAddS64(addwarpedArgs, PS_LIST_TAIL, "-warp_id", 0, "define warptool ID (required)", 0); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-skycell_id", 0, "define skycell ID (required)", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-tess_id", 0, "define tessellation ID (required)", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-uri", 0, "define URI of file", 0); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-path_base", 0, "define base output location", 0); + psMetadataAddF64(addwarpedArgs, PS_LIST_TAIL, "-bg", 0, "define exposure background", NAN); + psMetadataAddF64(addwarpedArgs, PS_LIST_TAIL, "-bg_stdev", 0, "define exposure background stdev", NAN); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-dtime_warp", 0, "define elapsed processing time", NAN); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-dtime_script", 0, "define elapsed time in script (seconds)", NAN); + psMetadataAddS32(addwarpedArgs, PS_LIST_TAIL, "-xmin", 0, "define minimum x value", INT_MAX); + psMetadataAddS32(addwarpedArgs, PS_LIST_TAIL, "-xmax", 0, "define maximum x value", -INT_MAX); + psMetadataAddS32(addwarpedArgs, PS_LIST_TAIL, "-ymin", 0, "define minimum y value", INT_MAX); + psMetadataAddS32(addwarpedArgs, PS_LIST_TAIL, "-ymax", 0, "define maximum y value", -INT_MAX); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-hostname", 0, "define hostname", 0); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-good_frac", 0, "define %% of good pixels", NAN); + psMetadataAddS64(addwarpedArgs, PS_LIST_TAIL, "-set_magicked", 0, "define if this skycell has been magicked", 0); + psMetadataAddS16(addwarpedArgs, PS_LIST_TAIL, "-fault", 0, "set fault code", 0); + psMetadataAddS16(addwarpedArgs, PS_LIST_TAIL, "-quality", 0, "set quality", 0); + + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-ver_pslib", 0, "define psLib version", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-ver_psmodules", 0, "define psModules version", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-ver_psphot", 0, "define psphot version", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-ver_ppstats", 0, "define ppStats version", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-ver_pswarp", 0, "define pswarp version", NULL); + psMetadataAddStr(addwarpedArgs, PS_LIST_TAIL, "-ver_streaks", 0, "define streaksremove version", NULL); + + psMetadataAddS32(addwarpedArgs, PS_LIST_TAIL, "-maskfrac_npix", 0, "define number of pixels used for maskstats", 0); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-maskfrac_static", 0, "define static mask fraction", NAN); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-maskfrac_dynamic", 0, "define dynamic mask fraction", NAN); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-maskfrac_magic", 0, "define magic mask fraction", NAN); + psMetadataAddF32(addwarpedArgs, PS_LIST_TAIL, "-maskfrac_advisory", 0, "define advisory mask fraction", NAN); + + psMetadataAddS16(addwarpedArgs, PS_LIST_TAIL, "-background_model", 0, "define background model version", 0); + + // -warped + psMetadata *warpedArgs = psMetadataAlloc(); + psMetadataAddS64(warpedArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddS64(warpedArgs, PS_LIST_TAIL, "-warp_skyfile_id", 0, "search by skyfile ID", 0); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddS64(warpedArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure tag", 0); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure tag", 0); + psMetadataAddS64(warpedArgs, PS_LIST_TAIL, "-fake_id", 0, "search by phase 3 version of exposure tag", 0); + psMetadataAddTime(warpedArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(warpedArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-filter", 0, "search for exposures by filter", NULL); + psMetadataAddS16(warpedArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by warpRun data_group", NULL); + psMetadataAddStr(warpedArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by warpRun dist_group", NULL); + psMetadataAddS16(warpedArgs, PS_LIST_TAIL, "-background_model", 0, "search by background model version", 0); + // add magic related arguments + pxmagicAddArguments(warpedArgs); + // add arguments for spatial search + pxspaceAddArguments(warpedArgs); + psMetadataAddBool(warpedArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + + psMetadataAddBool(warpedArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(warpedArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(warpedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -listrun + psMetadata *listrunArgs = psMetadataAlloc(); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure id", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure tag", NULL); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-chip_id", 0, "search by chip id", 0); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-cam_id", 0, "search by cam id", 0); + psMetadataAddS64(listrunArgs, PS_LIST_TAIL, "-fake_id", 0, "search by phase 3 version of exposure tag (fake_id)", 0); + psMetadataAddTime(listrunArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(listrunArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-filter", 0, "search for exposures by filter", NULL); + psMetadataAddS16(listrunArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by warpRun data_group", NULL); + psMetadataAddStr(listrunArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by warpRun dist_group", NULL); + // add magic related arguments + pxmagicAddArguments(listrunArgs); + // add arguments for spatial search + pxspaceAddArguments(listrunArgs); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-pstamp_order", 0, "order results for postage stamp server", false); + + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(listrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(listrunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -revertwarped + // XXX need to allow multiple fake_ids + // XXX need to allow multiple fake_ids + // XXX need to allow multiple chip_ids + // XXX need to allow multiple exp_ids + psMetadata *revertwarpedArgs = psMetadataAlloc(); + psMetadataAddS64(revertwarpedArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(revertwarpedArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by skycell ID", NULL); + psMetadataAddStr(revertwarpedArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(revertwarpedArgs, PS_LIST_TAIL, "-reduction", 0, "search by warpRun reduction class", NULL); + psMetadataAddStr(revertwarpedArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddS16(revertwarpedArgs, PS_LIST_TAIL, "-fault", 0, "search by fault code", 0); + psMetadataAddBool(revertwarpedArgs, PS_LIST_TAIL, "-all", 0, "allow everything to be queued without search terms", false); + + // -advancerun + psMetadata *advancerunArgs = psMetadataAlloc(); + psMetadataAddS64(advancerunArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddStr(advancerunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by label ", NULL); + psMetadataAddU64(advancerunArgs, PS_LIST_TAIL, "-limit", 0, "limit exposures to advance to N items", 0); + + // -block + psMetadata *blockArgs = psMetadataAlloc(); + psMetadataAddStr(blockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to mask out (required)", NULL); + + // -masked + psMetadata *maskedArgs = psMetadataAlloc(); + psMetadataAddBool(maskedArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -unblock + psMetadata *unblockArgs = psMetadataAlloc(); + psMetadataAddStr(unblockArgs, PS_LIST_TAIL, "-label", 0, "name of a label to unmask (required)", NULL); + + // -tosummary + psMetadata *tosummaryArgs = psMetadataAlloc(); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID", 0); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure tag", 0); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure tag", 0); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-fake_id", 0, "search by phase 3 version of exposure tag", 0); + psMetadataAddTime(tosummaryArgs, PS_LIST_TAIL, "-dateobs_begin", 0, "search for exposures by time (>=)", NULL); + psMetadataAddTime(tosummaryArgs, PS_LIST_TAIL, "-dateobs_end", 0, "search for exposures by time (<=)", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-filter", 0, "search for exposures by filter", NULL); + psMetadataAddS64(tosummaryArgs, PS_LIST_TAIL, "-magicked", 0, "search by magic id", 0); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by warpRun data_group", NULL); + psMetadataAddStr(tosummaryArgs, PS_LIST_TAIL, "-dist_group", PS_META_DUPLICATE_OK, "search by warpRun dist_group", NULL); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-destreaked", 0, "search for runs that have been destreaked", false); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-not_destreaked", 0, "search for runs that have not been destreaked", false); + + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-all", 0, "search without arguments", false); + psMetadataAddU64(tosummaryArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(tosummaryArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -addsummary + psMetadata *addsummaryArgs = psMetadataAlloc(); + psMetadataAddS64(addsummaryArgs, PS_LIST_TAIL, "-warp_id", 0, "set warp ID", 0); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-projection_cell", 0, "set projection cell", NULL); + psMetadataAddStr(addsummaryArgs, PS_LIST_TAIL, "-path_base", 0, "set summary path base", NULL); + + // -pendingcleanuprun + psMetadata *pendingcleanuprunArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanuprunArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddBool(pendingcleanuprunArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanuprunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -pendingcleanupskyfile + psMetadata *pendingcleanupskyfileArgs = psMetadataAlloc(); + psMetadataAddStr(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "list blocks for specified label", NULL); + psMetadataAddS64(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warp ID (required)", 0); + psMetadataAddBool(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-all", 0, "list all skycells regardless of data_state", false); + psMetadataAddBool(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(pendingcleanupskyfileArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -revertcleanup + psMetadata *revertcleanupArgs = psMetadataAlloc(); + psMetadataAddS64(revertcleanupArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-data_group", PS_META_DUPLICATE_OK, "search by warpRun data_group", NULL); + psMetadataAddStr(revertcleanupArgs, PS_LIST_TAIL, "-state", 0, "search by state", NULL); + + // -donecleanup + psMetadata *donecleanupArgs = psMetadataAlloc(); + psMetadataAddStr(donecleanupArgs, PS_LIST_TAIL, "-label", 0, "list blocks for specified label", NULL); + psMetadataAddBool(donecleanupArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + psMetadataAddU64(donecleanupArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + + // -tocleanedskyfile + psMetadata *tocleanedskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(tocleanedskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "warptool ID to update", 0); + psMetadataAddStr(tocleanedskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + + // -topurgedskyfile + psMetadata *topurgedskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(topurgedskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "warptool ID to update", 0); + psMetadataAddStr(topurgedskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + + // -toscrubbedskyfile + psMetadata *toscrubbedskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(toscrubbedskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "warptool ID to update", 0); + psMetadataAddStr(toscrubbedskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + + // -tofullskyfile + psMetadata *tofullskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(tofullskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "warptool ID to update", 0); + psMetadataAddStr(tofullskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update", NULL); + psMetadataAddS64(tofullskyfileArgs, PS_LIST_TAIL, "-set_magicked", 0, "new value for magicked", 0); + psMetadataAddS16(tofullskyfileArgs, PS_LIST_TAIL, "-set_quality", 0, "new value for quality", 0); + psMetadataAddS16(tofullskyfileArgs, PS_LIST_TAIL, "-set_background_model", 0, "new value for background_model", 0); + + // -updateskyfile + psMetadata *updateskyfileArgs = psMetadataAlloc(); + psMetadataAddS64(updateskyfileArgs, PS_LIST_TAIL, "-warp_id", 0, "warptool ID to update (required)", 0); + psMetadataAddStr(updateskyfileArgs, PS_LIST_TAIL, "-skycell_id", 0, "skycell ID to update (required)", NULL); + psMetadataAddS16(updateskyfileArgs, PS_LIST_TAIL, "-fault", 0, "new fault code", 0); + psMetadataAddS16(updateskyfileArgs, PS_LIST_TAIL, "-set_quality", 0,"new quality value", 0); + psMetadataAddStr(updateskyfileArgs, PS_LIST_TAIL, "-set_state", 0, "set state", 0); + psMetadataAddS16(updateskyfileArgs, PS_LIST_TAIL, "-set_background_model", 0, "set the background_model value", 0); + // -exportrun + psMetadata *exportrunArgs = psMetadataAlloc(); + psMetadataAddS64(exportrunArgs, PS_LIST_TAIL, "-warp_id", 0, "export this warp ID (required)", 0); + psMetadataAddStr(exportrunArgs, PS_LIST_TAIL, "-outfile", 0, "export to this file (required)", NULL); + psMetadataAddU64(exportrunArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(exportrunArgs, PS_LIST_TAIL, "-clean", 0, "export run in cleaned state", false); + + // -importrun + psMetadata *importrunArgs = psMetadataAlloc(); + psMetadataAddStr(importrunArgs, PS_LIST_TAIL, "-infile", 0, "import from this file (required)", NULL); + + // -runstate + psMetadata *runstateArgs = psMetadataAlloc(); + psMetadataAddS64(runstateArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID", 0); +// psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-tess_id", 0, "search by tessellation ID", NULL); + psMetadataAddS64(runstateArgs, PS_LIST_TAIL, "-exp_id", 0, "search by exposure tag", 0); + psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-exp_name", 0, "search by exposure tag", 0); + psMetadataAddStr(runstateArgs, PS_LIST_TAIL, "-label", PS_META_DUPLICATE_OK, "search by warpRun label", NULL); + psMetadataAddBool(runstateArgs, PS_LIST_TAIL, "-no_magic", 0, "magic is not necessary for result", false); + + psMetadataAddU64(runstateArgs, PS_LIST_TAIL, "-limit", 0, "limit result set to N items", 0); + psMetadataAddBool(runstateArgs, PS_LIST_TAIL, "-simple", 0, "use the simple output format", false); + + // -setskyfiletoupdate + psMetadata *setskyfiletoupdateArgs = psMetadataAlloc(); + psMetadataAddS64(setskyfiletoupdateArgs, PS_LIST_TAIL, "-warp_id", 0, "search by warptool ID (required)", 0); + psMetadataAddStr(setskyfiletoupdateArgs, PS_LIST_TAIL, "-skycell_id", 0, "search by tessellation ID", NULL); + psMetadataAddStr(setskyfiletoupdateArgs, PS_LIST_TAIL, "-set_label", 0, "new value for warpRun.label", NULL); + + + psFree(now); + psMetadata *argSets = psMetadataAlloc(); + psMetadata *modes = psMetadataAlloc(); + + PXOPT_ADD_MODE("-definebyquery", "", WARPTOOL_MODE_DEFINEBYQUERY, definebyqueryArgs); + PXOPT_ADD_MODE("-definerun", "", WARPTOOL_MODE_DEFINERUN, definerunArgs); + PXOPT_ADD_MODE("-updaterun", "", WARPTOOL_MODE_UPDATERUN, updaterunArgs); + PXOPT_ADD_MODE("-exp", "", WARPTOOL_MODE_EXP, expArgs); + PXOPT_ADD_MODE("-imfile", "", WARPTOOL_MODE_IMFILE, imfileArgs); + PXOPT_ADD_MODE("-tooverlap", "", WARPTOOL_MODE_TOOVERLAP, tooverlapArgs); + PXOPT_ADD_MODE("-addoverlap", "", WARPTOOL_MODE_ADDOVERLAP, addoverlapArgs); + PXOPT_ADD_MODE("-revertoverlap", "", WARPTOOL_MODE_REVERTOVERLAP, revertoverlapArgs); + PXOPT_ADD_MODE("-scmap", "", WARPTOOL_MODE_SCMAP, scmapArgs); + PXOPT_ADD_MODE("-towarped", "", WARPTOOL_MODE_TOWARPED, towarpedArgs); + PXOPT_ADD_MODE("-addwarped", "", WARPTOOL_MODE_ADDWARPED, addwarpedArgs); + PXOPT_ADD_MODE("-advancerun", "", WARPTOOL_MODE_ADVANCERUN, advancerunArgs); + PXOPT_ADD_MODE("-warped", "", WARPTOOL_MODE_WARPED, warpedArgs); + PXOPT_ADD_MODE("-listrun", "", WARPTOOL_MODE_LISTRUN, listrunArgs); + PXOPT_ADD_MODE("-runstate", "", WARPTOOL_MODE_RUNSTATE, runstateArgs); + PXOPT_ADD_MODE("-revertwarped", "", WARPTOOL_MODE_REVERTWARPED, revertwarpedArgs); + PXOPT_ADD_MODE("-block", "set a label block", WARPTOOL_MODE_BLOCK, blockArgs); + PXOPT_ADD_MODE("-masked", "show blocked lables", WARPTOOL_MODE_MASKED, maskedArgs); + PXOPT_ADD_MODE("-unblock", "remove a label block", WARPTOOL_MODE_UNBLOCK, unblockArgs); + PXOPT_ADD_MODE("-pendingcleanuprun", "show runs that need to be cleaned up", WARPTOOL_MODE_PENDINGCLEANUPRUN, pendingcleanuprunArgs); + PXOPT_ADD_MODE("-pendingcleanupskyfile", "show runs that need to be cleaned up", WARPTOOL_MODE_PENDINGCLEANUPSKYFILE, pendingcleanupskyfileArgs); + PXOPT_ADD_MODE("-revertcleanup", "revert cleanup runs with errors", WARPTOOL_MODE_REVERTCLEANUP, revertcleanupArgs); + PXOPT_ADD_MODE("-donecleanup", "show runs that have been cleaned", WARPTOOL_MODE_DONECLEANUP, donecleanupArgs); + PXOPT_ADD_MODE("-tocleanedskyfile", "set skyfile as cleaned", WARPTOOL_MODE_TOCLEANEDSKYFILE, tocleanedskyfileArgs); + PXOPT_ADD_MODE("-topurgedskyfile", "set skyfile as purged", WARPTOOL_MODE_TOPURGEDSKYFILE, topurgedskyfileArgs); + PXOPT_ADD_MODE("-toscrubbedskyfile", "set skyfile as scrubbed", WARPTOOL_MODE_TOSCRUBBEDSKYFILE, toscrubbedskyfileArgs); + PXOPT_ADD_MODE("-tofullskyfile", "set skyfile as full (updated)", WARPTOOL_MODE_TOFULLSKYFILE, tofullskyfileArgs); + PXOPT_ADD_MODE("-updateskyfile", "update fault code for skyfile", WARPTOOL_MODE_UPDATESKYFILE, updateskyfileArgs); + PXOPT_ADD_MODE("-setskyfiletoupdate", "set cleaned skyfile to be updated", WARPTOOL_MODE_SETSKYFILETOUPDATE, setskyfiletoupdateArgs); + PXOPT_ADD_MODE("-tosummary", "show runs that can be summarized", WARPTOOL_MODE_TOSUMMARY, tosummaryArgs); + PXOPT_ADD_MODE("-addsummary", "add entry to the summary table", WARPTOOL_MODE_ADDSUMMARY, addsummaryArgs); + + PXOPT_ADD_MODE("-exportrun", "export run for import on other database", WARPTOOL_MODE_EXPORTRUN, exportrunArgs); + PXOPT_ADD_MODE("-importrun", "import run from metadata file", WARPTOOL_MODE_IMPORTRUN, importrunArgs); + + if (!pxGetOptions(stderr, argc, argv, config, modes, argSets)) { + psError(PS_ERR_UNKNOWN, true, "option parsing failed"); + psFree(argSets); + psFree(modes); + psFree(config); + return NULL; + } + + psFree(argSets); + psFree(modes); + + // define Database handle, if used + // do this last so we don't setup a connection before CLI options are + // validated + config->dbh = psMemIncrRefCounter(pmConfigDB(config->modules)); + if (!config->dbh) { + psError(PS_ERR_UNKNOWN, false, "Can't configure database"); + psFree(config); + return NULL; + } + + return config; +}