Index: /trunk/Ohana/doc/README =================================================================== --- /trunk/Ohana/doc/README (revision 2508) +++ /trunk/Ohana/doc/README (revision 2508) @@ -0,0 +1,46 @@ +Documenation for Ohana: + + Ohana includes: + + 1) fits - FITS I/O library (Eugene Magnier) (V 1.0) + These are clean, C-like FITS interaction routines which are + meant to be easy to program with and easy to read, etc. + + 2) readline - the GNU readline library. (FSF / GNU) (V1.1) + This routines allow for easy emacs / vi-like cursor interactions. + + 3) opihi - a shell template program (Eugene Magnier) (V1.0) + This program provides a tcsh-like interaction with macros + and math capabilities. It is meant to be a template for any + program which may require a shell-like interaction, macros, and math. + + 4) mana - an image manipulation program. (Eugene Magnier) (V1.0) + This program is in someways similar to MIDAS, IRAF, or Vista (the standard + astrophysics analysis packages). However, mana is meant to be more + stripped down (it is not a "do everything in the world" program!). + Mana also has three major advantages over the large programs: + a) it is easy to modify (clean, friendly code!) + b) it has a nice macro language with math at the shell input + c) it has a tcsh-like cursor interaction + + 5) kii - display program for mana. + This is the X window display tool used by mana. It is not meant to be run + by the user, only by mana. + + + +----------------------------------------------------------------------------------------- +Notes + +Compiled successfully for HP730, gcc & cc + +Compiled successfully for Sparc + + + + +----------------------------------------------------------------------------------------- +Problems / Bugs + +mana / kii are rather slow in general - can we do anything about this? + Index: /trunk/Ohana/doc/README.mana =================================================================== --- /trunk/Ohana/doc/README.mana (revision 2508) +++ /trunk/Ohana/doc/README.mana (revision 2508) @@ -0,0 +1,133 @@ + + April 26, 2000 + + I am providing the first release of 'mana' and the supporting + packages. Here I will describe the needed steps to get the program + compiled and running. + +Directory organization and Architecture selection: + + Mana is part of a large collection of software called 'ohana'. This + distribution of mana only includes the components of ohana needed for + mana. The ohana collection is contained within a directory tree that + eases the use of the package on a multi-system network. To this end, + there are separate bin and lib directories for different architecture + machines. At the top level of 'ohana' there are the src, bin, lib, + and include directories. bin and lib have subdirectories named + e.g. bin/linux, bin/sol, or bin/hp. The Makefiles throughout 'ohana' + use an environment variable ARCH to decide which of these directories + is the appropriate destination. Therefore: + + BEFORE compiling any of the ohana programs, please set the ARCH + environment variable appropriately for your machine. To do this + automatically, I use the following lines in my .cshrc: + + ### automatically setting the ARCH variable: + set sys=`uname -s` + switch ($sys) + case IRIX64: + setenv ARCH irix; + breaksw; + case SunOS: + set ver=`uname -r | awk '{print substr($1,1,1)}'`; + if ($ver == 5) then + setenv ARCH sol + else + setenv ARCH sun4 + endif + breaksw; + case Linux: + setenv ARCH linux; + breaksw; + case HP-UX: + setenv ARCH hp; + breaksw; + default: + echo "unknown architecture"; + setenv ARCH unknown; + breaksw; + endsw + + This mechanism also lets you set your PATH correctly, no matter what + architecture you are on. just include in your PATH + /h/fred/src/ohana/bin/{$ARCH} (or whatever the path is to the ohana + package!). + + In order to configure the ohana package for your local installation, + edit the file ohana/Configure. Change the entry for ROOT to point + to the correct location of the ohana package. Also in Configure, + you can tell ohana to install things in a different location from + the ohana directory structure. + + +Necessary Libraries: + + I've included in this distribution all of the necessary libraries to + compile mana. Two of these are part of the ohana package: the fits + and ohana libraries. These two should compile trivially when you + run make. There are several other libraries that are typically + included in various UNIX installations, but are sometimes missing. + I've included them in this distribution in case you need them, but + the Makefiles are not well tuned to fit with the ohana layout. In + particular, libjpeg and libpng you will have to compile by hand (ie, + by going into those directories and compiling there, not from the + top level ohana directory). Install the resulting libraries and + include files in the appropriate locations (either in your system, + or just in the appropriate lib and include directores in ohana). + The readline library might compile correctly from the ohana + Makefile, but I am not sure of this. + + It is a good idea to try the mana compilation without dealing with + these libraries. You'll find out soon enough which ones your + systems doesn't have! + + The other oddity is the situation with libnsl and libsocket. These + are required libraries for Suns, but not available (not even needed) + under linux. To keep the same link lines in the Makefiles, I have + made a dummy version of libnsl and libsocket, which should be put in + ohana/lib/linux. see the ohana/src/libdummy/Makefile for details. + +Compiling and installing: + + First of all, move Makefile.mana to Makefile. + + If the libaries are all OK, then you should be able to compile and + install mana and the display programs kapa and kii with the command: + + make install + + +Running Mana: + + Just a couple of notes about running mana. I'm hoping the commands + are generally obvious and the help files helpful... + + When mana starts, it loads a resource file ~/.manarc. This is just + a file with a bunch of mana commands. They can include common macro + definitions, defining variables, or whatever. I've included an + example as mana.rc in ohana/src/mana/doc/ + + Mana uses two external programs to display images and vector plots, + nominally called 'kii' and 'kapa'. By default, it assumes those two + names, but user may change the names if needed. Mana looks at the + variables (mana variables, not env variables) KAPA and KII to decide + the program names. By default these are set to 'kapa' and 'kii'. + Type "echo $KAPA" or "echo $KII" in mana to see the current values. + If you need to place these programs outside your path or give them + different names, just assign these variables (perhaps in the + .manarc) to the correct names and they should work fine. + + In the same way, mana looks for the help files in a preset location, + specifically the directory of the mana executable, subdirectory + mana.hlp. So, if you have mana in /usr/local/bin, mana will look + for the help files in /usr/local/bin/mana.hlp. But again, this can + be reassigned by the user by changing the mana variable HELPDIR. + + Two useful mana commands are ? and ??. The first lists all commands + and gives a little helpline. the second lists are currently defined + mana variables. + + + + Eugene Magnier + Index: /trunk/Ohana/doc/Todo =================================================================== --- /trunk/Ohana/doc/Todo (revision 2508) +++ /trunk/Ohana/doc/Todo (revision 2508) @@ -0,0 +1,21 @@ +Things to do / fix in the ohana release + +1- error handling in FITS (fits_error_message) + +2- ohana PS overlay needs to be cleaned + +3- kii needs to handle ieee errors in the image without crshing + +4- check for kii errors if there is no image displayed + +5- loops in mana + +6- better mana image math? (parse expressions into RPN) + +7- mana logical image assignments + +8- mana: function to read a line from file into a variable + +9- variables from the image header + +10- more than 1 mana at a time Index: /trunk/Ohana/doc/allocate.txt =================================================================== --- /trunk/Ohana/doc/allocate.txt (revision 2508) +++ /trunk/Ohana/doc/allocate.txt (revision 2508) @@ -0,0 +1,40 @@ + +# define MIN(X,Y) ((X) < (Y) ? (X) : (Y)) +# define MAX(X,Y) ((X) > (Y) ? (X) : (Y)) + +# ifndef ALLOCATE +# define ALLOCATE(X,T,S) \ + X=(T *)malloc((unsigned) ((S)*sizeof(T)));\ + if(X==NULL) \ + { \ + fprintf(stderr,"failed to malloc\n");\ + exit(0);\ + } +# define REALLOCATE(X,T,S) \ + X=(T *)realloc(X,(unsigned) ((S)*sizeof(T))); \ + if(X==NULL) \ + { \ + fprintf(stderr,"failed to realloc\n"); \ + exit(0); \ + } +# endif /* ALLOCATE */ + +example () { + + float *fred; + int Nfred, NFRED, i; + + NFRED = 100; + ALLOCATE (fred, float, NFRED); + + for (i = 0; fscanf (stdin, "%f", &fred[i]) != EOF; i++) { + if (i == NFRED - 1) { + NFRED += 100; + REALLOCATE (fred, float, NFRED); + } + } + Nfred = i; + REALLOCATE (fred, float, MAX (Nfred, 1)); + +} + Index: /trunk/Ohana/doc/analysis.txt =================================================================== --- /trunk/Ohana/doc/analysis.txt (revision 2508) +++ /trunk/Ohana/doc/analysis.txt (revision 2508) @@ -0,0 +1,25 @@ + +major analysis steps: + +bias, flat, dark: images (*.f) +check images +doscript: parameter files (*.par_in) +dophot: object lists (*.obj_out), misc files (*.par_out, *.dout, *.shd_out) +striphead: header files (*.head) +astro +rastro +relphot +lightcurve +absphot + + +1) bias, dark, flat, zero bad columns +2) create a file containing all image names (*.f) +3) dosetup files: this will + 1) strip the headers (*.head), + 2) run mana and ask you to choose a star for every image + (a good time to check the quality of each image) + 3) create a script to run dophot (dophot.script) +4) dophot.script +5) + Index: /trunk/Ohana/doc/arguments.txt =================================================================== --- /trunk/Ohana/doc/arguments.txt (revision 2508) +++ /trunk/Ohana/doc/arguments.txt (revision 2508) @@ -0,0 +1,63 @@ +# include + +int get_argument (int argc, char **argv, char *arg) { + + int i; + + for (i = 0; i < argc; i++) { + if (!strcmp(argv[i], arg)) + return (i); + } + + return ((int) NULL); +} + +int remove_argument (int N, int *argc, char **argv) { + + int i; + + if ((N != (int) NULL) && (N != 0)) { + (*argc)--; + for (i = N; i < *argc; i++) { + argv[i] = argv[i+1]; + } + } + + return (N); + +} + +main (int argc, char **argv) { + + int VERBOSE; + float size; + + /* example of an optional argument */ + VERBOSE = FALSE; + if (N = get_argument (argc, argv, "-v")) { + remove_argument (i, &argc, argv); + VERBOSE = TRUE; + } + + /* one way to do an required argument */ + size = -1; + if (N = get_argument (argc, argv, "-s")) { + remove_argument (i, &argc, argv); + size = atof (argv[N]); + remove_argument (i, &argc, argv); + } + if (size < 0) { + fprintf (stderr, "USAGE: program (file) -s (size) \n"); + exit (1); + } + + /* another way to demand an argument */ + if (argc != 2) { + fprintf (stderr, "USAGE: program (file) -s (size) \n"); + exit (1); + } + +} + + + Index: /trunk/Ohana/doc/compiling.txt =================================================================== --- /trunk/Ohana/doc/compiling.txt (revision 2508) +++ /trunk/Ohana/doc/compiling.txt (revision 2508) @@ -0,0 +1,24 @@ + +There are a few steps needed to compile spicam. First, untar the +ohana.tgz file. Edit the file Configure in the top-level directory. +ROOT should be the path of the top-level directory. There are some +destination directories, places where compiled things will go. there +are also two lines for the X library and include directories, though +spicam does not require this information. + +There are a couple of libraries that need to be compiled to make +spicam. first, type "make install.ohana". this will compile a set +of miscellaneous libraries and install the file libohana.a in your +LDIR (defined in Configure). Next, type "make install.fits". this +will compile the FITS library and install libFITS.a in LDIR (note that +this is not the Goddard fitsio package...). Next, you need readline, +so if you don't have it already, type "make install.readline". same +thing. + +finally, you are ready to compile spicam. type "make install.spicam" +and it should compile OK. + +good luck! + +gene + Index: /trunk/Ohana/doc/elixir-todo.txt =================================================================== --- /trunk/Ohana/doc/elixir-todo.txt (revision 2508) +++ /trunk/Ohana/doc/elixir-todo.txt (revision 2508) @@ -0,0 +1,26 @@ + +Cleaning the Ohana portion of Elixir: + +Goals: + - enable CVS + - allow for tarball releases of ohana & subsets + - clean up the software configuration + +- identify external dependencies + * libpng + * libjpeg + * readline + +- move notes & documentation to a single location + +- fix the compile system for 'misc' to match the others + +- remove the links / save links in a tarball + +- compilation should be possible with just: + + configure + make + + do this by hand + Index: /trunk/Ohana/doc/exclude =================================================================== --- /trunk/Ohana/doc/exclude (revision 2508) +++ /trunk/Ohana/doc/exclude (revision 2508) @@ -0,0 +1,4 @@ +find ohana -name "*.o" -print > exclude.files +find ohana -name "bin" -print >> exclude.files +find ohana -name "*.a" -print >> exclude.files +tar cvfX - exclude.files ohana | gzip > ohana.tgz.8.97 Index: /trunk/Ohana/doc/exclude.list =================================================================== --- /trunk/Ohana/doc/exclude.list (revision 2508) +++ /trunk/Ohana/doc/exclude.list (revision 2508) @@ -0,0 +1,19 @@ +Configure +bin +*.sol* +*.sid* +*.linux* +*.lin64* +*.o +*.a +*.so +*~ +#* +.mana +.status +.dvo +RCS +lib/sol +lib/sid +lib/linux +lib/lin64 Index: /trunk/Ohana/doc/gnu.license =================================================================== --- /trunk/Ohana/doc/gnu.license (revision 2508) +++ /trunk/Ohana/doc/gnu.license (revision 2508) @@ -0,0 +1,344 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 675 Mass Ave, Cambridge, MA 02139, USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + + Appendix: How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) 19yy + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) 19yy name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. Index: /trunk/Ohana/doc/graphics.info =================================================================== --- /trunk/Ohana/doc/graphics.info (revision 2508) +++ /trunk/Ohana/doc/graphics.info (revision 2508) @@ -0,0 +1,1595 @@ +This is Info file graphics.info, produced by Makeinfo-1.55 from the +input file graphics.texi. + + This file documents GNU graphics. + + Copyright (C) 1989 Free Software Foundation, Inc. + + +File: graphics.info, Node: Top, Up: (dir) + +* Menu: + +* introduction:: About the GNU graphics utilities +* graph examples:: Using Graph to create plot files from data +* graph invocation:: Command line options for Graph. +* plot2ps examples:: converting plot files to PostScript +* plot2ps invocation:: Plot2ps Command Line Options +* xplot:: a plot file previewer for X11 +* xplot invocation:: xplot command line options and X11 resources +* plot2fig examples:: converting plot files to Fig graphics editor format +* plot2fig invocation:: command line options for plot2fig +* plot2tek invocation:: Tektronix 4010 output +* latex example:: Including a Figure in an Article +* psfig:: Options to `psfig' for Including Figures +* atend.pl:: A perl script for use with psfig +* dvips:: How to Get `dvips' +* idraw:: How to Get `idraw' +* ghostscript:: an X11 PostScript interpreter/previewer +* libps:: The Library of Plot Functions for PostScript Output +* files:: The Plot File Format +* Acknowledgements:: The Contributors +* Function Index:: An Index to the libps Functions + + +File: graphics.info, Node: introduction, Next: graph examples, Up: Top + +The GNU Graphics Utilities +************************** + + The GNU graphics utilities are a set of programs for plotting +scientific data. The program `graph' reads data files and writes a +stream of plotting commands in a device independent format refered to +below as a GNU plot file. The remaining programs provide support for +displaying GNU plot files on tektronix 4010, PostScript (TM)*, and X +window system compatible output devices. + + `graph' reads both ascii and binary data files and writes a plot +file with or without axes and labels. You can specify labels and ranges +for the axes, and you can set the sizes and position of the plot on the +page. Each invocation of graph produces a plot with single set of axes +and data. You can place an arbitrary number of plots on the page by +concatenating the plot output of several invocations. + + `plot2ps' is a utility for converting plot files into PostScript. +The `plot2ps' utility reads plotting commands from named files or the +standard input and writes PostScript to the standard output. You can +then print the PostScript output on a printer, or edit it using the +`idraw' graphics editor. You can also easily include the output in +LaTeX documents using `dvips' and the LaTeX command `psfig'. + + Why is this useful? The plot file format is a common standard on un*x +systems. To produces figures for publication, you might need to take +data sets, and produce labeled figures from them. This can be done +using `graph', `plot2ps' and the `idraw' editor. You can also include +these figures in LaTeX documents using the `dvips' utility. All of +these utilities, as well as the plot file format and plot library, are +discussed in the following sections. + + This documentation is under revision. Any comments, suggestions, or +additions would greatly benifit GNU users. Please mail them to either +`bug-gnu-utils@prep.ai.mit.edu' or `Rich@Rice.edu'. + + * PostScript is a trademark of Adobe Systems Incorporated. + + +File: graphics.info, Node: graph examples, Next: graph invocation, Prev: introduction, Up: Top + +`graph' Examples +================ + + Each invocation of `graph' plots data read from stdin or named files +togeather in a single plot with or without axes and labels. The +following sections show common usage of `graph'. + +* Menu: + +* Invocation:: Simple examples using graph +* Input Files:: What the input looks like +* Overlaying:: Combining several data sets in one plot +* Multiple Plots:: Multiple plots on one page +* Binary Data:: Reading other data formats + + +File: graphics.info, Node: Invocation, Next: Input Files, Up: graph examples + +Simple examples using graph +=========================== + + By default, `graph' reads ascii data from the standard input or +files specified on the command line. `graph' reads pairs of values, x +and y coordinates: + + 0.0 0.0 + 1.0 0.2 + 2.0 0.0 + 3.0 0.4 + 4.0 0.2 + 5.0 0.6 + + To plot this data, you might use + + graph < ascii_data_file |plot + + where ascii_data_file could contain data similar to the above +example. You can replace the command `plot' with `plot2tek' if you have +a tektronix 4010 compatible graphics terminal, `plot2ps' if you have a +postscript compatible printer or previewer, or `xplot' if you have an X +window system display. + + Note that `graph' is commonly supplied with some operating systems. +If so, some confusion may arise if the system supplied version is +executed mistakenly. On unix systems, you can determine which version +you invoke by typing the command `which graph', which prints the file +name of the version you invoke by default. + + To reduce the change the size of the plot and position it in the +middle of the display, you could use + + graph -h .4 -w .4 -r .2 -u .2 < ascii_data_file |plot + + where `h' and `w' are the height and width of the plot and `r' and +`u' indicate how far up and to the right the plot is positioned. + + You can put symbols at each data point using + + graph -S 2 .01 < ascii_data_file |plot + + where 2 indicates which symbol to plot, and .01 indicates it's size. + + You can choose the type of line draw on each curve: + + graph -m 2 < ascii_data_file |plot + + where 2 indicates what kind of line connects the data points. + + +File: graphics.info, Node: Input Files, Next: Overlaying, Prev: Invocation, Up: graph examples + +The format of input to `graph' +============================== + + As mentioned above, by default `graph' reads ascii pairs of values, +x and y coordinates, from the standard input or files specified on the +command line. Optional labels may be placed after each coordinate in +the data file. The label ends at the end of the line: + + 3.0 0.4 this is a label for point (3.0, 0.4). + + The label must be enclosed in double quotes if it would otherwise be +appear to be a coordinate: + + 3.0 0.4 "5.0 looks like a value." + + You can use the `-b' to break lines after each label in the input. +Use a pair of empty double quotes if you need to break a line, but do +not need a label. + + 0.0 0.0 + 2.0 0.0 + 1.0 0.2 + "" + 0.0 0.1 + 2.0 0.2 + 4.0 0.3 + + You can also break cuves using `-M' option to break lines whenever +the abscissal values between successive pairs of points decrease. When +using `-M', each continuous curve has monotonicly increasing abscissal +values. + + 0.0 0.0 first data set + 2.0 0.0 + 4.0 0.2 + 0.0 0.1 second data set + 2.0 0.2 + 4.0 0.3 + + `graph' will automaticly generate abscissal values for you if you +specify the `-a' option. Only ordinate values are given in the data, +and the data is then assumed to be equally sampled along the abscissa. +The values following `-a' on the command line specify the sampling +interval and the abscissal value of the first data point. + + 0.0 + 0.1 + 0.2 label for point (2.0, 0.2) + 0.3 + 0.2 + 0.3 + + +File: graphics.info, Node: Overlaying, Next: Multiple Plots, Prev: Input Files, Up: graph examples + +Combining several data sets in one plot +======================================= + + There are cases where you will want to superimpose several data sets +or several plots on top of each other. If for example, the data sets +are in seperate files, you can specify each by it's name on the command +line. Since `graph' reads the standard input only if no files are +named on the command line, you must add the name `--' if you want +`graph' to read the standard input as well. + + graph data-file-one data-file-two data-file-three |plot + + For comparison sake, you might wish to distinguish the data in one +set from another either by using different symbols at each point or by +distinguishing the type of line draw. You can do this by preceeding +each file name with options affecting the synbol or line style used to +plot that data. + + graph -S 1 data-file-one -S 3 data-file-two -S 4 data-file-three |plot + + or + + graph -m 1 data-file-one -m 3 data-file-two -S 4 data-file-three |plot + + If you need to superimpose several data sets, but must invoke graph +seperately for each, you will have to specify the limits of the axes. + + graph -x 0 100 -y -3 3 -S 3 -m -1 < ascii_data_file_1 >> plot_file + + where `-x 0 100' specifies the limits on the x axis, `-y -3 3' +specifies the limits on the y axis, `-S 2' specifies a box to be drawn +at each point, and `-m -1' specifies that no line is to be draw +connecting the points. You can overlay a second data set on the first +by using: + + graph -s -g 0 -x 0 100 -y -3 3 -m 0 < ascii_data_file_1 >> plot_file + + where `-s' avoids erasing the page, `-g 0' avoids drawing the axis, +tick marks and labels which were drawn previously, and `-m 0' specifies +that solid lines are draw connecting the points. + + +File: graphics.info, Node: Multiple Plots, Next: Binary Data, Prev: Overlaying, Up: graph examples + +How to put multiple plots on one page +===================================== + + The command + graph -h .4 -w .4 -r .1 -u .1 < ascii_data_file_1 > plot_file + + will put a single box containing the plot in the lower left hand +quarter of the page. You can add another plot to the upper left hand +corner of the page using the command + + graph -s -h .4 -w .4 -r .1 -u .6 < ascii_data_file_2 >> plot_file + + Be sure you use the `-s' option so the the first plot isn't erased. + + Likewise you can add plots to the right hand side of the page using + + graph -s -h .4 -w .4 -r .6 -u .1 < ascii_data_file_3 >> plot_file + graph -s -h .4 -w .4 -r .6 -u .6 < ascii_data_file_4 >> plot_file + + The tick marks can be moved inside the box and labels moved to the +opposite sides using + + graph -T -.005 < ascii_data_file >> plot_file + + +File: graphics.info, Node: Binary Data, Prev: Multiple Plots, Up: graph examples + +Reading other data formats +========================== + + `graph' will read binary data in integer, short integer, float, and +double float format when you use the `-d' option followed by `f', or +`d', respectively. There are two advantage to using binary data: 1) +`graph' runs significantly faster because the computational overhead +for converting data from ascii to binary is eliminated, and 2) the +input files can be significantly smaller than the ascii format would +be. Double float is the fastest format to read, while short integer is +the most space conservative. If you have very large data sets, using a +binary format can reduce storage and runtime costs. + + For example, you can create double float data as output from C +language programs: + + #include + void write_point (x, y) + double x, y; + { + fwrite(&x, sizeof (double), 1, stdout); + fwrite(&y, sizeof (double), 1, stdout); + } + + You can then plot data written this way using: + + graph -d d plotfile + + +File: graphics.info, Node: graph invocation, Next: plot2ps examples, Prev: graph examples, Up: Top + +Graph Command Line Options +========================== + + The following table describes each of the command line arguments to +graph. Each option which takes an argument is followed by the type and +default values of the argument in parentheses. + +`-a [STEP_SIZE [LOWER_LIMIT]]' +`+auto-abscissa [STEP_SIZE [LOWER_LIMIT]]' + (floats, defaults 1 and 0) Automaticly generate abscissa (x) + values. This option specifies that the data contains only + ordinate (y) values. The difference between successive x values + will be STEP_SIZE, and the first x value will be LOWER_LIMIT. To + return to reading abscissal values from the input you can specify + `-a 0', which disables automatic generation of the abscissa and + returns STEP_SIZE and LOWER_LIMIT to their default values. + +`-b' +`+break-on-labels' + Assumes multiple data sets are in the data file, and the data sets + are separated by a label. The default is don't break on labels. + +`-M' +`+break-non-monotone' + When successive abscissa (x) values decrease, a separate data set + is assumed. This allows multiple data sets in each file. Similar + to `-b'. + +`-f SIZE' +`+fontsize SIZE' + Specify the size of the desired font as SIZE points. Not all + display devices will honor this command. + +`-N X|Y' +`+no-label X|Y' + By default, values at each tick mark are labeled beside the axis. + This option removes the labeling of the tick marks on the specified + axis. + +`-R' +`+dont-round-to-next-tick' + By default, the ends of the axes are extended to the next tick + mark. This option prohibits rounding the limits of the axes to + the next tick mark. + +`-c STRING' +`+point-label STRING' + This option defines STRING as the default label for each point. + Any label in the input will override this default. + +`-S SYMBOL_NUMBER [SYMBOL_SIZE]' +`+symbol SYMBOL_NUMBER [SYMBOL_SIZE]' + (integer and float, defaults -1 and 0.01) Draw a symbol at each + point in the data. SYMBOL_NUMBER specifies the shape of the symbol + according to the following table and SYMBOL_SIZE specifies the + fractional size of the symbol with respect to the height and width + of the plot. Note that you can specify symbols to be drawn + without any line connecting them by specifying the option `-m -1'. + -1 no symbol at all + 0 plus sign (+) + 1 cross (x) + 2 star (*) + 3 box + 4 diamond + 5 circle +`-T TICK_SIZE' +`+ticksize TICK_SIZE' + (float, default .01) TICK_SIZE is the fractional size of the tick + marks on each axis. A value of 1.0 produces tick marks on the x + (y) axis whose length is equal to the width (height) of the plot. + +`-X X_LABEL' +`+xtitle X_LABEL' + (string, default blank) X_LABEL is a label printed below the x + axis. + +`-Y Y_LABEL' +`+ytitle Y_LABEL' + (string, default blank) Y_LABEL is a label printed to the right of + the y axis. + +`-d DATA-FORMAT' +`+data-format DATA-FORMAT' + This specifies what format the input data is in. Note labels can + be used only in ascii format input files. + `a' + `A' + ascii data + + `i' + `I' + binary integer data + + `s' + `S' + binary short integer data + + `f' + `F' + binary float data + + `d' + `D' + binary double data + +`+debug' + Debugging information, including the data read in, is sent to the + standard error output. + +`-g GRID_STYLE' +`+grid GRID_STYLE' + (integer, default 1) GRID_STYLE specifies the type of box framing + the plot and whether grid lines are drawn inside the box. + 0 no box around plot, no axes, no labels. + 1 box containing a grid and axes with tick marks and labels. + 2 box around plot, tick marks around the box and labels. + 3 box around plot, ticks on left and lower sides only and labels. + 4 axes intersect at the origin without a box or grid. +`-h HEIGHT' +`+height-plot HEIGHT' + (float, default 0.8) HEIGHT specifies the fractional height of the + plot with respect to the height of the plotting area. A value of + 1.0 will produce a box which fills the available area. Note that + the tick marks and labels are outside this area so that values + less than 1.0 are generally used. + +`-L TOP_LABEL' +`+toptitle TOP_LABEL' + (string, default blank) TOP_LABEL is a label placed above the plot. + +`-m LINE_MODE' +`+linestyle LINE_MODE' + (integer, default 0) LINE_MODE specifies the mode (or style) of + lines drawn between data points. + -1 no line at all + 0 solid + 1 dotted + 2 shortdashed + 3 dotdashed + 4 longdashed + 5 disconnected +`-r RIGHT' +`+right-margin-posn RIGHT' + (float, default 0.1) Move the plot to the right by a fractional + amount RIGHT with respect to the width of the plotting area. This + produces a margin on the left hand side of the plot. A value of + 0.5 will produce a margin half the width of the available area. + Note that the tick marks and labels are drawn in the margin. + +`-u UP' +`+bottom-margin-posn UP' + (float, default 0.1) Move the plot up by a fractional amount UP + with respect to the height of the plotting area. This produces a + margin below the plot. A value of 0.5 will produce a margin half + the height of the available area. Note that the tick marks and + labels are drawn in the margin. + +`-s' +`+save-screen' + Save the screen. This option prevent graph from erasing the + previous contents of the graphics window or device. + +`-t' +`+transpose' + Transpose the abscissa and ordinate. This option causes the axes + to be interchanged, and the options which apply to each axis to be + applied to the opposite axis. That is, data is read in as (y, x) + pairs and `-x', `-X' and `-lx' apply to the y axis. + +`-w WIDTH' +`+width-plot WIDTH' + (float, default 0.8) WIDTH specifies the fractional width of the + plot with respect to the width of the plotting area. A value of + 1.0 will produce a box which fills the available area. Note that + the tick marks and labels are outside this area, so values less + than 1.0 are generally used. + +`-x LOWER_LIMIT UPPER_LIMIT' +`+xlimits LOWER_LIMIT UPPER_LIMIT' + (floats) The arguments LOWER_LIMIT and UPPER_LIMIT specify the + limits of the x axis. By default the upper and lower limits are + taken from the data. If unspecified the limits of the data are + used. + +`-y LOWER_LIMIT UPPER_LIMIT' +`+ylimits LOWER_LIMIT UPPER_LIMIT' + These arguments specify the scale and limits of the y axis as for + the x axis above. + +`-l X|Y' +`+log-axis X|Y' + The argument indicates which axis should be a log axis. Either + one or both x- and y-axes can be specified by using the + appropriate letter. Use XY or YX to specify both. + +`+high-byte-first' +`+low-byte-first' + These options force graph to use the specified byte order when + writing out the plot file. By default the byte order is host + dependent. + + +File: graphics.info, Node: plot2ps examples, Next: plot2ps invocation, Prev: graph invocation, Up: Top + +Examples Using `plot2ps' +======================== + + To produce a plot of data arranged in ordered pairs of x and y +coordinates in an ASCII file, you can use: + + graph test.ps + + To edit the plot: + + idraw test.ps + + To use the previewer to look at the plot: + + gs test.ps + + +File: graphics.info, Node: plot2ps invocation, Next: xplot, Prev: plot2ps examples, Up: Top + +Plot2ps Command Line Options +============================ + + `plot2ps' is a relatively simple utility in that there are few +command line options to choose from. The plot file format does not +contain methods for specifying font or font size, so you must specify +these things with options. There are no other options for controlling +the picture. + + The plot file format is machine dependent on the byte order of +unformatted, signed, two byte integer coordinates contained in plot +commands. The `-high-byte-first' or `-low-byte-first' option specifies +this order explicitly. `plot2ps' attempts to determine the byte order +from commands early in the plot file, but the method is heuristic and +is not foolproof. Several standard plot sizes specified by the `open' +command are used to recognize byte order by `plot2ps'. If these sizes +are recognized in byte reversed order, `plot2ps' adjusts accordingly. +These sizes include 504x504, 2048x2048 (versatek plotters), 2100x2100, +3120x3120(tektronix 4010 terminals) and 4096x4096 (gsi 300 terminals). + + The remaining command line options may be used specify an alternate +PostScript prologue and to print the licensing information. + + Input plot files names may be specified anywhere on the command line. +If no file names are specified, or the name `-' is specified, the +standard input is read for plotting instructions. Only the font or +font size options which precede a file name will affect the text for +that file. +`-H' +`+help' + The help option prints a summary of command line syntax for + `plot2ps', a list of the font names (the standard builtin + PostScript fonts), and version, copyright and warranty information. + Specifying this options causes plot2ps to ignore files on the + standard input. You can specify a file on the standard input + explicitly with the option `-' if you want it to read the standard + input as well. + +`-v' +`-V' +`+version' + This option prints version, copyright and warranty information. + +`-fontsize SIZE' +`-f SIZE' + The fontsize options specifies the default size in printer's points + (1/72 inch) of all text appearing in the plot. If unspecified, the + size defaults to 14 points. + + Some sizes are supported better than others under X windows. The + standard sizes distributed with X windows are 8, 10, 12, 14, 18, + and 24 points. Text at these point sizes will display correctly + in the `idraw' editor. Other font sizes will print correctly on a + PostScript device such as the laserwriter, but may not appear at + the correct size in the `idraw' editor. + +`-font NAME' +`-fo NAME' + The font name option specifies the name of the default font for all + text appearing in the plot. `plot2ps -help' prints a listing of + the font names on the standard output. These names include the + available builtin fonts on standard PostScript printers. + +`-high-byte-first' +`-h' + The high-byte-first option specifies explicitly that the higher + order byte of each signed, two byte integer occurs first in the + file. It disables determination of byte order from the file + itself. + +`-line-width WIDTH' + WIDTH is the width of lines drawn in the plot, and defaults to a + value of 0. A value of 0 will produce the thinest line possible + in a device dependent fashion, however this is known to cause + problems for older versions of idraw. The line width is device + independent for a positive values of WIDTH. + +`-low-byte-first' +`-l' + The low-byte-first option specifies explicitly that the lower order + byte of each signed, two byte integer occurs first in the file. It + disables determination of byte order from the file itself. + +`-prologue FILENAME' +`-p FILENAME' + The prologue option specifies the name of an alternate PostScript + prologue FILENAME to be used in place of the default `idraw' + prologue. The prologue declares procedures used to draw each + graphic object. The default prologue was generously provided by + John Interante and is a part of the InterViews distribution, + version 2.5. + +`-copying' +`-warranty' + The copying and warranty options print a copy of the GNU General + Public License on the standard error output. Included is + conditions for copying `plot2ps' and information on the lack of any + warranty. + + These conditions do not cover the output of `plot2ps'. The only + conditions imposed on the output are those which come from the + prologue that you are using. + +`-signed' +`-unsigned' + The signed and unsigned options specify whether coordinates in the + plot file are signed. By convention, coordinates are always + signed. Some plot files do not follow this convention, and you + can use the unsigned option to convert those files. + +`-bbox' + The bbox option specifies that a bounding box comment will be + written at the end of the output file. This information is useful + for document preparation systems which determine how to size and + place the figure using the bounding box. See also the atend + script. + + +File: graphics.info, Node: xplot, Next: xplot invocation, Prev: plot2ps invocation, Up: Top + +A plot file previewer for X11 +============================= + + `xplot' is a plot file previewer for the X window system. It reads +GNU plot commands from its standard input and draws the resulting +graphics in an X window. + + After xplot reaches the end-of-file on the input, it puts itself in +the background (forks). Control returns to the calling program, while +xplot continues, remaining on screen. + + To exit, click the left mouse button in the xplot window. Note that +xplot ignores SIGHUP signals, so you must use another signal to kill +xplot if necessary. + + +File: graphics.info, Node: xplot invocation, Next: Plot2fig Examples, Prev: xplot, Up: Top + +Xplot Options +============= + + `xplot' accepts all of the standard X toolkit command line options, +and the initial geometry specification determines the resolution, with a +default geometry of 500x500 pixels. + + The following standard X Toolkit command line arguments may be used +with `xplot': +`-bg COLOR' + This specifies the color to use for the background of the window. + The default is WHITE. + +`-bd COLOR' + This specifies the color to use for the border of the window. The + default is BLACK. + +`-bw NUMBER' + This specifies the width in pixels of the border surrounding the + window. + +`-fg COLOR' + This specifies the color to use for displaying text. The default + is BLACK. + +`-fn FONT' + This specifies the font to be used for displaying normal text. The + default is 6X10. + +`-rv' + This indicates that reverse video should be simulated by swapping + the foreground and background colors. + +`-geometry GEOMETRY' + This specifies the preferred size and position of the plot window. + +`-display HOST:DISPLAY' + This specifies the X server to contact. + +`-xrm RESOURCESTRING' + This specifies a toolkit resource property. See the manual page + for xrdb. + + `xplot' uses the athena Command widget in the X Toolkit. So, it +understands all of the core resource names and classes as well as: +`reverseVideo' + (class ReverseVideo) Specifies that the foreground and background + colors should be reversed. + +Example +------- + + The resources: + + Xplot.font: 6x9 + Xplot.geometry: 300x300 + + will set the font used in the plot window to `6x9' and the size of +the window to 300 by 300 pixels. + + +File: graphics.info, Node: Plot2fig Examples, Next: plot2fig invocation, Prev: xplot invocation, Up: Top + +Using Plot2fig +============== + + To create a simple plot file one can use: + + echo 0 0 1 1 2 0 | spline | graph | plot2fig > test.fig + + To edit the plot: + + fig test.fig + + To convert the fig file into dvi code, create a latex file containing +a document which includes the figure: + + \\documentstyle\[\]{article} + \\begin{document} + \\input{test} + \\end{document} + + Then, run transfig on the figure and latex on the document: + + % transfig -L latex test.fig + % make test.tex + % latex t.tex + + To edit a plot of data arranged in ordered pairs of x and y +coordinates in an ascii file, one can use: + + % graph file.fig + % fig file.fig + + +File: graphics.info, Node: plot2fig invocation, Next: plot2tek invocation, Prev: plot2fig examples, Up: Top + +A plot file to fig file translator +================================== + + `plot2fig' reads plotting instructions from the specified input +files and/or the standard input and produces `Fig' compatible code on +it's standard output. This output file can be edited with the fig +(Facility for Interactive Generation of figures) graphics editor. The +output can subsequently be converted to pictex, PostScript, latex, epic, +eepic, and tpic languages using the `transfig' translator. + + Any unrecognized options on the command line are assumed to be input +files. The standard input is read by default only if no other files +specified on the command line are successfully opened. A single dash +(-) on the command line indicates the standard input is to be read. +Each option is set and each file read in the order they are specified on +the command line. + + For compatibility with pic2fig, plot2fig ignores leading white space +in labels. Labels containing all white space are ignored. + +`-fn NAME' +`-fontname NAME' + Default: the default font of the transfig output device. This + option sets the font for all subsequent text to NAME. Recognized + font names are typewriter, modern, italic, bold, and times. In + addition, courier is an alias for typewriter and roman is an alias + for times. Note that the fonts are device dependent. + +`-fs SIZE' +`-fontsize SIZE' + Default: 12. This option sets the size of subsequent text to SIZE + (in printer's points). + +`-h' +`-high-byte-first' + This option specifies that the byte ordering of two byte integers + in the input plot file is high byte first. + +`-l' +`-low-byte-first' + This option specifies that the byte ordering of two byte integers + in the input plot file is low byte first. + +`-warranty' +`-copying' + This option prints out the copying conditions and warranty + information. + +`-signed' +`-unsigned' + Default: signed. This option specifies whether two byte integers + in the input plot file are unsigned or signed. + +`-' + This option specifies explicitly that the standard input should be + read for plotting instructions. + + +File: graphics.info, Node: plot2tek invocation, Next: latex example, Prev: plot2fig invocation, Up: Top + +Tektronix 4010 output +===================== + +`-H' +`+help' + The help option prints a summary of command line syntax, a list of + the known font names, and version, copyright and warranty + information. Specifying this options causes plot to ignore the + standard input, so you must specify the option `-' if you want it + to read the standard input as well. + +`-v' +`-V' +`+version' + This option prints version, copyright and warranty information. + +`-f SIZE' +`+fontsize SIZE' + The fontsize options specifies the default size in printer's points + (1/72 inch) of all text appearing in the plot. If unspecified, the + size defaults to 14 points. + +`+high-byte-first' +`+low-byte-first' + These options force graph to use the specified byte order when + writing out the plot file. By default the byte order is host + dependent. + +`-F NAME' +`+fontname NAME' + The font name option specifies the name of the default font for all + text appearing in the plot. `plot -help' prints a listing of the + font names on the standard output. These names include the + available builtin fonts on standard PostScript printers. + +`-h' +`+high-byte-first' + The high-byte-first option specifies explicitly that the higher + order byte of each signed, two byte integer occurs first in the + file. It disables determination of byte order from the file + itself. + +`-l' +`+low-byte-first' + The low-byte-first option specifies explicitly that the lower order + byte of each signed, two byte integer occurs first in the file. It + disables determination of byte order from the file itself. + +`-p FILENAME' +`+prologue FILENAME' + The prologue option specifies the name of an alternate PostScript + prologue FILENAME to be used in place of the default `idraw' + prologue. The prologue declares procedures used to draw each + graphic object. The default prologue was generously provided by + John Interante and is a part of the InterViews distribution, + version 2.5. + +`+copying' +`+C' +`+warranty' +`+W' + The copying and warranty options print a copy of the GNU General + Public License on the standard error output. Included is + conditions for copying `plot' and information on the lack of any + warranty. + +`-s' +`+signed-input' +`-u' +`+unsigned-input' + The signed and unsigned options specify whether coordinates in the + plot file are signed. By convention, coordinates are always + signed. Some plot files do not follow this convention, and you + can use the unsigned option to convert those files. + + +File: graphics.info, Node: latex example, Next: psfig, Prev: plot2tek invocation, Up: Top + +Including a Figure in an Article +================================ + + This is an example of LaTeX code which places the figure generated in +the previous example in a page of text. + + \documentstyle[]{article} + \input{psfig} + \begin{document} + \title{Title of the article.} + \author{The Author's name} + \maketitle + This is an example of how to include PostScript figures in LaTeX documents. + \begin{figure}[h] + \centerline{\psfig{figure=test.ps,height=3in}} + \caption{Here is a description of the figure which will appear below it.} + \end{figure} + Note that the above figure was included using dvips. + \end{document} + + If the above LaTeX code is contained in a file called `mytext.tex' +you can use the commands + + latex mytext + dvips mytext.dvi >mytext.ps + lpr -Plw mytext.ps + + to format and print the example text. + + +File: graphics.info, Node: psfig, Next: atend.pl, Prev: latex example, Up: Top + +Options to `psfig' for Including Figures +======================================== + + `psfig' is a LaTeX command used to insert a PostScript figure into a +document. + + `psfig' can be used to insert `plot2ps' generated PostScript into a +LaTeX document. The placement of the `psfig' command tells LaTeX where +in the document to place the PostScript, and arguments to the command +give the name of the file containing the PostScript, and the desired +size of the figure. Arguments are separated by commas or blanks, and +are of the form `KEYWORD=VALUE'. The following is a list of valid +arguments for the `psfig' command: + +`file=NAME' + The file name of the PostScript figure. + +`height=SIZE' + The height of the figure (eg. 3in). If you specify only a height + or only a width, the width and height are scaled equally. If you + specify both a width and a height the aspect ratio will be + affected. + +`width=SIZE' + The width of the figure (eg. 3in). + +`bbllx=COORDINATE' + The bounding box lower left-hand x coordinate. Any PostScript file + which conforms to the PostScript Document Structuring Conventions + version 2.0 should contain a bounding box information at the head + of the file. `plot2ps' output conforms to the version 2.0 + conventions so that you should not need to use any of the bounding + box options. + +`bblly=COORDINATE' + The bounding box lower left-hand y coordinate. + +`bburx=COORDINATE' + The bounding box upper right-hand x coordinate. + +`bbury=COORDINATE' + The bounding box upper right-hand y coordinate. + +`rheight=SIZE' + Horizontal space to reserve for the figure. + +`rwidth=SIZE' + Vertical space to reserve for the figure. + +`clip=' + Clip the figure. `clip=' is a switch and takes no value, but the + `=' must be present. This option is useful for including + PostScript figures which use the size of the clipping path to size + themselves. + + +File: graphics.info, Node: atend.pl, Next: dvips, Prev: psfig, Up: Top + +A perl script for use with psfig +================================ + + `atend.pl' is a perl script which moves the bounding box comment +from the trailer to the header. Although either is legal, most document +preparation software, such as `psfig', will only accept bounding box +comments in the header. If you use `psfig' and the `-bbox' option +togeather, run `atend.pl' on the output of `plot2ps' before importing +the graphics using `psfig'. + + `atend.pl' can be used as a filter: + + echo 0 0 1 1 2 0 | spline | graph >spline.pl + plot2ps -bbox |atend.pl - >spline.ps + + Or, `atend.pl' can be used to fix the output file in place: + + echo 0 0 1 1 2 0 | spline | graph |plot2ps -bbox >spline.ps + atend.pl spline.ps + + +File: graphics.info, Node: dvips, Next: idraw, Prev: atend.pl, Up: Top + +How to Get `dvips' +================== + + The `dvips' utility mentioned previously is used convert dvi files +generated by LaTeX into post-script. It also has support for inclusion +of PostScript figures into LaTeX documents. It is available via +anonymous ftp from `labrea.stanford.edu' (36.8.0.47). Look for +`pub/dvips.tar.Z'. + + +File: graphics.info, Node: idraw, Next: ghostscript, Prev: dvips, Up: Top + +How to Get `idraw' +================== + + The `idraw' utility mentioned previously is an interactive graphics +editor which is distributed with InterViews. InterViews is available +via anonymous ftp from `interviews.stanford.edu' (36.22.0.175) in the +file `InterViews/2.5.tar.Z'. + + +File: graphics.info, Node: ghostscript, Next: libps, Prev: idraw, Up: Top + +How to Get `ghostscript' +======================== + + Ghostscript is a previewer which is intended to be compatible with +the PostScript language. It supports several output devices including +the X window system and ega displays. Version 2.0 is suitable for +previewing LaTeX documents with imbedded encapsulated PostScript +figures, the type of PostScript figures generated by plot2ps and idraw. +It is available via anonymous ftp from `prep.ai.mit.edu' (18.71.0.38) +- Look for `pub/gnu/ghostscript.tar.Z'. + + +File: graphics.info, Node: libps, Next: files, Prev: ghostscript, Up: Top + +`libps', a Library of Plot Functions +==================================== + + Libps is a library of plot functions for drawing graphic object +using PostScript. Before drawing any objects or using any of the other +functions, a program should call `openpl'. Before exiting and after +all other libps calls a program should call `closepl'. + +* Menu: + +The standard plot library includes: + +* arc:: draw an arc +* circle:: draw a circle +* closepl:: close the device for output +* cont:: continue a line +* erase:: erase the page +* label:: print a label +* line:: draw a line +* linemod:: change the line mode (style) +* move:: move to a new coordinate +* openpl:: open the device for output +* point:: draw a point +* space:: define the user's plot space (size) + +The extensions provided only in libps to take advantage of PostScript +features include: + +* alabel:: print a vertically or horizontally justified label +* color:: change the color +* fill:: fill pattern for closed paths +* fontname:: set the font name +* fontsize:: set the font size +* rotate:: rotate subsequent text + + +File: graphics.info, Node: alabel, Next: arc, Up: libps + +`alabel' +-------- + + int `alabel' (char X_JUSTIFY, char Y_JUSTIFY, char *LABEL); + + `alabel' takes three arguments X_JUSTIFY, Y_JUSTIFY, and LABEL and +places the label according to the x and y axis adjustments specified in +X_JUSTIFY and Y_JUSTIFY respectively. X_JUSTIFY is a character +containing either `l', `c', or `r' for left, center or right justified +with respect to the current x coordinate. Y_JUSTIFY is a character +containing either `b', `c', or `t' for placing the bottom center or top +of the label even with the current y coordinate. *LABEL is a string +containing the label. The current point is moved to follow the end of +the text. + + *Note fontname:: on how to change the default font. *Note +fontsize:: on how to change the font size. + + +File: graphics.info, Node: arc, Next: circle, Prev: alabel, Up: libps + +`arc' +----- + + int `arc' (int X, int Y, int X0, int Y0, int X1, int Y1) + + `arc' takes six integer arguments specifying the coordinates of the +center (X, Y), beginning (X0, Y0), and ending (X1, Y1) of a circular +arc. The current point becomes (X, Y). + + +File: graphics.info, Node: circle, Next: closepl, Prev: arc, Up: libps + +`circle' +-------- + + int `circle' (int X, int Y, int R) + + `circle' takes three integer arguments specifying the center (X, Y) +of the circle and its radius (R). The current point becomes (X, Y). + + +File: graphics.info, Node: closepl, Next: color, Prev: circle, Up: libps + +`closepl' +--------- + + int `closepl' () + + `closepl' takes no arguments. It merely outputs the PostScript +trailer containing a `showpage' command. + + +File: graphics.info, Node: color, Next: cont, Prev: closepl, Up: libps + +`color' +------- + + int `color' (int RED, int GREEN, int BLUE); + + `color' sets the foreground color of all the following objects. The +arguments RED, GREEN and BLUE indicate the intensity of red, green and +blue components of the foreground color respectively. Each is a +unsigned integer specifying an intensity in the range from 0 to 0xFFFF. +A value of (0, 0, 0) represents black and a value of (0xFFFF, 0xFFFF, +0xFFFF) indicates white. + + +File: graphics.info, Node: cont, Next: erase, Prev: color, Up: libps + +`cont' +------ + + int `cont' (int X, int Y) + + `cont' takes two integer arguments specifying the coordinate (X, Y) +for the continuation of a line. This draws a line segment from the +current point to the point (X, Y). The current point then becomes (X, +Y). + + +File: graphics.info, Node: erase, Next: fill, Prev: cont, Up: libps + +`erase' +------- + + int `erase' () + + `erase' normally erases all the graphics from the display before a +plot is viewed. Since we start off with a blank page in PostScript and +`idraw' this function does nothing. + + +File: graphics.info, Node: fill, Next: fontname, Prev: erase, Up: libps + +`fill' +------ + + int `fill' (int LEVEL); + + `fill' sets the intensity of the filler for closed paths. The +argument LEVEL indicates the grey level of the fill pattern. It's +value ranges from 1 to 0xFFFF. A value of 1 represents black and a +value of 0xFFFF indicates white. A value of 0 represents no fill, or +transparent. + + +File: graphics.info, Node: fontname, Next: fontsize, Prev: fill, Up: libps + +`fontname' +---------- + + int `fontname' (char *FONT_NAME); + + `fontname' takes a single string argument, FONT_NAME, specifying the +name of the font to be used for following text. The laser writer +builtin fonts are supported: + + courier-bold + courier-boldoblique + courier-oblique + courier + helvetica-bold + helvetica-boldoblique + helvetica-oblique + helvetica + symbol + times-bold + times-bolditalic + times-italic + times-roman + + +File: graphics.info, Node: fontsize, Next: label, Prev: fontname, Up: libps + +`fontsize' +---------- + + int `fontsize' (int SIZE); + + `fontsize' takes a single integer argument SIZE in printer's points +(1/72 inch) and sets the font size accordingly. + + +File: graphics.info, Node: label, Next: line, Prev: fontsize, Up: libps + +`label' +------- + + int `label' (char *S) + + `label' takes a single string argument S and draws the text +contained in S at the most recently used coordinate in the current +font. By default the text is left justified and centered vertically +with respect to the current coordinate. + + +File: graphics.info, Node: line, Next: linemod, Prev: label, Up: libps + +`line' +------ + + int `line' (int X1, int y1, int X2, int Y2) + + `line' takes four integer arguments specifying the beginning (X1, +Y1) and ending (X2, Y2) points of a line. The current point becomes +(X2, Y2). + + *Note linemod:: for how to specify the style or pattern of line. + + +File: graphics.info, Node: linemod, Next: move, Prev: line, Up: libps + +`linemod' +--------- + + int `linemod' (char *S) + + `linemod' takes a single string argument S containing the name of +the line style desired. The names supported are longdashed, +disconnected, dotdashed, dotted, solid and shortdashed. These +correspond to the following sixteen bit patterns: + + solid -------------------------------- + longdashed ------- ------- + disconnected - - + dotdashed ----------- - ----------- - + dotted - - - - - - - - - - - - - - - - + shortdashed -- -- + + +File: graphics.info, Node: move, Next: openpl, Prev: linemod, Up: libps + +`move' +------ + + int `move' (int X, int Y) + + `move' takes two integer arguments specifying the coordinate (X, Y) +for the beginning of a new line. This is equivalent to lifting the pen +on a plotter and moving it to a new position without drawing any line. +The current point becomes (X, Y). + + +File: graphics.info, Node: openpl, Next: point, Prev: move, Up: libps + +`openpl' +-------- + + int `openpl' () + + `openpl' normally opens the device. For PostScript we just print +out the PostScript prologue. The following global variables defined in +`openpl' specify what prologue is written to the output. + + USER_HAS_PROLOGUE is a flag. If it is non-zero then the open +routine should output the user specified prologue contained in the file +specified in the string USERS_PROLOGUE. + + USERS_PROLOGUE is a string containing the file name for any user +specified PostScript prologue. This file is a substitute for the +default prologue. + + +File: graphics.info, Node: point, Next: rotate, Prev: openpl, Up: libps + +`point' +------- + + int `point' (int X, int Y) + + `point' takes a pair of integer arguments specifying the coordinate +(X, Y) for a single point. The current point then becomes (X, Y). + + +File: graphics.info, Node: rotate, Next: space, Prev: point, Up: libps + +`rotate' +-------- + + int `rotate' (int ANGLE); + + `rotate' takes three integer arguments. The last argument, ANGLE, +specifies the angle in degrees counter-clockwise from the x +(horizontal) axis following text. + + +File: graphics.info, Node: space, Prev: rotate, Up: libps + +`space' +------- + + int `space' (int X0, int Y0, int X1, int Y1) + + `space' takes two pair of integers arguments specifying the lower, +left-hand and upper, right-hand limits of the range of plot +coordinates. The scaling of input to output coordinate conversion is +adjusted to fit these ranges into the page. Note however that if the +ranges of x and y coordinates are different the smallest scaling of the +two is used to avoid affecting the aspect ratio of the plot. This +means that although the plot is scaled to fit on the page, the axes are +not stretched with respect to each other. + + +File: graphics.info, Node: files, Next: Acknowledgements, Prev: libps, Up: Top + +The Plot File Format +==================== + + The plot file is a set of plotting commands and data. Each command +is a single ascii character indicating which operation is to be +performed. The data following a command is either a newline terminated +ascii string or several signed, two byte integers in binary format. +For example, the command to move the current point to the coordinate +(3,5) would be `m\000\003\000\005'. + + Note that the byte order of the binary representation of the signed, +two byte integers is machine dependent, so on some machines, this +command might appear as `m\003\000\005\000'. `plot2ps' tries to guess +the byte order from the arguments to the `openpl' command and adjust +the order accordingly. + + The following table lists each single character commands followed by +the name of the corresponding libps function called to handle the data +and a description of the command and data. + +`Command' + Description + +`a' + The arc command is followed by three pair of signed, two byte + integers indicating the center, starting and ending points for a + circular arc. The center becomes the the current point. This is + equivalent to the `arc' function (*note arc::.). + +`c' + The circle command is followed by three signed, two byte integers. + The first two indicate the x and y coordinates of the center of + the circle and the third indicates the radius of the circle. The + center becomes the the current point. This is equivalent to the + `circle' function (*note circle::.). + +`C' + The color command is followed by three unsigned, two byte integer + which indicate the intensity of RED, GREEN and BLUE components + respectively of the background color. For each component the + range of intensity is from 0 to 65535. A value of (0, 0, 0) + represents black and (65535, 65535, 65535) represents white. This + is equivalent to the `color' function (*note color::.). + +`e' + The erase command is followed by no data. The erase command is not + needed since in `idraw' and PostScript we start off with a blank + page. For this reason the erase command does not actually output + any PostScript. This is equivalent to the `erase' function (*note + erase::.). + +`f' + The linemod command is followed by a newline terminated string + containing the name of the line mode (or style) for all subsequent + lines, circles and arcs. This is equivalent to the `linemod' + function (*note linemod::.) which describes the line styles and + their names. + +`F' + The the fontname command is followed by a newline terminated string + containing the name of the font to be used for all subsequent + text. This is equivalent to the `fontname' function (*note + fontname::.). + +`l' + The line command is followed by two pair of signed, two byte + integers which indicate the starting and ending points of the + line. The second pair becomes the the current point. This is + equivalent to the `line' function (*note line::.). + +`L' + The fill command is followed by an unsigned, two byte integer + indicating the intensity of the fill for closed paths. A value of + 1 represents black and a value of 0xFFFF indicates white. The + value 0 is special in that is indicates that no solid fill should + occur, and that the interior of the respective path is + transparent. This is equivalent to the `fill' function (*note + fill::.). + +`m' + The move command is followed by a pair of signed, two byte integers + containing the location of the new current point. No line is + drawn to this point as opposed to the continue command (`c') which + draws a line. This is equivalent to the `move' function (*note + move::.). + +`n' + The continue command is followed by pair of signed, two byte + integers containing the coordinates of the endpoint of a line + segment. A line is drawn from the previous current point if it + was set using a command such as move or continue. This then + becomes the the current point. This is equivalent to the `cont' + function (*note cont::.). + +`p' + The point command is followed by pair of signed, two byte integers + containing the location of single point to be drawn. This then + becomes the the current point. This is equivalent to the `point' + function (*note point::.). + +`r' + The rotate command is followed by one signed, two byte integer. It + indicates the rotation of all subsequent text. The rotation is in + degrees counter-clockwise from the x (horizontal) axis. This is + equivalent to the `rotate' function (*note rotate::.). + +`s' + The space command is followed by two pair of signed, two byte + integers which indicate the the lower right-hand and upper + left-hand corners of the range of plot coordinate space. + `plot2ps' uses the third signed, two byte integer (the right-hand + limit) to try to determine the byte order. This is equivalent to + the `space' function (*note space::.) which describes the + recognized sizes. + +`S' + The fontsize command is followed by an signed, two byte integer + containing the size in printers points of all subsequent text. + This is equivalent to the `fontsize' function (*note fontsize::.). + +`t' + The label command is followed by a newline terminated string + contains a label which is printed at the current point. It is + left justified and centered vertically with respect to the current + point. The current point is then set at the end of the text. This + is equivalent to the `label' function (*note label::.). + +`T' + The adjusted label command is followed by two characters which + indicate the horizontal and vertical justification respectively + and a newline terminated string containing the label. The label + is drawn with the specified justification and the current point is + set at the end of the text. This is equivalent to the `alabel' + function (*note alabel::.) which describes how to specify + justification. + + +File: graphics.info, Node: Acknowledgements, Next: Function Index, Prev: files, Up: Top + +Acknowledgements +================ + + Rich Murphey wrote the first version of the graph, +plot2tek, plot2ps and tek2plot and the documentation. Richard Stallman + further directed development of the programs and +editorial support for the documentation. John Interrante generously +provided the PostScript prologue and helpful comments on the program. + + Arthur Smith (Lassp, Cornell University) + has generously provided his code for the +xplot utility. + + Ray Toy provided code for graph and tek 4011 +and rewrote the tick mark spacing code, incorporated gnu getopt and +provided the statistics package. + + David B. Rosen , jeffrey templon + and David W. Forslund + tested alpha versions. + + +File: graphics.info, Node: Function Index, Prev: Acknowledgements, Up: Top + +Function Index +============== + +* Menu: + +* alabel (char X_JUSTIFY, char Y_JUSTIFY, char *LABEL): alabel. +* arc (int X, int Y, int X0, int Y0, int X1, int Y1): arc. +* circle (int X, int Y, int R): circle. +* closepl (): closepl. +* color (int RED, int GREEN, int BLUE): color. +* cont (int X, int Y): cont. +* erase(): erase. +* fill (int LEVEL): fill. +* fontname (char *FONT_NAME): fontname. +* fontsize (int SIZE): fontsize. +* label (char *S): label. +* linemod (char *S): linemod. +* line (int X1, int y1, int X2, int Y2): line. +* move (int X, int Y): move. +* openpl(): openpl. +* point (int X, int Y): point. +* rotate (int W, int H, int ANGLE): rotate. +* space (int X0, int Y0, int X1, int Y1): space. + + + +Tag Table: +Node: Top187 +Node: introduction1407 +Node: graph examples3457 +Node: Invocation4024 +Node: Input Files5739 +Node: Overlaying7374 +Node: Multiple Plots9252 +Node: Binary Data10198 +Node: graph invocation11307 +Node: plot2ps examples18376 +Node: plot2ps invocation18914 +Node: xplot24137 +Node: xplot invocation24803 +Node: Plot2fig Examples26556 +Node: plot2fig invocation27374 +Node: plot2tek invocation29619 +Node: latex example32322 +Node: psfig33301 +Node: atend.pl35322 +Node: dvips36130 +Node: idraw36538 +Node: ghostscript36899 +Node: libps37487 +Node: alabel38741 +Node: arc39560 +Node: circle39892 +Node: closepl40169 +Node: color40400 +Node: cont40919 +Node: erase41255 +Node: fill41545 +Node: fontname41952 +Node: fontsize42615 +Node: label42872 +Node: line43234 +Node: linemod43593 +Node: move44259 +Node: openpl44632 +Node: point45278 +Node: rotate45544 +Node: space45836 +Node: files46490 +Node: Acknowledgements52660 +Node: Function Index53597 + +End Tag Table Index: /trunk/Ohana/doc/keywords.txt =================================================================== --- /trunk/Ohana/doc/keywords.txt (revision 2508) +++ /trunk/Ohana/doc/keywords.txt (revision 2508) @@ -0,0 +1,129 @@ + +here are keywords used in Elixir: + +Elixir defined keywords: +(detrend dB) +TVSTART +TVSTOP +FILTER +EXPTIME +OBSTYPE +NENTRY - values in dB + +(phot dB) +NIMAGES + +(imregister / iminfo) +EXPTIME-KEYWORD : ExptimeKeyword : EXPTIME : Exposure time +IMAGETYPE-KEYWORD : ImagetypeKeyword : OBSTYPE / EXPTYPE : Image type (DARK, OBJECT, etc) +CCDNUM-KEYWORD : CCDnumKeyword : IMAGEID : CCD number +FILTER-KEYWORD : FilterKeyword : FILTER : filter name +AIRMASS-KEYWORD : AirmassKeyword : AIRMASS : airmass +FOCUS-KEYWORD : FocusKeyword : TELFOCUS: telescope focus setting +ROTATION-KEYWORD : RotationKeyword : ROTANGLE: rotation angle +DETTEMP-KEYWORD : DettempKeyword : DETTEM : detector temperature +TELDATA1-KEYWORD : Teldata1Keyword : XPROBE : bonnette probe x position +TELDATA2-KEYWORD : Teldata2Keyword : YPROBE : bonnette probe y position +TELDATA3-KEYWORD : Teldata3Keyword : ZPROBE : bonnette focus +RA-DDD-KEYWORD : RADecDegKeyword : CRVAL1 : RA in dec. degrees +DEC-DDD-KEYWORD : DECDecDegKeyword : CRVAL2 : DEC in dec. degrees +RA-HMS-KEYWORD : RASexigKeyword : CRVAL1 : RA in hour,min,sec +DEC-DMS-KEYWORD : DECSexigKeyword : CRVAL2 : DEC in dec,min,sec +CAMERA-KEYWORD : CameraKeyword : INSTRUME: unique name of camera + +(time keywords already abstracted) +DATE-KEYWORD : DateKeyword : DATE-OBS : date +DATE-MODE : DateMode : yyyy-mm-dd : format for DATE-KEYWORD +UT-KEYWORD : UTKeyword : UTC-OBS : UT +JD-KEYWORD : JDKeyword : NONE : JD +MJD-KEYWORD : MJDKeyword : NONE : MJD + +(photcode) +CAMERA-KEYWORD : CameraKeyword : DETECTOR: unique name of camera +FILTER-KEYWORD : FilterKeyword : FILTER : filter name +CCDNUM-KEYWORD : CCDnumKeyword : IMAGEID : CCD number + + +EXTEND : extensions present (FITS required keyword for MEF) +NEXTEND : extension number (not necessarily IMAGEID) + +(gophot) +SATVALUE: saturation value for this image +GAIN : gain (e/DN) for this image +NEWGAIN : a stupid name for an altered gain. +RDNOISE : read noise (in electrons) *** +DNNOISE : read noise (in DN) *** + +(gastro / coordops in ohana) +WCS keywords (these do not need to be abstracted) +CD1_1 +CD1_2 +CD2_1 +CD2_2 +CDELT1 +CDELT2 +CROTA2 +CRPIX1 +CRPIX2 +CRVAL1 +CRVAL2 +CTYPE2 +NPLYTERM +PC001001 +PC001002 +PC002001 +PC002002 +PCA1X0Y2 +PCA1X0Y3 +PCA1X1Y1 +PCA1X1Y2 +PCA1X2Y0 +PCA1X2Y1 +PCA1X3Y0 +PCA2X0Y2 +PCA2X0Y3 +PCA2X1Y1 +PCA2X1Y2 +PCA2X2Y0 +PCA2X2Y1 +PCA2X3Y0 +DEC_O +DEC_X +DEC_Y +RA_O +RA_X +RA_Y +EPOCH +EQUINOX + +NAXIS1 : FITS mandatory keyword +NAXIS2 : FITS mandatory keyword +RA : RA in sexigesimal (for rough astrometry) +DEC : DEC in sexigesimal (for rough astrometry) +NSTARS : elixir-added keyword (number of stars, from imclean) + +EXPTIME +AIRMASS +IMAGEID + +ZERO_PT : Elixir-added keyword (imclean) +NASTRO : Elixir-added keyword (imclean) +CERROR : Elixir-added keyword (gastro) +PHOTCODE: Elixir-added keyword (imclean) +APMIFIT : Elixir-added keyword (imclean) +dAPMIFIT: Elixir-added keyword (imclean) +FLIMIT : Elixir-added keyword (imclean) +FSATUR : Elixir-added keyword (imclean) +FWHM_X : Elixir-added keyword (imclean) +FWHM_Y : Elixir-added keyword (imclean) +TZERO : Elixir-added keyword (imclean) +TRATE : Elixir-added keyword (imclean) +NSTARS : Elixir-added keyword (imclean) + +NIMAGES : (Elixir keyword for phot db) + +parse_time.c: fits_scan (header, JDKeyword, "%lf", 1, &jd); +parse_time.c: fits_scan (header, MJDKeyword, "%lf", 1, &jd); +parse_time.c: fits_scan (header, UTKeyword, "%s", 1, line); +parse_time.c: fits_scan (header, DateKeyword, "%s", 1, line); +wimage.c: fits_scan (&header, "NIMAGES", "%d", 1, &Nimages); Index: /trunk/Ohana/doc/linux.txt =================================================================== --- /trunk/Ohana/doc/linux.txt (revision 2508) +++ /trunk/Ohana/doc/linux.txt (revision 2508) @@ -0,0 +1,148 @@ +From jht@ing.iac.es Mon Jul 21 03:58 PDT 1997 +Return-Path: +Received: from ing.iac.es by pikake.astro.washington.edu + (8.6.12/UW-NDC Revision: 2.33 ) id DAA16312; + Mon, 21 Jul 1997 03:58:40 -0700 +Received: by ing.iac.es (4.1/1.0) + id AA20304; Mon, 21 Jul 97 11:58:27 BST +Date: Mon, 21 Jul 1997 11:58:26 +0100 (BST) +From: John Telting +To: Eugene Magnier +Cc: Saskia Prins , John Telting +Subject: Re: fits tools on linux +In-Reply-To: <199706101748.KAA22094@pikake.astro.washington.edu> +Message-Id: +Mime-Version: 1.0 +Content-Type: TEXT/PLAIN; charset=US-ASCII +Content-Length: 3744 +Status: O + +hi gene, + +HOWZIT? +we saw lots of dreamcatchers on indian territory, but only few +were 3D like the one you gave us. Stunning views in and in between +the national parks/monuments. Great vacation behind us!! + +gdb is useful! +apparently the linux C compiler reads the FITS bytes into the +variables in the reverse order (NOT a byteswap!!). +i only tested SIGNED bitpix=16,32,-32. +for bitpix=8 there is a sign problem. + +we had a closer look at sas's linux mana version, and all data values +are affected by the same problem. (So Sas cannot really use it conveniently.) + +below a hack in the 1D vector routine you made especially for my use +when we both were in amsterdam. I guess you can make use of my hacking. +if/when you are going to do a port to linux, please let us know!!! + +See You!! +John + +ps. the compiler gives warnings about the way i coded the pointers. +dont know how to do it the way it likes it, sorry. + +---------------------------------------------------------------- +file: F_get_V_value.c +---------------------------------------------------------------- +# include "fits.h" + +/***************** fits get matrix value *******************/ +double fits_get_vector_value (matrix, x) +Matrix *matrix; +int x; +{ + + double value; + float float1, float2; + int pixel, int1, int2; + short short1, short2; + char yyy[3], zzz[3]; + + if ((x < 0) || (x >= matrix[0].Naxis[0])) { + fprintf (stderr, "%6d %d\n", x, matrix[0].Naxis[0]); + return (0.0); + } + pixel = x; + value = fmod (1.0,0.0); + + if (pixel==0) { + fprintf(stderr,"\n\n!! fits_get_vector_value !!\n"); + fprintf(stderr,"!! USE THIS hack ONLY UNDER linux !!\n"); + fprintf(stderr,"!! DONT USE FOR bitpix=8 !!\n"); + fprintf(stderr,"!! DONT USE FOR bitpix=-64 !!\n"); + fprintf(stderr,"!! DONT USE FOR UNSIGNED TYPES !!\n"); + } + + if (matrix[0].unsign) { + switch (matrix[0].bitpix) { + case 8: + value = *((unsigned char *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR UNSIGNED TYPES !!\n"); + break; + case 16: + value = *((unsigned short *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR UNSIGNED TYPES !!\n"); + break; + case 32: + value = *((unsigned int *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR UNSIGNED TYPES !!\n"); + break; + case -32: + value = *((float *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR UNSIGNED TYPES !!\n"); + break; + case -64: + value = *((double *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR UNSIGNED TYPES !!\n"); + break; + } + } + else { + switch (matrix[0].bitpix) { + case 8: + value = *((char *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR bitpix=8 !!\n"); + break; + case 16: + short1 = *((short *) matrix[0].buffer + pixel); + short2 = 0; + swab(&short1,&short2,2); + value = short2; + break; + case 32: + int1 = *((int *) matrix[0].buffer + pixel); + int2 = 0; + bcopy(&int1,&zzz[0],4); + yyy[0] = zzz[3]; + yyy[1] = zzz[2]; + yyy[2] = zzz[1]; + yyy[3] = zzz[0]; + bcopy(&yyy[0],&int2,4); + value = int2; + break; + case -32: + float1 = *((float *) matrix[0].buffer + pixel); + float2 = 0; + bcopy(&float1,&zzz[0],4); + yyy[0] = zzz[3]; + yyy[1] = zzz[2]; + yyy[2] = zzz[1]; + yyy[3] = zzz[0]; + bcopy(&yyy[0],&float2,4); + value = float2; + break; + case -64: + value = *((double *) matrix[0].buffer + pixel); + fprintf(stderr,"!! DONT USE FOR bitpix=-64 !!\n"); + break; + } + } + + value = matrix[0].bscale*value + matrix[0].bzero; + return (value); + +} + + Index: /trunk/Ohana/doc/locks.txt =================================================================== --- /trunk/Ohana/doc/locks.txt (revision 2508) +++ /trunk/Ohana/doc/locks.txt (revision 2508) @@ -0,0 +1,70 @@ + +NFS locks & file locks in ohana / elixir + +There are three types of locks in use in the elixir system: + +SOFT, XCLD, HARD + +SOFT - a soft lock on a file prevents a xcld or hard lock on that + file, but allows other soft locks to be set. the soft lock + uses NFS locking and is ephmeral, vanishing if the holding + program ends, whether or not the program explicitly removes the + lock. + +XCLD - an xcld lock on a file prevents all other locks: soft, xcld, or + hard; only one xcld lock may be active on a file at a time. + The xcld lock uses NFS locking is ephemeral, vanisihing when + the program exits. + +HARD - a hard lock on a file prevents all other locks: soft, xcld, or + hard; only one hard lock may be active on a file at a time. + The hard lock is sticky and will remain unless the program + actively removes the lock. It is implemented by first setting + an XCLD lock in NFS and then creating a lock file (.file.lck) + which contains the work BUSY. until this file is removed or + the contents changed, the lock remains. + +basic lock APIs: + +int setlockfile2 (char *filename, double timeout, int type, int *state) +FILE *fsetlockfile (char *filename, double timeout, int type, int *state) + +setlockfile2 sets a lock of the requested type on the given file, + returning an open file descriptor to the file. If the requested + lock is SOFT and the file does not exist, setlockfile2 returns -1 + and sets the state to LCK_EMPTY - the lock is not set. In the case + of XCLD or HARD locks, if the file does not exist, it is created. + If the lock cannot be set immediately, setlockfile2 will block for + up to 'timeout' seconds, at which point it will return -1 and exit + with state LCK_TIMEOUT. + +fsetlockfile behaves identically to setlockfile2, but generates and +returns a stream associated with the open file descriptor. An error +will result in the return of (FILE *) NULL. + +int clearlockfile2 (char *filename, int fd, int type, int *state) +int fclearlockfile (char *filename, FILE *f, int type, int *state) + +clearlockfile2 clears the lock associated with a file and file +descriptor. If the type is HARD, it will clear the associated hard +lock file before removing the NFS level file lock. If only the +filename is passed to clearlockfile2, the NFS-level locks cannot be +unset, but the hardlock will be unset. this is useful if a program is +known to then exit. + +fclearlockfile behaves identically to setlockfile2, but requires the +associated stream, rather than the file descriptor. + + +----- + + +int lock_catalog (Catalog *catalog, int lockmode) + +lock_catalog attempts to set a lock on the specified photometry +catalog file. If successful, the catalog file pointer is set to the +beginning of the catalog and the function returns the value 1. If the +file is empty, the return value is 2. This only indicates an error if +the file pointer is NULL, which occurs if the requested locktype is +SOFT. + Index: /trunk/Ohana/doc/notes.trange.txt =================================================================== --- /trunk/Ohana/doc/notes.trange.txt (revision 2508) +++ /trunk/Ohana/doc/notes.trange.txt (revision 2508) @@ -0,0 +1,119 @@ + +---- info on locks ---- + +addrefs/src/addrefs.c: fd = setlockfile2 (ImageCat, 3600.0, LCK_HARD, &dbstate); +addspphot/src/addspphot.c: fd = setlockfile2 (ImageCat, 3600.0, LCK_HARD, &dbstate); +addstar/src/addstar.c: fd = setlockfile2 (ImageCat, 3600.0, LCK_HARD, &dbstate); +elixir/src/DumpStatus.c: fd = setlockfile2 (filename, 0.1, LCK_XCLD, &state); +elixir/src/LoadPending.c: fd = setlockfile2 (filename, 1.0, LCK_XCLD, &lockstate); +elixir/src/MachineOps.c: fd = setlockfile2 (filename, 0.1, LCK_XCLD, &state); +elixir/src/MsgOps.c: fd = setlockfile2 (fifo, 0.1, LCK_XCLD, &state); +elixir/src/MsgOps.c: fd = setlockfile2 (fifo, 0.1, LCK_XCLD, &state); +imregister/old/imregister1.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister/old/imsort1.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister/old/imstatreg1.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister/old/photreg1.c: fd = setlockfile2 (dBFile, 120.0, LCK_HARD, &dbstate); +imregister/old/photsearch1.c: fd = setlockfile2 (dBFile, 300.0, lockstate, &dbstate); +imregister/old/transearch1.c: fd = setlockfile2 (dBFile, 300.0, lockstate, &dbstate); +imregister/old/transreg1.c: fd = setlockfile2 (dBFile, 120.0, LCK_HARD, &dbstate); +imregister/old/imregister1b.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister/old/imsearch.c: fd = setlockfile2 (DataBase, 300.0, lockstate, &dbstate); +imregister/old/imsort.c: fd = setlockfile2 (DataBase, 600.0, LCK_HARD, &dbstate); +imregister/old/imstatreg.c: fd = setlockfile2 (DataBase, 600.0, LCK_HARD, &dbstate); +imregister/base/WriteFIFO.c: fd = setlockfile2 (filename, 20.0, LCK_XCLD, &state); +imregister/detrend/detregister2.c: fd = setlockfile2 (fullname, 2.0, LCK_XCLD, &filestate); +imregister-1.0/src/old/imdbset.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/old/detsearch1.c: fd = setlockfile2 (dBFile, 300.0, lockstate, &dbstate); +imregister-1.0/src/WriteFIFO.c: fd = setlockfile2 (filename, 20.0, LCK_XCLD, &state); +imregister-1.0/src/photsearch1.c: fd = setlockfile2 (dBFile, 300.0, lockstate, &dbstate); +imregister-1.0/src/detsearch1.c: fd = setlockfile2 (dBFile, 300.0, lockstate, &dbstate); +imregister-1.0/src/imstatreg1.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/imregister1.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/photreg1.c: fd = setlockfile2 (dBFile, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/detregister1.c: fd = setlockfile2 (dBFile, 600.0, LCK_HARD, &dbstate); +imregister-1.0/src/transearch1.c: fd = setlockfile2 (dBFile, 300.0, lockstate, &dbstate); +imregister-1.0/src/transreg1.c: fd = setlockfile2 (dBFile, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/detregister.c: fd = setlockfile2 (fullname, 2.0, LCK_XCLD, &filestate); +imregister-1.0/src/imsort1.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/imregister.c: fd = setlockfile2 (DataBase, 120.0, LCK_HARD, &dbstate); +imregister-1.0/src/imsearch.c: fd = setlockfile2 (DataBase, 300.0, lockstate, &dbstate); +imregister-1.0/src/imsort.c: fd = setlockfile2 (DataBase, 600.0, LCK_HARD, &dbstate); +imregister-1.0/src/detregister2.c: fd = setlockfile2 (fullname, 2.0, LCK_XCLD, &filestate); +imregister-1.0/src/imstatreg.c: fd = setlockfile2 (DataBase, 600.0, LCK_HARD, &dbstate); +imregister-1.0/src/mknames.c: fd = setlockfile2 (argv[1], 60.0, LCK_XCLD, &dbstate); +libohana/src/glockfile.c: setlockfile2 returns a file descriptor to an open READ/WRITE access file, +libohana/src/glockfile.c: fd = setlockfile2 (filename, timeout, type, state); +libohana/src/glockfile.c: /* setlockfile2 returns a pointer to a vile open for update */ +libohana/src/glockfile.c:int setlockfile2 (char *filename, double timeout, int type, int *state) { +libohana/src/phot_catalog.c: fd = setlockfile2 (catalog[0].filename, 3600.0, lockmode, &dbstate); +misc/src/dbtest.c:int setlockfile2 (char *filename, double timeout, int hard, int *state); +misc/src/dbtest.c: if ((fd = setlockfile2 (dbfile, 60.0, LCK_HARD, &dbstate)) == -1) { +misc/src/glockfile2.c:int setlockfile2 (char *filename, double timeout, int type, int *state) { +status/user/elixir.c: fd = setlockfile2 (fifo, 2.0, LCK_XCLD, &state); +status/user/elixir.c: fd = setlockfile2 (fifo, 2.0, LCK_XCLD, &state); + +---- info on -time -trange ---- + + +convert to a single usage: -trange (start) (stop||range), -time (time) +allow multiple entries + +-time vs -trange: + +./imregister/src/photsearch.c: if (N = get_argument (argc, argv, "-trange")) { +./imregister/src/transreg.c: if (N = get_argument (argc, argv, "-trange")) { +./imregister/src/imsearch.c: if (N = get_argument (argc, argv, "-trange")) { +./imregister/src/photsearch2.c: if (N = get_argument (argc, argv, "-trange")) { +./status/user/detrend.c: if (N = get_argument (argc, argv, "-trange")) { +./status/user/images.c: if (N = get_argument (argc, argv, "-trange")) { +./status/user/imrough.c: if (N = get_argument (argc, argv, "-trange")) { + +elixir2: grep -- -time `find . -name "*.c"` +./addrefs/src/args.c: if ((i = get_argument (argc, argv, "-time"))) { +./imregister/src/old/detsearch.new.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/old/imsearchrough.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/old/detregister.old2.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/old/detregister.old.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/old/detsearch1.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/detsearch.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/detregister.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/photsearch.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/transearch.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/imsearch.c: if (N = get_argument (argc, argv, "-time")) { +./imregister/src/photsearch2.c: if (N = get_argument (argc, argv, "-time")) { +./scatfit/src/initialize.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/gimages.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/detrend.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/dmags.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/elixir.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/ddmags.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/resid.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/photresid.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/images.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/imdata.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/imextract.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/imlist.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/imrough.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/imsearch.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/photcodes.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/zeropts.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/ddmagsextract.c: if (N = get_argument (argc, argv, "-time")) { +./status/user/ddmagextract.c: if (N = get_argument (argc, argv, "-time")) { + + +haka1: grep -- -trange * +dt.select: $line = "-type flat -filter $name -mode $mode -trange $start $stop"; +fr.select: $line = "-type object -mode $mode -trange $start $stop -filter $name"; +imselect: $line = "-type flat -filter $name -mode $mode -trange $ARGV[3] $ARGV[4]"; +mktrans: @list = `photsearch -trange $start $stop -photcode $code`; + +haka1: grep -- -time * +normalize: if ($ARGV[0] eq "-time") { + +haka1: grep -- -trange * +mk.ptlist: imsearch $config $type -trange $start $stop | awk '{n=substr($6,0,7)}{printf "%s/%s %s/%s%s %s %s\n", $5, $6, n, n, $4, $4, $3}' +phot.by.night: transreg $code $zp $dzp $date 1d -trange + +haka1: grep -- -time * +fr.smooth:set mask=`detsearch -ccd $ccd -type mask -time $start -select -quiet` +phot.by.night:set answer=`photsearch -time $date 1d -photcode $code | awk '{f+=$3}{f2+=$3*$3}END{print f/NR, sqrt (f2/NR - f*f/NR/NR)}'` Index: /trunk/Ohana/doc/pipeline/.mana =================================================================== --- /trunk/Ohana/doc/pipeline/.mana (revision 2508) +++ /trunk/Ohana/doc/pipeline/.mana (revision 2508) @@ -0,0 +1,2 @@ +99/11/12 12:22:43: echo {1280 / (25.4*8)} +99/11/12 12:46:39: quit Index: /trunk/Ohana/doc/pipeline/elixir.fig =================================================================== --- /trunk/Ohana/doc/pipeline/elixir.fig (revision 2508) +++ /trunk/Ohana/doc/pipeline/elixir.fig (revision 2508) @@ -0,0 +1,76 @@ +#FIG 3.2 +Portrait +Center +Inches +Letter +90.00 +Single +-2 +1200 2 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 5175 3000 1650 975 5175 3000 6825 3975 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 8400 10575 1350 825 8400 10575 9750 11400 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 2475 10650 1350 825 2475 10650 3825 11475 +2 4 0 1 0 7 0 0 20 0.000 0 0 7 0 0 5 + 8400 8700 6600 8700 6600 6375 8400 6375 8400 8700 +2 4 0 1 0 7 2 0 20 0.000 0 0 7 0 0 5 + 9450 8250 7650 8250 7650 5925 9450 5925 9450 8250 +2 4 0 1 0 7 0 0 20 0.000 0 0 7 0 0 5 + 7425 9150 5625 9150 5625 6825 7425 6825 7425 9150 +2 4 0 1 0 7 0 0 -1 4.000 0 0 7 0 0 5 + 3300 9150 3300 6825 1500 6825 1500 9150 3300 9150 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 0 0 1 + 5850 3900 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 6225 3750 7950 5925 +2 1 0 1 0 7 20 0 -1 0.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 4275 3900 2400 6825 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 1 1 2 + 1 0 1.00 60.00 120.00 + 1 0 1.00 60.00 120.00 + 2475 9150 2475 9825 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 7350 9150 7875 9825 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 9150 8250 8475 9750 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 7950 8700 8175 9750 +2 1 1 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 3300 9975 5625 8325 +2 1 1 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 7050 10500 3300 8325 +2 2 0 1 7 7 1 0 20 0.000 0 0 -1 0 0 5 + 2925 4875 4275 4875 4275 5625 2925 5625 2925 4875 +2 1 0 1 0 7 1 0 -1 0.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 5550 3975 6000 6825 +2 1 0 1 0 7 2 0 -1 0.000 0 0 -1 1 0 2 + 1 0 1.00 60.00 120.00 + 5850 3900 6975 6375 +2 2 0 1 7 7 1 0 20 0.000 0 0 -1 0 0 5 + 5775 4575 7125 4575 7125 5400 5775 5400 5775 4575 +4 0 0 0 0 0 36 0.0000 4 375 8385 1200 1500 Elixir: Calibration and Data Evaluation\001 +4 0 0 0 0 0 18 0.0000 4 255 915 5850 7425 Ptolemy\001 +4 0 0 0 0 0 18 0.0000 4 195 465 5850 8175 data\001 +4 0 0 0 0 0 18 0.0000 4 195 1155 5850 8550 evaluation\001 +4 0 0 0 0 0 18 0.0000 4 195 720 1650 7425 FLIPS\001 +4 0 0 0 0 0 18 0.0000 4 255 1320 7650 10350 Photometry\001 +4 0 0 0 0 0 18 0.0000 4 255 1320 7725 10875 Astrometry\001 +4 0 0 0 0 0 18 0.0000 4 195 1260 1800 10425 Calibration\001 +4 0 0 0 0 0 18 0.0000 4 255 780 2025 10875 Images\001 +4 0 0 0 0 0 18 0.0000 4 225 720 4800 3075 and/or\001 +4 0 0 0 0 0 18 0.0000 4 255 1110 3975 2625 Telescope\001 +4 0 0 0 0 0 18 0.0000 4 195 930 5325 3525 Archive\001 +4 0 0 0 0 0 18 0.0000 4 195 1185 1650 8550 creation &\001 +4 0 0 0 0 0 18 0.0000 4 195 1155 1650 8865 evaluation\001 +4 0 0 0 0 0 18 0.0000 4 195 1200 1650 8175 calibration\001 +4 0 0 0 0 0 18 0.0000 4 255 765 6225 5250 images\001 +4 0 0 0 0 0 18 0.0000 4 255 765 3225 5475 images\001 +4 0 0 0 0 0 18 0.0000 4 195 1200 3000 5175 calibration\001 +4 0 0 0 0 0 18 0.0000 4 195 870 6000 4875 science \001 Index: /trunk/Ohana/doc/pipeline/flags.fig =================================================================== --- /trunk/Ohana/doc/pipeline/flags.fig (revision 2508) +++ /trunk/Ohana/doc/pipeline/flags.fig (revision 2508) @@ -0,0 +1,29 @@ +#FIG 3.2 +Landscape +Center +Inches +Letter +100.00 +Single +-2 +1200 2 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 3 + 2400 975 4050 975 4050 4125 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 3 + 6525 1425 7650 1425 7650 4200 +4 0 0 0 0 2 14 0.0000 4 195 1695 600 975 Bit 15: ID_Moving\001 +4 0 0 0 0 2 14 0.0000 4 180 1590 600 1425 Bit 14: ID_USNO\001 +4 0 0 0 0 2 14 0.0000 4 195 2085 4275 1425 Bit 14: ID_Bad_Object\001 +4 0 0 0 0 2 14 0.0000 4 195 1695 7875 1875 Lower Byte Values\001 +4 0 0 0 0 2 14 0.0000 4 150 1095 8025 2325 0x01: Ghost\001 +4 0 0 0 0 2 14 0.0000 4 150 1005 8025 2625 0x02: Trail\001 +4 0 0 0 0 2 14 0.0000 4 150 1050 8025 2925 0x03: Bleed\001 +4 0 0 0 0 2 14 0.0000 4 180 1785 600 1875 Bit 13: ID_Variable\001 +4 0 0 0 0 2 14 0.0000 4 180 1875 600 2325 Bit 12: ID_Transient\001 +4 0 0 0 0 2 14 0.0000 4 180 1905 600 2775 Bit 11: ID_Bad_Data\001 +4 0 0 0 0 2 14 0.0000 4 195 1635 600 3225 Bit 10: ID_Proper\001 +4 0 0 0 0 2 14 0.0000 4 150 1620 600 3675 Bit 09: Undefined\001 +4 0 0 0 0 2 14 0.0000 4 150 1620 600 4125 Bit 08: Undefined\001 +4 0 0 0 0 0 12 0.0000 4 180 510 6750 1350 (if yes)\001 +4 0 0 0 0 0 12 0.0000 4 180 510 2925 900 (if yes)\001 +4 0 0 0 0 2 14 0.0000 4 180 1770 4275 1875 Bit 13: ID_Asteroid\001 Index: /trunk/Ohana/doc/pipeline/loneos.fig =================================================================== --- /trunk/Ohana/doc/pipeline/loneos.fig (revision 2508) +++ /trunk/Ohana/doc/pipeline/loneos.fig (revision 2508) @@ -0,0 +1,174 @@ +#FIG 3.2 +Portrait +Center +Inches +Letter +90.00 +Single +-2 +1200 2 +6 2775 9450 8325 11250 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 5475 10500 825 450 5475 10500 6300 10950 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 3675 10500 825 450 3675 10500 4500 10950 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 7275 10500 825 450 7275 10500 8100 10950 +2 4 1 1 0 7 0 0 -1 4.000 0 0 7 0 0 5 + 8325 11250 8325 9450 2775 9450 2775 11250 8325 11250 +2 1 1 1 0 7 0 0 -1 4.000 0 0 -1 0 0 2 + 6375 9450 6375 11250 +4 0 0 0 0 0 18 0.0000 4 195 840 5025 10650 Star dB\001 +4 0 0 0 0 0 18 0.0000 4 255 1050 3150 10650 Image dB\001 +4 0 0 0 0 0 18 0.0000 4 195 945 6825 10650 Rock dB\001 +4 0 0 0 0 0 18 0.0000 4 195 1440 3825 9900 Photom dB\001 +-6 +6 3975 8400 5175 8850 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 3975 8400 5175 8400 5175 8850 3975 8850 3975 8400 +4 0 0 0 0 0 18 0.0000 4 195 810 4125 8775 addstar\001 +-6 +6 2100 3150 3075 3525 +2 2 0 1 0 7 0 0 -1 4.000 0 0 -1 0 0 5 + 2100 3150 3075 3150 3075 3525 2100 3525 2100 3150 +4 0 0 0 0 0 18 0.0000 4 195 735 2175 3450 flatten\001 +-6 +6 375 10050 1725 11250 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 1050 10650 675 600 1050 10650 1725 11250 +4 0 0 0 0 0 18 0.0000 4 195 750 750 10725 USNO\001 +-6 +6 375 8625 1725 9825 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 1050 9225 675 600 1050 9225 1725 9825 +4 0 0 0 0 0 18 0.0000 4 195 1125 450 9300 HST GSC\001 +-6 +6 3225 6525 4425 6975 +6 3225 6525 4425 6975 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 3225 6525 4425 6525 4425 6975 3225 6975 3225 6525 +4 0 0 0 0 0 18 0.0000 4 195 495 3525 6825 fstat\001 +-6 +-6 +6 2550 4800 3750 5250 +6 2550 4800 3750 5250 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 2550 4800 3750 4800 3750 5250 2550 5250 2550 4800 +4 0 0 0 0 0 18 0.0000 4 255 780 2700 5175 dophot\001 +-6 +-6 +6 2925 5475 4125 6225 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 3525 5850 600 375 3525 5850 4125 6225 +4 0 0 0 0 0 18 0.0000 4 255 345 3300 5925 obj\001 +-6 +6 2250 3825 3450 4575 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 2850 4200 600 375 2850 4200 3450 4575 +4 0 0 0 0 0 18 0.0000 4 195 255 2700 4275 flt\001 +-6 +6 3600 7275 4800 8025 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 4200 7650 600 375 4200 7650 4800 8025 +4 0 0 0 0 0 18 0.0000 4 195 480 3975 7725 cmp\001 +-6 +6 5925 8325 7125 8775 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 5925 8325 7125 8325 7125 8775 5925 8775 5925 8325 +4 0 0 0 0 0 18 0.0000 4 195 660 6075 8625 filters\001 +-6 +6 6150 6375 7800 7575 +6 6150 6375 7800 7575 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 6975 6975 825 600 6975 6975 7800 7575 +4 0 0 0 0 0 18 0.0000 4 195 510 6750 6900 alert\001 +-6 +4 0 0 0 0 0 18 0.0000 4 195 465 6750 7275 files\001 +-6 +6 1650 7425 2850 7875 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 1650 7425 2850 7425 2850 7875 1650 7875 1650 7425 +4 0 0 0 0 0 18 0.0000 4 225 675 1800 7800 gastro\001 +-6 +6 2775 12600 3975 13050 +6 2925 12750 3825 12975 +4 0 0 0 0 0 18 0.0000 4 195 915 2925 12975 addusno\001 +-6 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 2775 12600 3975 12600 3975 13050 2775 13050 2775 12600 +-6 +6 4200 12600 5400 13050 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4200 12600 5400 12600 5400 13050 4200 13050 4200 12600 +4 0 0 0 0 0 18 0.0000 4 195 1005 4350 12975 markstar\001 +-6 +6 5625 12600 6825 13050 +6 5625 12600 6825 13050 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 5625 12600 6825 12600 6825 13050 5625 13050 5625 12600 +-6 +4 0 0 0 0 0 18 0.0000 4 195 1080 5775 12975 markrock\001 +-6 +6 7050 12600 8250 13050 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 7050 12600 8250 12600 8250 13050 7050 13050 7050 12600 +4 0 0 0 0 0 18 0.0000 4 255 765 7200 12975 nrphot\001 +-6 +6 8100 6450 9750 7650 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 8925 7050 825 600 8925 7050 9750 7650 +4 0 0 0 0 0 18 0.0000 4 255 885 8475 7290 pictures\001 +4 0 0 0 0 0 18 0.0000 4 240 795 8475 6975 results,\001 +-6 +6 7725 8250 8925 8700 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 7725 8250 8925 8250 8925 8700 7725 8700 7725 8250 +4 0 0 0 0 0 18 0.0000 4 165 645 7875 8625 status\001 +-6 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 1425 1950 750 525 1425 1950 2175 2475 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 3750 1950 750 525 3750 1950 4500 2475 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 3525 7650 2850 7650 +2 4 1 1 0 7 0 0 -1 4.000 0 0 7 0 0 5 + 2025 12375 2025 8550 150 8550 150 12375 2025 12375 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 3375 11250 3375 12600 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 1 2 + 0 0 1.00 60.00 120.00 + 2250 3075 1425 2475 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 1 2 + 0 0 1.00 60.00 120.00 + 2850 3075 3750 2475 +2 1 1 1 0 7 0 0 -1 4.000 0 0 -1 1 0 2 + 0 0 1.20 100.00 200.00 + 2550 3525 4875 9450 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 4800 11250 4800 12600 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 6225 11250 6225 12600 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 7575 11250 7575 12600 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 1 3 + 0 0 1.00 60.00 120.00 + 2775 12825 1050 12825 1050 12375 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 1 3 + 0 0 1.00 60.00 120.00 + 1650 7650 1050 7650 1050 8550 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 6525 8325 6675 7500 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 6225 9450 6361 8772 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 8039 9453 8189 8703 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 8339 8253 8489 7503 +4 0 0 0 0 0 18 0.0000 4 195 1155 525 12075 references\001 +4 0 0 0 0 0 18 0.0000 4 165 840 675 11775 astrom.\001 +4 0 0 0 0 0 18 0.0000 4 195 825 3375 2025 archive\001 +4 0 0 0 0 0 18 0.0000 4 255 1020 900 2025 telescope\001 +4 0 0 0 0 0 36 0.0000 4 495 4200 5250 4425 Variability Pipeline\001 +4 0 0 0 0 0 36 0.0000 4 435 3870 5550 3450 U.W. / LONEOS \001 Index: /trunk/Ohana/doc/pipeline/params.txt =================================================================== --- /trunk/Ohana/doc/pipeline/params.txt (revision 2508) +++ /trunk/Ohana/doc/pipeline/params.txt (revision 2508) @@ -0,0 +1,125 @@ +# Configuration file for Loneos analysis pipeline + +# important files and directories +IN_DIR /pallas +OUT_DIR /irene/d11/workspace +# +CATDIR /hebe/d27/database +IMAGE_CATALOG /hebe/d27/database/Images.dat +IMAGE_CATALOG_TEMPLATE /hebe/d27/database/template.cat +CATALOG_TEMPLATE /hebe/d27/database/template.cat +ROCK_CATALOG /hebe/d27/database/Rocks.dat +# +GSCFILE /metis/d11/references/GSC/GSCregions.tbl +GSC_DIR /metis/d11/references/GSC +USNO_CDROM /metis/d11/references/USNO +DOPHOT_PARAMS /metis/d11/references/config/default_parameters +FLATTEN_SCRIPT /metis/d11/references/config/flatten.pro +LONEOS_REGIONS /metis/d11/references/config/regions.map + +# parameters for "fstat" +# instrumental mag zero point +ZERO_PT 24.5 +DOPHOT_CHAR_LINE 129 +DOPHOT_TYPE_FIELD 5 +DOPHOT_AP_FIELD 91 +DOPHOT_PSF_FIELD 52 +MIN_SN_FSTAT 6.0 + +# parameters for "gastro" +DEFAULT_RADIUS 32.0 +MINIMUM_RADIUS 1.5 +MIN_ERROR 0.6 +MIN_PRECISE 0.12 +CCD_PC1_1 1 +CCD_PC2_2 -1 # for loneos +CCD_PC1_2 0 +CCD_PC2_1 0 +ASEC_PIX 2.82 +NFIELD 3.0 +MMIN 6.0 +ROT_ZERO 0 +dROT 1.0 +NROT 2 +POLAR_AXIS_RA -90.0 +POLAR_AXIS_DEC 89.88 +RA_OFFSET -90.0 +DEC_OFFSET 0.0 + +# parameters for "addstar" +# airmass extinction coefficient (in mag / airmass) +NSIGMA 3.0 +ALPHA 0.03 +XOVERSCAN 60 +YOVERSCAN 50 + +# parameters for "controller" +NEW_IMAGES images.dat +USERNAME gene +PASSWORD gene + +# here we list all available machines +NMACHINES 6 +MACHINE0 metis +MACHINE1 hebe +MACHINE2 iris +MACHINE3 irene +#MACHINE4 juno +MACHINE4 ceres +MACHINE5 vesta + +# here we define which machines which are always free +fMACHINE0 1 +fMACHINE1 1 +fMACHINE2 1 +fMACHINE3 1 +fMACHINE4 0 +fMACHINE5 0 +fMACHINE6 0 + +LOCAL_MACHINE 1 # machine with catalog on local disk + +# parameters for "status" +TIME_REFERENCE 90/01/01 00:00:00 + +# parameters for "nrphot" +# make this one small to boost change of getting an early region +TAU 3.0 +SCATTER_LIM 15.0 +MAG_LIM 17.0 +IMAGE_SCATTER 0.075 +NIMAGE_SCATTER 1.5 +STAR_SCATTER 0.05 + +### parameters for "markstar" +SEARCH_RADIUS 360 # region for initial trail hunt, in arcsec +TRAIL_WIDTH 5.0 # expected trail width, in arcsec +NANGLE_BINS 180 # Number of angular bins in 180 degrees +NPTSINLINE 5 # minimum points needed for trail +MIN_DENSITY 0.025 # minimum linear density in star/arcsec +SPACE_SIGMA 10.0 # how much denser than average for trail? +# parameters defining bright star exclusion regions +BRIGHT_XTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_XTRAIL_MAG 9.5 # faintest sat. star +BRIGHT_XTRAIL_SLOPE -80.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_YTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_YTRAIL_MAG 11.0 # faintest sat. star +BRIGHT_YTRAIL_SLOPE -400.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_HALO_MAG 8.5 # faintest sat. star +BRIGHT_HALO_SLOPE -28.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +# parameters which define the ghosts +GHOST_MAG 7.5 +GHOST_RADIUS 200 # in arcsec +OPTICAL_AXIS1 2154.0 +OPTICAL_AXIS2 2193.0 + +# parameters for "addusno" +USNO_RADIUS 5.0 # in arcsec +USNO_PROPER 20.0 # in arcsec +USNO_RED 1000 # code for USNO Red data +USNO_BLUE 1001 # code for USNO Blue data + +# parameters for "markrock" +ROCK_RADIUS 2.0 # in arcsec +ROCK_MAX_RADIUS 20.0 # in arcsec +ROCK_MAX_SPEED 0.1 # in arcsec / sec Index: /trunk/Ohana/doc/pipeline/pipeline.c =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline.c (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline.c (revision 2508) @@ -0,0 +1,22 @@ +\documentstyle[astro]{article] +\GoodMargins +\title{Variability Pipeline} +\author{Eugene Magnier} +\begin{document} +\maketitle + +\section{Overview} + +\section{Photometry -- Dophot} + +\section{Astrometry -- gastro} + +\section{The Photometry Database} + +\section{Visualization Tools} + +\section{Software Distribution} + +\section{Programming Details} + +\end{document} Index: /trunk/Ohana/doc/pipeline/pipeline.tex =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline.tex (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline.tex (revision 2508) @@ -0,0 +1,997 @@ +%\documentstyle[astro,psfig]{article} +\documentclass{article} +\usepackage{psfig} +\usepackage{astro} +\usepackage{times} +\GoodMargins +\newcommand{\note}[1]{({\bf #1})} +\title{Variability Pipeline} +\author{Eugene Magnier} +\begin{document} +\maketitle + +\section{Overview} + +The Loneos Pipeline provides automated reduction of images, including +photometry and astrometry, with the final data stored in a photometry +database organized by star position on the sky. + +Fig.~\ref{pipeline} shows a schematic of the analysis pipeline. +Images are written to the disk from the camera by the software which +runs it, {\tt astrocam}. The resulting images are FITS files (which +may be arbitrarily sized) and have header keyword parameters RA and +DEC specifying rough celestial coordinates. Images are first analyzed +by {\tt dophot}, which produces instrumental photometry and pixel +coordinate positions for stars and other objects in the images, +writing files with a particular format, called OBJ files in this +discussion. The results from {\tt dophot} are cleaned up by a program +called {\tt fstat} which merges the header information from the image +with the most relevant parts of the OBJ file, creating a new file with +the CMP format. Astrometry is performed on the CMP file with the +program {\tt gastro}. {\tt Gastro} makes a comparison with an +astrometric reference database and only changes CMP header keywords as +a result. Finally, the processed image CMP file is added to the +photometry database with the program {\tt addstar}. The photometry +database is maintained in a large number of files (CPT files), each +representing a small region of sky. There is no relationship between +the image locations and the boundaries of CPT file. All of these +processes occur on individual image, and would typically be performed +while the observations are occuring, or during the day following, but +may be performed at any time to add new or archived data to the +database. + +Once large batches of images have been processed, a series of programs +should be run to clean up the photometry database. Typically, these +routines would be run at the end of a night on all of the CPT files to +which new data were added. First, {\tt markstar} identifies bad data +(the trails of satelites, the bleeding columns from stars, ghost +images), and flags these data points appropriately. Second, {\tt + addusno} matches the catalog stars with stars from the USNO database +and adds the USNO magnitudes to the photometry database. Next, {\tt + markrock} searches for likely asteroids in the CPT file, marks them, +and writes the asteroid observations to an asteroid database. It also +notes measurements which are likely to be noise and cosmic rays. +Finally, {\tt nrphot} determines relative photometric calibrations for +observations in a set of CPT files. + +With the exception of {\tt dophot}, all of the programs in the +pipeline refer to a single configuration file to determine the large +number of parameters necessary in the process. The default parameter +file is hardwired into the programs, but an alternate configuration +file may be specified with the environment variable {\tt + PIPE\_CONFIG}. This option makes independent processing of multiple +data sources trivial. For example, if the pipeline is to be used for +both Loneos and SDSS image analysis and display, there could be two +configuration files (eg, {\tt loneos.txt} and {\tt sdss.txt}), one for +each data source. Several important parameters would be different +between these two, for example, the initial plate scale guess for the +astrometry routine, or the parameters which define the rough chip +orientation. This scheme allows for analysis of different types of +images, with a common destination photometry database, or for +independent photometry databases. + +\begin{figure} +\psfig{file=schematic.eps,width=9cm} +\caption{\label{pipeline} +Schematic of the photometry pipeline. Rectangles represent programs, +while ovals represent data products. Arrows show the direction of +travel of information. The flatten routine is a filter whose position +may change as the pipeline matures. +} +\end{figure} + +Data from different sources can be processed by the pipeline and +stored in the same photometry database, since all photometry is tagged +with a source ID in the database (as are the images in the image +database). This allows us to include, for example, the USNO catalog +magnitude measurements (or a subset) in the photometry database for a +quick comparion. Different sources may be considered different +filters, so color-color and color-magnitude diagrams could be +extracted by specifying choices for source catalog. Currently each +object has a single primary photometric system, and individual +observations may be combined to determine that average (eg, if they +are different photometry sources with roughly the filter bandpass) or +they may be ignored when calculating relative magnitudes (eg, the USNO +magnitudes would be ignored). This may be expanded upon by defining +multiple average magnitudes and allowing different measurements to +contribute only to the appropriate average. The determination of the +photometry code for a particular image is currently not well defined. +There are several points in the analysis where the user (or the +controlling program) may specify the photometry source. A table +should be kept in the photometry catalog to define the various +photometry codes. + +\section{Image Analysis Pipeline} +\subsection{Photometry -- Dophot} + +Photometry is performed using a varient of {\tt Dophot}. The version +of dophot is based on Dophot 2.0, but it has been adapted to +streamline processing of many files. First, with this version of +dophot, any FITS format data is safely read in. Instead of compiling +to a fixed maximum size, only the memory needed to contain the image +matrix is allocated. The image is also converted to REAL*32 from +whatever BITPIX for processing. Finally, only a single, complete +parameter file is loaded by dophot. There is no option for a modified +and a default parameter file. The parameter file to be used is +defined, along with the input image and the output object list on the +command line when the program is run: \\ +{\tt dophot image.fits image.obj parameter\_file} \\ +To achive this, the code from dophot was wrapped within a C program +which interprets the command line arguments, allocates the necessary +memory, and reads in the image. The rest of the processing is +performed by dophot in the standad manner. Notice that we use a +consistent naming convention through the pipeline reductions. The +image is called {\tt foo.fits}, while the resulting dophot-produced +photometry list is called {\tt foo.obj}. + +\subsection{Cleanup -- fstat} + +After an image is processed by dophot, the object file ({\tt foo.obj}) +is converted to a more-compact, more-complete file called a CMP file, +with a name of the form ({\tt foo.cmp}). This conversion is done with +the program {\tt fstat}. The CMP file consists of the FITS header +from the original image ({\tt foo.fits}), with some additional +keywords to be used at later stages in the analysis, followed by an +ASCII list of the interesting data from ({\tt foo.obj}). In this +list, types 6 and 8 are excluded, and only the following values are +kept: +\begin{verbatim} +X Y Mag dMag t log(sky) +1342.0 106.1 14.166 000 4 3.2 +\end{verbatim} +Objects with a signal-to-noise ratio lower than a specified cutoff +({\tt MIN\_SN\_FSTAT}) are also excluded. Some general information +about the image is derived by {\tt fstat} (FWHM, saturation and +completeness limits, number of each dophot type) and stored as +keywords in the header. The photometry source must be specified at +this stage as one of the command line arguments, though the value used +may be overridden in one of the stages belowe. The resulting file can +now stand on its own without reference to the original image. The +keywords used by {\tt fstat} include the minimum signal to noise, a +rough guess at the zero point ({\tt ZERO\_PT}), and four numbers +defining the format of the {\tt dophot} OBJ file. + +\subsection{Astrometry -- gastro} + +Astrometry is performed automatically by the program {\tt gastro}. +{\tt gastro} loads a CMP file and determines an initial guess for the +center coordinates (based on the RA and DEC header keywords). The +program also uses the configuration information about the plate scale +and rough orientation of the image to get a close to the final +solution. Also, the true sky position of the telescope pole may be +defined to allow astrometry on images taken close to the pole. The +comparison is made with the astrometric catalog, which may be the HST +Guide Star Catalog, or it may be any source if the data is placed in +the correct format. + +\section{The Photometry Database} + +\subsection{Photometry File Format} +The photometry database consists of a large number of photometry files +representing sections of the sky and a file containing information on +the images incorporated in the database. These files are called +alternatively ``region files'' or CPT files (because they have the +extension {\tt .cpt}). The photometry files are sorted by Declination +into different directories, each consisting of a band 7.5\degree\ +wide. These directories have names like n0730, representing the band +starting at a Declination of +07:30. The image database (Images.dat) +is stored in the upper level directory of this directory tree. In +fact, the locations of the Images.dat file, the photometry database, +and such are all flexible and may be changed by altering the +configuration file. + +The name and location of each sky region comes from the Hubble Space +Telescope Guide Star Catalog. There is a file, GSCregions.tbl, which +can be used to find the region file appropriate for any location on +the sky. A number of routines exist to make such a query. + +Each photometry file in the photometry database contains all +observations and all average values for all stars observed within the +appropriate region on the sky. The identity of stars is essentially +determined by the RA and DEC coordinates. In very crowded regions, +the end user may need to double check that a neighboring star is not +contaminating individual detections (see the discusion on {\tt + addstar} below). + +The photometry files are stored in a binary format, both to reduce the +volume and to speed access. As a result, the interpretation of the +data is machine dependent: little endian machines need to swap the +data intelligently. Fortunately, this operation is taken care of +appropriately, if the programs are compiled with the BYTESWAP option +set as needed, which is automatically set by the Makefile in the case +of a Linux machine. For reference, Suns, SGI, and HP are all big +endian, while PCs and DECs are little endian. The automatic +conversion takes place by using the funcions {\tt Fread} and {\tt Fwrite} +to substitute for the standard C library {\tt fread} and {\tt fwrite} +functions. {\tt Fread} and {\tt Fwrite} are told the datatype being read, +and they know the layout of the bytes for a particular datatype. +Anytime the definitions of the relevant structures (see below) are +changed, {\tt Fread} and {\tt Fwrite} must be updated. Fortunately, there +is only one file ({\tt Fread.c}), and it has a single byteswapping +section, making it easy to adjust the code appropriately. + +The format of a single photometry database file consists four +sections: +\begin{itemize} +\item {\bf header} -- this is a standard FITS header with room for comments + about the file or whatever. Since the rest of the file is not a + standard FITS file, the SIMPLE keyword is set to False. The three + necessary keywords are NSTARS, NMEAS, and NMISS, which define the + sizes of the next three sections. +\item {\bf average} -- this section contains all average measurement + quantities for each star in the file. There are NSTARS entries, and + each entry is the data from a structure of type Average (defined in + loneos.h, and discussed in more detail below). Thus, the total size + in bytes of this section can be found by NSTARS*sizeof(Average). +\item {\bf measure} -- this section contains the individual + measurements for each star in the database. There are NMEAS + entries, and like the average section, each entry is a single + structure, in this case of type Measure (in loneos.h, see below). + All measurements of a single star are consecutive. The number of + measurements for a single star is defined by average.Nm and the + starting entry for a single star is given by average.offset. + Therefore, the first measurement of a specific star average[500] + would be given by measure[average[500].offset], while the second is + measure[average[500].offset + 1], and so forth. +\item {\bf missing} -- this section contains references to all missing + measurements of a star. This means all of those occasions when an + image enclosed the location of a star in the database, but no star + was detected on the image. Similar to the case of the measure + section, each star has a number of missing entries (which may be 0) + given by average.Nn, and the first missing entry is given by + average.missing. Each missing entry value is again a structure of + type Missing. Currently, the only data in the structure is the time + of the observation, which allows for an unambiguous recovery of the + image where the star was missed. +\end{itemize} + +\subsubsection*{Average Structure defined in loneos.h} +\begin{verbatim} +typedef struct { + float R; /* RA in decimal degrees */ + float D; /* DEC in decimal degrees */ + short int M; /* thousandths of mag (-32.000 to 32.000 valid range) */ + unsigned short int Nm, Nn; /* number of measurements, missing */ + short int Xp, Xm; /* log(chisq) values in tenths */ + unsigned short int code; /* an ID code (ie, star, ghost, satelite, etc) */ + signed int offset; /* offset to first measurement */ + signed int missing; /* offset to first missing obs */ +} Average; /* 28 bytes / Average */ +\end{verbatim} +The structure above defines the average data stored in the photometry +database for each star. Most of the entries are self-explantory, but +some need a bit of clarification. The two $\chi^2$ entries ({\tt Xp} +and {\tt Xm}) are the $\chi^2$ values of the average position and the +average magnitude. These are stored as logarthmic quantities because +high dynamic range is not needed. For the magnitude, the assumption +is that the star brightness is constant and the $\chi^2$ incorporates +the photometric errors on each measurement. For the postion, the +a default constant error is used \note{True?} and again the assumption +is that the postion is constant. As discussed above, the values of +{\tt Nm} and {\tt Nn} give the number of measurements and missing +observations for this star, while the entries {\tt offset} and {\tt + missing} point to the first {\bf measure} and {\bf missing} entry +for this star. The entry {\tt M} is the current best average +magnitude solution for this star (see the section on Relative +Photometry). Finally, the entry {\tt code} is an ID code for each +object, which may define a variety of things about the object. For +example, if the object is known to be variable, or if the object is +associated with a USNO star, or if the object is a measurement of an +asteroid. See below for details of these definitions. + +\subsubsection*{Measure Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + short int dR, dD; /* 1/100 of arcsec (-327.67 to +327.67 valid range) */ + short int M; /* thousandths of mag (-32.767 to 32.767 valid range) */ + short int Mcal; /* image cal mag, thousandths of mag (-32.767 to 32.767 valid range) */ + unsigned char dM; /* thousandths of mag (0.000 to 0.255 valid range) */ + char dophot; /* dophot type (1-9) */ + unsigned short int source; /* code to identify photometry source */ + unsigned int t; /* time in seconds (0 - 143 years valid range) */ + unsigned int average; /* reference to corresponding average entry */ + /* upper byte of Measure.average stores flags: + limit of 16,777,215 stars (Naverage) in a file (=0xFFFFFF) + flags: average & 0x1000000 */ +} Measure; /* 20 bytes / Measure */ +\end{verbatim} +} +The structure above defines the individual measurement data stored for +each observation of each star. The first two entries, dR and dD, are +the difference between the average coordinates for this star and the +coordinates determined for this observation. Clearly, if this +residual is too large, the individual measurement will not be +successfully matched with the average star position. {\tt M} and {\tt + dM} are the instrumental magnitude and error for this measurement, +after correction for the exposure time and a zero point (defined in +the configuration file as ZERO\_PT). The entry {\tt t} determines the +time of the individual observation. This is important not only for +timing purposes, but also to determine the source image for this +observation. The time and the entry {\tt source} uniquely determine +the image in the image database that generated this measurement. +Furthermore, this correspondence allows for determination of the pixel +coordinates on the chip, by going throught the image astrometry stored +in the Image database (see below). The entry {\tt source} is a code which +identifies the particular CCD/Telescope/Filter setup. Each unique +{\tt source} should be treated as an independent filter set for the +purposes of accurate relative photometric comparisons. This entry +also allows external catalog data, such as the USNO or Sloan +photometry, to be added to the database, as desired. The entry {\tt + average} is a pointer back to the {\bf average} section to allow +determination of the star from the measurements, in addition to the +reverse. The bits of the upper byte of this value are reserved for +flags used to define particular situations for this measurement. +Possible values are: +\begin{itemize} +\item 1 = BLEND\_IMAGE - the star on the image matched more than one + catalog star +\item 2 = BLEND\_CATALOG - the star in the catalog matched more than + one image star +\item 4 = UPPER\_LIMIT - \note{not really used?} +\item 8 = CALIBRATED - relative photometry has been performed at least + once. +\end{itemize} +The entry {\tt dophot} stores the dophot type for this particular +measurement (1-9: four extra bits in this field). +Finally, the value {\tt Mcal} determines the photometric calibration +of this specific measurment. This value is an offset appropriate to +this image (and if needed, this point in time and this chip position) +to bring the observations of the stars on multiple images to a common +system (see the section on relative photometry). + +\subsection{Image Database} + +All images for which photometry has been included in the photometry +database also have entries in an image database. The image database +is (currently) a single file in the upper directory of the photometry +database. The name of the file is determined by the IMAGE\_CATALOG +entry in the configuration file, and is currently set to Images.dat. +Like the photometry database files, the image database consists of a +FITS header followed by binary data. There is only one type of binary +data: each image has an entry, which is the data from a structure of +type Image (loneos.h). The number of images in the file is determined +by the {\tt NIMAGES} keyword. If the number of images becomes too +large, extrapolation of the system to a set of image database files +with a reference list is quite straightforeward, and could be modeled +on the organization of the region files (eg, one image database file +for each Declination directory). + +\subsubsection*{Image Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + Coords coords; /* 120 bytes */ + unsigned int tzero; /* readout time row 0 in sec (0 - 142 years valid range) */ + unsigned int nstar; /* number of stars on image */ + short int secz; /* thousanths of airmass (valid range -32.000 -- 32.000) */ + short int NX, NY; /* dimensions of image */ + short int apmifit, dapmifit; /* aperture correction and error in thousandths of mag */ + short int source; /* identifier for CCD (each ever used will have a unique letter) */ + short int Mcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int dMcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int Xm; /* 10*log(image chi-square) */ + char name[32]; /* name of original image */ + unsigned char detection_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char saturation_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char cerror; /* astrometric error: 1/50 of arcsec (0 -- 5.12 valid range) */ + unsigned char fwhm_x, fwhm_y; /* PSF terms in 25*arcsec (valid range 0.0 -- 10.2" ") */ + unsigned char trate; /* 10000 * scan rate in sec/pix (0 -- 0.0256 valid range, + typically 0.0146. this is used only to determine the time + of the observation, not to find the coordinates. 1 byte + gives 0.11 sec accuracy */ + char code; /* flag to mark an image as bad or whatever */ + char dummy[7]; /* extra space for the future? (seems like a lot!) */ + float exptime; /* exposure time, seconds */ +} Image; /* 192 bytes / Image */ +\end{verbatim}} +The structure above lists all of the data stored for each image. A +substantial amount of information is stored for each image. First, +the astrometric calibration information is stored in a {\tt Coords} +structure (see coordinate systems, above). The upper and lower limits +of the photometry in the database (in instrumental magnitudes) is +stored in {\tt detection\_limit} and {\tt saturation\_limit}. These +values are useful for comparison for those stars where observations +are missing. The relationship between time and pixel coordinates is +given by {\tt tzero} and {\tt trate}. Drift images will have a scan +rate stored in {\tt trate}, while staring images will have a value of +0.0 for this entry ({\tt tzero} then applies to the whole image). +Other important parameters are kept: the astrometric error ({\tt + cerror}), the airmass ({\tt secz}), the exposure time ({\tt + exptime}), the number of stars detected ({\tt nstar}), the FWHM +values ({\tt fwhm\_x, fwhm\_y}), the image name ({\tt name}), and the +size of the image ({\tt NX, NY}). The entry {\tt source} is used as +above to define the CCD/Telescope/Filter used for this observation. +The entry {\tt code} is used to flag images in various ways. +Currently, only one bit is used, to define if the image has +unacceptable photometric scatter. Seven extra bytes are stored for +future additions. + +\subsection{Data Incorporation -- {\tt addstar}} + +Images which are completely processed ({\tt dophot, fstat, gastro}) are +then incorporated into the photometry database with the program {\tt +addstar}. This program decides which region files are appropriate +for this particular image, then one-by-one adds stars from the image +to the appropriate region file. Stars already in the catalog are +matched with stars in the new image purely by a positional comparison. +In order to avoid the difficulty of comparisons in the RA and DEC +coordinate frame, a cartesian projection is performed. The stars from +the image being processed and the database stars in the same area are +projected onto a tangent plane and positional comparisons made in this +(locally cartesian) coordinate frame. This avoids the dangerous +singularities at the pole and also makes the RA 0,360\degree\ boundary +a trivial problem. + +Several choices must be made in the comparison process. If a star in +the catalog is matched with a star in the image, the new measurements +of that star are added to the database. If a star in the database +lies in the field of the image, but is not detected, this information +is added to the list of missing data. The only data stored in this +case is the time and source, which is sufficient to unambiguously +identify the source image from the image database. This allows later +programs to find relevant statistics from the image database, if +necessary. If a star is detected in the image, but is not already in +the database, a new entry is added to the database. In addition, the +same ``missing data'' from all previous observations images which have +covered this location (ie, images already in the image database) are +included. This last step is necessary so that the image processing +order is not important. + +Stars also run the danger of being crowded together. Since all +comparisons are performed on the basis of position alone, crowded +fields may make for ambiguous cross-identifications. A pair of stars +are matched if the difference in their positions is less than a +specified search radius (usually dependent on the astrometric for the +image). If more than one catalog stars is correlated with a star in +an image, the new measurement is added to {\em both} catalog stars, +and a flag is set noting that this observation had a blended IMAGE. +Conversely, if a single catalog star is matched with more than one +image star, both new measurements are added to the one catalog star, +and a different flat is set, noting that this observation had a +blended CATALOG. + +\section{Catalog Update and Cleaning} + +Occasionally, once a large number of images have been added to the +photometry database, a set of programs should be run to clean and +update the CPT files. This may best be done on CPT files after a +night's worth of data is incorporated. These routines define bad +data, identify the USNO catalog stars, and search for asteroids and +other junk. Most of the results from these routines are the addition +of object flags to the Average data structures (Average.code). Most +of these routines process data within a single CPT file at a time. + +\subsection{Markstar} + +The first of the update programs, {\tt markstar}, identifies bad data +and flags it. It makes three types of identifications: bright stars, +ghost stars, and trails. In the first case, it searches the HST guide +star catalog for bright stars within the field of the current CPT +file. Three types of pixels are flagged for bright stars. Any +measurements within a circular region centered on the star are suspect +(the ``halo'' of the bright star). Second, points along the X axis +are likely to contain flux from the diffraction spikes. Finally, +points along the Y axis are likely to have bleeding in addition to the +diffraction spikes. Each of these three regions has a different size +and range which must be defined in the configuration file. The +regions are represented by a representative magnitude and a scale +parameter. For example, the diameter $d$ of the halo for a star with +magnitude $m$ is defined as +$d = \mbox{BRIGHT\_HALO\_SLOPE} (m - \mbox{BRIGHT\_HALO\_MAG})$. The X and +Y axis points are defined by the length of the diffraction spike, with +an equivalent formula, and the width of the spike. Since these are +each independent, there are 8 parameters to define the exclusion +regions of bright stars. One of the current drawbacks of this system +is the reliance on the HST Guide Star photometry. In the HST Guide +Star Catalog, only one photometry band is given, so no color +information is available. The parameters currently used were defined +based on a large number of bright stars, with no adjustment for the +different effective filter system of the Loneos telescope and the HST +GSC. Thus, occasionally very red stars will be several magnitudes +brighter in the real data than reported in the catalog. As a result, +large numbers of bad data points in these halos and spikes may be +missed. + +The second type of bad data flagged by {\tt markstar} are the trails +from satelite and other space junk. This portion of the program +searches for objects which land along a line and which have a high +linear density. Parameters may be adjusted to define the width and +the necessary density or number of points in the line. Objects in +lines are also only identified if all observations come from a common +image. If an object only has one measurement and it is in a trail, +the entire Average is flagged to be bad. However, if an object has +many measurements and only one is bad, the object is only flagged as +having some bad data points. + +The third type of bad dat are ghost star images. The HST GSC is +searched for stars projected across the telescope optical axis, which +is defined by the user. All objects within a region of fixed size is +marked for ghost stars brighter than a specific cutoff magnitude. +Similar to the trails, the average is marked as bad only if all +detections of an object are ghost images. Otherwise, it is noted that +some data may be bad. + +\subsection{Addusno} + +The program {\tt addusno} makes two types of identifications of stars +with the USNO catalog. First, it searches for coincidences within a +small, specified radius. The choice of this radius is somewhat +tricky: it is important to catch the association, but also to avoid +making too many associations. Unfortunately, astrometric errors in +the USNO catalog make it necessary to choose a surprisingly large +radius of 5\asec. To avoid making double matches (matching the two +USNO stars to the same photometry database object), only the USNO +object closest to the object is associated. There are also a +substantial number of stars with significant proper motion between the +USNO epochs and the current date. It is not unusual to see 8-15\asec\ +discrepancy for specific stars. To catch these objects, any objects +which do not match the USNO database, but which have more than a +minimum number of measurements (2 or 3?) are searched for more distant +USNO companions. A large, user-specified radius is used for this +search. Of course, only USNO stars which are not already matched to +the database may be candidates for the proper motion match. This +stage is suceptible to errors in that a slow-moving solar-system +object may be associated with a faint USNO star. For this reason, +objects flagged as high proper motion stars should be taken with some +caution. Of course, during the {\tt addusno} stage, no objects +already flagged as bad by {\tt markstar} will be matched with USNO +stars. + +\subsection{Markrock} + +Once stars have been identified with the USNO database, it is possible +to search for asteroid detections. The program {\tt markrock} +searches within a specific CPT file for objects which are moving along +straight lines in (X,Y,t) space. In order for an object to be found +by {\tt markrock}, it must have three points, each with only single +measurements. The third point must land within a specified distance +of the line projected from the first two data points. These +comparisons are made in the three dimensional space of image position +and time. Also, objects are only accepted if they have a speed less +than a threshold. If an object is moving too fast, it will make a +streak on each individual image and will probably be missed anyway. +Moving objects detected in this way are flagged in the photometry +database and are also written to a Rock database. Currently, the Rock +database just contains ASCII lists of the RA, DEC, Mag, and time for +each detection. In fact, the data are stored in pairs of detections, +with the middle detection saved twice. This format makes it easy to +plot lines connecting the points using the connect-the-dots plotting +style of status/kapa (see below). + +One of the current drawbacks of {\tt markrock} is the insistence on +three observations with only 1 measurement each. If an object is +moving too slowly, two of the dectections of the object may be merged +together. Enough data is stored in the database to extract these +individual measurements, so a more sophisticated search is possible. +It is also currently the case that USNO proper motion stars are +ignored, but some of these will be asteroid detections which are +simply too close to a faint USNO stars (it has to be faint because it +would otherwise be associated with a star in the photometry +database). + +\begin{figure} +\psfig{file=flags.eps,width=9cm} +\caption{\label{flags} +Pictoral representation of the Average.code flags. The lower byte +value is used for specific definitions of object types. +} +\end{figure} + +\subsection{Data Flags} + +The identifications made by the above routines are noted as a set of +flags in the Average.code structure entry. Some of these flags are +mutually exclusive, so a certain bit may have more than one meaning, +depending on the value of other bits. Figure~\ref{flags} shows a +pictoral representation of the meaning of the different bit fields. +The uppermost bit (bit 15) determines if the object may be considered +a ``fixed'' star or something which has no fixed location. If it is +not fixed, and this bit is 1, then there are several options. The +second uppermost bit (bit 14) will be set if this object is one of the +three types of bad data flagged by {\tt markstar}. The lower byte of +code is used to define the type of bad data: ghost, trail, or bleed. +Thus, any datapoints determined to be a satelite trail will have a +code value of 0xc002 (49154 decimal). If an object is a moving +object, but not a bad datapoint, it may be identified as a ``rock'' +(an asteroid) by the program {\tt markrock}. In this case, the flag +for an asteroid is turned on (bit 13). Thus anything identified as a +rock by {\tt markrock} will have a code value of 0xa000. The lower +byte of code is not yet defined for asteroids, but could be reserved +for distinguishing different classes of asteroids (ie, on the basis of +orbital speeds, and so forth). Objects which are fixed may have a +variety of possible flags. First, an object may be identified with a +USNO star (bit 14). It may be found to exhibit variability (bit 13), +implying that the relative photometry routine should ignore it. If +may be a transient object (bit 12). Note that, while an object +associated with a USNO object may not be transient, an object which is +variable may also not be associated with a USNO object. Thus, we need +seperate bits for variable vs transient. Different types of variable +objects may be represented with different values of the lower byte. +For example, a Cepheid may have a specific lower byte code of 0x01, +which an RR Lyra may have a lower byte code of 0x02. Thus a Cepheid +associated with the USNO catalog would have a total code value of +0x6001. Finally, any of these types of non-moving objects may have +some bad individual measurements (bit 12) or may be found to have a +significant proper motion (bit 11). + +\section{Visualization -- Status} + +Visualization of the data in the photometry database can best be +performed with the program {\tt status}. This is a command-line +driven program with a math and macro language which makes it easy to +perform complex tasks. + +First, a few notes about the user interface. The interface has an +interaction similar to {\tt tcsh}. The arrows allow editing of +previous commands. You can also use emacs-like commands such as +cntl-a to reach the beginning of the line and cntl-e to reach the end. +There is command and file completion: if you type part of a command +(as the first thing on a line) and then type tab, it will fill in as +much as possible, until the word is not unique. Typing tab twice at +that point will list the possible endings. For any but the first word +on a line, the same thing will happen for the files in the current +directory. It is also possible to type just a fraction of a command, +as long as it is unique. An ambiguous command will list the possible +alternatives. For example: +\begin{verbatim} +status: c +ambiguous command: c ( catalog cgrid clear create cursor ) +\end{verbatim} + +The shell is essentially an interpretive programming language. +Variables are set as follows: +\begin{verbatim} +status: $fred = 10 +\end{verbatim} +Any expression within curly brackets \{\} is assumed +to be an arithmetical expression and is evaluated before the line is +executed. For example: +\begin{verbatim} +echo {$fred*dcos(45)} +\end{verbatim} +would give the response 7.07107. There are math functions cos, sin, +and tan, which operate on radian expressions, and also dcos, dsin, +dtan, which operate on degree expressions. There are also the +equivalent inverse functions: eg., asin and dasin return radians and +degrees, respectively. The help section on Math defines all of the +available math functions. + +\subsection{Miscellaneous Commands} +\begin{verbatim} +! -- system call +? -- list commands +?? -- list variables +echo -- type this line +exec -- system call +exit -- exit program +help -- get help on a function +output -- redirect output to file +quit -- exit program +scan -- scan line from keyboard or file to variable +wait -- wait until return is typed +which -- show command +\end{verbatim} +Most of these are self-explanatory. The command ?? prints the system +variables. The {\tt help} command will provide help on a single +command or, without any arguments, will list all available help +files (this includes general help not associated with a specific +command). + +\subsection{Shell Programing} +\begin{verbatim} +break -- escape from function +for -- loops +if -- logical cases +input -- read command lines from a file +macro -- deal with the macros +\end{verbatim} + +There are several options for programming in {\tt status}. First, a +file which contains a series of commands can be executed with {\tt + input (filename)}. It is also possible to define macros which will +behave much like regular commands. A macro is defined by typing {\tt + macro name} or {\tt macro create name} followed by the commands. +Arguments to the macro are assigned to the variables \$1 .. \$N and +the number of arguments is given by \$0. Macros may be defined in +{\tt input} files, and in fact when {\tt status} is started, it loads +the file {\tt \~/.statusrc} which may contain default macros. Simple +loops and if statements can be performed, and are quite useful for +complex macros. + +``If'' statements are similar in syntax to C if statements, but only +the following logical operators are available: $>$, $<$, $=$, !, $|$, +and \&. Notice that (currently) there is no $>=$ or $<=$ symbol. The +operator ! means ``not equal to'', but cannot be used to negate a +logical value. The operators $|$ and \& have the meaning of ``or'' and +``and'' respectively. Math expresions in the if statement must be +contained in curly braces, as elsewhere. Variables with string values +may use the logical $=$ operator to test if two strings are the same. +``For'' loops are quite simplistic. The form is: +\begin{verbatim} +for var first last delta + (commands) +end +\end{verbatim} +The value of {\tt \$var} will start at the value {\tt first} and increment by +{\tt delta} after each loop. The loop will stop after {\tt \$var} is greater +than {\tt stop}. The value {\tt delta} is optional, with 1 assumed. +The value of {\tt \$var} may be changed during the loop, and if set +beyong the value of {\tt last} will end the loop early. + +\subsection{Vector Plotting} + +\begin{verbatim} +box -- draw a box on the plot +clear -- erase plot +create -- create a new vector +cursor -- get coords from cursor +grid -- plot cartesian grid +hist -- create histogram from a vector +labels -- define labels for plot +limits -- define plot limits +plot -- plot a pair of vectors +print -- write vectors to file +ps -- define labels for plot +set -- vector math +style -- set the style for graph plots +vectors -- list vectors +zplot -- plot scaled points +\end{verbatim} + +In addition to scalar variables, {\tt status} can manipulate and +display 1-D vector variables. Many of the commands which extract data +from the photometry database place the data in vectors as well as +plotting them. A vector can also be created based on a number +sequence with the command {\tt create name Nelements start delta}. +The resulting vector has $Nelements$ entries, starting at a value of +$start$ and running until $start + delta*Nelements$. If $delta$ is +0.0, all elements will have the value of $start$. A histogram of a vector +may be made with the command {\tt hist}, which creates a new vector +containing the histogram of the first vector. The data range and bin +size of the histogram are defined in same way as with create. This +makes it easy to create the index vector that goes with a histogram +vector: +\begin{verbatim} +hist y Ny 1 100 0.1 +create dx 1 100 0.1 +\end{verbatim} +The above will create a histogram of y in Ny and the index in dx. +Plotting this with {\tt plot dx Ny} will show the histogram. + +Vector math is performed with a command of the form {\tt set new = + (expression)}. The expression is some math function employing +vectors and scalars. A complete listing of the math operators +available in {\tt set} can be found in the help for {\tt set}. + +Once vectors are defined, they may be plotted. A pair of vectors can +be plotted against each other if they have the same number of entries. +The plotting is performed on the graphics window, Kapa. There are +actually several graphics windows available to {\tt status}, any of +which may be used to plot at any time. Some of the more complex +operations default to either graphics window 0 or 1, depending on the +context. Except for those functions with a pre-defined window, all +plotting functions apply to the current graphics window unless an +option {\tt -n N} is given to specify a different window. The +plotting style is determined by the command {\tt style} which can set +the line width, the line type (solid, dashed, dotted, etc), the point +type (box, cross, etc), the point size, the color, and whether a pair +of vectors is plotted as a sequence of points, a set of connected +lines, or a histogram. Some functions which make plots use their own +styles, as discussed below. The function {\tt limits} lets the user +set the range of the plot axes, or check the current setting. The +command {\tt plot} will plot a pair of vectors on the current graphics +window using the current plotting style for that window. The command +{\tt zplot} will plot a pair of vectors with the point size scaled by +a third vector, with maximum and minimum point sizes representing +specified values. The {\tt cursor} command goes the other way: this +command puts the Kapa window in cursor mode and waits for input from +Kapa. The user can then type any alphanumeric key on the graphics +windows and will be told both the pointer location (in the graphics +coordinates) and will have the coordinates stored in {\tt status} +variables. For example, by typing ``1'' in the sky display window, +the RA and DEC of the pointer are stored in the variables {\tt \$R1} +and {\tt \$D1}. This command can be used to let the user define +locations or regions of interest on the Kapa window. (Future addition: +{\tt button}, which does the same with the mouse buttons). + +\subsection{Database Functions} + +\begin{verbatim} +gcat -- get catalog at location +gimages -- get images at location +gstar -- get star statistics +extract -- extract average vectors from catalogs +mextract -- extract measurement vectors from catalogs +imstats -- plot image statistics +imextract -- extract image vectors from database +lcat -- list catalogs in display region +cmatch -- match two catalogs +\end{verbatim} + +There are a variety of other commands which directly refer to the +photometry database. Some of these functions extract data of various +types from the database, others perform more complex plotting +operations. The commands listed above are those which simply extract +data from the database. The first three list information relevant to +a specific RA, DEC location on the sky: {\tt gcat (RA) (DEC)} lists +the catalog at the specified location and places the name in the +variable {\tt \$CATNAME}, {\tt gimages (RA) (DEC)} lists all images +which overlap the specified location, , {\tt gstars (RA) (DEC) + (RADIUS)} lists data about the stars within a specified radius of +the specified location (all numbers above are given in decimal +degrees). Similarly, {\tt lcat} lists the catalogs in the region. +Imstats lists statistics about each image + +The next three commands extract a specific piece of information from +the photometry database and places it in a vector. First, {\tt + extract} will extract average values for each star and place it in a +vector. Next, {\tt mextract} will extract measurement values for each +star and place it in a vector: as a result a single star may have +multiple entries in the measurement vectors. Finally, {\tt imextract} +will extract image statistics into vectors (not yet implemented). + + +\begin{verbatim} +catalog -- plot catalog stars +cgrid -- plot sky coordinate grid +cplot -- plot vectors in sky coordinates +czplot -- plot scaled vectors in sky coordinates +images -- plot image boxes +imdense -- image density plot +lcurve -- plot lightcurve for a star +pcat -- plot catalog boundaries +region -- define sky region for plot +resid -- plot residuals +simage -- plot stars in an image +\end{verbatim} + +There are two types of database plotting functions: those that display +or refer to the spatial charateristics of the data and those that +refer to other types of charatersitics, such as the time domain. The +graphics window 0 is reserved for all plots of objects on the sky. +The command {\tt region} defines the current sky coordinates for plots +in graphic window 0. The command {\tt pcat} plots the outline of all +photometry database files which are within the currently defined +region (and by default, only those with data). {\tt images} plots the +outline of the images in the image database, while {\tt imdense} shows +the number of images at a location by randomly spacing dots within the +boundary of the images. The command {\tt cgrid} +draws a grid in celestial coordinates on the for the current region. + +The most complex, but also one of the most useful command is {\tt + catalog}, which plots the positions of stars in the photometry +database (and others) on the sky. There are many options to this +command. One set allows the user to plot stars from the photometry +database (the default), from the HST GSC, or from an ASCII text file +with RA, DEC, and Mag in specified columns. If the ASCII file has a +fixed number of bytes per line, the data can be more quickly loaded. +The size of the points may be scaled by the star magnitude, by the +number of observations of the star, or by the number of missing +datapoints for the star. In addition, points may be plotted only if +they land in specified magnitude ranges, or with specified numbers of +measurements, or missed measurements. Also, objects may be plotted +only if they have a specified Average.code, so that only asteroids or +only perfect stars may be plotted. The plotted vectors may be saved, +if desired, and the source catalog epoch may be specified as different +from J2000 (only valid for ASCII data). + +Several other commands relate to non-spatial charateristics of images +and stars. {\tt lcurve} will plot a light curve for all stars within +some radius of a point. {\tt resid} plots the photometry residuals +for a particular region file. + +\section{Some Examples} + +\begin{figure} +\psfig{file=allsky.ps,width=16cm} +\caption{\label{allsky} +Map of the entire sky, and images added to database. +} +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location +of the images currently in the database. This picture was made with +the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 gls +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=catalog.ps,width=9cm} +\caption{\label{catalog} +Comparison between HST GSC and photometry database astrometry. +} +\end{figure} + +Fig.~\ref{catalog} shows an example comparison of the photometry +database star positions and the HST Guide Star Catalog star +positions. The crosses are from the photometry database while the +boxes are from the HST GSC. The size of both points is a function of +brightness. This plot was made with the following commands (starting +from the previous image): +\begin{verbatim} +status: cursor (typed 1 on region of interest) +1 137.097858 22.698305 +q 137.097858 22.698305 +status: region $R1 $D1 0.5 TAN +status: cgrid +status: box +status: style -pt 0 +status: gcat $R1 $D1 + 0 n2230/1951.cpt * +status: catalog n2230/1951.cpt -m 12 18 +status: style -pt 2 +status: catalog n2230/1951.cpt -m 12 18 -g +\end{verbatim} + +\begin{figure} +\psfig{file=lightcurve.ps,width=9cm} +\caption{\label{lightcurve} +Lightcurve for several image stars. +} +\end{figure} + +Fig.~\ref{lightcurve} shows an example lightcurve from three stars in +the above field. The x axis shows the time in seconds, which the y +axis shows the star magnitudes. This plot was made with the following commands (starting +from the previous image): +{\small +\begin{verbatim} +status: cursor +1 137.062562 22.791860 +q 137.062562 22.791860 +status: lcurve $R1 $D1 0.01 -l +status: box +status: gstar $R1 $D1 0.01 +read 11318 stars from catalog file /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt (22462 total measurements) +508 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.068619 22.785400 18.438000 1 (2) 1.000000 1.000000 +498 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.062820 22.792747 12.882000 3 (0) 1.000000 1.000000 +501 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.064499 22.800817 15.480000 3 (0) 1.000000 1.000000 +\end{verbatim}} +The last lines show the database information on the three stars. + +\section{Software Distribution} + +\section{Programming Details} + +\section {Comparison with USNO catalog} + +I have made several comparisons between the USNO catalog and results +from running images from Loneos through the photometry pipeline. In +general, there is a good agreement between the astrometry of the two +datasets. There are specfic areas where the two disagree, however. +First, as discussed above, the USNO star positions may be +significantly different from the observed star positions simply +because of proper motion. The first picure in the ``Comparisons +between Loneos + USNO'' pages shows several stars with significant +proper motion. These are currently being flagged in the photometry +database as discussed. Other differences evident in this first +picture include the classification of bright stars and the splitting +of faint and bright stars. In the upper right of the image, two +bright stars are visible, one of which is actually a double. The USNO +catalog does a good job of detecting both, but it also detects and +includes as stars several diffraction spike points. The HST GSC only +detected a single point and did not split the star at all. The Loneos +data in our pipeline did not detect the star at all due to saturation. +A star in the upper left corner shows the ability of {\tt dophot} in +the Loneos pipeline to split double stars. This star is, on close +inspection, clearly a double. The USNO catalog only detects one. The +Pipeline not only splits the two stars, but it also makes the correct +cross-identification of the stars. + +The second set of pictures shows the ability of the Loneos pipeline to +deal with small galaxies. There are two small galaxies in this field, +both of which are broken into 4 or 5 objects by the USNO catalog, but +maintained as single objects in the Loneos pipeline. In another +example (not shown) a bright star within the disk of a small galaxies +was clearly split from the galaxy by the pipeline (and also by USNO). + +\end{document} Index: /trunk/Ohana/doc/pipeline/pipeline2.tex =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline2.tex (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline2.tex (revision 2508) @@ -0,0 +1,1284 @@ +%\documentstyle[astro,psfig]{article} +\documentclass{article} +\usepackage{psfig} +\usepackage{astro} +\usepackage{times} +\GoodMargins +\newcommand{\note}[1]{({\bf #1})} +\title{Variability Pipeline} +\author{Eugene Magnier} +\begin{document} +\maketitle + +\section{Overview} + +The Loneos Pipeline provides automated reduction of images, including +photometry and astrometry, with the final data stored in a photometry +database organized by star position on the sky. + +Fig.~\ref{pipeline} shows a schematic of the analysis pipeline. +Images are written to the disk from the camera by the software which +runs it, {\tt astrocam}. The resulting images are FITS files (which +may be arbitrarily sized) and have header keyword parameters RA and +DEC specifying rough celestial coordinates. Images are first analyzed +by {\tt dophot}, which produces instrumental photometry and pixel +coordinate positions for stars and other objects in the images, +writing files with a particular format, called OBJ files in this +discussion. The results from {\tt dophot} are cleaned up by a program +called {\tt fstat} which merges the header information from the image +with the most relevant parts of the OBJ file, creating a new file with +the CMP format. Astrometry is performed on the CMP file with the +program {\tt gastro}. {\tt Gastro} makes a comparison with an +astrometric reference database and only changes CMP header keywords as +a result. Finally, the processed image CMP file is added to the +photometry database with the program {\tt addstar}. The photometry +database is maintained in a large number of files (CPT files), each +representing a small region of sky. There is no relationship between +the image locations and the boundaries of CPT file. All of these +processes occur on individual image, and would typically be performed +while the observations are occuring, or during the day following, but +may be performed at any time to add new or archived data to the +database. + +Once large batches of images have been processed, a series of programs +should be run to clean up the photometry database. Typically, these +routines would be run at the end of a night on all of the CPT files to +which new data were added. First, {\tt markstar} identifies bad data +(the trails of satelites, the bleeding columns from stars, ghost +images), and flags these data points appropriately. Second, {\tt + addusno} matches the catalog stars with stars from the USNO database +and adds the USNO magnitudes to the photometry database. Next, {\tt + markrock} searches for likely asteroids in the CPT file, marks them, +and writes the asteroid observations to an asteroid database. It also +notes measurements which are likely to be noise and cosmic rays. +Finally, {\tt nrphot} determines relative photometric calibrations for +observations in a set of CPT files. + +With the exception of {\tt dophot}, all of the programs in the +pipeline refer to a single configuration file to determine the large +number of parameters necessary in the process. The default parameter +file is hardwired into the programs, but an alternate configuration +file may be specified with the environment variable {\tt + PIPE\_CONFIG}. This option makes independent processing of multiple +data sources trivial. For example, if the pipeline is to be used for +both Loneos and SDSS image analysis and display, there could be two +configuration files (eg, {\tt loneos.txt} and {\tt sdss.txt}), one for +each data source. Several important parameters would be different +between these two, for example, the initial plate scale guess for the +astrometry routine, or the parameters which define the rough chip +orientation. This scheme allows for analysis of different types of +images, with a common destination photometry database, or for +independent photometry databases. + +\begin{figure} +\psfig{file=schematic.eps,width=9cm} +\caption{\label{pipeline} +Schematic of the photometry pipeline. Rectangles represent programs, +while ovals represent data products. Arrows show the direction of +travel of information. The flatten routine is a filter whose position +may change as the pipeline matures. +} +\end{figure} + +Data from different sources can be processed by the pipeline and +stored in the same photometry database, since all photometry is tagged +with a source ID in the database (as are the images in the image +database). This allows us to include, for example, the USNO catalog +magnitude measurements (or a subset) in the photometry database for a +quick comparion. Different sources may be considered different +filters, so color-color and color-magnitude diagrams could be +extracted by specifying choices for source catalog. Currently each +object has a single primary photometric system, and individual +observations may be combined to determine that average (eg, if they +are different photometry sources with roughly the filter bandpass) or +they may be ignored when calculating relative magnitudes (eg, the USNO +magnitudes would be ignored). This may be expanded upon by defining +multiple average magnitudes and allowing different measurements to +contribute only to the appropriate average. The determination of the +photometry code for a particular image is currently not well defined. +There are several points in the analysis where the user (or the +controlling program) may specify the photometry source. A table +should be kept in the photometry catalog to define the various +photometry codes. + +\section{Image Analysis Pipeline} +\label{parallel} +\subsection{Photometry -- Dophot} + +Photometry is performed using a varient of {\tt Dophot}. The version +of dophot is based on Dophot 2.0, but it has been adapted to +streamline processing of many files. First, with this version of +dophot, any FITS format data is safely read in. Instead of compiling +to a fixed maximum size, only the memory needed to contain the image +matrix is allocated. The image is also converted to REAL*32 from +whatever BITPIX for processing. Finally, only a single, complete +parameter file is loaded by dophot. There is no option for a modified +and a default parameter file. The parameter file to be used is +defined, along with the input image and the output object list on the +command line when the program is run: \\ +{\tt dophot image.fits image.obj parameter\_file} \\ +To achive this, the code from dophot was wrapped within a C program +which interprets the command line arguments, allocates the necessary +memory, and reads in the image. The rest of the processing is +performed by dophot in the standad manner. Notice that we use a +consistent naming convention through the pipeline reductions. The +image is called {\tt foo.fits}, while the resulting dophot-produced +photometry list is called {\tt foo.obj}. + +\subsection{Cleanup -- fstat} + +After an image is processed by dophot, the object file ({\tt foo.obj}) +is converted to a more-compact, more-complete file called a CMP file, +with a name of the form ({\tt foo.cmp}). This conversion is done with +the program {\tt fstat}. The CMP file consists of the FITS header +from the original image ({\tt foo.fits}), with some additional +keywords to be used at later stages in the analysis, followed by an +ASCII list of the interesting data from ({\tt foo.obj}). In this +list, types 6 and 8 are excluded, and only the following values are +kept: +\begin{verbatim} +X Y Mag dMag t log(sky) +1342.0 106.1 14.166 000 4 3.2 +\end{verbatim} +Objects with a signal-to-noise ratio lower than a specified cutoff +({\tt MIN\_SN\_FSTAT}) are also excluded. Some general information +about the image is derived by {\tt fstat} (FWHM, saturation and +completeness limits, number of each dophot type) and stored as +keywords in the header. The photometry source must be specified at +this stage as one of the command line arguments, though the value used +may be overridden in one of the stages belowe. The resulting file can +now stand on its own without reference to the original image. The +keywords used by {\tt fstat} include the minimum signal to noise, a +rough guess at the zero point ({\tt ZERO\_PT}), and four numbers +defining the format of the {\tt dophot} OBJ file. + +\subsection{Astrometry -- gastro} + +Astrometry is performed automatically by the program {\tt gastro}. +{\tt gastro} loads a CMP file and determines an initial guess for the +center coordinates (based on the RA and DEC header keywords). The +program also uses the configuration information about the plate scale +and rough orientation of the image to get a close to the final +solution. Also, the true sky position of the telescope pole may be +defined to allow astrometry on images taken close to the pole. The +comparison is made with the astrometric catalog, which may be the HST +Guide Star Catalog, or it may be any source if the data is placed in +the correct format. + +\section{The Photometry Database} + +\subsection{Photometry File Format} +The photometry database consists of a large number of photometry files +representing sections of the sky and a file containing information on +the images incorporated in the database. These files are called +alternatively ``region files'' or CPT files (because they have the +extension {\tt .cpt}). The photometry files are sorted by Declination +into different directories, each consisting of a band 7.5\degree\ +wide. These directories have names like n0730, representing the band +starting at a Declination of +07:30. The image database (Images.dat) +is stored in the upper level directory of this directory tree. In +fact, the locations of the Images.dat file, the photometry database, +and such are all flexible and may be changed by altering the +configuration file. + +The name and location of each sky region comes from the Hubble Space +Telescope Guide Star Catalog. There is a file, GSCregions.tbl, which +can be used to find the region file appropriate for any location on +the sky. A number of routines exist to make such a query. + +Each photometry file in the photometry database contains all +observations and all average values for all stars observed within the +appropriate region on the sky. The identity of stars is essentially +determined by the RA and DEC coordinates. In very crowded regions, +the end user may need to double check that a neighboring star is not +contaminating individual detections (see the discusion on {\tt + addstar} below). + +The photometry files are stored in a binary format, both to reduce the +volume and to speed access. As a result, the interpretation of the +data is machine dependent: little endian machines need to swap the +data intelligently. Fortunately, this operation is taken care of +appropriately, if the programs are compiled with the BYTESWAP option +set as needed, which is automatically set by the Makefile in the case +of a Linux machine. For reference, Suns, SGI, and HP are all big +endian, while PCs and DECs are little endian. The automatic +conversion takes place by using the funcions {\tt Fread} and {\tt Fwrite} +to substitute for the standard C library {\tt fread} and {\tt fwrite} +functions. {\tt Fread} and {\tt Fwrite} are told the datatype being read, +and they know the layout of the bytes for a particular datatype. +Anytime the definitions of the relevant structures (see below) are +changed, {\tt Fread} and {\tt Fwrite} must be updated. Fortunately, there +is only one file ({\tt Fread.c}), and it has a single byteswapping +section, making it easy to adjust the code appropriately. + +The format of a single photometry database file consists four +sections: +\begin{itemize} +\item {\bf header} -- this is a standard FITS header with room for comments + about the file or whatever. Since the rest of the file is not a + standard FITS file, the SIMPLE keyword is set to False. The three + necessary keywords are NSTARS, NMEAS, and NMISS, which define the + sizes of the next three sections. +\item {\bf average} -- this section contains all average measurement + quantities for each star in the file. There are NSTARS entries, and + each entry is the data from a structure of type Average (defined in + loneos.h, and discussed in more detail below). Thus, the total size + in bytes of this section can be found by NSTARS*sizeof(Average). +\item {\bf measure} -- this section contains the individual + measurements for each star in the database. There are NMEAS + entries, and like the average section, each entry is a single + structure, in this case of type Measure (in loneos.h, see below). + All measurements of a single star are consecutive. The number of + measurements for a single star is defined by average.Nm and the + starting entry for a single star is given by average.offset. + Therefore, the first measurement of a specific star average[500] + would be given by measure[average[500].offset], while the second is + measure[average[500].offset + 1], and so forth. +\item {\bf missing} -- this section contains references to all missing + measurements of a star. This means all of those occasions when an + image enclosed the location of a star in the database, but no star + was detected on the image. Similar to the case of the measure + section, each star has a number of missing entries (which may be 0) + given by average.Nn, and the first missing entry is given by + average.missing. Each missing entry value is again a structure of + type Missing. Currently, the only data in the structure is the time + of the observation, which allows for an unambiguous recovery of the + image where the star was missed. +\end{itemize} + +\subsubsection*{Average Structure defined in loneos.h} +\begin{verbatim} +typedef struct { + float R; /* RA in decimal degrees */ + float D; /* DEC in decimal degrees */ + short int M; /* thousandths of mag (-32.000 to 32.000 valid range) */ + unsigned short int Nm, Nn; /* number of measurements, missing */ + short int Xp, Xm; /* log(chisq) values in tenths */ + unsigned short int code; /* an ID code (ie, star, ghost, satelite, etc) */ + signed int offset; /* offset to first measurement */ + signed int missing; /* offset to first missing obs */ +} Average; /* 28 bytes / Average */ +\end{verbatim} +The structure above defines the average data stored in the photometry +database for each star. Most of the entries are self-explantory, but +some need a bit of clarification. The two $\chi^2$ entries ({\tt Xp} +and {\tt Xm}) are the $\chi^2$ values of the average position and the +average magnitude. These are stored as logarthmic quantities because +high dynamic range is not needed. For the magnitude, the assumption +is that the star brightness is constant and the $\chi^2$ incorporates +the photometric errors on each measurement. For the postion, the +a default constant error is used \note{True?} and again the assumption +is that the postion is constant. As discussed above, the values of +{\tt Nm} and {\tt Nn} give the number of measurements and missing +observations for this star, while the entries {\tt offset} and {\tt + missing} point to the first {\bf measure} and {\bf missing} entry +for this star. The entry {\tt M} is the current best average +magnitude solution for this star (see the section on Relative +Photometry). Finally, the entry {\tt code} is an ID code for each +object, which may define a variety of things about the object. For +example, if the object is known to be variable, or if the object is +associated with a USNO star, or if the object is a measurement of an +asteroid. See below for details of these definitions. + +\subsubsection*{Measure Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + short int dR, dD; /* 1/100 of arcsec (-327.67 to +327.67 valid range) */ + short int M; /* thousandths of mag (-32.767 to 32.767 valid range) */ + short int Mcal; /* image cal mag, thousandths of mag (-32.767 to 32.767 valid range) */ + unsigned char dM; /* thousandths of mag (0.000 to 0.255 valid range) */ + char dophot; /* dophot type (1-9) */ + unsigned short int source; /* code to identify photometry source */ + unsigned int t; /* time in seconds (0 - 143 years valid range) */ + unsigned int average; /* reference to corresponding average entry */ + /* upper byte of Measure.average stores flags: + limit of 16,777,215 stars (Naverage) in a file (=0xFFFFFF) + flags: average & 0x1000000 */ +} Measure; /* 20 bytes / Measure */ +\end{verbatim} +} +The structure above defines the individual measurement data stored for +each observation of each star. The first two entries, dR and dD, are +the difference between the average coordinates for this star and the +coordinates determined for this observation. Clearly, if this +residual is too large, the individual measurement will not be +successfully matched with the average star position. {\tt M} and {\tt + dM} are the instrumental magnitude and error for this measurement, +after correction for the exposure time and a zero point (defined in +the configuration file as ZERO\_PT). The entry {\tt t} determines the +time of the individual observation. This is important not only for +timing purposes, but also to determine the source image for this +observation. The time and the entry {\tt source} uniquely determine +the image in the image database that generated this measurement. +Furthermore, this correspondence allows for determination of the pixel +coordinates on the chip, by going throught the image astrometry stored +in the Image database (see below). The entry {\tt source} is a code which +identifies the particular CCD/Telescope/Filter setup. Each unique +{\tt source} should be treated as an independent filter set for the +purposes of accurate relative photometric comparisons. This entry +also allows external catalog data, such as the USNO or Sloan +photometry, to be added to the database, as desired. The entry {\tt + average} is a pointer back to the {\bf average} section to allow +determination of the star from the measurements, in addition to the +reverse. The bits of the upper byte of this value are reserved for +flags used to define particular situations for this measurement. +Possible values are: +\begin{itemize} +\item 1 = BLEND\_IMAGE - the star on the image matched more than one + catalog star +\item 2 = BLEND\_CATALOG - the star in the catalog matched more than + one image star +\item 4 = UPPER\_LIMIT - \note{not really used?} +\item 8 = CALIBRATED - relative photometry has been performed at least + once. +\end{itemize} +The entry {\tt dophot} stores the dophot type for this particular +measurement (1-9: four extra bits in this field). +Finally, the value {\tt Mcal} determines the photometric calibration +of this specific measurment. This value is an offset appropriate to +this image (and if needed, this point in time and this chip position) +to bring the observations of the stars on multiple images to a common +system (see the section on relative photometry). + +\subsection{Image Database} + +All images for which photometry has been included in the photometry +database also have entries in an image database. The image database +is (currently) a single file in the upper directory of the photometry +database. The name of the file is determined by the IMAGE\_CATALOG +entry in the configuration file, and is currently set to Images.dat. +Like the photometry database files, the image database consists of a +FITS header followed by binary data. There is only one type of binary +data: each image has an entry, which is the data from a structure of +type Image (loneos.h). The number of images in the file is determined +by the {\tt NIMAGES} keyword. If the number of images becomes too +large, extrapolation of the system to a set of image database files +with a reference list is quite straightforeward, and could be modeled +on the organization of the region files (eg, one image database file +for each Declination directory). + +\subsubsection*{Image Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + Coords coords; /* 120 bytes */ + unsigned int tzero; /* readout time row 0 in sec (0 - 142 years valid range) */ + unsigned int nstar; /* number of stars on image */ + short int secz; /* thousanths of airmass (valid range -32.000 -- 32.000) */ + short int NX, NY; /* dimensions of image */ + short int apmifit, dapmifit; /* aperture correction and error in thousandths of mag */ + short int source; /* identifier for CCD (each ever used will have a unique letter) */ + short int Mcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int dMcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int Xm; /* 10*log(image chi-square) */ + char name[32]; /* name of original image */ + unsigned char detection_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char saturation_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char cerror; /* astrometric error: 1/50 of arcsec (0 -- 5.12 valid range) */ + unsigned char fwhm_x, fwhm_y; /* PSF terms in 25*arcsec (valid range 0.0 -- 10.2" ") */ + unsigned char trate; /* 10000 * scan rate in sec/pix (0 -- 0.0256 valid range, + typically 0.0146. this is used only to determine the time + of the observation, not to find the coordinates. 1 byte + gives 0.11 sec accuracy */ + char code; /* flag to mark an image as bad or whatever */ + char dummy[7]; /* extra space for the future? (seems like a lot!) */ + float exptime; /* exposure time, seconds */ +} Image; /* 192 bytes / Image */ +\end{verbatim}} +The structure above lists all of the data stored for each image. A +substantial amount of information is stored for each image. First, +the astrometric calibration information is stored in a {\tt Coords} +structure (see coordinate systems, above). The upper and lower limits +of the photometry in the database (in instrumental magnitudes) is +stored in {\tt detection\_limit} and {\tt saturation\_limit}. These +values are useful for comparison for those stars where observations +are missing. The relationship between time and pixel coordinates is +given by {\tt tzero} and {\tt trate}. Drift images will have a scan +rate stored in {\tt trate}, while staring images will have a value of +0.0 for this entry ({\tt tzero} then applies to the whole image). +Other important parameters are kept: the astrometric error ({\tt + cerror}), the airmass ({\tt secz}), the exposure time ({\tt + exptime}), the number of stars detected ({\tt nstar}), the FWHM +values ({\tt fwhm\_x, fwhm\_y}), the image name ({\tt name}), and the +size of the image ({\tt NX, NY}). The entry {\tt source} is used as +above to define the CCD/Telescope/Filter used for this observation. +The entry {\tt code} is used to flag images in various ways. +Currently, only one bit is used, to define if the image has +unacceptable photometric scatter. Seven extra bytes are stored for +future additions. + +\subsection{Data Incorporation -- {\tt addstar}} + +Images which are completely processed ({\tt dophot, fstat, gastro}) are +then incorporated into the photometry database with the program {\tt +addstar}. This program decides which region files are appropriate +for this particular image, then one-by-one adds stars from the image +to the appropriate region file. Stars already in the catalog are +matched with stars in the new image purely by a positional comparison. +In order to avoid the difficulty of comparisons in the RA and DEC +coordinate frame, a cartesian projection is performed. The stars from +the image being processed and the database stars in the same area are +projected onto a tangent plane and positional comparisons made in this +(locally cartesian) coordinate frame. This avoids the dangerous +singularities at the pole and also makes the RA 0,360\degree\ boundary +a trivial problem. + +Several choices must be made in the comparison process. If a star in +the catalog is matched with a star in the image, the new measurements +of that star are added to the database. If a star in the database +lies in the field of the image, but is not detected, this information +is added to the list of missing data. The only data stored in this +case is the time and source, which is sufficient to unambiguously +identify the source image from the image database. This allows later +programs to find relevant statistics from the image database, if +necessary. If a star is detected in the image, but is not already in +the database, a new entry is added to the database. In addition, the +same ``missing data'' from all previous observations images which have +covered this location (ie, images already in the image database) are +included. This last step is necessary so that the image processing +order is not important. + +Stars also run the danger of being crowded together. Since all +comparisons are performed on the basis of position alone, crowded +fields may make for ambiguous cross-identifications. A pair of stars +are matched if the difference in their positions is less than a +specified search radius (usually dependent on the astrometric for the +image). If more than one catalog stars is correlated with a star in +an image, the new measurement is added to {\em both} catalog stars, +and a flag is set noting that this observation had a blended IMAGE. +Conversely, if a single catalog star is matched with more than one +image star, both new measurements are added to the one catalog star, +and a different flat is set, noting that this observation had a +blended CATALOG. + +\section{Catalog Update and Cleaning} + +Occasionally, once a large number of images have been added to the +photometry database, a set of programs should be run to clean and +update the CPT files. This may best be done on CPT files after a +night's worth of data is incorporated. These routines define bad +data, identify the USNO catalog stars, and search for asteroids and +other junk. Most of the results from these routines are the addition +of object flags to the Average data structures (Average.code). Most +of these routines process data within a single CPT file at a time. + +\subsection{Markstar} + +The first of the update programs, {\tt markstar}, identifies bad data +and flags it. It makes three types of identifications: bright stars, +ghost stars, and trails. In the first case, it searches the HST guide +star catalog for bright stars within the field of the current CPT +file. Three types of pixels are flagged for bright stars. Any +measurements within a circular region centered on the star are suspect +(the ``halo'' of the bright star). Second, points along the X axis +are likely to contain flux from the diffraction spikes. Finally, +points along the Y axis are likely to have bleeding in addition to the +diffraction spikes. Each of these three regions has a different size +and range which must be defined in the configuration file. The +regions are represented by a representative magnitude and a scale +parameter. For example, the diameter $d$ of the halo for a star with +magnitude $m$ is defined as +$d = \mbox{BRIGHT\_HALO\_SLOPE} (m - \mbox{BRIGHT\_HALO\_MAG})$. The X and +Y axis points are defined by the length of the diffraction spike, with +an equivalent formula, and the width of the spike. Since these are +each independent, there are 8 parameters to define the exclusion +regions of bright stars. One of the current drawbacks of this system +is the reliance on the HST Guide Star photometry. In the HST Guide +Star Catalog, only one photometry band is given, so no color +information is available. The parameters currently used were defined +based on a large number of bright stars, with no adjustment for the +different effective filter system of the Loneos telescope and the HST +GSC. Thus, occasionally very red stars will be several magnitudes +brighter in the real data than reported in the catalog. As a result, +large numbers of bad data points in these halos and spikes may be +missed. + +The second type of bad data flagged by {\tt markstar} are the trails +from satelite and other space junk. This portion of the program +searches for objects which land along a line and which have a high +linear density. Parameters may be adjusted to define the width and +the necessary density or number of points in the line. Objects in +lines are also only identified if all observations come from a common +image. If an object only has one measurement and it is in a trail, +the entire Average is flagged to be bad. However, if an object has +many measurements and only one is bad, the object is only flagged as +having some bad data points. + +The third type of bad dat are ghost star images. The HST GSC is +searched for stars projected across the telescope optical axis, which +is defined by the user. All objects within a region of fixed size is +marked for ghost stars brighter than a specific cutoff magnitude. +Similar to the trails, the average is marked as bad only if all +detections of an object are ghost images. Otherwise, it is noted that +some data may be bad. + +\subsection{Addusno} + +The program {\tt addusno} makes two types of identifications of stars +with the USNO catalog. First, it searches for coincidences within a +small, specified radius. The choice of this radius is somewhat +tricky: it is important to catch the association, but also to avoid +making too many associations. Unfortunately, astrometric errors in +the USNO catalog make it necessary to choose a surprisingly large +radius of 5\asec. To avoid making double matches (matching the two +USNO stars to the same photometry database object), only the USNO +object closest to the object is associated. There are also a +substantial number of stars with significant proper motion between the +USNO epochs and the current date. It is not unusual to see 8-15\asec\ +discrepancy for specific stars. To catch these objects, any objects +which do not match the USNO database, but which have more than a +minimum number of measurements (2 or 3?) are searched for more distant +USNO companions. A large, user-specified radius is used for this +search. Of course, only USNO stars which are not already matched to +the database may be candidates for the proper motion match. This +stage is suceptible to errors in that a slow-moving solar-system +object may be associated with a faint USNO star. For this reason, +objects flagged as high proper motion stars should be taken with some +caution. Of course, during the {\tt addusno} stage, no objects +already flagged as bad by {\tt markstar} will be matched with USNO +stars. + +\subsection{Markrock} + +Once stars have been identified with the USNO database, it is possible +to search for asteroid detections. The program {\tt markrock} +searches within a specific CPT file for objects which are moving along +straight lines in (X,Y,t) space. In order for an object to be found +by {\tt markrock}, it must have three points, each with only single +measurements. The third point must land within a specified distance +of the line projected from the first two data points. These +comparisons are made in the three dimensional space of image position +and time. Also, objects are only accepted if they have a speed less +than a threshold. If an object is moving too fast, it will make a +streak on each individual image and will probably be missed anyway. +Moving objects detected in this way are flagged in the photometry +database and are also written to a Rock database. Currently, the Rock +database just contains ASCII lists of the RA, DEC, Mag, and time for +each detection. In fact, the data are stored in pairs of detections, +with the middle detection saved twice. This format makes it easy to +plot lines connecting the points using the connect-the-dots plotting +style of status/kapa (see below). + +One of the current drawbacks of {\tt markrock} is the insistence on +three observations with only 1 measurement each. If an object is +moving too slowly, two of the dectections of the object may be merged +together. Enough data is stored in the database to extract these +individual measurements, so a more sophisticated search is possible. +It is also currently the case that USNO proper motion stars are +ignored, but some of these will be asteroid detections which are +simply too close to a faint USNO stars (it has to be faint because it +would otherwise be associated with a star in the photometry +database). + +\begin{figure} +\psfig{file=flags.eps,width=9cm} +\caption{\label{flags} +Pictoral representation of the Average.code flags. The lower byte +value is used for specific definitions of object types. +} +\end{figure} + +\subsection{Data Flags} + +The identifications made by the above routines are noted as a set of +flags in the Average.code structure entry. Some of these flags are +mutually exclusive, so a certain bit may have more than one meaning, +depending on the value of other bits. Figure~\ref{flags} shows a +pictoral representation of the meaning of the different bit fields. +The uppermost bit (bit 15) determines if the object may be considered +a ``fixed'' star or something which has no fixed location. If it is +not fixed, and this bit is 1, then there are several options. The +second uppermost bit (bit 14) will be set if this object is one of the +three types of bad data flagged by {\tt markstar}. The lower byte of +code is used to define the type of bad data: ghost, trail, or bleed. +Thus, any datapoints determined to be a satelite trail will have a +code value of 0xc002 (49154 decimal). If an object is a moving +object, but not a bad datapoint, it may be identified as a ``rock'' +(an asteroid) by the program {\tt markrock}. In this case, the flag +for an asteroid is turned on (bit 13). Thus anything identified as a +rock by {\tt markrock} will have a code value of 0xa000. The lower +byte of code is not yet defined for asteroids, but could be reserved +for distinguishing different classes of asteroids (ie, on the basis of +orbital speeds, and so forth). Objects which are fixed may have a +variety of possible flags. First, an object may be identified with a +USNO star (bit 14). It may be found to exhibit variability (bit 13), +implying that the relative photometry routine should ignore it. If +may be a transient object (bit 12). Note that, while an object +associated with a USNO object may not be transient, an object which is +variable may also not be associated with a USNO object. Thus, we need +seperate bits for variable vs transient. Different types of variable +objects may be represented with different values of the lower byte. +For example, a Cepheid may have a specific lower byte code of 0x01, +which an RR Lyra may have a lower byte code of 0x02. Thus a Cepheid +associated with the USNO catalog would have a total code value of +0x6001. Finally, any of these types of non-moving objects may have +some bad individual measurements (bit 12) or may be found to have a +significant proper motion (bit 11). + +\section{Holding it Together} + +The preceeding sections describe the step-by-step analysis and +incorporation of the data from individual images. However, the power +of the Pipeline comes in apply it to large numbers of images. At +LONEOS, we are using four Pentinum II computers to analyse the roughly 200 +images that are observed each night. + +It is best to think of the analysis of images occurring on groups of +images from a given night. Of the set of processes discussed above, +those in section \ref{parallel} are performed on individual images +essentially independently of the other images. The rest of the steps, +both the data incorporation ({\tt addstar} and the database cleaning +routines occur on the entire batch, one image at a time. Thus, the +former routines can easily be run in parallel, but the latter routines +are run in serial to avoid having two programs writing to same part of +the database at the same time. In this model, we would process all of +the images from a night through the parallel stage of the analysis and +then run the serial stage only after the entire night is finished in +the parallel stage. We thus have a natural division of the pipeline +into two stages. + +The implementation of the two stages involves several programs, some +of which are specific to the LONEOS problem, and others which are more +general. There are two layers of programs which enable batch +processing of many images and repeated batch processing over many +nights. At the highest level, are Perl scripts which decide on a new +set of data. At the next level down, these scripts call two main C +programs which run the images of a given night through the pipeline. +A fixed set of files keeps a log of the nights and images that have +been processed, so the programs may know if a given night has already +been analysed. + +\subsection{lastnight} + +The Perl script {\tt lastnight} looks at the disks used by the +observers to write the images from a given night. Images are written +to a directory called /pallas/d[1-6]/NNNNNN where NNNNNN is an +abbreviation for the current night's date in the form YYMMDD (ie, +990223). All of the images are stored in this directory with names +yyyyMMDDxxxxb.fits where yyyy is the full year, MM is month, DD is +date, and xxxx is a sequence number. The 'b' refers to the 'b' CCD, +and the .fits extension is required. The script {\tt lastnight} is +called by a cron job scheduled for 7am on {\tt hebe.lowell.edu}. The +script searches the disks /pallas/d[1-6] for a directory of the right +form (basically any directory beginning with a number). By default, +the program demands that the directory have a name of today's date, +but with the flag -any, it will accept any directory. It then checks +to see if this directory have been already included in the +'processed.log' file. If not, it lists all files with the right +ending (*b.fits) and places these names in a file. It also writes the +name of the directory (along with some other data) in a file +'extracted.log'. When it has successfully ended and identified some +data, it calls the next script, {\tt stage1} and then quits. + +\subsection{slurp} + +An alternative to {\tt lastnight} is the Perl script {\tt slurp}, +which looks for data on the tape archive. This script looks at the +tape.log file and compares it to the file processed.log. It tries to +identify nights listed in the tape.log which have not been included in +the processed.log. If it finds any, it tries to download as much of +the data as will fit on the disk /pallas/d1. This script is called +from within the {\tt stage1} script and is invoked if {\tt stage1} is +run without any data already in extracted.log. + +\subsection{stage1} + +The Perl script {\tt stage1} takes the image list produced by {\tt +lastnight} or {\tt slurp} and passes the list to the controller +program {\tt control1} which performs the parallel analysis. The +script has a few features to decide which files have been analysed +through the pipeline and to what extent. If {\tt stage1} is called, +it should be able to figure out 1) if there is data ready to be +analysed, 2) if the data has been partially analysed, 3) which data +needs to be run through which part of the pipeline, and so forth. +These features are most useful if the analysis crashes in the middle +or needs to be restarted. The {\tt stage1} script is meant to be run +without any arguments and it will figure out what needs to be done. +The one possible argument is the -once flag. This flag tells {\tt +stage1} to analyse the data it finds and quit when it finishes, and +not call another instance of itself. The script writes a log file +called \$scriptdir/stage1.NN.log where NN is a number in sequence +between all runs of {\tt stage1}. A file stage1.count keeps track of +which number NN we are on. When {\tt stage1} has finished the +analysis of a night's data, it writes the date in processed.log to +keep {\tt slurp} and {\tt lastnight} from re-running a night. + +\subsection{stage2} +The serial stage processes are run by the Perl scrip {\tt stage2}. +Normally, this script is always running. It waits for the name of a +night to be added to the processed.log file. Once if finds a new name +in this file, it looks for the appropriate images in the file +YYMMDD.addlist. It then submits the list to the second controlling +program {\tt control2}, which runs the images through {\tt addstar}. +Images which are successfully run through addstar have their *bf.fits +(flattened image) and *.obj (dophot output) files deleted. When the +entire night is done, the rest of the *.obj and *bf.fits images are +deleted by {\tt stage2}, the directory with the *.cmp and *.log files +is tared and gzipped into the file YYMMNN.tgz. The directory is then +deleted. Finally, {\tt stage2} calls the script {\tt cleaning.pl} +which runs the cleanup programs ({\tt markstar - nrphot}) on the +database. After a successful run, {\tt stage2} will automatically +respawn a new copy that waits until more data arrives. + +\subsection{control1 / control2} +These two C-programs perform the tasks of farming the images out to +the different machines for analysis. The two programs are essentially +identical, but {\tt control1} is allowed to use all machines and {\tt +control2} can only use one (hebe.lowell.edu, which has the database +local to it, mostly). The two programs also run a different set of +program. These controller programs start off by executing remote +shells (using ssh) on a list of machines. These shells are then used +to execute the remote commands and are kept open the entire time the +process runs. They maintain a list of images in a set of queues. A +set of rules defines the order in which each image travels through the +queues. Each queue is associated with a single process (flatten, +astrometry, etc). Images which successfully run through a queue are +sent on to the defined 'next' queue, while images which fail are +instead sent to the defined 'fail' queue. In most cases, the 'fail' +queue is a process which marks the image as bad and write the name in +a file, and then puts the image in the 'done' queue. However, it is +possible for the 'fail' queue to be another attempt, such as in the +case of astrometry. In this example, images which fail the default +astrometry are run through a second queue which tries again with the +-loneos flag set (this flag makes the assumption that the header +information about the LONEOS region number for the image is correct, +not the RA/DEC information). + +The connections on the remote machines are handled as child processes +with communication through the stdin and stdout/stderr pipes. +Messages can be sent back and forth though these pipes, and messages +which are received from a process associated with an image are written +to a log file with the name imagename.log (where imagename is the root +name of the image). The only required messages from the program are +messages saying ``SUCCESS'' or ``ERROR''. The programs therefore all +end with either of these words. The programs are started within the +controller as a command to the shell followed by and echo ``PROCESS +DONE''. If the controller sees the message ``PROCESS DONE'' without +either ``SUCCESS'' or ``FAILURE'' (after waiting an appropriately long +time), it assumes the program crashed. Currently, no timeout is +available for the programs. The flat-field process is run in mana and +is a slightly special case. We avoid re-loading the same flat many +times by starting a mana process once and running the flatten +procedures as mana function calls. The images which are analysed +(with an indicator of success or failure at each stage) are written to +a file named YYMMDD.addlist by the {\tt control1} and YYMMDD.outlist +by {\tt control2}. + +\section{Lock files} + +A variety of locking mechanisms are used to avoid overwriting the +database or running more than one version of the controlling +programs. there is a perl program, locks, which allows the user to +set or check the state of the different locks. + +\section{File locations} + +\begin{verbatim} +database: /hebe/d27/database (also in /metis/d27/database as linked directories). + lock files also go in this directory +workspace: /irene/d11/workspace - temporary storage of result files +dumpspace: /pallas/d1 - disk to store data dumped from the archive by slurp. +scriptdir: /metis/d11/logs - all log files +\end{verbatim} + +\section{Visualization -- Status} + +Visualization of the data in the photometry database can best be +performed with the program {\tt status}. This is a command-line +driven program with a math and macro language which makes it easy to +perform complex tasks. + +First, a few notes about the user interface. The interface has an +interaction similar to {\tt tcsh}. The arrows allow editing of +previous commands. You can also use emacs-like commands such as +cntl-a to reach the beginning of the line and cntl-e to reach the end. +There is command and file completion: if you type part of a command +(as the first thing on a line) and then type tab, it will fill in as +much as possible, until the word is not unique. Typing tab twice at +that point will list the possible endings. For any but the first word +on a line, the same thing will happen for the files in the current +directory. It is also possible to type just a fraction of a command, +as long as it is unique. An ambiguous command will list the possible +alternatives. For example: +\begin{verbatim} +status: c +ambiguous command: c ( catalog cgrid clear create cursor ) +\end{verbatim} + +The shell is essentially an interpretive programming language. +Variables are set as follows: +\begin{verbatim} +status: $fred = 10 +\end{verbatim} +Any expression within curly brackets \{\} is assumed +to be an arithmetical expression and is evaluated before the line is +executed. For example: +\begin{verbatim} +echo {$fred*dcos(45)} +\end{verbatim} +would give the response 7.07107. There are math functions cos, sin, +and tan, which operate on radian expressions, and also dcos, dsin, +dtan, which operate on degree expressions. There are also the +equivalent inverse functions: eg., asin and dasin return radians and +degrees, respectively. The help section on Math defines all of the +available math functions. + +\subsection{Miscellaneous Commands} +\begin{verbatim} +! -- system call +? -- list commands +?? -- list variables +echo -- type this line +exec -- system call +exit -- exit program +help -- get help on a function +output -- redirect output to file +quit -- exit program +scan -- scan line from keyboard or file to variable +wait -- wait until return is typed +which -- show command +\end{verbatim} +Most of these are self-explanatory. The command ?? prints the system +variables. The {\tt help} command will provide help on a single +command or, without any arguments, will list all available help +files (this includes general help not associated with a specific +command). + +\subsection{Shell Programing} +\begin{verbatim} +break -- escape from function +for -- loops +if -- logical cases +input -- read command lines from a file +macro -- deal with the macros +\end{verbatim} + +There are several options for programming in {\tt status}. First, a +file which contains a series of commands can be executed with {\tt + input (filename)}. It is also possible to define macros which will +behave much like regular commands. A macro is defined by typing {\tt + macro name} or {\tt macro create name} followed by the commands. +Arguments to the macro are assigned to the variables \$1 .. \$N and +the number of arguments is given by \$0. Macros may be defined in +{\tt input} files, and in fact when {\tt status} is started, it loads +the file {\tt \~/.statusrc} which may contain default macros. Simple +loops and if statements can be performed, and are quite useful for +complex macros. + +``If'' statements are similar in syntax to C if statements, but only +the following logical operators are available: $>$, $<$, $=$, !, $|$, +and \&. Notice that (currently) there is no $>=$ or $<=$ symbol. The +operator ! means ``not equal to'', but cannot be used to negate a +logical value. The operators $|$ and \& have the meaning of ``or'' and +``and'' respectively. Math expresions in the if statement must be +contained in curly braces, as elsewhere. Variables with string values +may use the logical $=$ operator to test if two strings are the same. +``For'' loops are quite simplistic. The form is: +\begin{verbatim} +for var first last delta + (commands) +end +\end{verbatim} +The value of {\tt \$var} will start at the value {\tt first} and increment by +{\tt delta} after each loop. The loop will stop after {\tt \$var} is greater +than {\tt stop}. The value {\tt delta} is optional, with 1 assumed. +The value of {\tt \$var} may be changed during the loop, and if set +beyong the value of {\tt last} will end the loop early. + +\subsection{Vector Plotting} + +\begin{verbatim} +box -- draw a box on the plot +clear -- erase plot +create -- create a new vector +cursor -- get coords from cursor +grid -- plot cartesian grid +hist -- create histogram from a vector +labels -- define labels for plot +limits -- define plot limits +plot -- plot a pair of vectors +print -- write vectors to file +ps -- define labels for plot +set -- vector math +style -- set the style for graph plots +vectors -- list vectors +zplot -- plot scaled points +\end{verbatim} + +In addition to scalar variables, {\tt status} can manipulate and +display 1-D vector variables. Many of the commands which extract data +from the photometry database place the data in vectors as well as +plotting them. A vector can also be created based on a number +sequence with the command {\tt create name Nelements start delta}. +The resulting vector has $Nelements$ entries, starting at a value of +$start$ and running until $start + delta*Nelements$. If $delta$ is +0.0, all elements will have the value of $start$. A histogram of a vector +may be made with the command {\tt hist}, which creates a new vector +containing the histogram of the first vector. The data range and bin +size of the histogram are defined in same way as with create. This +makes it easy to create the index vector that goes with a histogram +vector: +\begin{verbatim} +hist y Ny 1 100 0.1 +create dx 1 100 0.1 +\end{verbatim} +The above will create a histogram of y in Ny and the index in dx. +Plotting this with {\tt plot dx Ny} will show the histogram. + +Vector math is performed with a command of the form {\tt set new = + (expression)}. The expression is some math function employing +vectors and scalars. A complete listing of the math operators +available in {\tt set} can be found in the help for {\tt set}. + +Once vectors are defined, they may be plotted. A pair of vectors can +be plotted against each other if they have the same number of entries. +The plotting is performed on the graphics window, Kapa. There are +actually several graphics windows available to {\tt status}, any of +which may be used to plot at any time. Some of the more complex +operations default to either graphics window 0 or 1, depending on the +context. Except for those functions with a pre-defined window, all +plotting functions apply to the current graphics window unless an +option {\tt -n N} is given to specify a different window. The +plotting style is determined by the command {\tt style} which can set +the line width, the line type (solid, dashed, dotted, etc), the point +type (box, cross, etc), the point size, the color, and whether a pair +of vectors is plotted as a sequence of points, a set of connected +lines, or a histogram. Some functions which make plots use their own +styles, as discussed below. The function {\tt limits} lets the user +set the range of the plot axes, or check the current setting. The +command {\tt plot} will plot a pair of vectors on the current graphics +window using the current plotting style for that window. The command +{\tt zplot} will plot a pair of vectors with the point size scaled by +a third vector, with maximum and minimum point sizes representing +specified values. The {\tt cursor} command goes the other way: this +command puts the Kapa window in cursor mode and waits for input from +Kapa. The user can then type any alphanumeric key on the graphics +windows and will be told both the pointer location (in the graphics +coordinates) and will have the coordinates stored in {\tt status} +variables. For example, by typing ``1'' in the sky display window, +the RA and DEC of the pointer are stored in the variables {\tt \$R1} +and {\tt \$D1}. This command can be used to let the user define +locations or regions of interest on the Kapa window. (Future addition: +{\tt button}, which does the same with the mouse buttons). + +\subsection{Database Functions} + +\begin{verbatim} +gcat -- get catalog at location +gimages -- get images at location +gstar -- get star statistics +extract -- extract average vectors from catalogs +mextract -- extract measurement vectors from catalogs +imstats -- plot image statistics +imextract -- extract image vectors from database +lcat -- list catalogs in display region +cmatch -- match two catalogs +\end{verbatim} + +There are a variety of other commands which directly refer to the +photometry database. Some of these functions extract data of various +types from the database, others perform more complex plotting +operations. The commands listed above are those which simply extract +data from the database. The first three list information relevant to +a specific RA, DEC location on the sky: {\tt gcat (RA) (DEC)} lists +the catalog at the specified location and places the name in the +variable {\tt \$CATNAME}, {\tt gimages (RA) (DEC)} lists all images +which overlap the specified location, , {\tt gstars (RA) (DEC) + (RADIUS)} lists data about the stars within a specified radius of +the specified location (all numbers above are given in decimal +degrees). Similarly, {\tt lcat} lists the catalogs in the region. +Imstats lists statistics about each image + +The next three commands extract a specific piece of information from +the photometry database and places it in a vector. First, {\tt + extract} will extract average values for each star and place it in a +vector. Next, {\tt mextract} will extract measurement values for each +star and place it in a vector: as a result a single star may have +multiple entries in the measurement vectors. Finally, {\tt imextract} +will extract image statistics into vectors (not yet implemented). + + +\begin{verbatim} +catalog -- plot catalog stars +cgrid -- plot sky coordinate grid +cplot -- plot vectors in sky coordinates +czplot -- plot scaled vectors in sky coordinates +images -- plot image boxes +imdense -- image density plot +lcurve -- plot lightcurve for a star +pcat -- plot catalog boundaries +region -- define sky region for plot +resid -- plot residuals +simage -- plot stars in an image +\end{verbatim} + +There are two types of database plotting functions: those that display +or refer to the spatial charateristics of the data and those that +refer to other types of charatersitics, such as the time domain. The +graphics window 0 is reserved for all plots of objects on the sky. +The command {\tt region} defines the current sky coordinates for plots +in graphic window 0. The command {\tt pcat} plots the outline of all +photometry database files which are within the currently defined +region (and by default, only those with data). {\tt images} plots the +outline of the images in the image database, while {\tt imdense} shows +the number of images at a location by randomly spacing dots within the +boundary of the images. The command {\tt cgrid} +draws a grid in celestial coordinates on the for the current region. + +The most complex, but also one of the most useful command is {\tt + catalog}, which plots the positions of stars in the photometry +database (and others) on the sky. There are many options to this +command. One set allows the user to plot stars from the photometry +database (the default), from the HST GSC, or from an ASCII text file +with RA, DEC, and Mag in specified columns. If the ASCII file has a +fixed number of bytes per line, the data can be more quickly loaded. +The size of the points may be scaled by the star magnitude, by the +number of observations of the star, or by the number of missing +datapoints for the star. In addition, points may be plotted only if +they land in specified magnitude ranges, or with specified numbers of +measurements, or missed measurements. Also, objects may be plotted +only if they have a specified Average.code, so that only asteroids or +only perfect stars may be plotted. The plotted vectors may be saved, +if desired, and the source catalog epoch may be specified as different +from J2000 (only valid for ASCII data). + +Several other commands relate to non-spatial charateristics of images +and stars. {\tt lcurve} will plot a light curve for all stars within +some radius of a point. {\tt resid} plots the photometry residuals +for a particular region file. + +\section{Some Examples} + +\begin{figure} +\psfig{file=allsky.ps,width=16cm} +\caption{\label{allsky} +Map of the entire sky, and images added to database. +} +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location +of the images currently in the database. This picture was made with +the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 gls +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=catalog.ps,width=9cm} +\caption{\label{catalog} +Comparison between HST GSC and photometry database astrometry. +} +\end{figure} + +Fig.~\ref{catalog} shows an example comparison of the photometry +database star positions and the HST Guide Star Catalog star +positions. The crosses are from the photometry database while the +boxes are from the HST GSC. The size of both points is a function of +brightness. This plot was made with the following commands (starting +from the previous image): +\begin{verbatim} +status: cursor (typed 1 on region of interest) +1 137.097858 22.698305 +q 137.097858 22.698305 +status: region $R1 $D1 0.5 TAN +status: cgrid +status: box +status: style -pt 0 +status: gcat $R1 $D1 + 0 n2230/1951.cpt * +status: catalog n2230/1951.cpt -m 12 18 +status: style -pt 2 +status: catalog n2230/1951.cpt -m 12 18 -g +\end{verbatim} + +\begin{figure} +\psfig{file=lightcurve.ps,width=9cm} +\caption{\label{lightcurve} +Lightcurve for several image stars. +} +\end{figure} + +Fig.~\ref{lightcurve} shows an example lightcurve from three stars in +the above field. The x axis shows the time in seconds, which the y +axis shows the star magnitudes. This plot was made with the following commands (starting +from the previous image): +{\small +\begin{verbatim} +status: cursor +1 137.062562 22.791860 +q 137.062562 22.791860 +status: lcurve $R1 $D1 0.01 -l +status: box +status: gstar $R1 $D1 0.01 +read 11318 stars from catalog file /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt (22462 total measurements) +508 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.068619 22.785400 18.438000 1 (2) 1.000000 1.000000 +498 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.062820 22.792747 12.882000 3 (0) 1.000000 1.000000 +501 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.064499 22.800817 15.480000 3 (0) 1.000000 1.000000 +\end{verbatim}} +The last lines show the database information on the three stars. + +\section{Software Distribution} + +\section{Programming Details} + +\section {Comparison with USNO catalog} + +I have made several comparisons between the USNO catalog and results +from running images from Loneos through the photometry pipeline. In +general, there is a good agreement between the astrometry of the two +datasets. There are specfic areas where the two disagree, however. +First, as discussed above, the USNO star positions may be +significantly different from the observed star positions simply +because of proper motion. The first picure in the ``Comparisons +between Loneos + USNO'' pages shows several stars with significant +proper motion. These are currently being flagged in the photometry +database as discussed. Other differences evident in this first +picture include the classification of bright stars and the splitting +of faint and bright stars. In the upper right of the image, two +bright stars are visible, one of which is actually a double. The USNO +catalog does a good job of detecting both, but it also detects and +includes as stars several diffraction spike points. The HST GSC only +detected a single point and did not split the star at all. The Loneos +data in our pipeline did not detect the star at all due to saturation. +A star in the upper left corner shows the ability of {\tt dophot} in +the Loneos pipeline to split double stars. This star is, on close +inspection, clearly a double. The USNO catalog only detects one. The +Pipeline not only splits the two stars, but it also makes the correct +cross-identification of the stars. + +The second set of pictures shows the ability of the Loneos pipeline to +deal with small galaxies. There are two small galaxies in this field, +both of which are broken into 4 or 5 objects by the USNO catalog, but +maintained as single objects in the Loneos pipeline. In another +example (not shown) a bright star within the disk of a small galaxies +was clearly split from the galaxy by the pipeline (and also by USNO). + +\appendix +\section{Parameter file params.txt} +\begin{verbatim} +# Configuration file for Loneos analysis pipeline + +# important files and directories +GSCFILE /host/pikake/d1/gene/gsc/GSCregions.tbl +GSC_DIR /host/pikake/d1/gene/gsc +CATDIR /host/radon/rad3/gene/photcat2 +IMAGE_CATALOG /host/radon/rad3/gene/photcat2/Images.dat +IMAGE_CATALOG_TEMPLATE /host/radon/rad3/gene/photcat2/template.cat +CATALOG_TEMPLATE /host/radon/rad3/gene/photcat2/template.cat +ROCK_CATALOG /host/radon/rad3/gene/photcat2/Rocks.dat +USNO_CDROM /host/tycho/mnt/md0/kevin/zones +LONEOS_REGIONS /host/pikake/h1/gene/loneos3/regions.map + +# parameters for "fstat" +# instrumental mag zero point +ZERO_PT 24.5 +DOPHOT_CHAR_LINE 129 +DOPHOT_TYPE_FIELD 5 +DOPHOT_AP_FIELD 91 +DOPHOT_PSF_FIELD 52 +MIN_SN_FSTAT 6.0 + +# parameters for "gastro" +DEFAULT_RADIUS 32.0 +MINIMUM_RADIUS 1.5 +MIN_ERROR 0.6 +MIN_PRECISE 0.12 +CCD_PC1_1 1 +CCD_PC2_2 -1 # for loneos +CCD_PC1_2 0 +CCD_PC2_1 0 +ASEC_PIX 2.82 +NFIELD 3.0 +MMIN 6.0 +ROT_ZERO 0 +dROT 1.0 +NROT 2 +POLAR_AXIS_RA -90.0 +POLAR_AXIS_DEC 89.88 +RA_OFFSET -90.0 +DEC_OFFSET 0.0 + +# parameters for "addstar" +# airmass extinction coefficient (in mag / airmass) +NSIGMA 3.0 +ALPHA 0.1 +XOVERSCAN 60 +YOVERSCAN 50 + +# parameters for "controller" +IN_DIR /host/pikake/h1/gene +OUT_DIR /host/pikake/h1/gene/loneos4/ +DOPHOT_PARAMS /host/pikake/h1/gene/src/ohana/config/default_parameters +FLATTEN_SCRIPT /host/pikake/h1/gene/src/ohana/config/test.pro +NEW_IMAGES images.dat +USERNAME gene +PASSWORD gene + +# here we list all available machines +NMACHINES 2 +MACHINE0 pikake.astro.washington.edu +MACHINE1 mildew.astro.washington.edu + +# here we define which machines which are always free +fMACHINE0 1 +fMACHINE1 0 + +# here we define the machine which runs the database +LOCAL_MACHINE 0 + +# parameters for "status" +TIME_REFERENCE 90/01/01 00:00:00 + +# parameters for "nrphot" +# make this one small to boost change of getting an early region +TAU 3.0 +SCATTER_LIM 15.0 +MAG_LIM 17.0 +IMAGE_SCATTER 0.075 +NIMAGE_SCATTER 1.5 +STAR_SCATTER 0.05 + +### parameters for "markstar" +SEARCH_RADIUS 360 # region for initial trail hunt, in arcsec +TRAIL_WIDTH 5.0 # expected trail width, in arcsec +NANGLE_BINS 180 # Number of angular bins in 180 degrees +NPTSINLINE 5 # minimum points needed for trail +MIN_DENSITY 0.025 # minimum linear density in star/arcsec +SPACE_SIGMA 10.0 # how much denser than average for trail? +# parameters defining bright star exclusion regions +BRIGHT_XTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_XTRAIL_MAG 9.5 # faintest sat. star +BRIGHT_XTRAIL_SLOPE -80.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_YTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_YTRAIL_MAG 11.0 # faintest sat. star +BRIGHT_YTRAIL_SLOPE -400.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_HALO_MAG 8.5 # faintest sat. star +BRIGHT_HALO_SLOPE -28.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +# parameters which define the ghosts +GHOST_MAG 7.5 +GHOST_RADIUS 200 # in arcsec +OPTICAL_AXIS1 2154.0 +OPTICAL_AXIS2 2193.0 + +# parameters for "addusno" +USNO_RADIUS 5.0 # in arcsec +USNO_PROPER 20.0 # in arcsec +USNO_RED 1000 # code for USNO Red data +USNO_BLUE 1001 # code for USNO Blue data +# /mnt/cdrom + +# parameters for "markrock" +ROCK_RADIUS 2.0 # in arcsec +ROCK_MAX_RADIUS 20.0 # in arcsec +ROCK_MAX_SPEED 0.1 # in arcsec / sec +\end{verbatim} + +\end{document} Index: /trunk/Ohana/doc/pipeline/pipeline3.tex =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline3.tex (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline3.tex (revision 2508) @@ -0,0 +1,1463 @@ +%\documentstyle[astro,psfig]{article} +\documentclass{article} +\usepackage{psfig} +\usepackage{astro} +\usepackage{times} +\GoodMargins +\newcommand{\note}[1]{({\bf #1})} +\title{Variability Pipeline} +\author{Eugene Magnier} +\begin{document} +\maketitle + +\section{Overview} + +The Loneos Pipeline provides automated reduction of images, including +photometry and astrometry, with the final data stored in a photometry +database organized by star position on the sky. + +Fig.~\ref{pipeline} shows a schematic of the analysis pipeline. +Images are written to the disk from the camera by the software which +runs it, {\tt astrocam}. The resulting images are FITS files (which +may be arbitrarily sized) and have header keyword parameters RA and +DEC specifying rough celestial coordinates. Images are first analyzed +by {\tt dophot}, which produces instrumental photometry and pixel +coordinate positions for stars and other objects in the images, +writing files with a particular format, called OBJ files in this +discussion. The results from {\tt dophot} are cleaned up by a program +called {\tt fstat} which merges the header information from the image +with the most relevant parts of the OBJ file, creating a new file with +the CMP format. Astrometry is performed on the CMP file with the +program {\tt gastro}. {\tt Gastro} makes a comparison with an +astrometric reference database and only changes CMP header keywords as +a result. Finally, the processed image CMP file is added to the +photometry database with the program {\tt addstar}. The photometry +database is maintained in a large number of files (CPT files), each +representing a small region of sky. There is no relationship between +the image locations and the boundaries of CPT file. All of these +processes occur on individual image, and would typically be performed +while the observations are occuring, or during the day following, but +may be performed at any time to add new or archived data to the +database. + +Once large batches of images have been processed, a series of programs +should be run to clean up the photometry database. Typically, these +routines would be run at the end of a night on all of the CPT files to +which new data were added. First, {\tt markstar} identifies bad data +(the trails of satelites, the bleeding columns from stars, ghost +images), and flags these data points appropriately. Second, {\tt + addusno} matches the catalog stars with stars from the USNO database +and adds the USNO magnitudes to the photometry database. Next, {\tt + markrock} searches for likely asteroids in the CPT file, marks them, +and writes the asteroid observations to an asteroid database. It also +notes measurements which are likely to be noise and cosmic rays. +Finally, {\tt nrphot} determines relative photometric calibrations for +observations in a set of CPT files. + +With the exception of {\tt dophot}, all of the programs in the +pipeline refer to a single configuration file to determine the large +number of parameters necessary in the process. The default parameter +file is hardwired into the programs, but an alternate configuration +file may be specified with the environment variable {\tt + PIPE\_CONFIG}. This option makes independent processing of multiple +data sources trivial. For example, if the pipeline is to be used for +both Loneos and SDSS image analysis and display, there could be two +configuration files (eg, {\tt loneos.txt} and {\tt sdss.txt}), one for +each data source. Several important parameters would be different +between these two, for example, the initial plate scale guess for the +astrometry routine, or the parameters which define the rough chip +orientation. This scheme allows for analysis of different types of +images, with a common destination photometry database, or for +independent photometry databases. + +\begin{figure} +\psfig{file=schematic.eps,width=9cm} +\caption{\label{pipeline} +Schematic of the photometry pipeline. Rectangles represent programs, +while ovals represent data products. Arrows show the direction of +travel of information. The flatten routine is a filter whose position +may change as the pipeline matures. +} +\end{figure} + +Data from different sources can be processed by the pipeline and +stored in the same photometry database, since all photometry is tagged +with a source ID in the database (as are the images in the image +database). This allows us to include, for example, the USNO catalog +magnitude measurements (or a subset) in the photometry database for a +quick comparion. Different sources may be considered different +filters, so color-color and color-magnitude diagrams could be +extracted by specifying choices for source catalog. Currently each +object has a single primary photometric system, and individual +observations may be combined to determine that average (eg, if they +are different photometry sources with roughly the filter bandpass) or +they may be ignored when calculating relative magnitudes (eg, the USNO +magnitudes would be ignored). This may be expanded upon by defining +multiple average magnitudes and allowing different measurements to +contribute only to the appropriate average. The determination of the +photometry code for a particular image is currently not well defined. +There are several points in the analysis where the user (or the +controlling program) may specify the photometry source. A table +should be kept in the photometry catalog to define the various +photometry codes. + +\section{Image Analysis Pipeline} +\label{parallel} +\subsection{Photometry -- Dophot} + +Photometry is performed using a varient of {\tt Dophot}. The version +of dophot is based on Dophot 2.0, but it has been adapted to +streamline processing of many files. First, with this version of +dophot, any FITS format data is safely read in. Instead of compiling +to a fixed maximum size, only the memory needed to contain the image +matrix is allocated. The image is also converted to REAL*32 from +whatever BITPIX for processing. Finally, only a single, complete +parameter file is loaded by dophot. There is no option for a modified +and a default parameter file. The parameter file to be used is +defined, along with the input image and the output object list on the +command line when the program is run: \\ +{\tt dophot image.fits image.obj parameter\_file} \\ +To achive this, the code from dophot was wrapped within a C program +which interprets the command line arguments, allocates the necessary +memory, and reads in the image. The rest of the processing is +performed by dophot in the standad manner. Notice that we use a +consistent naming convention through the pipeline reductions. The +image is called {\tt foo.fits}, while the resulting dophot-produced +photometry list is called {\tt foo.obj}. + +\subsection{Cleanup -- fstat} + +After an image is processed by dophot, the object file ({\tt foo.obj}) +is converted to a more-compact, more-complete file called a CMP file, +with a name of the form ({\tt foo.cmp}). This conversion is done with +the program {\tt fstat}. The CMP file consists of the FITS header +from the original image ({\tt foo.fits}), with some additional +keywords to be used at later stages in the analysis, followed by an +ASCII list of the interesting data from ({\tt foo.obj}). In this +list, types 6 and 8 are excluded, and only the following values are +kept: +\begin{verbatim} +X Y Mag dMag t log(sky) +1342.0 106.1 14.166 000 4 3.2 +\end{verbatim} +Objects with a signal-to-noise ratio lower than a specified cutoff +({\tt MIN\_SN\_FSTAT}) are also excluded. Some general information +about the image is derived by {\tt fstat} (FWHM, saturation and +completeness limits, number of each dophot type) and stored as +keywords in the header. The photometry source must be specified at +this stage as one of the command line arguments, though the value used +may be overridden in one of the stages belowe. The resulting file can +now stand on its own without reference to the original image. The +keywords used by {\tt fstat} include the minimum signal to noise, a +rough guess at the zero point ({\tt ZERO\_PT}), and four numbers +defining the format of the {\tt dophot} OBJ file. + +\subsection{Astrometry -- gastro} + +Astrometry is performed automatically by the program {\tt gastro}. +{\tt gastro} loads a CMP file and determines an initial guess for the +center coordinates (based on the RA and DEC header keywords). The +program also uses the configuration information about the plate scale +and rough orientation of the image to get a close to the final +solution. Also, the true sky position of the telescope pole may be +defined to allow astrometry on images taken close to the pole. The +comparison is made with the astrometric catalog, which may be the HST +Guide Star Catalog, or it may be any source if the data is placed in +the correct format. + +\section{The Photometry Database} + +\subsection{Photometry File Format} +The photometry database consists of a large number of photometry files +representing sections of the sky and a file containing information on +the images incorporated in the database. These files are called +alternatively ``region files'' or CPT files (because they have the +extension {\tt .cpt}). The photometry files are sorted by Declination +into different directories, each consisting of a band 7.5\degree\ +wide. These directories have names like n0730, representing the band +starting at a Declination of +07:30. The image database (Images.dat) +is stored in the upper level directory of this directory tree. In +fact, the locations of the Images.dat file, the photometry database, +and such are all flexible and may be changed by altering the +configuration file. + +The name and location of each sky region comes from the Hubble Space +Telescope Guide Star Catalog. There is a file, GSCregions.tbl, which +can be used to find the region file appropriate for any location on +the sky. A number of routines exist to make such a query. + +Each photometry file in the photometry database contains all +observations and all average values for all stars observed within the +appropriate region on the sky. The identity of stars is essentially +determined by the RA and DEC coordinates. In very crowded regions, +the end user may need to double check that a neighboring star is not +contaminating individual detections (see the discusion on {\tt + addstar} below). + +The photometry files are stored in a binary format, both to reduce the +volume and to speed access. As a result, the interpretation of the +data is machine dependent: little endian machines need to swap the +data intelligently. Fortunately, this operation is taken care of +appropriately, if the programs are compiled with the BYTESWAP option +set as needed, which is automatically set by the Makefile in the case +of a Linux machine. For reference, Suns, SGI, and HP are all big +endian, while PCs and DECs are little endian. The automatic +conversion takes place by using the funcions {\tt Fread} and {\tt Fwrite} +to substitute for the standard C library {\tt fread} and {\tt fwrite} +functions. {\tt Fread} and {\tt Fwrite} are told the datatype being read, +and they know the layout of the bytes for a particular datatype. +Anytime the definitions of the relevant structures (see below) are +changed, {\tt Fread} and {\tt Fwrite} must be updated. Fortunately, there +is only one file ({\tt Fread.c}), and it has a single byteswapping +section, making it easy to adjust the code appropriately. + +The format of a single photometry database file consists four +sections: +\begin{itemize} +\item {\bf header} -- this is a standard FITS header with room for comments + about the file or whatever. Since the rest of the file is not a + standard FITS file, the SIMPLE keyword is set to False. The three + necessary keywords are NSTARS, NMEAS, and NMISS, which define the + sizes of the next three sections. +\item {\bf average} -- this section contains all average measurement + quantities for each star in the file. There are NSTARS entries, and + each entry is the data from a structure of type Average (defined in + loneos.h, and discussed in more detail below). Thus, the total size + in bytes of this section can be found by NSTARS*sizeof(Average). +\item {\bf measure} -- this section contains the individual + measurements for each star in the database. There are NMEAS + entries, and like the average section, each entry is a single + structure, in this case of type Measure (in loneos.h, see below). + All measurements of a single star are consecutive. The number of + measurements for a single star is defined by average.Nm and the + starting entry for a single star is given by average.offset. + Therefore, the first measurement of a specific star average[500] + would be given by measure[average[500].offset], while the second is + measure[average[500].offset + 1], and so forth. +\item {\bf missing} -- this section contains references to all missing + measurements of a star. This means all of those occasions when an + image enclosed the location of a star in the database, but no star + was detected on the image. Similar to the case of the measure + section, each star has a number of missing entries (which may be 0) + given by average.Nn, and the first missing entry is given by + average.missing. Each missing entry value is again a structure of + type Missing. Currently, the only data in the structure is the time + of the observation, which allows for an unambiguous recovery of the + image where the star was missed. +\end{itemize} + +\subsubsection*{Average Structure defined in loneos.h} +\begin{verbatim} +typedef struct { + float R; /* RA in decimal degrees */ + float D; /* DEC in decimal degrees */ + short int M; /* thousandths of mag (-32.000 to 32.000 valid range) */ + unsigned short int Nm, Nn; /* number of measurements, missing */ + short int Xp; /* scatter in 1/100 arcsec (-327.67 to +327.67 valid range) */ + short int Xm; /* 1000*log(chisq) for magnitude measurement */ + unsigned short int code; /* an ID code (ie, star, ghost, satelite, etc) */ + signed int offset; /* offset to first measurement */ + signed int missing; /* offset to first missing obs */ +} Average; /* 28 bytes / Average */ +\end{verbatim} +The structure above defines the average data stored in the photometry +database for each star. Most of the entries are self-explantory, but +some need a bit of clarification. The two $\chi^2$ entries ({\tt Xp} +and {\tt Xm}) are the $\chi^2$ values of the average position and the +average magnitude. These are stored as logarthmic quantities because +high dynamic range is not needed. For the magnitude, the assumption +is that the star brightness is constant and the $\chi^2$ incorporates +the photometric errors on each measurement. For the postion, the +a default constant error is used \note{True?} and again the assumption +is that the postion is constant. As discussed above, the values of +{\tt Nm} and {\tt Nn} give the number of measurements and missing +observations for this star, while the entries {\tt offset} and {\tt + missing} point to the first {\bf measure} and {\bf missing} entry +for this star. The entry {\tt M} is the current best average +magnitude solution for this star (see the section on Relative +Photometry). Finally, the entry {\tt code} is an ID code for each +object, which may define a variety of things about the object. For +example, if the object is known to be variable, or if the object is +associated with a USNO star, or if the object is a measurement of an +asteroid. See below for details of these definitions. + +\subsubsection*{Measure Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + short int dR, dD; /* 1/100 of arcsec (-327.67 to +327.67 valid range) */ + short int M; /* thousandths of mag (-32.767 to 32.767 valid range) */ + short int Mcal; /* image cal mag, thousandths of mag (-32.767 to 32.767 valid range) */ + unsigned char dM; /* thousandths of mag (0.000 to 0.255 valid range) */ + char dophot; /* dophot type (1-9) */ + unsigned short int source; /* code to identify photometry source */ + unsigned int t; /* time in seconds (0 - 143 years valid range) */ + unsigned int average; /* reference to corresponding average entry */ + /* upper byte of Measure.average stores flags: + limit of 16,777,215 stars (Naverage) in a file (=0xFFFFFF) + flags: average & 0x1000000 */ +} Measure; /* 20 bytes / Measure */ +\end{verbatim} +} +The structure above defines the individual measurement data stored for +each observation of each star. The first two entries, dR and dD, are +the difference between the average coordinates for this star and the +coordinates determined for this observation. Clearly, if this +residual is too large, the individual measurement will not be +successfully matched with the average star position. {\tt M} and {\tt + dM} are the instrumental magnitude and error for this measurement, +after correction for the exposure time and a zero point (defined in +the configuration file as ZERO\_PT). The entry {\tt t} determines the +time of the individual observation. This is important not only for +timing purposes, but also to determine the source image for this +observation. The time and the entry {\tt source} uniquely determine +the image in the image database that generated this measurement. +Furthermore, this correspondence allows for determination of the pixel +coordinates on the chip, by going throught the image astrometry stored +in the Image database (see below). The entry {\tt source} is a code which +identifies the particular CCD/Telescope/Filter setup. Each unique +{\tt source} should be treated as an independent filter set for the +purposes of accurate relative photometric comparisons. This entry +also allows external catalog data, such as the USNO or Sloan +photometry, to be added to the database, as desired. The entry {\tt + average} is a pointer back to the {\bf average} section to allow +determination of the star from the measurements, in addition to the +reverse. The bits of the upper byte of this value are reserved for +flags used to define particular situations for this measurement. +Possible values are: +\begin{itemize} +\item 1 = BLEND\_IMAGE - the star on the image matched more than one + catalog star +\item 2 = BLEND\_CATALOG - the star in the catalog matched more than + one image star +\item 4 = UPPER\_LIMIT - \note{not really used?} +\item 8 = CALIBRATED - relative photometry has been performed at least + once. +\end{itemize} +The entry {\tt dophot} stores the dophot type for this particular +measurement (1-9: four extra bits in this field). +Finally, the value {\tt Mcal} determines the photometric calibration +of this specific measurment. This value is an offset appropriate to +this image (and if needed, this point in time and this chip position) +to bring the observations of the stars on multiple images to a common +system (see the section on relative photometry). + +\subsection{Image Database} + +All images for which photometry has been included in the photometry +database also have entries in an image database. The image database +is (currently) a single file in the upper directory of the photometry +database. The name of the file is determined by the IMAGE\_CATALOG +entry in the configuration file, and is currently set to Images.dat. +Like the photometry database files, the image database consists of a +FITS header followed by binary data. There is only one type of binary +data: each image has an entry, which is the data from a structure of +type Image (loneos.h). The number of images in the file is determined +by the {\tt NIMAGES} keyword. If the number of images becomes too +large, extrapolation of the system to a set of image database files +with a reference list is quite straightforeward, and could be modeled +on the organization of the region files (eg, one image database file +for each Declination directory). + +\subsubsection*{Image Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + Coords coords; /* 120 bytes */ + unsigned int tzero; /* readout time row 0 in sec (0 - 142 years valid range) */ + unsigned int nstar; /* number of stars on image */ + short int secz; /* thousanths of airmass (valid range -32.000 -- 32.000) */ + short int NX, NY; /* dimensions of image */ + short int apmifit, dapmifit; /* aperture correction and error in thousandths of mag */ + short int source; /* identifier for CCD (each ever used will have a unique letter) */ + short int Mcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int dMcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int Xm; /* 10*log(image chi-square) */ + char name[32]; /* name of original image */ + unsigned char detection_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char saturation_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char cerror; /* astrometric error: 1/50 of arcsec (0 -- 5.12 valid range) */ + unsigned char fwhm_x, fwhm_y; /* PSF terms in 25*arcsec (valid range 0.0 -- 10.2" ") */ + unsigned char trate; /* 10000 * scan rate in sec/pix (0 -- 0.0256 valid range, + typically 0.0146. this is used only to determine the time + of the observation, not to find the coordinates. 1 byte + gives 0.11 sec accuracy */ + float exptime; /* exposure time, seconds */ + char code; /* flag to mark an image as bad or whatever */ + char dummy[21]; /* extra space for the future */ + short int order; /* number of terms used for Mrel */ + short int Mx, My; + short int Mxx, Mxy, Myy; + short int Mxxx, Mxxy, Mxyy, Myyy; + short int Mxxxx, Mxxxy, Mxxyy, Mxyyy, Myyyy; +} Image; /* 240 bytes / Image */ +\end{verbatim}} +The structure above lists all of the data stored for each image. A +substantial amount of information is stored for each image. First, +the astrometric calibration information is stored in a {\tt Coords} +structure (see coordinate systems, above). The upper and lower limits +of the photometry in the database (in instrumental magnitudes) is +stored in {\tt detection\_limit} and {\tt saturation\_limit}. These +values are useful for comparison for those stars where observations +are missing. The relationship between time and pixel coordinates is +given by {\tt tzero} and {\tt trate}. Drift images will have a scan +rate stored in {\tt trate}, while staring images will have a value of +0.0 for this entry ({\tt tzero} then applies to the whole image). +Other important parameters are kept: the astrometric error ({\tt + cerror}), the airmass ({\tt secz}), the exposure time ({\tt + exptime}), the number of stars detected ({\tt nstar}), the FWHM +values ({\tt fwhm\_x, fwhm\_y}), the image name ({\tt name}), and the +size of the image ({\tt NX, NY}). The entry {\tt source} is used as +above to define the CCD/Telescope/Filter used for this observation. +The entry {\tt code} is used to flag images in various ways. +Currently, only one bit is used, to define if the image has +unacceptable photometric scatter. Seven extra bytes are stored for +future additions. + +\subsection{Data Incorporation -- {\tt addstar}} + +Images which are completely processed ({\tt dophot, fstat, gastro}) are +then incorporated into the photometry database with the program {\tt +addstar}. This program decides which region files are appropriate +for this particular image, then one-by-one adds stars from the image +to the appropriate region file. Stars already in the catalog are +matched with stars in the new image purely by a positional comparison. +In order to avoid the difficulty of comparisons in the RA and DEC +coordinate frame, a cartesian projection is performed. The stars from +the image being processed and the database stars in the same area are +projected onto a tangent plane and positional comparisons made in this +(locally cartesian) coordinate frame. This avoids the dangerous +singularities at the pole and also makes the RA 0,360\degree\ boundary +a trivial problem. + +Several choices must be made in the comparison process. If a star in +the catalog is matched with a star in the image, the new measurements +of that star are added to the database. If a star in the database +lies in the field of the image, but is not detected, this information +is added to the list of missing data. The only data stored in this +case is the time and source, which is sufficient to unambiguously +identify the source image from the image database. This allows later +programs to find relevant statistics from the image database, if +necessary. If a star is detected in the image, but is not already in +the database, a new entry is added to the database. In addition, the +same ``missing data'' from all previous observations images which have +covered this location (ie, images already in the image database) are +included. This last step is necessary so that the image processing +order is not important. + +Stars also run the danger of being crowded together. Since all +comparisons are performed on the basis of position alone, crowded +fields may make for ambiguous cross-identifications. A pair of stars +are matched if the difference in their positions is less than a +specified search radius (usually dependent on the astrometric for the +image). If more than one catalog stars is correlated with a star in +an image, the new measurement is added to {\em both} catalog stars, +and a flag is set noting that this observation had a blended IMAGE. +Conversely, if a single catalog star is matched with more than one +image star, both new measurements are added to the one catalog star, +and a different flat is set, noting that this observation had a +blended CATALOG. + +\section{Catalog Update and Cleaning} + +Occasionally, once a large number of images have been added to the +photometry database, a set of programs should be run to clean and +update the CPT files. This may best be done on CPT files after a +night's worth of data is incorporated. These routines define bad +data, identify the USNO catalog stars, and search for asteroids and +other junk. Most of the results from these routines are the addition +of object flags to the Average data structures (Average.code). Most +of these routines process data within a single CPT file at a time. + +\subsection{Markstar} + +The first of the update programs, {\tt markstar}, identifies bad data +and flags it. It makes three types of identifications: bright stars, +ghost stars, and trails. In the first case, it searches the HST guide +star catalog for bright stars within the field of the current CPT +file. Three types of pixels are flagged for bright stars. Any +measurements within a circular region centered on the star are suspect +(the ``halo'' of the bright star). Second, points along the X axis +are likely to contain flux from the diffraction spikes. Finally, +points along the Y axis are likely to have bleeding in addition to the +diffraction spikes. Each of these three regions has a different size +and range which must be defined in the configuration file. The +regions are represented by a representative magnitude and a scale +parameter. For example, the diameter $d$ of the halo for a star with +magnitude $m$ is defined as +$d = \mbox{BRIGHT\_HALO\_SLOPE} (m - \mbox{BRIGHT\_HALO\_MAG})$. The X and +Y axis points are defined by the length of the diffraction spike, with +an equivalent formula, and the width of the spike. Since these are +each independent, there are 8 parameters to define the exclusion +regions of bright stars. One of the current drawbacks of this system +is the reliance on the HST Guide Star photometry. In the HST Guide +Star Catalog, only one photometry band is given, so no color +information is available. The parameters currently used were defined +based on a large number of bright stars, with no adjustment for the +different effective filter system of the Loneos telescope and the HST +GSC. Thus, occasionally very red stars will be several magnitudes +brighter in the real data than reported in the catalog. As a result, +large numbers of bad data points in these halos and spikes may be +missed. + +The second type of bad data flagged by {\tt markstar} are the trails +from satelite and other space junk. This portion of the program +searches for objects which land along a line and which have a high +linear density. Parameters may be adjusted to define the width and +the necessary density or number of points in the line. Objects in +lines are also only identified if all observations come from a common +image. If an object only has one measurement and it is in a trail, +the entire Average is flagged to be bad. However, if an object has +many measurements and only one is bad, the object is only flagged as +having some bad data points. + +The third type of bad dat are ghost star images. The HST GSC is +searched for stars projected across the telescope optical axis, which +is defined by the user. All objects within a region of fixed size is +marked for ghost stars brighter than a specific cutoff magnitude. +Similar to the trails, the average is marked as bad only if all +detections of an object are ghost images. Otherwise, it is noted that +some data may be bad. + +\subsection{Addusno} + +The program {\tt addusno} makes two types of identifications of stars +with the USNO catalog. First, it searches for coincidences within a +small, specified radius. The choice of this radius is somewhat +tricky: it is important to catch the association, but also to avoid +making too many associations. Unfortunately, astrometric errors in +the USNO catalog make it necessary to choose a surprisingly large +radius of 5\asec. To avoid making double matches (matching the two +USNO stars to the same photometry database object), only the USNO +object closest to the object is associated. There are also a +substantial number of stars with significant proper motion between the +USNO epochs and the current date. It is not unusual to see 8-15\asec\ +discrepancy for specific stars. To catch these objects, any objects +which do not match the USNO database, but which have more than a +minimum number of measurements (2 or 3?) are searched for more distant +USNO companions. A large, user-specified radius is used for this +search. Of course, only USNO stars which are not already matched to +the database may be candidates for the proper motion match. This +stage is suceptible to errors in that a slow-moving solar-system +object may be associated with a faint USNO star. For this reason, +objects flagged as high proper motion stars should be taken with some +caution. Of course, during the {\tt addusno} stage, no objects +already flagged as bad by {\tt markstar} will be matched with USNO +stars. + +\subsection{Markrock} + +Once stars have been identified with the USNO database, it is possible +to search for asteroid detections. The program {\tt markrock} +searches within a specific CPT file for objects which are moving along +straight lines in (X,Y,t) space. In order for an object to be found +by {\tt markrock}, it must have three points, each with only single +measurements. The third point must land within a specified distance +of the line projected from the first two data points. These +comparisons are made in the three dimensional space of image position +and time. Also, objects are only accepted if they have a speed less +than a threshold. If an object is moving too fast, it will make a +streak on each individual image and will probably be missed anyway. +Moving objects detected in this way are flagged in the photometry +database and are also written to a Rock database. Currently, the Rock +database just contains ASCII lists of the RA, DEC, Mag, and time for +each detection. In fact, the data are stored in pairs of detections, +with the middle detection saved twice. This format makes it easy to +plot lines connecting the points using the connect-the-dots plotting +style of status/kapa (see below). + +One of the current drawbacks of {\tt markrock} is the insistence on +three observations with only 1 measurement each. If an object is +moving too slowly, two of the dectections of the object may be merged +together. Enough data is stored in the database to extract these +individual measurements, so a more sophisticated search is possible. +It is also currently the case that USNO proper motion stars are +ignored, but some of these will be asteroid detections which are +simply too close to a faint USNO stars (it has to be faint because it +would otherwise be associated with a star in the photometry +database). + +\begin{figure} +\psfig{file=flags.eps,width=9cm} +\caption{\label{flags} +Pictoral representation of the Average.code flags. The lower byte +value is used for specific definitions of object types. +} +\end{figure} + +\subsection{Data Flags} + +The identifications made by the above routines are noted as a set of +flags in the Average.code structure entry. Some of these flags are +mutually exclusive, so a certain bit may have more than one meaning, +depending on the value of other bits. Figure~\ref{flags} shows a +pictoral representation of the meaning of the different bit fields. +The uppermost bit (bit 15) determines if the object may be considered +a ``fixed'' star or something which has no fixed location. If it is +not fixed, and this bit is 1, then there are several options. The +second uppermost bit (bit 14) will be set if this object is one of the +three types of bad data flagged by {\tt markstar}. The lower byte of +code is used to define the type of bad data: ghost, trail, or bleed. +Thus, any datapoints determined to be a satelite trail will have a +code value of 0xc002 (49154 decimal). If an object is a moving +object, but not a bad datapoint, it may be identified as a ``rock'' +(an asteroid) by the program {\tt markrock}. In this case, the flag +for an asteroid is turned on (bit 13). Thus anything identified as a +rock by {\tt markrock} will have a code value of 0xa000. The lower +byte of code is not yet defined for asteroids, but could be reserved +for distinguishing different classes of asteroids (ie, on the basis of +orbital speeds, and so forth). Objects which are fixed may have a +variety of possible flags. First, an object may be identified with a +USNO star (bit 14). It may be found to exhibit variability (bit 13), +implying that the relative photometry routine should ignore it. If +may be a transient object (bit 12). Note that, while an object +associated with a USNO object may not be transient, an object which is +variable may also not be associated with a USNO object. Thus, we need +seperate bits for variable vs transient. Different types of variable +objects may be represented with different values of the lower byte. +For example, a Cepheid may have a specific lower byte code of 0x01, +which an RR Lyra may have a lower byte code of 0x02. Thus a Cepheid +associated with the USNO catalog would have a total code value of +0x6001. Finally, any of these types of non-moving objects may have +some bad individual measurements (bit 12) or may be found to have a +significant proper motion (bit 11). + +\section{Holding it Together} + +The preceeding sections describe the step-by-step analysis and +incorporation of the data from individual images. However, the power +of the Pipeline comes in applying it to large numbers of images. At +LONEOS, we are using four Pentinum II computers to analyse the roughly 200 +images that are observed each night. + +It is best to think of the analysis of images occurring on groups of +images from a given night. Of the set of processes discussed above, +those in section \ref{parallel} are performed on individual images +essentially independently of the other images. The rest of the steps, +both the data incorporation ({\tt addstar} and the database cleaning +routines occur on the entire batch, one image at a time. Thus, the +former routines can easily be run in parallel, but the latter routines +are run in serial to avoid having two programs writing to same part of +the database at the same time. In this model, we would process all of +the images from a night through the parallel stage of the analysis and +then run the serial stage only after the entire night is finished in +the parallel stage. We thus have a natural division of the pipeline +into two stages. + +The implementation of the two stages involves several programs, some +of which are specific to the LONEOS situation, and others which are +more general. There are two layers of programs which enable batch +processing of many images and repeated batch processing over many +nights. At the highest level, are Perl scripts which choose what data +needs to be analysed and when. These scripts tend to be more +dependent on the details of the LONEOS implementation. At the next +level down, these scripts call two main C programs which run the +images of a given night through the pipeline. These C programs are +more general and do not depend on the details of the LONEOS +implementation. A defined set of files holds a log of the nights and +images that have been processed, both for the information of people +who might want to monitor the progress, but also used by the programs +to guide decisions about what to analyse next. + +\subsection{data and file organization} + +There are a set of standard locations for all of the files used and +created by the Pipeline. These locations have both abstract names +used by the pipeline programs and specific definitions used in the +LONEOS implementation. In general, the definitions of the files are +given in the configuration file, 'config.txt' (currently located in a +hard-wired location, but this may soon change to the mechanism used by +the lastest version of spicam (3.0)). The C programs all refer to the +config file for the definitions, but the Perl scripts currently have +the definitions hardwired as variable names defined in the beginning +of each file. This should be changed to a mechanism that looks in +the config file, perhaps implemented by a standard Perl function. + +Here is a list of the main directories and their meanings: + +logdir: /metis/d11/logs - all log files and the files used by +the Perl scripts to control their actions (except locks) go here. + +workspace: /irene/d11/workspace - the intermediate analysis +stages go here, and the tar, gzipped directory files stay here. + +dumpspace: /pallad/d1 - used by slurp to store images +downloaded from the tape archive. + +database: /hebe/d27/database - the result database, as also +the locks and the Rocks.dat file. + +references: /metis/d11/references - a variety of reference data, +including the HST GSC, the USNO catalog, the config files, and so +forth. + +source: /metis/d11/src/ohana - all of the programs source, +binaries, and scripts relevant to running the Pipeline are contained +in these directories. + +\begin{verbatim} +database: /hebe/d27/database (also in /metis/d27/database as linked directories). + lock files also go in this directory +workspace: /irene/d11/workspace - temporary storage of result files +dumpspace: /pallas/d1 - disk to store data dumped from the archive by slurp. +scriptdir: /metis/d11/logs - all log files +\end{verbatim} + +\subsection{lastnight} + +The first step in the process is the Perl script {\tt lastnight}, +which is highly dependent on the specifics of the LONEOS setup. It +looks at the disks used by the observers to write the images from a +given night. Images are written to a directory called +/pallas/d[1-6]/YYMMDD where 'YYMMDD' is an abbreviation for the current +night's date (ie, 990223). Many files use this naming scheme, so we +will refer to such a name as 'YYMMDD'. All of the images are stored +in this directory with names yyyyMMDDxxxxb.fits where yyyy is the full +year, MM is month, DD is date, and xxxx is a sequence number. The 'b' +refers to the 'b' CCD, and the .fits extension is required. The +script {\tt lastnight} is called by a cron job scheduled for 7am on +{\tt hebe.lowell.edu} (note that the clock on hebe is set to PST, not +local MST - fix this some day!). The script searches the disks +{\tt /pallas/d[1-6]} for a directory of the right form (basically any +directory beginning with a number). By default, the program demands +that the directory have the name derived from today's date, but with +the flag -any, it will accept any directory. It then checks to see if +this directory has been already been analysed (ie, it is included in +the 'processed.log' file). If not, it lists all files in that +directory with the right ending (*b.fits) and places these names in a +file, {\tt /metis/d11/logs/YYMMDD.inlist}. It also writes the name of the +directory (along with some other data) in a file 'extracted.log'. +When it has successfully ended and identified some data, it calls the +next script, {\tt stage1} and then quits. + +\subsection{slurp} + +An alternative to {\tt lastnight} is the Perl script {\tt slurp}, +which looks for data on the tape archive. This script looks at the +tape.log file and compares it to the file 'processed.log'. It tries +to identify nights listed in the file 'tape.log' which have not been +included in the 'processed.log'. If it finds any, it tries to +download as much of the data as will fit on the disk {\tt /pallas/d1}. +This script is called from within the {\tt stage1} script and is +invoked if {\tt stage1} is run without any data already in +the file 'extracted.log'. + +\subsection{stage1} + +The Perl script {\tt stage1} takes the image list produced by {\tt +lastnight} or {\tt slurp} (YYMMDD.inlist) and passes the list to the +controller program {\tt control1} which performs the parallel +analysis. The script has a few features to decide which files have +been analysed through the pipeline and to what extent. If {\tt +stage1} is called, it should be able to figure out 1) if there is data +ready to be analysed (ie, there is a new date reference in the file +'extracted.log'), 2) if the data has been partially analysed (ie, the +date reference in 'extracted.log' has not been put in 'processed.log', +but there are entries in the file YYMMDD.outlist), and 3) which data +needs to be run through which part of the pipeline (ie, it decides +which of the images in YYMMDD.outlist have succeeded at which stages +of the analysis, and starts them at the appropriate spot). These +features are most useful if the analysis crashes in the middle or +needs to be restarted. The {\tt stage1} script is meant to be run +without any arguments and it will figure out what needs to be done. +The one possible argument is the -once flag. This flag tells {\tt +stage1} to analyse the data it finds and quit when it finishes, and +not call another instance of itself. Otherwise, when it is done, it +will restart itself, and since there is no new data in +'extracted.log', it will start a slurp process to get data from the +archive. If there is no data in the archive either, this process will +sleep for an hour and try again. After a few attempts (2 days?), {\tt +stage1} will give up and quit without restarting itself. The script +writes a log file called \$scriptdir/stage1.NN.log where NN is a +number in sequence between all runs of {\tt stage1}. A file +stage1.count keeps track of which number NN we are on. When {\tt +stage1} has finished the analysis of a night's data, it writes the +date in 'processed.log' to let other processes know what has been +analysed. The bulk of {\tt stage1} is not dependent on details of the +LONEOS setup, but it does expect the files to have the naming scheme +described above. + +\subsection{stage2} +The serial stage processes are run by the Perl scrip {\tt stage2}. +Normally, this script is always running. It waits for the name of a +night to be added to the 'processed.log file, presumably be {\tt +stage1}. Once if finds a new name in this file, it looks for the +appropriate images in the file 'YYMMDD.addlist'. It then submits the +list to the second controlling program {\tt control2}, which runs the +images through {\tt addstar}. Images which are successfully run +through addstar have their *bf.fits (flattened image) and *.obj +(dophot output) files deleted. When the entire night is done, the +rest of the *.obj and *bf.fits images are deleted by {\tt stage2}, the +directory with the *.cmp and *.log files is tared and gzipped into the +file YYMMNN.tgz. The directory is then deleted. Finally, {\tt +stage2} calls the scripts {\tt cleaning.pl} and {\tt run.phot} which +runs the cleanup programs ({\tt markstar - nrphot}) on the database. +After a successful run, {\tt stage2} will automatically respawn a new +copy that waits until more data arrives. + +\subsection{control1 / control2} +These two C-programs perform the tasks of farming the images out to +the different machines for analysis. The two programs are essentially +identical, but {\tt control1} is allowed to use all machines and {\tt +control2} can only use one (hebe.lowell.edu, which has the database +local to it). These controller programs start off by executing remote +shells (using ssh) on a set of machines. These shells are then used +to execute the remote commands and are kept open the entire time the +process runs. + +Both control programs maintain a list of images in a set of queues. A +set of rules defines the order in which each image travels through the +queues. Each queue is associated with a single process (ie, flatten, +astrometry, etc). Images which successfully run through a queue are +sent on to the defined 'next' queue, while images which fail are +instead sent to the defined 'fail' queue. In most cases, the 'fail' +queue is a process which marks the image as bad and write the name in +a file, and then puts the image in the 'done' queue. However, it is +possible for the 'fail' queue to be another attempt, such as in the +case of astrometry. In this example, images which fail the default +astrometry are run through a second queue which tries again with the +-loneos flag set (this flag makes the assumption that the header +information about the LONEOS region number for the image is correct, +not the RA/DEC information). + +The connections on the remote machines are handled as child processes +with communication through the stdin and stdout/stderr pipes. +Messages can be sent back and forth though these pipes, and messages +which are received from a process associated with an image are written +to a log file with the name imagename.log (where imagename is the root +name of the image). The only required messages from the program are +messages saying ``SUCCESS'' or ``ERROR''. The programs therefore all +end with either of these words. The programs are started within the +controller as a command to the shell followed by and echo ``PROCESS +DONE''. If the controller sees the message ``PROCESS DONE'' without +either ``SUCCESS'' or ``FAILURE'' (after waiting an appropriately long +time), it assumes the program crashed. Currently, no timeout is +available for the programs. The flat-field process is run in mana and +is a slightly special case. We avoid re-loading the same flat many +times by starting a mana process once and running the flatten +procedures as mana function calls. The images which are analysed +(with an indicator of success or failure at each stage) are written to +a file named 'YYMMDD.addlist' by the {\tt control1} and 'YYMMDD.outlist' +by {\tt control2}. + +\subsection{Lock files} + +A variety of locking mechanisms are used to avoid overwriting the +database or running more than one version of the controlling +programs. There is a perl program, {\tt locks}, which allows the user to +set or check the state of the different locks. All programs which +write to the database check for the existence of a lock on the +database and refuse to run if it exists. The control1/2 programs and the +stage1/2 programs check for their own lock files and only run if they +don't exist. Finally, the use can set the lock files 'kill' or 'halt' +for either stage1 or stage2. The first tells the process to stop +immediately, taking care of bookkeeping first. The second tells the +process to finish the current set of images but not start a new +version. + +\subsection{Other Perl Scripts} + +there are several other scripts which are either used to do some +simple function or are used by the maintainer to check up / work on +the pipeline. The two programs 'checkoutlist' and 'checkaddlist' take +a list of YYMMDD.outlist or YYMMDD.addlist files and list the number +of images in the file and the number of successes at each analysis +stage. There programs 'cleaning.pl' and 'run.phot' deal with the +cleanup programs and the running of nrphot. the program +'backup.stuff' take the first 4 characters of the year/month file +names and backs up to the jukebox tape all tar files for the given +month. + +\section{Operational Issues and Subtleties} + +The current LONEOS system runs principally on four Intel machines +under linux. The existing system has been generally quite stable over +the past year of operation, but there are some subtleties. One +significant issue is the RAID disk attached to hebe.lowell.edu. This +device is a set of three 9 GB disks merged into one 27GB raid disk. +The database is stored mostly on this dist and the RAID disk mounted +on metis. It is easy to split the database across multiple file +systems by using links. Within the database directory, there are +directories for each of 24 Declination bands (7.5 degrees tall). The +easiest way of splitting up the data is to move some of these +subdirectories to another device and creating links to them at the +appropriate place. This is currently done by having a directory +/metis/d27/database with the remote directories. + +The current problem with the /hebe/d27 RAID results from the poor way +it was setup. It was set up without much knowledge of the (at the +time) rather rudimentary linux RAID system. As a result, the +implementation is such that the disk will not automatically be +configured and mounted at boot time. There is a script /root/mdstart +which lists the step to setup the RAID correctly. It would be helpful +if the rc.local or rc.sysinit files were fixed to start, fsck and +mount the RAID automatically at boot. The setup on metis is +substantially more mature and is correctly implemented. It comes up +when the machine boots. + +\section{Visualization -- Status} + +Visualization of the data in the photometry database can best be +performed with the program {\tt status}. This is a command-line +driven program with a math and macro language which makes it easy to +perform complex tasks. + +First, a few notes about the user interface. The interface has an +interaction similar to {\tt tcsh}. The arrows allow editing of +previous commands. You can also use emacs-like commands such as +cntl-a to reach the beginning of the line and cntl-e to reach the end. +There is command and file completion: if you type part of a command +(as the first thing on a line) and then type tab, it will fill in as +much as possible, until the word is not unique. Typing tab twice at +that point will list the possible endings. For any but the first word +on a line, the same thing will happen for the files in the current +directory. It is also possible to type just a fraction of a command, +as long as it is unique. An ambiguous command will list the possible +alternatives. For example: +\begin{verbatim} +status: c +ambiguous command: c ( catalog cgrid clear create cursor ) +\end{verbatim} + +The shell is essentially an interpretive programming language. +Variables are set as follows: +\begin{verbatim} +status: $fred = 10 +\end{verbatim} +Any expression within curly brackets \{\} is assumed +to be an arithmetical expression and is evaluated before the line is +executed. For example: +\begin{verbatim} +echo {$fred*dcos(45)} +\end{verbatim} +would give the response 7.07107. There are math functions cos, sin, +and tan, which operate on radian expressions, and also dcos, dsin, +dtan, which operate on degree expressions. There are also the +equivalent inverse functions: eg., asin and dasin return radians and +degrees, respectively. The help section on Math defines all of the +available math functions. + +\subsection{Miscellaneous Commands} +\begin{verbatim} +! -- system call +? -- list commands +?? -- list variables +echo -- type this line +exec -- system call +exit -- exit program +help -- get help on a function +output -- redirect output to file +quit -- exit program +scan -- scan line from keyboard or file to variable +wait -- wait until return is typed +which -- show command +\end{verbatim} +Most of these are self-explanatory. The command ?? prints the system +variables. The {\tt help} command will provide help on a single +command or, without any arguments, will list all available help +files (this includes general help not associated with a specific +command). + +\subsection{Shell Programing} +\begin{verbatim} +break -- escape from function +for -- loops +if -- logical cases +input -- read command lines from a file +macro -- deal with the macros +\end{verbatim} + +There are several options for programming in {\tt status}. First, a +file which contains a series of commands can be executed with {\tt + input (filename)}. It is also possible to define macros which will +behave much like regular commands. A macro is defined by typing {\tt + macro name} or {\tt macro create name} followed by the commands. +Arguments to the macro are assigned to the variables \$1 .. \$N and +the number of arguments is given by \$0. Macros may be defined in +{\tt input} files, and in fact when {\tt status} is started, it loads +the file {\tt \~/.statusrc} which may contain default macros. Simple +loops and if statements can be performed, and are quite useful for +complex macros. + +``If'' statements are similar in syntax to C if statements, but only +the following logical operators are available: $>$, $<$, $=$, !, $|$, +and \&. Notice that (currently) there is no $>=$ or $<=$ symbol. The +operator ! means ``not equal to'', but cannot be used to negate a +logical value. The operators $|$ and \& have the meaning of ``or'' and +``and'' respectively. Math expresions in the if statement must be +contained in curly braces, as elsewhere. Variables with string values +may use the logical $=$ operator to test if two strings are the same. +``For'' loops are quite simplistic. The form is: +\begin{verbatim} +for var first last delta + (commands) +end +\end{verbatim} +The value of {\tt \$var} will start at the value {\tt first} and increment by +{\tt delta} after each loop. The loop will stop after {\tt \$var} is greater +than {\tt stop}. The value {\tt delta} is optional, with 1 assumed. +The value of {\tt \$var} may be changed during the loop, and if set +beyong the value of {\tt last} will end the loop early. + +\subsection{Vector Plotting} + +\begin{verbatim} +box -- draw a box on the plot +clear -- erase plot +create -- create a new vector +cursor -- get coords from cursor +grid -- plot cartesian grid +hist -- create histogram from a vector +labels -- define labels for plot +limits -- define plot limits +plot -- plot a pair of vectors +print -- write vectors to file +ps -- define labels for plot +set -- vector math +style -- set the style for graph plots +vectors -- list vectors +zplot -- plot scaled points +\end{verbatim} + +In addition to scalar variables, {\tt status} can manipulate and +display 1-D vector variables. Many of the commands which extract data +from the photometry database place the data in vectors as well as +plotting them. A vector can also be created based on a number +sequence with the command {\tt create name Nelements start delta}. +The resulting vector has $Nelements$ entries, starting at a value of +$start$ and running until $start + delta*Nelements$. If $delta$ is +0.0, all elements will have the value of $start$. A histogram of a vector +may be made with the command {\tt hist}, which creates a new vector +containing the histogram of the first vector. The data range and bin +size of the histogram are defined in same way as with create. This +makes it easy to create the index vector that goes with a histogram +vector: +\begin{verbatim} +hist y Ny 1 100 0.1 +create dx 1 100 0.1 +\end{verbatim} +The above will create a histogram of y in Ny and the index in dx. +Plotting this with {\tt plot dx Ny} will show the histogram. + +Vector math is performed with a command of the form {\tt set new = + (expression)}. The expression is some math function employing +vectors and scalars. A complete listing of the math operators +available in {\tt set} can be found in the help for {\tt set}. + +Once vectors are defined, they may be plotted. A pair of vectors can +be plotted against each other if they have the same number of entries. +The plotting is performed on the graphics window, Kapa. There are +actually several graphics windows available to {\tt status}, any of +which may be used to plot at any time. Some of the more complex +operations default to either graphics window 0 or 1, depending on the +context. Except for those functions with a pre-defined window, all +plotting functions apply to the current graphics window unless an +option {\tt -n N} is given to specify a different window. The +plotting style is determined by the command {\tt style} which can set +the line width, the line type (solid, dashed, dotted, etc), the point +type (box, cross, etc), the point size, the color, and whether a pair +of vectors is plotted as a sequence of points, a set of connected +lines, or a histogram. Some functions which make plots use their own +styles, as discussed below. The function {\tt limits} lets the user +set the range of the plot axes, or check the current setting. The +command {\tt plot} will plot a pair of vectors on the current graphics +window using the current plotting style for that window. The command +{\tt zplot} will plot a pair of vectors with the point size scaled by +a third vector, with maximum and minimum point sizes representing +specified values. The {\tt cursor} command goes the other way: this +command puts the Kapa window in cursor mode and waits for input from +Kapa. The user can then type any alphanumeric key on the graphics +windows and will be told both the pointer location (in the graphics +coordinates) and will have the coordinates stored in {\tt status} +variables. For example, by typing ``1'' in the sky display window, +the RA and DEC of the pointer are stored in the variables {\tt \$R1} +and {\tt \$D1}. This command can be used to let the user define +locations or regions of interest on the Kapa window. (Future addition: +{\tt button}, which does the same with the mouse buttons). + +\subsection{Database Functions} + +\begin{verbatim} +gcat -- get catalog at location +gimages -- get images at location +gstar -- get star statistics +extract -- extract average vectors from catalogs +mextract -- extract measurement vectors from catalogs +imstats -- plot image statistics +imextract -- extract image vectors from database +lcat -- list catalogs in display region +cmatch -- match two catalogs +\end{verbatim} + +There are a variety of other commands which directly refer to the +photometry database. Some of these functions extract data of various +types from the database, others perform more complex plotting +operations. The commands listed above are those which simply extract +data from the database. The first three list information relevant to +a specific RA, DEC location on the sky: {\tt gcat (RA) (DEC)} lists +the catalog at the specified location and places the name in the +variable {\tt \$CATNAME}, {\tt gimages (RA) (DEC)} lists all images +which overlap the specified location, , {\tt gstars (RA) (DEC) + (RADIUS)} lists data about the stars within a specified radius of +the specified location (all numbers above are given in decimal +degrees). Similarly, {\tt lcat} lists the catalogs in the region. +Imstats lists statistics about each image + +The next three commands extract a specific piece of information from +the photometry database and places it in a vector. First, {\tt + extract} will extract average values for each star and place it in a +vector. Next, {\tt mextract} will extract measurement values for each +star and place it in a vector: as a result a single star may have +multiple entries in the measurement vectors. Finally, {\tt imextract} +will extract image statistics into vectors (not yet implemented). + + +\begin{verbatim} +catalog -- plot catalog stars +cgrid -- plot sky coordinate grid +cplot -- plot vectors in sky coordinates +czplot -- plot scaled vectors in sky coordinates +images -- plot image boxes +imdense -- image density plot +lcurve -- plot lightcurve for a star +pcat -- plot catalog boundaries +region -- define sky region for plot +resid -- plot residuals +simage -- plot stars in an image +\end{verbatim} + +There are two types of database plotting functions: those that display +or refer to the spatial charateristics of the data and those that +refer to other types of charatersitics, such as the time domain. The +graphics window 0 is reserved for all plots of objects on the sky. +The command {\tt region} defines the current sky coordinates for plots +in graphic window 0. The command {\tt pcat} plots the outline of all +photometry database files which are within the currently defined +region (and by default, only those with data). {\tt images} plots the +outline of the images in the image database, while {\tt imdense} shows +the number of images at a location by randomly spacing dots within the +boundary of the images. The command {\tt cgrid} +draws a grid in celestial coordinates on the for the current region. + +The most complex, but also one of the most useful command is {\tt + catalog}, which plots the positions of stars in the photometry +database (and others) on the sky. There are many options to this +command. One set allows the user to plot stars from the photometry +database (the default), from the HST GSC, or from an ASCII text file +with RA, DEC, and Mag in specified columns. If the ASCII file has a +fixed number of bytes per line, the data can be more quickly loaded. +The size of the points may be scaled by the star magnitude, by the +number of observations of the star, or by the number of missing +datapoints for the star. In addition, points may be plotted only if +they land in specified magnitude ranges, or with specified numbers of +measurements, or missed measurements. Also, objects may be plotted +only if they have a specified Average.code, so that only asteroids or +only perfect stars may be plotted. The plotted vectors may be saved, +if desired, and the source catalog epoch may be specified as different +from J2000 (only valid for ASCII data). + +Several other commands relate to non-spatial charateristics of images +and stars. {\tt lcurve} will plot a light curve for all stars within +some radius of a point. {\tt resid} plots the photometry residuals +for a particular region file. + +\section{Some Examples} + +\begin{figure} +\psfig{file=allsky.ps,width=16cm} +\caption{\label{allsky} +Map of the entire sky, and images added to database. +} +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location +of the images currently in the database. This picture was made with +the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 gls +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=catalog.ps,width=9cm} +\caption{\label{catalog} +Comparison between HST GSC and photometry database astrometry. +} +\end{figure} + +Fig.~\ref{catalog} shows an example comparison of the photometry +database star positions and the HST Guide Star Catalog star +positions. The crosses are from the photometry database while the +boxes are from the HST GSC. The size of both points is a function of +brightness. This plot was made with the following commands (starting +from the previous image): +\begin{verbatim} +status: cursor (typed 1 on region of interest) +1 137.097858 22.698305 +q 137.097858 22.698305 +status: region $R1 $D1 0.5 TAN +status: cgrid +status: box +status: style -pt 0 +status: gcat $R1 $D1 + 0 n2230/1951.cpt * +status: catalog n2230/1951.cpt -m 12 18 +status: style -pt 2 +status: catalog n2230/1951.cpt -m 12 18 -g +\end{verbatim} + +\begin{figure} +\psfig{file=lightcurve.ps,width=9cm} +\caption{\label{lightcurve} +Lightcurve for several image stars. +} +\end{figure} + +Fig.~\ref{lightcurve} shows an example lightcurve from three stars in +the above field. The x axis shows the time in seconds, which the y +axis shows the star magnitudes. This plot was made with the following commands (starting +from the previous image): +{\small +\begin{verbatim} +status: cursor +1 137.062562 22.791860 +q 137.062562 22.791860 +status: lcurve $R1 $D1 0.01 -l +status: box +status: gstar $R1 $D1 0.01 +read 11318 stars from catalog file /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt (22462 total measurements) +508 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.068619 22.785400 18.438000 1 (2) 1.000000 1.000000 +498 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.062820 22.792747 12.882000 3 (0) 1.000000 1.000000 +501 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.064499 22.800817 15.480000 3 (0) 1.000000 1.000000 +\end{verbatim}} +The last lines show the database information on the three stars. + +\section{Software Distribution} + +All of the code, binaries, and scripts for the Pipeline are stored in +the ohana tree at /metis/d11/src/ohana. The ohana directory contains +subdirs of bin, lib, include, src, config, and doc. There is a +Makefile at the top level which calls lower-level Makefiles to build +the programs desired. One of the important concepts in the ohana +distribution is the design of the directory layout to enable binary +support for multiple architectures. The Makefiles use the environment +variable ARCH to determine the appropriate architecture. The binary +and library files are stored in subdirectories of the form bin/\$ARCH +and lib/\$ARCH. It is therefore necessary that users of the ohana +system architecture define the ARCH variable correctly. It is also +necessary to include in the user's PATH the directory +/metis/d11/ohana/bin/\$ARCH. At LONEOS, +with the solaris sparc station and the linux machines both in use, +ARCH can take on the values of 'linux' and 'sol'. The values 'sun4', +'irix', and 'hp' have also been used at various sites. A simple bit +of csh script in the users .cshrc or equivalent can make this +assignment transparently: + +\begin{verbatim} +set sys=`uname -s` +switch ($sys) + case IRIX64: + setenv ARCH irix; + breaksw; + case SunOS: + set ver=`uname -r | awk '{print substr($1,1,1)}'`; + if ($ver == 5) then + setenv ARCH sol + else + setenv ARCH sun4 + endif + breaksw; + case Linux: + setenv ARCH linux; + breaksw; + default: + echo "unknown architecture"; + setenv ARCH unknown; + breaksw; +endsw +\end{verbatim} + +The Makefiles use the ARCH variable not only for destination +directories, but also for destination binary names. All files of the +form fred.c are compiled to fred.ARCH.o (instead of fred.o), +uninstalled libraries get the names libfred.ARCH.a (instead of +libfred.a), while uninstalled programs are called fred.ARCH. + +The perl scripts are kept in ohana/src/perl, with links to the +appropriate bin directories. All programs are stored in directories +of the form ohana/src/program. The exceptions are gastro (stored in +ohana/src/astro), and control1/2 (stored in ohana/src/astro). Also, a +variety of simple, user-level functions are available in +ohana/src/misc. + +Most of the ohana package of programs use several common libraries. +These include the readline library (except for dophot, this is the only +part of ohana that is not written by Eugene Magnier), which is used +for command-line interface systems. Other important libraries are the +fits library for implementing the FITS specifications in a convenient +C-friendly environment, and the ohana library, consisting of basic +string and other basic operations. + +For more details on the programming aspects of parts of the pipeline, +see the associated document, 'A Programmer's Guide to the Loneos +Pipeline' + +\section {Comparison with USNO catalog} + +I have made several comparisons between the USNO catalog and results +from running images from Loneos through the photometry pipeline. In +general, there is a good agreement between the astrometry of the two +datasets. There are specfic areas where the two disagree, however. +First, as discussed above, the USNO star positions may be +significantly different from the observed star positions simply +because of proper motion. The first picure in the ``Comparisons +between Loneos + USNO'' pages shows several stars with significant +proper motion. These are currently being flagged in the photometry +database as discussed. Other differences evident in this first +picture include the classification of bright stars and the splitting +of faint and bright stars. In the upper right of the image, two +bright stars are visible, one of which is actually a double. The USNO +catalog does a good job of detecting both, but it also detects and +includes as stars several diffraction spike points. The HST GSC only +detected a single point and did not split the star at all. The Loneos +data in our pipeline did not detect the star at all due to saturation. +A star in the upper left corner shows the ability of {\tt dophot} in +the Loneos pipeline to split double stars. This star is, on close +inspection, clearly a double. The USNO catalog only detects one. The +Pipeline not only splits the two stars, but it also makes the correct +cross-identification of the stars. + +The second set of pictures shows the ability of the Loneos pipeline to +deal with small galaxies. There are two small galaxies in this field, +both of which are broken into 4 or 5 objects by the USNO catalog, but +maintained as single objects in the Loneos pipeline. In another +example (not shown) a bright star within the disk of a small galaxies +was clearly split from the galaxy by the pipeline (and also by USNO). + +\appendix +\section{Parameter file params.txt} +\begin{verbatim} +# Configuration file for Loneos analysis pipeline + +# important files and directories +GSCFILE /host/pikake/d1/gene/gsc/GSCregions.tbl +GSC_DIR /host/pikake/d1/gene/gsc +CATDIR /host/radon/rad3/gene/photcat2 +IMAGE_CATALOG /host/radon/rad3/gene/photcat2/Images.dat +IMAGE_CATALOG_TEMPLATE /host/radon/rad3/gene/photcat2/template.cat +CATALOG_TEMPLATE /host/radon/rad3/gene/photcat2/template.cat +ROCK_CATALOG /host/radon/rad3/gene/photcat2/Rocks.dat +USNO_CDROM /host/tycho/mnt/md0/kevin/zones +LONEOS_REGIONS /host/pikake/h1/gene/loneos3/regions.map + +# parameters for "fstat" +# instrumental mag zero point +ZERO_PT 24.5 +DOPHOT_CHAR_LINE 129 +DOPHOT_TYPE_FIELD 5 +DOPHOT_AP_FIELD 91 +DOPHOT_PSF_FIELD 52 +MIN_SN_FSTAT 6.0 + +# parameters for "gastro" +DEFAULT_RADIUS 32.0 +MINIMUM_RADIUS 1.5 +MIN_ERROR 0.6 +MIN_PRECISE 0.12 +CCD_PC1_1 1 +CCD_PC2_2 -1 # for loneos +CCD_PC1_2 0 +CCD_PC2_1 0 +ASEC_PIX 2.82 +NFIELD 3.0 +MMIN 6.0 +ROT_ZERO 0 +dROT 1.0 +NROT 2 +POLAR_AXIS_RA -90.0 +POLAR_AXIS_DEC 89.88 +RA_OFFSET -90.0 +DEC_OFFSET 0.0 + +# parameters for "addstar" +# airmass extinction coefficient (in mag / airmass) +NSIGMA 3.0 +ALPHA 0.1 +XOVERSCAN 60 +YOVERSCAN 50 + +# parameters for "controller" +IN_DIR /host/pikake/h1/gene +OUT_DIR /host/pikake/h1/gene/loneos4/ +DOPHOT_PARAMS /host/pikake/h1/gene/src/ohana/config/default_parameters +FLATTEN_SCRIPT /host/pikake/h1/gene/src/ohana/config/test.pro +NEW_IMAGES images.dat +USERNAME gene +PASSWORD gene + +# here we list all available machines +NMACHINES 2 +MACHINE0 pikake.astro.washington.edu +MACHINE1 mildew.astro.washington.edu + +# here we define which machines which are always free +fMACHINE0 1 +fMACHINE1 0 + +# here we define the machine which runs the database +LOCAL_MACHINE 0 + +# parameters for "status" +TIME_REFERENCE 90/01/01 00:00:00 + +# parameters for "nrphot" +# make this one small to boost change of getting an early region +TAU 3.0 +SCATTER_LIM 15.0 +MAG_LIM 17.0 +IMAGE_SCATTER 0.075 +NIMAGE_SCATTER 1.5 +STAR_SCATTER 0.05 + +### parameters for "markstar" +SEARCH_RADIUS 360 # region for initial trail hunt, in arcsec +TRAIL_WIDTH 5.0 # expected trail width, in arcsec +NANGLE_BINS 180 # Number of angular bins in 180 degrees +NPTSINLINE 5 # minimum points needed for trail +MIN_DENSITY 0.025 # minimum linear density in star/arcsec +SPACE_SIGMA 10.0 # how much denser than average for trail? +# parameters defining bright star exclusion regions +BRIGHT_XTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_XTRAIL_MAG 9.5 # faintest sat. star +BRIGHT_XTRAIL_SLOPE -80.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_YTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_YTRAIL_MAG 11.0 # faintest sat. star +BRIGHT_YTRAIL_SLOPE -400.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_HALO_MAG 8.5 # faintest sat. star +BRIGHT_HALO_SLOPE -28.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +# parameters which define the ghosts +GHOST_MAG 7.5 +GHOST_RADIUS 200 # in arcsec +OPTICAL_AXIS1 2154.0 +OPTICAL_AXIS2 2193.0 + +# parameters for "addusno" +USNO_RADIUS 5.0 # in arcsec +USNO_PROPER 20.0 # in arcsec +USNO_RED 1000 # code for USNO Red data +USNO_BLUE 1001 # code for USNO Blue data +# /mnt/cdrom + +# parameters for "markrock" +ROCK_RADIUS 2.0 # in arcsec +ROCK_MAX_RADIUS 20.0 # in arcsec +ROCK_MAX_SPEED 0.1 # in arcsec / sec +\end{verbatim} + +\end{document} Index: /trunk/Ohana/doc/pipeline/pipeline4.tex =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline4.tex (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline4.tex (revision 2508) @@ -0,0 +1,1520 @@ +%\documentstyle[astro,psfig]{article} +\documentclass{article} +\usepackage{psfig} +\usepackage{astro} +\usepackage{times} +\GoodMargins +\newcommand{\note}[1]{({\bf #1})} +\title{Variability Pipeline} +\author{Eugene Magnier} +\begin{document} +\maketitle + +\section{Overview} + +The Loneos Pipeline provides automated reduction of images, including +photometry and astrometry, with the final data stored in a photometry +database organized by star position on the sky. + +Fig.~\ref{pipeline} shows a schematic of the analysis pipeline. +Images are written to the disk from the camera by the software which +runs it, {\tt astrocam}. The resulting images are FITS files (which +may be arbitrarily sized) and have header keyword parameters RA and +DEC specifying rough celestial coordinates. Images are first analyzed +by {\tt dophot}, which produces instrumental photometry and pixel +coordinate positions for stars and other objects in the images, +writing files with a particular format, called OBJ files in this +discussion. The results from {\tt dophot} are cleaned up by a program +called {\tt fstat} which merges the header information from the image +with the most relevant parts of the OBJ file, creating a new file with +the CMP format. Astrometry is performed on the CMP file with the +program {\tt gastro}. {\tt Gastro} makes a comparison with an +astrometric reference database and only changes CMP header keywords as +a result. Finally, the processed image CMP file is added to the +photometry database with the program {\tt addstar}. The photometry +database is maintained in a large number of files (CPT files), each +representing a small region of sky. There is no relationship between +the image locations and the boundaries of CPT file. All of these +processes occur on individual image, and would typically be performed +while the observations are occuring, or during the day following, but +may be performed at any time to add new or archived data to the +database. + +Once large batches of images have been processed, a series of programs +should be run to clean up the photometry database. Typically, these +routines would be run at the end of a night on all of the CPT files to +which new data were added. First, {\tt markstar} identifies bad data +(the trails of satelites, the bleeding columns from stars, ghost +images), and flags these data points appropriately. Second, {\tt + addusno} matches the catalog stars with stars from the USNO database +and adds the USNO magnitudes to the photometry database. Next, {\tt + markrock} searches for likely asteroids in the CPT file, marks them, +and writes the asteroid observations to an asteroid database. It also +notes measurements which are likely to be noise and cosmic rays. +Finally, {\tt nrphot} determines relative photometric calibrations for +observations in a set of CPT files. + +With the exception of {\tt dophot}, all of the programs in the +pipeline refer to a single configuration file to determine the large +number of parameters necessary in the process. The default parameter +file is hardwired into the programs, but an alternate configuration +file may be specified with the environment variable {\tt + PIPE\_CONFIG}. This option makes independent processing of multiple +data sources trivial. For example, if the pipeline is to be used for +both Loneos and SDSS image analysis and display, there could be two +configuration files (eg, {\tt loneos.txt} and {\tt sdss.txt}), one for +each data source. Several important parameters would be different +between these two, for example, the initial plate scale guess for the +astrometry routine, or the parameters which define the rough chip +orientation. This scheme allows for analysis of different types of +images, with a common destination photometry database, or for +independent photometry databases. + +\begin{figure} +\psfig{file=schematic.eps,width=9cm} +\caption{\label{pipeline} +Schematic of the photometry pipeline. Rectangles represent programs, +while ovals represent data products. Arrows show the direction of +travel of information. The flatten routine is a filter whose position +may change as the pipeline matures. +} +\end{figure} + +Data from different sources can be processed by the pipeline and +stored in the same photometry database, since all photometry is tagged +with a source ID in the database (as are the images in the image +database). This allows us to include, for example, the USNO catalog +magnitude measurements (or a subset) in the photometry database for a +quick comparion. Different sources may be considered different +filters, so color-color and color-magnitude diagrams could be +extracted by specifying choices for source catalog. Currently each +object has a single primary photometric system, and individual +observations may be combined to determine that average (eg, if they +are different photometry sources with roughly the filter bandpass) or +they may be ignored when calculating relative magnitudes (eg, the USNO +magnitudes would be ignored). This may be expanded upon by defining +multiple average magnitudes and allowing different measurements to +contribute only to the appropriate average. The determination of the +photometry code for a particular image is currently not well defined. +There are several points in the analysis where the user (or the +controlling program) may specify the photometry source. A table +should be kept in the photometry catalog to define the various +photometry codes. + +\section{Image Analysis Pipeline} +\label{parallel} +\subsection{Flat Fielding -- Mana / Flatten} + +Flat fielding is performed on each image after it has been written to +disk by the acquisition software. The processing is performed with +mana (see mana user's guide) using a script written to work with the +analysis pipeline processing software. There are several important +aspects of the script which make it work with the containing programs. +First, the script is contained within a file, flatten.pro, in the +references directory. the script is invoked by calling it on the +command line wehn mana is started. If mana is started with a file +name on the command line, that file is loaded with in 'input'. The +file defines a macro 'flatten', and also loads the flat-field file. +The flattening process is actually done by calling within mana the +macro flatten, with the input and output file names as arguments. By +loading the flat-field at the start of mana, instead of on each +execution of flatten, we save the load time of each flat-field image +(which is 2x the size of the basic image, being 4byte floats per +pixel). The location of the flat field is currently hard-wired in the +flatten script - this should probably be changed in the future. One +of the aspects of this is that the macro loads a flatfield with a +specific name and location. This file is actually a link; The +higher-level controlling process (stage1) creates a link to the +correct flatfield. This allows the analysis process to use different +flatfields for different situations. At LONEOS, we currently use a +flat field generated from a well-understood night. This is probably +not the best solution as the flatfield changes with time. As +discussed below, the photometry solution program, nrphot, fits a +high-order polynomial to the relative photometry, allowing for +differences from the correct flatfield. + +\subsection{Photometry -- Dophot} + +Photometry is performed using a varient of {\tt Dophot}. The version +of dophot is based on Dophot 2.0, but it has been adapted to +streamline processing of many files. First, with this version of +dophot, any FITS format data is safely read in. Instead of compiling +to a fixed maximum size, only the memory needed to contain the image +matrix is allocated. The image is also converted to REAL*32 from +whatever BITPIX for processing. Finally, only a single, complete +parameter file is loaded by dophot. There is no option for a modified +and a default parameter file. The parameter file to be used is +defined, along with the input image and the output object list on the +command line when the program is run: \\ +{\tt dophot image.fits image.obj parameter\_file} \\ +To achive this, the code from dophot was wrapped within a C program +which interprets the command line arguments, allocates the necessary +memory, and reads in the image. The rest of the processing is +performed by dophot in the standad manner. Notice that we use a +consistent naming convention through the pipeline reductions. The +image is called {\tt foo.fits}, while the resulting dophot-produced +photometry list is called {\tt foo.obj}. + +\subsection{Cleanup -- fstat} + +After an image is processed by dophot, the object file ({\tt foo.obj}) +is converted to a more-compact, more-complete file called a CMP file, +with a name of the form ({\tt foo.cmp}). This conversion is done with +the program {\tt fstat}. The CMP file consists of the FITS header +from the original image ({\tt foo.fits}), with some additional +keywords to be used at later stages in the analysis, followed by an +ASCII list of the interesting data from ({\tt foo.obj}). In this +list, types 6 and 8 are excluded, and only the following values are +kept: +\begin{verbatim} +X Y Mag dMag t log(sky) +1342.0 106.1 14.166 000 4 3.2 +\end{verbatim} +Objects with a signal-to-noise ratio lower than a specified cutoff +({\tt MIN\_SN\_FSTAT}) are also excluded. Some general information +about the image is derived by {\tt fstat} (FWHM, saturation and +completeness limits, number of each dophot type) and stored as +keywords in the header. The photometry source must be specified at +this stage as one of the command line arguments, though the value used +may be overridden in one of the stages belowe. The resulting file can +now stand on its own without reference to the original image. The +keywords used by {\tt fstat} include the minimum signal to noise, a +rough guess at the zero point ({\tt ZERO\_PT}), and four numbers +defining the format of the {\tt dophot} OBJ file. + +\subsection{Astrometry -- gastro} + +Astrometry is performed automatically by the program {\tt gastro}. +{\tt gastro} loads a CMP file and determines an initial guess for the +center coordinates (based on the RA and DEC header keywords). The +program also uses the configuration information about the plate scale +and rough orientation of the image to get a close to the final +solution. Also, the true sky position of the telescope pole may be +defined to allow astrometry on images taken close to the pole. The +comparison is made with the astrometric catalog, which may be the HST +Guide Star Catalog, or it may be any source if the data is placed in +the correct format. + +\section{The Photometry Database} + +\subsection{Photometry File Format} +The photometry database consists of a large number of photometry files +representing sections of the sky and a file containing information on +the images incorporated in the database. These files are called +alternatively ``region files'' or CPT files (because they have the +extension {\tt .cpt}). The photometry files are sorted by Declination +into different directories, each consisting of a band 7.5\degree\ +wide. These directories have names like n0730, representing the band +starting at a Declination of +07:30. The image database (Images.dat) +is stored in the upper level directory of this directory tree. In +fact, the locations of the Images.dat file, the photometry database, +and such are all flexible and may be changed by altering the +configuration file. + +The name and location of each sky region comes from the Hubble Space +Telescope Guide Star Catalog. There is a file, GSCregions.tbl, which +can be used to find the region file appropriate for any location on +the sky. A number of routines exist to make such a query. + +Each photometry file in the photometry database contains all +observations and all average values for all stars observed within the +appropriate region on the sky. The identity of stars is essentially +determined by the RA and DEC coordinates. In very crowded regions, +the end user may need to double check that a neighboring star is not +contaminating individual detections (see the discusion on {\tt + addstar} below). + +The photometry files are stored in a binary format, both to reduce the +volume and to speed access. As a result, the interpretation of the +data is machine dependent: little endian machines need to swap the +data intelligently. Fortunately, this operation is taken care of +appropriately, if the programs are compiled with the BYTESWAP option +set as needed, which is automatically set by the Makefile in the case +of a Linux machine. For reference, Suns, SGI, and HP are all big +endian, while PCs and DECs are little endian. The automatic +conversion takes place by using the funcions {\tt Fread} and {\tt Fwrite} +to substitute for the standard C library {\tt fread} and {\tt fwrite} +functions. {\tt Fread} and {\tt Fwrite} are told the datatype being read, +and they know the layout of the bytes for a particular datatype. +Anytime the definitions of the relevant structures (see below) are +changed, {\tt Fread} and {\tt Fwrite} must be updated. Fortunately, there +is only one file ({\tt Fread.c}), and it has a single byteswapping +section, making it easy to adjust the code appropriately. + +The format of a single photometry database file consists four +sections: +\begin{itemize} +\item {\bf header} -- this is a standard FITS header with room for comments + about the file or whatever. Since the rest of the file is not a + standard FITS file, the SIMPLE keyword is set to False. The three + necessary keywords are NSTARS, NMEAS, and NMISS, which define the + sizes of the next three sections. +\item {\bf average} -- this section contains all average measurement + quantities for each star in the file. There are NSTARS entries, and + each entry is the data from a structure of type Average (defined in + loneos.h, and discussed in more detail below). Thus, the total size + in bytes of this section can be found by NSTARS*sizeof(Average). +\item {\bf measure} -- this section contains the individual + measurements for each star in the database. There are NMEAS + entries, and like the average section, each entry is a single + structure, in this case of type Measure (in loneos.h, see below). + All measurements of a single star are consecutive. The number of + measurements for a single star is defined by average.Nm and the + starting entry for a single star is given by average.offset. + Therefore, the first measurement of a specific star average[500] + would be given by measure[average[500].offset], while the second is + measure[average[500].offset + 1], and so forth. +\item {\bf missing} -- this section contains references to all missing + measurements of a star. This means all of those occasions when an + image enclosed the location of a star in the database, but no star + was detected on the image. Similar to the case of the measure + section, each star has a number of missing entries (which may be 0) + given by average.Nn, and the first missing entry is given by + average.missing. Each missing entry value is again a structure of + type Missing. Currently, the only data in the structure is the time + of the observation, which allows for an unambiguous recovery of the + image where the star was missed. +\end{itemize} + +\subsubsection*{Average Structure defined in loneos.h} +{\footnotesize +\begin{verbatim} +typedef struct { + float R; /* RA in decimal degrees */ + float D; /* DEC in decimal degrees */ + short int M; /* thousandths of mag (-32.000 to 32.000 valid range) */ + unsigned short int Nm, Nn; /* number of measurements, missing */ + short int Xp; /* scatter in 1/100 arcsec (-327.67 to +327.67 valid range) */ + short int Xm; /* 1000*log(chisq) for magnitude measurement */ + unsigned short int code; /* an ID code (ie, star, ghost, satelite, etc) */ + signed int offset; /* offset to first measurement */ + signed int missing; /* offset to first missing obs */ +} Average; /* 28 bytes / Average */ +\end{verbatim} } +The structure above defines the average data stored in the photometry +database for each star. Most of the entries are self-explantory, but +some need a bit of clarification. The two $\chi^2$ entries ({\tt Xp} +and {\tt Xm}) are the $\chi^2$ values of the average position and the +average magnitude. These are stored as logarthmic quantities because +high dynamic range is not needed. For the magnitude, the assumption +is that the star brightness is constant and the $\chi^2$ incorporates +the photometric errors on each measurement. For the postion, the +a default constant error is used \note{True?} and again the assumption +is that the postion is constant. As discussed above, the values of +{\tt Nm} and {\tt Nn} give the number of measurements and missing +observations for this star, while the entries {\tt offset} and {\tt + missing} point to the first {\bf measure} and {\bf missing} entry +for this star. The entry {\tt M} is the current best average +magnitude solution for this star (see the section on Relative +Photometry). Finally, the entry {\tt code} is an ID code for each +object, which may define a variety of things about the object. For +example, if the object is known to be variable, or if the object is +associated with a USNO star, or if the object is a measurement of an +asteroid. See below for details of these definitions. + +\subsubsection*{Measure Structure defined in loneos.h} +{\footnotesize +\begin{verbatim} +typedef struct { + short int dR, dD; /* 1/100 of arcsec (-327.67 to +327.67 valid range) */ + short int M; /* thousandths of mag (-32.767 to 32.767 valid range) */ + short int Mcal; /* image cal mag, thousandths of mag (-32.767 to 32.767 valid range) */ + unsigned char dM; /* thousandths of mag (0.000 to 0.255 valid range) */ + char dophot; /* dophot type (1-9) */ + unsigned short int source; /* code to identify photometry source */ + unsigned int t; /* time in seconds (0 - 143 years valid range) */ + unsigned int average; /* reference to corresponding average entry */ + /* upper byte of Measure.average stores flags: + limit of 16,777,215 stars (Naverage) in a file (=0xFFFFFF) + flags: average & 0x1000000 */ +} Measure; /* 20 bytes / Measure */ +\end{verbatim} } +The structure above defines the individual measurement data stored for +each observation of each star. The first two entries, dR and dD, are +the difference between the average coordinates for this star and the +coordinates determined for this observation. Clearly, if this +residual is too large, the individual measurement will not be +successfully matched with the average star position. {\tt M} and {\tt + dM} are the instrumental magnitude and error for this measurement, +after correction for the exposure time and a zero point (defined in +the configuration file as ZERO\_PT). The entry {\tt t} determines the +time of the individual observation. This is important not only for +timing purposes, but also to determine the source image for this +observation. The time and the entry {\tt source} uniquely determine +the image in the image database that generated this measurement. +Furthermore, this correspondence allows for determination of the pixel +coordinates on the chip, by going throught the image astrometry stored +in the Image database (see below). The entry {\tt source} is a code which +identifies the particular CCD/Telescope/Filter setup. Each unique +{\tt source} should be treated as an independent filter set for the +purposes of accurate relative photometric comparisons. This entry +also allows external catalog data, such as the USNO or Sloan +photometry, to be added to the database, as desired. The entry {\tt + average} is a pointer back to the {\bf average} section to allow +determination of the star from the measurements, in addition to the +reverse. The bits of the upper byte of this value are reserved for +flags used to define particular situations for this measurement. +Possible values are: +\begin{itemize} +\item 1 = BLEND\_IMAGE - the star on the image matched more than one + catalog star +\item 2 = BLEND\_CATALOG - the star in the catalog matched more than + one image star +\item 4 = UPPER\_LIMIT - \note{not really used?} +\item 8 = CALIBRATED - relative photometry has been performed at least + once. +\end{itemize} +The entry {\tt dophot} stores the dophot type for this particular +measurement (1-9: four extra bits in this field). +Finally, the value {\tt Mcal} determines the photometric calibration +of this specific measurment. This value is an offset appropriate to +this image (and if needed, this point in time and this chip position) +to bring the observations of the stars on multiple images to a common +system (see the section on relative photometry). + +\subsection{Image Database} + +All images for which photometry has been included in the photometry +database also have entries in an image database. The image database +is (currently) a single file in the upper directory of the photometry +database. The name of the file is determined by the IMAGE\_CATALOG +entry in the configuration file, and is currently set to Images.dat. +Like the photometry database files, the image database consists of a +FITS header followed by binary data. There is only one type of binary +data: each image has an entry, which is the data from a structure of +type Image (loneos.h). The number of images in the file is determined +by the {\tt NIMAGES} keyword. If the number of images becomes too +large, extrapolation of the system to a set of image database files +with a reference list is quite straightforeward, and could be modeled +on the organization of the region files (eg, one image database file +for each Declination directory). + +\subsubsection*{Image Structure defined in loneos.h} +{\footnotesize +\begin{verbatim} +typedef struct { + Coords coords; /* 120 bytes */ + unsigned int tzero; /* readout time row 0 in sec (0 - 142 years valid range) */ + unsigned int nstar; /* number of stars on image */ + short int secz; /* thousanths of airmass (valid range -32.000 -- 32.000) */ + short int NX, NY; /* dimensions of image */ + short int apmifit, dapmifit; /* aperture correction and error in thousandths of mag */ + short int source; /* identifier for CCD (each ever used will have a unique letter) */ + short int Mcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int dMcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int Xm; /* 10*log(image chi-square) */ + char name[32]; /* name of original image */ + unsigned char detection_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char saturation_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char cerror; /* astrometric error: 1/50 of arcsec (0 -- 5.12 valid range) */ + unsigned char fwhm_x, fwhm_y; /* PSF terms in 25*arcsec (valid range 0.0 -- 10.2" ") */ + unsigned char trate; /* 10000 * scan rate in sec/pix (0 -- 0.0256 valid range, + typically 0.0146. this is used only to determine the time + of the observation, not to find the coordinates. 1 byte + gives 0.11 sec accuracy */ + float exptime; /* exposure time, seconds */ + char code; /* flag to mark an image as bad or whatever */ + char dummy[21]; /* extra space for the future */ + short int order; /* number of terms used for Mrel */ + short int Mx, My; + short int Mxx, Mxy, Myy; + short int Mxxx, Mxxy, Mxyy, Myyy; + short int Mxxxx, Mxxxy, Mxxyy, Mxyyy, Myyyy; +} Image; /* 240 bytes / Image */ +\end{verbatim}} +The structure above lists all of the data stored for each image. A +substantial amount of information is stored for each image. First, +the astrometric calibration information is stored in a {\tt Coords} +structure (see coordinate systems, above). The upper and lower limits +of the photometry in the database (in instrumental magnitudes) is +stored in {\tt detection\_limit} and {\tt saturation\_limit}. These +values are useful for comparison for those stars where observations +are missing. The relationship between time and pixel coordinates is +given by {\tt tzero} and {\tt trate}. Drift images will have a scan +rate stored in {\tt trate}, while staring images will have a value of +0.0 for this entry ({\tt tzero} then applies to the whole image). +Other important parameters are kept: the astrometric error ({\tt + cerror}), the airmass ({\tt secz}), the exposure time ({\tt + exptime}), the number of stars detected ({\tt nstar}), the FWHM +values ({\tt fwhm\_x, fwhm\_y}), the image name ({\tt name}), and the +size of the image ({\tt NX, NY}). The entry {\tt source} is used as +above to define the CCD/Telescope/Filter used for this observation. +The entry {\tt code} is used to flag images in various ways. +Currently, only one bit is used, to define if the image has +unacceptable photometric scatter. Seven extra bytes are stored for +future additions. + +\subsection{Data Incorporation -- {\tt addstar}} + +Images which are completely processed ({\tt dophot, fstat, gastro}) are +then incorporated into the photometry database with the program {\tt +addstar}. This program decides which region files are appropriate +for this particular image, then one-by-one adds stars from the image +to the appropriate region file. Stars already in the catalog are +matched with stars in the new image purely by a positional comparison. +In order to avoid the difficulty of comparisons in the RA and DEC +coordinate frame, a cartesian projection is performed. The stars from +the image being processed and the database stars in the same area are +projected onto a tangent plane and positional comparisons made in this +(locally cartesian) coordinate frame. This avoids the dangerous +singularities at the pole and also makes the RA 0,360\degree\ boundary +a trivial problem. + +Several choices must be made in the comparison process. If a star in +the catalog is matched with a star in the image, the new measurements +of that star are added to the database. If a star in the database +lies in the field of the image, but is not detected, this information +is added to the list of missing data. The only data stored in this +case is the time and source, which is sufficient to unambiguously +identify the source image from the image database. This allows later +programs to find relevant statistics from the image database, if +necessary. If a star is detected in the image, but is not already in +the database, a new entry is added to the database. In addition, the +same ``missing data'' from all previous observations images which have +covered this location (ie, images already in the image database) are +included. This last step is necessary so that the image processing +order is not important. + +Stars also run the danger of being crowded together. Since all +comparisons are performed on the basis of position alone, crowded +fields may make for ambiguous cross-identifications. A pair of stars +are matched if the difference in their positions is less than a +specified search radius (usually dependent on the astrometric for the +image). If more than one catalog stars is correlated with a star in +an image, the new measurement is added to {\em both} catalog stars, +and a flag is set noting that this observation had a blended IMAGE. +Conversely, if a single catalog star is matched with more than one +image star, both new measurements are added to the one catalog star, +and a different flat is set, noting that this observation had a +blended CATALOG. + +\section{Catalog Update and Cleaning} + +Occasionally, once a large number of images have been added to the +photometry database, a set of programs should be run to clean and +update the CPT files. This may best be done on CPT files after a +night's worth of data is incorporated. These routines define bad +data, identify the USNO catalog stars, and search for asteroids and +other junk. Most of the results from these routines are the addition +of object flags to the Average data structures (Average.code). Most +of these routines process data within a single CPT file at a time. + +\subsection{Markstar} + +The first of the update programs, {\tt markstar}, identifies bad data +and flags it. It makes three types of identifications: bright stars, +ghost stars, and trails. In the first case, it searches the HST guide +star catalog for bright stars within the field of the current CPT +file. Three types of pixels are flagged for bright stars. Any +measurements within a circular region centered on the star are suspect +(the ``halo'' of the bright star). Second, points along the X axis +are likely to contain flux from the diffraction spikes. Finally, +points along the Y axis are likely to have bleeding in addition to the +diffraction spikes. Each of these three regions has a different size +and range which must be defined in the configuration file. The +regions are represented by a representative magnitude and a scale +parameter. For example, the diameter $d$ of the halo for a star with +magnitude $m$ is defined as +$d = \mbox{BRIGHT\_HALO\_SLOPE} (m - \mbox{BRIGHT\_HALO\_MAG})$. The X and +Y axis points are defined by the length of the diffraction spike, with +an equivalent formula, and the width of the spike. Since these are +each independent, there are 8 parameters to define the exclusion +regions of bright stars. One of the current drawbacks of this system +is the reliance on the HST Guide Star photometry. In the HST Guide +Star Catalog, only one photometry band is given, so no color +information is available. The parameters currently used were defined +based on a large number of bright stars, with no adjustment for the +different effective filter system of the Loneos telescope and the HST +GSC. Thus, occasionally very red stars will be several magnitudes +brighter in the real data than reported in the catalog. As a result, +large numbers of bad data points in these halos and spikes may be +missed. + +The second type of bad data flagged by {\tt markstar} are the trails +from satelite and other space junk. This portion of the program +searches for objects which land along a line and which have a high +linear density. Parameters may be adjusted to define the width and +the necessary density or number of points in the line. Objects in +lines are also only identified if all observations come from a common +image. If an object only has one measurement and it is in a trail, +the entire Average is flagged to be bad. However, if an object has +many measurements and only one is bad, the object is only flagged as +having some bad data points. + +The third type of bad dat are ghost star images. The HST GSC is +searched for stars projected across the telescope optical axis, which +is defined by the user. All objects within a region of fixed size is +marked for ghost stars brighter than a specific cutoff magnitude. +Similar to the trails, the average is marked as bad only if all +detections of an object are ghost images. Otherwise, it is noted that +some data may be bad. + +\subsection{Addusno} + +The program {\tt addusno} makes two types of identifications of stars +with the USNO catalog. First, it searches for coincidences within a +small, specified radius. The choice of this radius is somewhat +tricky: it is important to catch the association, but also to avoid +making too many associations. Unfortunately, astrometric errors in +the USNO catalog make it necessary to choose a surprisingly large +radius of 5\asec. To avoid making double matches (matching the two +USNO stars to the same photometry database object), only the USNO +object closest to the object is associated. There are also a +substantial number of stars with significant proper motion between the +USNO epochs and the current date. It is not unusual to see 8-15\asec\ +discrepancy for specific stars. To catch these objects, any objects +which do not match the USNO database, but which have more than a +minimum number of measurements (2 or 3?) are searched for more distant +USNO companions. A large, user-specified radius is used for this +search. Of course, only USNO stars which are not already matched to +the database may be candidates for the proper motion match. This +stage is suceptible to errors in that a slow-moving solar-system +object may be associated with a faint USNO star. For this reason, +objects flagged as high proper motion stars should be taken with some +caution. Of course, during the {\tt addusno} stage, no objects +already flagged as bad by {\tt markstar} will be matched with USNO +stars. + +\subsection{Markrock} + +Once stars have been identified with the USNO database, it is possible +to search for asteroid detections. The program {\tt markrock} +searches within a specific CPT file for objects which are moving along +straight lines in (X,Y,t) space. In order for an object to be found +by {\tt markrock}, it must have three points, each with only single +measurements. The third point must land within a specified distance +of the line projected from the first two data points. These +comparisons are made in the three dimensional space of image position +and time. Also, objects are only accepted if they have a speed less +than a threshold. If an object is moving too fast, it will make a +streak on each individual image and will probably be missed anyway. +Moving objects detected in this way are flagged in the photometry +database and are also written to a Rock database. Currently, the Rock +database just contains ASCII lists of the RA, DEC, Mag, and time for +each detection. In fact, the data are stored in pairs of detections, +with the middle detection saved twice. This format makes it easy to +plot lines connecting the points using the connect-the-dots plotting +style of status/kapa (see below). + +One of the current drawbacks of {\tt markrock} is the insistence on +three observations with only 1 measurement each. If an object is +moving too slowly, two of the dectections of the object may be merged +together. Enough data is stored in the database to extract these +individual measurements, so a more sophisticated search is possible. +It is also currently the case that USNO proper motion stars are +ignored, but some of these will be asteroid detections which are +simply too close to a faint USNO stars (it has to be faint because it +would otherwise be associated with a star in the photometry +database). + +\section{Relative Photometry -- nrphot} + +Once the data has been incorporated in the photometry database and the +data validation routines have been run, it is possible to determine +photometric solutions for all of the images. The program which does +this is nrphot. It tries to find a set of solutions for each image +which reduce both the scatter per image and the scatter per star. To +see how this works, we start by defining a measurement of star $j$ on +image $i$ as $m_{i,j}$. Every star has an apparent magnitude of +$M_j$. Every image has a correction magnitude of $M_i$, defined as +$m_{i,j}= M_i + M_j$. (USE SOME OF THE TEXT FROM THE M31 paper). + +The Pipeline implementation of relative photometry includes the +possibility for up to third order polynomial terms across the field. +This is needed to account for the different flat fields, particularly +between images taken with and without the red-block filter. + +\begin{figure} +\psfig{file=flags.eps,width=9cm} +\caption{\label{flags} +Pictoral representation of the Average.code flags. The lower byte +value is used for specific definitions of object types. +} +\end{figure} + +\subsection{Data Flags} + +The identifications made by the above routines are noted as a set of +flags in the Average.code structure entry. Some of these flags are +mutually exclusive, so a certain bit may have more than one meaning, +depending on the value of other bits. Figure~\ref{flags} shows a +pictoral representation of the meaning of the different bit fields. +The uppermost bit (bit 15) determines if the object may be considered +a ``fixed'' star or something which has no fixed location. If it is +not fixed, and this bit is 1, then there are several options. The +second uppermost bit (bit 14) will be set if this object is one of the +three types of bad data flagged by {\tt markstar}. The lower byte of +code is used to define the type of bad data: ghost, trail, or bleed. +Thus, any datapoints determined to be a satelite trail will have a +code value of 0xc002 (49154 decimal). If an object is a moving +object, but not a bad datapoint, it may be identified as a ``rock'' +(an asteroid) by the program {\tt markrock}. In this case, the flag +for an asteroid is turned on (bit 13). Thus anything identified as a +rock by {\tt markrock} will have a code value of 0xa000. The lower +byte of code is not yet defined for asteroids, but could be reserved +for distinguishing different classes of asteroids (ie, on the basis of +orbital speeds, and so forth). Objects which are fixed may have a +variety of possible flags. First, an object may be identified with a +USNO star (bit 14). It may be found to exhibit variability (bit 13), +implying that the relative photometry routine should ignore it. If +may be a transient object (bit 12). Note that, while an object +associated with a USNO object may not be transient, an object which is +variable may also not be associated with a USNO object. Thus, we need +seperate bits for variable vs transient. Different types of variable +objects may be represented with different values of the lower byte. +For example, a Cepheid may have a specific lower byte code of 0x01, +which an RR Lyra may have a lower byte code of 0x02. Thus a Cepheid +associated with the USNO catalog would have a total code value of +0x6001. Finally, any of these types of non-moving objects may have +some bad individual measurements (bit 12) or may be found to have a +significant proper motion (bit 11). + +\section{Holding it Together} + +The preceeding sections describe the step-by-step analysis and +incorporation of the data from individual images. However, the power +of the Pipeline comes in applying it to large numbers of images. At +LONEOS, we are using four Pentinum II computers to analyse the roughly 200 +images that are observed each night. + +It is best to think of the analysis of images occurring on groups of +images from a given night. Of the set of processes discussed above, +those in section \ref{parallel} are performed on individual images +essentially independently of the other images. The rest of the steps, +both the data incorporation ({\tt addstar} and the database cleaning +routines occur on the entire batch, one image at a time. Thus, the +former routines can easily be run in parallel, but the latter routines +are run in serial to avoid having two programs writing to same part of +the database at the same time. In this model, we would process all of +the images from a night through the parallel stage of the analysis and +then run the serial stage only after the entire night is finished in +the parallel stage. We thus have a natural division of the pipeline +into two stages. + +The implementation of the two stages involves several programs, some +of which are specific to the LONEOS situation, and others which are +more general. There are two layers of programs which enable batch +processing of many images and repeated batch processing over many +nights. At the highest level, are Perl scripts which choose what data +needs to be analysed and when. These scripts tend to be more +dependent on the details of the LONEOS implementation. At the next +level down, these scripts call two main C programs which run the +images of a given night through the pipeline. These C programs are +more general and do not depend on the details of the LONEOS +implementation. A defined set of files holds a log of the nights and +images that have been processed, both for the information of people +who might want to monitor the progress, but also used by the programs +to guide decisions about what to analyse next. + +\subsection{data and file organization} + +There are a set of standard locations for all of the files used and +created by the Pipeline. These locations have both abstract names +used by the pipeline programs and specific definitions used in the +LONEOS implementation. In general, the definitions of the files are +given in the configuration file, 'config.txt' (currently located in a +hard-wired location, but this may soon change to the mechanism used by +the lastest version of spicam (3.0)). The C programs all refer to the +config file for the definitions, but the Perl scripts currently have +the definitions hardwired as variable names defined in the beginning +of each file. This should be changed to a mechanism that looks in +the config file, perhaps implemented by a standard Perl function. + +Here is a list of the main directories and their meanings: + +logdir: /metis/d11/logs - all log files and the files used by +the Perl scripts to control their actions (except locks) go here. + +workspace: /irene/d11/workspace - the intermediate analysis +stages go here, and the tar, gzipped directory files stay here. + +dumpspace: /pallad/d1 - used by slurp to store images +downloaded from the tape archive. + +database: /hebe/d27/database - the result database, as also +the locks and the Rocks.dat file. + +references: /metis/d11/references - a variety of reference data, +including the HST GSC, the USNO catalog, the config files, and so +forth. + +source: /metis/d11/src/ohana - all of the programs source, +binaries, and scripts relevant to running the Pipeline are contained +in these directories. + +\begin{verbatim} +database: /hebe/d27/database (also in /metis/d27/database as linked directories). + lock files also go in this directory +workspace: /irene/d11/workspace - temporary storage of result files +dumpspace: /pallas/d1 - disk to store data dumped from the archive by slurp. +scriptdir: /metis/d11/logs - all log files +\end{verbatim} + +\subsection{lastnight} + +The first step in the process is the Perl script {\tt lastnight}, +which is highly dependent on the specifics of the LONEOS setup. It +looks at the disks used by the observers to write the images from a +given night. Images are written to a directory called +/pallas/d[1-6]/YYMMDD where 'YYMMDD' is an abbreviation for the current +night's date (ie, 990223). Many files use this naming scheme, so we +will refer to such a name as 'YYMMDD'. All of the images are stored +in this directory with names yyyyMMDDxxxxb.fits where yyyy is the full +year, MM is month, DD is date, and xxxx is a sequence number. The 'b' +refers to the 'b' CCD, and the .fits extension is required. The +script {\tt lastnight} is called by a cron job scheduled for 7am on +{\tt hebe.lowell.edu} (note that the clock on hebe is set to PST, not +local MST - fix this some day!). The script searches the disks +{\tt /pallas/d[1-6]} for a directory of the right form (basically any +directory beginning with a number). By default, the program demands +that the directory have the name derived from today's date, but with +the flag -any, it will accept any directory. It then checks to see if +this directory has been already been analysed (ie, it is included in +the 'processed.log' file). If not, it lists all files in that +directory with the right ending (*b.fits) and places these names in a +file, {\tt /metis/d11/logs/YYMMDD.inlist}. It also writes the name of the +directory (along with some other data) in a file 'extracted.log'. +When it has successfully ended and identified some data, it calls the +next script, {\tt stage1} and then quits. + +\subsection{slurp} + +An alternative to {\tt lastnight} is the Perl script {\tt slurp}, +which looks for data on the tape archive. This script looks at the +tape.log file and compares it to the file 'processed.log'. It tries +to identify nights listed in the file 'tape.log' which have not been +included in the 'processed.log'. If it finds any, it tries to +download as much of the data as will fit on the disk {\tt /pallas/d1}. +This script is called from within the {\tt stage1} script and is +invoked if {\tt stage1} is run without any data already in +the file 'extracted.log'. + +A related pair program is the load/unload C programs which are needed +for loading and unloading the tape jukebox. The first, load N, takes +tape number N and loads it into the tape drive. The second takes the +tape a returns it to its slot. It is important to check on the other +users of the tape drive to be sure it is not occupied. This is done +with the llock, lunlock and llockstat programs. llockstat lists the +devices which have been allocated. llock lets you lock a particular +service, etc. + +\subsection{stage1} + +The Perl script {\tt stage1} takes the image list produced by {\tt +lastnight} or {\tt slurp} (YYMMDD.inlist) and passes the list to the +controller program {\tt control1} which performs the parallel +analysis. The script has a few features to decide which files have +been analysed through the pipeline and to what extent. If {\tt +stage1} is called, it should be able to figure out 1) if there is data +ready to be analysed (ie, there is a new date reference in the file +'extracted.log'), 2) if the data has been partially analysed (ie, the +date reference in 'extracted.log' has not been put in 'processed.log', +but there are entries in the file YYMMDD.outlist), and 3) which data +needs to be run through which part of the pipeline (ie, it decides +which of the images in YYMMDD.outlist have succeeded at which stages +of the analysis, and starts them at the appropriate spot). These +features are most useful if the analysis crashes in the middle or +needs to be restarted. The {\tt stage1} script is meant to be run +without any arguments and it will figure out what needs to be done. +The one possible argument is the -once flag. This flag tells {\tt +stage1} to analyse the data it finds and quit when it finishes, and +not call another instance of itself. Otherwise, when it is done, it +will restart itself, and since there is no new data in +'extracted.log', it will start a slurp process to get data from the +archive. If there is no data in the archive either, this process will +sleep for an hour and try again. After a few attempts (2 days?), {\tt +stage1} will give up and quit without restarting itself. The script +writes a log file called \$scriptdir/stage1.NN.log where NN is a +number in sequence between all runs of {\tt stage1}. A file +stage1.count keeps track of which number NN we are on. When {\tt +stage1} has finished the analysis of a night's data, it writes the +date in 'processed.log' to let other processes know what has been +analysed. The bulk of {\tt stage1} is not dependent on details of the +LONEOS setup, but it does expect the files to have the naming scheme +described above. + +\subsection{stage2} +The serial stage processes are run by the Perl scrip {\tt stage2}. +Normally, this script is always running. It waits for the name of a +night to be added to the 'processed.log file, presumably be {\tt +stage1}. Once if finds a new name in this file, it looks for the +appropriate images in the file 'YYMMDD.addlist'. It then submits the +list to the second controlling program {\tt control2}, which runs the +images through {\tt addstar}. Images which are successfully run +through addstar have their *bf.fits (flattened image) and *.obj +(dophot output) files deleted. When the entire night is done, the +rest of the *.obj and *bf.fits images are deleted by {\tt stage2}, the +directory with the *.cmp and *.log files is tared and gzipped into the +file YYMMNN.tgz. The directory is then deleted. Finally, {\tt +stage2} calls the scripts {\tt cleaning.pl} and {\tt run.phot} which +runs the cleanup programs ({\tt markstar - nrphot}) on the database. +After a successful run, {\tt stage2} will automatically respawn a new +copy that waits until more data arrives. + +\subsection{control1 / control2} +These two C-programs perform the tasks of farming the images out to +the different machines for analysis. The two programs are essentially +identical, but {\tt control1} is allowed to use all machines and {\tt +control2} can only use one (hebe.lowell.edu, which has the database +local to it). These controller programs start off by executing remote +shells (using ssh) on a set of machines. These shells are then used +to execute the remote commands and are kept open the entire time the +process runs. + +Both control programs maintain a list of images in a set of queues. A +set of rules defines the order in which each image travels through the +queues. Each queue is associated with a single process (ie, flatten, +astrometry, etc). Images which successfully run through a queue are +sent on to the defined 'next' queue, while images which fail are +instead sent to the defined 'fail' queue. In most cases, the 'fail' +queue is a process which marks the image as bad and write the name in +a file, and then puts the image in the 'done' queue. However, it is +possible for the 'fail' queue to be another attempt, such as in the +case of astrometry. In this example, images which fail the default +astrometry are run through a second queue which tries again with the +-loneos flag set (this flag makes the assumption that the header +information about the LONEOS region number for the image is correct, +not the RA/DEC information). + +The connections on the remote machines are handled as child processes +with communication through the stdin and stdout/stderr pipes. +Messages can be sent back and forth though these pipes, and messages +which are received from a process associated with an image are written +to a log file with the name imagename.log (where imagename is the root +name of the image). The only required messages from the program are +messages saying ``SUCCESS'' or ``ERROR''. The programs therefore all +end with either of these words. The programs are started within the +controller as a command to the shell followed by and echo ``PROCESS +DONE''. If the controller sees the message ``PROCESS DONE'' without +either ``SUCCESS'' or ``FAILURE'' (after waiting an appropriately long +time), it assumes the program crashed. Currently, no timeout is +available for the programs. The flat-field process is run in mana and +is a slightly special case. We avoid re-loading the same flat many +times by starting a mana process once and running the flatten +procedures as mana function calls. The images which are analysed +(with an indicator of success or failure at each stage) are written to +a file named 'YYMMDD.addlist' by the {\tt control1} and 'YYMMDD.outlist' +by {\tt control2}. + +\subsection{Lock files} + +A variety of locking mechanisms are used to avoid overwriting the +database or running more than one version of the controlling +programs. There is a perl program, {\tt locks}, which allows the user to +set or check the state of the different locks. All programs which +write to the database check for the existence of a lock on the +database and refuse to run if it exists. The control1/2 programs and the +stage1/2 programs check for their own lock files and only run if they +don't exist. Finally, the use can set the lock files 'kill' or 'halt' +for either stage1 or stage2. The first tells the process to stop +immediately, taking care of bookkeeping first. The second tells the +process to finish the current set of images but not start a new +version. + +\subsection{Other Perl Scripts} + +there are several other scripts which are either used to do some +simple function or are used by the maintainer to check up / work on +the pipeline. The two programs 'checkoutlist' and 'checkaddlist' take +a list of YYMMDD.outlist or YYMMDD.addlist files and list the number +of images in the file and the number of successes at each analysis +stage. There programs 'cleaning.pl' and 'run.phot' deal with the +cleanup programs and the running of nrphot. the program +'backup.stuff' take the first 4 characters of the year/month file +names and backs up to the jukebox tape all tar files for the given +month. + +\section{Operational Issues and Subtleties} + +The current LONEOS system runs principally on four Intel machines +under linux. The existing system has been generally quite stable over +the past year of operation, but there are some subtleties. One +significant issue is the RAID disk attached to hebe.lowell.edu. This +device is a set of three 9 GB disks merged into one 27GB raid disk. +The database is stored mostly on this dist and the RAID disk mounted +on metis. It is easy to split the database across multiple file +systems by using links. Within the database directory, there are +directories for each of 24 Declination bands (7.5 degrees tall). The +easiest way of splitting up the data is to move some of these +subdirectories to another device and creating links to them at the +appropriate place. This is currently done by having a directory +/metis/d27/database with the remote directories. + +The current problem with the /hebe/d27 RAID results from the poor way +it was setup. It was set up without much knowledge of the (at the +time) rather rudimentary linux RAID system. As a result, the +implementation is such that the disk will not automatically be +configured and mounted at boot time. There is a script /root/mdstart +which lists the step to setup the RAID correctly. It would be helpful +if the rc.local or rc.sysinit files were fixed to start, fsck and +mount the RAID automatically at boot. The setup on metis is +substantially more mature and is correctly implemented. It comes up +when the machine boots. + +\section{Visualization -- Status} + +Visualization of the data in the photometry database can best be +performed with the program {\tt status}. This is a command-line +driven program with a math and macro language which makes it easy to +perform complex tasks. + +First, a few notes about the user interface. The interface has an +interaction similar to {\tt tcsh}. The arrows allow editing of +previous commands. You can also use emacs-like commands such as +cntl-a to reach the beginning of the line and cntl-e to reach the end. +There is command and file completion: if you type part of a command +(as the first thing on a line) and then type tab, it will fill in as +much as possible, until the word is not unique. Typing tab twice at +that point will list the possible endings. For any but the first word +on a line, the same thing will happen for the files in the current +directory. It is also possible to type just a fraction of a command, +as long as it is unique. An ambiguous command will list the possible +alternatives. For example: +\begin{verbatim} +status: c +ambiguous command: c ( catalog cgrid clear create cursor ) +\end{verbatim} + +The shell is essentially an interpretive programming language. +Variables are set as follows: +\begin{verbatim} +status: $fred = 10 +\end{verbatim} +Any expression within curly brackets \{\} is assumed +to be an arithmetical expression and is evaluated before the line is +executed. For example: +\begin{verbatim} +echo {$fred*dcos(45)} +\end{verbatim} +would give the response 7.07107. There are math functions cos, sin, +and tan, which operate on radian expressions, and also dcos, dsin, +dtan, which operate on degree expressions. There are also the +equivalent inverse functions: eg., asin and dasin return radians and +degrees, respectively. The help section on Math defines all of the +available math functions. + +\subsection{Miscellaneous Commands} +\begin{verbatim} +! -- system call +? -- list commands +?? -- list variables +echo -- type this line +exec -- system call +exit -- exit program +help -- get help on a function +output -- redirect output to file +quit -- exit program +scan -- scan line from keyboard or file to variable +wait -- wait until return is typed +which -- show command +\end{verbatim} +Most of these are self-explanatory. The command ?? prints the system +variables. The {\tt help} command will provide help on a single +command or, without any arguments, will list all available help +files (this includes general help not associated with a specific +command). + +\subsection{Shell Programing} +\begin{verbatim} +break -- escape from function +for -- loops +if -- logical cases +input -- read command lines from a file +macro -- deal with the macros +\end{verbatim} + +There are several options for programming in {\tt status}. First, a +file which contains a series of commands can be executed with {\tt + input (filename)}. It is also possible to define macros which will +behave much like regular commands. A macro is defined by typing {\tt + macro name} or {\tt macro create name} followed by the commands. +Arguments to the macro are assigned to the variables \$1 .. \$N and +the number of arguments is given by \$0. Macros may be defined in +{\tt input} files, and in fact when {\tt status} is started, it loads +the file {\tt \~/.statusrc} which may contain default macros. Simple +loops and if statements can be performed, and are quite useful for +complex macros. + +``If'' statements are similar in syntax to C if statements, but only +the following logical operators are available: $>$, $<$, $=$, !, $|$, +and \&. Notice that (currently) there is no $>=$ or $<=$ symbol. The +operator ! means ``not equal to'', but cannot be used to negate a +logical value. The operators $|$ and \& have the meaning of ``or'' and +``and'' respectively. Math expresions in the if statement must be +contained in curly braces, as elsewhere. Variables with string values +may use the logical $=$ operator to test if two strings are the same. +``For'' loops are quite simplistic. The form is: +\begin{verbatim} +for var first last delta + (commands) +end +\end{verbatim} +The value of {\tt \$var} will start at the value {\tt first} and increment by +{\tt delta} after each loop. The loop will stop after {\tt \$var} is greater +than {\tt stop}. The value {\tt delta} is optional, with 1 assumed. +The value of {\tt \$var} may be changed during the loop, and if set +beyong the value of {\tt last} will end the loop early. + +\subsection{Vector Plotting} + +\begin{verbatim} +box -- draw a box on the plot +clear -- erase plot +create -- create a new vector +cursor -- get coords from cursor +grid -- plot cartesian grid +hist -- create histogram from a vector +labels -- define labels for plot +limits -- define plot limits +plot -- plot a pair of vectors +print -- write vectors to file +ps -- define labels for plot +set -- vector math +style -- set the style for graph plots +vectors -- list vectors +zplot -- plot scaled points +\end{verbatim} + +In addition to scalar variables, {\tt status} can manipulate and +display 1-D vector variables. Many of the commands which extract data +from the photometry database place the data in vectors as well as +plotting them. A vector can also be created based on a number +sequence with the command {\tt create name Nelements start delta}. +The resulting vector has $Nelements$ entries, starting at a value of +$start$ and running until $start + delta*Nelements$. If $delta$ is +0.0, all elements will have the value of $start$. A histogram of a vector +may be made with the command {\tt hist}, which creates a new vector +containing the histogram of the first vector. The data range and bin +size of the histogram are defined in same way as with create. This +makes it easy to create the index vector that goes with a histogram +vector: +\begin{verbatim} +hist y Ny 1 100 0.1 +create dx 1 100 0.1 +\end{verbatim} +The above will create a histogram of y in Ny and the index in dx. +Plotting this with {\tt plot dx Ny} will show the histogram. + +Vector math is performed with a command of the form {\tt set new = + (expression)}. The expression is some math function employing +vectors and scalars. A complete listing of the math operators +available in {\tt set} can be found in the help for {\tt set}. + +Once vectors are defined, they may be plotted. A pair of vectors can +be plotted against each other if they have the same number of entries. +The plotting is performed on the graphics window, Kapa. There are +actually several graphics windows available to {\tt status}, any of +which may be used to plot at any time. Some of the more complex +operations default to either graphics window 0 or 1, depending on the +context. Except for those functions with a pre-defined window, all +plotting functions apply to the current graphics window unless an +option {\tt -n N} is given to specify a different window. The +plotting style is determined by the command {\tt style} which can set +the line width, the line type (solid, dashed, dotted, etc), the point +type (box, cross, etc), the point size, the color, and whether a pair +of vectors is plotted as a sequence of points, a set of connected +lines, or a histogram. Some functions which make plots use their own +styles, as discussed below. The function {\tt limits} lets the user +set the range of the plot axes, or check the current setting. The +command {\tt plot} will plot a pair of vectors on the current graphics +window using the current plotting style for that window. The command +{\tt zplot} will plot a pair of vectors with the point size scaled by +a third vector, with maximum and minimum point sizes representing +specified values. The {\tt cursor} command goes the other way: this +command puts the Kapa window in cursor mode and waits for input from +Kapa. The user can then type any alphanumeric key on the graphics +windows and will be told both the pointer location (in the graphics +coordinates) and will have the coordinates stored in {\tt status} +variables. For example, by typing ``1'' in the sky display window, +the RA and DEC of the pointer are stored in the variables {\tt \$R1} +and {\tt \$D1}. This command can be used to let the user define +locations or regions of interest on the Kapa window. (Future addition: +{\tt button}, which does the same with the mouse buttons). + +\subsection{Database Functions} + +\begin{verbatim} +gcat -- get catalog at location +gimages -- get images at location +gstar -- get star statistics +extract -- extract average vectors from catalogs +mextract -- extract measurement vectors from catalogs +imstats -- plot image statistics +imextract -- extract image vectors from database +lcat -- list catalogs in display region +cmatch -- match two catalogs +\end{verbatim} + +There are a variety of other commands which directly refer to the +photometry database. Some of these functions extract data of various +types from the database, others perform more complex plotting +operations. The commands listed above are those which simply extract +data from the database. The first three list information relevant to +a specific RA, DEC location on the sky: {\tt gcat (RA) (DEC)} lists +the catalog at the specified location and places the name in the +variable {\tt \$CATNAME}, {\tt gimages (RA) (DEC)} lists all images +which overlap the specified location, , {\tt gstars (RA) (DEC) + (RADIUS)} lists data about the stars within a specified radius of +the specified location (all numbers above are given in decimal +degrees). Similarly, {\tt lcat} lists the catalogs in the region. +Imstats lists statistics about each image + +The next three commands extract a specific piece of information from +the photometry database and places it in a vector. First, {\tt + extract} will extract average values for each star and place it in a +vector. Next, {\tt mextract} will extract measurement values for each +star and place it in a vector: as a result a single star may have +multiple entries in the measurement vectors. Finally, {\tt imextract} +will extract image statistics into vectors (not yet implemented). + + +\begin{verbatim} +catalog -- plot catalog stars +cgrid -- plot sky coordinate grid +cplot -- plot vectors in sky coordinates +czplot -- plot scaled vectors in sky coordinates +images -- plot image boxes +imdense -- image density plot +lcurve -- plot lightcurve for a star +pcat -- plot catalog boundaries +region -- define sky region for plot +resid -- plot residuals +simage -- plot stars in an image +\end{verbatim} + +There are two types of database plotting functions: those that display +or refer to the spatial charateristics of the data and those that +refer to other types of charatersitics, such as the time domain. The +graphics window 0 is reserved for all plots of objects on the sky. +The command {\tt region} defines the current sky coordinates for plots +in graphic window 0. The command {\tt pcat} plots the outline of all +photometry database files which are within the currently defined +region (and by default, only those with data). {\tt images} plots the +outline of the images in the image database, while {\tt imdense} shows +the number of images at a location by randomly spacing dots within the +boundary of the images. The command {\tt cgrid} +draws a grid in celestial coordinates on the for the current region. + +The most complex, but also one of the most useful command is {\tt + catalog}, which plots the positions of stars in the photometry +database (and others) on the sky. There are many options to this +command. One set allows the user to plot stars from the photometry +database (the default), from the HST GSC, or from an ASCII text file +with RA, DEC, and Mag in specified columns. If the ASCII file has a +fixed number of bytes per line, the data can be more quickly loaded. +The size of the points may be scaled by the star magnitude, by the +number of observations of the star, or by the number of missing +datapoints for the star. In addition, points may be plotted only if +they land in specified magnitude ranges, or with specified numbers of +measurements, or missed measurements. Also, objects may be plotted +only if they have a specified Average.code, so that only asteroids or +only perfect stars may be plotted. The plotted vectors may be saved, +if desired, and the source catalog epoch may be specified as different +from J2000 (only valid for ASCII data). + +Several other commands relate to non-spatial charateristics of images +and stars. {\tt lcurve} will plot a light curve for all stars within +some radius of a point. {\tt resid} plots the photometry residuals +for a particular region file. + +\section{Some Examples} + +\begin{figure} +\psfig{file=allsky.ps,width=16cm} +\caption{\label{allsky} +Map of the entire sky, and images added to database. +} +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location +of the images currently in the database. This picture was made with +the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 gls +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=catalog.ps,width=9cm} +\caption{\label{catalog} +Comparison between HST GSC and photometry database astrometry. +} +\end{figure} + +Fig.~\ref{catalog} shows an example comparison of the photometry +database star positions and the HST Guide Star Catalog star +positions. The crosses are from the photometry database while the +boxes are from the HST GSC. The size of both points is a function of +brightness. This plot was made with the following commands (starting +from the previous image): +\begin{verbatim} +status: cursor (typed 1 on region of interest) +1 137.097858 22.698305 +q 137.097858 22.698305 +status: region $R1 $D1 0.5 TAN +status: cgrid +status: box +status: style -pt 0 +status: gcat $R1 $D1 + 0 n2230/1951.cpt * +status: catalog n2230/1951.cpt -m 12 18 +status: style -pt 2 +status: catalog n2230/1951.cpt -m 12 18 -g +\end{verbatim} + +\begin{figure} +\psfig{file=lightcurve.ps,width=9cm} +\caption{\label{lightcurve} +Lightcurve for several image stars. +} +\end{figure} + +Fig.~\ref{lightcurve} shows an example lightcurve from three stars in +the above field. The x axis shows the time in seconds, which the y +axis shows the star magnitudes. This plot was made with the following commands (starting +from the previous image): +{\footnotesize +\begin{verbatim} +status: cursor +1 137.062562 22.791860 +q 137.062562 22.791860 +status: lcurve $R1 $D1 0.01 -l +status: box +status: gstar $R1 $D1 0.01 +read 11318 stars from catalog file /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt (22462 total measurements) +508 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.068619 22.785400 18.438000 1 (2) 1.000000 1.000000 +498 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.062820 22.792747 12.882000 3 (0) 1.000000 1.000000 +501 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.064499 22.800817 15.480000 3 (0) 1.000000 1.000000 +\end{verbatim}} +The last lines show the database information on the three stars. + +\section{Software Distribution} + +All of the code, binaries, and scripts for the Pipeline are stored in +the ohana tree at /metis/d11/src/ohana. The ohana directory contains +subdirs of bin, lib, include, src, config, and doc. There is a +Makefile at the top level which calls lower-level Makefiles to build +the programs desired. One of the important concepts in the ohana +distribution is the design of the directory layout to enable binary +support for multiple architectures. The Makefiles use the environment +variable ARCH to determine the appropriate architecture. The binary +and library files are stored in subdirectories of the form bin/\$ARCH +and lib/\$ARCH. It is therefore necessary that users of the ohana +system architecture define the ARCH variable correctly. It is also +necessary to include in the user's PATH the directory +/metis/d11/ohana/bin/\$ARCH. At LONEOS, +with the solaris sparc station and the linux machines both in use, +ARCH can take on the values of 'linux' and 'sol'. The values 'sun4', +'irix', and 'hp' have also been used at various sites. A simple bit +of csh script in the users .cshrc or equivalent can make this +assignment transparently: + +\begin{verbatim} +set sys=`uname -s` +switch ($sys) + case IRIX64: + setenv ARCH irix; + breaksw; + case SunOS: + set ver=`uname -r | awk '{print substr($1,1,1)}'`; + if ($ver == 5) then + setenv ARCH sol + else + setenv ARCH sun4 + endif + breaksw; + case Linux: + setenv ARCH linux; + breaksw; + default: + echo "unknown architecture"; + setenv ARCH unknown; + breaksw; +endsw +\end{verbatim} + +The Makefiles use the ARCH variable not only for destination +directories, but also for destination binary names. All files of the +form fred.c are compiled to fred.ARCH.o (instead of fred.o), +uninstalled libraries get the names libfred.ARCH.a (instead of +libfred.a), while uninstalled programs are called fred.ARCH. + +The perl scripts are kept in ohana/src/perl, with links to the +appropriate bin directories. All programs are stored in directories +of the form ohana/src/program. The exceptions are gastro (stored in +ohana/src/astro), and control1/2 (stored in ohana/src/astro). Also, a +variety of simple, user-level functions are available in +ohana/src/misc. + +Most of the ohana package of programs use several common libraries. +These include the readline library (except for dophot, this is the only +part of ohana that is not written by Eugene Magnier), which is used +for command-line interface systems. Other important libraries are the +fits library for implementing the FITS specifications in a convenient +C-friendly environment, and the ohana library, consisting of basic +string and other basic operations. + +For more details on the programming aspects of parts of the pipeline, +see the associated document, 'A Programmer's Guide to the Loneos +Pipeline' + +\section {Comparison with USNO catalog} + +I have made several comparisons between the USNO catalog and results +from running images from Loneos through the photometry pipeline. In +general, there is a good agreement between the astrometry of the two +datasets. There are specfic areas where the two disagree, however. +First, as discussed above, the USNO star positions may be +significantly different from the observed star positions simply +because of proper motion. The first picure in the ``Comparisons +between Loneos + USNO'' pages shows several stars with significant +proper motion. These are currently being flagged in the photometry +database as discussed. Other differences evident in this first +picture include the classification of bright stars and the splitting +of faint and bright stars. In the upper right of the image, two +bright stars are visible, one of which is actually a double. The USNO +catalog does a good job of detecting both, but it also detects and +includes as stars several diffraction spike points. The HST GSC only +detected a single point and did not split the star at all. The Loneos +data in our pipeline did not detect the star at all due to saturation. +A star in the upper left corner shows the ability of {\tt dophot} in +the Loneos pipeline to split double stars. This star is, on close +inspection, clearly a double. The USNO catalog only detects one. The +Pipeline not only splits the two stars, but it also makes the correct +cross-identification of the stars. + +The second set of pictures shows the ability of the Loneos pipeline to +deal with small galaxies. There are two small galaxies in this field, +both of which are broken into 4 or 5 objects by the USNO catalog, but +maintained as single objects in the Loneos pipeline. In another +example (not shown) a bright star within the disk of a small galaxies +was clearly split from the galaxy by the pipeline (and also by USNO). + +\appendix +\section{Parameter file params.txt} +{\footnotesize +\begin{verbatim} +# Configuration file for Loneos analysis pipeline + +# important files and directories +GSCFILE /host/pikake/d1/gene/gsc/GSCregions.tbl +GSC_DIR /host/pikake/d1/gene/gsc +CATDIR /host/radon/rad3/gene/photcat2 +IMAGE_CATALOG /host/radon/rad3/gene/photcat2/Images.dat +IMAGE_CATALOG_TEMPLATE /host/radon/rad3/gene/photcat2/template.cat +CATALOG_TEMPLATE /host/radon/rad3/gene/photcat2/template.cat +ROCK_CATALOG /host/radon/rad3/gene/photcat2/Rocks.dat +USNO_CDROM /host/tycho/mnt/md0/kevin/zones +LONEOS_REGIONS /host/pikake/h1/gene/loneos3/regions.map + +# parameters for "fstat" +# instrumental mag zero point +ZERO_PT 24.5 +DOPHOT_CHAR_LINE 129 +DOPHOT_TYPE_FIELD 5 +DOPHOT_AP_FIELD 91 +DOPHOT_PSF_FIELD 52 +MIN_SN_FSTAT 6.0 + +# parameters for "gastro" +DEFAULT_RADIUS 32.0 +MINIMUM_RADIUS 1.5 +MIN_ERROR 0.6 +MIN_PRECISE 0.12 +CCD_PC1_1 1 +CCD_PC2_2 -1 # for loneos +CCD_PC1_2 0 +CCD_PC2_1 0 +ASEC_PIX 2.82 +NFIELD 3.0 +MMIN 6.0 +ROT_ZERO 0 +dROT 1.0 +NROT 2 +POLAR_AXIS_RA -90.0 +POLAR_AXIS_DEC 89.88 +RA_OFFSET -90.0 +DEC_OFFSET 0.0 + +# parameters for "addstar" +# airmass extinction coefficient (in mag / airmass) +NSIGMA 3.0 +ALPHA 0.1 +XOVERSCAN 60 +YOVERSCAN 50 + +# parameters for "controller" +IN_DIR /host/pikake/h1/gene +OUT_DIR /host/pikake/h1/gene/loneos4/ +DOPHOT_PARAMS /host/pikake/h1/gene/src/ohana/config/default_parameters +FLATTEN_SCRIPT /host/pikake/h1/gene/src/ohana/config/test.pro +NEW_IMAGES images.dat +USERNAME gene +PASSWORD gene + +# here we list all available machines +NMACHINES 2 +MACHINE0 pikake.astro.washington.edu +MACHINE1 mildew.astro.washington.edu + +# here we define which machines which are always free +fMACHINE0 1 +fMACHINE1 0 + +# here we define the machine which runs the database +LOCAL_MACHINE 0 + +# parameters for "status" +TIME_REFERENCE 90/01/01 00:00:00 + +# parameters for "nrphot" +# make this one small to boost change of getting an early region +TAU 3.0 +SCATTER_LIM 15.0 +MAG_LIM 17.0 +IMAGE_SCATTER 0.075 +NIMAGE_SCATTER 1.5 +STAR_SCATTER 0.05 + +### parameters for "markstar" +SEARCH_RADIUS 360 # region for initial trail hunt, in arcsec +TRAIL_WIDTH 5.0 # expected trail width, in arcsec +NANGLE_BINS 180 # Number of angular bins in 180 degrees +NPTSINLINE 5 # minimum points needed for trail +MIN_DENSITY 0.025 # minimum linear density in star/arcsec +SPACE_SIGMA 10.0 # how much denser than average for trail? +# parameters defining bright star exclusion regions +BRIGHT_XTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_XTRAIL_MAG 9.5 # faintest sat. star +BRIGHT_XTRAIL_SLOPE -80.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_YTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_YTRAIL_MAG 11.0 # faintest sat. star +BRIGHT_YTRAIL_SLOPE -400.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_HALO_MAG 8.5 # faintest sat. star +BRIGHT_HALO_SLOPE -28.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +# parameters which define the ghosts +GHOST_MAG 7.5 +GHOST_RADIUS 200 # in arcsec +OPTICAL_AXIS1 2154.0 +OPTICAL_AXIS2 2193.0 + +# parameters for "addusno" +USNO_RADIUS 5.0 # in arcsec +USNO_PROPER 20.0 # in arcsec +USNO_RED 1000 # code for USNO Red data +USNO_BLUE 1001 # code for USNO Blue data +# /mnt/cdrom + +# parameters for "markrock" +ROCK_RADIUS 2.0 # in arcsec +ROCK_MAX_RADIUS 20.0 # in arcsec +ROCK_MAX_SPEED 0.1 # in arcsec / sec +\end{verbatim}} + +\end{document} Index: /trunk/Ohana/doc/pipeline/pipeline5.tex =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline5.tex (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline5.tex (revision 2508) @@ -0,0 +1,1990 @@ +%\documentstyle[astro,psfig]{article} +\documentclass{article} +\usepackage{psfig} +\usepackage{astro} +\usepackage{times} +\GoodMargins +\newcommand{\note}[1]{({\bf #1})} +\title{Variability Pipeline} +\author{Eugene Magnier} +\begin{document} +\maketitle + +\section{Overview} + +The Loneos Pipeline provides automated reduction of images, including +photometry and astrometry, with the final data stored in a photometry +database organized by star position on the sky. + +Fig.~\ref{pipeline} shows a schematic of the analysis pipeline. +Images are written to the disk from the camera by the software which +runs it, {\tt astrocam}. The resulting images are FITS files (which +may be arbitrarily sized) and have header keyword parameters RA and +DEC specifying rough celestial coordinates. Images are first analyzed +by {\tt dophot}, which produces instrumental photometry and pixel +coordinate positions for stars and other objects in the images, +writing files with a particular format, called OBJ files in this +discussion. The results from {\tt dophot} are cleaned up by a program +called {\tt fstat} which merges the header information from the image +with the most relevant parts of the OBJ file, creating a new file with +the CMP format. Astrometry is performed on the CMP file with the +program {\tt gastro}. {\tt Gastro} makes a comparison with an +astrometric reference database and only changes CMP header keywords as +a result. Finally, the processed image CMP file is added to the +photometry database with the program {\tt addstar}. The photometry +database is maintained in a large number of files (CPT files), each +representing a small region of sky. There is no relationship between +the image locations and the boundaries of CPT file. All of these +processes occur on individual images, and would typically be performed +while the observations are occuring, or during the day following, but +may be performed at any time to add new or archived data to the +database. + +Once large batches of images have been processed, a series of programs +should be run to clean up the photometry database. Typically, these +routines would be run at the end of a night on all of the CPT files to +which new data were added. First, {\tt markstar} identifies bad data +(the trails of satelites, the bleeding columns from stars, ghost +images), and flags these data points appropriately. Second, {\tt + addusno} matches the catalog stars with stars from the USNO database +and adds the USNO magnitudes to the photometry database. Next, {\tt + markrock} searches for likely asteroids in the CPT file, marks them, +and writes the asteroid observations to an asteroid database. It also +notes measurements which are likely to be noise and cosmic rays. +Finally, {\tt nrphot} determines relative photometric calibrations for +observations in a set of CPT files. + +With the exception of {\tt dophot}, all of the programs in the +pipeline refer to a single configuration file to determine the large +number of parameters necessary in the process. The default parameter +file is hardwired into the programs, but an alternate configuration +file may be specified with the environment variable {\tt + PIPE\_CONFIG}. This option makes independent processing of multiple +data sources trivial. For example, if the pipeline is to be used for +both Loneos and SDSS image analysis and display, there could be two +configuration files (eg, {\tt loneos.txt} and {\tt sdss.txt}), one for +each data source. Several important parameters would be different +between these two, for example, the initial plate scale guess for the +astrometry routine, or the parameters which define the rough chip +orientation. This scheme allows for analysis of different types of +images, with a common destination photometry database, or for +independent photometry databases. + +\begin{figure} +\psfig{file=schematic.eps,width=9cm} +\caption{\label{pipeline} \small + Schematic of the photometry pipeline. Rectangles represent + programs, while ovals represent data products. Arrows show the + direction of travel of information. The flatten routine is a filter + whose position may change as the pipeline matures. } +\end{figure} + +Data from different sources can be processed by the pipeline and +stored in the same photometry database, since all photometry is tagged +with a source ID in the database (as are the images in the image +database). This allows us to include, for example, the USNO catalog +magnitude measurements (or a subset) in the photometry database for a +quick comparion. Different sources may be considered different +filters, so color-color and color-magnitude diagrams could be +extracted by specifying choices for source catalog. Currently each +object has a single primary photometric system, and individual +observations may be combined to determine that average (eg, if they +are different photometry sources with roughly the filter bandpass) or +they may be ignored when calculating relative magnitudes (eg, the USNO +magnitudes would be ignored). This may be expanded upon by defining +multiple average magnitudes and allowing different measurements to +contribute only to the appropriate average. The determination of the +photometry code for a particular image is currently not well defined. +There are several points in the analysis where the user (or the +controlling program) may specify the photometry source. A table +should be kept in the photometry catalog to define the various +photometry codes. + +\section{Image Analysis Pipeline} +\label{parallel} +\subsection{Flat Fielding -- Mana / Flatten} + +Flat fielding is performed on each image after it has been written to +disk by the acquisition software. The processing is performed with +mana (see mana user's guide) using a script written to work with the +analysis pipeline processing software. There are several important +aspects of the script which make it work with the containing programs. +First, the script is contained within a file, flatten.pro, in the +references directory. The script is invoked by calling it on the +command line when mana is started. If mana is started with a file +name on the command line, that file is loaded with in 'input'. The +file defines a macro 'flatten', and also loads the flat-field file. +The flattening process is actually done by calling within mana the +macro flatten, with the input and output file names as arguments. By +loading the flat-field at the start of mana, instead of on each +execution of flatten, we save the load time of each flat-field image +(which is 2x the size of the basic image, being 4byte floats per +pixel). \note{The location of the flat field is currently hard-wired in the +flatten script - this should probably be changed in the future.} One +of the aspects of this is that the macro loads a flatfield with a +specific name and location. This file is actually a link; The +higher-level controlling process (stage1) creates a link to the +correct flatfield. This allows the analysis process to use different +flatfields for different situations. At LONEOS, we currently use a +flat field generated from a well-understood night. This is probably +not the best solution as the flatfield changes with time. As +discussed below, the photometry solution program, nrphot, fits a +high-order polynomial to the relative photometry, allowing for +differences from the correct flatfield. + +\subsection{Photometry -- Dophot} + +Photometry is performed using a varient of {\tt Dophot}. The version +of dophot is based on Dophot 2.0, but it has been adapted to +streamline processing of many files. First, with this version of +dophot, any FITS format data is safely read in. Instead of compiling +to a fixed maximum size, only the memory needed to contain the image +matrix is allocated. The image is also converted to REAL*32 from +whatever BITPIX for processing. Finally, only a single, complete +parameter file is loaded by dophot. There is no option for a modified +and a default parameter file. The parameter file to be used is +defined, along with the input image and the output object list on the +command line when the program is run: \\ +{\tt dophot image.fits image.obj parameter\_file} \\ +To achive this, the code from dophot was wrapped within a C program +which interprets the command line arguments, allocates the necessary +memory, and reads in the image. The rest of the processing is +performed by dophot in the standad manner. Notice that we use a +consistent naming convention through the pipeline reductions. The +image is called {\tt foo.fits}, while the resulting dophot-produced +photometry list is called {\tt foo.obj}. + +\subsection{Cleanup -- fstat} + +After an image is processed by dophot, the object file ({\tt foo.obj}) +is converted to a more-compact, more-complete file called a CMP file, +with a name of the form ({\tt foo.cmp}). This conversion is done with +the program {\tt fstat}. The CMP file consists of the FITS header +from the original image ({\tt foo.fits}), with some additional +keywords to be used at later stages in the analysis, followed by an +ASCII list of the interesting data from ({\tt foo.obj}). In this +list, types 6 and 8 are excluded, and only the following values are +kept: +\begin{verbatim} +X Y Mag dMag t log(sky) +1342.0 106.1 14.166 000 4 3.2 +\end{verbatim} +Objects with a signal-to-noise ratio lower than a specified cutoff +({\tt MIN\_SN\_FSTAT}) are also excluded. Some general information +about the image is derived by {\tt fstat} (FWHM, saturation and +completeness limits, number of each dophot type) and stored as +keywords in the header. The photometry source must be specified at +this stage as one of the command line arguments, though the value used +may be overridden in one of the stages belowe. The resulting file can +now stand on its own without reference to the original image. The +keywords used by {\tt fstat} include the minimum signal to noise, a +rough guess at the zero point ({\tt ZERO\_PT}), and four numbers +defining the format of the {\tt dophot} OBJ file. + +\subsection{Astrometry -- gastro} + +Astrometry is performed automatically by the program {\tt gastro}. +{\tt gastro} loads a CMP file and determines an initial guess for the +center coordinates (based on the RA and DEC header keywords). The +program also uses the configuration information about the plate scale +and rough orientation of the image to get close to the final solution. +Also, the true sky position of the telescope pole may be defined to +allow astrometry on images taken close to the pole. The comparison is +made with the astrometric catalog, which may be the HST Guide Star +Catalog, or it may be any source if the data is placed in the correct +format. + +\section{The Photometry Database} + +\subsection{Photometry File Format} +The photometry database consists of a large number of photometry files +representing sections of the sky and a file containing information on +the images incorporated in the database. These files are called +alternatively ``region files'' or CPT files (because they have the +extension {\tt .cpt}). The photometry files are sorted by Declination +into different directories, each consisting of a band 7.5\degree\ +wide. These directories have names like n0730, representing the band +starting at a Declination of +07:30. The image database (Images.dat) +is stored in the upper level directory of this directory tree. In +fact, the locations of the Images.dat file, the photometry database, +and such are all flexible and may be changed by altering the +configuration file. + +The name and location of each sky region comes from the Hubble Space +Telescope Guide Star Catalog. There is a file, GSCregions.tbl, which +can be used to find the region file appropriate for any location on +the sky. A number of routines exist to make such a query. + +Each photometry file in the photometry database contains all +observations and all average values for all stars observed within the +appropriate region on the sky. The identity of stars is essentially +determined by the RA and DEC coordinates. In very crowded regions, +the end user may need to double check that a neighboring star is not +contaminating individual detections (see the discusion on {\tt + addstar} below). + +The photometry files are stored in a binary format, both to reduce the +volume and to speed access. As a result, the interpretation of the +data is machine dependent: little endian machines need to swap the +data intelligently. Fortunately, this operation is taken care of +appropriately, if the programs are compiled with the BYTESWAP option +set as needed, which is automatically set by the Makefile in the case +of a Linux machine. For reference, Suns, SGI, and HP are all big +endian, while PCs and DECs are little endian. The automatic +conversion takes place by using the funcions {\tt Fread} and {\tt Fwrite} +to substitute for the standard C library {\tt fread} and {\tt fwrite} +functions. {\tt Fread} and {\tt Fwrite} are told the datatype being read, +and they know the layout of the bytes for a particular datatype. +Anytime the definitions of the relevant structures (see below) are +changed, {\tt Fread} and {\tt Fwrite} must be updated. Fortunately, there +is only one file ({\tt Fread.c}), and it has a single byteswapping +section, making it easy to adjust the code appropriately. + +The format of a single photometry database file consists four +sections: +\begin{itemize} +\item {\bf header} -- this is a standard FITS header with room for comments + about the file or whatever. Since the rest of the file is not a + standard FITS file, the SIMPLE keyword is set to False. The three + necessary keywords are NSTARS, NMEAS, and NMISS, which define the + sizes of the next three sections. +\item {\bf average} -- this section contains all average measurement + quantities for each star in the file. There are NSTARS entries, and + each entry is the data from a structure of type Average (defined in + loneos.h, and discussed in more detail below). Thus, the total size + in bytes of this section can be found by NSTARS*sizeof(Average). +\item {\bf measure} -- this section contains the individual + measurements for each star in the database. There are NMEAS + entries, and like the average section, each entry is a single + structure, in this case of type Measure (in loneos.h, see below). + All measurements of a single star are consecutive. The number of + measurements for a single star is defined by average.Nm and the + starting entry for a single star is given by average.offset. + Therefore, the first measurement of a specific star average[500] + would be given by measure[average[500].offset], while the second is + measure[average[500].offset + 1], and so forth. +\item {\bf missing} -- this section contains references to all missing + measurements of a star. This means all of those occasions when an + image enclosed the location of a star in the database, but no star + was detected on the image. Similar to the case of the measure + section, each star has a number of missing entries (which may be 0) + given by average.Nn, and the first missing entry is given by + average.missing. Each missing entry value is again a structure of + type Missing. Currently, the only data in the structure is the time + of the observation, which allows for an unambiguous recovery of the + image where the star was missed. +\end{itemize} + +\subsubsection*{Average Structure defined in loneos.h} +\begin{verbatim} +typedef struct { + float R; /* RA in decimal degrees */ + float D; /* DEC in decimal degrees */ + short int M; /* thousandths of mag (-32.000 to 32.000 valid range) */ + unsigned short int Nm, Nn; /* number of measurements, missing */ + short int Xp; /* scatter in 1/100 arcsec (-327.67 to +327.67 valid range) */ + short int Xm; /* 1000*log(chisq) for magnitude measurement */ + unsigned short int code; /* an ID code (ie, star, ghost, satelite, etc) */ + signed int offset; /* offset to first measurement */ + signed int missing; /* offset to first missing obs */ +} Average; /* 28 bytes / Average */ +\end{verbatim} +The structure above defines the average data stored in the photometry +database for each star. Most of the entries are self-explantory, but +some need a bit of clarification. The entry {\tt Xm} is the magnitude +$\chi^2$ value for the star, under the assumption that the star +brightness is constant. This $\chi^2$ incorporates the photometric +errors on each measurement and is stored as 1000 times the logarithm +of the $\chi^2$ because high dynamic range is not needed. The entry +{\tt Xp} is the scatter about the average postion, in 10 +milli-arcseconds. As discussed above, the values of {\tt Nm} and {\tt + Nn} give the number of measurements and missing observations for +this star, while the entries {\tt offset} and {\tt missing} point to +the first {\bf measure} and {\bf missing} entry for this star. The +entry {\tt M} is the current best average magnitude solution for this +star (see the section on Relative Photometry). Finally, the entry +{\tt code} is an ID code for each object, which may define a variety +of things about the object. For example, if the object is known to be +variable, or if the object is associated with a USNO star, or if the +object is a measurement of an asteroid. See below for details of +these definitions. + +\subsubsection*{Measure Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + short int dR, dD; /* 1/100 of arcsec (-327.67 to +327.67 valid range) */ + short int M; /* thousandths of mag (-32.767 to 32.767 valid range) */ + short int Mcal; /* image cal mag, thousandths of mag (-32.767 to 32.767 valid range) */ + unsigned char dM; /* thousandths of mag (0.000 to 0.255 valid range) */ + char dophot; /* dophot type (1-9) */ + unsigned short int source; /* code to identify photometry source */ + unsigned int t; /* time in seconds (0 - 143 years valid range) */ + unsigned int average; /* reference to corresponding average entry */ + /* upper byte of Measure.average stores flags: + limit of 16,777,215 stars (Naverage) in a file (=0xFFFFFF) + flags: average & 0x1000000 */ +} Measure; /* 20 bytes / Measure */ +\end{verbatim} +} +The structure above defines the individual measurement data stored for +each observation of each star. The first two entries, dR and dD, are +the difference between the average coordinates for this star and the +coordinates determined for this observation. Clearly, if this +residual is too large, the individual measurement will not be +successfully matched with the average star position. {\tt M} and {\tt + dM} are the instrumental magnitude and error for this measurement, +after correction for the exposure time and a zero point (defined in +the configuration file as ZERO\_PT). The entry {\tt t} determines the +time of the individual observation. This is important not only for +timing purposes, but also to determine the source image for this +observation. The time and the entry {\tt source} uniquely determine +the image in the image database that generated this measurement. +Furthermore, this correspondence allows for determination of the pixel +coordinates on the chip, by going throught the image astrometry stored +in the Image database (see below). The entry {\tt source} is a code which +identifies the particular CCD/Telescope/Filter setup. Each unique +{\tt source} should be treated as an independent filter set for the +purposes of accurate relative photometric comparisons. This entry +also allows external catalog data, such as the USNO or Sloan +photometry, to be added to the database, as desired. The entry {\tt + average} is a pointer back to the {\bf average} section to allow +determination of the star from the measurements, in addition to the +reverse. The bits of the upper byte of this value are reserved for +flags used to define particular situations for this measurement. +Possible values are: +\begin{itemize} +\item 1 = BLEND\_IMAGE - the star on the image matched more than one + catalog star +\item 2 = BLEND\_CATALOG - the star in the catalog matched more than + one image star +\item 4 = UPPER\_LIMIT - \note{not really used?} +\item 8 = CALIBRATED - relative photometry has been performed at least + once. +\end{itemize} +The entry {\tt dophot} stores the dophot type for this particular +measurement (1-9: four extra bits in this field). +Finally, the value {\tt Mcal} determines the photometric calibration +of this specific measurment. This value is an offset appropriate to +this image (and if needed, this point in time and this chip position) +to bring the observations of the stars on multiple images to a common +system (see the section on relative photometry). + +\subsection{Image Database} + +All images for which photometry has been included in the photometry +database also have entries in an image database. The image database +is (currently) a single file in the upper directory of the photometry +database. The name of the file is determined by the IMAGE\_CATALOG +entry in the configuration file, and is currently set to Images.dat. +Like the photometry database files, the image database consists of a +FITS header followed by binary data. There is only one type of binary +data: each image has an entry, which is the data from a structure of +type Image (loneos.h). The number of images in the file is determined +by the {\tt NIMAGES} keyword. If the number of images becomes too +large, extrapolation of the system to a set of image database files +with a reference list is quite straightforeward, and could be modeled +on the organization of the region files (eg, one image database file +for each Declination directory). + +\subsubsection*{Image Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + Coords coords; /* 120 bytes */ + unsigned int tzero; /* readout time row 0 in sec (0 - 142 years valid range) */ + unsigned int nstar; /* number of stars on image */ + short int secz; /* thousanths of airmass (valid range -32.000 -- 32.000) */ + short int NX, NY; /* dimensions of image */ + short int apmifit, dapmifit; /* aperture correction and error in thousandths of mag */ + short int source; /* identifier for CCD (each ever used will have a unique letter) */ + short int Mcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int dMcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int Xm; /* 10*log(image chi-square) */ + char name[32]; /* name of original image */ + unsigned char detection_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char saturation_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char cerror; /* astrometric error: 1/50 of arcsec (0 -- 5.12 valid range) */ + unsigned char fwhm_x, fwhm_y; /* PSF terms in 25*arcsec (valid range 0.0 -- 10.2" ") */ + unsigned char trate; /* 10000 * scan rate in sec/pix (0 -- 0.0256 valid range, + typically 0.0146. this is used only to determine the time + of the observation, not to find the coordinates. 1 byte + gives 0.11 sec accuracy */ + float exptime; /* exposure time, seconds */ + char code; /* flag to mark an image as bad or whatever */ + char dummy[21]; /* extra space for the future */ + short int order; /* number of terms used for Mrel */ + short int Mx, My; + short int Mxx, Mxy, Myy; + short int Mxxx, Mxxy, Mxyy, Myyy; + short int Mxxxx, Mxxxy, Mxxyy, Mxyyy, Myyyy; +} Image; /* 240 bytes / Image */ +\end{verbatim}} + +The structure above lists all of the data stored for each image. A +substantial amount of information is stored for each image. First, +the astrometric calibration information is stored in a {\tt Coords} +structure (see coordinate systems, above). The upper and lower limits +of the photometry in the database (in instrumental magnitudes) is +stored in {\tt detection\_limit} and {\tt saturation\_limit}. These +values are useful for comparison for those stars where observations +are missing. The relationship between time and pixel coordinates is +given by {\tt tzero} and {\tt trate}. Drift images will have a scan +rate stored in {\tt trate}, while staring images will have a value of +0.0 for this entry ({\tt tzero} then applies to the whole image). +Other important parameters are kept: the astrometric error ({\tt + cerror}), the airmass ({\tt secz}), the exposure time ({\tt + exptime}), the number of stars detected ({\tt nstar}), the FWHM +values ({\tt fwhm\_x, fwhm\_y}), the image name ({\tt name}), and the +size of the image ({\tt NX, NY}). The entry {\tt source} is used as +above to define the CCD/Telescope/Filter used for this observation. +The entry {\tt code} is used to flag images in various ways. +Currently, only two bits are used, to define if the image has been +photometrically calibrated and if the calibration is acceptable. +There are 21 extra bytes are stored for future additions. + +\subsection{Data Incorporation -- {\tt addstar}} + +Images which are completely processed ({\tt dophot, fstat, gastro}) are +then incorporated into the photometry database with the program {\tt +addstar}. This program decides which region files are appropriate +for this particular image, then one-by-one adds stars from the image +to the appropriate region file. Stars already in the catalog are +matched with stars in the new image purely by a positional comparison. +In order to avoid the difficulty of comparisons in the RA and DEC +coordinate frame, a cartesian projection is performed. The stars from +the image being processed and the database stars in the same area are +projected onto a tangent plane and positional comparisons made in this +(locally cartesian) coordinate frame. This avoids the dangerous +singularities at the pole and also makes the RA 0,360\degree\ boundary +a trivial problem. + +Several choices must be made in the comparison process. If a star in +the catalog is matched with a star in the image, the new measurements +of that star are added to the database. If a star in the database +lies in the field of the image, but is not detected, this information +is added to the list of missing data. The only data stored in this +case is the time and source, which is sufficient to unambiguously +identify the source image from the image database. This allows later +programs to find relevant statistics from the image database, if +necessary. If a star is detected in the image, but is not already in +the database, a new entry is added to the database. In addition, the +same ``missing data'' from all previous observations images which have +covered this location (ie, images already in the image database) are +included. This last step is necessary so that the image processing +order is not important. + +Stars also run the danger of being crowded together. Since all +comparisons are performed on the basis of position alone, crowded +fields may make for ambiguous cross-identifications. A pair of stars +are matched if the difference in their positions is less than a +specified search radius (dependent on the scatter in the astrometric +solution for the image). If more than one catalog stars are correlated +with a star in an image, the new measurement is added to {\em both} +catalog stars, and a flag is set noting that this observation had a +blended IMAGE. Conversely, if a single catalog star is matched with +more than one image star, both new measurements are added to the one +catalog star, and a different flat is set, noting that this +observation had a blended CATALOG. + +\section{Catalog Update and Cleaning} + +Occasionally, once a large number of images have been added to the +photometry database, a set of programs should be run to clean and +update the CPT files. This may best be done on CPT files after a +night's worth of data is incorporated. These routines define bad +data, identify the USNO catalog stars, and search for asteroids and +other junk. Most of the results from these routines are the addition +of object flags to the Average data structures (Average.code). Most +of these routines process data within a single CPT file at a time. + +\subsection{Markstar} + +The first of the update programs, {\tt markstar}, identifies bad data +and flags it. It makes three types of identifications: bright stars, +ghost stars, and trails. In the first case, it searches the HST guide +star catalog for bright stars within the field of the current CPT +file. Three types of pixels are flagged for bright stars. Any +measurements within a circular region centered on the star are suspect +(the ``halo'' of the bright star). Second, points along the X axis +are likely to contain flux from the diffraction spikes. Finally, +points along the Y axis are likely to have bleeding in addition to the +diffraction spikes. Each of these three regions has a different size +and range which must be defined in the configuration file. The +regions are represented by a representative magnitude and a scale +parameter. For example, the diameter $d$ of the halo for a star with +magnitude $m$ is defined as +$d = \mbox{BRIGHT\_HALO\_SLOPE} (m - \mbox{BRIGHT\_HALO\_MAG})$. The X and +Y axis points are defined by the length of the diffraction spike, with +an equivalent formula, and the width of the spike. Since these are +each independent, there are 8 parameters to define the exclusion +regions of bright stars. One of the current drawbacks of this system +is the reliance on the HST Guide Star photometry. In the HST Guide +Star Catalog, only one photometry band is given, so no color +information is available. The parameters currently used were defined +based on a large number of bright stars, with no adjustment for the +different effective filter system of the Loneos telescope and the HST +GSC. Thus, occasionally very red stars will be several magnitudes +brighter in the real data than reported in the catalog. As a result, +large numbers of bad data points in these halos and spikes may be +missed. + +The second type of bad data flagged by {\tt markstar} are the trails +from satelite and other space junk. This portion of the program +searches for objects which land along a line and which have a high +linear density. Parameters may be adjusted to define the width and +the necessary density or number of points in the line. Objects in +lines are also only identified if all observations come from a common +image. If an object only has one measurement and it is in a trail, +the entire Average is flagged to be bad. However, if an object has +many measurements and only one is bad, the object is only flagged as +having some bad data points. + +The third type of bad data are ghost star images. The HST GSC is +searched for stars projected across the telescope optical axis, which +is defined by the user. All objects within a region of fixed size are +marked for ghost stars brighter than a specific cutoff magnitude. +Similar to the trails, the average is marked as bad only if all +detections of an object are ghost images. Otherwise, it is noted that +some data may be bad. + +To run 'markstar' by hand, the syntax is: {\tt markstar (file.cpt)}. +There are several optional flags, which are listed if you type +markstar by itself. + +\subsection{Addusno} + +The program {\tt addusno} makes two types of identifications of stars +with the USNO catalog. First, it searches for coincidences within a +small, specified radius. The choice of this radius is somewhat +tricky: it is important to catch the association, but also to avoid +making too many associations. Unfortunately, astrometric errors in +the USNO catalog make it necessary to choose a surprisingly large +radius of 5\asec. To avoid making double matches (matching the two +USNO stars to the same photometry database object), only the USNO +object closest to the object is associated. There are also a +substantial number of stars with significant proper motion between the +USNO epochs and the current date. It is not unusual to see 8-15\asec\ +discrepancy for specific stars. To catch these objects, any objects +which do not match the USNO database, but which have more than a +minimum number of measurements (2 or 3?) are searched for more distant +USNO companions. A large, user-specified radius is used for this +search. Of course, only USNO stars which are not already matched to +the database may be candidates for the proper motion match. This +stage is susceptible to errors in that a slow-moving solar-system +object may be associated with a faint USNO star. For this reason, +objects flagged as high proper motion stars should be taken with some +caution. Of course, during the {\tt addusno} stage, no objects +already flagged as bad by {\tt markstar} will be matched with USNO +stars. + +To run 'addusno' by hand, the syntax is: {\tt addusno (file.cpt)}. +There are several optional flags, which are listed if you type +addusno by itself. + +\subsection{Markrock} + +Once stars have been identified with the USNO database, it is possible +to search for asteroid detections. The program {\tt markrock} +searches within a specific CPT file for objects which are moving along +straight lines in (X,Y,t) space. In order for an object to be found +by {\tt markrock}, it must have three points, each with only single +measurements. The third point must land within a specified distance +of the line projected from the first two data points. These +comparisons are made in the three dimensional space of image position +and time. Also, objects are only accepted if they have a speed less +than a threshold. If an object is moving too fast, it will make a +streak on each individual image and will probably be missed anyway. +Moving objects detected in this way are flagged in the photometry +database and are also written to a Rock database. Currently, the Rock +database just contains ASCII lists of the RA, DEC, Mag, and time for +each detection. In fact, the data are stored in pairs of detections, +with the middle detection saved twice. This format makes it easy to +plot lines connecting the points using the connect-the-dots plotting +style of status/kapa (see below). + +One of the current drawbacks of {\tt markrock} is the insistence on +three observations with only 1 measurement each. If an object is +moving too slowly, two of the dectections of the object may be merged +together. Enough data is stored in the database to extract these +individual measurements, so a more sophisticated search is possible. +It is also currently the case that USNO proper motion stars are +ignored, but some of these will be asteroid detections which are +simply too close to a faint USNO stars (it has to be faint because it +would otherwise be associated with a star in the photometry +database). + +To run 'markrock' by hand, the syntax is: {\tt markrock (file.cpt)}. +There are several optional flags, which are listed if you type +markrock by itself. + +\section{Relative Photometry -- nrphot} + +Once the data has been incorporated in the photometry database and the +data validation routines have been run, it is possible to determine +photometric solutions for all of the images. The program which does +this is nrphot. It tries to find a set of solutions for each image +which reduce both the scatter per image and the scatter per star. +\note{the following section is quoted verbatim from the M31 paper + Magnier et al 1992, A\&A Supp 96, 379. It could use a little + editing to fit the reality of the LONEOS implementation.} + +In order to convert from instrumental magnitudes reported by DOPHOT +to a standard photometric system, relative photometry was used, and +all values converted to one internal magnitude system for the entire +set of data taken with each filter. Note that this is not {\em + differential}\ photometry on a single image, but rather {\em + relative}\ photometry between many images. This method was +advantageous for several reasons. First, since many of our nights +were not photometric, relative photometry allowed us to use all of our +data, even that obtained on the cloudy nights. Second, it allowed us +to use photometry from several of the recent CCD photometric studies +(Massey \etal\ 1986; Hodge \& Lee 1988; Freedman 1989; Richer \etal\ +1990) in small fields of M31 to calibrate our data to a standard +system. Third, relative photometry allowed us to check the agreement +of the photometry of the calibration data. + + In order to perform relative photometry, we used multiple +measurements of individual stars on overlapping images to connect +neighboring images. A standard method of converting between +instrumental magnitudes and a standard photometric system is to use a +photometry equation of the form: +\begin{equation} +M_{app} = c_{\lambda} + m + a_{\lambda}*\zeta + \gamma_{color}*(color) \\ +\label{M_mag} +\end{equation} +where $m$ is the instrumental magnitude, +\begin{equation} +m = -2.5\log(N_{el}) + 2.5\log(t). \\ +\label{inst_mag} +\end{equation} +$N_{el}$ is the number of electrons measured in a single star, $t$ is +the exposure time for the image, $M_{app}$ is the apparent magnitude +of the star in the standard system, $\zeta$ is the airmass of the +image, and $color$ is the value of a color index (\ie\ \B\ - \V\ or +\V\ - \R) for the particular star. The subscript $\lambda$ refers to +the filter and the subscript $color$ refers to the color index. For a +given filter, several possible color indices could be used (\ie\, for +calibration of \V\ data, both \B\ - \V\ and \V\ - \R\ could be used to +measure the response of the detector with the color of the star); +however the colors of stars are highly correlated, so only one color +index need be used at a time. $c_{\lambda}$, $a_{\lambda}$, and +$\gamma_{color}$ are calibration coefficients dependent on the +instruments and the site. $c_{\lambda}$ is a factor to scale the +response of the detector. In fact, the value determined for +$c_{\lambda}$ depends on what color index is used in the calibration, +so one might in fact write $c_{\lambda}$ as $c_{\lambda, color}$. +However, for the sake of readability, this second subscript will be +dropped. The terms $\gamma_{color}$ and $a_{\lambda}$ give the +variation of the sensitivity with the color of the star and the +airmass. For a detailed discussion of photometric calibrations, see +Hardie (1962). Equation (\ref{M_mag}) is only valid for photometric +conditions. Under non-photometric conditions, we need to add another +term to account for clouds: +\begin{equation} +M_{app} = c_{\lambda} + m + a_{\lambda}*\zeta + \gamma_{color}*(color) + clouds \\ +\label{M_cloud} +\end{equation} +Now, we can regroup the terms in (\ref{M_cloud}) as follows: +\begin{equation} +m = [M_{app} - \gamma_{color}*(color)] - [c_{\lambda} + a_{\lambda}*\zeta + clouds] +\label{M_sep} +\end{equation} +Of the two terms on the right side of equation (\ref{M_sep}), the variables +in the first pair of brackets are dependent on the properties of the +star, those in the second pair of brackets are dependent on the image parameters, +except for $c_{\lambda}$ which is independent of both image and star. +We can now define the ``relative magnitude'' and the ``calibration +magnitude'' from (\ref{M_sep}) as follows: +\begin{eqnarray} + m & = & M_{rel} + M_{cal} \label{rel_phot1}\\ + M_{rel} & = & [M_{app} - \gamma_{color}*(color) + \Delta] \label{color}\\ + M_{cal} & = & - [c_{\lambda} + a_{\lambda}*\zeta + clouds + \Delta] \label{M_cal} +\end{eqnarray} +where $M_{rel}$ is the relative photometric magnitude of the star in +the internal system and $M_{cal}$ is a correction factor to convert +instrumental magnitudes to relative magnitudes. The terms $\Delta$ +are included in eqs. (\ref{color}) and (\ref{M_cal}) to make explicit +the fact that an arbitrary constant can be added to $M_{rel}$ and +subtracted from $M_{cal}$, as this fact will be important (see below). +The goal of relative photometry is to determine $M_{cal}$ for each +image, then use this $M_{cal}$ to find $M_{rel}$ for all stars from +eq. (\ref{rel_phot1}). Once one has $M_{rel}$ for each star, one can +then convert them to a standard system ($M_{app}$) using equation +(\ref{color}). + + An equation of the form (\ref{rel_phot1}) exists for each star +detected on each image. As described, the value $m$ depends on both +the particular image and the particular star, the value of $M_{cal}$ +only depends on the image, and the value of $M_{rel}$ only depends on +the particular star. Let us label the images with the subscript $i$ +and the stars with the subscript $j$. Equation (\ref{rel_phot1}) can +then be written in the form: +\begin{equation} +m_{i,j} = M_{rel,j} + M_{cal,i} \label{rel_phot} +\end{equation} +Of course, $m_{i,j}$ only exists for certain combinations of images +and stars: not all stars appear on all images. In the entire system +of equations, only one value of $\Delta$ (from eqs. \ref{color} and +\ref{M_cal}) can be used, but, as noted above, the {\em value}\ of +$\Delta$ is arbitrary. + + In equation (\ref{rel_phot}), both $M_{rel,j}$ and $M_{cal,i}$ +are unknown quantities. We can use a method of least squares to find +these values for the entire set of data. We try to minimize the chi +square, defined as: +\begin{equation} +\chisq = \sum_{i,j}(m_{i,j} - M_{rel,j} - M_{cal,i})^2 / \sigma_{i,j}^2 \label{chisquare} +\end{equation} +where $\sigma_{i,j}$ is the error in the measurement $m_{i,j}$, and +the index $i$ runs over all images, while the index $j$ runs over all +stars which are multiply measured, \ie\ those stars which appear on more +than one image. + + We attempt to minimize \chisq\ analytically by finding the +derivatives of \chisq\ with respect to $M_{rel,j}$ and $M_{cal,i}$ +and setting them equal to zero: +\begin{eqnarray} +\frac{\partial\chisq}{\partial M_{rel,j}} = \sum_{i}-2(m_{i,j} - +M_{rel,j} - M_{cal,i}) / \sigma_{i,j}^2 = 0 \nonumber \\ +\frac{\partial\chisq}{\partial M_{cal,i}} = \sum_{j}-2(m_{i,j} - +M_{rel,j} - M_{cal,i}) / \sigma_{i,j}^2 = 0 +\end{eqnarray} +In these summations, the index $i$ runs over all images in which star +$j$ appears, and the index $j$ runs over all stars which appear on +image $i$ which are multiply measured. Solving the equality leads to +the following system of equations: +\begin{eqnarray} +R_{j}M_{rel,j} = \sum_{i}(m_{i,j} - M_{cal,i}) / \sigma_{i,j}^2 \label{system1} \\ +R_{i}M_{cal,i} = \sum_{j}(m_{i,j} - M_{rel,j}) / \sigma_{i,j}^2 \label{system2} +\end{eqnarray} +where +\begin{eqnarray} +R_{i} = \sum_{j}\frac{1}{\sigma_{i,j}^2} \nonumber \\ +R_{j} = \sum_{i}\frac{1}{\sigma_{i,j}^2} +\end{eqnarray} +Again, the index $i$ runs over all images in which star $j$ appears, +and the index $j$ runs over all stars which appear on image $i$, with +the restriction of multiple measurements. The term $\Delta$ from +eqs. (\ref{color}) and (\ref{M_cal}) acts as an arbitrary zero point +between the relative and apparent photometric systems. + + Equations (\ref{system1}) and (\ref{system2}) are a set of +$N_{images} + N_{stars}$ linear equations with $N_{images} + +N_{stars}$ unknowns. The unknown parameters are the terms $M_{rel,j}$ +and $M_{cal,i}$. Unfortunately, $N_{images} + N_{stars}$ is a number +on the order of 5000 - 10000, thus an analytical solution is +impractical and an iteration method must be used. A simple iteration +method is as follows: + +%\SingleSpace +\begin{enumerate} +\setlength{\parskip}{-0.05in} +\item Guess at values for each $M_{cal,i}$. +\item Use equation (\ref{system1}) to find values for $M_{rel,j}$. +\item Substitute these $M_{rel,j}$ values into (\ref{system2}) to get a new +set of $M_{cal,i}$ values. +\item Return to step 1. +\end{enumerate} +%\DoubleSpace +This process is repeated until some criterion is reached, such as the +\chisq\ no longer changes or reaches a desired minimum. Although a +minimum can be found, error propagation causes severe problems +since images are only connected to neighboring images. The end point +images are connected by \approx 50 intermediate images, and errors +propagating over such a long distance can cause severe errors. Also, +$M_{rel,j}$ and $M_{cal,i}$ are not uniquely defined, because of the +term $\Delta$. Therefore, during the iteration process, the values of +$M_{rel,j}$ and $M_{cal,i}$ tend to wander and do not tend to converge +($M_{cal,i}$ may be incorrect by typically 0.5 mag). These problems +can be overcome if some of the values $M_{cal,i}$ or $M_{rel,j}$ are +known. In this situation, the known values can be held fixed, +providing a reference point for the remaining values. If enough +values are known across a large enough portion of the data, no image +is separated from the fixed values by many images, and errors will not +build up. + + Although many values $M_{rel,j}$ were known from other CCD +photometry studies, we chose not to use them. The reason for this +decision was threefold: 1) By not using them, we are able to check the +effectiveness of our method. 2) We are able to check the accuracy of +the reference data. 3) The known $M_{rel,j}$ values were in a very +small number of highly concentrated regions (the locations of the CCD +photometry studies). This was particularly a problem in \R\ and \I, +where we only had one or two reference fields. We therefore chose to +fix certain values of $M_{cal,i}$ + + Images taken under photometric conditions provide a set of +known $M_{cal,i}$ which can be held fixed. This is true since, if the +image is taken under photometric conditions, the term $clouds$ in eq. +(\ref{M_cal}) is by definition 0. The other terms in this equation +are either known ($a_{\lambda}*\zeta$) or are constant for all images +($c_{\lambda}$) and can thus be assigned to a set value, which can +later be removed with the zero point calibration. We used the values +of $a_{\lambda}$ reported by Pilachowski \etal\ (1991): $a_B$, $a_V$, +$a_R$, $a_I$ = (0.24, 0.15, 0.11, 0.07). We used the long-term +average coefficients, not those measured in October 1991, since the +observations were performed before the eruption of Mount Pinatubo. +The task that remains is to determine which images were photometric. + + Since the conditions during observations varied greatly from +night to night, we decided to determine which images were photometric +from the images themselves. To do this, we used a variation on the +iteration method described above. Before starting the iterations, we +assume a value for $c_{\lambda}$. We can choose this arbitrarily, and +correct for our choice in the correction to an absolute photometric +system. Now, we perform the following iteration: + +%\SingleSpace +\begin{enumerate} +\setlength{\parskip}{-0.05in} +\item Assume that all images are photometric and determine $M_{cal,i}$. +\item Find the corresponding values of $M_{rel,j}$ from equation (\ref{system1}). +\item Use these $M_{rel,j}$ values to find new values for $M_{cal,i}$. +\item Find the values of $clouds$ for each image from these $M_{cal,i}$ +\item Assume that any image with a value $clouds$ more than 1 standard +deviation from the mean value (over images) has a substantial cloud +layer. +\item Return to step 1, but use the iterated $M_{cal,i}$ values for those +images with substantial clouds. +\end{enumerate} +%\DoubleSpace +This process is continued until \chisq (eq. \ref{chisquare}) does not +change substantially. Figure (\ref{fig:Mrel}) shows a histogram of +the occurrences of different $clouds$ values. There are a few +important points to make about this figure. First, there is a +concentrated peak at 0. All of these images have essentially zero +cloud level. As one would expect, there are a large number of images +to the right of this peak. These are the images which were taken +through some amount of clouds. Since the exposures were stopped if +the clouds got too thick, it is not surprising that these images taper +off at higher cloud levels. The most curious and telling part of the +plot is the area below the 0-cloud peak. There are many images on the +``negative cloud'' side of the peak. Because ``negative clouds'' are +unphysical, there must be a simple explanation for this. In fact, 112 +of the 122 images more than 0.02 mag below the peak had exposure times +of less than 20 seconds. Since the shutter at MDM is accurate to only +1 - 2 seconds, it is likely that these images had an exposure time +which was longer than reported, and therefore have artificially +enhanced values of $M_{cal,i}$. + + We calculate the error on the relative calibration of each +image by finding the scatter of the $M_{cal,i}$ values for each image +and dividing by $\sqrt{N stars}$. A histogram of this error for all +images is shown in Figure (\ref{fig:dMrel}). We also show the +variation of the formal photometric error reported by \dophot\ with +magnitude for each filter in Figure (\ref{fig:dm}). Now that we have +the values $M_{cal,i}$, we can calculate the values $M_{rel,j}$ for +every star on every image. + +The Pipeline implementation of relative photometry includes the +possibility for up to third order polynomial terms across the field. +This is needed to account for the different flat fields, particularly +between images taken with and without the red-block filter. + +\begin{figure} +\psfig{file=flags.eps,width=9cm} +\caption{\label{flags} \small + Pictoral representation of the Average.code flags. The lower byte + value is used for specific definitions of object types. } +\end{figure} + +To run 'nrphot' by hand, the syntax is: {\tt nrphot (file.cpt)}. +There are several optional flags, which are listed if you type +nrphot by itself. + +\subsection{Data Flags} + +The identifications made by the above routines are noted as a set of +flags in the Average.code structure entry. Some of these flags are +mutually exclusive, so a certain bit may have more than one meaning, +depending on the value of other bits. Figure~\ref{flags} shows a +pictoral representation of the meaning of the different bit fields. +The uppermost bit (bit 15) determines if the object may be considered +a ``fixed'' star or something which has no fixed location. If it is +not fixed, and this bit is 1, then there are several options. The +second uppermost bit (bit 14) will be set if this object is one of the +three types of bad data flagged by {\tt markstar}. The lower byte of +code is used to define the type of bad data: ghost, trail, or bleed. +Thus, any datapoints determined to be a satelite trail will have a +code value of 0xc002 (49154 decimal). If an object is a moving +object, but not a bad datapoint, it may be identified as a ``rock'' +(an asteroid) by the program {\tt markrock}. In this case, the flag +for an asteroid is turned on (bit 13). Thus anything identified as a +rock by {\tt markrock} will have a code value of 0xa000. The lower +byte of code is not yet defined for asteroids, but could be reserved +for distinguishing different classes of asteroids (ie, on the basis of +orbital speeds, and so forth). Objects which are fixed may have a +variety of possible flags. First, an object may be identified with a +USNO star (bit 14). It may be found to exhibit variability (bit 13), +implying that the relative photometry routine should ignore it. If +may be a transient object (bit 12). Note that, while an object +associated with a USNO object may not be transient, an object which is +variable may also not be associated with a USNO object. Thus, we need +seperate bits for variable vs transient. Different types of variable +objects may be represented with different values of the lower byte. +For example, a Cepheid may have a specific lower byte code of 0x01, +which an RR Lyrae may have a lower byte code of 0x02. Thus a Cepheid +associated with the USNO catalog would have a total code value of +0x6001. Finally, any of these types of non-moving objects may have +some bad individual measurements (bit 12) or may be found to have a +significant proper motion (bit 11). It should be noted that the above +guidelines for variable stars are suggestions only, and to date (July +7, 1999), no programs currently assign or define these variability flags. + +\section{Holding it Together} + +The preceeding sections describe the step-by-step analysis and +incorporation of the data from individual images. However, the power +of the Pipeline comes in applying it to large numbers of images. At +LONEOS, we are using four Pentinum II computers to analyse the roughly 200 +images that are observed each night. + +It is best to think of the analysis of images occurring on groups of +images from a given night. Of the set of processes discussed above, +those in section \ref{parallel} are performed on individual images +essentially independently of the other images. The rest of the steps, +both the data incorporation ({\tt addstar} and the database cleaning +routines occur on the entire batch, one image at a time. Thus, the +former routines can easily be run in parallel, but the latter routines +are run in serial to avoid having two programs writing to same part of +the database at the same time. In this model, we would process all of +the images from a night through the parallel stage of the analysis and +then run the serial stage only after the entire night is finished in +the parallel stage. We thus have a natural division of the pipeline +into two stages. + +The implementation of the two stages involves several programs, some +of which are specific to the LONEOS situation, and others which are +more general. There are two layers of programs which enable batch +processing of many images and repeated batch processing over many +nights. At the highest level, are Perl scripts which choose what data +needs to be analysed and when. These scripts tend to be more +dependent on the details of the LONEOS implementation. At the next +level down, these scripts call two main C programs which run the +images of a given night through the pipeline. These C programs are +more general and do not depend on the details of the LONEOS +implementation. A defined set of files holds a log of the nights and +images that have been processed, both for the information of people +who might want to monitor the progress, but also used by the programs +to guide decisions about what to analyse next. + +\subsection{data and file organization} + +There are a set of standard locations for all of the files used and +created by the Pipeline. These locations have both abstract names +used by the pipeline programs and specific definitions used in the +LONEOS implementation. In general, the definitions of the files are +given in the configuration file, 'config.txt' (currently located in a +hard-wired location, but this may soon change to the mechanism used by +the lastest version of spicam (3.0)). The C programs all refer to the +config file for the definitions, but the Perl scripts currently have +the definitions hardwired as variable names defined in the beginning +of each file. This should be changed to a mechanism that looks in +the config file, perhaps implemented by a standard Perl function. + +Here is a list of the main directories and their meanings: + +logdir: /metis/d11/logs - all log files and the files used by +the Perl scripts to control their actions (except locks) go here. + +workspace: /irene/d11/workspace - the intermediate analysis +stages go here, and the tar, gzipped directory files stay here. + +dumpspace: /pallad/d1 - used by slurp to store images +downloaded from the tape archive. + +database: /hebe/d27/database - the result database, as also +the locks and the Rocks.dat file. + +references: /metis/d11/references - a variety of reference data, +including the HST GSC, the USNO catalog, the config files, and so +forth. + +source: /metis/d11/src/ohana - all of the programs source, +binaries, and scripts relevant to running the Pipeline are contained +in these directories. + +\begin{verbatim} +database: /hebe/d27/database (also in /metis/d27/database as linked directories). + lock files also go in this directory +workspace: /irene/d11/workspace - temporary storage of result files +dumpspace: /pallas/d1 - disk to store data dumped from the archive by slurp. +scriptdir: /metis/d11/logs - all log files +\end{verbatim} + +\subsection{lastnight} + +The first step in the process is the Perl script {\tt lastnight}, +which is highly dependent on the specifics of the LONEOS setup. It +looks at the disks used by the observers to write the images from a +given night. Images are written to a directory called +/pallas/d[1-6]/YYMMDD where 'YYMMDD' is an abbreviation for the current +night's date (ie, 990223). Many files use this naming scheme, so we +will refer to such a name as 'YYMMDD'. All of the images are stored +in this directory with names yyyyMMDDxxxxb.fits where yyyy is the full +year, MM is month, DD is date, and xxxx is a sequence number. The 'b' +refers to the 'b' CCD, and the .fits extension is required. The +script {\tt lastnight} is called by a cron job scheduled for 7am on +{\tt hebe.lowell.edu} (note that the clock on hebe is set to PST, not +local MST - fix this some day!). The script searches the disks +{\tt /pallas/d[1-6]} for a directory of the right form (basically any +directory beginning with a number). By default, the program demands +that the directory have the name derived from today's date, but with +the flag -any, it will accept any directory. It then checks to see if +this directory has been already been analysed (ie, it is included in +the 'processed.log' file). If not, it lists all files in that +directory with the right ending (*b.fits) and places these names in a +file, {\tt /metis/d11/logs/YYMMDD.inlist}. It also writes the name of the +directory (along with some other data) in a file 'extracted.log'. +When it has successfully ended and identified some data, it calls the +next script, {\tt stage1} and then quits. + +\subsection{slurp} + +An alternative to {\tt lastnight} is the Perl script {\tt slurp}, +which looks for data on the tape archive. This script looks at the +tape.log file and compares it to the file 'processed.log'. It tries +to identify nights listed in the file 'tape.log' which have not been +included in the 'processed.log'. If it finds any, it tries to +download as much of the data as will fit on the disk {\tt /pallas/d1}. +This script is called from within the {\tt stage1} script and is +invoked if {\tt stage1} is run without any data already in +the file 'extracted.log'. + +A related pair program is the load/unload C programs which are needed +for loading and unloading the tape jukebox. The first, load N, takes +tape number N and loads it into the tape drive. The second takes the +tape a returns it to its slot. It is important to check on the other +users of the tape drive to be sure it is not occupied. This is done +with the llock, lunlock and llockstat programs. llockstat lists the +devices which have been allocated. llock lets you lock a particular +service, etc. + +\subsection{stage1} + +The Perl script {\tt stage1} takes the image list produced by {\tt +lastnight} or {\tt slurp} (YYMMDD.inlist) and passes the list to the +controller program {\tt control1} which performs the parallel +analysis. The script has a few features to decide which files have +been analysed through the pipeline and to what extent. If {\tt +stage1} is called, it should be able to figure out 1) if there is data +ready to be analysed (ie, there is a new date reference in the file +'extracted.log'), 2) if the data has been partially analysed (ie, the +date reference in 'extracted.log' has not been put in 'processed.log', +but there are entries in the file YYMMDD.outlist), and 3) which data +needs to be run through which part of the pipeline (ie, it decides +which of the images in YYMMDD.outlist have succeeded at which stages +of the analysis, and starts them at the appropriate spot). These +features are most useful if the analysis crashes in the middle or +needs to be restarted. The {\tt stage1} script is meant to be run +without any arguments and it will figure out what needs to be done. +The one possible argument is the -once flag. This flag tells {\tt +stage1} to analyse the data it finds and quit when it finishes, and +not call another instance of itself. Otherwise, when it is done, it +will restart itself, and since there is no new data in +'extracted.log', it will start a slurp process to get data from the +archive. If there is no data in the archive either, this process will +sleep for an hour and try again. After a few attempts (2 days?), {\tt +stage1} will give up and quit without restarting itself. The script +writes a log file called \$scriptdir/stage1.NN.log where NN is a +number in sequence between all runs of {\tt stage1}. A file +stage1.count keeps track of which number NN we are on. When {\tt +stage1} has finished the analysis of a night's data, it writes the +date in 'processed.log' to let other processes know what has been +analysed. The bulk of {\tt stage1} is not dependent on details of the +LONEOS setup, but it does expect the files to have the naming scheme +described above. + +\subsection{stage2} +The serial stage processes are run by the Perl scrip {\tt stage2}. +Normally, this script is always running. It waits for the name of a +night to be added to the 'processed.log file, presumably by {\tt +stage1}. Once it finds a new name in this file, it looks for the +appropriate images in the file 'YYMMDD.addlist'. It then submits the +list to the second controlling program {\tt control2}, which runs the +images through {\tt addstar}. Images which are successfully run +through addstar have their *bf.fits (flattened image) and *.obj +(dophot output) files deleted. When the entire night is done, the +rest of the *.obj and *bf.fits images are deleted by {\tt stage2}, the +directory with the *.cmp and *.log files is tared and gzipped into the +file YYMMNN.tgz. The directory is then deleted. Finally, {\tt +stage2} calls the scripts {\tt cleaning.pl} and {\tt run.phot} which +runs the cleanup programs ({\tt markstar - nrphot}) on the database. +After a successful run, {\tt stage2} will automatically respawn a new +copy that waits until more data arrives. + +\subsection{control1 / control2} +These two C-programs perform the tasks of farming the images out to +the different machines for analysis. The two programs are essentially +identical, but {\tt control1} is allowed to use all machines and {\tt +control2} can only use one (hebe.lowell.edu, which has the database +local to it). These controller programs start off by executing remote +shells (using ssh) on a set of machines. These shells are then used +to execute the remote commands and are kept open the entire time the +process runs. + +Both control programs maintain a list of images in a set of queues. A +set of rules defines the order in which each image travels through the +queues. Each queue is associated with a single process (ie, flatten, +astrometry, etc). Images which successfully run through a queue are +sent on to the defined 'next' queue, while images which fail are +instead sent to the defined 'fail' queue. In most cases, the 'fail' +queue is a process which marks the image as bad and write the name in +a file, and then puts the image in the 'done' queue. However, it is +possible for the 'fail' queue to be another attempt, such as in the +case of astrometry. In this example, images which fail the default +astrometry are run through a second queue which tries again with the +-loneos flag set (this flag makes the assumption that the header +information about the LONEOS region number for the image is correct, +not the RA/DEC information). + +The connections on the remote machines are handled as child processes +with communication through the stdin and stdout/stderr pipes. +Messages can be sent back and forth though these pipes, and messages +which are received from a process associated with an image are written +to a log file with the name imagename.log (where imagename is the root +name of the image). The only required messages from the program are +messages saying ``SUCCESS'' or ``ERROR''. The programs therefore all +end with either of these words. The programs are started within the +controller as a command to the shell followed by and echo ``PROCESS +DONE''. If the controller sees the message ``PROCESS DONE'' without +either ``SUCCESS'' or ``FAILURE'' (after waiting an appropriately long +time), it assumes the program crashed. Currently, no timeout is +available for the programs. The flat-field process is run in mana and +is a slightly special case. We avoid re-loading the same flat many +times by starting a mana process once and running the flatten +procedures as mana function calls. The images which are analysed +(with an indicator of success or failure at each stage) are written to +a file named 'YYMMDD.addlist' by the {\tt control1} and 'YYMMDD.outlist' +by {\tt control2}. + +\subsection{Lock files} + +A variety of locking mechanisms are used to avoid overwriting the +database or running more than one version of the controlling +programs. There is a perl program, {\tt locks}, which allows the user to +set or check the state of the different locks. All programs which +write to the database check for the existence of a lock on the +database and refuse to run if it exists. The control1/2 programs and the +stage1/2 programs check for their own lock files and only run if they +don't exist. Finally, the use can set the lock files 'kill' or 'halt' +for either stage1 or stage2. The first tells the process to stop +immediately, taking care of bookkeeping first. The second tells the +process to finish the current set of images but not start a new +version. + +\subsection{Other Perl Scripts} + +there are several other scripts which are either used to do some +simple function or are used by the maintainer to check up or work on +the pipeline. The two programs 'checkoutlist' and 'checkaddlist' take +a list of YYMMDD.outlist or YYMMDD.addlist files and list the number +of images in the file and the number of successes at each analysis +stage. There programs 'cleaning.pl' and 'run.phot' deal with the +cleanup programs and the running of nrphot. the program +'backup.stuff' takes the first 4 characters of the year/month file +names and backs up to the jukebox tape all tar files for the given +month. + +\section{Operational Issues and Subtleties} + +The current LONEOS system runs principally on four Intel machines +under linux. The existing system has been generally quite stable over +the past year of operation, but there are some subtleties. One +significant issue is the RAID disk attached to hebe.lowell.edu. This +device is a set of three 9 GB disks merged into one 27GB raid disk. +The database is stored mostly on this disk and the RAID disk mounted +on metis. It is easy to split the database across multiple file +systems by using links. Within the database directory, there are +directories for each of 24 Declination bands (7.5 degrees tall). The +easiest way of splitting up the data is to move some of these +subdirectories to another device and creating links to them at the +appropriate place. This is currently done by having a directory +/metis/d27/database with the remote directories. + +The current problem with the /hebe/d27 RAID results from the poor way +it was setup. It was set up without much knowledge of the (at the +time) rather rudimentary linux RAID system. As a result, the +implementation is such that the disk will not automatically be +configured and mounted at boot time. There is a script /root/mdstart +which lists the step to setup the RAID correctly. It would be helpful +if the rc.local or rc.sysinit files were fixed to start, fsck and +mount the RAID automatically at boot. The setup on metis is +substantially more mature and is correctly implemented. It comes up +when the machine boots. + +\section{Visualization -- Status} + +Visualization of the data in the photometry database can best be +performed with the program {\tt status}. This is a command-line +driven program with a math and macro language which makes it easy to +perform complex tasks. + +First, a few notes about the user interface. The interface has an +interaction similar to {\tt tcsh}. The arrows allow editing of +previous commands. You can also use emacs-like commands such as +cntl-a to reach the beginning of the line and cntl-e to reach the end. +There is command and file completion: if you type part of a command +(as the first thing on a line) and then type tab, it will fill in as +much as possible, until the word is not unique. Typing tab twice at +that point will list the possible endings. For any but the first word +on a line, the same thing will happen for the files in the current +directory. It is also possible to type just a fraction of a command, +as long as it is unique. An ambiguous command will list the possible +alternatives. For example: +\begin{verbatim} +status: c +ambiguous command: c ( catalog cgrid clear create cursor ) +\end{verbatim} + +The shell is essentially an interpretive programming language. +Variables are set as follows: +\begin{verbatim} +status: $fred = 10 +\end{verbatim} +Any expression within curly brackets \{\} is assumed +to be an arithmetical expression and is evaluated before the line is +executed. For example: +\begin{verbatim} +echo {$fred*dcos(45)} +\end{verbatim} +would give the response 7.07107. There are math functions cos, sin, +and tan, which operate on radian expressions, and also dcos, dsin, +dtan, which operate on degree expressions. There are also the +equivalent inverse functions: eg., asin and dasin return radians and +degrees, respectively. The help section on Math defines all of the +available math functions. + +\subsection{Miscellaneous Commands} +\begin{verbatim} +! -- system call +? -- list commands +?? -- list variables +echo -- type this line +exec -- system call +exit -- exit program +help -- get help on a function +output -- redirect output to file +quit -- exit program +scan -- scan line from keyboard or file to variable +wait -- wait until return is typed +which -- show command +\end{verbatim} +Most of these are self-explanatory. The command ?? prints the system +variables. The {\tt help} command will provide help on a single +command or, without any arguments, will list all available help +files (this includes general help not associated with a specific +command). + +\subsection{Shell Programing} +\begin{verbatim} +break -- escape from function +for -- loops +if -- logical cases +input -- read command lines from a file +macro -- deal with the macros +\end{verbatim} + +There are several options for programming in {\tt status}. First, a +file which contains a series of commands can be executed with {\tt + input (filename)}. It is also possible to define macros which will +behave much like regular commands. A macro is defined by typing {\tt + macro name} or {\tt macro create name} followed by the commands. +Arguments to the macro are assigned to the variables \$1 .. \$N and +the number of arguments is given by \$0. Macros may be defined in +{\tt input} files, and in fact when {\tt status} is started, it loads +the file {\tt \~/.statusrc} which may contain default macros. Simple +loops and if statements can be performed, and are quite useful for +complex macros. + +``If'' statements are similar in syntax to C if statements, but only +the following logical operators are available: $>$, $<$, $=$, !, $|$, +and \&. Notice that (currently) there is no $>=$ or $<=$ symbol. The +operator ! means ``not equal to'', but cannot be used to negate a +logical value. The operators $|$ and \& have the meaning of ``or'' and +``and'' respectively. Math expresions in the if statement must be +contained in curly braces, as elsewhere. Variables with string values +may use the logical $=$ operator to test if two strings are the same. +``For'' loops are quite simplistic. The form is: +\begin{verbatim} +for var first last delta + (commands) +end +\end{verbatim} +The value of {\tt \$var} will start at the value {\tt first} and increment by +{\tt delta} after each loop. The loop will stop after {\tt \$var} is greater +than {\tt stop}. The value {\tt delta} is optional, with 1 assumed. +The value of {\tt \$var} may be changed during the loop, and if set +beyong the value of {\tt last} will end the loop early. + +\subsection{Vector Plotting} + +\begin{verbatim} +box -- draw a box on the plot +clear -- erase plot +create -- create a new vector +cursor -- get coords from cursor +grid -- plot cartesian grid +hist -- create histogram from a vector +labels -- define labels for plot +limits -- define plot limits +plot -- plot a pair of vectors +print -- write vectors to file +ps -- define labels for plot +set -- vector math +style -- set the style for graph plots +vectors -- list vectors +zplot -- plot scaled points +\end{verbatim} + +In addition to scalar variables, {\tt status} can manipulate and +display 1-D vector variables. Many of the commands which extract data +from the photometry database place the data in vectors as well as +plotting them. A vector can also be created based on a number +sequence with the command {\tt create name Nelements start delta}. +The resulting vector has $Nelements$ entries, starting at a value of +$start$ and running until $start + delta*Nelements$. If $delta$ is +0.0, all elements will have the value of $start$. A histogram of a vector +may be made with the command {\tt hist}, which creates a new vector +containing the histogram of the first vector. The data range and bin +size of the histogram are defined in same way as with create. This +makes it easy to create the index vector that goes with a histogram +vector: +\begin{verbatim} +hist y Ny 1 100 0.1 +create dx 1 100 0.1 +\end{verbatim} +The above will create a histogram of y in Ny and the index in dx. +Plotting this with {\tt plot dx Ny} will show the histogram. + +Vector math is performed with a command of the form {\tt set new = + (expression)}. The expression is some math function employing +vectors and scalars. A complete listing of the math operators +available in {\tt set} can be found in the help for {\tt set}. + +Once vectors are defined, they may be plotted. A pair of vectors can +be plotted against each other if they have the same number of entries. +The plotting is performed on the graphics window, Kapa. There are +actually several graphics windows available to {\tt status}, any of +which may be used to plot at any time. Some of the more complex +operations default to either graphics window 0 or 1, depending on the +context. Except for those functions with a pre-defined window, all +plotting functions apply to the current graphics window unless an +option {\tt -n N} is given to specify a different window. The +plotting style is determined by the command {\tt style} which can set +the line width, the line type (solid, dashed, dotted, etc), the point +type (box, cross, etc), the point size, the color, and whether a pair +of vectors is plotted as a sequence of points, a set of connected +lines, or a histogram. Some functions which make plots use their own +styles, as discussed below. The function {\tt limits} lets the user +set the range of the plot axes, or check the current setting. The +command {\tt plot} will plot a pair of vectors on the current graphics +window using the current plotting style for that window. The command +{\tt zplot} will plot a pair of vectors with the point size scaled by +a third vector, with maximum and minimum point sizes representing +specified values. The {\tt cursor} command goes the other way: this +command puts the Kapa window in cursor mode and waits for input from +Kapa. The user can then type any alphanumeric key on the graphics +windows and will be told both the pointer location (in the graphics +coordinates) and will have the coordinates stored in {\tt status} +variables. For example, by typing ``1'' in the sky display window, +the RA and DEC of the pointer are stored in the variables {\tt \$R1} +and {\tt \$D1}. This command can be used to let the user define +locations or regions of interest on the Kapa window. (Future addition: +{\tt button}, which does the same with the mouse buttons). + +\subsection{Database Functions} + +\begin{verbatim} +gcat -- get catalog at location +gimages -- get images at location +gstar -- get star statistics +extract -- extract average vectors from catalogs +mextract -- extract measurement vectors from catalogs +imstats -- plot image statistics +imextract -- extract image vectors from database +lcat -- list catalogs in display region +cmatch -- match two catalogs +\end{verbatim} + +There are a variety of other commands which directly refer to the +photometry database. Some of these functions extract data of various +types from the database, others perform more complex plotting +operations. The commands listed above are those which simply extract +data from the database. The first three list information relevant to +a specific RA, DEC location on the sky: {\tt gcat (RA) (DEC)} lists +the catalog at the specified location and places the name in the +variable {\tt \$CATNAME}, {\tt gimages (RA) (DEC)} lists all images +which overlap the specified location, {\tt gstars (RA) (DEC) (RADIUS)} +lists data about the stars within a specified radius of the specified +location (all numbers above are given in decimal degrees). Similarly, +{\tt lcat} lists the catalogs in the region. Imstats lists statistics +about each image + +The next three commands extract a specific piece of information from +the photometry database and places it in a vector. First, {\tt + extract} will extract average values for each star and place it in a +vector. Next, {\tt mextract} will extract measurement values for each +star and place it in a vector: as a result a single star may have +multiple entries in the measurement vectors. Finally, {\tt imextract} +will extract image statistics into vectors (not yet implemented). + + +\begin{verbatim} +catalog -- plot catalog stars +cgrid -- plot sky coordinate grid +cplot -- plot vectors in sky coordinates +czplot -- plot scaled vectors in sky coordinates +images -- plot image boxes +imdense -- image density plot +lcurve -- plot lightcurve for a star +pcat -- plot catalog boundaries +region -- define sky region for plot +resid -- plot residuals +simage -- plot stars in an image +\end{verbatim} + +There are two types of database plotting functions: those that display +or refer to the spatial charateristics of the data and those that +refer to other types of charatersitics, such as the time domain. The +graphics window 0 is reserved for all plots of objects on the sky. +The command {\tt region} defines the current sky coordinates for plots +in graphic window 0. The command {\tt pcat} plots the outline of all +photometry database files which are within the currently defined +region (and by default, only those with data). {\tt images} plots the +outline of the images in the image database, while {\tt imdense} shows +the number of images at a location by randomly spacing dots within the +boundary of the images. The command {\tt cgrid} +draws a grid in celestial coordinates on the for the current region. + +The most complex, but also one of the most useful command is {\tt + catalog}, which plots the positions of stars in the photometry +database (and others) on the sky. There are many options to this +command. One set allows the user to plot stars from the photometry +database (the default), from the HST GSC, or from an ASCII text file +with RA, DEC, and Mag in specified columns. If the ASCII file has a +fixed number of bytes per line, the data can be more quickly loaded. +The size of the points may be scaled by the star magnitude, by the +number of observations of the star, or by the number of missing +datapoints for the star. In addition, points may be plotted only if +they land in specified magnitude ranges, or with specified numbers of +measurements, or missed measurements. Also, objects may be plotted +only if they have a specified Average.code, so that only asteroids or +only perfect stars may be plotted. The plotted vectors may be saved, +if desired, and the source catalog epoch may be specified as different +from J2000 (only valid for ASCII data). + +Several other commands relate to non-spatial charateristics of images +and stars. {\tt lcurve} will plot a light curve for all stars within +some radius of a point. {\tt resid} plots the photometry residuals +for a particular region file. + +\section{Some Examples} + +\begin{figure} +\psfig{file=allsky.ps,width=16cm} +\caption{\label{allsky} \small + Map of the entire sky, and images added to database. } +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location +of the images currently in the database. This picture was made with +the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 gls +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=catalog.ps,width=9cm} +\caption{\label{catalog} \small + Comparison between HST GSC and photometry database astrometry. } +\end{figure} + +Fig.~\ref{catalog} shows an example comparison of the photometry +database star positions and the HST Guide Star Catalog star +positions. The crosses are from the photometry database while the +boxes are from the HST GSC. The size of both points is a function of +brightness. This plot was made with the following commands (starting +from the previous image): +\begin{verbatim} +status: cursor (typed 1 on region of interest) +1 137.097858 22.698305 +q 137.097858 22.698305 +status: region $R1 $D1 0.5 TAN +status: cgrid +status: box +status: style -pt 0 +status: gcat $R1 $D1 + 0 n2230/1951.cpt * +status: catalog n2230/1951.cpt -m 12 18 +status: style -pt 2 +status: catalog n2230/1951.cpt -m 12 18 -g +\end{verbatim} + +\begin{figure} +\psfig{file=lightcurve.ps,width=9cm} +\caption{\label{lightcurve} \small + Lightcurve for several image stars. } +\end{figure} + +Fig.~\ref{lightcurve} shows an example lightcurve from three stars in +the above field. The x axis shows the time in seconds, which the y +axis shows the star magnitudes. This plot was made with the following +commands (starting from the previous image): {\small +\begin{verbatim} +status: cursor +1 137.062562 22.791860 +q 137.062562 22.791860 +status: lcurve $R1 $D1 0.01 -l +status: box +status: gstar $R1 $D1 0.01 +read 11318 stars from catalog file /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt (22462 total measurements) +508 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.068619 22.785400 18.438000 1 (2) 1.000000 1.000000 +498 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.062820 22.792747 12.882000 3 (0) 1.000000 1.000000 +501 /host/pikake/d1/gene/loneos/allstars/n2230/1951.cpt 137.064499 22.800817 15.480000 3 (0) 1.000000 1.000000 +\end{verbatim}} +The last lines show the database information on the three stars. + +\section{Software Distribution} + +All of the code, binaries, and scripts for the Pipeline are stored in +the ohana tree at /metis/d11/src/ohana. The ohana directory contains +subdirs of bin, lib, include, src, config, and doc. There is a +Makefile at the top level which calls lower-level Makefiles to build +the programs desired. One of the important concepts in the ohana +distribution is the design of the directory layout to enable binary +support for multiple architectures. The Makefiles use the environment +variable ARCH to determine the appropriate architecture. The binary +and library files are stored in subdirectories of the form bin/\$ARCH +and lib/\$ARCH. It is therefore necessary that users of the ohana +system architecture define the ARCH variable correctly. It is also +necessary to include in the user's PATH the directory +/metis/d11/ohana/bin/\$ARCH. At LONEOS, +with the solaris sparc station and the linux machines both in use, +ARCH can take on the values of 'linux' and 'sol'. The values 'sun4', +'irix', and 'hp' have also been used at various sites. A simple bit +of csh script in the users .cshrc or equivalent can make this +assignment transparently: + +\begin{verbatim} +set sys=`uname -s` +switch ($sys) + case IRIX64: + setenv ARCH irix; + breaksw; + case SunOS: + set ver=`uname -r | awk '{print substr($1,1,1)}'`; + if ($ver == 5) then + setenv ARCH sol + else + setenv ARCH sun4 + endif + breaksw; + case Linux: + setenv ARCH linux; + breaksw; + default: + echo "unknown architecture"; + setenv ARCH unknown; + breaksw; +endsw +\end{verbatim} + +The Makefiles use the ARCH variable not only for destination +directories, but also for destination binary names. All files of the +form fred.c are compiled to fred.ARCH.o (instead of fred.o), +uninstalled libraries get the names libfred.ARCH.a (instead of +libfred.a), while uninstalled programs are called fred.ARCH. + +The perl scripts are kept in ohana/src/perl, with links to the +appropriate bin directories. All programs are stored in directories +of the form ohana/src/program. The exceptions are gastro (stored in +ohana/src/astro), and control1/2 (stored in ohana/src/astro). Also, a +variety of simple, user-level functions are available in +ohana/src/misc. + +Most of the ohana package of programs use several common libraries. +These include the readline library (except for dophot, this is the only +part of ohana that is not written by Eugene Magnier), which is used +for command-line interface systems. Other important libraries are the +fits library for implementing the FITS specifications in a convenient +C-friendly environment, and the ohana library, consisting of basic +string and other basic operations. + +For more details on the programming aspects of parts of the pipeline, +see the associated document, 'A Programmer's Guide to the Loneos +Pipeline' + +\section {Comparison with USNO catalog} + +I have made several comparisons between the USNO catalog and results +from running images from Loneos through the photometry pipeline. In +general, there is a good agreement between the astrometry of the two +datasets. There are specfic areas where the two disagree, however. +First, as discussed above, the USNO star positions may be +significantly different from the observed star positions simply +because of proper motion. The first picure in the ``Comparisons +between Loneos + USNO'' pages shows several stars with significant +proper motion. These are currently being flagged in the photometry +database as discussed. Other differences evident in this first +picture include the classification of bright stars and the splitting +of faint and bright stars. In the upper right of the image, two +bright stars are visible, one of which is actually a double. The USNO +catalog does a good job of detecting both, but it also detects and +includes as stars several diffraction spike points. The HST GSC only +detected a single point and did not split the star at all. The Loneos +data in our pipeline did not detect the star at all due to saturation. +A star in the upper left corner shows the ability of {\tt dophot} in +the Loneos pipeline to split double stars. This star is, on close +inspection, clearly a double. The USNO catalog only detects one. The +Pipeline not only splits the two stars, but it also makes the correct +cross-identification of the stars. + +The second set of pictures shows the ability of the Loneos pipeline to +deal with small galaxies. There are two small galaxies in this field, +both of which are broken into 4 or 5 objects by the USNO catalog, but +maintained as single objects in the Loneos pipeline. In another +example (not shown) a bright star within the disk of a small galaxies +was clearly split from the galaxy by the pipeline (and also by USNO). + +\appendix +\section{Parameter file params.txt} +\begin{verbatim} +# Configuration file for Loneos analysis pipeline + +# important files and directories +IN_DIR /pallas +OUT_DIR /irene/d11/workspace +# +CATDIR /hebe/d27/database +IMAGE_CATALOG /hebe/d27/database/Images.dat +IMAGE_CATALOG_TEMPLATE /hebe/d27/database/template.cat +CATALOG_TEMPLATE /hebe/d27/database/template.cat +ROCK_CATALOG /hebe/d27/database/Rocks.dat +# +GSCFILE /metis/d11/references/GSC/GSCregions.tbl +GSC_DIR /metis/d11/references/GSC +USNO_CDROM /metis/d11/references/USNO +DOPHOT_PARAMS /metis/d11/references/config/default_parameters +FLATTEN_SCRIPT /metis/d11/references/config/flatten.pro +LONEOS_REGIONS /metis/d11/references/config/regions.map + +# parameters for "fstat" +# instrumental mag zero point +ZERO_PT 24.5 +DOPHOT_CHAR_LINE 129 +DOPHOT_TYPE_FIELD 5 +DOPHOT_AP_FIELD 91 +DOPHOT_PSF_FIELD 52 +MIN_SN_FSTAT 6.0 + +# parameters for "gastro" +DEFAULT_RADIUS 32.0 +MINIMUM_RADIUS 1.5 +MIN_ERROR 0.6 +MIN_PRECISE 0.12 +CCD_PC1_1 1 +CCD_PC2_2 -1 # for loneos +CCD_PC1_2 0 +CCD_PC2_1 0 +ASEC_PIX 2.82 +NFIELD 3.0 +MMIN 6.0 +ROT_ZERO 0 +dROT 1.0 +NROT 2 +POLAR_AXIS_RA -90.0 +POLAR_AXIS_DEC 89.88 +RA_OFFSET -90.0 +DEC_OFFSET 0.0 + +# parameters for "addstar" +# airmass extinction coefficient (in mag / airmass) +NSIGMA 3.0 +ALPHA 0.03 +XOVERSCAN 60 +YOVERSCAN 50 + +# parameters for "controller" +NEW_IMAGES images.dat +USERNAME gene +PASSWORD gene + +# here we list all available machines +NMACHINES 6 +MACHINE0 metis +MACHINE1 hebe +MACHINE2 iris +MACHINE3 irene +#MACHINE4 juno +MACHINE4 ceres +MACHINE5 vesta + +# here we define which machines which are always free +fMACHINE0 1 +fMACHINE1 1 +fMACHINE2 1 +fMACHINE3 1 +fMACHINE4 0 +fMACHINE5 0 +fMACHINE6 0 + +LOCAL_MACHINE 1 # machine with catalog on local disk + +# parameters for "status" +TIME_REFERENCE 90/01/01 00:00:00 + +# parameters for "nrphot" +# make this one small to boost change of getting an early region +TAU 3.0 +SCATTER_LIM 15.0 +MAG_LIM 17.0 +IMAGE_SCATTER 0.075 +NIMAGE_SCATTER 1.5 +STAR_SCATTER 0.05 + +### parameters for "markstar" +SEARCH_RADIUS 360 # region for initial trail hunt, in arcsec +TRAIL_WIDTH 5.0 # expected trail width, in arcsec +NANGLE_BINS 180 # Number of angular bins in 180 degrees +NPTSINLINE 5 # minimum points needed for trail +MIN_DENSITY 0.025 # minimum linear density in star/arcsec +SPACE_SIGMA 10.0 # how much denser than average for trail? +# parameters defining bright star exclusion regions +BRIGHT_XTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_XTRAIL_MAG 9.5 # faintest sat. star +BRIGHT_XTRAIL_SLOPE -80.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_YTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_YTRAIL_MAG 11.0 # faintest sat. star +BRIGHT_YTRAIL_SLOPE -400.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_HALO_MAG 8.5 # faintest sat. star +BRIGHT_HALO_SLOPE -28.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +# parameters which define the ghosts +GHOST_MAG 7.5 +GHOST_RADIUS 200 # in arcsec +OPTICAL_AXIS1 2154.0 +OPTICAL_AXIS2 2193.0 + +# parameters for "addusno" +USNO_RADIUS 5.0 # in arcsec +USNO_PROPER 20.0 # in arcsec +USNO_RED 1000 # code for USNO Red data +USNO_BLUE 1001 # code for USNO Blue data + +# parameters for "markrock" +ROCK_RADIUS 2.0 # in arcsec +ROCK_MAX_RADIUS 20.0 # in arcsec +ROCK_MAX_SPEED 0.1 # in arcsec / sec +\end{verbatim} + +\section{Creating New Flats} + +The LONEOS implementation of the analysis pipeline has a limited +ability to choose a flatfield image. There is a mechanism in the +stage1 script to select a flatfield on the basis of the photcode +entry. It works like this: there is a fixed directory for all +flatfield files, currently /metis/d11/reference/. This directory is +written in the file flatten.pro, and also in the script stage1. Each +flatfield has a name like flatcodeN.fits, where N is the relevant +photcode. The mana macro 'flatten' in flatten.pro loads the file +\$flatdir/flatfield.fits (OR SOMETHING LIKE THAT), which is a link to +the correct flat. The script stage1 has the job of deciding which is +the correct photcode and setting the link appropriately. + +This then begs the question of how this flatfield file should be +created and assigned a new photcode. The flatfield is NOT made +nightly from the entire set of data, as one might expect. This is not +done because of the problems of scattered light in the telescope. The +widefield of the LONEOS telescope makes it somewhat succeptible to +light at relatively large incident angles. As a result, a median +image made from images of the night sky will have different amounts of +scattered light depending on the overall sky brightness. In our +experience, the presence of light clouds and just a small moon (as +little as a quarter) make the resulting median image a very poor +representation of the chip response. Our solution to this problem is +to choose a good night, in which the sky is very dark, and make a +flatfield from that collection of images. As a result, there is too +much decision-making involved to allow the pipeline to make this +decision. Instead, we have chosen to only infrequently make a new +flat from the data, choosing those nights that we are confident will +not introduce excess scattered light. + +There are a set of scripts and programs to make the flatfield image. +The process involves first removing the bias and dark from the image, +by subtracting a dark image, then medianing the resulting images. +The mana script FRED performs the dark-subtraction and then +medianfilter performs the medianing. In fact, instead of medianing, +medianfilter returns the average of the fraction of pixels with values +between f1 and f2 percent of the total range of the pixel data. + +\section{Daily Monitoring of the Pipeline} + +On a daily basis, the pipeline generally runs by itself without any +intervention. There are a few checks that can be made to be sure the +process is running OK. + +First, it is good to see if the disk space is sufficient. There is a +component to stage1 which will keep it from running the analysis if +there is less space on the destination disk (/irene/d11/workspace) +than 1.1 times the amount of space in the source directory +(/pallas/d?/YYMMDD). + +Second, it is a good idea to see if the nightly cron job is checking +for new data and if the data is getting analysed. If data was created +lastnight, there should be a set of files in the log directory +(/metis/d11/logs) of the form YYMMDD.inlist, YYMMDD.addlist, and +YYMMDD.outlist. If these files do not exist, it may just mean that no +data was created last night. In that case, the tail of the file +/metis/d11/logs/lastnight should say something like 'there is no new +data to analyse'. + +If data was created, and these files exist, it is a good idea then to +check that the analysis of the images was reasonable and didn't fail +on most of the images. There are two Perl scripts to check on this. +First, 'checkaddlist [filenames]' will take a list of *.addlist files +and, for each file, list the total number of images, then the number +that failed for each of the four stage1 processes, flatten, dophot, +fstat, and gastro. A typical run might look like this: + +\begin{verbatim} +hebe: checkaddlist 9907??.addlist +990701: 206 0 0 0 0 +990703: 199 5 0 0 0 +990705: 150 10 1 0 1 +\end{verbatim} + +It is actually unusual for any except the flatten process to fail, but +it might be typical for 1 or 2 to fail, since they will represent +aborted images or images which failed to write everything to disk. +The second Perl script is 'checkoutlist' which works just the same +with the YYMMDD.outlist files, but only give statistics on the +'addstar' process. Here, it is more common for several images to fail +'addstar'. At the current time, 'gastro' does not return a fail +status if it finds a bogus solution. Images which fail 'gastro' are +mostly those which were taken when the sky was too bright and all that +is seen is the sky itself, or the focus frames, or the very short +images taken of bright stars to check the coords. If the images fail +'gastro', they are rejected by 'addstar'. + +\section{Restarting the Pipeline} + +It is difficult to predict the future, so it is hard to give guidance +on solving future problems. Some of the things which might happen +would include having one of the machines which serve a disk go down or +having the disks fill up. If this happens, it is possible that the +run of a given night will have failed completely, or mostly. There +are a variety of things to do to recover from these situations. + +\subsection{Deleting data from the archive} + +The program 'delstar' lets the user delete a set of data from the +database. There are several options for this. The first example is: +{\tt delstar 199902040015b.cmp}. In this form, delstar uses the +astrometry information in the image header to find the measurements +and delete them from the database, along with the image from the image +database. This form requires the datafile *.cmp to exist in the local +directory. The second form, {\tt delstar 911012323 1}, uses the given +time in standard FORM (NEED TO DISCUSS TIME FORMATS AND GIVE THEM +NAMES) to identify the image in the image database and delete both the +image and the measurements. It is easy to associate time with image +with the status function 'findimages' or 'gimage' which list images +covering a specific location. + +The third form of delstar is used to clean up the database if an image +is deleted but not all measurements from that image are removed. The +routine 'addstar' uses the astrometric information for an image and +the X,Y coordinates of each object in the image to locate the object +in the database. Several other routines which do the reverse process of +finding location of objects within the images. The function which +performs this association needs to find all database *.cpt files which +are covered by the image. The existing implemenation is relatively +quick, but not the most general possible. If an image has extreme +astrometric problems, yet somehow makes it through the pipeline, it +will introduce stars which are not easily found by the inversion +process. This may be because the image has astrometric parameters +which make it extremely non-rectilinear. If such an image is deleted, +which will probably be desired, some of the stars from that image may +get left behind. These orphaned stars can be deleted with the command +{\tt delstar -orphan STUFF}. + +\subsection{Deleting a night from the pipeline} + +If the analysis for a specific night fails miserably, it is probably +necessary to delete the entire night and then encourage the system to +redo that night. It is easy to delete an entire night with delstar by +giving a time interval which includes the night in question. It is +also easy to find the value of the time in seconds with the status +function 'ctimes'. (At some point, delstar should incorporate the +code from status which does the time format conversion automatically). +This process will remove the night in question from the database. But +before the pipeline can be re-run, the pipeline needs to be forget +that it ran that night before. Every night which has been analysed +gets included in the log files 'processed.log' and 'addstared.log'. +The appropriate entry from these files needs to be deleted. It is +also necessary to delete the files 'YYMMDD.addlist' and +'YYMMDD.outlist' from the log directory. If the 'extracted.log' file +exists, it should be deleted as well. All of this is predicated on +the assumption that the user has halted all processing by the pipeline +before trying to fix anything. At this point, it is possible to +restart the database. This can be done in several possible ways. If +the night in question has already been put on tape, it has to be +extracted. The program 'slurp' will automatically find data which has +not been extracted from the tape, download it, and set up the +appropriate files. Unfortunately, 'slurp' uses the file 'tape.log' +which is not automatically generated. It is therefore necessary to +add a line to 'tape.log' (in /metis/d11/logs) for the night in +question. Instead of actually running 'slurp', however, the user +should just run 'stage1 -once' which will perform one run of stage1 +and then end. If there is no pending data (nothing in 'extracted.log' +that has not been analysed), then stage1 will automatically call +'slurp' in the correct fashion (ie, on pallas), run through the +analysis, and then exit without restarting. + +If the night's data has not been deleted yet, then it is not necessary +to run 'slurp' as the script 'lastnight' will do the job. The +cronscript for 'lastnight' uses the -today option so that the +automatic analysis will not reattempt any nights still on pallas. +Instead, it will be necessary for the user to run 'lastnight' with the +-any flag, which will also take care of running 'stage1'. This should +be safe at anytime, as the lock functions keep multiple instances of +the programs from running, but it is better to do this with enough +time to allow the process to finish before the next night's data is +ready for processing. + +If the pipeline has been interrupted, it will probably be necessary to +restart 'stage2' as well. Since 'stage2' always restarts itself on +exit, there is no cron job for it to make the initial start. (This +probably should be introduced into the /etc/rc.local file, or some +cron implementation for stage2 worked out). Note that both 'stage1' +and 'stage2' take no arguments except the -once flag. They make their +own decisions about the correct log file number and so forth. + +If the 'lastnight' cron job gets lost, there is a copy of the +necessary cron script /metis/d11/src/ohana/src/perl/lastnight.cron. +This can be submitted with 'crontab lastnight.cron'. + +\section{Data backup} + +Backups are the still rather primitive in implementation for the +LONEOS system. There are two things which should be backed up, but +only one currently has a good scheme. The database itself does not +have a backup process at the moment. This is a bad situation, but not +an easy one to solve. The database is very large, over 30 GB, which +is larger than the space on a single AIC tape. At the moment, we are +satisfied with simply saving the intermediate results directories, all +of the YYMMDD.tgz files. If the entire database is destroyed, these +files can be used to rebuild the database in a relatively short time +(currently, about a week). The script 'backup.stuff' will backup a +group of files. A good method is to backup a month at a time. These +will become a single tar file on the AIC tape, and a month's worth of +data is a reasonable amount of disk space for future downloads. + +\end{document} Index: /trunk/Ohana/doc/pipeline/pipeline6.tex =================================================================== --- /trunk/Ohana/doc/pipeline/pipeline6.tex (revision 2508) +++ /trunk/Ohana/doc/pipeline/pipeline6.tex (revision 2508) @@ -0,0 +1,2047 @@ +%\documentstyle[astro,psfig]{article} +\documentclass{article} +\usepackage{psfig} +\usepackage{astro} +\usepackage{times} +\GoodMargins +\newcommand{\note}[1]{({\bf #1})} +\newcommand\pr{\em} % font for program names +\newcommand\ff{\bf} % font for file names +\newcommand\cc{\tt} % font for program code +\newcommand\ty{\sc} % font for file types + +\title{Variability Pipeline} +\author{Eugene Magnier} + +\begin{document} +\maketitle + +\section{Overview} + +Ptolemy is a collection of programs which act together to provide +automated reduction of images, including photometry and astrometry, +data validation, and incorporation of the resulting measurements in a +photometry database organized by star position on the sky. Ptolemy +was designed to deal with images produced by LONEOS, the Lowell +Observatory Near Earth Object Search, to study the data stream with +astronomical goals in mind beyond just the Near-Earth Objects. The +document deals with both the generalities of Ptolemy and also the +specifics of the LONEOS implementation. Why Ptolemy? Two reasons: +First, Ptolemy, the Roman Astronomery (AD 100-170) was the the first +to make a systematic map of the entire sky visible from Greece, much +like LONEOS is doing. Second, Ptolemy I (367-283 BC), as the founder +of the Library of Alexandria, represents the ideals of collection and +organization of large quantities of knowledge). + +Fig.~\ref{pipeline} shows a schematic of the {\pr Ptolemy} program +organzation. Images to be analysed may be provided to Ptolemy either +from the telescope or from a tape archive. Images are first analyzed +by {\pr dophot}, which produces instrumental photometry and pixel +coordinate positions for stars and other objects in the images, +writing files with a particular format, called {\ty obj} files in this +discussion. The results from {\pr dophot} are cleaned up by a program +called {\pr fstat} which merges the header information from the image +with the most relevant parts of the {\ty obj} file, creating a new +file with the {\ty cmp} format. Astrometry is performed on the {\ty +cmp} file with the program {\pr gastro}. The program {\pr gastro} +makes a comparison with an astrometric reference database and only +changes {\ty cmp} header keywords as a result. Finally, the processed +image {\ty cmp} file is added to the photometry database with the +program {\pr addstar}. The photometry database is maintained in a +large number of files ({\ty cpt} files), each representing a small +region of sky. There is no relationship between the image locations +and the boundaries of {\ty cpt} file. All of the preceeding processes +occur on individual images, and would typically be performed while the +observations are occuring, or during the day following, but may be +performed at any time to add new or archived data to the database. + +\begin{figure} +\psfig{file=schematic.eps,width=9cm} +\caption{\label{pipeline} \small + Schematic of the photometry pipeline. Rectangles represent + programs, while ovals represent data products. Arrows show the + direction of travel of information. The flatten routine is a filter + whose position may change as the pipeline matures. } +\end{figure} + +Once large batches of images have been processed, a series of programs +should be run to clean up the photometry database. First, {\pr +markstar} identifies bad data (the trails of satelites, the bleeding +columns from stars, ghost images), and flags these data points +appropriately. Second, {\pr addusno} matches the catalog stars with +stars from the USNO database and adds the USNO magnitudes to the +photometry database. Next, {\pr markrock} searches for likely +asteroids in the {\ty cpt} file, marks them, and writes the asteroid +observations to an asteroid database. It also notes measurements +which are likely to be noise and cosmic rays. Finally, {\pr nrphot} +determines relative photometric calibrations for observations in a set +of {\ty cpt} files. + +\section{Image Analysis Pipeline} +\label{parallel} + +\subsection{Configuration and Generalization} +With the exception of {\pr dophot}, all of the Ptolemy programs refer +to a single configuration file to determine the large number of +parameters necessary in the process. The default parameter file is +hardwired into the programs, but an alternate configuration file may +be specified with the environment variable {\ff PIPE\_CONFIG}. This +option makes independent processing of multiple data sources trivial. +For example, if the pipeline is to be used for both Loneos and SDSS +image analysis and display, there could be two configuration files +(eg, {\ff loneos.txt} and {\ff sdss.txt}), one for each data source. +Several important parameters would be different between these two, for +example, the initial plate scale guess for the astrometry routine, or +the parameters which define the rough chip orientation. This scheme +allows for analysis of different types of images, with a common +destination photometry database, or for independent photometry +databases. + +Data from different sources can be processed by the pipeline and +stored in the same photometry database, since all photometry is tagged +with a source ID in the database (as are the images in the image +database). This allows us to include, for example, the USNO catalog +magnitude measurements (or a subset) in the photometry database for a +quick comparion. Different sources may be considered different +filters, so color-color and color-magnitude diagrams could be +extracted by specifying choices for source catalog. Currently each +object has a single primary photometric system, and individual +observations may be combined to determine that average (eg, if they +are different photometry sources with roughly the filter bandpass) or +they may be ignored when calculating relative magnitudes (eg, the USNO +magnitudes would be ignored). This may be expanded upon by defining +multiple average magnitudes and allowing different measurements to +contribute only to the appropriate average. The determination of the +photometry code for a particular image is currently not well defined. +There are several points in the analysis where the user (or the +controlling program) may specify the photometry source. A table +should be kept in the photometry catalog to define the various +photometry codes. + +file names: Notice that we use a consistent naming convention through +the pipeline reductions. The image is called {\tt foo.fits}, while +the resulting dophot-produced photometry list is called {\tt foo.obj}. + +\subsection{Time representations in Ptolemy} + +Every measurement in the Ptolemy database has a time stamp associated +with it, as do all of the images. Throughout the software, the time +is represented consistently using the standard UNIX representation of +time, which consists of an unsigned integer representing the number of +seconds since 1970 Jan 01, midnight. The time used roughly represents +UT time, though there is some subtlety here. In reality, the time +reported at LONEOS is reported by a GPS clock, and therefore uses the +GPS zero point. There are small, well defined differences between UT, +GPS, and WHAT IS THE THIRD ONE? GPS - UT is a constant NN seconds, +while UT - OTHER is a function of time, as leap seconds are +accumulated in one and not the other. While the programs all refer to +this seconds representation of GPS time, the user of the programs +usually has a few more choices. In {\pr status}, all(?) functions +which require a time argument expect to see one of three formats. If +there is a number of the form NNNNNNNNNs, this is assumed to be the +UNIX seconds form. If there is a number of the form NNNNNNj, this is +assumed to be a Julian date. If there is a number of the form +YY:MM:DD,hh:mm:ss, this is assumed to be a UT date string. Note that +the difference is in the presence of either 's', 'j', or something +else at the end of the string. In the case of a date, it is +acceptable for the separators to be any non-numerical character, and +there may even be a trailing separator, as long as it is neither 's' +nor 'j'. Command arguments which expect a time range may provide one +of the following units: 's' (seconds), 'm' (minutes), 'h' (hours), 'd' +(days). Also, it is only necessary to provide as many significant +places in the date as desired. So, if you are interested in the +images taken on UT 1999 March 03, you could run {\pr images -im +99:03:03 1d}. + +The validity range of the UNIX seconds representation of time is 143 +years, so this measurement representation will become ambiguous in the +year 2113. Note that the Y2K problem is dealt in the date string by +assuming any date with year smaller that 70 means 21st century, and +others are 19th century. Of course, the user may explicitly type the +full 4 digit year to ensure an unambiguous answer. Finally, the user +may refer to the current time as NOW and the current date as TODAY. + +\subsection{Flat Fielding -- Mana / Flatten} + +Flat fielding is performed on each image with {\pr mana} (see {\pr +mana} user's guide) using a script which interacts with the Ptolemy +controlling software. The script is contained within a file, {\ff +flatten.pro}, in the references directory. The script is invoked by +calling it on the command line when {\pr mana} is started. If {\pr +mana} is started with a file name on the command line, that file is +loaded with in 'input'. The file defines a macro '{\pr flatten}', and +also loads the flat-field file. The flattening process is actually +done by calling within {\pr mana} the macro {\pr flatten}, with the +input and output file names as arguments. By loading the flat-field +at the start of {\pr mana}, instead of on each execution of {\pr +flatten}, we save the load time of each flat-field image (which is 2x +the size of the basic image, being 4byte floats per pixel). \note{The +location of the flat field is currently hard-wired in the {\pr +flatten} script - this should probably be changed in the future.} The +macro loads a flatfield with a specific name and location, which is +actually a link; The higher-level controlling process ({\pr stage1}) +creates a link to the correct flatfield. This allows the analysis +process to use different flatfields for different situations. At +LONEOS, we currently use a flat field generated from a well-understood +night. This is probably not the best solution as the flatfield +changes with time. As discussed below, the photometry solution +program, nrphot, fits a high-order polynomial to the relative +photometry, allowing for differences from the correct flatfield. + +\subsection{Photometry -- Dophot} + +Photometry is performed using a varient of {\pr dophot}, based on {\pr +Dophot 2.0} and adapted to streamline processing of many files. With +this version of {\pr dophot}, any FITS format data is safely read in. +Instead of using staticly defined memory blocks for the images, the +necessary memory is allocated dynamically. The image is also +converted to REAL*32 from the given BITPIX for processing. Finally, +only a single, complete parameter file is loaded by dophot. There is +no option for a modified and a default parameter file. The parameter +file to be used is given, along with the input image and the output +object list on the command line when the program is run: {\cc dophot +image.fits image.obj parameter\_file}. To achive these changes, the +code from {\pr dophot} was wrapped within a C program which interprets +the command line arguments, allocates the necessary memory, and reads +in the image. The rest of the processing is performed by {\pr dophot} in +the standard manner. + +\subsection{Cleanup -- fstat} + +After an image is processed by {\pr dophot}, the object file ({\ff +foo.obj}) is converted to a more-compact, more-complete file called a +{\ty cmp} file, with a name of the form ({\ff foo.cmp}). This +conversion is done with the program {\pr fstat}. The {\ty cmp} file +consists of the FITS header from the original image ({\ff foo.fits}), +with some additional keywords to be used at later stages in the +analysis, followed by an ASCII list of the interesting data from ({\ff +foo.obj}). In this list, types 6 and 8 are excluded, and only the +following values are kept: +\begin{verbatim} +X Y Mag dMag t log(sky) +1342.0 106.1 14.166 000 4 3.2 +\end{verbatim} +Objects with a signal-to-noise ratio lower than a specified cutoff +({\cc MIN\_SN\_FSTAT}) are also excluded. Some general information +about the image is derived by {\pr fstat} (FWHM, saturation and +completeness limits, number of each dophot type) and stored as +keywords in the header. The photometry source must be specified at +this stage as one of the command line arguments, though the value used +may be overridden in one of the stages belowe. The resulting file can +now stand on its own without reference to the original image. The +keywords used by {\pr fstat} include the minimum signal to noise, a +rough guess at the zero point ({\cc ZERO\_PT}), and four numbers +defining the format of the {\pr dophot} {\ty obj} file. + +\subsection{Astrometry -- gastro} + +Astrometry is performed automatically by the program {\pr gastro}. +{\pr gastro} loads a {\ty cmp} file and determines an initial guess +for the center coordinates (based on the RA and DEC header keywords). +The program also uses the configuration information about the plate +scale and rough orientation of the image to get close to the final +solution. Also, the true sky position of the telescope pole may be +defined to allow astrometry on images taken close to the pole. The +comparison is made with the astrometric catalog, which may be the HST +Guide Star Catalog, or it may be any source if the data is placed in +the correct format. \note{the astrometric reference catalog +is currently stored in an ASCII format with very limited efficiency. +It would be nice to make a standard format that has a bit of meta-data +documentation and probably a binary format to save space.} + +\section{The Photometry Database} + +\subsection{Photometry File Format} +The photometry database consists of a large number of photometry files +representing sections of the sky and a file containing information on +the images incorporated in the database. These files are called +alternatively ``region files'' or {\ty cpt} files (because they have +the extension {\cc .cpt}). The photometry files are sorted by +Declination into different directories, each consisting of a band +7.5\degree\ wide. These directories have names like n0730, +representing the band starting at a Declination of +07:30. The image +database ({\ff Images.dat}) is stored in the upper level directory of +this directory tree. In fact, the locations of the {\ff Images.dat} +file, the photometry database, and such are all flexible and may be +changed by altering the configuration file. + +At LONEOS, the database is stored mostly on a RAID disk on {\ff hebe} +and a RAID disk mounted on {\ff metis}. It is easy to split the +database across multiple file systems by using links. Within the +database directory, there are directories for each of 24 Declination +bands (7.5 degrees tall). The easiest way of splitting up the data is +to move some of these subdirectories to another device and creating +links to them at the appropriate place. This is currently done by +having a directory{\ff /metis/d27/database} with the remote +directories. + +The name and location of each sky region comes from the Hubble Space +Telescope Guide Star Catalog. There is a file, {\ff GSCregions.tbl}, +which can be used to find the region file appropriate for any location +on the sky. A number of routines exist to make such a query. + +Each photometry file in the photometry database contains all +observations and all average values for all stars observed within the +appropriate region on the sky. The identity of stars is essentially +determined by their RA and DEC coordinates. In very crowded regions, +the end user may need to double check that a neighboring star is not +contaminating individual detections (see the discusion on {\pr +addstar} below). + +The photometry files are stored in a binary format, both to reduce the +volume and to speed access. As a result, the interpretation of the +data is machine dependent: little endian machines need to swap the +data intelligently. Fortunately, this operation is taken care of +appropriately, if the programs are compiled with the BYTESWAP option +set as needed, which is automatically set by the Makefile in the case +of a Linux machine. For reference, Suns, SGI, and HP are all big +endian, while PCs and DECs are little endian. The automatic +conversion takes place by using the funcions {\tt Fread} and {\tt Fwrite} +to substitute for the standard C library {\tt fread} and {\tt fwrite} +functions. {\tt Fread} and {\tt Fwrite} are told the datatype being read, +and they know the layout of the bytes for a particular datatype. +Anytime the definitions of the relevant structures (see below) are +changed, {\tt Fread} and {\tt Fwrite} must be updated. Fortunately, there +is only one file ({\tt Fread.c}), and it has a single byteswapping +section, making it easy to adjust the code appropriately. + +The format of a single photometry database file consists four +sections: +\begin{itemize} +\item {\bf header} -- this is a standard FITS header with room for comments + about the file or whatever. Since the rest of the file is not a + standard FITS file, the SIMPLE keyword is set to False. The three + necessary keywords are NSTARS, NMEAS, and NMISS, which define the + sizes of the next three sections. +\item {\bf average} -- this section contains all average measurement + quantities for each star in the file. There are NSTARS entries, and + each entry is the data from a structure of type Average (defined in + loneos.h, and discussed in more detail below). Thus, the total size + in bytes of this section can be found by NSTARS*sizeof(Average). +\item {\bf measure} -- this section contains the individual + measurements for each star in the database. There are NMEAS + entries, and like the average section, each entry is a single + structure, in this case of type Measure (in loneos.h, see below). + All measurements of a single star are consecutive. The number of + measurements for a single star is defined by average.Nm and the + starting entry for a single star is given by average.offset. + Therefore, the first measurement of a specific star average[500] + would be given by measure[average[500].offset], while the second is + measure[average[500].offset + 1], and so forth. +\item {\bf missing} -- this section contains references to all missing + measurements of a star. This means all of those occasions when an + image enclosed the location of a star in the database, but no star + was detected on the image. Similar to the case of the measure + section, each star has a number of missing entries (which may be 0) + given by average.Nn, and the first missing entry is given by + average.missing. Each missing entry value is again a structure of + type Missing. Currently, the only data in the structure is the time + of the observation, which allows for an unambiguous recovery of the + image where the star was missed. +\end{itemize} + +\subsubsection*{Average Structure defined in loneos.h} +\begin{verbatim} +typedef struct { + float R; /* RA in decimal degrees */ + float D; /* DEC in decimal degrees */ + short int M; /* thousandths of mag (-32.000 to 32.000 valid range) */ + unsigned short int Nm, Nn; /* number of measurements, missing */ + short int Xp; /* scatter in 1/100 arcsec (-327.67 to +327.67 valid range) */ + short int Xm; /* 1000*log(chisq) for magnitude measurement */ + unsigned short int code; /* an ID code (ie, star, ghost, satelite, etc) */ + signed int offset; /* offset to first measurement */ + signed int missing; /* offset to first missing obs */ +} Average; /* 28 bytes / Average */ +\end{verbatim} +The structure above defines the average data stored in the photometry +database for each star. Most of the entries are self-explantory, but +some need a bit of clarification. The entry {\tt Xm} is the magnitude +$\chi^2$ value for the star, under the assumption that the star +brightness is constant. This $\chi^2$ incorporates the photometric +errors on each measurement and is stored as 1000 times the logarithm +of the $\chi^2$ because high dynamic range is not needed. The entry +{\tt Xp} is the scatter about the average postion, in 10 +milli-arcseconds. As discussed above, the values of {\tt Nm} and {\tt + Nn} give the number of measurements and missing observations for +this star, while the entries {\tt offset} and {\tt missing} point to +the first {\bf measure} and {\bf missing} entry for this star. The +entry {\tt M} is the current best average magnitude solution for this +star (see the section on Relative Photometry). Finally, the entry +{\tt code} is an ID code for each object, which may define a variety +of things about the object. For example, if the object is known to be +variable, or if the object is associated with a USNO star, or if the +object is a measurement of an asteroid. See below for details of +these definitions. + +\subsubsection*{Measure Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + short int dR, dD; /* 1/100 of arcsec (-327.67 to +327.67 valid range) */ + short int M; /* thousandths of mag (-32.767 to 32.767 valid range) */ + short int Mcal; /* image cal mag, thousandths of mag (-32.767 to 32.767 valid range) */ + unsigned char dM; /* thousandths of mag (0.000 to 0.255 valid range) */ + char dophot; /* dophot type (1-9) */ + unsigned short int source; /* code to identify photometry source */ + unsigned int t; /* time in seconds (0 - 143 years valid range) */ + unsigned int average; /* reference to corresponding average entry */ + /* upper byte of Measure.average stores flags: + limit of 16,777,215 stars (Naverage) in a file (=0xFFFFFF) + flags: average & 0x1000000 */ +} Measure; /* 20 bytes / Measure */ +\end{verbatim} +} +The structure above defines the individual measurement data stored for +each observation of each star. The first two entries, dR and dD, are +the difference between the average coordinates for this star and the +coordinates determined for this observation. Clearly, if this +residual is too large, the individual measurement will not be +successfully matched with the average star position. {\tt M} and {\tt + dM} are the instrumental magnitude and error for this measurement, +after correction for the exposure time and a zero point (defined in +the configuration file as ZERO\_PT). The entry {\tt t} determines the +time of the individual observation. This is important not only for +timing purposes, but also to determine the source image for this +observation. The time and the entry {\tt source} uniquely determine +the image in the image database that generated this measurement. +Furthermore, this correspondence allows for determination of the pixel +coordinates on the chip, by going throught the image astrometry stored +in the Image database (see below). The entry {\tt source} is a code which +identifies the particular CCD/Telescope/Filter setup. Each unique +{\tt source} should be treated as an independent filter set for the +purposes of accurate relative photometric comparisons. This entry +also allows external catalog data, such as the USNO or Sloan +photometry, to be added to the database, as desired. The entry {\tt + average} is a pointer back to the {\bf average} section to allow +determination of the star from the measurements, in addition to the +reverse. The bits of the upper byte of this value are reserved for +flags used to define particular situations for this measurement. +Possible values are: +\begin{itemize} +\item 1 = BLEND\_IMAGE - the star on the image matched more than one + catalog star +\item 2 = BLEND\_CATALOG - the star in the catalog matched more than + one image star +\item 4 = UPPER\_LIMIT - \note{not really used?} +\item 8 = CALIBRATED - relative photometry has been performed at least + once. +\end{itemize} +The entry {\tt dophot} stores the dophot type for this particular +measurement (1-9: four extra bits in this field). +Finally, the value {\tt Mcal} determines the photometric calibration +of this specific measurment. This value is an offset appropriate to +this image (and if needed, this point in time and this chip position) +to bring the observations of the stars on multiple images to a common +system (see the section on relative photometry). + +\subsection{Image Database} + +All images for which photometry has been included in the photometry +database also have entries in an image database. The image database +is (currently) a single file in the upper directory of the photometry +database. The name of the file is determined by the IMAGE\_CATALOG +entry in the configuration file, and is currently set to Images.dat. +Like the photometry database files, the image database consists of a +FITS header followed by binary data. There is only one type of binary +data: each image has an entry, which is the data from a structure of +type Image (loneos.h). The number of images in the file is determined +by the {\tt NIMAGES} keyword. If the number of images becomes too +large, extrapolation of the system to a set of image database files +with a reference list is quite straightforeward, and could be modeled +on the organization of the region files (eg, one image database file +for each Declination directory). + +\subsubsection*{Image Structure defined in loneos.h} +{\small +\begin{verbatim} +typedef struct { + Coords coords; /* 120 bytes */ + unsigned int tzero; /* readout time row 0 in sec (0 - 142 years valid range) */ + unsigned int nstar; /* number of stars on image */ + short int secz; /* thousanths of airmass (valid range -32.000 -- 32.000) */ + short int NX, NY; /* dimensions of image */ + short int apmifit, dapmifit; /* aperture correction and error in thousandths of mag */ + short int source; /* identifier for CCD (each ever used will have a unique letter) */ + short int Mcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int dMcal; /* thousandths of mag (-32.000 -- 32.000 valid range) */ + short int Xm; /* 10*log(image chi-square) */ + char name[32]; /* name of original image */ + unsigned char detection_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char saturation_limit; /* tenths of mag (0.0 - 25.6 valid range) */ + unsigned char cerror; /* astrometric error: 1/50 of arcsec (0 -- 5.12 valid range) */ + unsigned char fwhm_x, fwhm_y; /* PSF terms in 25*arcsec (valid range 0.0 -- 10.2" ") */ + unsigned char trate; /* 10000 * scan rate in sec/pix (0 -- 0.0256 valid range, + typically 0.0146. this is used only to determine the time + of the observation, not to find the coordinates. 1 byte + gives 0.11 sec accuracy */ + float exptime; /* exposure time, seconds */ + char code; /* flag to mark an image as bad or whatever */ + char dummy[21]; /* extra space for the future */ + short int order; /* number of terms used for Mrel */ + short int Mx, My; + short int Mxx, Mxy, Myy; + short int Mxxx, Mxxy, Mxyy, Myyy; + short int Mxxxx, Mxxxy, Mxxyy, Mxyyy, Myyyy; +} Image; /* 240 bytes / Image */ +\end{verbatim}} + +The structure above lists all of the data stored for each image. A +substantial amount of information is stored for each image. First, +the astrometric calibration information is stored in a {\tt Coords} +structure (see coordinate systems, above). The upper and lower limits +of the photometry in the database (in instrumental magnitudes) is +stored in {\tt detection\_limit} and {\tt saturation\_limit}. These +values are useful for comparison for those stars where observations +are missing. The relationship between time and pixel coordinates is +given by {\tt tzero} and {\tt trate}. Drift images will have a scan +rate stored in {\tt trate}, while staring images will have a value of +0.0 for this entry ({\tt tzero} then applies to the whole image). +Other important parameters are kept: the astrometric error ({\tt + cerror}), the airmass ({\tt secz}), the exposure time ({\tt + exptime}), the number of stars detected ({\tt nstar}), the FWHM +values ({\tt fwhm\_x, fwhm\_y}), the image name ({\tt name}), and the +size of the image ({\tt NX, NY}). The entry {\tt source} is used as +above to define the CCD/Telescope/Filter used for this observation. +The entry {\tt code} is used to flag images in various ways. +Currently, only two bits are used, to define if the image has been +photometrically calibrated and if the calibration is acceptable. +There are 21 extra bytes are stored for future additions. + +\subsection{Data Incorporation -- {\pr addstar}} + +Images which are completely processed ({\pr dophot, fstat, gastro}) are +then incorporated into the photometry database with the program {\pr +addstar}. This program decides which region files are appropriate +for this particular image, then one-by-one adds stars from the image +to the appropriate region file. Stars already in the catalog are +matched with stars in the new image purely by a positional comparison. +In order to avoid the difficulty of comparisons in the RA and DEC +coordinate frame, a cartesian projection is performed. The stars from +the image being processed and the database stars in the same area are +projected onto a tangent plane and positional comparisons made in this +(locally cartesian) coordinate frame. This avoids the dangerous +singularities at the pole and also makes the RA 0,360\degree\ boundary +a trivial problem. + +Several choices must be made in the comparison process. If a star in +the catalog is matched with a star in the image, the new measurements +of that star are added to the database. If a star in the database +lies in the field of the image, but is not detected, this information +is added to the list of missing data. The only data stored in this +case is the time and source, which is sufficient to unambiguously +identify the source image from the image database. This allows later +programs to find relevant statistics from the image database, if +necessary. If a star is detected in the image, but is not already in +the database, a new entry is added to the database. In addition, the +same ``missing data'' from all previous observations images which have +covered this location (ie, images already in the image database) are +included. This last step is necessary so that the image processing +order is not important. + +Stars also run the danger of being crowded together. Since all +comparisons are performed on the basis of position alone, crowded +fields may make for ambiguous cross-identifications. A pair of stars +are matched if the difference in their positions is less than a +specified search radius (dependent on the scatter in the astrometric +solution for the image). If more than one catalog stars are correlated +with a star in an image, the new measurement is added to {\em both} +catalog stars, and a flag is set noting that this observation had a +blended IMAGE. Conversely, if a single catalog star is matched with +more than one image star, both new measurements are added to the one +catalog star, and a different flat is set, noting that this +observation had a blended CATALOG. + +\section{Catalog Update and Cleaning} + +Occasionally, once a large number of images have been added to the +photometry database, a set of programs should be run to clean and +update the {\ty cpt} files. This may best be done on {\ty cpt} files after a +night's worth of data is incorporated. These routines define bad +data, identify the USNO catalog stars, and search for asteroids and +other junk. Most of the results from these routines are the addition +of object flags to the Average data structures (Average.code). Most +of these routines process data within a single {\ty cpt} file at a time. + +\subsection{Markstar} + +The first of the update programs, {\pr markstar}, identifies bad data +and flags it. It makes three types of identifications: bright stars, +ghost stars, and trails. In the first case, it searches the HST guide +star catalog for bright stars within the field of the current {\ty cpt} +file. Three types of pixels are flagged for bright stars. Any +measurements within a circular region centered on the star are suspect +(the ``halo'' of the bright star). Second, points along the X axis +are likely to contain flux from the diffraction spikes. Finally, +points along the Y axis are likely to have bleeding in addition to the +diffraction spikes. Each of these three regions has a different size +and range which must be defined in the configuration file. The +regions are represented by a representative magnitude and a scale +parameter. For example, the diameter $d$ of the halo for a star with +magnitude $m$ is defined as +$d = \mbox{BRIGHT\_HALO\_SLOPE} (m - \mbox{BRIGHT\_HALO\_MAG})$. The X and +Y axis points are defined by the length of the diffraction spike, with +an equivalent formula, and the width of the spike. Since these are +each independent, there are 8 parameters to define the exclusion +regions of bright stars. One of the current drawbacks of this system +is the reliance on the HST Guide Star photometry. In the HST Guide +Star Catalog, only one photometry band is given, so no color +information is available. The parameters currently used were defined +based on a large number of bright stars, with no adjustment for the +different effective filter system of the Loneos telescope and the HST +GSC. Thus, occasionally very red stars will be several magnitudes +brighter in the real data than reported in the catalog. As a result, +large numbers of bad data points in these halos and spikes may be +missed. + +The second type of bad data flagged by {\pr markstar} are the trails +from satelite and other space junk. This portion of the program +searches for objects which land along a line and which have a high +linear density. Parameters may be adjusted to define the width and +the necessary density or number of points in the line. Objects in +lines are also only identified if all observations come from a common +image. If an object only has one measurement and it is in a trail, +the entire Average is flagged to be bad. However, if an object has +many measurements and only one is bad, the object is only flagged as +having some bad data points. + +The third type of bad data are ghost star images. The HST GSC is +searched for stars projected across the telescope optical axis, which +is defined by the user. All objects within a region of fixed size are +marked for ghost stars brighter than a specific cutoff magnitude. +Similar to the trails, the average is marked as bad only if all +detections of an object are ghost images. Otherwise, it is noted that +some data may be bad. + +To run 'markstar' by hand, the syntax is: {\pr markstar (file.cpt)}. +There are several optional flags, which are listed if you type +markstar by itself. + +\subsection{Addusno} + +The program {\pr addusno} makes two types of identifications of stars +with the USNO catalog. First, it searches for coincidences within a +small, specified radius. The choice of this radius is somewhat +tricky: it is important to catch the association, but also to avoid +making too many associations. Unfortunately, astrometric errors in +the USNO catalog make it necessary to choose a surprisingly large +radius of 5\asec. To avoid making double matches (matching the two +USNO stars to the same photometry database object), only the USNO +object closest to the object is associated. There are also a +substantial number of stars with significant proper motion between the +USNO epochs and the current date. It is not unusual to see 8-15\asec\ +discrepancy for specific stars. To catch these objects, any objects +which do not match the USNO database, but which have more than a +minimum number of measurements (2 or 3?) are searched for more distant +USNO companions. A large, user-specified radius is used for this +search. Of course, only USNO stars which are not already matched to +the database may be candidates for the proper motion match. This +stage is susceptible to errors in that a slow-moving solar-system +object may be associated with a faint USNO star. For this reason, +objects flagged as high proper motion stars should be taken with some +caution. Of course, during the {\pr addusno} stage, no objects +already flagged as bad by {\pr markstar} will be matched with USNO +stars. + +To run 'addusno' by hand, the syntax is: {\pr addusno (file.cpt)}. +There are several optional flags, which are listed if you type +addusno by itself. + +\subsection{Markrock} + +Once stars have been identified with the USNO database, it is possible +to search for asteroid detections. The program {\pr markrock} +searches within a specific {\ty cpt} file for objects which are moving along +straight lines in (X,Y,t) space. In order for an object to be found +by {\pr markrock}, it must have three points, each with only single +measurements. The third point must land within a specified distance +of the line projected from the first two data points. These +comparisons are made in the three dimensional space of image position +and time. Also, objects are only accepted if they have a speed less +than a threshold. If an object is moving too fast, it will make a +streak on each individual image and will probably be missed anyway. +Moving objects detected in this way are flagged in the photometry +database and are also written to a Rock database. Currently, the Rock +database just contains ASCII lists of the RA, DEC, Mag, and time for +each detection. In fact, the data are stored in pairs of detections, +with the middle detection saved twice. This format makes it easy to +plot lines connecting the points using the connect-the-dots plotting +style of status/kapa (see below). + +One of the current drawbacks of {\pr markrock} is the insistence on +three observations with only 1 measurement each. If an object is +moving too slowly, two of the dectections of the object may be merged +together. Enough data is stored in the database to extract these +individual measurements, so a more sophisticated search is possible. +It is also currently the case that USNO proper motion stars are +ignored, but some of these will be asteroid detections which are +simply too close to a faint USNO stars (it has to be faint because it +would otherwise be associated with a star in the photometry +database). + +To run 'markrock' by hand, the syntax is: {\pr markrock (file.cpt)}. +There are several optional flags, which are listed if you type +markrock by itself. + +\section{Relative Photometry -- nrphot} + +Once the data has been incorporated in the photometry database and the +data validation routines have been run, it is possible to determine +photometric solutions for all of the images. The program which does +this is nrphot. It tries to find a set of solutions for each image +which reduce both the scatter per image and the scatter per star. +\note{the following section is taken from Magnier et al +1992, A\&A Supp 96, 379. It could use a little editing to fit the +reality of the LONEOS implementation.} + +In order bring the different measurements to a common photometric +system, relative photometry was performed to determine calibrations +for each image. We used multiple measurements of individual stars on +overlapping images to connect neighboring images. It is useful to +discuss some of the details of this process. + +A typical photometry equation can be written in a form such as: +\begin{equation} +M_{app} = c_{\lambda} + m + a_{\lambda}*\zeta + \gamma_{\Delta\lambda}*(\Delta\lambda) \\ +\label{M_mag} +\end{equation} +where $m$ is the instrumental magnitude, +\begin{equation} +m = -2.5\log(N_{e}) + 2.5\log(t). \\ +\label{inst_mag} +\end{equation} +$N_{e}$ is the number of electrons measured in a single star, $t$ is +the exposure time for the image, $M_{app}$ is the apparent magnitude +of the star in the standard system, $\zeta$ is the airmass of the +image, and $\Delta\lambda$ is the value of a color index (\ie\ $B-V$ +or $V-I$) for the particular star. The subscript $\lambda$ refers to +the filter and the subscript $\Delta\lambda$ refers to the color +index. Notice that we are ignoring the second-order color-airmass +crossterm, and any higher order terms in $\Delta\lambda$ or $\zeta$. +In all situations, $c_{\lambda}$, $a_{\lambda}$, and +$\gamma_{\Delta\lambda}$ are calibration coefficients dependent on the +instruments and the site. The term $c_{\lambda}$ is a factor to scale +the response of the detector. The terms $\gamma_{\Delta\lambda}$ and +$a_{\lambda}$ give the variation of the sensitivity with the color of +the star and the airmass. Also, Eqn. (\ref{M_mag}) is only valid for +photometric conditions. Under non-photometric conditions, we need to +add another term to account for clouds: +\begin{equation} +M_{app} = c_{\lambda} + m + a_{\lambda}*\zeta + \gamma_{\Delta\lambda}*(\Delta\lambda) + clouds \\ +\label{M_cloud} +\end{equation} +Now, we can regroup the terms in (\ref{M_cloud}) as follows: +\begin{equation} +m = [M_{app} - \gamma_{\Delta\lambda}*(\Delta\lambda)] - [c_{\lambda} + a_{\lambda}*\zeta + clouds] +\label{M_sep} +\end{equation} +Of the two terms on the right side of equation (\ref{M_sep}), the variables +in the first pair of brackets are dependent on the properties of the +star, those in the second pair of brackets are dependent on the image parameters, +except for $c_{\lambda}$ which is independent of both image and star. +We can now define the ``relative magnitude'' and the ``calibration +magnitude'' from (\ref{M_sep}) as follows: +\begin{eqnarray} + m & = & M_{rel} + M_{cal} \label{rel_phot1}\\ + M_{rel} & = & [M_{app} - \gamma_{\Delta\lambda}*(\Delta\lambda) + \Delta] \label{color}\\ + M_{cal} & = & - [c_{\lambda} + a_{\lambda}*\zeta + clouds + \Delta] \label{M_cal} +\end{eqnarray} +where $M_{rel}$ is the relative photometric magnitude of the star in +the internal system and $M_{cal}$ is a correction factor to convert +instrumental magnitudes to relative magnitudes. The terms $\Delta$ +are included in eqs. (\ref{color}) and (\ref{M_cal}) to make explicit +the fact that an arbitrary constant can be added to $M_{rel}$ and +subtracted from $M_{cal}$. The goal of relative photometry is to +determine $M_{cal}$ for each image, then use this $M_{cal}$ to find +$M_{rel}$ for all stars from eq. (\ref{rel_phot1}). Once one has +$M_{rel}$ for each star, one can then convert them to a standard +system ($M_{app}$) using an equation of the form of +Eqn. (\ref{color}). In the case of the LONEOS photometry, the absence +of a filter makes an effective bandpass which is so wide that it may +be meaningless to force it to a specific, standard filter. For the +sake of variability measurements, it is probably more sensible simply +to refer to the LONEOS system magnitudes and present a set of color +terms which give the rough relationship between stars of various +colors and the LONEOS system. + +In fact, the discussion above make a significant simplification which +assumes that all images will have a single calibration magnitude +$M_{cal}$. In fact, it is the case in the LONEOS system that a +variety of effects introduce systematic variations across the images. +For this reason, in practice we extrapolate the above discussion to a +situation where the value of $M_{cal}$ is allowed to be a function of +position, and is fit with a 2-D polynomial of some degree. With the +LONEOS system, we allow the fit to take polynomials of up to 4th +order, but only accept the minimum order which significantly reduces +the scatter for the particular image. + + An equation of the form (\ref{rel_phot1}) exists for each star +detected on each image. As described, the value $m$ depends on both +the particular image and the particular star, the value of $M_{cal}$ +only depends on the image, and the value of $M_{rel}$ only depends on +the particular star. Let us label the images with the subscript $i$ +and the stars with the subscript $j$. Equation (\ref{rel_phot1}) can +then be written in the form: +\begin{equation} +m_{i,j} = M_{rel,j} + M_{cal,i} \label{rel_phot} +\end{equation} +Of course, $m_{i,j}$ only exists for certain combinations of images +and stars: not all stars appear on all images. In the entire system +of equations, only one value of $\Delta$ (from eqs. \ref{color} and +\ref{M_cal}) can be used, but, as noted above, the {\em value}\ of +$\Delta$ is arbitrary. + + In equation (\ref{rel_phot}), both $M_{rel,j}$ and $M_{cal,i}$ +are unknown quantities. We can use a method of least squares to find +these values for the entire set of data. We try to minimize the chi +square, defined as: +\begin{equation} +\chisq = \sum_{i,j}(m_{i,j} - M_{rel,j} - M_{cal,i})^2 / \sigma_{i,j}^2 \label{chisquare} +\end{equation} +where $\sigma_{i,j}$ is the error in the measurement $m_{i,j}$, and +the index $i$ runs over all images, while the index $j$ runs over all +stars which are multiply measured, \ie\ those stars which appear on more +than one image. + + We attempt to minimize \chisq\ analytically by finding the +derivatives of \chisq\ with respect to $M_{rel,j}$ and $M_{cal,i}$ +and setting them equal to zero: +\begin{eqnarray} +\frac{\partial\chisq}{\partial M_{rel,j}} = \sum_{i}-2(m_{i,j} - +M_{rel,j} - M_{cal,i}) / \sigma_{i,j}^2 = 0 \nonumber \\ +\frac{\partial\chisq}{\partial M_{cal,i}} = \sum_{j}-2(m_{i,j} - +M_{rel,j} - M_{cal,i}) / \sigma_{i,j}^2 = 0 +\end{eqnarray} +In these summations, the index $i$ runs over all images in which star +$j$ appears, and the index $j$ runs over all stars which appear on +image $i$ which are multiply measured. Solving the equality leads to +the following system of equations: +\begin{eqnarray} +R_{j}M_{rel,j} = \sum_{i}(m_{i,j} - M_{cal,i}) / \sigma_{i,j}^2 \label{system1} \\ +R_{i}M_{cal,i} = \sum_{j}(m_{i,j} - M_{rel,j}) / \sigma_{i,j}^2 \label{system2} +\end{eqnarray} +where +\begin{eqnarray} +R_{i} = \sum_{j}\frac{1}{\sigma_{i,j}^2} \nonumber \\ +R_{j} = \sum_{i}\frac{1}{\sigma_{i,j}^2} +\end{eqnarray} +Again, the index $i$ runs over all images in which star $j$ appears, +and the index $j$ runs over all stars which appear on image $i$, with +the restriction of multiple measurements. The term $\Delta$ from +eqs. (\ref{color}) and (\ref{M_cal}) acts as an arbitrary zero point +between the relative and apparent photometric systems. + + Equations (\ref{system1}) and (\ref{system2}) are a set of +$N_{images} + N_{stars}$ linear equations with $N_{images} + +N_{stars}$ unknowns. The unknown parameters are the terms $M_{rel,j}$ +and $M_{cal,i}$. Unfortunately, $N_{images} + N_{stars}$ is a number +on the order of 5000 - 10000, thus an analytical solution is +impractical and an iteration method must be used. A simple iteration +method is as follows: + +%\SingleSpace +\begin{enumerate} +\setlength{\parskip}{-0.05in} +\item Guess at values for each $M_{cal,i}$. +\item Use equation (\ref{system1}) to find values for $M_{rel,j}$. +\item Substitute these $M_{rel,j}$ values into (\ref{system2}) to get a new +set of $M_{cal,i}$ values. +\item Return to step 1. +\end{enumerate} +%\DoubleSpace +This process is repeated until some criterion is reached, such as the +\chisq\ no longer changes or reaches a desired minimum. Although a +minimum can be found, error propagation causes severe problems +since images are only connected to neighboring images. The end point +images are connected by \approx 50 intermediate images, and errors +propagating over such a long distance can cause severe errors. Also, +$M_{rel,j}$ and $M_{cal,i}$ are not uniquely defined, because of the +term $\Delta$. Therefore, during the iteration process, the values of +$M_{rel,j}$ and $M_{cal,i}$ tend to wander and do not tend to converge +($M_{cal,i}$ may be incorrect by typically 0.5 mag). These problems +can be overcome if some of the values $M_{cal,i}$ or $M_{rel,j}$ are +known. In this situation, the known values can be held fixed, +providing a reference point for the remaining values. If enough +values are known across a large enough portion of the data, no image +is separated from the fixed values by many images, and errors will not +build up. + + Images taken under photometric conditions provide a set of +known $M_{cal,i}$ which can be held fixed. This is true since, if the +image is taken under photometric conditions, the term $clouds$ in eq. +(\ref{M_cal}) is by definition 0. The other terms in this equation +are either known ($a_{\lambda}*\zeta$) or are constant for all images +($c_{\lambda}$) and can thus be assigned to a set value, which can +later be removed with the zero point calibration. We make an initial +assumption of the value of $a_\lambda$, but use the data themselves to +determine a best fit choice for $a_\lambda$. The task that remains is +to determine which images were photometric. + + Since the conditions during observations varied greatly from +night to night, we decided to determine which images were photometric +from the images themselves. To do this, we used a variation on the +iteration method described above. Before starting the iterations, we +assume a value for $c_{\lambda}$. We can choose this arbitrarily, and +correct for our choice in the correction to an absolute photometric +system. Now, we perform the following iteration: + +%\SingleSpace +\begin{enumerate} +\setlength{\parskip}{-0.05in} +\item Assume that all images are photometric and determine $M_{cal,i}$. +\item Find the corresponding values of $M_{rel,j}$ from equation (\ref{system1}). +\item Use these $M_{rel,j}$ values to find new values for $M_{cal,i}$. +\item Find the values of $clouds$ for each image from these $M_{cal,i}$ +\item Assume that any image with a value $clouds$ more than 1 standard +deviation from the mean value (over images) has a substantial cloud +layer. +\item Return to step 1, but use the iterated $M_{cal,i}$ values for those +images with substantial clouds. +\end{enumerate} +%\DoubleSpace +This process is continued until \chisq (eq. \ref{chisquare}) does not +change substantially. Figure (\ref{fig:Mrel}) shows a histogram of +the occurrences of different $clouds$ values. There are a few +important points to make about this figure. First, there is a +concentrated peak at 0. All of these images have essentially zero +cloud level. As one would expect, there are a large number of images +to the right of this peak. These are the images which were taken +through some amount of clouds. Since the exposures were stopped if +the clouds got too thick, it is not surprising that these images taper +off at higher cloud levels. The most curious and telling part of the +plot is the area below the 0-cloud peak. There are many images on the +``negative cloud'' side of the peak. Because ``negative clouds'' are +unphysical, there must be a simple explanation for this. + + We calculate the error on the relative calibration of each +image by finding the scatter of the $M_{cal,i}$ values for each image +and dividing by $\sqrt{N stars}$. A histogram of this error for all +images is shown in Figure (\ref{fig:dMrel}). We also show the +variation of the formal photometric error reported by \dophot\ with +magnitude for each filter in Figure (\ref{fig:dm}). Now that we have +the values $M_{cal,i}$, we can calculate the values $M_{rel,j}$ for +every star on every image. + +To run 'nrphot' by hand, the syntax is: {\pr nrphot (file.cpt)}. +There are several optional flags, which are listed if you type nrphot +by itself. \note{discuss the optional flags in some detail} + +\begin{figure} +\psfig{file=flags.eps,width=9cm} +\caption{\label{flags} \small + Pictoral representation of the Average.code flags. The lower byte + value is used for specific definitions of object types. } +\end{figure} + +\subsection{Data Flags} + +The identifications made by the above routines are noted as a set of +flags in the Average.code structure entry. Some of these flags are +mutually exclusive, so a certain bit may have more than one meaning, +depending on the value of other bits. Figure~\ref{flags} shows a +pictoral representation of the meaning of the different bit fields. +The uppermost bit (bit 15) determines if the object may be considered +a ``fixed'' star or something which has no fixed location. If it is +not fixed, and this bit is 1, then there are several options. The +second uppermost bit (bit 14) will be set if this object is one of the +three types of bad data flagged by {\pr markstar}. The lower byte of +code is used to define the type of bad data: ghost, trail, or bleed. +Thus, any datapoints determined to be a satelite trail will have a +code value of 0xc002 (49154 decimal). If an object is a moving +object, but not a bad datapoint, it may be identified as a ``rock'' +(an asteroid) by the program {\pr markrock}. In this case, the flag +for an asteroid is turned on (bit 13). Thus anything identified as a +rock by {\pr markrock} will have a code value of 0xa000. The lower +byte of code is not yet defined for asteroids, but could be reserved +for distinguishing different classes of asteroids (ie, on the basis of +orbital speeds, and so forth). Objects which are fixed may have a +variety of possible flags. First, an object may be identified with a +USNO star (bit 14). It may be found to exhibit variability (bit 13), +implying that the relative photometry routine should ignore it. If +may be a transient object (bit 12). Note that, while an object +associated with a USNO object may not be transient, an object which is +variable may also not be associated with a USNO object. Thus, we need +seperate bits for variable vs transient. Different types of variable +objects may be represented with different values of the lower byte. +For example, a Cepheid may have a specific lower byte code of 0x01, +which an RR Lyrae may have a lower byte code of 0x02. Thus a Cepheid +associated with the USNO catalog would have a total code value of +0x6001. Finally, any of these types of non-moving objects may have +some bad individual measurements (bit 12) or may be found to have a +significant proper motion (bit 11). It should be noted that the above +guidelines for variable stars are suggestions only, and to date (July +7, 1999), no programs currently assign or define these variability flags. + +\section{Holding it Together} + +The preceeding sections describe the step-by-step analysis and +incorporation of the data from individual images. However, the power +of the Pipeline comes in applying it to large numbers of images. At +LONEOS, we are using four Pentinum II computers to analyse the roughly 200 +images that are observed each night. + +It is best to think of the analysis of images occurring on groups of +images from a given night. Of the set of processes discussed above, +those in section \ref{parallel} are performed on individual images +essentially independently of the other images. The rest of the steps, +both the data incorporation ({\pr addstar}) and the database cleaning +routines occur on the entire batch, one image at a time. Thus, the +former routines can easily be run in parallel, but the latter routines +are run in serial to avoid having two programs writing to same part of +the database at the same time. In this model, we would process all of +the images from a night through the parallel stage of the analysis and +then run the serial stage only after the entire night is finished in +the parallel stage. We thus have a natural division of the pipeline +into two stages. + +The implementation of the two stages involves several programs, some +of which are specific to the LONEOS situation, and others which are +more general. There are two layers of programs which enable batch +processing of many images and repeated batch processing over many +nights. At the highest level, are Perl scripts which choose what data +needs to be analysed and when. These scripts tend to be more +dependent on the details of the LONEOS implementation. At the next +level down, these scripts principally call two C programs which run +the images of a given night through the pipeline. These C programs +are more general and do not depend on the details of the LONEOS +implementation. A defined set of files holds a log of the nights and +images that have been processed, both for the information of people +who might want to monitor the progress, but also used by the programs +to guide decisions about what to analyse next. + +\subsection{Data and File Organization} + +There are a set of standard locations for all of the files used and +created by Ptolemy. These locations have both abstract names used by +the pipeline programs and specific definitions used in the LONEOS +implementation. In general, the definitions of the files are given in +the configuration file, 'config.txt' (currently located in a +hard-wired location, but this may soon change to the mechanism used by +the lastest version of {\pr spicam} (3.0)). The C programs all refer +to the config file for the definitions, but the Perl scripts currently +have the definitions hardwired as variable names defined in the +beginning of each file. This should be changed to a mechanism that +looks in the config file, perhaps implemented by a standard Perl +function. + +Here is a list of the main directories and their meanings: + +{\bf logdir: {\ff /metis/d11/logs} - all log files and the files used +by the Perl scripts to control their actions (except locks) go here. + +{\bf workspace}: {\ff /irene/d11/workspace} - the intermediate +analysis stages go here, and the tarred, gzipped directory files stay +here. + +{\bf dumpspace}: {\ff /pallas/d1} - used by slurp to store images +downloaded from the tape archive. + +{\bf database}: {\ff /hebe/d27/database} - the result database, and +also the locks and the {\ff Rocks.dat} file. + +{\bf references}: {\ff /metis/d11/references} - a variety of reference +data, including the HST GSC, the USNO catalog, the config files, and +so forth. + +{\bf source}: {\ff /metis/d11/src/ohana} - all of the programs source, +binaries, and scripts relevant to running the Pipeline are contained +in these directories. + +\subsection{lastnight} + +The first step in the process is the Perl script {\pr lastnight}, +which is highly dependent on the specifics of the LONEOS setup. It +looks at the disks used by the observers to write the images from a +given night. Images are written to a directory called {\ff +/pallas/d[1-6]/YYMMDD} where {\ff YYMMDD} is an abbreviation for the +current night's date (ie, 990223). Many files use this naming scheme, +so we will refer to such a name as {\ff YYMMDD}. All of the images +are stored in this directory with names {\ff yyyyMMDDxxxxb.fits} where +{\ff yyyy} is the full year, {\ff MM} is month, {\ff DD} is date, and +{\ff xxxx} is a sequence number. The 'b' refers to the 'b' CCD, and +the '.fits' extension is required. The script {\pr lastnight} is +called by a cron job scheduled for 7am on {\ff hebe.lowell.edu} (note +that the clock on hebe is set to PST, not local MST - fix this some +day! see also man crontab for details on setting up a cron job). The +script searches the disks {\ff /pallas/d[1-6]} for a directory of the +right form (basically any directory beginning with a number). By +default, the program demands that the directory have the name derived +from today's date, but with the flag {\pr -any}, it will accept any +directory. It then checks to see if this directory has been already +been analysed (ie, it is included in the {\ff processed.log} file). +If not, it lists all files in that directory with the right ending +(\ff b.fits}) and places these names in a file, {\ff +/metis/d11/logs/YYMMDD.inlist}. It also writes the name of the +directory (along with some other data) in a file {\ff extracted.log}. +When it has successfully ended and identified some data, it calls the +next script, {\pr stage1} and then quits. + +\subsection{slurp} + +An alternative to {\pr lastnight} is the Perl script {\pr slurp}, +which looks for data on the tape archive. This script looks at the +{\ff tape.log} file and compares it to the file {\ff processed.log}|. +It tries to identify nights listed in the file {\ff tape.log} which +have not been included in the {\ff processed.log}. If it finds any, +it tries to download as much of the data as will fit on the disk {\ff +/pallas/d1}. This script is called from within the {\pr stage1} +script and is invoked if {\pr stage1} is run without any data already +in the file {\ff extracted.log. + +A related pair program are the {\pr load} \& {\pr unload} C programs +which are needed for loading and unloading the tape jukebox. The +first is called with, {\pr load N}, which takes tape number N and +loads it into the tape drive. The second version, called with {\pr +unload N} takes the tape and returns it to its slot. Note that the +jukebox does not know by itself what number tape is in the drive, so +{\pr unload} needs to be told the correct one. If it is mis-led, it +will complain and refuse to return the tape. It is important to check +on the other users of the tape drive to be sure it is not occupied. +This is done with the {\pr llock}, {\pr lunlock} and {\pr llockstat} +programs. The first, {\pr llock} lets you lock a particular service, +etc, while the last {\pr llockstat} lists the devices which have been +allocated. + +\subsection{stage1} + +The Perl script {\pr stage1} takes the image list produced by {\pr +lastnight} or {\pr slurp} ({\ff YYMMDD.inlist}) and passes the list to +the controller program {\pr control1} which performs the parallel +analysis. The script has a few features to decide which files have +been analysed through the pipeline and to what extent. If {\pr +stage1} is called, it should be able to figure out 1) if there is data +ready to be analysed (ie, there is a new date reference in the file +{\ff extracted.log}), 2) if the data has been partially analysed (ie, +the date reference in {\ff extracted.log} has not been put in {\ff +processed.log}, but there are entries in the file {\ff +YYMMDD.outlist}), and 3) which data needs to be run through which part +of the pipeline (ie, it decides which of the images in {\ff +YYMMDD.outlist} have succeeded at which stages of the analysis, and +starts them at the appropriate spot). These features are most useful +if the analysis crashes in the middle or needs to be restarted. The +{\pr stage1} script is meant to be run without any arguments and it +will figure out what needs to be done. The only possible argument is +the -once flag. This flag tells {\pr stage1} to analyse the data it +finds and quit when it finishes, and not call another instance of +itself. Otherwise, when it is done, it will restart itself, and since +there is no new data in {\ff extracted.log}, it will start a {\pr +slurp} process to get data from the archive. If there is no data in +the archive either, this process will sleep for an hour and try again. +After a few attempts (2 days?), {\pr stage1} will give up and quit +without restarting itself. The script writes a log file called {\ff +\$scriptdir/stage1.NN.log} where NN is a number in sequence between +all runs of {\pr stage1}. A file {\ff stage1.count} keeps track of +which number NN we are on. When {\pr stage1} has finished the +analysis of a night's data, it writes the date in {\ff processed.log} +to let other processes know what has been analysed. The bulk of {\pr +stage1} is not dependent on details of the LONEOS setup, but it does +expect the files to have the naming scheme described above. + +\subsection{stage2} +The serial stage processes are run by the Perl scrip {\pr stage2}. +Normally, this script is always running. It waits for the name of a +night to be added to the {\ff processed.log} file, presumably by {\pr +stage1}. Once it finds a new name in this file, it looks for the +appropriate images in the file {\ff YYMMDD.addlist}. It then submits +the list to the second controlling program {\pr control2}, which runs +the images through {\pr addstar}. Images which are successfully run +through addstar have their {\ff *bf.fits} (flattened image) and {\ff +*.obj} (dophot output) files deleted. When the entire night is done, +the rest of the {\ff *.obj} and {\ff *bf.fits} images are deleted by +{\pr stage2}, the directory with the {\ff *.cmp} and {\ff *.log} files +is tared and gzipped into the file {\ff YYMMNN.tgz}. The directory is +then deleted. Finally, {\pr stage2} calls the scripts {\pr +cleaning.pl} and {\pr run.phot} which runs the cleanup programs ({\pr +markstar - nrphot}) on the database. After a successful run, {\pr +stage2} will automatically respawn a new copy that waits until more +data arrives. + +\subsection{control1 / control2} +These two C-programs perform the tasks of farming the images out to +the different machines for analysis. The two programs are essentially +identical, but {\pr control1} is allowed to use all machines and {\pr +control2} can only use one ({\ff hebe.lowell.edu}, which has the database +local to it). These controller programs start off by executing remote +shells (using {\pr ssh}) on a set of machines. These shells are then used +to execute the remote commands and are kept open the entire time the +process runs. + +Both control programs maintain a list of images in a set of queues. A +set of rules defines the order in which each image travels through the +queues. Each queue is associated with a single process (ie, {\pr +flatten}, astrometry, etc). Images which successfully run through a +queue are sent on to the defined 'next' queue, while images which fail +are instead sent to the defined 'fail' queue. In most cases, the +'fail' queue is a process which marks the image as bad and write the +name in a file, and then puts the image in the 'done' queue. However, +it is possible for the 'fail' queue to be another attempt, such as in +the case of astrometry. In this example, images which fail the +default astrometry are run through a second queue which tries again +with the {\pr -loneos} flag set (this flag makes the assumption that +the header information about the LONEOS region number for the image is +correct, not the RA/DEC information). + +The {\pr ssh} connections on the remote machines are handled as child +processes with communication through the {\ff stdin} and {\ff +stdout/stderr} pipes. Messages can be sent back and forth though +these pipes, and messages which are received from a process associated +with an image are written to a log file with the name {\ff +imagename.log} (where imagename is the root name of the image). The +only required messages from the program are messages saying +``SUCCESS'' or ``ERROR''. The programs therefore all end with either +of these words. The programs are started within the controller as a +command to the shell followed by an echo of ``PROCESS DONE''. If the +controller sees the message ``PROCESS DONE'' without either +``SUCCESS'' or ``FAILURE'' (after waiting an appropriately long time), +it assumes the program crashed. Currently, no timeout is available +for the programs. The flat-field process is run in {\pr mana} and is +a slightly special case. We avoid re-loading the same flat many times +by starting a {\pr mana} process once and running the flatten +procedures as {\pr mana} function calls. The images which are +analysed (with an indicator of success or failure at each stage) are +written to a file named {\ff YYMMDD.addlist} by the {\pr control1} and +\ff YYMMDD.outlist} by {\pr control2}. + +\subsection{Lock files} + +A variety of locking mechanisms are used to avoid overwriting the +database or running more than one version of the controlling programs. +There is a Perl program, {\pr locks}, which allows the user to set or +check the state of the different locks. All programs which write to +the database check for the existence of a lock on the database and +refuse to run if it exists. The {\pr control1/2} programs and the +{\pr stage1/2} programs check for their own lock files and only run if +they don't exist. Finally, the user can set the locks 'kill' or +'halt' for either {\pr stage1} or {\pr stage2}. The first tells the +process to stop immediately, taking care of bookkeeping first. The +second tells the process to finish the current set of images but not +start a new version. + +\subsection{Other Perl Scripts} + +there are several other scripts which are either used to do some +simple function or may be used by the maintainer to check on the +status of the Ptolemy exectution. The two programs {\pr checkoutlist} +and {\pr checkaddlist} take a list of {\ff YYMMDD.outlist} or {\ff +YYMMDD.addlist} files and list the number of images in the file and +the number of successes at each analysis stage. There programs {\pr +cleaning.pl} and {\pr run.phot} deal with the cleanup programs and the +running of {\pr nrphot}. The program {\pr backup.stuff} takes the +first 4 characters of the year/month file names and backs up to the +jukebox tape all tar files for the given month. + +\section{Operational Issues and Subtleties} + +The current LONEOS system runs principally on four Intel machines +under Linux. The existing system has been generally quite stable over +the past year of operation, but there are some subtleties. One +significant issue is the RAID disk attached to {\ff hebe.lowell.edu}. +This device is a set of three 9 GB disks merged into one 27GB RAID +disk. The current problem with the /hebe/d27 RAID results from the +poor way it was setup. It was set up without much knowledge of the +(at the time) rather rudimentary linux RAID system. As a result, the +implementation is such that the disk will not automatically be +configured and mounted at boot time. There is a script {\ff +/root/mdstart} which lists the step to setup the RAID correctly. It +would be helpful if the {\ff rc.local} or {\ff rc.sysinit} files were +fixed to start, fsck and mount the RAID automatically at boot. The +setup on {\ff metis} is substantially more mature and is correctly +implemented. It comes up when the machine boots. + +\section{Visualization -- Status} + +Visualization of the data in the photometry database can best be +performed with the program {\pr status}. This is a command-line +driven program with a math and macro language which makes it easy to +perform complex tasks. + +First, a few notes about the user interface. The interface has an +interaction similar to {\pr tcsh}. The arrows allow editing of +previous commands. You can also use emacs-like commands such as +cntl-a to reach the beginning of the line and cntl-e to reach the end. +There is command and file completion: if you type part of a command +(as the first thing on a line) and then type tab, it will fill in as +much as possible, until the word is not unique. Typing tab twice at +that point will list the possible endings. For any but the first word +on a line, the same thing will happen for the files in the current +directory. It is also possible to type just a fraction of a command, +as long as it is unique. An ambiguous command will list the possible +alternatives. For example: +\begin{verbatim} +status: c +ambiguous command: c ( catalog cgrid clear create cursor ) +\end{verbatim} + +The shell is essentially an interpretive programming language. +Variables are set as follows: +\begin{verbatim} +status: $fred = 10 +\end{verbatim} +Any expression within curly brackets \{\} is assumed +to be an arithmetical expression and is evaluated before the line is +executed. For example: +\begin{verbatim} +echo {$fred*dcos(45)} +\end{verbatim} +would give the response 7.07107. There are math functions cos, sin, +and tan, which operate on radian expressions, and also dcos, dsin, +dtan, which operate on degree expressions. There are also the +equivalent inverse functions: eg., asin and dasin return radians and +degrees, respectively. The help section on Math defines all of the +available math functions. + +\subsection{Miscellaneous Commands} +\begin{verbatim} +! -- system call +? -- list commands +?? -- list variables +echo -- type this line +exec -- system call +exit -- exit program +help -- get help on a function +output -- redirect output to file +quit -- exit program +scan -- scan line from keyboard or file to variable +wait -- wait until return is typed +which -- show command +\end{verbatim} +Most of these are self-explanatory. The command ?? prints the system +variables. The {\tt help} command will provide help on a single +command or, without any arguments, will list all available help +files (this includes general help not associated with a specific +command). + +\subsection{Shell Programing} +\begin{verbatim} +break -- escape from function +for -- loops +if -- logical cases +input -- read command lines from a file +macro -- deal with the macros +\end{verbatim} + +There are several options for programming in {\pr status}. First, a +file which contains a series of commands can be executed with {\tt + input (filename)}. It is also possible to define macros which will +behave much like regular commands. A macro is defined by typing {\tt + macro name} or {\tt macro create name} followed by the commands. +Arguments to the macro are assigned to the variables \$1 .. \$N and +the number of arguments is given by \$0. Macros may be defined in +{\tt input} files, and in fact when {\tt status} is started, it loads +the file {\tt \~/.statusrc} which may contain default macros. Simple +loops and if statements can be performed, and are quite useful for +complex macros. + +``If'' statements are similar in syntax to C if statements, but only +the following logical operators are available: $>$, $<$, $=$, !, $|$, +and \&. Notice that (currently) there is no $>=$ or $<=$ symbol. The +operator ! means ``not equal to'', but cannot be used to negate a +logical value. The operators $|$ and \& have the meaning of ``or'' and +``and'' respectively. Math expresions in the if statement must be +contained in curly braces, as elsewhere. Variables with string values +may use the logical $=$ operator to test if two strings are the same. +``For'' loops are quite simplistic. The form is: +\begin{verbatim} +for var first last delta + (commands) +end +\end{verbatim} +The value of {\tt \$var} will start at the value {\tt first} and increment by +{\tt delta} after each loop. The loop will stop after {\tt \$var} is greater +than {\tt stop}. The value {\tt delta} is optional, with 1 assumed. +The value of {\tt \$var} may be changed during the loop, and if set +beyong the value of {\tt last} will end the loop early. + +\subsection{Vector Plotting} + +\begin{verbatim} +box -- draw a box on the plot +clear -- erase plot +create -- create a new vector +cursor -- get coords from cursor +grid -- plot cartesian grid +hist -- create histogram from a vector +labels -- define labels for plot +limits -- define plot limits +plot -- plot a pair of vectors +print -- write vectors to file +ps -- define labels for plot +set -- vector math +style -- set the style for graph plots +vectors -- list vectors +zplot -- plot scaled points +\end{verbatim} + +In addition to scalar variables, {\tt status} can manipulate and +display 1-D vector variables. Many of the commands which extract data +from the photometry database place the data in vectors as well as +plotting them. A vector can also be created based on a number +sequence with the command {\tt create name Nelements start delta}. +The resulting vector has $Nelements$ entries, starting at a value of +$start$ and running until $start + delta*Nelements$. If $delta$ is +0.0, all elements will have the value of $start$. A histogram of a vector +may be made with the command {\tt hist}, which creates a new vector +containing the histogram of the first vector. The data range and bin +size of the histogram are defined in same way as with create. This +makes it easy to create the index vector that goes with a histogram +vector: +\begin{verbatim} +hist y Ny 1 100 0.1 +create dx 1 100 0.1 +\end{verbatim} +The above will create a histogram of y in Ny and the index in dx. +Plotting this with {\tt plot dx Ny} will show the histogram. + +Vector math is performed with a command of the form {\tt set new = + (expression)}. The expression is some math function employing +vectors and scalars. A complete listing of the math operators +available in {\tt set} can be found in the help for {\tt set}. + +Once vectors are defined, they may be plotted. A pair of vectors can +be plotted against each other if they have the same number of entries. +The plotting is performed on the graphics window, Kapa. There are +actually several graphics windows available to {\tt status}, any of +which may be used to plot at any time. Some of the more complex +operations default to either graphics window 0 or 1, depending on the +context. Except for those functions with a pre-defined window, all +plotting functions apply to the current graphics window unless an +option {\tt -n N} is given to specify a different window. The +plotting style is determined by the command {\tt style} which can set +the line width, the line type (solid, dashed, dotted, etc), the point +type (box, cross, etc), the point size, the color, and whether a pair +of vectors is plotted as a sequence of points, a set of connected +lines, or a histogram. Some functions which make plots use their own +styles, as discussed below. The function {\tt limits} lets the user +set the range of the plot axes, or check the current setting. The +command {\tt plot} will plot a pair of vectors on the current graphics +window using the current plotting style for that window. The command +{\tt zplot} will plot a pair of vectors with the point size scaled by +a third vector, with maximum and minimum point sizes representing +specified values. The {\tt cursor} command goes the other way: this +command puts the Kapa window in cursor mode and waits for input from +Kapa. The user can then type any alphanumeric key on the graphics +windows and will be told both the pointer location (in the graphics +coordinates) and will have the coordinates stored in {\tt status} +variables. For example, by typing ``1'' in the sky display window, +the RA and DEC of the pointer are stored in the variables {\tt \$R1} +and {\tt \$D1}. This command can be used to let the user define +locations or regions of interest on the Kapa window. (Future addition: +{\tt button}, which does the same with the mouse buttons). + +\subsection{Database Functions} + +\begin{verbatim} +gcat -- get catalog at location +gimages -- get images at location +gstar -- get star statistics +extract -- extract average vectors from catalogs +mextract -- extract measurement vectors from catalogs +imstats -- plot image statistics +imextract -- extract image vectors from database +lcat -- list catalogs in display region +cmatch -- match two catalogs +\end{verbatim} + +There are a variety of other commands which directly refer to the +photometry database. Some of these functions extract data of various +types from the database, others perform more complex plotting +operations. The commands listed above are those which simply extract +data from the database. The first three list information relevant to +a specific RA, DEC location on the sky: {\tt gcat (RA) (DEC)} lists +the catalog at the specified location and places the name in the +variable {\tt \$CATNAME}, {\tt gimages (RA) (DEC)} lists all images +which overlap the specified location, {\tt gstars (RA) (DEC) (RADIUS)} +lists data about the stars within a specified radius of the specified +location (all numbers above are given in decimal degrees). Similarly, +{\tt lcat} lists the catalogs in the region. Imstats lists statistics +about each image + +The next three commands extract a specific piece of information from +the photometry database and places it in a vector. First, {\tt + extract} will extract average values for each star and place it in a +vector. Next, {\tt mextract} will extract measurement values for each +star and place it in a vector: as a result a single star may have +multiple entries in the measurement vectors. Finally, {\tt imextract} +will extract image statistics into vectors (not yet implemented). + + +\begin{verbatim} +catalog -- plot catalog stars +cgrid -- plot sky coordinate grid +cplot -- plot vectors in sky coordinates +czplot -- plot scaled vectors in sky coordinates +images -- plot image boxes +imdense -- image density plot +lcurve -- plot lightcurve for a star +pcat -- plot catalog boundaries +region -- define sky region for plot +resid -- plot residuals +simage -- plot stars in an image +\end{verbatim} + +There are two types of database plotting functions: those that display +or refer to the spatial charateristics of the data and those that +refer to other types of charatersitics, such as the time domain. The +graphics window 0 is reserved for all plots of objects on the sky. +The command {\tt region} defines the current sky coordinates for plots +in graphic window 0. The command {\tt pcat} plots the outline of all +photometry database files which are within the currently defined +region (and by default, only those with data). {\tt images} plots the +outline of the images in the image database, while {\tt imdense} shows +the number of images at a location by randomly spacing dots within the +boundary of the images. The command {\tt cgrid} +draws a grid in celestial coordinates on the for the current region. + +The most complex, but also one of the most useful command is {\tt + catalog}, which plots the positions of stars in the photometry +database (and others) on the sky. There are many options to this +command. One set allows the user to plot stars from the photometry +database (the default), from the HST GSC, or from an ASCII text file +with RA, DEC, and Mag in specified columns. If the ASCII file has a +fixed number of bytes per line, the data can be more quickly loaded. +The size of the points may be scaled by the star magnitude, by the +number of observations of the star, or by the number of missing +datapoints for the star. In addition, points may be plotted only if +they land in specified magnitude ranges, or with specified numbers of +measurements, or missed measurements. Also, objects may be plotted +only if they have a specified Average.code, so that only asteroids or +only perfect stars may be plotted. The plotted vectors may be saved, +if desired, and the source catalog epoch may be specified as different +from J2000 (only valid for ASCII data). + +Several other commands relate to non-spatial charateristics of images +and stars. {\tt lcurve} will plot a light curve for all stars within +some radius of a point. {\tt resid} plots the photometry residuals +for a particular region file. + +\section{Some Examples} + +\note{we need some better and more relevant examples} + +\begin{figure} +\psfig{file=fullsky.ps,width=16cm} +\caption{\label{allsky} \small + Map of the entire sky, and images added to database. } +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location +of the images currently in the database. This picture was made with +the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 gls +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=polar.ps,width=16cm} +\caption{\label{polar} \small + Map of the sky in polar project, and images added to database. } +\end{figure} + +Fig.~\ref{allsky} shows a map of the entire sky, and the location of +the images currently in the database from a polar project. This +picture was made with the following commands: (output is not shown) \\ +\begin{verbatim} +status: region 0 0 90 zea +status: cgrid +status: style -lw 2 -c red +status: images +status: ps +\end{verbatim} +In this example, on the graphics window, the image boxes are shown in +red. The user now has the possiblitiy of using the cursor command to +narrow in on a specific region, and so forth. + +\begin{figure} +\psfig{file=catalog.ps,width=9cm} +\caption{\label{catalog} \small + Comparison between HST GSC and photometry database astrometry. } +\end{figure} + +Fig.~\ref{catalog} shows an example comparison of the photometry +database star positions and the HST Guide Star Catalog star positions. +The crosses are all objects in the photometry database, while the +boxes are only the stars identified as USNO stars. The circles are +the stars from the HST GSC. The size of both points is a function of +brightness. This plot was made with the following commands (starting +from the previous image): +\begin{verbatim} +status: cursor (typed 1 on region of interest) +1 137.097858 22.698305 +q 137.097858 22.698305 +status: region $R1 $D1 0.2 TAN +status: cgrid +status: box +status: style -pt 0 +status: gcat $R1 $D1 + 0 n2230/1951.cpt * +status: style -pt 2; cat -all -m 12 18 +status: style -pt 1; cat -all -m 12 18 -ID $USNO +status: style -pt 7; cat -all -m 12 18 -g +\end{verbatim} + +\section{Software Distribution} + +All of the code, binaries, and scripts for the Pipeline are stored in +the ohana tree at {\ff /metis/d11/src/ohana}. The ohana directory contains +subdirs of bin, lib, include, src, config, and doc. There is a +Makefile at the top level which calls lower-level Makefiles to build +the programs desired. One of the important concepts in the ohana +distribution is the design of the directory layout to enable binary +support for multiple architectures. The Makefiles use the environment +variable ARCH to determine the appropriate architecture. The binary +and library files are stored in subdirectories of the form {\ff bin/\$ARCH} +and {\ff lib/\$ARCH}. It is therefore necessary that users of the ohana +system architecture define the ARCH variable correctly. It is also +necessary to include in the user's PATH the directory +{\ff /metis/d11/ohana/bin/\$ARCH}. At LONEOS, +with the solaris sparc station and the linux machines both in use, +ARCH can take on the values of 'linux' and 'sol'. The values 'sun4', +'irix', and 'hp' have also been used at various sites. A simple bit +of csh script in the users .cshrc or equivalent can make this +assignment transparently: + +\begin{verbatim} +set sys=`uname -s` +switch ($sys) + case IRIX64: + setenv ARCH irix; + breaksw; + case SunOS: + set ver=`uname -r | awk '{print substr($1,1,1)}'`; + if ($ver == 5) then + setenv ARCH sol + else + setenv ARCH sun4 + endif + breaksw; + case Linux: + setenv ARCH linux; + breaksw; + default: + echo "unknown architecture"; + setenv ARCH unknown; + breaksw; +endsw +\end{verbatim} + +The Makefiles use the ARCH variable not only for destination +directories, but also for destination binary names. All files of the +form fred.c are compiled to fred.ARCH.o (instead of fred.o), +uninstalled libraries get the names libfred.ARCH.a (instead of +libfred.a), while uninstalled programs are called fred.ARCH. + +The perl scripts are kept in {\ff ohana/src/perl}, with links to the +appropriate bin directories. All programs are stored in directories +of the form {\ff ohana/src/program}. The exceptions are {\pr gastro} +(stored in {\ff ohana/src/astro}), and {\pr control1/2} (stored in +{\ff ohana/src/astro}). Also, a variety of simple, user-level +functions are available in {\ff ohana/src/misc}. + +Most of the ohana package of programs use several common libraries. +These include the {\pr readline} library (except for {\pr dophot}, +this is the only part of ohana that is not written by Eugene Magnier), +which is used for command-line interface systems. Other important +libraries are the {\pr fits} library for implementing the FITS +specifications in a convenient C-friendly environment, and the {\pr +ohana} library, consisting of basic string and other basic operations. + +\section {Comparison with USNO catalog} + +I have made several comparisons between the USNO catalog and results +from running images from Loneos through the photometry pipeline. In +general, there is a good agreement between the astrometry of the two +datasets. There are specfic areas where the two disagree, however. +First, as discussed above, the USNO star positions may be +significantly different from the observed star positions simply +because of proper motion. The first picure in the ``Comparisons +between Loneos + USNO'' pages shows several stars with significant +proper motion. These are currently being flagged in the photometry +database as discussed. Other differences evident in this first +picture include the classification of bright stars and the splitting +of faint and bright stars. In the upper right of the image, two +bright stars are visible, one of which is actually a double. The USNO +catalog does a good job of detecting both, but it also detects and +includes as stars several diffraction spike points. The HST GSC only +detected a single point and did not split the star at all. The Loneos +data in our pipeline did not detect the star at all due to saturation. +A star in the upper left corner shows the ability of {\pr dophot} in +the Loneos pipeline to split double stars. This star is, on close +inspection, clearly a double. The USNO catalog only detects one. The +Pipeline not only splits the two stars, but it also makes the correct +cross-identification of the stars. + +The second set of pictures shows the ability of the Loneos pipeline to +deal with small galaxies. There are two small galaxies in this field, +both of which are broken into 4 or 5 objects by the USNO catalog, but +maintained as single objects in the Loneos pipeline. In another +example (not shown) a bright star within the disk of a small galaxies +was clearly split from the galaxy by the pipeline (and also by USNO). + +\appendix +\section{Parameter file params.txt} +\note{replace with up-to-date version from Lowell} +\begin{verbatim} +# Configuration file for Loneos analysis pipeline + +# important files and directories +IN_DIR /pallas +OUT_DIR /irene/d11/workspace +# +CATDIR /hebe/d27/database +IMAGE_CATALOG /hebe/d27/database/Images.dat +IMAGE_CATALOG_TEMPLATE /hebe/d27/database/template.cat +CATALOG_TEMPLATE /hebe/d27/database/template.cat +ROCK_CATALOG /hebe/d27/database/Rocks.dat +# +GSCFILE /metis/d11/references/GSC/GSCregions.tbl +GSC_DIR /metis/d11/references/GSC +USNO_CDROM /metis/d11/references/USNO +DOPHOT_PARAMS /metis/d11/references/config/default_parameters +FLATTEN_SCRIPT /metis/d11/references/config/flatten.pro +LONEOS_REGIONS /metis/d11/references/config/regions.map + +# parameters for "fstat" +# instrumental mag zero point +ZERO_PT 24.5 +DOPHOT_CHAR_LINE 129 +DOPHOT_TYPE_FIELD 5 +DOPHOT_AP_FIELD 91 +DOPHOT_PSF_FIELD 52 +MIN_SN_FSTAT 6.0 + +# parameters for "gastro" +DEFAULT_RADIUS 32.0 +MINIMUM_RADIUS 1.5 +MIN_ERROR 0.6 +MIN_PRECISE 0.12 +CCD_PC1_1 1 +CCD_PC2_2 -1 # for loneos +CCD_PC1_2 0 +CCD_PC2_1 0 +ASEC_PIX 2.82 +NFIELD 3.0 +MMIN 6.0 +ROT_ZERO 0 +dROT 1.0 +NROT 2 +POLAR_AXIS_RA -90.0 +POLAR_AXIS_DEC 89.88 +RA_OFFSET -90.0 +DEC_OFFSET 0.0 + +# parameters for "addstar" +# airmass extinction coefficient (in mag / airmass) +NSIGMA 3.0 +ALPHA 0.03 +XOVERSCAN 60 +YOVERSCAN 50 + +# parameters for "controller" +NEW_IMAGES images.dat +USERNAME gene +PASSWORD gene + +# here we list all available machines +NMACHINES 6 +MACHINE0 metis +MACHINE1 hebe +MACHINE2 iris +MACHINE3 irene +#MACHINE4 juno +MACHINE4 ceres +MACHINE5 vesta + +# here we define which machines which are always free +fMACHINE0 1 +fMACHINE1 1 +fMACHINE2 1 +fMACHINE3 1 +fMACHINE4 0 +fMACHINE5 0 +fMACHINE6 0 + +LOCAL_MACHINE 1 # machine with catalog on local disk + +# parameters for "status" +TIME_REFERENCE 90/01/01 00:00:00 + +# parameters for "nrphot" +# make this one small to boost change of getting an early region +TAU 3.0 +SCATTER_LIM 15.0 +MAG_LIM 17.0 +IMAGE_SCATTER 0.075 +NIMAGE_SCATTER 1.5 +STAR_SCATTER 0.05 + +### parameters for "markstar" +SEARCH_RADIUS 360 # region for initial trail hunt, in arcsec +TRAIL_WIDTH 5.0 # expected trail width, in arcsec +NANGLE_BINS 180 # Number of angular bins in 180 degrees +NPTSINLINE 5 # minimum points needed for trail +MIN_DENSITY 0.025 # minimum linear density in star/arcsec +SPACE_SIGMA 10.0 # how much denser than average for trail? +# parameters defining bright star exclusion regions +BRIGHT_XTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_XTRAIL_MAG 9.5 # faintest sat. star +BRIGHT_XTRAIL_SLOPE -80.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_YTRAIL_WIDTH 5.0 # faintest sat. star +BRIGHT_YTRAIL_MAG 11.0 # faintest sat. star +BRIGHT_YTRAIL_SLOPE -400.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +BRIGHT_HALO_MAG 8.5 # faintest sat. star +BRIGHT_HALO_SLOPE -28.0 # exclusion radius = BRIGHT_SLOPE * (mag - BRIGHT_MAG) +# parameters which define the ghosts +GHOST_MAG 7.5 +GHOST_RADIUS 200 # in arcsec +OPTICAL_AXIS1 2154.0 +OPTICAL_AXIS2 2193.0 + +# parameters for "addusno" +USNO_RADIUS 5.0 # in arcsec +USNO_PROPER 20.0 # in arcsec +USNO_RED 1000 # code for USNO Red data +USNO_BLUE 1001 # code for USNO Blue data + +# parameters for "markrock" +ROCK_RADIUS 2.0 # in arcsec +ROCK_MAX_RADIUS 20.0 # in arcsec +ROCK_MAX_SPEED 0.1 # in arcsec / sec +\end{verbatim} + +\section{Creating New Flats} + +The LONEOS implementation of the analysis pipeline has a limited +ability to choose a flatfield image. There is a mechanism in the +stage1 script to select a flatfield on the basis of the photcode +entry. It works like this: there is a fixed directory for all +flatfield files, currently {\ff /metis/d11/reference}. This directory +is written in the file {\ff flatten.pro}, and also in the script {\pr +stage1}. Each flatfield has a name like {\ff flatcodeN.fits}, where N +is the relevant photcode. The {\pr mana} macro '{\pr flatten}' in +{\ff flatten.pro} loads the file {\ff \$flatdir/flatfield.fits} (OR +SOMETHING LIKE THAT), which is a link to the correct flat. The script +{\pr stage1} has the job of deciding which is the correct photcode and +setting the link appropriately. + +This then begs the question of how this flatfield file should be +created and assigned a new photcode. The flatfield is NOT made +nightly from the entire set of data, as one might expect. This is not +done because of the problems of scattered light in the telescope. The +widefield of the LONEOS telescope makes it somewhat succeptible to +light at relatively large incident angles. As a result, a median +image made from images of the night sky will have different amounts of +scattered light depending on the overall sky brightness. In our +experience, the presence of light clouds and just a small moon (as +little as a quarter) make the resulting median image a very poor +representation of the chip response. Our solution to this problem is +to choose a good night, in which the sky is very dark, and make a +flatfield from that collection of images. As a result, there is too +much decision-making involved to allow the pipeline to make this +decision. Instead, we have chosen to only infrequently make a new +flat from the data, choosing those nights that we are confident will +not introduce excess scattered light. + +There are a set of scripts and programs to make the flatfield image. +The process involves first removing the bias and dark from the image, +by subtracting a dark image, then medianing the resulting images. +\note{find these scripts...} In fact, instead of medianing, +medianfilter returns the average of the fraction of pixels with values +between f1 and f2 percent of the total range of the pixel data. + +\section{Daily Monitoring of the Pipeline} + +On a daily basis, the pipeline generally runs by itself without any +intervention. There are a few checks that can be made to be sure the +process is running OK. + +First, it is good to see if the disk space is sufficient. There is a +component to stage1 which will keep it from running the analysis if +there is less space on the destination disk ({\ff /irene/d11/workspace}) +than 1.1 times the amount of space in the source directory +({\ff /pallas/d?/YYMMDD}). + +Second, it is a good idea to see if the nightly cron job is checking +for new data and if the data is getting analysed. If data was created +last night, there should be a set of files in the log directory ({\ff +/metis/d11/logs}) of the form {\ff YYMMDD.inlist}, {\ff +YYMMDD.addlist}, and {\ff YYMMDD.outlist}. If these files do not +exist, it may just mean that no data was created last night. In that +case, the tail of the file {\ff /metis/d11/logs/lastnight} should say +something like 'there is no new data to analyse'. \note{It would be +nice to include some notification scripts at the end of {\pr +lastnight}, {\pr stage1} and {\pr stage2} which sent some mail +describing what had been accomplished the previous night}. + +If data was created, and these files exist, it is a good idea then to +check that the analysis of the images was reasonable and didn't fail +on most of the images. There are two Perl scripts to check on this. +First, {\pr checkaddlist [filenames]} will take a list of {\ff +*.addlist} files and, for each file, list the total number of images, +then the number that failed for each of the four {\pr stage1} +processes, {\pr flatten}, {\pr dophot}, {\pr fstat}, and {\pr gastro}. +A typical run might look like this: + +\begin{verbatim} +hebe: checkaddlist 9907??.addlist +990701: 206 0 0 0 0 +990703: 199 5 0 0 0 +990705: 150 10 1 0 1 +\end{verbatim} + +It is actually unusual for any except the {\pr flatten} process to +fail, and it might be typical for 1 or 2 of those to fail, since they +will represent aborted images or images which failed to write +everything to disk. The second Perl script is {\pr checkoutlist} +which works just the same with the {\ff YYMMDD.outlist} files, but +only give statistics on the {\pr addstar} process. Here, it is more +common for several images to fail {\pr addstar}. At the current time, +{\pr gastro} does not always return a fail status if it finds a bogus +solution (recent work in late July may have fixed all of those fail +status messages). Images which fail {\pr gastro} are mostly those +which were taken when the sky was too bright and all that is seen is +the sky itself, or the focus frames, or the very short images taken of +bright stars to check the coords. If the images fail {\pr gastro} but +do not get rejected with an error status, they are rejected by {\pr +addstar} on the basis of header keywords which will say 0 stars used +astrometry. It is also the case that {\pr gastro} should return a +failure status if the astrometric solution is sufficiently far from +Cartesian. In general, the astrometry does not fail with a solution +close to the correct solution. I have found several examples of +images with extremely wrong astrometric solutions, with very large +non-Cartesian images on the sky. There are also a number of failures +near the pole with signficantly off astrometry, but close to an +accurate solution. The should now be rejected by {\pr gastro}. + +\section{Restarting the Pipeline} + +It is difficult to predict the future, so it is hard to give guidance +on solving future problems. Some of the things which might happen +would include having one of the machines which serve a disk go down or +having the disks fill up. If this happens, it is possible that the +run of a given night will have failed completely, or mostly. There +are a variety of things to do to recover from these situations. + +\subsection{Deleting data from the archive} + +The program {\pr delstar} lets the user delete a set of data from the +database. There are several options for this. The first example is: +{\pr delstar 199902040015b.cmp}. In this form, delstar uses the +astrometry information in the image header to find the measurements +and delete them from the database, along with the image from the image +database. This form requires the datafile *.cmp to exist in the local +directory. The second form, {\pr delstar 911012323 1}, uses the given +time in standard UNIX seconds form to identify the image in the image +database and delete both the image and the measurements. \note{does +delstar expect a UNIX seconds date, or can it take any of the +variations available in 'status'?} It is easy to associate time with +image with the status function {\pr findimages} or {\pr gimage} which +list images covering a specific location. + +The third form of {\pr delstar} is used to clean up the database if an +image is deleted but not all measurements from that image are removed. +The routine {\pr addstar} uses the astrometric information for an +image and the X,Y coordinates of each object in the image to locate +the object in the database. Several other routines do the reverse +process of finding the location of objects within the images. The +function which performs this association needs to find all database +*.cpt files which are covered by the image. The existing +implemenation is relatively quick, but not the most general possible. +In some rare cases, parts of an image will not be found. This is +normally not an issue since few functions need to find all stars in a +given image from scratch. However, if such an image is deleted, some +of the stars from that image may get left behind. These orphaned +stars can be deleted with the command {\pr delstar region.cpt -orphan +STUFF??}. + +\subsection{Deleting a night from the pipeline} + +If the analysis for a specific night fails miserably, it is probably +necessary to delete the entire night and then encourage the system to +redo that night. It is easy to delete an entire night with {\pr +delstar} by giving a time interval which includes (only) the night in +question. It is also easy to find the value of the time in seconds +with the {\pr status} function {\pr ctimes}. (At some point, {\pr +delstar} should incorporate the code from {\pr status} which does the +time format conversion automatically). This process will remove the +night in question from the database. But before the analysis can be +re-run, Ptolemy needs to forget that it already ran that night before. +Every night which has been analysed gets included in the log files +{\ff processed.log} and {\pr addstared.log}. The appropriate entry +from these files needs to be deleted. It is also necessary to delete +the files {\ff YYMMDD.addlist} and {\ff YYMMDD.outlist} from the log +directory. If the {\ff extracted.log} file exists, it should be +deleted as well. All of this is predicated on the assumption that the +user has halted all processing by Ptolemy before trying to fix +anything. At this point, it is possible to restart the analysis. +This can be done in several possible ways. If the night in question +has already been put on tape, it has to be extracted. The program +{\pr slurp} will automatically find data which has not been extracted +from the tape, download it, and set up the appropriate files. +Unfortunately, {\pr slurp} uses the file {\ff tape.log} which is not +automatically generated. It is therefore necessary to add a line to +{\ff tape.log} (in {\ff /metis/d11/logs}) for the night in question, +which should look just like the entry in the directory {\ff +pallas:/root/tape.log}. Instead of actually running {\pr slurp}, +however, the user should just run {\pr stage1 -once} which will +perform one run of {\pr stage1} and then end. If there is no pending +data (nothing in {\ff extracted.log} that has not been analysed), then +{\pr stage1} will automatically call {\pr slurp} in the correct +fashion (ie, on {\ff pallas}), run through the analysis, and then exit +without restarting. + +If the night's data has not been deleted yet, then it is not necessary +to run {\pr slurp} as the script {\pr lastnight} will do the job. The +cronscript for {\pr lastnight} uses the -today option so that the +automatic analysis will not reattempt any nights still on pallas. +Instead, it will be necessary for the user to run {\pr lastnight +-any}, which will also take care of running {\pr stage1}. This should be +safe at anytime, as the lock functions keep multiple instances of the +programs from running, but it is better to do this with enough time to +allow the process to finish before the next night's data is ready for +processing. + +If the pipeline has been interrupted, it will probably be necessary to +restart {\pr stage2} as well. Since {\pr stage2} always restarts +itself on exit, there is no cron job for it to make the initial start. +(This probably should be introduced into the /etc/rc.local file, or +some cron implementation for {\pr stage2} worked out). Note that both +{\pr stage1} and {\pr stage2} take no arguments except the optional +{\pr -once} flag. They make their own decisions about the correct log +file number and so forth. + +If the {\pr lastnight} cron job gets lost, there is a copy of the +necessary cron script {\ff +/metis/d11/src/ohana/src/perl/lastnight.cron}. This can be submitted +with {\pr crontab lastnight.cron}. + +\section{Data backup} + +Backups are the still rather primitive in implementation for the +LONEOS system. There are two things which should be backed up, but +only one currently has a good scheme. The database itself does not +have a backup process at the moment. This is a bad situation, but not +an easy one to solve. The database is very large, over 30 GB, which +is comparable to the space on a single AIC tape. At the moment, we +are satisfied with simply saving the intermediate results directories, +all of the {\ff YYMMDD.tgz} files. If the entire database is +destroyed, these files can be used to rebuild the database in a +relatively short time (currently, about a week). The script {\pr +backup.stuff} will backup a group of files. A good method is to +backup a month at a time. These will become a single tar file on the +AIC tape, and a month's worth of data is a reasonable amount of disk +space for future downloads. + +\end{document} + +At LONEOS, the images are +written to the disk from the camera by {\pr astrocam}, the camera +controller user interface. The resulting images are FITS files (which +may be arbitrarily sized) and have header keyword parameters RA and +DEC specifying rough celestial coordinates. Index: /trunk/Ohana/doc/pipeline/schematic.fig =================================================================== --- /trunk/Ohana/doc/pipeline/schematic.fig (revision 2508) +++ /trunk/Ohana/doc/pipeline/schematic.fig (revision 2508) @@ -0,0 +1,214 @@ +#FIG 3.2 +Landscape +Center +Inches +Letter +100.00 +Single +-2 +1200 2 +6 2400 375 3600 825 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 2400 375 3600 375 3600 825 2400 825 2400 375 +4 0 0 0 0 0 18 0.0000 4 150 975 2550 750 astrocam\001 +-6 +6 2400 2250 3600 2700 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 2400 2250 3600 2250 3600 2700 2400 2700 2400 2250 +4 0 0 0 0 0 18 0.0000 4 240 750 2550 2625 dophot\001 +-6 +6 2400 1125 3600 1875 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 3000 1500 600 375 3000 1500 3600 1875 +4 0 0 0 0 0 18 0.0000 4 180 540 2625 1575 *.fits\001 +-6 +6 2475 2925 3675 3675 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 3075 3300 600 375 3075 3300 3675 3675 +4 0 0 0 0 0 18 0.0000 4 240 540 2700 3375 *.obj\001 +-6 +6 2475 4650 3675 5400 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 3075 5025 600 375 3075 5025 3675 5400 +4 0 0 0 0 0 18 0.0000 4 240 660 2700 5100 *.cmp\001 +-6 +6 600 4800 1800 5250 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 600 4800 1800 4800 1800 5250 600 5250 600 4800 +4 0 0 0 0 0 18 0.0000 4 210 660 750 5175 gastro\001 +-6 +6 4425 4800 5625 5250 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4425 4800 5625 4800 5625 5250 4425 5250 4425 4800 +4 0 0 0 0 0 18 0.0000 4 180 780 4575 5175 addstar\001 +-6 +6 8775 4425 9975 4875 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 8775 4425 9975 4425 9975 4875 8775 4875 8775 4425 +4 0 0 0 0 0 18 0.0000 4 150 615 8925 4800 status\001 +-6 +6 525 2850 1875 4050 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 1200 3450 675 600 1200 3450 1875 4050 +4 0 0 0 0 0 18 0.0000 4 210 975 750 3525 Astro dB\001 +4 0 0 0 0 0 18 0.0000 4 225 690 825 3825 (HST)\001 +-6 +6 8475 1725 10125 2925 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 9300 2325 825 600 9300 2325 10125 2925 +4 0 0 0 0 0 18 0.0000 4 240 855 8850 2565 pictures\001 +4 0 0 0 0 0 18 0.0000 4 210 765 8850 2250 results,\001 +-6 +6 6300 1725 7950 2925 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 7125 2325 825 600 7125 2325 7950 2925 +4 0 0 0 0 0 18 0.0000 4 180 480 6900 2250 alert\001 +-6 +6 4200 750 5850 1950 +1 1 0 1 0 7 0 0 -1 4.000 1 0.0000 5025 1350 825 600 5025 1350 5850 1950 +4 0 0 0 0 0 18 0.0000 4 210 885 4575 1275 postage \001 +4 0 0 0 0 0 18 0.0000 4 210 750 4575 1590 stamps\001 +-6 +6 750 1275 1725 1650 +2 2 0 1 0 7 0 0 -1 4.000 0 0 -1 0 0 5 + 750 1275 1725 1275 1725 1650 750 1650 750 1275 +4 0 0 0 0 0 18 0.0000 4 180 690 825 1575 flatten\001 +-6 +6 2475 3900 3675 4350 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 2475 3900 3675 3900 3675 4350 2475 4350 2475 3900 +4 0 0 0 0 0 18 0.0000 4 180 465 2625 4275 fstat\001 +-6 +6 4425 6225 5625 6675 +6 4575 6375 5475 6600 +4 0 0 0 0 0 18 0.0000 4 180 900 4575 6600 addusno\001 +-6 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4425 6225 5625 6225 5625 6675 4425 6675 4425 6225 +-6 +6 4425 5550 5625 6000 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4425 5550 5625 5550 5625 6000 4425 6000 4425 5550 +4 0 0 0 0 0 18 0.0000 4 180 945 4575 5925 markstar\001 +-6 +6 4425 6900 5625 7350 +6 4425 6900 5625 7350 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4425 6900 5625 6900 5625 7350 4425 7350 4425 6900 +-6 +4 0 0 0 0 0 18 0.0000 4 180 1035 4575 7275 markrock\001 +-6 +6 4425 7575 5625 8025 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4425 7575 5625 7575 5625 8025 4425 8025 4425 7575 +4 0 0 0 0 0 18 0.0000 4 240 705 4575 7950 nrphot\001 +-6 +6 6225 4425 8025 7725 +6 6450 5925 7800 7125 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 7125 6525 675 600 7125 6525 7800 7125 +4 0 0 0 0 0 18 0.0000 4 210 810 6675 6600 Star dB\001 +-6 +6 6450 4650 7800 5850 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 7125 5250 675 600 7125 5250 7800 5850 +4 0 0 0 0 0 18 0.0000 4 240 1050 6675 5325 Image dB\001 +-6 +2 4 1 1 0 7 0 0 -1 4.000 0 0 7 0 0 5 + 8025 7725 8025 4425 6225 4425 6225 7725 8025 7725 +4 0 0 0 0 0 18 0.0000 4 210 1395 6375 7575 Photom dB\001 +-6 +6 1500 5700 2850 6900 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 2175 6300 675 600 2175 6300 2850 6900 +4 0 0 0 0 0 18 0.0000 4 210 1110 1650 6375 USNO dB\001 +-6 +6 1500 7200 2850 8400 +1 1 0 1 0 7 0 0 -1 0.000 1 0.0000 2175 7800 675 600 2175 7800 2850 8400 +4 0 0 0 0 0 18 0.0000 4 210 945 1725 7875 Rock dB\001 +-6 +2 2 1 1 0 7 0 0 -1 4.000 0 0 -1 0 0 5 + 2775 975 3150 975 3150 1050 2775 1050 2775 975 +2 2 1 1 0 7 0 0 -1 4.000 0 0 -1 0 0 5 + 2775 2025 3225 2025 3225 2100 2775 2100 2775 2025 +2 1 0 1 0 7 0 0 -1 4.000 0 0 -1 0 0 2 + 3075 3675 3075 3900 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3075 4350 3075 4650 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3000 2700 3000 2925 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3000 1875 3000 2250 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3000 825 3000 1125 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 2475 5025 1800 5025 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3675 5025 4425 5025 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 5625 5025 6225 5025 +2 1 3 1 0 7 0 0 -1 4.000 0 0 -1 0 1 3 + 0 0 1.00 60.00 120.00 + 2775 975 1350 975 1350 1275 +2 1 3 1 0 7 0 0 -1 4.000 0 0 -1 0 1 4 + 0 0 1.00 60.00 120.00 + 2775 2025 1425 2025 1350 2025 1350 1650 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 6450 3525 7650 3525 7650 3975 6450 3975 6450 3525 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 5625 5775 6225 5775 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 7050 3975 7050 4425 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3600 4875 4575 3150 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 3600 1575 4575 2850 +2 2 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 5 + 4575 2775 5775 2775 5775 3225 4575 3225 4575 2775 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 4950 3225 4950 4800 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 7050 3525 7050 2925 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 4950 2775 4950 1950 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 1 2 + 0 0 1.00 60.00 120.00 + 8775 4650 8025 4650 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 0 0 1 + 9375 4425 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 9300 4425 9300 2925 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 5625 6450 6225 6450 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 5625 7125 6225 7125 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 5625 7800 6225 7575 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 2850 7800 4425 7125 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 2850 6300 4425 6450 +2 1 0 1 0 7 0 0 -1 0.000 0 0 -1 1 0 2 + 0 0 1.00 60.00 120.00 + 1200 4050 1200 4800 +4 0 0 0 0 0 18 0.0000 4 180 465 6900 2625 files\001 +4 0 0 0 0 0 18 0.0000 4 225 810 6675 3825 (filters)\001 +4 0 0 0 0 0 18 0.0000 4 180 675 4800 3075 imdiff\001 Index: /trunk/Ohana/doc/pipeline/survey.fig =================================================================== --- /trunk/Ohana/doc/pipeline/survey.fig (revision 2508) +++ /trunk/Ohana/doc/pipeline/survey.fig (revision 2508) @@ -0,0 +1,16 @@ +#FIG 3.2 +Portrait +Center +Inches +Letter +100.00 +Single +-2 +1200 2 +2 5 0 1 0 -1 0 0 -1 0.000 0 0 -1 0 0 5 + 0 polar.ps + 975 823 9127 823 9127 9375 975 9375 975 823 +2 5 0 1 0 -1 0 0 -1 0.000 0 0 -1 0 0 5 + 0 fullsky.ps + 975 8273 9127 8273 9127 12225 975 12225 975 8273 +4 0 0 0 0 0 36 0.0000 4 375 6030 2250 1125 LONEOS SURVEY MAPS\001 Index: /trunk/Ohana/doc/projection.txt =================================================================== --- /trunk/Ohana/doc/projection.txt (revision 2508) +++ /trunk/Ohana/doc/projection.txt (revision 2508) @@ -0,0 +1,51 @@ + +/* +here is some C code to project from R,D to x,y (all in decimal +degrees) in a SIN projection, with Ro, Do as projection center: +*/ + +# define DEG_RAD 57.295779513082322 +# define RAD_DEG 0.017453292519943 + +project (double *x, double *y, double R, double D, double Ro, double Do) { + + double sdp, cdp, salp, calp, sdel, cdel, stht, sphi, cphi; + + sdp = sin(RAD_DEG*Do); + cdp = cos(RAD_DEG*Do); + salp = sin(RAD_DEG*(ra - Ro)); + calp = cos(RAD_DEG*(ra - Ro)); + sdel = sin(RAD_DEG*dec); + cdel = cos(RAD_DEG*dec); + + stht = sdel*sdp + cdel*cdp*calp; /* sin(theta) */ + sphi = cdel*salp; /* = cos(theta)*sin(phi) */ + cphi = cdel*sdp*calp - sdel*cdp; /* = cos(theta)*cos(phi) */ + if (stht < 0) { return 0; /* projection from the wrong side of the sphere */ } + + X = DEG_RAD * sphi; + Y = -DEG_RAD * cphi; + +# if (0) + + /* + these lines allow for a rotation / distortion 2x2 matrix (pci_j), + a (two direction) plate-scale shift (cdelt1, cdelt2), + and a reference center offset of Xo, Yo, if desired. + */ + + tmp_d = 1.0 / (pc_1_1*pc_2_2 - pc_1_2*pc_2_1); + *x = tmp_d * (pc_2_2*X - pc_1_2*Y) / cdelt1 + Xo; + *y = tmp_d * (pc_1_1*Y - pc_2_1*X) / cdelt2 + Yo; + +# else + + *x = X; + *y = Y; + +# endif + + + return (1); + +} Index: /trunk/Ohana/doc/status.txt =================================================================== --- /trunk/Ohana/doc/status.txt (revision 2508) +++ /trunk/Ohana/doc/status.txt (revision 2508) @@ -0,0 +1,72 @@ + +hi christian, + +The path you need depends on the machine type, either: + +/h/eugene/src/ohana/bin/linux or /h/eugene/src/ohana/bin/sol + +(the linux version is usually a little more up to date, but both +should work just fine for you). + +you also need to copy the following files to you home directory: + + ~dougados/.ptolemyrc + ~eugene/.statusrc + +The configuration scheme is much too complex to go into here, but +basically, the first defines the location of certain data, and in this +case, by choosing Catherine's .ptolemyrc file, you'll be looking at +the Taurus data by default, instead of the archive analysis I've been +running. + +The second file defines various macros for you, in particular +'ecliptic' and 'galactic', which draw the ecliptic and galactic +planes. + +start the program by running 'status'. It is command-line driven, and +has some limited help files. type '?' to see a list of commands and +'help' to get a list of help files. + +I think most things you'd want to do, you can do. In some cases the +syntax can be a little cumbersome. To get you started, here is a +macro to generate a box from (10,20) to (15,25), with an offset. +this could be more easily done by loading the reference coordinates +from a file with the 'data (filename)' and 'read' commands. I'll try +to help you figure things out when you have had a chance to play with +it a bit. + +gene + +the syntax for somr +macro skybox + create ra 0 2 + set ra = ra * 0 + set dec = ra * 0 + concat {10+$1} ra + concat {15+$1} ra + + concat {15+$1} ra + concat {15+$1} ra + + concat {15+$1} ra + concat {10+$1} ra + + concat {10+$1} ra + concat {10+$1} ra + + concat {20+$2} dec + concat {20+$2} dec + + concat {20+$2} dec + concat {25+$2} dec + + concat {25+$2} dec + concat {25+$2} dec + + concat {25+$2} dec + concat {20+$2} dec + + style -x 2 -c red -lw 3 -pt 100 + cplot ra dec +end + Index: /trunk/Ohana/doc/systest.c =================================================================== --- /trunk/Ohana/doc/systest.c (revision 2508) +++ /trunk/Ohana/doc/systest.c (revision 2508) @@ -0,0 +1,98 @@ +# include + +main (argc, argv) +int argc; +char **argv; +{ + + int i; + FILE *f; + + if (argc != 2) { + fprintf (stderr, "USAGE: %s testfile \n", argv[0]); + exit (0); + } + + fprintf (stderr, "char: %d bytes\n", sizeof (char)); + fprintf (stderr, "short: %d bytes\n", sizeof (short)); + fprintf (stderr, "int: %d bytes\n", sizeof (int)); + fprintf (stderr, "long: %d bytes\n", sizeof (long)); + fprintf (stderr, "float: %d bytes\n", sizeof (float)); + fprintf (stderr, "double: %d bytes\n", sizeof (double)); + + f = fopen (argv[1], "w"); + +# define TYPE char + { + TYPE I[20]; + + I[0] = 1; + for (i = 1; i < 20; i++) { + I[i] = I[i-1] * 2; + } + fprintf (f, "20 of type TYPE, %d bytes each\n", sizeof(TYPE)); + fwrite (I, sizeof(TYPE), 20, f); + } + +# define TYPE short + { + TYPE I[20]; + + I[0] = 1; + for (i = 1; i < 20; i++) { + I[i] = I[i-1] * 2; + } + fprintf (f, "20 of type TYPE, %d bytes each\n", sizeof(TYPE)); + fwrite (I, sizeof(TYPE), 20, f); + } + +# define TYPE int + { + TYPE I[20]; + + I[0] = 1; + for (i = 1; i < 20; i++) { + I[i] = I[i-1] * 2; + } + fprintf (f, "20 of type TYPE, %d bytes each\n", sizeof(TYPE)); + fwrite (I, sizeof(TYPE), 20, f); + } + +# define TYPE long + { + TYPE I[20]; + + I[0] = 1; + for (i = 1; i < 20; i++) { + I[i] = I[i-1] * 2; + } + fprintf (f, "20 of type TYPE, %d bytes each\n", sizeof(TYPE)); + fwrite (I, sizeof(TYPE), 20, f); + } + +# define TYPE float + { + TYPE I[20]; + + I[0] = 1; + for (i = 1; i < 20; i++) { + I[i] = I[i-1] * 2; + } + fprintf (f, "20 of type TYPE, %d bytes each\n", sizeof(TYPE)); + fwrite (I, sizeof(TYPE), 20, f); + } + +# define TYPE double + { + TYPE I[20]; + + I[0] = 1; + for (i = 1; i < 20; i++) { + I[i] = I[i-1] * 2; + } + fprintf (f, "20 of type TYPE, %d bytes each\n", sizeof(TYPE)); + fwrite (I, sizeof(TYPE), 20, f); + } + + +} Index: /trunk/Ohana/doc/thread.c =================================================================== --- /trunk/Ohana/doc/thread.c (revision 2508) +++ /trunk/Ohana/doc/thread.c (revision 2508) @@ -0,0 +1,36 @@ +# include +# include + +static int loop1, loop2; + +void *subloop (void *arg) { + + char *input; + + input = (char *) arg; + fprintf (stderr, "starting thread %s\n", input); + + while (loop1) { + fprintf (stderr, "loop2: %d\n", loop2); + usleep (300000); + } + pthread_exit (0); +} + +main () { + + int var; + pthread_t thread1; + pthread_attr_t thread_attr; + + loop1 = 1; + loop2 = 10; + pthread_create (&thread1, NULL, subloop, "test"); + + while (fscanf (stdin, "%d", &var) != EOF) { + loop2 = var; + if (loop2 == 0) loop1 = 0; + } +} + +// a comment Index: /trunk/Ohana/doc/wcs.txt =================================================================== --- /trunk/Ohana/doc/wcs.txt (revision 2508) +++ /trunk/Ohana/doc/wcs.txt (revision 2508) @@ -0,0 +1,54 @@ + +DETSEC - It looks like it is defining the real pixel coordinates of +each chip relative to the mosaic, including the gaps. This means it +is not affected by binning. For now, let's leave it off from +subrastering until we can calculate the correct coornates in that +case. + +CRPIX1,2 - these need to be adjusted to the effective coordinate of +the same pixel in the new coordinate frame: + +if you read a subraster (x1 - x2, y1 - y2) [unbinned pixels] with +binning dx, + +CRPIX1 = (CRPIX1' - x1) / dx +CRPIX2 = (CRPIX2' - y1) / dy + +CD1_1, CD1_2, CD2_1, CD2_2 just need to be rescaled: + +CD1_1' = CD1_1 / dx +CD2_1' = CD2_1 / dx + +CD1_2' = CD1_2 / dy +CD2_2' = CD2_2 / dy + +GAIN stays fixed (we are providing the sum of the binned pixels, not +the average). + +what about the terms PIXSIZEn, PIXSCALn (I don't know whether these +refer to physical pixels or image pixels. in the former case, they +stay constant, in the later case, they get scaled by dx, dy). + +As for the issue of the number of valid pixels... Well, I can see +arguments both ways. Perhaps we *should* stick with the old +definition of the valid data area. In that case, make it: + +X1 = 6 +X2 = (Nx + 1) / dx +Y1 = 5 +Y2 = (Ny + 4) / dy + +for DATASEC, which seems to match what we use now. + +you are right about the BIASSEC, we want the vertical region, not the corner. + +X1 = Nx / dx + 2 +X2 = Nx / dx + Sx +Y1 = 5 +Y2 = (Ny + 4) / dy + +(currently we don't use the full Y-direction overscan -- BIASSEC is +[6:2049,5:4100]). + +gene +