http://misc.ipp.ifa.hawaii.edu/getsmf.php+ +When this page is accessed without parameters it simply displays an +error message. +With valid parameters the tool returns the contents of the selected +smf file. +
+ curl 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1780283' -o 1780283.smf + + wget 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1780283' -O 1780283-wget.smf ++ +The previous method allows the user to control the name of the local copy. +If one would like to use the original file name supplied by the server +the commands may be changed to: +
+ curl 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1780283' -O --remote-header-name + + wget 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1780283' --content-disposition + ++In this example the name of the downloaded file is "o7616g0008o.1131492.cm.1780283.smf". +
exp_name=o7616g0008o+gets the same result as the examples listed above. (As of the time +of this writing). +
exp_name=o5275g0401o&release=3PI.PV3retrieves the +smf processed for exposure o5275g0401o in 3PI.PV3. +If the release were changed to 3PI.nightly +the smf from the original nightly procecessing would be retrieved. +(At the time of this writing, if the release parameter were omitted +the 3PI.PV3 version would also be retrieved since it is currently +the latest processing.) +
+ curl 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=99999999999' -o test.smf ++After this command succeeds, the file test.smf is not a FITS file. +Instead it contains text. This fact and its contents may be found +with the following +commands. +
+ $ file test.smf + test.smf: ASCII text + $ cat test.smf + Could not find smf for cam_id: 99999999999. +Index: trunk/DataStoreServer/web/doc/smfs/index.html =================================================================== --- trunk/DataStoreServer/web/doc/smfs/index.html (revision 39660) +++ trunk/DataStoreServer/web/doc/smfs/index.html (revision 39660) @@ -0,0 +1,29 @@ + + +
+This page contains links to information about accessing +IPP SMF files +which contain the results of IPP camera stage processing. +
+These files are FITS format files containing multiple binary table extension +sections. These which contain the photometric and astronometric measurements +for the sources detected in a single Pan-STARRS exposure. +
+These files are the inital inputs to the PS calibration process. +
+The following links describe how to use the IPP-MISC SMF Access tools +to discover and retrive SMF files. +
+ + Index: trunk/DataStoreServer/web/doc/smfs/listsmfs.html =================================================================== --- trunk/DataStoreServer/web/doc/smfs/listsmfs.html (revision 39660) +++ trunk/DataStoreServer/web/doc/smfs/listsmfs.html (revision 39660) @@ -0,0 +1,141 @@ + + +
+listsmf.php is a tool that may be used to retrieve certain information
+about SMF files from the IPP database.
+
The url of the tool is
+http://misc.ipp.ifa.hawaii.edu/listsmfs.php
+
+
+The list returned is controlled by parameters supplied +as part of the http request used to access the page. + +By default listsmfs displays the list of +smfs corresponding to the PS1 science exposures collected on +the current date (UTC). The output may be fine tuned using parameters. +The use of parameters is discussed below. +
+Here is some sample output. +
+#cam_id smf_name quality state exp_name label release filter obs_date obs_time comment +1783121 o7622g0062o.1134677.cm.1783121.smf 0 full o7622g0062o OSS.nightlyscience SSS.nightly i.00000 2016-08-22 07:43:12 'OSSR.R19S6.16.Q.i ps1_32_2599 visit 1' +1783120 o7622g0061o.1134678.cm.1783120.smf 0 full o7622g0061o OSS.nightlyscience SSS.nightly i.00000 2016-08-22 07:42:16 'OSSR.R19S6.16.Q.i ps1_32_2493 visit 1' +1783122 o7622g0063o.1134679.cm.1783122.smf 0 full o7622g0063o OSS.nightlyscience SSS.nightly i.00000 2016-08-22 07:44:08 'OSSR.R19S6.16.Q.i ps1_32_2614 visit 1' +... and so on. ++ +
+The first line, which begins with the "comment" character '#', is a +header line. This line lists the names of the columns shown +in the subsequent lines. +
+If no exposures are found that match the supplied parameters (and their +defaults) only the header line is displayed. +
The first 4 columns are the most important. They are very useful +for constructing requests to retrieve individual SMFs using +getsmf.php. + +
+For example the sample output shown above was generated by a request +with the parameter string '?release=SSS.nightly' appended to the URL +(on the date that this page was written of course). +
+If multiple parameters are to be supplied they should be separated +by a '&' character. For example '?release=SSS.nightly&filter=r' +would list exposures with the named release in the r band filter.. +
+Here is the list of the parameters. + + +
YYYY-MM-DDTHH:MM:SS+where the '-', 'T', and ':' characters are literal and the other characters are +numeric values for year, month, day, hours, minutes, and seconds. +
+For example 'date_min=2016-08-22T08:00:00&date_max=2016-08-22T09:00:00' +would select exposures observed between 0800 UTC and before 0900 UTC +on Augst 22. +
+Notes on date limits.
+The time portion may be omitted. For example the parameter pair +'date_min=2016-08-21&date_max=2016-08-22' will cause all exposures observed +on 2016-08-21 or 2016-08-22 to be listed. +
+If date_min is not supplied, any date_max value supplied is silently ignored. +Instead page uses the defaults for the date parameter: exposures for the current date are listed (if any). + +
+The process of retrieving SMF files is straightforward. +
+This page is primarily concerned with the first two items on this list. + +
+The two pages listsmfs.php and getsmf.php have a relatively straightforward +interface. UNIX command line tools such as curl or wget may be used to +access the interface. + +
+The perl script below gives an example of how to use curl to get +a list of the current day's smf files from listsmf.php and print curl +commands that may be used to download the files. + +
+ #!/usr/bin/env perl
+
+ use strict;
+ use warnings;
+
+ my $verbose = 0;
+
+ my $misc_url = 'http://misc.ipp.ifa.hawaii.edu';
+ my $list_url = "$misc_url/listsmfs.php";
+ my $get_url = "$misc_url/getsmf.php";
+
+ my $silent = ' --silent';
+ if ($verbose) {
+ $silent = '';
+ }
+
+ my $list_cmd = "curl $silent http://misc.ipp.ifa.hawaii.edu/listsmfs.php";
+
+ my $output = `$list_cmd`;
+
+ if ($? != 0) {
+ my $rc = $?;
+ print STDERR "Ooops something went wrong with the command $list_cmd\n";
+ print STDERR "command returned $rc\n";
+ exit $rc >> 8;
+ }
+
+ my @lines = split "\n", $output;
+
+ foreach my $line (@lines) {
+ # skip the header line and any others that begin with '#'
+ next if $line =~ '#';
+
+ # split the line at the single quote character that preceeds the
+ # comment string
+ my ($left, $right) = split "'", $line;
+
+ # the right side of this split is the comment string ...
+ my $comment = $right;
+
+ # ... and the left side of this split contains a string
+ # with the other parameters.
+ # The other parameters are separated by whitespace
+
+ # NOTE: One might have thought we could parse the header string
+ # and find the column names but that implies more than the
+ # listsmf "interface" commits to at this time.
+
+ my ($cam_id, $smf_name, $quality, $state, $exp_name, $label, $release, $filter, $obs_date, $obs_time) = split " ", $left;
+
+ if ($quality != 0) {
+ print STDERR "camRun $cam_id had bad quality $quality\n";
+ next;
+ }
+
+ if ($state ne 'full') {
+ print STDERR "camRun is in state $state\n";
+ next;
+ }
+
+ # generate and print out a curl command that could be used to
+ # download the smf file to the current directory
+ my $this_cmd = "curl $silent '$get_url?cam_id=$cam_id' --output $smf_name";
+
+ print "$this_cmd\n";
+ }
+ # all done
+
+
+When executed the resulting output will look something like this:
++ # 1783100 o7622g0010o.1134626.cm.1783100.smf 0 ICECUBE-160731A.01 i band dither 1 visit 1 + curl --silent 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1783100' --output o7622g0010o.1134626.cm.1783100.smf + # 1783101 o7622g0011o.1134627.cm.1783101.smf 0 ICECUBE-160731A.01 i band dither 2 visit 1 + curl --silent 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1783101' --output o7622g0011o.1134627.cm.1783101.smf + # 1783102 o7622g0012o.1134628.cm.1783102.smf 0 ICECUBE-160731A.01 i band dither 3 visit 1 + curl --silent 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1783102' --output o7622g0012o.1134628.cm.1783102.smf + # 1783103 o7622g0013o.1134629.cm.1783103.smf 0 ICECUBE-160731A.01 i band dither 4 visit 1 + curl --silent 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1783103' --output o7622g0013o.1134629.cm.1783103.smf + # 1783104 o7622g0014o.1134630.cm.1783104.smf 0 ICECUBE-160731A.01 i band dither 5 visit 1 + curl --silent 'http://misc.ipp.ifa.hawaii.edu/getsmf.php?cam_id=1783104' --output o7622g0014o.1134630.cm.1783104.smf ++ +
+There are two lines for each exposure. The first (which begins with + a '#' character) lists prints selected metadata +for the smf and its source exposure. The second line gives +a curl command that would copy the file to the user's current directory. +
Note, while not suggested, one could take the output of listsmfs.php +and source it into a UNIX shell to execute the commands. +
+This script demonstrates how to +
+This simple script shows the most important steps necessary for successful +smf retrieval. + +
+For the sake of speed we chose to not include any information that +might be useful for detecting errors in the download process. The database +does contain anything (a checksum for example) +which could be used to gain confidence that a file was transmitted +successfully. + +
+In practice so far this has not shown to be an issue. getsmf.php has been +used to download hundreds of thousands of smf files. When users have +detected problem files, simply trying the download again seems to correct +the problem.