IPP Progress Report for the week 2011.01.10 - 2011.01.14

(Up to IPP Progress Reports)

Eugene Magnier

I have spent most of the past week (and more) working on the elevated magic masking. Exploration by Heather showed that the main culprit was NOT the covariance change from Paul before 8/23. It seems that one important culprit was a problem with the background modeling code. The details of this error are still unclear, but I have fixed the problem in my development branch.

Another important cause of the elevated magic masking results from the post-8/23 modification of the PSF matching code. The PSF matching code improvements were motivated by the tendency of the PSF matching to leave behind donuts and dipoles. These were both examples of the PSF-matching not actually matching very well -- in these cases, the solutions tended to under fit the difference (in other words, the PSF-matched images were not convolved by enough of a kernel to modify the PSF enough). The work over the Fall (updates in 8/23 and 10/29) introduced modifications to the dual convolution to achieve better solutions, with good success to that goal. However, my investigations last week showed that these solutions could also over fit the PSF differences: they allowed potentially so many free parameters that the solutions could add excess noise to the images. At a suggestion from John Tonry, I have modified the code to require higher-order fits (from the dual convolution and spatial variations) only be used if they were justified by improvements in the chi-square (this was actually trickier than it seems -- larger kernels also smooth the image, reducing the effective chisq if the impact of that smoothing on the variance is not included). I tested this code over a wide range of simulated images which either should require or not require dual and/or spatial variations, and found that the code did a good job of choosing the expected set of fit parameters.

I am still running tests with real GPC1 data before declaring victory on this.

Serge Chastel

  • MOPS processing czar
  • I'm still fighting with mysql-5.1 on ippc00
    • Note: mysql-5.5 is not a development version but the stable version. 5.1 is already considered outdated by mysql.com. The latest GA version is precisely MySQL Community Server 5.5.8.
    • I successfully installed it from source and make it run. The configuration is a real pain. For info, the "right" call to configure is:
      ./configure --sysconfdir=/etc/mysql --localstatedir=/var/lib/mysql --enable-profiling --enable-local-infile
                  --with-mysqld-user=mysql --with-big-tables --with-plugins=partition,innodb_plugin,innobase
                  --enable-thread-safe-client --with-unix-socket-path=/var/run/mysql/mysql.sock
      Note that it is installed in /usr/local (I tried with --prefix=/usr)
    • To make comparisons, I ingested a nebulous dump (from mysql-neb-ippdb00-2011-01-07T00:30:05.dump.gz). Ingestion took about 38 hours!
    • I ran OPTIMIZE TABLE on each of the table. Some results:
      OPTIMIZE TABLE directory -> 29.84 sec / 4409811 entries
      OPTIMIZE TABLE instance -> 4 hours 44 min 42.60 sec / 367254059 entries
      OPTIMIZE TABLE storage_object -> 3 days 7 hours 15 min 53.60 sec / 332149048 entries
      I was running another ingestion process of the same dump when storage_object was optimized. This might explain why it took longer but I don't understand why it took so long (I would have accepted a total of 38 hours + 5 hours (4h44 with a similar table in size)).
    • I'm working now on representative nebulous SQL statements to see if optimize changed anything. I'm also about to partition the tables.

Heather Flewelling

Roy Henderson

  • PSPS
    • discussions about, and small amount of research into, the design of a new DRL
    • loading continued, with some issues: batches with duplicate objIDs and dodgy flux values discovered by loader
    • PSVO
      • new classes, with common base-class, for slow and fast queries so that they can be handled polymorphically, meaning much cleaner code
      • PSVO can now receive messages via SAMP. Whole tables and table selections, including graphical selections on sky-plots in TOPCAT, can be passed back to PSVO for further query to PSPS
      • now remembers username/password combo to avoid logging-in every-time
      • began documentation page on PSPS system wiki
  • IPP
    • Czar wed/thurs. Minor issues
    • czartool
      • substantial time spent introducing a long-overdue XML format configuration file across the czartool code
      • brought documentation up-to-date

Bill Sweeney

  • worked on updating the destreak implementation to save excised pixels to a file.
  • worked on updating the postage stamp server to optionally restore excised pixels before making postage stamps (for authorized users)
  • two days as processing czar. Spent a lot of time hunting for space.
  • Monitored the STS data processing and kept it flowing.
  • Fixed some cleanup faults and other errors that are not apparent on czartool so we don't see them.
  • one day vacation

Chris Waters

  • Back from vacation, caught up on status.
  • Fixed bugs in new nightly science code:
    • race condition identified in burntool processing due to images that exist at the summit but are not downloaded returning a NULL instead of a valid class_id. Using the summitImfile.class_id value fixes this issue, ensuring proper ordering of images.
    • updated regtool -pendingburntoolimfile calls to use a book of dates instead of just calculating the current UTC date. This solves the issue of images from one day being ignored if they did not complete registration correctly by 14:00 HST.
    • updated the register_imfile.pl script to not bother burntooling images taken during the day to speed up their registration. This is controlled by two variables in the register.pro task: $sunset = 03:30 and $sunrise = 17:30. These should be generous enough to not cause problems.
    • identified bug that was likely preventing stacks and diffs from being generated (the script was not correctly checking if more processing was needed at end of night, and was claiming it was finished queuing stack and diff tasks). Fixed this, and determined that some diffs were being done out of order due to a quirk of the SQL dropping other incompletely processed data. Fixed this as well.
  • Looked at neb-cull, and determined that there was no bug in the code. neb-cull requires a volume, not a host name, which explains why the ROC script that was ran previously did not correctly cull the appropriate instance. Wrote correction script for the ROC files that moves the bad instances to the appropriate volume, and then corrects the ROC file to point at this new instance. Added roc -validate mode to read a ROC file, and check each input file and calculate if there are any data errors.
  • Fixed log(flux) scaling to generate images with the expected scaling.