IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:IPP_Progress_Report_20110110

Version 9 (modified by bills, 15 years ago) ( diff )

--

IPP Progress Report for the week 2011.01.10 - 2011.01.14

    1. IPP Progress Report for the week 2011.01.10 - 2011.01.14
      1. Eugene Magnier
      2. Serge Chastel
      3. Heather Flewelling
      4. Roy Henderson
      5. Bill Sweeney
      6. Chris Waters

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

  • MOPS processing czar
  • I'm still fighting with mysql-5.1 on ippc00
    • Note: mysql-5.5 is not a development version but the stable version. 5.1 is already considered outdated by mysql.com. The latest GA version is precisely MySQL Community Server 5.5.8.
    • I successfully installed it from source and make it run. The configuration is a real pain. For info, the "right" call to configure is: 
      ./configure --sysconfdir=/etc/mysql --localstatedir=/var/lib/mysql --enable-profiling --enable-local-infile
            --with-mysqld-user=mysql --with-big-tables --with-plugins=partition,innodb_plugin,innobase
            --enable-thread-safe-client --with-unix-socket-path=/var/run/mysql/mysql.sock
      Note that it is installed in /usr/local (I tried with --prefix=/usr)
    • To make comparisons, I ingested a nebulous dump (from mysql-neb-ippdb00-2011-01-07T00:30:05.dump.gz). Ingestion took about 38 hours!
    • I ran OPTIMIZE TABLE on each of the table. Some results:
      OPTIMIZE TABLE directory -> 29.84 sec / 4409811 entries
      OPTIMIZE TABLE instance -> 4 hours 44 min 42.60 sec / 367254059 entries
      OPTIMIZE TABLE storage_object -> 3 days 7 hours 15 min 53.60 sec / 332149048 entries
      I was running another ingestion process of the same dump when storage_object was optimized. This might explain why it took longer but I don't understand why it took so long (I would have accepted a total of 38 hours + 5 hours (4h44 with a similar table in size)).
    • I'm working now on representative nebulous SQL statements to see if optimize changed anything. I'm also about to partition the tables.

Heather Flewelling

Roy Henderson

  • PSPS
    • discussions about, and small amount of research into, the design of a new DRL
    • loading continued, with some issues: batches with duplicate objIDs and dodgy flux values discovered by loader
    • PSVO
      • new classes, with common base-class, for slow and fast queries so that they can be handled polymorphically, meaning much cleaner code
      • PSVO can now receive messages via SAMP. Whole tables and table selections, including graphical selections on sky-plots in TOPCAT, can be passed back to PSVO for further query to PSPS
      • now remembers username/password combo to avoid logging-in every-time
      • began documentation page on PSPS system wiki
  • IPP
    • Czar wed/thurs. Minor issues
    • czartool
      • substantial time spent introducing a long-overdue XML format configuration file across the czartool code
      • brought documentation up-to-date

Bill Sweeney

  • worked on updating the destreak implementation to save excised pixels to a file.
  • worked on updating the postage stamp server to optionally restore excised pixels before making postage stamps (for authorized users)
  • two days as processing czar. Spent a lot of time hunting for space.
  • Monitored the STS data processing and kept it flowing.
  • Fixed some cleanup faults and other errors that are not apparent on czartool so we don't see them.
  • one day vacation

Chris Waters

  • Back from vacation, caught up on status.
  • Fixed bugs in new nightly science code:
    • race condition identified in burntool processing due to images that exist at the summit but are not downloaded returning a NULL instead of a valid class_id. Using the summitImfile.class_id value fixes this issue, ensuring proper ordering of images.
    • updated regtool -pendingburntoolimfile calls to use a book of dates instead of just calculating the current UTC date. This solves the issue of images from one day being ignored if they did not complete registration correctly by 14:00 HST.
    • updated the register_imfile.pl script to not bother burntooling images taken during the day to speed up their registration. This is controlled by two variables in the register.pro task: $sunset = 03:30 and $sunrise = 17:30. These should be generous enough to not cause problems.
    • identified bug that was likely preventing stacks and diffs from being generated (the script was not correctly checking if more processing was needed at end of night, and was claiming it was finished queuing stack and diff tasks). Fixed this, and determined that some diffs were being done out of order due to a quirk of the SQL dropping other incompletely processed data. Fixed this as well.
  • Looked at neb-cull, and determined that there was no bug in the code. neb-cull requires a volume, not a host name, which explains why the ROC script that was ran previously did not correctly cull the appropriate instance. Wrote correction script for the ROC files that moves the bad instances to the appropriate volume, and then corrects the ROC file to point at this new instance. Added roc -validate mode to read a ROC file, and check each input file and calculate if there are any data errors.
  • Fixed log(flux) scaling to generate images with the expected scaling.
Note: See TracWiki for help on using the wiki.