IPP Progress Report for the week 2010.11.29 - 2010.12.03

(Up to IPP Progress Reports)

Eugene Magnier

I spent much of the week preparing for the board meeting. On friday, with no new data from the summit, we spent much of the day running an interesting experiment to judge the system throughput and bottlenecks. We queued up a bunch of exposures for processing, and only allowed the chip stage to run. We started with 12 active node connections, and every so often increased the number of available nodes. As the number of nodes went from 12 to 102, there was a nearly linear growth in the number of jobs processed per unit time. At 180, there was some degradation, and by 231, we were basically saturated. At that point, adding more nodes was not increasing the total processed exposures. There were three likely limiting factors: 1) the nebulous mysql, 2) the processing mysql, 3) pantasks. It turns out that an artificial limit in pantasks was the initial culprit: there is a limit to the number of jobs which request a specific machine (even if they revert to another). Increasing this limit increased the processing rate so that, with 273 nodes, we were able to process more jobs per node than in the first pass with 231 nodes. At this point, we were probably limited by the nebulous mysql throughput, but we ended the experiment for they day before exploring any further.

Serge Chastel

  • OC133.OSS data preparation for copy to MOPS data. Copy started on Friday;
  • Tried to understand how to change version of ppMops in IPP;
  • Tessellation: trying to find time to finish it;
  • Various meetings (Ken, IPP, MOPS);
  • Helped Heather with perl problems.
  • MOPS czar (and therefore IPP crypto-czar at least to speed up OSS data processing);

Heather Flewelling

  • boston all week (flew back all day friday)
  • installed ipp on harvard's cf cluster (and have accounts on cf and odyssey), with the intent to copy the dvo db to harvard once they find space
  • talked to pavlos about videophot - started a branch and started coding this (he needs to provide the algorithm)
  • talked to matt about various things (including latency of oss publication)
  • attended meetings with harvard panstarrs people and harvard transients people - they want better diff photometry

Roy Henderson

Vacation all week.

Bill Sweeney

  • Moved postage stamp server off of ippdb02 to its new (and hopefully permanent home) ippc17. First did a trial transfer Found many hundreds of GB old crufty data. Doing the transition and cleaning that stuff up took the better part of Monday. On Tuesday the actual transition appeared to go smoothly. Unfortunately one piece of the configuration was not updated so distribution posted file sets to the old database. Publishing and postage stamp server are configured differently so they worked fine. Since we had bad weather nobody noticed that nothing was getting distributed. Finally on Monday Dec 6 Mark Huber reported the problem. Fixed this by updating the incorrect table, setting faults on the incorrectly posted file sets and letting revert do the registration over again. All fixed within 15 minutes.
  • Investigated why some user's postage stamps at the position of a supernova were masked. It turned out that the coordinates were just outside the field of view of the exposures. Since they requested warp stamps they got masked postage stamps.
  • Made some refinements to the code to "replicate outputs". We will turn this on next time we update the production tag.
  • Investigated why the fraction of pixels masked by magic was higher than expected. It turns out something changed in the ipp since the last time tests were run that made things worse. Further investigation is needed to track down the cause.
  • Made changes to the database and scripts to store the resutls of the psphot detection efficiency analysis in the chip and camera database tables.
  • Tested detection efficiency changes using gpc1 data and simtest. Fixed some latent bugs revealed by simtest.
  • Selected exposures for MD03 reference stack. Started the processing. Something has changed in the software or the cluster. ppStack required lots of memory for many skycells. So much so that we had some systems crash. Worked around this by continuing the stack processing in a special pantasks instance which only uses 32GB nodes. This is slow going though.
  • Spent Friday and some time on the weekend as processing czar.

Chris Waters

  • Detrends: Reviewed the new darks, and checked that linearity was working correctly. This seems to be the case, so linearity development is now finished. The new darks look much better, although there are some residual dark signal in a few OTAs (ota15, ota45, ota43).
  • Burntool: Coding on registration for the realtime burntool is finished, and still needs to be tested.
  • Throughput: Looked at the apache/mysql logs for nebulous to see if we are overloading the database server. We should be able to easily handle 5e5 requests per hour, which should be more than enough to handle processing (translates to ~1400 exposures a day).