| | 7 | |
| | 8 | This has been a week of fighting fires and trying to understand problems. |
| | 9 | |
| | 10 | The MHPCC->ATRC copy is finally moving along smoothly. After further iterations with Chris on the neb-admin search sql, the system is now reliably picking up all of the chips across the cluster and correctly identifying the ones that need to be replicated. We also tuned the parameters for this pantasks so that the pipe stays full. Since about Wed of last week, we have been able to keep up ~120MB/sec nearly continuously. At this rate, we will fill the ATRC machines in 3-4 weeks (just in time for another box to arrive). |
| | 11 | |
| | 12 | A related issue: we had been having an unusual number of machine crashes over the past 2-3 weeks. This turned out to be caused by an error in the pantasks config for the shuffle to ATRC: the code was not correctly telling pantasks which machines would be used for the process. Uneven distribution of the source chips meant that individual source machines at MHPCC could be heavily overloaded (50 simultaneous copies). Fixing this configuration error has stopped the machine crashes plaguing us for the past few weeks. |
| | 13 | |
| | 14 | We have also been having trouble keeping up with the data each night. Until May we had been finishing nightly processing soon after the end of the night. Since May, on nights when we actually got data, we were taking most of the day to finish the processing. This problem had been masked by the weather (lack of data) and the extra LAP processing data (a likely scapegoat). Last week, we had lots of data and the problem was clearly in evidence. We limited or stopped LAP processing, but still we were clearly running slower than previously. Heather looked into timings and discovered that chip processing had gotten slower since May, but mostly in the supporting perl script. I discovered that the perl script, trying to extra a recipe value from the camera configuration, was taking an unreasonably long time to parse the metadata files. I hardcoded for one night the specific value, and the processing rate went up significantly. I will adapt the system to do the parsing in the ppConfigDump c code. |
| | 15 | |
| | 16 | The final big concern on the regular processing is the storage. Looking at the historical logs, it seems that we are filling up the disks more quickly than we expect. If we have no LAP processing running, we should be adding to our storage mostly by adding raw data, at a rate of about 35-40TB per month (assuming the processing buffer is roughly full). If LAP processing is running, then we also add by the amount of static sky we have generated. Going back over 6 months, however, it seems we are increasing the storage at close to 2x the raw data rate even when we are not running LAP processing. This is becoming critical as we can only shuffle data to ATRC at a finite rate. We are still investigating this particular problem. |
| | 17 | |
| | 18 | I also have been working on the slow parts of the extended source analysis, but have only been able to make modest progress given these other issues. |
