| | 1 | == 2014-03-19 == |
| | 2 | |
| | 3 | The newest concern is related to the speed of individual commands in the job bundle. Things that should not be a significant portion of the time request are running slower than expected, causing the job to fail due to taking too long. From the various time stamps and change stats: |
| | 4 | |
| | 5 | || try 1 || first directory timestamp || 2014-03-18 20:02:39 || |
| | 6 | || || directory 119 timestamp || 2014-03-18 20:14:18 || |
| | 7 | || || last directory (620) || 2014-03-18 20:40:50 || |
| | 8 | || try 2 || submit time || 2014-03-18 18:45:07 || |
| | 9 | || || complete time || 2014-03-18 20:59:53 || |
| | 10 | || || beginning date command || 2014-03-18 19:59:30 || |
| | 11 | || || first ppImage || 2014-03-19 02:00:51 || |
| | 12 | || || last ppImage || 2014-03-19 02:39:06 || |
| | 13 | || || first dbinfo || 2014-03-18 20:48:00 || |
| | 14 | || || last (119) dbinfo || 2014-03-18 20:59:00 || |
| | 15 | |
| | 16 | Based on these times (and shifting the ppImage times to a matching timezone), the time we would need to request for the job to complete successfully from scratch is far larger than expected: |
| | 17 | |
| | 18 | || step || time || notes || |
| | 19 | || mkdir || 26:32 || This took effectively zero time in the second try, as the directories already existed. || |
| | 20 | || ppImage || 38:15 || This is about 64% of the requested time, smaller than the 80% target value. || |
| | 21 | || dbinfo || 57:19 || This is a projection from the 119 that completed. || |
| | 22 | || total ||122:06 || || |
| | 23 | |
| | 24 | My suspicion is that writing to disk in a parallel fashion is far slower than expected, and that the jobs can pile up in a disk-wait state, making the parallel mkdir and echos overwhelming. As the writes for the ppImage stage are more randomized and less intense, this disk-wait isn't a problem. |
| | 25 | |
| | 26 | == 2014-03-17 == |
| | 27 | |
| | 28 | Due to the fact that ppImage doesn't make directories for output files, and a realization that the stats to database information step needs to be parallelized as well, I've reworked the command creation to operate in a pre/command/post split. For the chip stage, the pre commands are simply {{{mkdir -p}}} calls to ensure that all output directories exist. The command phase for chip is the detrend-calculated ppImage commands from before, and the post phase constructs dbinfo files. These files are created by an {{{echo -n}}} (to obtain information that is known when the command list is calculated, but is hard or unknowable on the remote cluster (database id values, etc), followed by a {{{ppStatsFromMetadata}}} that converts the stats file into the remaining database flags (execution time, object counts, background levels, etc). |
| | 29 | |
