Changeset 41242 for trunk/doc/release.2015/ps1.datasystem/response.txt
- Timestamp:
- Jan 28, 2020, 3:10:56 PM (6 years ago)
- File:
-
- 1 edited
-
trunk/doc/release.2015/ps1.datasystem/response.txt (modified) (1 diff)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/response.txt
r41235 r41242 1 2 ** We are resubmitting our article "The Pan-STARRS Data Processing 3 System" after addressing suggestions raised by the referee. We 4 thank the referee for detailed comments and suggestions. Below are 5 our responses to the referee's suggestions. (Our responses are 6 preceeded by "**") 1 We are resubmitting our article "The Pan-STARRS Data Processing System" after addressing suggestions raised by the referee. We thank the referee for detailed comments and suggestions. Below are our responses to the referee's suggestions. (Our responses are preceeded 2 by "**") 7 3 8 4 # General Notes 9 5 10 This is a well-written and important technical paper that succeeds 11 admirably at what I consider the most important goal of any pipeline 12 paper: providing a decription of the processing steps that are 13 relevant for downstream science users (in this case, by providing the 14 big picture that ties together a number of more detailed papers). With 15 only a handful of minor cleanups (see detailed notes below), I think 16 the paper is ready for publication, and most of my comments represent 17 ideas for improvement that I hope the authors will consider (but 18 should not feel obliged to act on). 6 This is a well-written and important technical paper that succeeds admirably at what I consider the most important goal of any pipeline paper: providing a decription of the processing steps that are relevant for downstream science users (in this case, by providing the big picture that ties together a number of more detailed papers). With only a handful of minor cleanups (see detailed notes below), I think the paper is ready for publication, and most of my comments represent ideas for improvement that I hope the authors will consider (but should not feel obliged to act on). 19 7 20 My only general concern is that the paper often misses the opportunity 21 to pass on lessons learned to the developers of future pipelines, and 22 this makes much of the detailed description of how the PS1 systems 23 work (particularly in Section 5) feel like it belongs more in operator 24 documentation rather than an article like this one. I suspect a small 25 amounof additional historical context - how different systems evolved 26 over the course of the survey - and commentary on what worked well and 27 what was a regular pain point would go a long way. 8 My only general concern is that the paper often misses the opportunity to pass on lessons learned to the developers of future pipelines, and this makes much of the detailed description of how the PS1 systems work (particularly in Section 5) feel like it belongs more in operator documentation rather than an article like this one. I suspect a small amounof additional historical context - how different systems evolved over the course of the survey - and commentary on what worked well and what was a regular pain point would go a long way. 28 9 29 In particular, the described system seems to involve a both fair 30 amount of duplication (e.g. multiple databases, sky-tiling systems, 31 and task orchestration layers) and a number of in-house solutions to 32 what seem like fairly general problems (the DVO database and 33 especially the pantask/opihi system stand out in this regard). This is 34 not intended as criticism; I am quite aware that there are many good 35 reasons for both duplication and keeping central components in-house, 36 from deliberately keeping components loosely coupled to taking into 37 account the often-brief shelf-life of off-the-shelf solutions, 38 especially as compared to the duration of a major astronomical 39 survey. But describing *which* of many potential reasons actually 40 played a role in each of various design choices (and which, if any, 41 look less good in hindsight) would make the paper much more 42 interesting. 10 In particular, the described system seems to involve a both fair amount of duplication (e.g. multiple databases, sky-tiling systems, and task orchestration layers) and a number of in-house solutions to what seem like fairly general problems (the DVO database and especially the pantask/opihi system stand out in this regard). This is not intended as criticism; I am quite aware that there are many good reasons for both duplication and keeping central components in-house, from deliberately keeping components loosely coupled to taking into account the often-brief shelf-life of off-the-shelf solutions, especially as compared to the duration of a major astronomical survey. But describing *which* of many potential reasons actually played a role in each of various design choices (and which, if any, look less good in hindsight) would make the paper much more interesting. 43 11 44 ** We have greatly expanded the Conclusion to address these questions, 45 and to identify choices we made which either turned out well or 46 which we would have done differently given changes to the software 47 landscape. 12 ** We have greatly expanded the Conclusion to address these questions, and to identify choices we made which either turned out well or which we would have done differently given changes to the software landscape. 48 13 49 14 # Detailed Notes 50 15 51 16 ## Section 2.4 17 18 Is the Distribution and Publication system mentioned in the text supposed to be part of Figure 1, either as an umbrella term or a (missing?) component? 52 19 53 Is the Distribution and Publication system mentioned in the text 54 supposed to be part of Figure 1, either as an umbrella term or a 55 (missing?) component? 56 57 ** We have adjusted this figure to put the publication "customers" at 58 the bottom and added a line to show where the distribution and 59 publication mechanisms interface to these customers. 20 ** We have adjusted this figure to put the publication "customers" at the bottom and added a line to show where the distribution and publication mechanisms interface to these customers. 60 21 61 22 ## Section 3.1 62 23 63 This is by no means necessary, but I'm curious to see a table or 64 discussion of what fraction of jobs of various types failed with bad 65 "quality". In other words, how much data could you not get through the 66 pipelines at all, and what was the most sensitive step? 24 This is by no means necessary, but I'm curious to see a table or discussion of what fraction of jobs of various types failed with bad "quality". In other words, how much data could you not get through the pipelines at all, and what was the most sensitive step? 67 25 68 ** We liked this suggestion and added a subsection 3.12 and a new 69 table (2) to discuss the failure rates. 26 ** We liked this suggestion and added a subsection 3.12 and a new table (2) to discuss the failure rates. 70 27 71 28 ## Section 3.3 72 29 73 Running "Registration" only once for each exposure would seem to 74 prohibit re-running "burntool" after updating the algorithm for that - 75 and I'm guessing you didn't get that fully stabilized until after you 76 had already processed some images and learned from thr experience. How 77 did that work? 30 Running "Registration" only once for each exposure would seem to prohibit re-running "burntool" after updating the algorithm for that - and I'm guessing you didn't get that fully stabilized until after you had already processed some images and learned from thr experience. How did that work? 78 31 79 ** We added a couple of sentences to explain that we used a 80 semi-manual task to re-run just the burntool analysis during 81 development and if the code ever needs to be changed. 32 ** We added a couple of sentences to explain that we used a semi-manual task to re-run just the burntool analysis during development and if the code ever needs to be changed. 82 33 83 ## Section 3.5 34 ## Section 3.5 35 36 Why a 3rd-order polynomial from chip to focal plane? Wouldn't an affine transform have been sufficient (and more than that degenerate with the focal plane to sky transform)? 84 37 85 Why a 3rd-order polynomial from chip to focal plane? Wouldn't an 86 affine transform have been sufficient (and more than that degenerate 87 with the focal plane to sky transform)? 38 ** We use the higher-order transformation for each chip to capture the small-scale astrometric signal present in the data. One could use an afine transformation for chip-to-focal plane and capture the same signal in a much higher-order model for focal-plane to sky, but that was not our development path. These would be equivalent solutions. (Note that degeneracies exist in both cases). We avoid the degeneracy of the chip positions in the focal plane solution by fitting the local gradient to get the initial distortion solution (and there are certain terms which are held fixed for the focal plane.) We then limit the impact of the degeneracy by fitting the two levels independently and fixing the focal-plane solution after a few iterations. 88 39 89 ** We use the higher-order transformation for each chip to capture the 90 small-scale astrometric signal present in the data. One could use 91 an afine transformation for chip-to-focal plane and capture the 92 same signal in a much higher-order model for focal-plane to sky, 93 but that was not our development path. These would be equivalent 94 solutions. (Note that degeneracies exist in both cases). We 95 avoid the degeneracy of the chip positions in the focal plane 96 solution by fitting the local gradient to get the initial 97 distortion solution (and there are certain terms which are held 98 fixed for the focal plane.) We then limit the impact of the 99 degeneracy by fitting the two levels independently and fixing the 100 focal-plane solution after a few iterations. 40 ** We have added some words to explain some of this, but leave the details to Paper IV. 101 41 102 We have added some words to explain some of this, but leave the 103 details to Paper IV. 42 What makes the masks generated in this step "dynamic"? Are they generated wholly from the reference catalog (i.e. predicting where a ghost will appear based on the position of a bright star)? It seems like the CAMERA step does not utilize any of the pixel data (just the pixel-level masks from CHIP). Is that correct? 104 43 105 What makes the masks generated in this step "dynamic"? Are they 106 generated wholly from the reference catalog (i.e. predicting where a 107 ghost will appear based on the position of a bright star)? It seems 108 like the CAMERA step does not utilize any of the pixel data (just the 109 pixel-level masks from CHIP). Is that correct? 44 ** correct: the dynamic masks are generated from the reference catalog and do not go back to the original pixels. We added a paragraph to clarify. 110 45 111 ** correct: the dynamic masks are generated from the reference 112 catalog and do not go back to the original pixels. We added a 113 paragraph to clarify. 46 ## Section 3.8 47 48 Is the selection of which warped images go into a stack driven by human operators, or are there automated systems to launch these jobs, too? 49 50 ** section 5.2 discusses how both the nightly stacks and large-scale reprocessing campaign stacks are automatically defined. We added some words to refer to this section in 3.8. 51 52 ## Section 3.10 53 54 How much of the PSF-convolved galaxy models do you re-fit in forced photometry? If you're fitting more than just the amplitude at that stage, and considering each exposure as independent, you're potentially throwing away a lot of S/N (at least in the many-exposure limit), even if you average later. If you're just fitting the amplitude, the structural parameters are still going to be the ones affected by poor PSFs in the stack. 55 56 ** the galaxy models are not fitted on each warp. rather we calculate the normalizations and chi-square values for a grid of galaxy model shape parameters for each warp image. The values for each grid point are combined across all warps to generate a total stack-equivalent grid. At this point, the best parameters are determined from the grid (interpolating to the chi-square minimum). This is mathematically equivalent to simultaneously fitting (via a grid search) the pixels from all warps to a single model, preserving the full signal-to-noise. We have updated the text to add some detail to the description of what is being measured to clarify this point. 57 58 ## Section 3.11 59 60 transient source -> transient sources 61 62 ** fixed. 63 64 ## Section 4.1.3 65 66 I was confused when first encountering the word "files" here because up to this point I had been thinking of the DVO as just another MySQL (or other SQL-ish) database, and I wasn't sure what kind of files were being referred to. I think it'd be helpful to briefly describe the overall architecture of the DVO as (mostly?) spatially sharded files at the beginning of section 4, even if the details of the partitioning aren't described until 4.1.3. 67 68 ** we added a sentence to 4.1.1 to note this point. 69 70 Missing punctuation in parenthetical HST GSC reference? 71 72 ** fixed 73 74 ## Section 4.1.4 75 76 There's some inconsistency here between "detID" and "det_id" (same for "image"), both referring to measurement IDs in DVO. If those are supposed to be meaningfully different, I'm confused. 77 78 ** in the DVO section (and in the DVO schema), these should all be 'detID' and 'imageID'. In the gpc1 database schema, the underscored versions are used. we have fixed the erroneous det_id and image_id entries in this section. 79 80 ## Section 4.2 81 82 I tend to associate the term "ubercal" specifically with the SDSS version of the algorithm that coined the term, and think it probably should be referenced here even if the actual algorithms used are only vaguely similar. 83 84 ** we agree and have added a sentence with reference. 85 86 ## Section 4.3 87 88 Is the PSPS database another spatially-shared, file-based database using custom technology, a MySQL database like the Processing Database, or something else? I assume the same system is used at both IFA and MAST? 89 90 ** PSPS is based on MS SQL Server. We have added a bit of description to 4.3. 91 92 ## Section 5.1.1 93 94 Apparent typo or missing text: macro ex- its job successfuly". 95 96 ** this should have read 'macro exits successfully' ("exits" was being hyphenated). fixed. 97 98 ## Section 5.1.4 99 100 "responsible to" -> "responsible for" 101 102 ** fixed 103 104 ## Section 5.2 105 106 > Pairing warps together is simplified by the observing strategy in which the same pointing is observed multiple times in a night. By limiting to warp-warp pairs from the same pointing, the problem is significantly reduced from the arbitrary case. 107 108 This (as well as the following paragraph) seems to imply that you typically generate differences between images taken in the same night, which of course limits you to detecting only very short-timescale transients and fast-moving objects. I suspect that's just not what you intended to imply, or is the nightly processing really not supposed to find e.g. supernovae? 109 110 ** the wording here was unclear that the nightly processing system generates warp-warp difference images (for asteroids), warp-stack difference images (for 3pi supernovae), and MD nightly stack - reference stacks difference images (for deep MD supernovae). We have updated the text to explain these differences. 111 112 ## Section 5.2 113 114 Are the `projection_cells` described here the same as or related to the DVO partition cells of 4.1.3, or the RINGS.V3 skycells of 3.7? 115 116 ** same as RINGS.V3. we have clarified this and also cleaned up the wording of this paragraph. 117 118 This is a more general concept, but it came to a head in this section: I found the use of so many notation styles for different concepts more distracting than helpful. I think I was able to infer that small caps were used for processing stages and non-bold italics were used for database tables, but it wasn't clear why some other stages were written in fixed-width mixed case instead (were these scripts, rather than stages?), or what the use of bold-italic meant (everything eles?). I'd recommend either adding a notation legend paragraph early in the paper or just cutting down on the number of styles used. 119 120 ** We agree and have simplified the typography a bit (using only a single face for both db tables and db columns), eliminating the use of boldface. We have also added a paragraph in the introduction section to define the type faces. 121 122 ## Section 5.3 123 124 Was Nebulous just used by the orchestration levels like pantasks, or was it used within the Perl scripts and C programs that constitute the algorithmic steps as well? 125 126 ** Nebulous is used by any level of the software that needs access to a specific file. The c-based processing programs have direct interfaces as do the Perl-based wrappers (ippScripts). We have added a paragraph to explain this. 127 128 Was the database used by Nebulous integrated with the Processing Database at all (or even part of the same server)? 129 130 ** these two databases are on separate machines and kept independent. A sentence was added to the end of 6.1 to note this. 131 132 It's a bit strange to first encounter what seems like a core part of the data access system this late in the description, given that it would have needed to be updated by all of the processing steps mentioned early. This would of course make more sense if Nebulous is in fact used by the lowest levels of the pipeline and hence a Nebulous database entry is created whenever a file is written to disk. 133 134 ** our organizational scheme is meant to place the details closest to the science analysis up front and leave the more general systems toward the end, with only a few necessary broad concepts introduced early on for context. Thus section 3 is about the analysis steps and the related programs, section 4 is about the science database and the calibration, section 5 is more generic operations concepts, and section 6 is the computing hardware. Within section 5, the processing organization comes first, while nebulous is left to the end since it seems (to us) to be very general and should not be driving the science decisions. 114 135 115 136 116 ## Section 3.8117 118 Is the selection of which warped images go into a stack driven by119 human operators, or are there automated systems to launch these jobs,120 too?121 122 ** section 5.2 discusses how both the nightly stacks and123 large-scale reprocessing campaign stacks are automatically124 defined. We added some words to refer to this section in 3.8.125 126 ## Section 3.10127 128 How much of the PSF-convolved galaxy models do you re-fit in forced129 photometry? If you're fitting more than just the amplitude at that130 stage, and considering each exposure as independent, you're131 potentially throwing away a lot of S/N (at least in the many-exposure132 limit), even if you average later. If you're just fitting the133 amplitude, the structural parameters are still going to be the ones134 affected by poor PSFs in the stack.135 136 ** the galaxy models are not fitted on each warp. rather we137 calculate the normalizations and chi-square values for a grid of138 galaxy model shape parameters for each warp image. The values139 for each grid point are combined across all warps to generate a140 total stack-equivalent grid. At this point, the best parameters141 are determined from the grid (interpolating to the chi-square142 minimum). This is mathematically equivalent to simultaneously143 fitting (via a grid search) the pixels from all warps to a single144 model, preserving the full signal-to-noise. We have updated the145 text to add some detail to the description of what is being146 measured to clarify this point.147 148 ## Section 3.11149 150 transient source -> transient sources151 152 ** fixed.153 154 ## Section 4.1.3155 156 I was confused when first encountering the word "files" here because157 up to this point I had been thinking of the DVO as just another MySQL158 (or other SQL-ish) database, and I wasn't sure what kind of files were159 being referred to. I think it'd be helpful to briefly describe the160 overall architecture of the DVO as (mostly?) spatially sharded files161 at the beginning of section 4, even if the details of the partitioning162 aren't described until 4.1.3.163 164 ** we added a sentence to 4.1.1 to note this point.165 166 Missing punctuation in parenthetical HST GSC reference?167 168 ** fixed169 170 ## Section 4.1.4171 172 There's some inconsistency here between "detID" and "det_id" (same for173 "image"), both referring to measurement IDs in DVO. If those are174 supposed to be meaningfully different, I'm confused.175 176 ** in the DVO section (and in the DVO schema), these should all be177 'detID' and 'imageID'. In the gpc1 database schema, the178 underscored versions are used. we have fixed the erroneous179 det_id and image_id entries in this section.180 181 ## Section 4.2182 183 I tend to associate the term "ubercal" specifically with the SDSS184 version of the algorithm that coined the term, and think it probably185 should be referenced here even if the actual algorithms used are only186 vaguely similar.187 188 ** we agree and have added a sentence with reference.189 190 ## Section 4.3191 192 Is the PSPS database another spatially-shared, file-based database193 using custom technology, a MySQL database like the Processing194 Database, or something else? I assume the same system is used at both195 IFA and MAST?196 197 ** PSPS is based on MS SQL Server. We have added a bit of198 description to 4.3.199 200 201 ## Section 5.1.1202 203 Apparent typo or missing text: macro ex- its job successfuly".204 205 ** this should have read 'macro exits successfully' ("exits" was206 beign hyphenated). fixed.207 208 ## Section 5.1.4209 210 "responsible to" -> "responsible for"211 212 ** fixed213 214 ## Section 5.2215 216 > Pairing warps together is simplified by the observing strategy in217 which the same pointing is observed multiple times in a night. By218 limiting to warp-warp pairs from the same pointing, the problem is219 significantly reduced from the arbitrary case.220 221 This (as well as the following paragraph) seems to imply that you222 typically generate differences between images taken in the same night,223 which of course limits you to detecting only very short-timescale224 transients and fast-moving objects. I suspect that's just not what you225 intended to imply, or is the nightly processing really not supposed to226 find e.g. supernovae?227 228 ** the wording here was unclear that the nightly processing system229 generates warp-warp difference images (for asteroids), warp-stack230 difference images (for 3pi supernovae), and MD nightly stack -231 reference stacks difference images (for deep MD supernovae). We232 have updated the text to explain these differences.233 234 ## Section 5.2235 236 Are the `projection_cells` described here the same as or related to237 the DVO partition cells of 4.1.3, or the RINGS.V3 skycells of 3.7?238 239 ** same as RINGS.V3. we have clarified this and also cleaned up the240 wording of this paragraph.241 242 This is a more general concept, but it came to a head in this section:243 I found the use of so many notation styles for different concepts more244 distracting than helpful. I think I was able to infer that small caps245 were used for processing stages and non-bold italics were used for246 database tables, but it wasn't clear why some other stages were247 written in fixed-width mixed case instead (were these scripts, rather248 than stages?), or what the use of bold-italic meant (everything249 eles?). I'd recommend either adding a notation legend paragraph early250 in the paper or just cutting down on the number of styles used.251 252 ** We agree and have simplified the typography a bit (using only aa253 single face for both db tables and db columns), eliminating the254 use of boldface. We have also added a paragraph in the255 introduction section to define the type faces.256 257 ## Section 5.3258 259 Was Nebulous just used by the orchestration levels like pantasks, or260 was it used within the Perl scripts and C programs that constitute the261 algorithmic steps as well?262 263 ** Nebulous is used by any level of the software that needs access264 to a specific file. the c-based processing programs have direct265 interfaces as do the Perl-based wrappers (ippScripts). We have266 added a paragraph to explain this.267 268 Was the database used by Nebulous integrated with the Processing269 Database at all (or even part of the same server)?270 271 ** these two databases are on separate machines and kept272 independent. A sentence was added to the end of 6.1 to note273 this.274 275 It's a bit strange to first encounter what seems like a core part of276 the data access system this late in the description, given that it277 would have needed to be updated by all of the processing steps278 mentioned early. This would of course make more sense if Nebulous is279 in fact used by the lowest levels of the pipeline and hence a Nebulous280 database entry is created whenever a file is written to disk.281 282 ** our organizational scheme is meant to place the details closest283 to the science analysis up front and leave the more general284 systems toward the end, with only a few necessary broad concepts285 introduced early on for context. Thus section 3 is about the286 analysis steps and the related programs, section 4 is about the287 science database and the calibration, section 5 is more generic288 operations concepts, and section 6 is the computing hardware.289 Within section 5, the processing organization comes first, while290 nebulous is left to the end since it seems (to us) to be very291 general and should not be driving the science decisions.292
Note:
See TracChangeset
for help on using the changeset viewer.
