Changes between Version 160 and Version 161 of ippToPsps

Timestamp:

Sep 29, 2011, 11:55:22 AM (15 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -138 +138 @@
 == Deletion policy ==
 
-When loading, two copies of the data exist: the batches on the datastore and the original files on disk. The deletion policy at present is:
-
- * when a batch has been loaded to the ODM and has a status of 'merge worthy', then the copy on the datastore is removed
- * when a batch has been successfully merged into the PSPS database, the copy on local disk is deleted
+When loading, three copies of the data exist: the original files on disk, the batches on the datastore and a third copy on the PSPS loading machine. The deletion policy at present is:
+
+ * when a batch has been loaded to the ODM and has a status of 'merge worthy', then the copy on the datastore and the DXLayer copy can be removed
+ * when a batch has been successfully merged into the PSPS database, the final copy on local disk is deleted
 
 The logic for this is that errors may occur during the merge phase and it is useful to have local copies of offending batches for debugging purposes.
 
+This standard behavior can be changed by setting the appropriate values in the ''deletion'' section of the config file.
+
 == Metrics ==
 
@@ -156 +158 @@
 
 {{{
-               ippToPsps loading summary
-
+                ippToPsps loading summary
+
+Time now                                2011-09-29 11:54:33
 Loading epoch                           2011-08-16
 DVO label                               LAP.ThreePi.20110809
 
-+------+----------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------+
-| Type |  Total in DVO  |   Processed    |  On datastore  | Loaded to ODM  |  Merge-worthy  |     Merged     |  Total missing |    Last published    |
-+------+----------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------+
-|  P2  |  226           |  49(177)       |  49            |  0(49)         |  0             |  0             |  226           |  16.0 minutes ago    |
-|  ST  |  0             |  0             |  0             |  0             |  0             |  0             |  0             |  Never               |
-+------+----------------+----------------+----------------+----------------+----------------+----------------+----------------+----------------------+
++----+------------------+---------------+-------------------+------------------+----------------+
+|Type| batches per hour | last 24 hours | per day this week | total detections | last published |
++----+------------------+---------------+-------------------+------------------+----------------+
+| P2 |              0.0 |           283 |             329.4 |       1855730051 |  4.6 hours ago |
+| ST |              0.0 |         12936 |            4539.0 |        185807608 |  3.7 hours ago |
++----+------------------+---------------+-------------------+------------------+----------------+
+
++----+-------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+|Type|  DVO  |          processed|loaded_to_datastore|      loaded_to_ODM|       merge_worthy|             merged|  deleted_datastore|    deleted_dxlayer|      deleted_local|
+|    |       | Pend  Succ  Fail  | Pend  Succ  Fail  | Pend  Succ  Fail  | Pend  Succ  Fail  | Pend  Succ  Fail  | Pend  Succ  Fail  | Pend  Succ  Fail  | Pend  Succ  Fail  |
++----+-------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
+| P2 |  8140 |       8101    39  |    2  8099        | 2041  6049     9  |       6049        |  313  5736        |       5736        | 5736              |                   |
+| ST | 32309 |      31684   625  |    5 31679        |24617  7062        |       7062        | 7062              |                   |                   |                   |
++----+-------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
 
 }}}
@@ -173 +184 @@
 
 [[Image(LAP.ThreePi.20110809_P2.png)]] 
-
-