Changes between Version 22 and Version 23 of PS1_IPP_CzarLog_20120514

Timestamp:

May 18, 2012, 8:51:56 AM (14 years ago)

Author:

bills

Comment:

--

PS1_IPP_CzarLog_20120514

@@ -28 +28 @@
  * 11:00 CZW: restarted stdscience, which was using a full processor for pcontrol.
  * 11:27 CZW: restarting stdscience again, because scrolling up to find a "status" command occasionally finds a "shutdown" command. These are not the same commands.
-=== Thursday : YYYY.MM.DD ===
+=== Thursday : 2012-05-17 ===
 
-=== Friday : 2012-05-17 ===
 
  * 07:20 EAM : burntool hung / failed on one exposure, fixed with: {{{regtool -updateprocessedimfile -exp_id 488814 -class_id XY71 -set_state pending_burntool -dbname gpc1}}}
@@ -41 +40 @@
  * 11:45 EAM : big memory spike on ipp058 with resulting crash; rebooted it
  * 11:59 EAM : restarted all pantasks to clear out cruft
+=== Friday : 2012-05-18 ===
+ * 08:30 - 08:45: Bill Some poor user entered coordinates in MD09 for a warp postage stamp request into the web form. Since the coordinates also overlap SAS 6684 jobs were queued. Since the web form has highest priority this has lead to a massive backlog. Changed the labels for that request from WEB to WEB.BIG which has
+lower priority. Found a QUB stack/stack diff request which was failing because one of the files in one of the nightly stacks from several years ago was lost in the ipp020X disaster of 2010. It eventually would have given up after it faulted 3 times but I put it out of it's misery.
+ * Space is getting to be a problem. From the graph it looks like we have a couple of weeks before we are down to zero. So many nodes are empty that we are getting quite a few job faults due to nfs errors. Reverted a few by hand. There are a couple of days worth of space used up for pstamp updates. We can clean that up once we catch up. 
 
 === Saturday : YYYY.MM.DD ===