PS1 IPP Czar Logs for the week 2014.07.28 - 2014.08.03

(Up to PS1 IPP Czar Logs)

Monday : 2014.07.28

• 07:05 Bill: Restarted distribution pantasks. (The rest of them could probably use a restart as well)
  • increased npending for rcserver.makefileset.run task to 10 from 5. It is about 200,000 behind the number of completed filesets. We should keep an eye on ippc17 to insure that this doesn't overload the datastore server ippc17.
• 08:45 MEH: w/ nightly processing finished, using the idle nodes for MOPS diffim reprocessing tests (couple hours to run) -- finished
• 09:11 Bill: Preparing to restart pstamp pantasks, letting pstamp.request.finish jobs finish.
  • 09:50 pstamp pantasks server restarted

Tuesday : 2014.07.29

• 05:30 Bill: changed label to allow another 50,000 skycal distRuns to be processed. There are also 50,000 filesets to be made.
• 11:00 MEH: going to start some MD.pv2 staticsky runs on underutilized s3+c2 -- turning off s3 before nightly processing
• 16:24 CZW: I've added ipp071 into nebulous, and after some time and a few spot tests, it seems to be accepting files correctly. I have therefore set it to 'up'.

Wednesday : 2014.07.30

Thursday : 2014.07.31

• 08:38 Bill: started up skycal as ~ipptest to process the pole skycells. Starting with the CNP version.
  • 08:42 that was quick. Now adding label for rings.v3 version
  • also set another 100,000 lap skycal distRuns to be processed. Here is the current status

    mysql> select label, state, count(dist_id) as num_runs, min(dist_id) as first_dist_id, max(dist_id) as last_dist_id 
    from distRun where label like 'lap.threepi.20130717%' and stage ='skycal'  group by label,state;
    +---------------------------+-------+----------+---------------+--------------+
    | label                     | state | num_runs | first_dist_id | last_dist_id |
    +---------------------------+-------+----------+---------------+--------------+
    | LAP.ThreePi.20130717      | full  |   550002 |       3465080 |      4015081 | 
    | LAP.ThreePi.20130717      | new   |   100000 |       4015082 |      4115081 | 
    | LAP.ThreePi.20130717.wait | new   |   333697 |       4115082 |      4677520 | 
    +---------------------------+-------+----------+---------------+--------------+
    3 rows in set (4.55 sec)
    
    

• 09:20 MEH: clearning another fault 5 diffim from the other night before the warps get cleaned up -- 579391,skycell.2054.023 -- too late, now have to manually update warps..
• 09:30 MEH: doing regular restart of stdscience

Friday : 2014.08.01

Saturday : 2014.08.02

Sunday : 2014.08.03