PS1 IPP Czar Logs for the week 2014.07.14 - 2014.07.20

(Up to PS1 IPP Czar Logs)

Monday : 2014.07.14

mark is czar

  • camera on pump-down over weekend, should be back up tonight.
  • 10:20 MEH: will start regular restart of pantasks in preparation for week -- looks like someone had started multiple pantasks_severs for summitcopy, registration, stdsicence, cleanup, stack. dummy one looks to have date July 10.
    • looks like someone restarted pstamp and distribution this morning, but pstamp still has a dummy pantask_server running, so shutting both down and restarting..
    • also need to rotate the apache logs and so will interrupt processing -- ippc02 has ~5G less than other machines on / and unclear why
  • 10:50 MEH: distribution looks to be stopped, no note so turning back to run
  • 16:00 MEH: summit not ready, no data tonight

Tuesday : YYYY.MM.DD

  • 15:45 CZW: Mark noticed that the new nebulous directories were causing write problems. The issue seems to be that ipp068 and ipp069 did not have their nebulous directories chowned properly, preventing the nebulous group from writing. I've chowned/chmoded these to match the other hosts. In addition, I've rsynced the /data/ippXXX.0/ipp/ directory contents (tess, tmp) onto ipp067-ipp070, so these match as well.
  • 17:25 CZW: Started side pantasks by ipp user on ippc20 to regenerate missing stack summary products. These should finish before processing starts tomorrow night.

Wednesday : 2014.07.16

  • 14:00 CZW: Haydn has moved ippdb01 to a (hopefully) better cooled location in the racks, and I've restarted the pantasks servers.

Thursday : 2014.07.17

  • 17:10 CZW: I've turned on s1, s2, s3 in stack pantasks so we can have some power devoted to stacks. I don't know what the correct full-rebalance solution is, but this should at least keep the jobs running.
  • NOTE: PS1 is down due to summit glycol / cooling problems. Down until at least next week Wed night.

Friday : 2014.07.18

  • 16:00 EAM : I launched skycal for the LAP.ThreePi?.20130717 data. I have a script running which is queuing data in 1h bands.

Saturday : 2014.07.19

Sunday : 2014.07.20