PS1 IPP Czar Logs for the week 2013.01.07 - 2013.01.13

(Up to PS1 IPP Czar Logs)

Monday : 2013.01.07

• 04:40 Gene tried power cycling ipp027 and no response, further details pending
• 10:00 Bill restarted pstamp and update pantasks. pcontrol for deepstack is using 30% of a cpu on ippc17. Since the task durations are so large this isn't too wasteful.
• 13:17 Bill restarted cleanup pantasks. Added label goto_cleaned.rerun. Setting all chip and warp runs from LAP.2011% to state = goto_cleaned, label = goto_cleaned.rerun. This will cause cleanup to be rerun using the new code which will cause chip warp cmfs to be deleted.

Tuesday : 2013.01.08

• 03:20 MEH: looks like registration pantasks has crashed, starting. also looks like distribution is stuck, restarting.
• 07:45 MEH: MD03 o6300g0570o is strange, looks like streaks. observation started while still slewing? nightly stack in progress, will it be okay? so far looks okay.
• 15:00 Serge: Restarted a bunch of mysql servers on storage nodes. Made a backup of the mysql configuration files in the operational repository: ​file:///data/ippc18.0/home/operational_repository/current/databases
• 15:33 Bill: fixed a bug in ipp_cleanup.pl that was causing cleanup to be run repeatedly on the same runs

Wednesday : 2013.01.09

mark is czar (calendar loaded again through end of month)

• 08:30 Serge: Stopped nebulous replication on ippdb02. Performing backup to ippc63 (through script being tested).

Thursday : 2013.01.10

mark is czar

• 10:25 Serge: stsci04.1 is full

  neb-host --volume stsci04.1 repair
  

• 10:45 Serge: The neb-df figure in czartool has not been updated for about two days. I restarted nebdiskd on ippdb00.
• 15:00 MEH: with Bill, am setting LAP staticsky outside of the galactic plane to run in another pantasks (lap_staticsky_notgp) on ippc13 when nightly science isn't running. please do not load for stdscience or stack pantasks to run
• 15:40 Serge: nebdiskd now relies on ssh+df instead of df-ing nfs-mounted partitions: when a host was down, nebdiskd was stuck because of the still mounted but not reachable partition (e.g. the "flat signal" observed in the czartool available space figure since ipp027 died two days ago). Restarted nebdiskd. Changes committed to tag (rev: 34906) and then merged into trunk (rev: 34907).
• 15:55 MEH: pausing cleanup for a while so ippdb02 can catch up
• 17:05 MEH: turning off wave3,compute2,stare in lap_staticsky_notgp so the longer 1-1.5 hrs jobs clear before nightly science

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD