PS1_IPP_Czarlog_20130107 – Pan-STARRS IPP

Context Navigation

Version 26 (modified by Mark Huber, 14 years ago) ( diff )
--

04:40 Gene tried power cycling ipp027 and no response, further details pending
10:00 Bill restarted pstamp and update pantasks. pcontrol for deepstack is using 30% of a cpu on ippc17. Since the task durations are so large this isn't too wasteful.
13:17 Bill restarted cleanup pantasks. Added label goto_cleaned.rerun. Setting all chip and warp runs from LAP.2011% to state = goto_cleaned, label = goto_cleaned.rerun. This will cause cleanup to be rerun using the new code which will cause chip warp cmfs to be deleted.

03:20 MEH: looks like registration pantasks has crashed, starting. also looks like distribution is stuck, restarting.
07:45 MEH: MD03 o6300g0570o is strange, looks like streaks. observation started while still slewing? nightly stack in progress, will it be okay? so far looks okay.
15:00 Serge: Restarted a bunch of mysql servers on storage nodes. Made a backup of the mysql configuration files in the operational repository: file:///data/ippc18.0/home/operational_repository/current/databases
15:33 Bill: fixed a bug in ipp_cleanup.pl that was causing cleanup to be run repeatedly on the same runs

mark is czar (calendar loaded again through end of month)

08:30 Serge: Stopped nebulous replication on ippdb02. Performing backup to ippc63 (through script being tested).

mark is czar

10:25 Serge: stsci04.1 is full
```
neb-host --volume stsci04.1 repair
```
- Gene cleared up some duplicate non-neb data from the ipp027-ipp030 rsync, setting neb-host up again
10:45 Serge: The neb-df figure in czartool has not been updated for about two days. I restarted nebdiskd on ippdb00.
15:00 MEH: with Bill, am setting LAP staticsky outside of the galactic plane to run in another pantasks (lap_staticsky_notgp) on ippc13 when nightly science isn't running. please do not load for stdscience or stack pantasks to run
15:40 Serge: nebdiskd now relies on ssh+df instead of df-ing nfs-mounted partitions: when a host was down, nebdiskd was stuck because of the still mounted but not reachable partition (e.g. the "flat signal" observed in the czartool available space figure since ipp027 died two days ago). Restarted nebdiskd. Changes committed to tag (rev: 34906) and then merged into trunk (rev: 34907).
15:55 MEH: pausing cleanup for a while so ippdb02 can catch up
17:05 MEH: turning off wave3,compute2,stare in lap_staticsky_notgp so the longer 1-1.5 hrs jobs clear before nightly science
- taking wave4 from stack pantasks (should still be able to keep up with nightly science) and keep 1x wave4 running lap_staticsky_notgp
20:00 MEH: looks like poor weather, working on queuing up WS diffims for MOPS fields 160.3PI.00.E-4.K.z, 160.3PI.00.E-4.K.y