PS1 IPP Czar Logs for the week 2013.12.02 - 2013.12.08

(Up to PS1 IPP Czar Logs)

Monday : 2013.12.02

Bill is czar today

• 10:47 Set all data with label like ps_ud% to be cleaned. Restarted cleanup pantasks.
• 13:30 Dropped the 6 STD.nightlyscience chipRuns from 11-27 and 11-29 for exposures with filter = 'OPEN' We don't process those because we have no flat detrends
• 13:35 set all pantasks to stop in preparation for full restart. (The only one doing anything currently is cleanup)
• 16:15 set STS.rp.2013 distribution bundles to be cleaned
• 19:05 MEH: a reminder when restarting any pantasks, hosts may need to be manually turned off to account for DVO/PSPS/mysql use on machines until mysql etc can be restarted or jobs/nightly processing is likely to get jammed -- currently looks like ipp048--ipp053, 039, 057 may still be problem childs.
  • ipp053 holding onto registration job >3ks, chip jobs >2ks so far..
  • turning these ipp048--ipp053 off

Tuesday : 2013.12.03

• 08:35 Bill: Serge reports that ipp is stuck. It appears that ipp057 is down and some registration jobs are stuck trying to access files there power cycling it
• 16:10 MEH: in addition to manually taking nodes out of processing if DVO/PSPS overloading, they may also need to be put into neb repair (more of an issue now that there is open space on many of the similar data nodes
• 16:20 MEH: tweak_ssdiff to get the SSdiff out for QUB

Wednesday : 2013.12.04

• 09:38 Bill: dropped chip_id 919320 and 919321 because they will fail to run. The filter for these STD exposures is OPEN and we don't have flats for that case.
• 12:25 MEH: ipp065 needed to be taken out of stack as well.. tweak_ssdiff for whatever MD stacks are available now
• 17:50 MEH: Heather finished restarting mysql on ipp048-ipp053, ipp065 so putting back into pantasks
  • ipp039 may need to have an eye kept on
• 18:00 CZW: I've set stsci13.X to repair in nebulous. Something is broken with NFS, and because of this, the log is full of "lockd: cannot monitor ipp046" entries. This prevents any new files from being written, as we lock files to ensure they're written correctly. This was forcing nearly all jobs to fail, and suggests that stsci13 needs to be rebooted to regain its sanity.

Thursday : 2013.12.05

mark is czar

• 00:40 MEH: something was slowly reducing processing until ~midnight -- registration got ~20 exposures behind, camera stage taking ~2ks, rates ~20 exp/hr. removed LAP label and lap.off @midnight and things slowly getting back to normal, LAP label back in and lap.on but such a backlog of chips/warp will have to wait and see if degrades again.
• 04:50 MEH: again found LAP overusing processing, >80 3PI chips to do, registration behind -- LAP label out.
• 08:00 MEH: nightly finished except for WS. cleared five fault 5 WS diffim (501812, skycell.1157.030; 501836, skycell.1156.007; 501847, skycell.1157.030; 501868, skycell.1156.007). cleared two distRun faults hanging around from 12/2. LAP chip.off for a bit to clear some warps for stacks, then chip.on.
  • unclear what the slowdown was last night, whether other jobs (dvo, ps2iq,?) were doing it or if mysql on ippdb01 just needs to be restarted again
• 11:40 MEH: notice 33G left on ippc18.0, will archive the small 10G or so of logs before the weekend..
• 18:00 MEH: doing regular restart stdsci before nightly -- will be watching processing again to see what could be causing the stalling seen last night with LAP running
  • cleanup was probably overloading, 2x w4,c,c2,c3 is probably overkill. noticed jobs taking >1ks to finish earlier so turned poll down to 30, turned off all computes and 1x wave4
  • lap.off helped to drop ippdb01 load, adjust.lap seems to set a better poll period so running with that for a bit
  • mysql on ippdb01 likely just needs a restart as well
• 20:30 MEH: lap.off until ~midnight (when ~half chunk LAP stacks should be finished) as ippdb01 loaded again and plenty of nightly to do
• 23:50 MEH: lap.on with increased poll period and loading new LAP jobs for past hour and seems ok, leave on for rest of night

Friday : 2013.12.06

mark is czar

• 08:00 MEH: change in LAP poll period seems to have helped keep nightly from backing up last night -- will make default in stdsci startup
• 10:30 MEH: after talking with Gavin, stsci13 just needs a reboot -- looks like SAS staticsky will take most of day still and don't want to interrupt and don't want to reboot at end of day on Friday.. leave for Monday
  • ippdb01 mysql could also likely use a reboot and will leave for Monday (bad idea on friday)

Saturday : 2013.12.07

Sunday : 2013.12.08