PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.22

• 07:40 MEH: warp fault -- cannot build growth curve (psf model is invalid everywhere)

  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1297048 -skycell_id skycell.2460.067
  

• 13:05 MEH: ipp006-031 neb-host repair, limit space available and try to help limit data across 10G link w/ note reflecting such (as discussed at morning meeting). ipp033-053 already in repair for similar reasons.
  • all except ipp008,012,013,014,016,018,109,020,021,037 which are actually on ippcore
• 18:00 CZW: restarting stdlocal, as it's old and not spawning jobs efficiently. -- be sure to re-adjust the poll to not overload the 10G link with chip processing
• 23:40 MEH: doesn't look like balance was done? poll>400 for chips and 10G overload harassing other processing (good thing no nightly), rate lower than before with fewer stacks. 8x:x nodes (not sure when changed) looks to be trying 6 stacks at a time.. probably not a good default to have.. -- setting back to 6x and reducing poll for chips to where 10G link isn't being as overloaded

Tuesday : 2014.12.23

• 09:05 MEH: ipp015 unresponsive, nothing on console, minimal load -- power cycle

Wednesday : YYYY.MM.DD

• 01:40 MEH: regtool on ipp062 stalling registration for past 1.5hr.. 20 exposures behind -- killed job and moving again
• 2:10 MEH: ippc28 crash -- GPF and something odd w/ swap /dev/sdb? -- try to powercycle but taking out of stdsci

  <Dec/24 01:42 am>[1507866.922239] end_request: I/O error, dev sdb, sector 198065
  <Dec/24 01:42 am>[1507866.928211] Read-error on swap-device (8:16:198073)
  

  • power cycled but not fully booting, stalling on * Starting local ...
• 06:15 MEH: fixed fault 5 diff

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 625690 -skycell_id skycell.1243.051 -dbname gpc1
  

Thursday : YYYY.MM.DD

• 06:00 MEH: nightly registration behind, stdlocal poll 200->150
• 06:25 EAM: ipp086 is getting overloaded by nfs and getting behind. nightly science is waiting for jobs that want ipp086. putting it in repair for now. update: putting stdlocal to control run reap while nightly science catches up a bit.
  • 086 load due to being in WT case, something is odd w/ battery stats
• 07:35 EAM : stdlocal back to run
• 08:10 MEH: diffim fault 5 clear

  difftool -updatediffskyfile -fault 0 -set_quality 42 -diff_id 625928 -skycell_id skycell.2048.094 -dbname gpc1
  

• 21:30 MEH: looks like stdlocal load has stalled nightly processing to the auto-shutoff point in stdlocal chip-warp -- setting stdlocal poll to 150 so if nightly catches up, both can maybe run more easily together
• 10:05 MEH: looks like stdsci polls also not staying filled -- needed its regular restart
  • ipp084 like 086 also in WT case and having load issues -- may need to trigger relearn cycle again..
  • w/ stdlocal auto-stopped, network over 10G link is underused ~5-6 Gb/s with just nightly and md processing

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD