PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.15

• 08:00 MEH: cleanup must be turned back on, the few number of nodes available for data is killing processing again
• 11:00 CZW: I've updated the database to prevent the exposures from the dead pixelserver from clogging up summitcopy:

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832987;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832988;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832989;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832990;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832991;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832992;
  

• 14:00 CZW: Restarting stdlocal. I'm going to see what happens with the "default" set of hosts.
  • 14:07 CZW: default set is underpowered. Turning on xnodes with xnode.hosts.on macro.
• 14:50 MEH: summitcopy stop -- camera group reporting ramping overload -- anyone else doing summitcopy type stuff?
• 14:45 CZW: I noticed more failing exposures in pzDownloadExp table, and dropped the camera controlled exposures in state 'run':

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834453;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834454;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834455;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834456;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834457;
  

• 14:55 MEH: Haydn working on ipp077, will be up/down and shouldn't be used until he gives the all-clear
  • back online -- setting to repair only so not to overload until we catchup
• 15:30 MEH: cleanup diff on only, doing double loading and poll 120 -- seems okay, will help move the larger size diffs from past few days out of the way
• 16:00 CZW: With ipp077 back online, the LANL remote stack prep commands are no longer failing. I've reverted the LAP.PV3.20140730.bigmem stacks locally, to see if adding stacks impacts the processing rate.
• 16:25 CZW: Restarting the ipp user pantasks. I will add 4x c2 to stdscience and ensure that cleanup is prioritizing the larger diffs.
• 16:55 CZW: ipp071 had a load spike, probably related to stdlocal having the stack poll set too high. I've set it to repair, and I'll move it back to up when the stack npend number drops.
• 18:20 CZW: After hearing back from Gene, I've added the stsciXX nodes to 'up' in nebulous to see if that helps with our processing throughput.
• 18:25 CZW: Load jumped on stsci machines running Gene's rsyncs. I'm hesitant to keep them up for nightly processing with this added load. Back to repair, and I'll try again tomorrow when it's easier to monitor.
• 18:30 MEH: ippx001 new kernel behavior essentially no different to old kernel version (though the overloading xNNN w/ stacks seems to work well) -- need more info on what the network to these machines is doing
• 18:35 MEH: will manually add new storage nodes to summitcopy as done for past nights -- however, before adding, seeing a large number of fault events in summitcopy, something wrong? stsci overload? stdlocal heavily loaded still?

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD