PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.15

• 08:00 MEH: cleanup must be turned back on, the few number of nodes available for data is killing processing again
• 11:00 CZW: I've updated the database to prevent the exposures from the dead pixelserver from clogging up summitcopy:

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832987;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832988;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832989;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832990;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832991;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 832992;
  

• 14:00 CZW: Restarting stdlocal. I'm going to see what happens with the "default" set of hosts.
  • 14:07 CZW: default set is underpowered. Turning on xnodes with xnode.hosts.on macro.
• 14:50 MEH: summitcopy stop -- camera group reporting ramping overload -- anyone else doing summitcopy type stuff?
• 14:45 CZW: I noticed more failing exposures in pzDownloadExp table, and dropped the camera controlled exposures in state 'run':

  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834453;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834454;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834455;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834456;
  UPDATE pzDownloadExp SET state = 'drop' WHERE summit_id = 834457;
  

• 14:55 MEH: Haydn working on ipp077, will be up/down and shouldn't be used until he gives the all-clear
  • back online -- setting to repair only so not to overload until we catchup
• 15:30 MEH: cleanup diff on only, doing double loading and poll 120 -- seems okay, will help move the larger size diffs from past few days out of the way
• 16:00 CZW: With ipp077 back online, the LANL remote stack prep commands are no longer failing. I've reverted the LAP.PV3.20140730.bigmem stacks locally, to see if adding stacks impacts the processing rate.
• 16:25 CZW: Restarting the ipp user pantasks. I will add 4x c2 to stdscience and ensure that cleanup is prioritizing the larger diffs.
• 16:55 CZW: ipp071 had a load spike, probably related to stdlocal having the stack poll set too high. I've set it to repair, and I'll move it back to up when the stack npend number drops.

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD