PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2019.01.14

• MEH: very old WW mops diffs and WS catalogs on datastore -- clean out (dsreg for WW and distribution goto_cleaned) to past couple months so page loads more easily for checking
• MEH: PS1 subset/interleaved OSS chunks interfering with normal MOPS operations -- comment string was changed from OSS.R% to EU.r%, manually alter nightly_science.config to proceed with processing
• MEH: PS2 back on sky -- WW diffs not queuing -- looks like the dark done in afternoon didn't have the expected comment of darks_afternoon?

  ENGINEERING	42707	o8498h0002d	2019-01-15 03:25:58	CLEAR.00002	30	darks_morn - Dark exp 30s 1 of 1
  
  nightly_science.pl --dbname gpc2 --camera GPC2 --verbose  --queue_diffs --date 2019-01-15
  

  • setup crontab to occasionally run this...
• MEH: PS2 WWdiffs taking very long time to run on a few skycells per diff_id -- manually killing ppSub on long running jobs and set quality 42 to allow the rest of the skycells to get to MOPS

Tuesday : 2019.01.15

• MEH: messed up PS1 EU config that blocked processing on the last OSS chunk, fixed for tonight tests okay -- new nightly processing label EU.nightlyscience added to stdscience with priority just below %SS
• MEH: PS2 WWdiffs still regularly taking >1-2hrs and delaying processing (also using excessive memory that overloads the heavily loaded nodes) -- create crontab to kill jobs after 3600s
  • manually add c2 group a 2-3x (more could also add more memory overload) to help move things along while waiting for problematic ppSub jobs to clear
  • looks like excessive detections on known odd group of cells from last Fall in chip near edge of camera -- looks like different set now, one in OTA in corner and one near middle (obvious poor cells in camera image) primarily, the previous one shows up from time to time as well -- needs to be masked until/if cause can be fixed

Wednesday : 2019.01.16

• MEH: bad exposure (GPC1 crash, multiple ota missing) stuck in summitcopy -- o8499g0264o -- should set to drop
• MEH: another example of pantasks autorestart borking job -- summitcopy at toadvance state o8500g0002d -- manually stop & restart summitcopy again is one way to clear (and clear any possible other oddities in the pantasks that may have happened)

Thursday : 2019.01.17

• MEH: another bad/incomplete exposure in summitcopy -- o8500g0392o
• MEH: reducing time to kill long running ppSub jobs to 1200s (average is 100-200s) until poor cells can be masked
• MEH: looks like the fortunate condition for XY36 that would trigger fault 4 has broken down and is also trying to run (>30 minutes) -- also manually killing ppImage if gets >600s -- this needs to be masked until/if cause can be fixed

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD