PS1 IPP Czar Logs for the week 2013.08.26 - 2013.08.31

(Up to PS1 IPP Czar Logs)

Monday : 2013.08.26

mark is czar

• 08:30 MEH: nightly is finished, going to do regular restart of stdsci and try fix some LAP chips again
• 09:04 Bill restarted update and pstamp pantasks (their pcontrols were spinning)
• 09:13 Bill recovered two missing raw files from virtual bit bucket ippb02: o5536g0128o.ota44.fits and o5536g0071o.ota44.fits
  • the remaining 4 m31 faults have bad burntool state due to failures rerunning burntool. These will have to wait until ipp047's disk is back up
• 09:25 MEH: turning compute3 back off in pstamp now that MOPS is finished. adding goto_cleaned.rerundiff label back to cleanup
• 10:30 MEH: more MOPS stamps, 1x compute3 off in stack and back on in pstamp (and 1x in cleanup)
• 10:40 MEH: diffim faults (from 8/24) -- setting diff.revert.off (chip.revert.off also still set while ipp047 down, will set back on for nightly ~8-9pm)

  -- two 3PI WWdiff before fix to use good_frac (ie these warps had <0.1 good_frac) -- good_frac still not in ops build..
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -fault 0 -diff_id 470166 -skycell_id skycell.0705.092
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -fault 0 -diff_id 470287 -skycell_id skycell.0795.078
  
  -- one 3PI WSdiff -- with only 4 stamps for the diffim but images well covered -- log says FWHM for warp is ~2.8 and warp has many sources with odd masking at cores (8448 CR+blend), suspect undersize psf determination on warp and need to retest -- 4 stamps are on good sources, so looks like issue in chip->warp and other skycells have similar issues it seems, so dropping diffim
          470401 	skycell.1517.042 	ThreePi.WS.nightlyscience
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -fault 0 -diff_id  470401 -skycell_id skycell.1517.042
  
  -- two MD09 WSdiff -- both warp and refstack have nan for PSF.. something wrong here -- warps look odd (odd blotches size of sources) and high background ~22k and other skycells swamped with likely false detections it seems as well, so exposure poor anyways and dropping these
   	470147 	skycell.026 	MD09.nightlyscience
   	470159 	skycell.076 	MD09.nightlyscience
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -fault 0 -diff_id 470147 -skycell_id skycell.026
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -fault 0 -diff_id 470159  -skycell_id skycell.076
  
  

• 19:00 MEH: as mentioned at meeting, adjusted compute3 allocation
  • suggestion was just+1x into stdsci (for total of 3x) while keeping stack same (total of 3x)
  • +3x (for total of 5) in stdsci, -1x (for total 2) in stack seems to be stable (and cleanup also using 1x right now) -- stack rate is ~500 and load is okay even when doing M31 reprocessing at same time -- nightly rate same ~50-60 while LAP updates being done as well
• 22:30 MEH: mentioned at meeting, ipp034 is @half RAM (12/24G) and so made hosts_wave2_weak group in pantasks_hosts.input. basically loads as normal wave2 but half as many in stdsci

Tuesday : 2013.08.27

• 08:50 MEH: Serge pointed out faulting diffim, another low good_frac warp -- the nightly_science.pl needs to be updated/rebuilt with the fix in the ops tag

  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -fault 0 -diff_id 471203 -skycell_id skycell.1582.032
  

• 09:10 MEH: while sitting in DRAVG, might as well do the regular restart of stdsci
• 09:20 MEH: chip.off and LAP prio to push warps into stacks after nightly
• 09:25 Bill: queued last of obs_mode = 'm31' exposures. Still need to find and queue the overlapping 3PI exposures.
• 09:33 Bill: rebuilt corrupt chip_bg file with tools/runbgchip.pl --chip_bg_id 12076 --class_id XY55 --redirect-output
• earlier Bill: queued m31.rp.2013.bgsub data to be cleaned. Did not change the label to goto_cleaned to avoid accidentally rerunning exposures in the future. Added label to the cleanup pantasks. Leaving the background preserved data for now.
• 09:50 MEH: testing overload with revised stdsci+stack with LAP -- can hit stsci machines a bit hard initially but recovers okay. preparing to fix ipp047 LAP chip faults
  • like nightly data, the setup is slightly overloaded and produces extra faults (expecting from nfs issues) -- taking one out of stdsci should be sufficient (or possibly even the manually added one in cleanup may be enough easement)
• 10:10 EAM : ipp064 had a kernel panic, rebooting

Wednesday : 2013.08.28

Thursday : 2013.08.29

Friday : 2013.08.30

Saturday : 2013.08.31

Sunday : 2013.09.01