PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

Continuing HAF suggestion to improve communication, so we know what's going on better -- adding a list in the czar pages for additional (non-standard) processing so that we all know what's going on.

Daily Czaring:

• MOPS has repeatedly requested NO changes be made to the ~ipp ops tag without being verified by their test set -- so currently there is a modified ops tag running diffs (WS labels only) as ippqub under ~ippqub/src/stdscience_ws on ippc06 -- if problems (Njobs>100k, power loss on ippc06 etc), it will need to be restarted like a normal nightly processing pantasks (or IF ANY OTHER CHANGES made for the ~ipp pantasks then likely same needs to be made for ~ippqub)

      ./start_server.sh stdscience_ws
  

  • even with the modified ops tag, there are still various files MIA in nebulous and fault problems that requires the daily czar to check and clear them as has been discussed before
  • ps_ud_QUB has also been moved to the ippqub:stdscience_ws pantasks to support updates possibly broken by missing cmf files, chip and warp updates will also be done in this pantasks as well

(Up to PS1 IPP Czar Logs)

Monday : 2015-12-07

• 19:55 CZW: restarted summitcopy and ippqub/stdscience_ws as they were crashed, and the email thing was complaining.

Tuesday : 2015.12.08

• 02:20 MEH: nightly processing got stuck, oddly some nodes just stalling jobs in reg+stdsci for 2+ hrs.. -- ipp065, 055, 005, 069, 089 (ipp055 common only to both)..
• 03:10 MEH: registration stuck on o7364g0449o and needed manual revert, ipp087 also having load spikes and high cpu wait so put neb-host repair for a bit (BBU seems fine)
• 06:50 Bill: Registration is 137 exposures behind. Burntool status shows a faulted imfile. Reverted it and data is flowing again

  % regtool -revertprocessedimfile -exp_id 1001097
        Updated 1 rawImfile
  

• 08:49 MEH: looks like stalled exposure o7364g0288o in chip and stalled exposure o7364g0289o in warp -- both should be cleared now
  • o7364g0288o was missing chipImfile, reran the chip_imfile.pl to finish the chipImfile
  • o7364g0289o was missing skycell in warpSkyfile causing revertoverlap to barf

     -> p_psDBRunQuery (psDB.c:812): Database error generated by the server
         Failed to execute SQL query.  Error: Cannot delete or update a parent row: a foreign key constraint fails (`gpc1/warpSkyfile`, CONSTRAINT `warpSkyfile_ibfk_1` FOREIGN KEY (`warp_id`, `skycell_id`, `tess_id`) REFERENCES `warpSkyCellMap` (`warp_id`, `skycell_id`, `tess_id`))
     -> revertoverlapMode (warptool.c:744): unknown psLib error
         database error
    

    • manually clear faults in db

      update warpSkyCellMap set fault=0 where warp_id=1649894 and fault>0;
      

• 09:28 MEH: restarting nightly pantasks that were having problems last night --
• 10:12 MEH: cleaning fault 5 diff (ref stack fwhm nan..)

  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.1702.060 -diff_id 1288693  -fault 0
  

• 12:23 MEH: starting full SAS reprocessing w/ PV3 tags under label SAS.20151208 using nightly nodes ~half power -- chip-warp,stack will use ~ipplanl/stdsas pantasks on ippc63 -- will suspend for nightly ~1930 depending on weather
• 13:06 MEH: ipp046 is back up, need back neb-host repair
• 20:07 MEH: ipp087 wait cpu dominating, so back to repair

Wednesday : 2015.12.09

• 08:54 MEH: clearing fault 5 diff (ref FWHM NAN)

  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.1234.046 -diff_id 1288757 -fault 0
  

• 09:30 MEH: SAS.20151208 stacks running on misc nightly nodes most of day

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD