PS1 IPP Czar Logs for the week 2014-04-14 - 2014-04-20

(Up to PS1 IPP Czar Logs)

Monday : 2014-04-14

  • 09:15 Bill has set the sas pantasks to run. There are ~200 faults to debug. Load should be low.

Tuesday : 2014-04-15

  • 12:45 Bill stopped distribution pantasks. I made a mistake in the SAS.20140404 distribution runs that caused them to not use nebulous for the bundles. I am cleaning this up.
    • all fixed
    • later: discovered a number of interesting things about distribution. We have been bundling chip, camera, warp, and diff pixels for the OSS, ESS, and MSS labels yet nobody has accessed any of this data this year. This is a contributor to our rapid data usage. Queued all of those data for cleanup and stopped them from being bundled in the future
    • also found that we aren't bundling the i band SS data. These are used by NCU and Harvard. Set this up.
  • 15:45 Bill restarted registration pantasks
  • 17:47 Bill queued lap.threepi.20120706 camera stage distribution runs for cleanup.

Wednesday : 2014-04-16

  • 08:40 Bill is running distribution of full force results in ~ippsky/sas-pantasks. Should run pretty quickly
  • 10:15 bill is running a skycal/fullforce test on Eddie Schlafey's test region using the sas-pantasks. 49 skycells staticsky input is LAP.20130717 staticsky and warps
    • this processing is complete sas-pantasks shut off
  • 15:00 CZW: stopped stdscience to merge new psastro with new CT masks into the tag. Shutdown and restarted the pantasks server when it got cranky.

Thursday : 2014-04-17

  • 11:50 Bill: briefly running 4 fullforce runs in the sas-pantasks

Friday : 2014-04-18

  • 01:00 MEH: fixing OSS fault 5 diffim so clears for mops (541953, skycell.1272.037)
  • 11:00 MEH: stdsci well underloaded, way past regular restart.. doing now
  • 11:20 MEH: clearing remaining fault 5 diffim from 3PI (typical few stamps non-robustness).
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2124.079 -diff_id 542011  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2123.091 -diff_id 542016  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2121.061 -diff_id 542039  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2258.000 -diff_id 542046  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.2191.065 -diff_id 542042 -fault 0
    • oddly, appears some fault 5 cleared/caught into another quality state after some number of reverts over the night.. another fault case? (542014, skycell.2123.036; 542015, skycell.2124.076)

Saturday : 2014-04-19

  • 19:40 MEH: stdsci poll becoming slightly underloaded for regular needed restart, doing before nightly starts

Sunday : 2014-04-20

  • 00:20 MEH: clearing more diffim fault 5 (few stamp non-robustness) -- 542391, skycell.1456.010; 542393, skycell.1457.086
  • 08:00 MEH: more fault 5 (same case above), clearing so mops has diffims
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1372.013 -diff_id 542446  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1283.059 -diff_id 542496  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1812.056 -diff_id 542563  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1812.063 -diff_id 542563  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1812.072 -diff_id 542563  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1726.051 -diff_id 542549  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1726.040 -diff_id 542559  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1727.029 -diff_id 542559  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1812.053 -diff_id 542564  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1812.062 -diff_id 542564  -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.1811.013 -diff_id 542565  -fault 0
  • 10:30 MEH: ipp055 put to repair since disk warning problems and over weekend