IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20150202

Version 13 (modified by Mark Huber, 11 years ago) ( diff )

--

PS1 IPP Czar Logs for the week 2015.02.02 - 2015.02.08

    1. PS1 IPP Czar Logs for the week 2015.02.02 - 2015.02.08
      1. Monday : 2015.02.02
      2. Tuesday : 2015.02.03
      3. Wednesday : 2015.02.04
      4. Thursday : YYYY.MM.DD
      5. Friday : YYYY.MM.DD
      6. Saturday : YYYY.MM.DD
      7. Sunday : YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2015.02.02

  • 08:50 EAM: stopped and restarted stdlocal, stdscience is basically done, so i've added in the storage hosts as well
  • 17:54 HAF: stop and restarting stdsci + friends for tonight.

Tuesday : 2015.02.03

  • 06:25 EAM: stopping and restarting stdlocal
  • 06:55 EAM: stsci13 hung on reads from lmsensors. I tried to use reboot, but it just hung. I power cycled and it is coming up
  • 10:55 MEH: reminder -- because we are an aggressive cleanup cycle of 2 days, any faults that need manual clearing have to be done ~daily or they won't clear w/o having to then also manually put the chips+warps back through updates -- for example faulted WS from 2/2 will have its warps cleaned up today at 2pm, so these need to get fixed now 
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654186
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654213
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654235
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 654286

difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 655221
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 655250
difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.0663.010 -diff_id 655267
  • 12:40 MEH: another modification to the data host targeting for nightly processing, adding nodes w/ poor BBU to accept raw OTA since worked with systems <ipp066 (ipp-20141024.lin64/share/pantasks/modules/ipphosts.mhpcc.config). will be used when summitcopy+registration+stdsci are restarted
  • 16:00 CZW stdlocal stopped to avoid possible race conditions as I batch run some LAP scripts.
  • 16:30 CZW stdlocal back up, stdlanl stopped to do the same LAP push, and also clear out runs that have stacks that finished today.

Wednesday : 2015.02.04

  • 07:35 EAM: we have still 5200 stdlocal stacks to do. since we are only doing stacks, I've (a) turned off half of the xnode connections (down to 5 from 9) and (b) bumped up the stack poll (to 750 from 500). I've also turned on the storage hosts.
  • 09:40 EAM: staticsky finished most of the skycells in the queued batch. There were a handful which were still running after >50k seconds. I've killed those jobs -- we can try to re-run them or debug them in the future. I'm am going to queue the galactic plane block for 0h now. The impact on the machines will likely be quite different, so the distribution of machines may need to be modified for the plane. 
      0   ippx001    BUSY  118106.27 0.0.0.2b0b  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 764564 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.1935.063/RINGS.V3.skycell.1935.063.sky.764564 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  1   ippx026    BUSY  58307.66 0.0.0.309d  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 765990 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.015/RINGS.V3.skycell.2159.015.sky.765990 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  2   ippx045    BUSY  58120.00 0.0.0.30a6  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 765999 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.024/RINGS.V3.skycell.2159.024.sky.765999 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  3    ipps08    BUSY  58036.93 0.0.0.30a7  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766000 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.025/RINGS.V3.skycell.2159.025.sky.766000 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  4    ipps14    BUSY  57880.35 0.0.0.30af  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766008 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.033/RINGS.V3.skycell.2159.033.sky.766008 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  5   ippx026    BUSY  57877.06 0.0.0.30b0  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766009 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.034/RINGS.V3.skycell.2159.034.sky.766009 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  6   ippx011    BUSY  57743.95 0.0.0.30b1  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766010 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.035/RINGS.V3.skycell.2159.035.sky.766010 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  7   ippx013    BUSY  57672.66 0.0.0.30b9  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766018 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.043/RINGS.V3.skycell.2159.043.sky.766018 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  8    ipps00    BUSY  57650.65 0.0.0.30ba  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766019 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.044/RINGS.V3.skycell.2159.044.sky.766019 --redirect-output --camera GPC1 --dbname gpc1 --verbose 
  9   ippx004    BUSY  57625.67 0.0.0.30bb  0 staticsky.pl --threads @MAX_THREADS@ --sky_id 766020 --outroot neb://any/gpc1/LAP.PV3.20140730/sky01.20141223//RINGS.V3/skycell.2159.045/RINGS.V3.skycell.2159.045.sky.766020 --redirect-output --cam
  • 12:55 MEH: Haydn needs to power down ipp077, ipp083, ippx044 to do some work but multiple jobs are using them
    • ipp077 neb repair, but likely long running jobs will still be writing to for a bit
    • ipp083 like ipp077 neb repair, but also active in stdlocal -- set to off and clearing
    • ippx044 active in ippsky -- set to off and slowly clearing
    • ippsky writing to any disk so long running jobs may be trying to write to ipp077,083 when go down
    • stdlocal and ippsky may also need to read from these

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD

Note: See TracWiki for help on using the wiki.