PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2012.03.12

Mark is czar

• 07:15 nightly science looks downloaded and processed. One DM07 stack (828514, skycell.011) has completely bad PSF and probably needs to be dropped. Fixed couple XY26 LAP chips.
• 08:30 stack ahoy.. cleared the 5 stalled LAP runs held up by error_cleaned in warp stage (common skycell_ids 2349.041,2349.042,2350.019,2350.053) with

  warptool -dbname gpc1 -tocleanedskyfile -warp_id 383567 -skycell_id skycell.2350.053
  warptool -dbname gpc1  -setskyfiletoupdate -warp_id 383567 -skycell_id skycell.2350.053 -set_label LAP.ThreePi.20110809
  

• 08:50 while most of LAP is in stack, going to restart stdscience.
• 10:50 LAP stack on ippc24 seems stalled >2ks. a suggestion of Chris, logging into the data host ipp008 and running sync seems to have cleared.

  -- stack pantasks
    0    ippc24    BUSY   2349.08 0.0.0.20e5  0 stack_skycell.pl --threads @MAX_THREADS@ --stack_id 829996 --outroot neb://any/gpc1/LAP.ThreePi.20110809/2012/03/12/RINGS.V3/skycell.2572.042/RINGS.V3.skycell.2572.042.stk.829996 --redirect-output --run-state new --reduction THREEPI_STACK --dbname gpc1 --verbose
  
  -- ps on ippc24
  ipp      15381 11400  0 10:38 ?        00:00:00 glockfile /data/ipp008.0/nebulous/e2/07/1954880509.gpc1:LAP.ThreePi.20110809:2012:03:12:RINGS.V3:skycell.2572.042:RINGS.V3.skycell.2572.042.stk.829996.ppStack.mdc xcld 0
  
  

• 12:10 many pending LAP jobs want data on ipp041 (only 1 CPU still and out of processing), Chris has set unwant=7 (5 is default) and 7 remote tasks running. Chris going to try bumping up to 10, system very busy and going to watch to see if anything breaks.
• 13:55 Serge fixed replication issue Replication_Issues
• 17:00 non-existent LAP OTA to drop

  neb://ipp040.0/gpc1/20100921/o5460g0133o/o5460g0133o.ota15.fits 
  
  chiptool -dropprocessedimfile -set_quality 42 -chip_id 426078 -class_id XY15 -dbname gpc1
  regtool -updateprocessedimfile -set_ignored -exp_id 228082 -class_id XY15 -dbname gpc1
  

• 17:30 Chris/Gene investigated ipp015 disk issues. ended up rebooting.
• 18:00 ipp015 back up but the disk not being exported (correctly?). ran exportfs -f. however, ganglia seems to still thinks it is down.
• 19:00 more xy26 LAP chips repaired. oops, actually have 18 more lap_id left to go..

Tuesday : 2012.03.13

Mark is czar

• 07:00 ipp015 is down, and not rebooting... looks like needs someone to physically go check out.
• 07:30 setting neb-host down ipp015, stopping and shutting down most pantasks while attempting to untangle things that needed ipp015. not sure what to do about addstar.
• 08:00 turning summitcopy back on (while going into office), seems to be downloading somewhat.
• 09:10 Serge: Nebulous snafu. Restarted the mysql server on ippdb00.
• 10:05 Serge: All apache/nebulous servers have been restarted.
• 11:00 still killing off hung processes.
• 11:10 notified by Rita that the AC @ATRC is down, room ~91F but now open and fans on. no temperature info on ​http://ps1wiki.ifa.hawaii.edu/cgi-bin/ippDashboard.py but not anticipating any load on those machines. should they be powered down anyways?
• 11:30 restarting things, putting ipp015 into the hosts_ignore_wave1 list (do we want to comment it out completely in the wave1 list?). Bill taking care of postage stamp server.
• 12:13 CZW: I've modified a copy of repair_bad_instance, and fed it with an input list generated from Serge's slocate databases of ota26 entries on ipp064. It's currently running singly threaded, and checking all of these raw fits files and repairing bad instances when possible. This is copying good copies from the ATRC, but as it should not be doing more than one at a time, I do not expect this to overly tax the ATRC computers/ATRC AC.
• 12:15 Serge: Killed ippdb03 scheduled mysqldump to perform a manual one that I intend to ingest on the broken ipp001. Replication on ippdb03 has been stopped on purpose.
• 12:40 nightly science cam/warp/stack faults mostly due to processed chip data needed that lives on ipp015. same goes for LAP so LAP is effectively stalled.
• 12:50 replication pantasks back running, target.on only.
• 13:00 Gavin setup access to the power control for us on compute3 nodes.
• 14:10 Serge: Restarted replication on ippdb03. CHANGE MASTER TO MASTER_LOG_FILE='mysqld-bin.020936', MASTER_LOG_POS=73345738;
• 14:30 registration has been very unhappy. ipp011 often having lockd server not responding for ipp008,ipp014 so set those two to repair and will watch with tonight's data.
• 15:00 all data downloaded and registered from last night, finally. There will be some data stuck (3 MD06 warps, 3 MD07 stacks, 14 3PI warps, 7 3PI camera, 1 3PI diff) that are waiting for access to intermediate processing products on ipp015. Turning chip/camera/warp.revert.off for now and will toggle manually every so often as remaining science data moves through.
• 16:00 extra LAP run queued to see if will process without needing any past processed products from ipp015.
• 16:30 Chris bumped up the stdscience unwant=10 again since it was reset back to 5 after the pantasks restart. May just remove ipp015 from pantasks (rather than off) tonight if still running slow waiting for targeted ipp015 jobs to be triggered remotely.
• 16:50 Rita reports that the ATRC room temperature is still ~90F, Gene is going to power down the ippb00-03 machines. AC service not until Thursday afternoon.
• 22:00 looks like lockd also having troubles with ipp009, ipp016 in addition to ipp008,ipp014 seen earlier.

Wednesday : 2012-03-13

Serge is czar. TODO list:

• If no news from Rita/Haydn at 930am, MHPCC phone number is 808-879-5077 (ask for server room working on PanSTARRS).
• Start mysql replication slave on ipp001
• Remove ippc50 from processing list

• 06:34 EAM: ipp014 hung, rebooted it (looks like a cpu error)
• 08:30 Serge: Shutting down ippc50
• 08:52 Serge: ippc50 is down

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD