(Up to PS1 IPP Czar Logs)

Monday : 2012.11.26

Serge is czar

• 09:30 MEH: pileup of warps in MD04 and LAP processing, nightly science done, chip.off
• 09:50 Set o6256g0498o quality to 0 (exposure time is 0). Reverted the failing 54 stack

  UPDATE rawImfile SET quality=8007 WHERE exp_name = 'o6256g0498o';
  UPDATE warpSkyfile SET quality=8007 WHERE warp_id = 666931;
  DELETE FROM stackInputSkyfile WHERE warp_id = 666931;
  

• 10:25 Serge: Queued about 500 exposures (Fri + Sat + Sun nights) for MOPS PS1_DV3 tests
• 12:05 Serge: Stopping replication from ippdb00 to ippdb02. Rsync /export/ippdb02.0/mysql to /export/ippc63.1/backup_nebulous/20121126 (from ippdb02, screen session: mysql_rsync)
• 12:25 Bill: set deepstack pantasks to stop. I want to modify the staticsky processing order a bit.
• 13:10 MEH: running local deepstack pantasks with a compute3 group to do the ref/deepstacks for the upcoming MD04 field -- moved over to using wave4, took 2x wave4 out of stack pantasks
  • in setting wave4 up, have noticed ipp055, ipp065 do not have full RAM after reboot many months ago and a few months ago respectively. Has happened before on other machines -- after rebooting systems, must check that all RAM is present and accounted for
• 13:42 Bill: set all lap staticskyRuns to state wait with ra > 150.5 and ra < 120 degrees except for the Kepler field. Queued 5 filter skycells 120 < ra < 150.5 set deepstack pantasks to run, then stop. Once the skycells near the galactic center finish processing, we will restart pantasks 2 x compute3.

  mysql> select state, count(sky_id) from staticskyRun left join staticskyResult using(sky_id) 
  where label ='lap.threepi.20120706' group by state;
  +-------+---------------+
  | state | count(sky_id) |
  +-------+---------------+
  | full  |         11897 | 
  | new   |          6417 | 
  | wait  |         34000 |           for simplicity, may drop these and re-queue later. haven't decided
  +-------+---------------+
  3 rows in set (0.47 sec)
  
  

• 14:00 Bill Decided to kill off the currently running psphotStack's so we can restart. Restarted deepstack pantasks with 2 x compute3. No pending runs are at low galactic latitude
• 18:10 MEH: chip.off still to push through more of the pile of LAP warps since no data tonight and stdscience at ~50% normal processing power
• 23:40 MEH: LAP warps down to ~1000, chip.on. MD01 at same priority, should go by _id

Tuesday : 2012-11-27

Heather is czar today

• 09:30 Serge: Restarted replication on ippdb02 (77310 seconds behind master). Monitoring how it catches up ~schastel/dev/ScanReplication/ippdb02.log (hopefully it will catch up).
• 10:00 Serge: Stopped publishing a MD03 (diff_id = 353088) is failing repeatedly
• 10:10 Serge: It's the exposure with a exp_time of 0. Dropped publication

  UPDATE publishRun SET state='drop' WHERE stage_id = 353088;
  

• 10:23 Serge: Started cleanup (I'd like to see how it influences replication rate).
• 11:20 processing shutdown for Haydyn to reboot ipp019, ipp055, ipp065 to attempt in recovering lost memory
• 11:47 Bill rebuilt with bug fixes to psphotDeblendSatstars and psphotReplaceAllSources. Improved a couple of assertions in psLib.
• 11:50 Serge: Setup wiki page for IPP-MOPS ICD comments
• 14:09 Bill ran check_system.sh stop because nebulous is not responding
• 15:20 Serge: Stopped apache on c01-c10 then restarted mysql server on ippdb00 (there were still pending connections after apache servers were stopped) then restarted apache servers.
• 20:00 MEH: looks like ipp031 is stalling jobs (summitcopy, registration, stdscience), taking out of processing. same with ipp052 -- both having mount troubles

Wednesday : 2012.11.28

Mark is czar

• 07:30 MEH: looks like ip040 and ipp047 have stalled registration for 5 hrs.. taken out of processing as well, restarted summitcopy, registration, stdscience as well. some 291 exposures left to register and process.
• 09:10 deepstack stop to reallocate compute3 back to stdscience to get backlog of nightly science through before noon
• 09:20 MEH: MD04 refstack continues to run using stacks wave4 allocation, will be another 24 hours or so. Then the MD05 refstack will need to use them..
• MEH: ipp031,040,047,052 may need to be rebooted after nightly science finished
• stdscience stopped also to fix more mount issues
• 12:20 Chris modified replication config to not do anything for the time being since ippb machines are full, restarting now.
• 12:40 MEH: ipp031 couldn't clear or fix mounts, rebooting.
• 13:05 same for ipp052, processing moving along more now -- while other systems were able to access the disk, looks like maybe for reading only. likely needed to set all problem ones to repair (normally do) or down. restarted registration.
• 13:10 ipp040, like previous 2, nfs reports tainted on a restart. rebooting
• 13:25 ipp047 mounts are back, so set neb-host back up and keep eye on
• 13:40 MEH: ipp010 and ipp011 also having mount troubles. ipp012 as well.
• 14:00 MEH: noticed after trying rpc.statd stop on ipp010, hung on stopping statd because needed to clear all the lost rpc.statd. restart rather than stop/start on rpc.statd, nfs may behave differently. ipp010,011,018 have excessive swap cleared now.
• 14:20 mounts recovered without any additional reboots, will try stop/start on nfs for future mount troubles to look at further. now just want to get nightly science through.
• 15:20 MEH: still using all compute3 for nightly stdscience, LAP and MD01 label out to only do nightly science
• 15:10 Chris made modification to cleanup input file, boost.stack (unboost.stack) to set poll period shorter to load more jobs
• 15:50 Bill rebuild psLib
• 16:00 MEH: nightly science finished, all pantasks restarted for the night. in cleanup do boost.stack to reset poll period faster.
  • the orphan rpc.statd can be cleared by restarting rpc.statd/nfs twice. will see if this helps clear any mount problems in future. ipp007,010,012,013,015,016,108,020 should be cleared

    sudo /etc/init.d/rpc.statd restart
    sudo /etc/init.d/rpc.statd restart
    --> will hang a bit on second restart as it clears the floating rpc.statd
    

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD