[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.11.26 ===
Serge is czar
 * 09:30 MEH: pileup of warps in MD04 and LAP processing, nightly science done, chip.off
 * 09:50 Set o6256g0498o quality to 0 (exposure time is 0). Reverted the failing 54 stack
{{{
UPDATE rawImfile SET quality=8007 WHERE exp_name = 'o6256g0498o';
UPDATE warpSkyfile SET quality=8007 WHERE warp_id = 666931;
DELETE FROM stackInputSkyfile WHERE warp_id = 666931;
}}}
 * 10:25 Serge: Queued about 500 exposures (Fri + Sat + Sun nights) for MOPS PS1_DV3 tests
 * 12:05 Serge: Stopping replication from ippdb00 to ippdb02. Rsync /export/ippdb02.0/mysql to /export/ippc63.1/backup_nebulous/20121126 (from ippdb02, screen session: mysql_rsync)
 * 12:25 Bill: set deepstack pantasks to stop. I want to modify the staticsky processing order a bit.
 * 13:10 MEH: running local deepstack pantasks with a compute3 group to do the ref/deepstacks for the upcoming MD04 field -- moved over to using wave4, took 2x wave4 out of stack pantasks 
  * in setting wave4 up, have noticed ipp055, ipp065 do not have full RAM after reboot many months ago and a few months ago respectively. Has happened before on other machines -- after rebooting systems,  must check that all RAM is present and accounted for
 * 13:42 Bill: set all lap staticskyRuns to state wait with ra > 150.5 and ra < 120 degrees except for the Kepler field. Queued 5 filter skycells 120 < ra < 150.5 set deepstack pantasks to run, then stop. Once the skycells near the galactic center finish processing, we will restart pantasks 2 x compute3. 
{{{
mysql> select state, count(sky_id) from staticskyRun left join staticskyResult using(sky_id) 
where label ='lap.threepi.20120706' group by state;
+-------+---------------+
| state | count(sky_id) |
+-------+---------------+
| full  |         11897 | 
| new   |          6417 | 
| wait  |         34000 |           for simplicity, may drop these and re-queue later. haven't decided
+-------+---------------+
3 rows in set (0.47 sec)

}}}
 * 14:00 Bill Decided to kill off the currently running psphotStack's so we can restart. Restarted deepstack pantasks with 2 x compute3. No pending runs are at low galactic latitude
 * 18:10 MEH: chip.off still to push through more of the pile of LAP warps since no data tonight and stdscience at ~50% normal processing power
 * 23:40 MEH: LAP warps down to ~1000, chip.on.  MD01 at same priority, should go by _id
=== Tuesday : 2012-11-27 ===
Heather is czar today

 * 09:30 Serge: Restarted replication on ippdb02 (77310 seconds behind master). Monitoring how it catches up ~schastel/dev/ScanReplication/ippdb02.log (hopefully it  will catch up).
 * 10:00 Serge: Stopped publishing a MD03 (diff_id = 353088) is failing repeatedly
 * 10:10 Serge: It's the exposure with a exp_time of 0. Dropped publication
{{{
UPDATE publishRun SET state='drop' WHERE stage_id = 353088;
}}}
 * 10:23 Serge: Started cleanup (I'd like to see how it influences replication rate).
 * 11:20 processing shutdown for Haydyn to reboot ipp019, ipp055, ipp065 to attempt in recovering lost memory
 * 11:47 Bill rebuilt with bug fixes to psphotDeblendSatstars and psphotReplaceAllSources. Improved a couple of assertions in psLib.
 * 11:50 Serge: Setup wiki page for [wiki:IPP_MOPS_ICD_comments IPP-MOPS ICD comments]
 * 14:09 Bill ran check_system.sh stop because nebulous is not responding
 * 15:20 Serge: Stopped apache on c01-c10 then restarted mysql server on ippdb00 (there were still pending connections after apache servers were stopped) then restarted apache servers.
 * 20:00 MEH: looks like ipp031 is stalling jobs (summitcopy, registration, stdscience), taking out of processing. same with ipp052 -- both having mount troubles
=== Wednesday : 2012.11.28 ===
 Mark is czar
 * 07:30 MEH: looks like ip040 and ipp047 have stalled registration for 5 hrs.. taken out of processing as well, restarted summitcopy, registration, stdscience as well. some 291 exposures left to register and process. 
 * 09:10 deepstack stop to reallocate compute3 back to stdscience to get backlog of nightly science through before noon
 * 09:20 MEH: MD04 refstack continues to run using stacks wave4 allocation, will be another 24 hours or so. Then the MD05 refstack will need to use them..
 * MEH: ipp031,040,047,052 may need to be rebooted after nightly science finished 
 * stdscience stopped also to fix more mount issues
 * 12:20 Chris modified replication config to not do anything for the time being since ippb machines are full, restarting now.
 * 12:40 MEH: ipp031 couldn't clear or fix mounts, rebooting. 
 * 13:05 same for ipp052, processing moving along more now -- while other systems were able to access the disk, looks like maybe for reading only. likely needed to set all problem ones to repair (normally do) or down. restarted registration.
 * 13:10 ipp040, like previous 2, nfs reports tainted on a restart. rebooting
 * 13:25 ipp047 mounts are back, so set neb-host back up and keep eye on
 * 13:40 MEH: ipp010 and ipp011 also having mount troubles. ipp012 as well.
 * 14:00 MEH: noticed after trying rpc.statd stop on ipp010, hung on stopping statd because needed to clear all the lost rpc.statd. restart rather than stop/start on rpc.statd, nfs may behave differently. ipp010,011,018 have excessive swap cleared now.
 * 14:20 mounts recovered without any additional reboots, will try stop/start on nfs for future mount troubles to look at further. now just want to get nightly science through. 
 * 15:20 MEH: still using all compute3 for nightly stdscience, LAP and MD01 label out to only do nightly science
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===