[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.11.26 ===
Serge is czar
 * 09:30 MEH: pileup of warps in MD04 and LAP processing, nightly science done, chip.off
 * 09:50 Set o6256g0498o quality to 0 (exposure time is 0). Reverted the failing 54 stack
{{{
UPDATE rawImfile SET quality=8007 WHERE exp_name = 'o6256g0498o';
UPDATE warpSkyfile SET quality=8007 WHERE warp_id = 666931;
DELETE FROM stackInputSkyfile WHERE warp_id = 666931;
}}}
 * 10:25 Serge: Queued about 500 exposures (Fri + Sat + Sun nights) for MOPS PS1_DV3 tests
 * 12:05 Serge: Stopping replication from ippdb00 to ippdb02. Rsync /export/ippdb02.0/mysql to /export/ippc63.1/backup_nebulous/20121126 (from ippdb02, screen session: mysql_rsync)
 * 12:25 Bill: set deepstack pantasks to stop. I want to modify the staticsky processing order a bit.
 * 13:10 MEH: running local deepstack pantasks with a compute3 group to do the ref/deepstacks for the upcoming MD04 field -- moved over to using wave4, took 2x wave4 out of stack pantasks 
  * in setting wave4 up, have noticed ipp055, ipp065 do not have full RAM after reboot many months ago and a few months ago respectively. Has happened before on other machines -- after rebooting systems,  must check that all RAM is present and accounted for
 * 13:42 Bill: set all lap staticskyRuns to state wait with ra > 150.5 and ra < 120 degrees except for the Kepler field. Queued 5 filter skycells 120 < ra < 150.5 set deepstack pantasks to run, then stop. Once the skycells near the galactic center finish processing, we will restart pantasks 2 x compute3. 
{{{
mysql> select state, count(sky_id) from staticskyRun left join staticskyResult using(sky_id) 
where label ='lap.threepi.20120706' group by state;
+-------+---------------+
| state | count(sky_id) |
+-------+---------------+
| full  |         11897 | 
| new   |          6417 | 
| wait  |         34000 |           for simplicity, may drop these and re-queue later. haven't decided
+-------+---------------+
3 rows in set (0.47 sec)

}}}
 * 14:00 Bill Decided to kill off the currently running psphotStack's so we can restart. Restarted deepstack pantasks with 2 x compute3. No pending runs are at low galactic latitude
 * 18:10 MEH: chip.off still to push through more of the pile of LAP warps since no data tonight and stdscience at ~50% normal processing power
 * 23:40 MEH: LAP warps down to ~1000, chip.on.  MD01 at same priority, should go by _id
=== Tuesday : YYYY.MM.DD ===
 * 09:30 Serge: Restarted replication on ippdb02 (77310 seconds behind master). Monitoring how it catches up ~schastel/dev/ScanReplication/ippdb02.log (hopefully it  will catch up).
 * 10:00 Serge: Stopped publishing a MD03 (diff_id = 353088) is failing repeatedly
 * 10:10 Serge: It's the exposure with a exp_time of 0. Dropped publication
{{{
UPDATE publishRun SET state='drop' WHERE stage_id = 353088;
}}}
 
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===