== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017.06.05 ===
 * MEH: again fixing stalled stamps for QUB
  
=== Tuesday : 2017.06.06 ===
 * MEH: Heather wondering issue w/ ippx012 being down -- ganglia shows offline early yesterday, console shows
{{{
<Jun/05 06:14 am>ippx012 login: [1679460.508553] ------------[ cut here ]------------
<Jun/05 06:14 am>[1679470.992940] kernel BUG at kernel/timer.c:926!
<Jun/05 06:14 am>[1679471.414703] invalid opcode: 0000 [#1] SMP 
<Jun/05 06:15 am>[1679473.103464] Modules linked in: adm1021 ipv6 dm_mod joydev usbhid microcode igb ehci_hcd uhci_hcd i2c_i801 ioatdma coretemp i2c_core pcspkr usbcore acpi_cpufreq mperf dca usb_common sg processor button thermal_sys
<Jun/05 06:15 am>[1679506.734049] CPU 10 
<Jun/05 06:15 am>[1679506.736226] Pid: 14368, comm: apache2 Not tainted 3.7.6 #1 Supermicro X8DTT/X8DTT
<Jun/05 06:15 am>[1679508.835724] RIP: 0010:[<ffffffff8103882d>]  [<ffffffff8103882d>] add_timer_on+0x34/0x78
<Jun/05 06:15 am>[1679510.520854] RSP: 0018:ffff88183f343e10  EFLAGS: 00010286
}}}
  * no remote power management yet, Haydn reminds would reboot 4 nodes ippx009-x012 and Heather would prefer to wait until next week to just individually reboot w/ physical button press
 * MEH: more fixing of error_cleaned data so pstamp and updates can work 
 * MEH: ippc18/homedir was quickly running out of space -- Heather cleaned up logs to make space for ongoing, regular batch running, the warning cron (ipp@ippc18) has been enabled to alert when gets to minimal levels 
 * MEH: K2.nightlyscience processing chip-cam-warp automated again and should be checked on by czars like the rest of nightly processing for next month or so again -- WSdiffs are still managed externally by me
=== Wednesday : 2017.06.07 ===
 * MEH: another case of nightly_science.pl not doing alternate visit diffims for MOPS -- v2-v3 manually queued
{{{
| exp_id  | exp_name    | filter  | fault | fwhm_major | quality | dateobs             | comment                               |
+---------+-------------+---------+-------+------------+---------+---------------------+---------------------------------------+
| 1255359 | o7911g0410o | i.00000 |     0 |    5.33247 |       0 | 2017-06-07 12:15:44 | OSSR.R21N2.17.Q.i ps1_33_0247 visit 1 | 
| 1255377 | o7911g0428o | i.00000 |     0 |    5.20372 |       0 | 2017-06-07 12:32:56 | OSSR.R21N2.17.Q.i ps1_33_0247 visit 2 | 
| 1255395 | o7911g0446o | i.00000 |     0 |    5.44651 |       0 | 2017-06-07 12:50:08 | OSSR.R21N2.17.Q.i ps1_33_0247 visit 3 | 
| 1255413 | o7911g0464o | i.00000 |     0 |    5.09305 |    4007 | 2017-06-07 13:07:15 | OSSR.R21N2.17.Q.i ps1_33_0247 visit 4 | 
}}}
=== Thursday : YYYY.MM.DD ===

=== Friday : 2017.06.09 ===
 * MEH: MOPS test diffs running on ipps nodes
 * 21:00 EAM : restarted the pantasks

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===