== PS1 IPP Czar Logs for the week 2014.05.26 - 2014.06.01 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.05.26 ===
  
=== Tuesday : 2014.05.27 ===

=== Wednesday : 2014.05.28 ===

 * 07:40 EAM : stdscience running a bit slow so I'm restarting it.
 * 11:45 EAM : graceful shutdown of cluster for planned power outage at MRTC-B
 * 16:15 CZW : All hosts back online, nebdiskd restarted, czartool scripts restarted, all pantasks servers restarted.
 * 16:30 CZW : I've stopped pantasks servers to see if memory issues Mark has noticed will be fixed.
 * 17:00 CZW : Haydn has fixed the ippdb01 memory issue, which was the biggest concern.  I've set the pantasks servers to run.
 * 17:15 MEH: rebooted ippc58 to get the memory back -- restarting staticsky now -- no, concerned with setup to use -- 
{{{
#PV2 staticsky build
set build = ipp-20140114
# sas test build
set build = ipp-trunk-20140404
}}}
=== Thursday : 2014.05.29 ===
 * 20:50 MEH: stack nodes idle w/o LAP stacks now, reallocating for MD refstack and staticsky unless nightly MD06 stacks need to be made as discussed at Tuesday's meeting
  * it wasn't noted that LAP staticsky was using the stack nodes.. having to clean up a slight overload going on now..
=== Friday : 2014.05.30 ===
 * 15:00 MEH: should clear fault 5 diffim..
{{{
difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0942.015 -diff_id 555513  -fault 0
}}}

=== Saturday : 2014.05.31 ===
 * 01:40 MEH: looks like registration stalled, >40 exposures backed up. reverting fault, cleared and nightly catching up. not sure why didn't revert as fault 2 -- from log
{{{
2014/05/31 00:54:57 | ipp011 | FATAL | Nebulous::Client::create - unhandled fault - error: mkdir /data/ipp053.0/nebulous: No such file or directory at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 2522
unable to instantiate neb://ipp053.0/gpc1/20140531/o6808g0324o.746034/o6808g0324o.746034.reg.ota76.log. at /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/register_imfile.pl line 62
}}}
=== Sunday : 2014.06.01 ===