PS1 IPP Czar Logs for the week 2014.05.26 - 2014.06.01

(Up to PS1 IPP Czar Logs)

Monday : 2014.05.26

Tuesday : 2014.05.27

Wednesday : 2014.05.28

  • 07:40 EAM : stdscience running a bit slow so I'm restarting it.
  • 11:45 EAM : graceful shutdown of cluster for planned power outage at MRTC-B
  • 16:15 CZW : All hosts back online, nebdiskd restarted, czartool scripts restarted, all pantasks servers restarted.
  • 16:30 CZW : I've stopped pantasks servers to see if memory issues Mark has noticed will be fixed.
  • 17:00 CZW : Haydn has fixed the ippdb01 memory issue, which was the biggest concern. I've set the pantasks servers to run.
  • 17:15 MEH: rebooted ippc58 to get the memory back -- restarting staticsky now -- no, concerned with setup to use --
    #PV2 staticsky build
    set build = ipp-20140114
    # sas test build
    set build = ipp-trunk-20140404
    

Thursday : 2014.05.29

  • 20:50 MEH: stack nodes idle w/o LAP stacks now, reallocating for MD refstack and staticsky unless nightly MD06 stacks need to be made as discussed at Tuesday's meeting
    • it wasn't noted that LAP staticsky was using the stack nodes.. having to clean up a slight overload going on now..

Friday : 2014.05.30

  • 15:00 MEH: should clear fault 5 diffim..
    difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0942.015 -diff_id 555513  -fault 0
    

Saturday : 2014.05.31

  • 01:40 MEH: looks like registration stalled, >40 exposures backed up. reverting fault, cleared and nightly catching up. not sure why didn't revert as fault 2 -- from log
    2014/05/31 00:54:57 | ipp011 | FATAL | Nebulous::Client::create - unhandled fault - error: mkdir /data/ipp053.0/nebulous: No such file or directory at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 2522
    unable to instantiate neb://ipp053.0/gpc1/20140531/o6808g0324o.746034/o6808g0324o.746034.reg.ota76.log. at /home/panstarrs/ipp/psconfig/ipp-20130712.lin64/bin/register_imfile.pl line 62
    

Sunday : 2014.06.01