PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.12.08

• 08:45 MEH: ippmd using ~280 nodes now that nightly is finished (ippsXX, 2x x2)
• 09:05 MEH: WS diffs queued fine today -- will be stdlocal ~300, ippmd~300, stdsci~300 -- except 600 diffs will shortly shut off chip-warp in stdlocal
• 12:55 MEH: the 20T data nodes with space that had put neb-host up late last week to fill and increase the number of data nodes have filled or behaving poorly -- neb-host repair on all 20T nodes now
  • for the 20T nodes w/o processing things seemed mostly okay -- net in was high >50MB/s and seemed to be writing okay, just forced to write constantly and probably could be put back in, but ones doing processing probably shouldn't be

• 20:35 HAF: various emails floating around, there is a problem with nebulous (mark / gene noticed), and summit copy /registration are faulty. The errors we see are like this:

-> pmConfigConvertFilename (pmConfig.c:1833): System error
     failed to create a new nebulous key: nebclient.c:1012 nebSetServerErr() - SOAP-ENV:Server - error: DBD::mysql::st execute failed: The table 'storage_object' is full at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 275,  line 12.
 -> pmConfigRead (pmConfig.c:618): System error
     Unable to resolve trace destination: neb://ipp015.0/gpc1/ThreePi.nt/2014/12/09//o7000g0063o.833489/o7000g0063o.833489.ch.1316586.XY23.trace
Unable to perform ppImage: 1 at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/chip_imfile.pl line 830
	main::my_die('Unable to perform ppImage: 1', 833489, 1316586, 'XY23', 1) called at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/chip_imfile.pl line 509

• 20:35 HAF: Serge is investigating, notes that ippdb00 is full and is finding the magical incantations to fix that.

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD