== PS1 IPP Czar Logs for the week 2013.01.28 - 2013.02.04 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.01.28 ===
 * 09:30 MEH: skycal off while fixing psastro/getstar being dealt with
 * 14:00 MEH: all pantasks (deepstack, pstamp, update) on ippc17 have crashed. restarting
=== Tuesday : 2013.01.30 ===
 * 13:35 Serge: Stopped cleanup
 * 14:10 Bill: queued 20 M31 frames for chip - warp processing

=== Wednesday : 2013.01.31 ===
Bill is czar today
 * 10:15 tempoarily changed neb-host ipp027 from down to repair. Should it be down? rsyncs are going on
 * Fixed 5 broken burntool instances, and reverted the faulted M31 chip runs that depended on them.
{{{
cp /data/ipp027.0/nebulous/b7/c5/370480873.gpc1:20100725:o5402g0452o:o5402g0452o.ota26.burn.tbl /data/ippb02.1/nebulous/b7/c5/899429635.gpc1:20100725:o5402g0452o:o5402g0452o.ota26.burn.tbl
cp /data/ipp027.0/nebulous/c1/ef/448745447.gpc1:20100915:o5454g0264o:o5454g0264o.ota26.burn.tbl /data/ippb02.0/nebulous/c1/ef/931041890.gpc1:20100915:o5454g0264o:o5454g0264o.ota26.burn.tbl
cp /data/ipp027.0/nebulous/71/bd/492194663.gpc1:20101015:o5484g0447o:o5484g0447o.ota26.burn.tbl /data/ippb02.0/nebulous/71/bd/901069396.gpc1:20101015:o5484g0447o:o5484g0447o.ota26.burn.tbl
cp /data/ipp027.0/nebulous/36/2d/535014565.gpc1:20101111:o5511g0093o:o5511g0093o.ota26.burn.tbl /data/ippb02.0/nebulous/36/2d/915499833.gpc1:20101111:o5511g0093o:o5511g0093o.ota26.burn.tbl
cp /data/ipp027.0/nebulous/c2/2a/573274958.gpc1:20101214:o5544g0015o:o5544g0015o.ota26.burn.tbl /data/ippb02.2/nebulous/c2/2a/904066045.gpc1:20101214:o5544g0015o:o5544g0015o.ota26.burn.tbl
}}}
 * 13:54 Serge: Restarted cleanup
 * 13:30 or so enabled skycal
 * 15:02 removed label goto_cleaned.rerun to allow the goto_cleaned runs to get cleaned (they have more bytes to recover)
 * 15:39 registration pantasks died. Decided to take this opportunity to restart the pantasks. check_system.sh stop
 * 15:52 all pantasks restarted (except for cleanup, deepstack, and the addstars of course) cleanup set to stop to prepare to restart (previous attempt failed due to czar error)
 * 21:16 many failures due to permission problems on the backup nebulous directories for ipp027 and 028. Setting to down for now.
=== Thursday : 2013.02.01 ===
Bill is czar again today
 * 07:30ish set ipp027 and 28 back to up
 * 10:30 Attempted to fix the permissions problem on the backup ipp027 directories
{{{
hopefully this wasn't rash
(stsci07:stsci07.2/ipp027.0/nebulous) # find . -maxdepth 2 -group users -exec chgrp nebulous {} \;
(stsci07:stsci07.2/ipp027.0/nebulous) # find . -maxdepth 2 -perm 700 -exec chmod 775 {} \;

Did same thing in
/data/stsci02.2/ipp028.0/nebulous
/data/stsci06.2/ipp027.0/nebulous

}}}
 * 10:46 we've been getting strange lock timeouts from stacktool -addsumskyfile. This has caused 23% of stacks to fail since we restarted the stack pantasks. As an experiment turned off skycal and staticsky to see whether their load queries might be triggering the problem
 * 11:15 stopping stack to attempt to sort out the database problem.
=== Friday : 2013.02.02 ===

=== Saturday : 2013.02.03 ===

=== Sunday : 2013.02.04 ===