PS1 IPP Czar Logs for the week 2013.01.28 - 2013.02.04

(Up to PS1 IPP Czar Logs)

Monday : 2013.01.28

• 09:30 MEH: skycal off while fixing psastro/getstar being dealt with
• 14:00 MEH: all pantasks (deepstack, pstamp, update) on ippc17 have crashed. restarting

Tuesday : 2013.01.30

• 13:35 Serge: Stopped cleanup
• 14:10 Bill: queued 20 M31 frames for chip - warp processing

Wednesday : 2013.01.31

Bill is czar today

• 10:15 tempoarily changed neb-host ipp027 from down to repair. Should it be down? rsyncs are going on
• Fixed 5 broken burntool instances, and reverted the faulted M31 chip runs that depended on them.

  cp /data/ipp027.0/nebulous/b7/c5/370480873.gpc1:20100725:o5402g0452o:o5402g0452o.ota26.burn.tbl /data/ippb02.1/nebulous/b7/c5/899429635.gpc1:20100725:o5402g0452o:o5402g0452o.ota26.burn.tbl
  cp /data/ipp027.0/nebulous/c1/ef/448745447.gpc1:20100915:o5454g0264o:o5454g0264o.ota26.burn.tbl /data/ippb02.0/nebulous/c1/ef/931041890.gpc1:20100915:o5454g0264o:o5454g0264o.ota26.burn.tbl
  cp /data/ipp027.0/nebulous/71/bd/492194663.gpc1:20101015:o5484g0447o:o5484g0447o.ota26.burn.tbl /data/ippb02.0/nebulous/71/bd/901069396.gpc1:20101015:o5484g0447o:o5484g0447o.ota26.burn.tbl
  cp /data/ipp027.0/nebulous/36/2d/535014565.gpc1:20101111:o5511g0093o:o5511g0093o.ota26.burn.tbl /data/ippb02.0/nebulous/36/2d/915499833.gpc1:20101111:o5511g0093o:o5511g0093o.ota26.burn.tbl
  cp /data/ipp027.0/nebulous/c2/2a/573274958.gpc1:20101214:o5544g0015o:o5544g0015o.ota26.burn.tbl /data/ippb02.2/nebulous/c2/2a/904066045.gpc1:20101214:o5544g0015o:o5544g0015o.ota26.burn.tbl
  

• 13:54 Serge: Restarted cleanup
• 13:30 or so enabled skycal
• 15:02 removed label goto_cleaned.rerun to allow the goto_cleaned runs to get cleaned (they have more bytes to recover)
• 15:39 registration pantasks died. Decided to take this opportunity to restart the pantasks. check_system.sh stop
• 15:52 all pantasks restarted (except for cleanup, deepstack, and the addstars of course) cleanup set to stop to prepare to restart (previous attempt failed due to czar error)
• 21:16 many failures due to permission problems on the backup nebulous directories for ipp027 and 028. Setting to down for now.

Thursday : 2013.02.01

Bill is czar again today

• 07:30ish set ipp027 and 28 back to up
• 10:30 Attempted to fix the permissions problem on the backup ipp027 directories

  hopefully this wasn't rash
  (stsci07:stsci07.2/ipp027.0/nebulous) # find . -maxdepth 2 -group users -exec chgrp nebulous {} \;
  (stsci07:stsci07.2/ipp027.0/nebulous) # find . -maxdepth 2 -perm 700 -exec chmod 775 {} \;
  
  Did same thing in
  /data/stsci02.2/ipp028.0/nebulous
  /data/stsci06.2/ipp027.0/nebulous
  
  

• 10:46 we've been getting strange lock timeouts from stacktool -addsumskyfile. This has caused 23% of stacks to fail since we restarted the stack pantasks. As an experiment turned off skycal and staticsky to see whether their load queries might be triggering the problem
• 11:15 stopping stack to attempt to sort out the database problem.

Friday : 2013.02.02

Saturday : 2013.02.03

Sunday : 2013.02.04