PS1 IPP Czar Logs for the week 2012-05-21 - 2012-05-27

(Up to PS1 IPP Czar Logs)

Monday : 2012-05-21

• heather is czar. I'm taking ippc09 out of processing from now until tomorrow - cindy wants to do some tests.

Tuesday : 2012-05-22

• Bill 09:48 dropped lost raw file with: repair_bad_instance -e 229421 -c XY31 -l
• CZW 17:52 After wondering why my chip images were vanishing, I've disabled lap.cleanup until I'm done with a quick test.
• 18:00 Mark: talked with Gene, he's no longer using ipp060 for his tests. taking out of neb-repair and also moving my larger deepstack tests to that disk.

Wednesday : 2012-05-23

• 13:28 CZW: Switched on balance mode in replication pantasks. This should begin migrating data to ippb03. Except they were all failing. Poke, poke, prod, prod, and the answer seems to be that the replicate phase hands the filecopy off to apache. However, the nebulous data directories were owned by ipp.users. apache user is not in the users group, but is in the nebulous group. Checking other hosts shows that those nebulous data directories are owned by ipp.nebulous. I've chowned the directories on ippb03, and things seem to be replicating there correctly. I've also changed the input file to turn on balance on server start.

Thursday : 2012-05-24

• 21:25 Bill We're getting data tonight. Restarted stdscience

Friday : 2012-05-25

Mark is czar

• 06:15 nightly science finished downloading and processing minus a few stuck STS warps (finished). tweaking the MD SSdiffs to run earlier so done before processing stopped @11am.
• 10:45 network maintenance at MHPCC, shutting processing down - all pantasks shutdown, apache on ippc17 stopped, apache on neb servers stopped (thanks Serge), czarpoll/server shutdown.
• 10:55 Serge: Stopped replication slaves (i.e. ippdb02, ippdb03, ippc19, ipp001).
• 12:55 Cindy finished network maintenance, restarting apache neb servers, pantask servers, apache on ippc17, czarpoll/server.

Saturday : 2012-05-26

Sunday : 2012-05-27