PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2012-06-04

Mark is czar

• 06:34 Serge: ingestion of the instance table finished at 5:26 this morning. The storage_object table is now being ingested.
• 06:50 Mark: limited nightly science data, downloaded and processed.
• 08:50 Serge: Estimate of 25% of storage_object table ingested (based on so_id assuming there are no "holes" in the table).
• 12:15 Serge: 50% of storage_object table ingested (based on so_id assuming there are no "holes" in the table).
• 14:35 Serge: Cleaned ipp009:/tmp
• 17:15 Mark: replication regularly show not running/up on czarpage when it is. been very busy, going to restart the pantasks.
• 20:00 Mark: turning chip.revert.on, please do not revert anything for MD09.
• 21:50 Serge: ingestion of storage_object table complete. The full ingestion should finish in a couple of hours, likely earlier.

Tuesday : 2012-06-05

Serge is czar

• 06:35 Serge: ingestion of nebulous complete (at 0:02am).
• 06:40 Serge: 5 exposures were not registered. I manually replayed the failing from ipp@ippc18

  register_imfile.pl --exp_id 493380 --tmp_class_id ota11 --tmp_exp_name o6082g0100o 
  --uri neb://ipp056.0/gpc1/20120604/o6082g0100o/o6082g0100o.ota11.fits 
  --logfile neb://ipp056.0/gpc1/20120604/o6082g0100o.493380/o6082g0100o.493380.reg.ota11.log --bytes 23716800 
  --md5sum 37827e9387ccfebffff5d115dabaddcf --sunset 03:30:00 --sunrise 17:30:00 
  --summit_dateobs 2012-06-04T07:00:13.000000 --dbname gpc1 --verbose
  

which fixed the registration problem.

• 07:10 Serge: Just forgot to mention that: I didn't revert anything for MD09 ;)
• 07:15 Mark: turning on the diffims for MD09, adding 2012-06-01 to pick up MD09-y observed then, tweaking SSdiff run time for 7:30am. (and thanks Serge)
• 10:30 Serge: Stopping all pantasks from ippc18; czarpoll and roboczar on ipp009; addstar/addstarlap as ippdbvo from ippc18
• 11:45 Serge: Replaying the binlogs is AWFULLY long! The first one isn't finished yet.
• 12:10 Serge: Restarted apache, pantasks but I did target.off and balance.off for replication, apache on ippc17, and czar monitoring on ipp009.
• 12:18 Serge: register.add.date 2012-06-05 gpc1 14 since no date was shown in ippMonitor.
• 13:30 Serge: First binlog ingested... 3 hours. I'm trying to play them without the COMMIT statements that happen every other line.
• 13:45 Serge: The second binlog ingestion is slow as well... because of the BEGIN statements now.
• 15:00 Serge: Second binlog ingestion complete. Further ingestions executed with: 'mysqlbinlog <binlog> | grep -v 'COMMIT/*!*/' | grep -v BEGIN | mysql -u root -p nebulous'.

Wednesday : 2012-06-06

Serge is czar

• 08:00 Serge: Nightly science processing complete

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD