PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2012-06-04

Mark is czar

• 06:34 Serge: ingestion of the instance table finished at 5:26 this morning. The storage_object table is now being ingested.
• 06:50 Mark: limited nightly science data, downloaded and processed.
• 08:50 Serge: Estimate of 25% of storage_object table ingested (based on so_id assuming there are no "holes" in the table).
• 12:15 Serge: 50% of storage_object table ingested (based on so_id assuming there are no "holes" in the table).
• 14:35 Serge: Cleaned ipp009:/tmp
• 17:15 Mark: replication regularly show not running/up on czarpage when it is. been very busy, going to restart the pantasks.
• 20:00 Mark: turning chip.revert.on, please do not revert anything for MD09.
• 21:50 Serge: ingestion of storage_object table complete. The full ingestion should finish in a couple of hours, likely earlier.

Tuesday : 2012-06-05

Serge is czar

• 06:35 Serge: ingestion of nebulous complete (at 0:02am).
• 06:40 Serge: 5 exposures were not registered. I manually replayed the failing from ipp@ippc18

  register_imfile.pl --exp_id 493380 --tmp_class_id ota11 --tmp_exp_name o6082g0100o 
  --uri neb://ipp056.0/gpc1/20120604/o6082g0100o/o6082g0100o.ota11.fits 
  --logfile neb://ipp056.0/gpc1/20120604/o6082g0100o.493380/o6082g0100o.493380.reg.ota11.log --bytes 23716800 
  --md5sum 37827e9387ccfebffff5d115dabaddcf --sunset 03:30:00 --sunrise 17:30:00 
  --summit_dateobs 2012-06-04T07:00:13.000000 --dbname gpc1 --verbose
  

which fixed the registration problem.

• 07:10 Serge: Just forgot to mention that: I didn't revert anything for MD09 ;)
• 07:15 Mark: turning on the diffims for MD09, adding 2012-06-01 to pick up MD09-y observed then, tweaking SSdiff run time for 7:30am. (and thanks Serge)
• 10:30 Serge: Stopping all pantasks from ippc18; czarpoll and roboczar on ipp009; addstar/addstarlap as ippdbvo from ippc18
• 11:45 Serge: Replaying the binlogs is AWFULLY long! The first one isn't finished yet.
• 12:10 Serge: Restarted apache, pantasks but I did target.off and balance.off for replication, apache on ippc17, and czar monitoring on ipp009.
• 12:18 Serge: register.add.date 2012-06-05 gpc1 14 since no date was shown in ippMonitor.
• 13:30 Serge: First binlog ingested... 3 hours. I'm trying to play them without the COMMIT statements that happen every other line.
• 13:45 Serge: The second binlog ingestion is slow as well... because of the BEGIN statements now.
• 15:00 Serge: Second binlog ingestion complete. Further ingestions executed with: 'mysqlbinlog <binlog> | grep -v 'COMMIT/*!*/' | grep -v BEGIN | mysql -u root -p nebulous'.
• 23:00 Mark: queuing some few 1000 MD09.GR0 nightly stack reprocessing. not putting label in stdscience to avoid problem chips reverting (both have same label), so will be ghost activity.

Wednesday : 2012-06-06

Serge is czar

• 08:00 Serge: Nightly science processing complete
• 08:50 Serge: nebulous binlogs: 9/13 completed.
• 10:40 Mark: MD09.GR0 nightly stacks finished. turning chip.revert.off to load onto datastore and not revert the problematic chips. final refstack i,r will be started in the deepstack pantasks to run over next day or so.
• 13:10 Serge: Changed ippdb00:/etc/nebdiskd.rc configuration so that nebulous.ipp.ifa.hawaii.edu is used. Restarted nebdiskd on ippdb00
• 16:15 Serge: ippc45 died around 15:00 but someone is already connected to it...
• 16:40 Serge: binlogs since yesterday (i.e. with balance and shuffling) have all been played. I'm playing now the 2 binlogs that have been generated since yesterday. Hopefully playing them will not be as long as the other ones...
• 16:45 Serge: From Gavin: "someone (onsite) disconnected ippc45 from the switch".
• 16:55 Serge: Looks like ippc45 is back
• 22:30 Serge: Playing mysqld-bin.000695 (ipp screen session 28803.pts-6.ippdb00)
• 23:15 Mark: looks like something pushed ipp062 into an odd state around 22:10 and has become unresponsive/unable to ssh to.
• 23:20 queuing MD09.refstack.20120603.5x20120606 staticsky with previous ops tag ipp-20120404 since new tag joins skycell table which doesn't have MD for tess_id loaded.
• 23:45 gave it 30 minutes and is blocking download/registration/processing. unable to login to system via console either. nothing on console display, cycling power

Thursday : 2012-06-07

Serge is czar

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD