PS1_IPP_CzarLog_20120528 – Pan-STARRS IPP

wiki:PS1_IPP_CzarLog_20120528

Context Navigation

Version 36 (modified by Mark Huber, 14 years ago) ( diff )
--

PS1 IPP Czar Logs for the week 2012.05.28 - 2012.06.03

1. PS1 IPP Czar Logs for the week 2012.05.28 - 2012.06.03

(Up to PS1 IPP Czar Logs)

Monday : 2012.05.28

few MD08 exposures last night, look to be through okay. looking at if the new masking Chris setup is being used and if a problem.

http://ippmonitor.ipp.ifa.hawaii.edu/ippMonitor/showimage.php?name=neb://any/gpc1/MD08.nt/2012/05/28//o6075g0034o.492281/o6075g0034o.492281.cm.443364&rule=PPIMAGE.JPEG1&camera=GPC1&class_id=NONE

http://ippmonitor.ipp.ifa.hawaii.edu/ippMonitor/showimage.php?name=neb://any/gpc1/MD08.nt/2012/05/20//o6067g0308o.490670/o6067g0308o.490670.cm.439923&rule=PPIMAGE.JPEG1&camera=GPC1&class_id=NONE

looks like more masking so should probably revert masks for now

| exp_id | exp_name    | label                      | ref_static | ref_dyna | ref_advis | max_static | max_dyna | max_advis |
+--------+-------------+----------------------------+------------+----------+-----------+------------+----------+-----------+
| 490670 | o6067g0308o | MD08.20120520              | 0.197      | 0.000234 | 0.0201    | 0.211      | 0.000223 | 0.0196    | 
| 490672 | o6067g0309o | MD08.20120520              | 0.197      | 0.000280 | 0.0213    | 0.211      | 0.000300 | 0.0208    | 

| 491784 | o6072g0163o | MD08.20120525              | 0.198      | 0.000255 | 0.0153    | 0.212      | 0.000275 | 0.0149    | 
| 491785 | o6072g0164o | MD08.20120525              | 0.199      | 0.000246 | 0.0164    | 0.213      | 0.000243 | 0.0159    | 

| 492281 | o6075g0034o | MD08.20120528              | 0.240      | 0.000245 | 0.0060    | 0.253      | 0.000234 | 0.0058    | 
| 492282 | o6075g0035o | MD08.20120528              | 0.238      | 0.000258 | 0.0079    | 0.252      | 0.000247 | 0.0076    |

Tuesday : 2012.05.29

Mark is czar

no data last night. reverted to previous mask set using a script Chris setup since his new ones to remove static issues due to crosstalk bleeds/etc was masking more than expected/desired.
11:00 Serge: stopped replication save on ippdb02 to give mysqldump a chance to finish. Changed nebulous backup copy on ipp@ipp001 to 20:05 (instead of 12:05).

15:40 re-ran SAS.Footprint to verify masks reverted. looks like they are.

| exp_id | exp_name    | label                      | ref_static | ref_dyna | ref_advis | max_static | max_dyna | max_advis |
+--------+-------------+----------------------------+------------+----------+-----------+------------+----------+-----------+
| 356282 | o5744g0329o | czw.footprint.test21       | 0.195      | 0.000031 | 0.0102    | 0.209      | 0.000029 | 0.0099    | 
| 356282 | o5744g0329o | czw.footprint.test120525   | 0.225      | 0.000029 | 0.0096    | 0.238      | 0.000027 | 0.0094    | 
| 356282 | o5744g0329o | czw.footprint.test120525v2 | 0.238      | 0.000029 | 0.0094    | 0.251      | 0.000027 | 0.0092    | 
| 356282 | o5744g0329o | czw.footprint.test120529   | 0.195      | 0.000031 | 0.0102    | 0.209      | 0.000029 | 0.0099    | 
| 356283 | o5744g0330o | czw.footprint.test21       | 0.195      | 0.000036 | 0.0106    | 0.209      | 0.000034 | 0.0104    | 
| 356283 | o5744g0330o | czw.footprint.test120525   | 0.224      | 0.000034 | 0.0101    | 0.237      | 0.000032 | 0.0099    | 
| 356283 | o5744g0330o | czw.footprint.test120525v2 | 0.238      | 0.000034 | 0.0099    | 0.251      | 0.000031 | 0.0097    | 
| 356283 | o5744g0330o | czw.footprint.test120529   | 0.195      | 0.000036 | 0.0106    | 0.209      | 0.000034 | 0.0104    |

15:50 shutting down all pantasks and turning apache off on ippc17 while Serge does some DB work.
16:15 Serge is now in charge.
16:30 Serge: restarted slave on ippdb02. All apache nebulous conf files are now pointing towards ippdb02. I will wait till 8pm for jt scripts running on ippc19 to finish.
16:35 Serge: nebulous slave is now synchronized with its master.
17:10 Serge: the things to be done are described here (just in case I have a moped accident)

Wednesday : 2012-05-30

Serge is czar

No observation last night due to a problem with the primary mirror actuator system. Repair is scheduled for 5-30-2012 HST (quoted from ps-obs report)
Nebulous maintenance is happening and details can be seen in the same reporting wikipage as yesterday
12:00 Started optimization of gpc1 on ippdb01
15:25 All but the last two versions of src/ipp-<tag> are being moved to /export/ippc18.1/ipp/src/Archives/
16:35 Moved ~ipp/iasc to ippc18.1. Symbolic link in ~ipp
17:08 End of gpc1 optimization: 195GB before. 167GB now.
17:10 No change for the ippdb00 ingestion completion: 6am.

Thursday :2012-05-31

Serge is czar

09:30 ippdb00 ingestion is still happening
11:00 Processing has been restarted
11:57 Processing stopped. The new tag is being built.
13:05 All pantasks servers (but ippdvo related) have been restarted

14:05 mysql in ippdb00 has crashed. From /var/log/message:

May 31 14:01:02 ippdb00 [26276891.033202] 3w-9xxx: scsi5: ERROR: (0x06:0x0010): Microcontroller Error: clearing.
May 31 14:01:32 ippdb00 [26276921.659024] sd 5:0:0:0: WARNING: (0x06:0x002C): Command (0x35) timed out, resetting card.
May 31 14:02:02 ippdb00 [26276951.032734] 3w-9xxx: scsi5: WARNING: (0x06:0x0037): Character ioctl (0x108) timed out, resetting card.
May 31 14:02:27 ippdb00 [26276975.998987] 3w-9xxx: scsi5: AEN: INFO (0x04:0x0063): Enclosure added:encl=0.
May 31 14:02:52 ippdb00 [26277001.122317] 3w-9xxx: scsi5: AEN: INFO (0x04:0x0063): Enclosure added:encl=0.
May 31 14:03:02 ippdb00 [26277011.439889] end_request: I/O error, dev sda, sector 2080656881
May 31 14:03:02 ippdb00 [26277011.439931] I/O error in filesystem ("sda4") meta-data dev sda4 block 0x74603200       ("xlog_iodone") error 5 buf count 262144
May 31 14:03:02 ippdb00 [26277011.439936] xfs_force_shutdown(sda4,0x2) called from line 1062 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa00c4652
May 31 14:03:02 ippdb00 [26277011.439961] Filesystem "sda4": Log I/O Error Detected.  Shutting down filesystem: sda4
May 31 14:03:02 ippdb00 [26277011.439964] Please umount the filesystem, and rectify the problem(s)
May 31 14:03:17 ippdb00 [26277025.812318] Filesystem "sda4": xfs_log_force: error 5 returned.
[...]
May 31 14:08:26 ippdb00 [26277335.389888] 3w-9xxx: scsi5: AEN: INFO (0x04:0x001A): Drive inserted:vport=23.
[...]
May 31 14:19:23 ippdb00 [26277992.304317] Filesystem "sda4": xfs_log_force: error 5 returned.
May 31 14:19:23 ippdb00 [26277992.304323] xfs_force_shutdown(sda4,0x1) called from line 420 of file fs/xfs/xfs_rw.c.  Return address = 0xffffffffa00d6a4d
[...]

15:40 Mark changed ~ipp/ippconfig/site.config and gpc1/psastro.config to use new refcat.20120524.v0, rebuilt ippconfig (stopped/restarted pstamp and replication pantasks)
23:00 Mark: yay nightly science data. SAStest.v5 chip->warp finished, running the 5x600 stacks. adding 2xcompute3 into stack pantasks (no deepstack running) and setting SAStest.v5 priority to 200.
23:40 Mark: fun.. looks like the MD diffims are failing, ends with "missing sources?". will look at images in the morning.

Friday : 2012.06.01