PS1 IPP Czar Logs for the week 2014.09.22 - 2014.09.28

(Up to PS1 IPP Czar Logs)

Monday : 2014.09.22

• 09:35 EAM : processing went OK last night, but there are a number of outstanding failures related to glockfile failures. I am stopping processing to clear out these lock issues.
• 10:40 EAM : I needed to reboot ipp058 so it could run glockfile on remote machines. nightly science processing then finished and I re-started stdlocal
• 13:15 EAM : ipp013 crashed. I power cycled it and it came back up with help from Haydn (BIOS setting?)
• 22:33 EAM : ipp013 was overloaded and crashed again. i rebooted it, but needed to take it out of all pantasks. i am stopping and restarting standard pantasks. i'm also restarting the ippdb01 db which has been extra sluggish.

Tuesday : YYYY.MM.DD

• HAF: fun times after gpc1 reboot - restarted czartool
• HAF: requeing manually diff for serge:

  o6923g0113o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_5599 visit 3
  o6923g0115o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_2310 visit 3
  o6923g0130o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_5599 visit 4
  o6923g0132o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_2310 visit 4
  

used these commands

[heather@ippc18 ~]$ difftool -dbname gpc1 -definewarpwarp -exp_id 798994 -template_exp_id 799008 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20140923 -set_reduction SWEETSPOT -simple -rerun 
597963 new neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 OSS.nightlyscience OSS.20140923 SweetSpot SWEETSPOT  2014-09-23T21:51:33.763077 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 
597964 new neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 OSS.nightlyscience OSS.20140923 SweetSpot SWEETSPOT  2014-09-23T21:51:33.763077 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 
[heather@ippc18 ~]$ difftool -dbname gpc1 -definewarpwarp -exp_id 798992  -template_exp_id 799009 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20140923 -set_reduction SWEETSPOT -simple -rerun -pretend
1041931 RINGS.V3 798992 OSS.20140923 1041973
[heather@ippc18 ~]$ difftool -dbname gpc1 -definewarpwarp -exp_id 798992 -template_exp_id 799009 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20140923 -set_reduction SWEETSPOT -simple -rerun 
597965 new neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 OSS.nightlyscience OSS.20140923 SweetSpot SWEETSPOT  2014-09-23T21:51:59.973567 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 

Wednesday : 2014.09.24

• HAF registration got stuck at like 2am - I kicked it. It's 7:20am, and we still have 60 images to download. Why is it so sloooooooow?
• 21:00 MEH: fallout from switching machines up and stalling and rebooting during nightly downloads, some files want to be on ipp077... stopping everything to slowly let those get placed in peace to avoid tediously correcting..
  • when brought back up, these should have been put into repair
  • cleanup was also stopped, until get nightly data might as well make space on 067,068 so set to run
  • might as well do the regular restart of stdscience while waiting for nightly

Thursday : 2014.09.25

• 06:00 MEH: no network connection into UH manoa or maui... -- though some email going through? PS1 not open last night so no nightly data
• 08:30 MEH: network access restored.. -- Gavin noted to Sidik's question about network that UH ITS had planned intermittent network on schedule for 0530-0630, we probably should keep eye on ITS info page (though if down won't be able to see the page anyways..) -- ​http://www.hawaii.edu/its/status/#0 and ​http://www.hawaii.edu/its
• 09:40 MEH: regular restart of summitcopy and registration, also to flush the logs of all the neb issues and ipp077 down yesterday
• morning -- ipp072-ipp082 down for PDU work

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD