PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

• 7:24 am Rob noticed the following exposures had problems:

  Exposure Status Comment
  o7524g0105o FAIL (Diff stage) OSSR.R12N6.14.Q.r ps1_30_4430 visit 1
  o7524g0126o FAIL (Diff1 stage) OSSR.R12N6.14.Q.r ps1_30_4430 visit 2
  
  Exposure Status Comment
  o7524g0392o FAIL (Diff stage) OSSR.R19S3.14.Q.w ps1_30_3423 visit 3
  o7524g0393o FAIL (Diff stage) OSSR.R19S3.14.Q.w ps1_30_3436 visit 3
  o7524g0410o FAIL (Cam (Bad Quality) stage) OSSR.R19S3.14.Q.w ps1_30_3423 visit 4
  o7524g0411o FAIL (Diff stage) OSSR.R19S3.14.Q.w ps1_30_3436 visit 4
  
  Exposure Status Comment
  o7524g0225o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4187 visit 3
  o7524g0226o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4353 visit 3
  o7524g0227o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4376 visit 3
  o7524g0228o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4186 visit 3
  o7524g0229o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4352 visit 3
  o7524g0230o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4176 visit 3
  o7524g0231o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4329 visit 3
  o7524g0232o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4350 visit 3
  o7524g0233o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4351 visit 3
  o7524g0234o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4185 visit 3
  o7524g0235o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4327 visit 3
  o7524g0236o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4174 visit 3
  o7524g0237o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4326 visit 3
  o7524g0238o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4165 visit 3
  o7524g0239o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4307 visit 3
  o7524g0240o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4328 visit 3
  o7524g0241o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4175 visit 3
  o7524g0242o FAIL (Diff stage) OSSR.R15N4.14.Q.r ps1_30_4187 visit 4
  

• 9:00am Serge says "I took the chainsaw and care of them (marked one warp and two diffs as invalid)."
• 10:40am HAF queued the following diff o7524g0393o-o7524g0411o for serge, also investigated/checked and set quality of 42 for a few diffs:

  difftool -dbname gpc1 -definewarpwarp -exp_id 1093036 -template_exp_id 1093055 -backwards 
  -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2016/05/16 -set_dist_group SweetSpot 
  -set_label OSS.nightlyscience -set_data_group OSS.20160516 -set_reduction SWEETSPOT -simple -rerun
  
  the following diff_ids were set to quality of 42
  
  1394181
  1394189
  1394230
  1394606
  

• 13:20 MEH: updating labels in ippMonitor/czartool
• 19:47 MEH: summitcopy having regular faults, seems to be related to ipp101 -- looks like it was put up from repair today -- looks like was also happening to afternoon darks and should have been dealt with then -- reminder, the darks in the afternoon provide a handy test for any changes to data system, only if checked... -- putting ipp101 back into repair to see if can clear...

  start copying file into nebulous  Mon May 16 17:21:49 HST 2016
  *** stderr ***
  Use of uninitialized value in string eq at /usr/lib64/perl5/5.8.8/File/Copy.pm line 76.
  Use of uninitialized value in stat at /usr/lib64/perl5/5.8.8/File/Copy.pm line 87.
  Use of uninitialized value in pattern match (m//) at /usr/lib64/perl5/5.8.8/File/Copy.pm line 131.
  Use of uninitialized value in concatenation (.) or string at /usr/lib64/perl5/5.8.8/File/Copy.pm line 133.
  2016/05/16 17:22:50 | ipp094 | FATAL | Nebulous::Client::replicate - can not copy instance file:///data/ipp101.0/nebulous/db/6f/8713342108.gpc1:20160517:o7525g0051d:o7525g0051d.ota22.fits
  can not copy instance file:///data/ipp101.0/nebulous/db/6f/8713342108.gpc1:20160517:o7525g0051d:o7525g0051d.ota22.fits at /usr/lib64/perl5/vendor_perl/5.8.8/Log/Log4perl/Logger.pm line 896
  Unable to perform dsget: 255 at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/summit_copy.pl line 246.
  
  failure for: summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o7525g0051d/o7525g0051d22.fits --filename neb://ipp060.0/gpc1/20160517/o7525g0051d/o7525g0051d.ota22.fits --summit_id 1088982 --exp_name o7525g0051d --inst gpc1 --telescope ps1 --class chip --class_id ota22 --bytes 49432320 --md5 80e639f1d9c99325dd84fbaeb8b6b631 --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous
  job exit status: 255
  job host: ipp094
  job dtime: 110.888797
  job exit date: Mon May 16 17:22:51 2016
  
  .....
  
  start copying file into nebulous  Mon May 16 19:47:55 HST 2016
  *** stderr ***
  Use of uninitialized value in string eq at /usr/lib64/perl5/5.8.8/File/Copy.pm line 76.
  Use of uninitialized value in stat at /usr/lib64/perl5/5.8.8/File/Copy.pm line 87.
  Use of uninitialized value in pattern match (m//) at /usr/lib64/perl5/5.8.8/File/Copy.pm line 131.
  Use of uninitialized value in concatenation (.) or string at /usr/lib64/perl5/5.8.8/File/Copy.pm line 133.
  2016/05/16 19:48:55 | ipp054 | FATAL | Nebulous::Client::replicate - can not copy instance file:///data/ipp101.0/nebulous/0a/f8/8713344299.gpc1:20160517:o7525g0062o:o7525g0062o.ota14.fits
  can not copy instance file:///data/ipp101.0/nebulous/0a/f8/8713344299.gpc1:20160517:o7525g0062o:o7525g0062o.ota14.fits at /usr/lib64/perl5/vendor_perl/5.8.8/Log/Log4perl/Logger.pm line 896
  Unable to perform dsget: 255 at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/summit_copy.pl line 246.
  
  failure for: summit_copy.pl --uri http://conductor.ifa.hawaii.edu/ds/gpc1/o7525g0062o/o7525g0062o14.fits --filename neb://ipp054.0/gpc1/20160517/o7525g0062o/o7525g0062o.ota14.fits --summit_id 1088993 --exp_name o7525g0062o --inst gpc1 --telescope ps1 --class chip --class_id ota14 --bytes 49432320 --md5 a41b6db19f64e1980f199d17e180a754 --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous
  job exit status: 255
  job host: ipp054
  job dtime: 110.480272
  job exit date: Mon May 16 19:48:56 2016
  

• 19:53 MEH: nightly processing needs its regular restart since not done over weekend??

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD