PS1 IPP Czar Logs for the week 2011.07.04 - 2011.07.10

(Up to PS1 IPP Czar Logs)

Monday : 2011.07.04

• 07:50 : Roy : Restarted summitcopy, which appeared to crash at 2am-ish
• 15:54 : Bill checked in to see whether the STS.2009.b data is finally done. Turned out that there were a couple of corrupt files that needed repair. Noticed that distribution pcontrrol was 1.3GB in size and not spawning the pending jobs very fast. Restarted it.

Tuesday : 2011-07-05

Serge is czar

• 10:41 Serge: stopping all servers
• 10:50 Serge: all servers stopped
• 12:00 Serge: processing resumed
• 13:15 Serge: chip.off
• 13:34 CZW: Serge noticed that diffs were not being queued. There was a registration fault this morning, and I suspect that fooled nightly_science into setting the nsDiffState to FINISHED_DIFFING. I launched most of the missing diffs with nightly_science.pl --date 2011-07-05 --verbose --queue_diffs. It looks like some diff inputs aren't ready, so this command will likely need to be run once more (alternatively, a restart of stdscience will reset the state).
• 15:03 Serge: from the file log file: /data/ipp041.0/nebulous/be/ef/1063727843.gpc1:IPP-MOPS-TEST:IPP-MOPS-TEST.231034.log, the task has been restarted 4464 times and finished with the roor:

  input file does not exist: neb://ipp020.0/gpc1/MD08.nightlyscience/2011/06/30/MD08.V2/skycell.094/SR_MD08.V2.skycell.094.WS.dif.141868.cmf 
  at /home/panstarrs/ipp/psconfig//ipp-20110622.lin64/bin/publish_file.pl line 374
  	main::my_die('input file does not exist: neb://ipp020.0/gpc1/MD08.nightlysc...', 231034, 2) 
          called at /home/panstarrs/ipp/psconfig//ipp-20110622.lin64/bin/publish_file.pl line 190
  

I tried:

difftool -revertdiffskyfile -dbname gpc1 -skycell_id skycell.094 -diff_id 141868

which had no effect.

• 15:20: Serge: warp.off
• 16:20: Serge: chip.on/warp.on. 5 exposures have been processed in the last 20 minutes
• 16:35: Serge: chip.off. Once again it can be observed that, when stare hosts are loaded, performance drops down.
• 17:25: Serge: warp.off
• 19:25: Serge: chip.on/warp.on

Wednesday : 2011-07-06

• heather restarted summitcopy - there was a chip that wasn't downloading (?). when I restarted it it looked like it downloaded (checkexp says so, mysql says no)
• registration is 'not happy' heather is investigating this still
• burntool is burntooling, no faults since heather started investigating (?)
• checkexp now reports things are going through registration.
• 11:10 Serge: MD data were not published to IPP-MOPS-TEST. I suspect it's because of the multiple declarations (e.g. survey.add.publish MD01 MD01.nightlyscience 5 NULL/survey.add.publish MD01 MD01.nightlyscience 1 NULL) in ~ipp/stdscience/input. I deleted the surveys (e.g. survey.del.publish MD01) and added them back with different "labels" (e.g. survey.add.publish MD01 MD01.nightlyscience 5 NULL/survey.add.publish MD01.ds MD01.nightlyscience 1 NULL). I modified ~ipp/stdscience/input accordingly.

• 11:22 CZW: registration stuck on exposure o5748g0233o: trunk/tool/regpeek.pl identified this as being a bt_state = -1 data_state = check_burntool, suggesting burntool failed to run correctly. Manually fixed with: ipp_apply_burntool_single.pl --exp_id 358617 --class_id XY52 --this_uri neb://ipp033.0/gpc1/20110706/o5748g0233o/o5748g0233o.ota52.fits --previous_uri neb://ipp033.0/gpc1/20110706/o5748g0232o/o5748g0232o.ota52.fits --verbose --dbname gpc1
• 12:40 heather: registration was stuck again, but this is a pantasks type of fault. it said on controller status:

  0    ipp015    RESP   4544.07 ipp_apply_burntool_single.pl --exp_id 358627 --class_id XY52 --this_uri neb://ipp033.0/gpc1/20110706/o5748g0246o/o5748g0246o.ota52.fits --continue 10 --previous_uri neb://ipp033.0/gpc1/20110706/o5748g0245o/o5748g0245o.ota52.fits --dbname gpc1 --verbose 0.0.8.4745 
  

  but when I checked on ipp015, there was no evidence of burntool. I ran this command by hand, and it worked (I don't know how/where/why it got lost in pantasks). Registration is now continuing.
• 12:41 Serge: added label for SAS data to stdscience/publishing
• 14:39 heather restarted stdsci at the request of chris (this should queue yesterday's stacks)
• 14:49 Serge (in stdscience):

  survey.add.publish SAS.footprint.123 SAS.footprint.123 5 SAS2_for_IPP-MOPS-TEST
  survey.add.publish SAS2.12 SAS2.12 5 SAS2_for_IPP-MOPS-TEST
  

• 16:00 heather deleted all the non-nightlyscience labels out of stdsci (LAP.ThreePi.20110621, LAP.ThreePi.test, SAS2.12, SAS.footprint.123)

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD