PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.04.25

• 23:50 MEH: another warp fault 4 that would have stalled until manually cleared -- cannot build growth curve (psf model is invalid everywhere)

  warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.0392.066 -warp_id 1721240 -fault 0
  

Tuesday : 2016.04.26

• 13:40 EAM : ipp041 fell over, rebooting now.
• 14:20 CZW : starting third batch of wave 3 rsync processes, concurrent with still running second batch. This set is ipp043,44,45,46,47. There are delays built into the jobs to try and stagger the impact.
• 20:15 EAM : stopping and restarting pantasks

Wednesday : YYYY.MM.DD

• 23:24 MEH: looks like registration crashed... restarting

  [2016-04-27 23:11:07] pantasks_server[6514]: segfault at 10f8ab8 ip 0000000000408a4e sp 00000000429b9f20 error 4 in pantasks_server[400000+16000]
  

Thursday : 2016.04.28

• 07:40 MEH: Serge/MOPS request for manual diffims since visit 4 cam quality fault, do v2-3

  difftool -dbname gpc1 -definewarpwarp -exp_id 1084724 -template_exp_id 1084735 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2016/04/28 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20160428.extra -set_reduction SWEETSPOT -simple -rerun 
  
  difftool -dbname gpc1 -definewarpwarp -exp_id 1084812 -template_exp_id 1084830 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2016/04/28 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20160428.extra -set_reduction SWEETSPOT -simple -rerun
  

• 08:20 MEH: the .multi diffims still being made, sending a large number to cleanup...
• 08:30 MEH: doing cleanup of ps_ud% to clear the cycling faulting chips and warps...

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD