== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2019.03.11 ===

 * MEH: simple one-liner to remove the faulted chip/warps in update.OSS to update.OSS.broke until ipp067 back online -- remember to add the user/password
{{{
mysql -hscidbm -uXXXXX -pXXXXX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.OSS" and warpRun.data_group like "OSS%" and warpRun.state like "update" and  ( warpSkyfile.fault>0 or chipProcessedImfile.fault>0 ) group by warp_id limit 500; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_label update.OSS.broke -chip_id  "$1" ;warptool -dbname gpc1  -updaterun -set_label update.OSS.broke -warp_id "$2 }'
}}}  

 * TdB: Following upgrades to Ubuntu and a series of EDAC errors, machines ippb16-23 are now behaving well again. They have been tested rigorously by Gene using rsyncs, and now put back to neb-host up.

 * CCL: set ipp134 down -> repair for testing then -> up, there are ipp134_bck.0, set it up only ATRC is offline
 * CCL: set ipp135 down -> repair for testing (ipp135_bck.o, same as ipp134_bck.0)
 * CCL: restart stdscience at 14:00

 * TdB: The ipp067 related faults on IPP monitor are being cleared out on a regular basis, when the total number of faulted exposures becomes too high (>300). In that case, stop the stdscience pantask and run the batch SQL query/chiptool command Mark gives above. Then, set the pantask back to run.

 * MEH: adding other updates to run in ~ippmops/stdscience targeting ipp123-129 for warps -- stop pantasks if needed
=== Tuesday : 2019.03.12 ===
 * MEH: ipp130 looks okay, setting neb-host down->repair->up and will just target in ippmops:stdscience for now
  * if it works fine, it could be kept neb-host up when nightly processing starts since it has serial console access in i12 (needs to be added to Consolerc)

 * TdB: Fixed a bunch of red on IPPmonitor by changing the labels to update.OSS.broke. However, a few exposures failed with fault=2 and state=full, which was somewhat unusual. An example is exposure o6982g0381o, XY41 which went through a chip update. The bottom of the update log reads:
{{{
      Complete ppImage run: 20.814865 sec
      Memory leaks: 0
Number of leaks to display: 500
Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/ppStatsFromMetadata /data/ipp018.0/nebulous/ee/8d/8954414172.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.stats - CHIP_IMFILE]...
-bg 314.070892 -bg_mean_stdev 1.714188 -bg_stdev 19.923737 -dtime_detrend 9.678738 -dtime_photom 0.000003 -dtime_total 20.751261 -quality 0 
Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/e2/46/8954413865.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.ch.fits > /dev/null]...
Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/58/d7/8954414001.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.ch.mk.fits > /dev/null]...
Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/3e/19/8954414035.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.ch.wt.fits > /dev/null]...
Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/fc/05/8954414163.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.mdl.fits > /dev/null]...
Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/chiptool -tofullimfile -chip_id 1262404 -class_id XY41 -dbname gpc1]... 
}}}
  The files in question cannot be accessed since ipp018 is down on nebulous.

 * TdB: ipp136 and ipp137 put from down to up, to test for files going there. it is currently not targeted for specific products, but will be tomorrow if all works out. 
 * CCL: ipp135 and ipp137 put from repair to up, for warp product targetting.
 * CCL: modify the configuration file of PS1 stdscience for warp stage targeting on ipp134 and ipp135 machine by following steps.
{{{
the files are under these two folders
/data/ippc64.1/ippitc/src/ipp-20170121/ippTasks
/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/share/pantasks/modules

make a copy for backing up and a new file for adding targetting machines from "ipphosts.mhpcc.config"
cp ipphosts.mhpcc.config ipphosts.mhpcc.config.190312ori
cp ipphosts.mhpcc.config ipphosts.mhpcc.config.190312_target134_135

add ipp134 & ipp135 to the new file
ipphosts METADATA
  camera STR skycell
  count S32 48
  sky00 STR  ipp123.1
  sky01 STR  ipp124.1
  sky02 STR  ipp125.1
  sky03 STR  ipp126.1
  sky04 STR  ipp129.0
...
  sky30 STR  ipp125.0
  sky31 STR  ipp126.0
  sky32 STR  ipp134.0
  sky33 STR  ipp134.1
  sky34 STR  ipp134.0
  sky35 STR  ipp134.1
  sky36 STR  ipp134.0
  sky37 STR  ipp134.1
  sky38 STR  ipp134.0
  sky39 STR  ipp134.1
  sky40 STR  ipp135.0
  sky41 STR  ipp135.1
  sky42 STR  ipp135.0
  sky43 STR  ipp135.1
  sky44 STR  ipp135.0
  sky45 STR  ipp135.1
  sky46 STR  ipp135.0
  sky47 STR  ipp135.1
 END

overwrite "ipphosts.mhpcc.config" by the new configuration file in the two folders
cp ipphosts.mhpcc.config.190312_target134_135 ipphosts.mhpcc.config

and then wait for "stdscience" restarting automatically at 3:30pm
}}}

 * MEH: in preparation for nightly data soon, sending some older update chips to cleanup (chip+warp full state, not broken labels) to open more space on the primary datanodes ipp067-112

 * JRF: ippc138 and ipp139 set to up in nebulous, and added some skycell targeting for them in the mhpcc config files (modified in src and lin64 build).

=== Wednesday : 2019.03.13 ===
 * TdB: pantasks has been restarted occassionally as update jobs reach 100k, and errors related to ipp067 have been cleared out to the broke label, awaiting repairs
 * TdB: machines ipp136,137 appear to be taking data well, and have now been added to the targeting queue for warp products, similar to the other ipp13* machines. 
 * MEH: ippb21 behaving badly again, EDAC errors, setting neb-host down since it is harassing update processing with primary copies on it
=== Thursday : 2019.03.14 ===
 * TdB: ipp067 is put back to repair following repairs by Haydn. This will allow the errors under the current label to clear out, which will be good testing. Following that, we can put the machine back to up state, and reprocess the IDs gathered under broke label.

 * TdB: With ipp067 back up, I will start clearing out the errors. First off, the broken files under ps_ud_MOPS.broke will be set back to the regular ps_ud_MOPS label in batches and set to update using:
{{{
mysql -hscidbm -uXXXXX -pXXXXX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "ps_ud_MOPS.broke" and warpRun.state like "update" group by warp_id limit 200; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id  "$1" ;warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id "$2 }' | tcsh
}}}
=== Friday : 2019.03.15 ===

 * TdB: following EDAC error warning, ippb22 and ippb17 have been put back to down in nebulous, so they can be checked out and not interfere with normal IPP operations.
 * TdB: all exposures under the ps_ud_MOPS.broke label have been cleared. Almost all went through without issues, with a total of 6 exposures failing due to files being located on ippb22. These have been reprocessed from scratch to allow Rob to get his stamps in a timely manner. Exposures under the update.OSS.broke will now be reprocessed intermixed with the regular update processing. A summary of the process is given below:
   * the broke labels for ps_ud_MOPS have been re-evaulated according to the following steps:

   step 1) use an update run command to move label back from broke to regular:
{{{
mysql -hscidbm -uXXXXX -pXXXXX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) joi camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "ps_ud_MOPS.broke" and warpRun.state like "update" group by warp_id limit 200; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  "$1" ;warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id "$2 }' | tcsh
}}}
   step 2) if the processing fails, use the revertprocessedimfile command to revert the processing and try again

   step 3) if the processing still fails, do a full reprocessing of the image

   In this instance, faults cleared from IPP monitor. However, it is important to check if that actually means the images needed to make the pstamp are in the full state. To that end, query the list of exposures Rob provided, and see if their most recent chip_ids and warp_ids are in the full state:
{{{
mysql -hXXXXX -uXXXXX -pippuser gpc1 -B -e 'SELECT suba.exp_name,suba.exp_id,subb.max_warpid,subc.max_warpstate,subc.max_chipid,subc.max_chipstate FROM (SELECT exp_id,exp_name FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name IN ("o5093g0045o","o5093g0054o","o5577g0554o","o5590g0733o","o5590g0740o","o5593g0599o","o5593g0622o","o5645g0512o","o5645g0513o","o5645g0529o","o5649g0305o","o5649g0321o","o5649g0322o","o5649g0339o","o5649g0340o","o5649g0356o","o5649g0357o","o5665g0138o","o5665g0156o","o5727g0153o","o5727g0176o","o5727g0200o","o5727g0223o","o6148g0466o","o6148g0489o","o6169g0172o","o6316g0103o","o6316g0114o","o6827g0269o","o6827g0289o","o6827g0309o","o6827g0329o","o6940g0066o","o7444g0312o","o7444g0330o","o7444g0348o","o7444g0366o","o7480g0136o","o7480g0155o","o7480g0174o","o7480g0193o","o7499g0102o","o7499g0120o","o7499g0138o","o7499g0156o","o7506g0103o","o7506g0122o","o7506g0141o","o7506g0160o","o7510g0106o","o7510g0124o","o7510g0142o","o7510g0160o","o7526g0096o","o7526g0118o","o7526g0140o","o7526g0162o") GROUP BY exp_id) as suba INNER JOIN (SELECT MAX(warp_id) as max_warpid,chipRun.exp_id,warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" GROUP BY chipRun.exp_id) AS subb ON (subb.exp_id = suba.exp_id) INNER JOIN (SELECT warp_id,chipRun.exp_id,warpRun.state as max_warpstate,chipRun.state as max_chipstate,chipRun.chip_id as max_chipid FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE  rawExp.camera LIKE "%GPC1%") AS subc ON (subc.exp_id = subb.exp_id AND subc.warp_id = subb.max_warpid);'
}}}
   This revealed that while most chip_IDs had processed to full state (1 had not), all the warp_IDs were either in update or cleaned state. The appropriate warp_IDs were subsequently processed to full by setting the skyfile to update, ensuring full-state recently processed pixel data is available for pstamps.

=== Saturday : 2019.03.16 ===
 * TdB: Compute node ippc67 was reported as down by nagios during the night. I went into the serial console, and tried to power cycle it. After repeated attempts, the machine was still unresponsive and not showing a boot screen. Therefore, the machine was powered down on the serial console while repairs are performed. 
 
 To remove the machine from pantasks, I edited the file ~ippitc/ippconfig/pantasks_hosts.input, commenting out ippc67 and noting the date and reason. Next, to remove it from the currently running pantasks, I ran ./check_system.sh hostoff ippc67 as user ippitc to clear the machine from the current tasks. This needed to be done 5 times in total. To see how many times are needed, log in to a pantasks using pantasks_client and run a controller status. That way, you can see the number of instances of the machine running in that pantask. 

=== Sunday : 2019.03.17 ===
 * MEH: sending chips to cleanup that have updated warps in full state to free up space on lower datanodes (>10k) again