PS1 IPP Czar Logs for the week 2019.02.18 - 2019.02.24

(Up to PS1 IPP Czar Logs)

Monday : 2019.02.18

• 11:30 EAM : stopped mysql on ipp121 & started rsync of mysql directory (/export/ipp121.0/mysql.nebulous.20180308 to /export/ipp121.1/mysql.nebulous.20190218/)
• TdB : There was 1 processing error on IPP monitor, which was due to missing detrend files. This was in turn due to both versions of the file being located on ippb06, which is without power at the ATRC. The exposure has been re-run using:

  pantasks_client
  add.label PSNSC.wfix.20180525
  exit
  
  chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o6980g0199o
  

  after running through, set the old runs to be cleaned to remove them from IPPmonitor

  chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXX
  warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
  

• TdB: ipp054-ipp066 should remain in repair until further notice.

Tuesday : 2019.02.19

• MEH: large warp update process continuing to run in ~ippmops/stdscience on the ippc31-c63, ipps00-14, ipp067-ipp123 nodes
  • ps_ud% chips and warps label changed to update.hold20190219 so not to be cleaned by accident

Wednesday : 2019.02.20

• TdB: John, Roger and myself went through the logfiles to figure out what went wrong yesterday with ipp113. We looked through the reports for ipp113 on Ganglia, noticing the spiking load after 18:00 until 20:00. We went into the serial console using:

  console --user xxxx -v ipp113
  

and check the rolling log information. We check the status (option 8) under power management (CTRL + P) and noticed it was back to normal now. Mark showed us how to powercycle the machine and check the boot messages in /var/log/messages (as sudo user)

When a machine has been power cycled, MYSQL is probably down. Check that by doing:

ps -ef | grep mysql

Check the config file under /etc/mysql/my.cnf for the location of the error log and see if MYSQL is what caused the issue. In this case, it did not.

To restart MYSQL do:

/etc/init.d/mysql start

After a reboot, the screen session also need to be checked and rebooted. As ippitc on ipp113 check them using:

screen -list

and restart screens using:

screen -S RoboCzar (or CzarPoll)

once in the screen session, launch the script using:

cd /data/ippc64.1/ippitc/src/ippMonitor/czartool
./roboczar.pl (or ./czarpoll.pl)

and get out using CTRL-a d

Finally, upon reboot, clear active processes from the pantasks by using:

./check_systems.sh stop

and set the pantasks back to run after jobs have cleared, using:

./check_systems.sh run

Remember, if there is any non-standard (i.e. not nightlyscience) processing happening, these are not affected by the check_systems script and need to be done by hand.

• TdB : There were a few processing errors on IPP monitor, due to missing files.These exposures have been re-run using the standard wfix set-up:

  pantasks_client
  add.label PSNSC.wfix.20180525
  exit
  
  chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o6980g0199o
  

Thursday : 2019.02.21

• TdB: ipp125 experienced stalling after high load and high wait yesterday night. messages in /var/log/messages showed some issues related to the swapper and the raid. Conferring with Haydn, the raid looks fine and a reboot is warranted.

To figure out which pantasks is running on a particular machine, run:

ps -ef | grep pclient

that will show which clients are running on that machine

We went into the pantasks_client in ippqub:stdscience_ws to stop all the current processing:

pantasks_client

stop

To check that all current processing was properly synced, we did:

sudo sync

This showed nothing, indicating everything is synced correctly. Following this, we logged in to the machine using the serial console and did a regular reboot using:

sudo reboot -f

boot messages indicate the Superblock last mount time was in the future and the clock was not set. Haydn confirmed the system clock does not automatically set on this motherboard/kernel combination, and set the clock manually using:

 ntpdate -u galileo

which fixes the clock issue. The machine has subsequently been set back to repair to monitor its activity and load levels, and finally put back to up.

• TdB: several machines need a kernel update, among which ippx052. We removed it for current processing by going into the ps2 stdscience pantasks_client at ~ippps2/stdscience and doing:

  controller status
  controller host off ippx052 (6 times)
  controller host delect ippx052 (6 times)
  

  After this, I let Gavin know the machine is ready for upgrade.

• 16:30 EAM : starting orphaned instance deletion for ipp005 (0.9M bad instances). should take a few hours.

• 20:03 TdB : There were several errors on IPP monitor. There were fault 4 chip stage errors for three exposures (o5364g0520o,o5366g0482o,o5366g0498o), which appear to be caused by missing .psf files on nebulous. I went to the directory in question and see that the .psf for most chips are there, but the offending chips have a file called .psf.gone instead. I tried first a setimfiletoupdate, but that did not seem to help the matter. Therefore, I did a full update using wfix (these are ecliptic.rp files) which went all the way through without fail. There were also several skycell failures, giving errors like:

  neb entry neb://ipp081.0/gpc1/LAP.PV3.20140730/2014/12/15/o5430g0390o.211135/o5430g0390o.211135.wrp.1273665.skycell.1685.070.skycell not found, not created 
   at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warp_skycell.pl line 210 
  Running [/home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/dvoImageExtract -D CATDIR /local/ipp/gpc1/tess/RINGS.V3 skycell.1685.070 -o /data/ipp081.0/nebulous/d4/80/8238504074.gpc1:LAP.PV3.20140730:2014:12:15:o5430g0390o.211135:o5430g0390o.211135.wrp.1273665.skycell.1685.070.skycell]...
  Couldn't find input file: neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/15/o5430g0390o.211135/o5430g0390o.211135.ch.1327597.ch.XY05.ch.fits at /home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warp_skycell.pl line 563. 
  Running [/home/panstarrs/ipp/psconfig/ipp-20141024.lin64/bin/warptool -updateskyfile -warp_id 1273665 -skycell_id skycell.1685.070 -fault 2 -dbname gpc1]... 
  08/07/8238511863.gpc1:LAP.PV3.20140730:2014:12:15:o5430g0390o.211135:o5430g0390o.211135.wrp.1273665.skycell.1685.070.log.update.bz2 lines 946-968/968 (END) 
  

  The filenames in question are quite odd, and the origin is unclear. The chip-stage files have been re-run, clearing their errors.

Friday : 2019.02.22

13:37 CCL: check and replicate one of the detrend file.

neb-stat neb:///gpc1/flatcorr.20081024/GPC1.FLAT.48/GPC1.FLAT.48.XY20.co.fits

===this is the output===
object id:             7422216
key:                   gpc1/flatcorr.20081024/GPC1.FLAT.48/GPC1.FLAT.48.XY20.co.fits
read lock:             0
write lock:
epoch:                 2008-10-25 10:21:27
mtime:                 0000-00-00 00:00:00
available instances:   2
requested instances:   2
total instances:       2
instance location:
    file:///data/ipp119.0/nebulous/a6/f9/10833539.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits
    file:///data/ipp119.1/nebulous/a6/f9/697091390.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits

CCL: replicate 4 copies for this file in order to have a total of 6 available instances

neb-replicate gpc1/flatcorr.20081024/GPC1.FLAT.48/GPC1.FLAT.48.XY20.co.fits --copies 4

neb-stat neb:///gpc1/flatcorr.20081024/GPC1.FLAT.48/GPC1.FLAT.48.XY20.co.fits
object id:             7422216
key:                   gpc1/flatcorr.20081024/GPC1.FLAT.48/GPC1.FLAT.48.XY20.co.fits
read lock:             0
write lock:
epoch:                 2008-10-25 10:21:27
mtime:                 0000-00-00 00:00:00
available instances:   6
requested instances:   2
total instances:       6
instance location:
    file:///data/ipp119.0/nebulous/a6/f9/10833539.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits
    file:///data/ipp119.1/nebulous/a6/f9/697091390.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits
    file:///data/ippb07.0/nebulous/a6/f9/11103528639.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits
    file:///data/ippb10.1/nebulous/a6/f9/11103528859.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits
    file:///data/ippb15.0/nebulous/a6/f9/11103528997.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits
    file:///data/ippb10.0/nebulous/a6/f9/11103529137.gpc1:flatcorr.20081024:GPC1.FLAT.48:GPC1.FLAT.48.XY20.co.fits

=== it seems that neb-replicate is smart enough to distribute new files on different b-nodes. ===

Saturday : 2019.02.23

• 07:58 TdB: There were 2 exposures stuck on IPP monitor, quoting two types of errors, one of which was nebulous instance of a file not available and the other one:

  DBI connect('host=nebulous.ipp.ifa.hawaii.edu;database=nebulous','ipp',...) failed: Can't connect to MySQL server on 'nebulous.ipp.ifa.hawaii.edu' (111) at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/whichnode line 77
  Cannot connect to database.
  Unable to perform whichnode: 255 at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 830
          main::my_die('Unable to perform whichnode: 255', 1197365, 1896246, 'XY51', 3) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 717
  

Setting the imfile to update fixed both errors, although the message of can't connect to MySQL server is worrying. I went to nebulous.ipp.ifa.hawaii.edu and the MySQL server there is up and running.

Sunday : 2019.02.24

• MEH: sending some chip updates to cleanup, will watch load on ippdb09