PS1 IPP Czar Logs for the week 2019.MM.DD - 2019.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2019.04.08

• TdB: The remaining update.LAP.PV3.broke label exposures have been dealt with. Some were simply reverted and some others needed to be sent to clean and then sent to update. The label should now be clear.

• TdB: there were some more requests from Rob for CTE removal:

  chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8577g0477o
  chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8577g0459o
  chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8577g0441o
  chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8577g0423o
  

• TdB: ipp102 need some small fix from haydn, and he asked for it to be put to down. I put it to neb-host down, but it turns out Mark was also running jobs on that machine on a different pantasks. SO, he turned the machine off across all pantasks. Now that it is finished, turn it back on using:

  ./check_system.sh hoston ipp102
  

  JRF: Haydn fixes complete -> moved ipp102 to repair

• TdB: There are some burntool error on ippmonitor:

  ipp_apply_burntool_fix.pl --exp_name o6457g0126o --class_id XY55 --verbose --dbname gpc1
  ipp_apply_burntool_fix.pl --exp_name o5306g0483o --class_id XY66 --verbose --dbname gpc1
  ipp_apply_burntool_fix.pl --exp_name o5306g0483o --class_id XY67 --verbose --dbname gpc1
  ipp_apply_burntool_fix.pl --exp_name o5306g0469o --class_id XY66 --verbose --dbname gpc1
  ipp_apply_burntool_fix.pl --exp_name o5306g0469o --class_id XY67 --verbose --dbname gpc1
  ipp_apply_burntool_fix.pl --exp_name o5327g0152o --class_id XY66 --verbose --dbname gpc1
  ipp_apply_burntool_fix.pl --exp_name o5327g0152o --class_id XY67 --verbose --dbname gpc1
  

  and revert files:

  chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1114864
  chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1115289
  chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1115326 
  chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1115409
  

• MEH: franken-chunk at end of night (XSS.nightlyscience) -- comment out OSS.WS.nightlyscience, Bright%.WS.nightlyscience from survey distribution and restart ~ippitc/stdscience pantasks

Tuesday : 2019.MM.DD

Diffs for breakfast guvner by JRF

Nightly processing went smoothly... for the most part

ippdb10 went down around 1am, no real explanation in the logs. Haydn restarted it, and noticed that there was a heat warning light on, seems fine now...

Maintenance by JRF

The rack containing ipp131-133 is offline, and I said it could remain so since the machines do not have any significant drives for data storage (the same case is true for ipp140-142).

Haydn continued with the replacement of ipp081. With Gavin's help they got it working again (the reason for the maintenance was a faulty SSD).

ippc64.1, which contains the homedirs and is where the pantasks are run from, was having some issues. It looks like the issues resolved themselves eventually, but Haydn would like to replace. We have given him the go-ahead.

Broken LAP.PV3 jobs by JRF

There are a bunch of chips stuck in the 'full' state. They mention that a log file is missing: e.g. log file (LOG.IMFILE, 'neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/10/05/o5763g0099o.365804/o5763g0099o.365804.ch.1121712') not found

Send them to clean:

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXX

Once they have finished cleaning, set the imfile to update:

chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXXXX

All good!

update.LAP.PV3 chips stuck in update by JRF

A number of jobs were failing saying they couldn't get specific files, I check on a bunch and the files are there and they seem normal (at least non-zero size). Just reverted them and they seemed to sovle themselves.

It looks like the CPU wait times are quite high across the computer nodes. Some of the b-nodes appear to be getting hammered too.

Wednesday : 2019.MM.DD

Root partition is critical ippc71 by JRF

Check the /tmp/nebulous_server.log size The root partition is getting pretty full on ippc71

71 - 71GB
72 - 67GB
73 - 63GB
74 - 70GB
75 - 60GB

It's also getting pretty large of the other apache servers too.

Follow the procedure from Wednesday : 2019.02.06 , see: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20190204 I first shutdown all the pantasks, then on each machine, stopped apaches, removed the log, replaced it with a fresh empty log with the correct permissions and started apache again.

JRF: ipp112.1 is getting pretty full on data so I've moved it into repair

No Mask for MOPS by JRF

The usual:

chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name XXXXXXXX

Thursday : 2019.MM.DD

JRF: put ipp102 back up after a disk change on Monday, as it seems to be behaving itself now.

• 16:00 EAM : I am running 4 exposures under user ippps2 as a reference set for tests of the candidate ippps2 tag.

Friday : 2019.MM.DD

Saturday : 2019.MM.DD

Sunday : 2019.MM.DD

• 10:15 EAM : ipp091 was not exporting to the new compute and storage nodes. /etc/exportfs was out of date. I modified to match the other systems and exported all. (exportfs -a)