| Version 5 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.05.03 - 2019.05.09
(Up to PS1 IPP Czar Logs)
Friday : 2019.05.03
pstamp requests by JRF
Received some error emails about pstamp requests. checking out the pstamp server we can see that over 800,000 jobs have been submitted by QUB.
The errors come from a script run by the crontab on ippc26 (this is mentioned in the emails). The script checks if pstamp has over 100,000 jobs done as this is were pantasks becomes flaky.
Checking out the status of the pstamp pantasks keep an eye on:
request.finish.run
as we DO NOT want to restart if this is running as it will cause funkiness.
So let's restart pstamp, doing the usual:
pantasks> stop pantasks> shutdown now --check that the jobs have finished first ~ippitc/pstamp> pantasks_server & pantasks_client pantasks> server input input pantasks> setup pantasks> run
I have added ippc76 to the hosts_apache group in the ~ippitc/ippconfig/pantasks_hosts.input file. This is so that when the nodes are allocated that ippc76 will be turned off when the apache group is removed.
Since over 800,000 requests were made the pstamp pantasks will need restarting about every hour or so.
- JRF: ippc72 has died, sent an email to Haydn, left the PDU off. Also commented it out of the
~ippitc/ippconfig/nebservers.cshfile.
CZAR HANDOVER by JRF
- ipp139 has started having issues so there will likely be some liaising with hardware people to fix that.
- ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
- 'wait' state: the past week has seen some chip states set to 'wait' which is not a valid state. They can appear in IPP processing at times, and the quickest way to fix them is: change label + imfile update:
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id XXXXXX chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id XXXXXX
the update second part is needed as the change from wait to update isn't usually recognized. - More updates will need to be queued, details of how to do this are given in the Thursday 2019-05-02 czarlog
- Guide for desperate diffs is in the czarlog for last week. HOWEVER, TdB likely has a better way of doing it by now
- QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
- CCL: need to modify /home/panstarrs/ipp/local/bin/apachedisk_chk.sh as ipp on ippc19 to solve the crontab error message, i.e., remove 72 & add 76 for host list.
{{{ssh: connect to host ippc72 port 22: No route to host /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 20: [: : integer expression expected /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 30: [: : integer expression expected }}}
Saturday : 2019.05.04
Sunday : 2019.05.05
Monday : 2019.05.06
