| Version 8 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.03.04 - 2019.03.10
(Up to PS1 IPP Czar Logs)
Monday : 2019.03.04
JRF (morning): ippb21 was unreachable this morning. This is after one of the SSDs had been replaced by Haydn. Rebooted it twice, seems to be behaving after the second reboot.
JRF (morning): ippb22 has also started reported similar problems. So we, JRF & EAM, have asked Haydn to have a look or think about what else might be a problem on those machines. (update) - Both b21 and b22 have now have disks replaced and been rebooted. They are behaving well at the moment so they have been set to repair in nebulous so that they get used.
Czar training:
Today we are changing the configuration of PS1 stdscience so that the warp stage will be targeting specific machine and
The file that contains the config for which machines are targeted for which stage is:
ipphosts.mhpcc.config
Located in the src directory directory:
~ippitc/src/ipp-20170121/ippTasks
Best practice is to create a backup of the current version, and another to modify and eventually replace the current one with.
At the moment ipp123-126 should now be targeted for any warp processing. Warps use the skycells, so this is the meta data we should change. We are mapping these skycells to these 16 specified machines. The remaining skycells will repeated with pattern. The new entry:
ipphosts METADATA camera STR skycell count S32 16 sky00 STR ipp123.0 sky01 STR ipp124.0 sky02 STR ipp125.0 sky03 STR ipp126.0 sky04 STR ipp123.1 sky05 STR ipp124.1 sky06 STR ipp125.1 sky07 STR ipp126.1 sky08 STR ipp123.0 sky09 STR ipp124.0 sky10 STR ipp125.0 sky11 STR ipp126.0 sky12 STR ipp123.1 sky13 STR ipp124.1 sky14 STR ipp125.1 sky15 STR ipp126.1 END
Note that there is a similar setup for the current mops tasks:
~ippmops/psconfig/ipp-20170121.lin64/share/pantasks/modules
Now that we have changed the source we would normally have to rebuild...however this is dangerous as it will incorporate any changes that have been checked in (the in production could should not really be modified, or at least this config should be separate...but these are separate issues). We can skip all that by just putting it into the production code:
~ippitc/psconfig/ipp-20170121.lin64/share/pantasks/modules
To incorporate the changes in stdscience pantasks we need to restart it.
Manually restarting a pantasks with a new label added to config (and also how not to):
On purpose, we have gone to the machine where ps1 pantasks stdscience is running and started the server (to know where it is running check the ippitc/stdscience/ptolemy.rc):
start_server.sh stdscience
This means there are two pantasks servers running now, which is not a great state to be in. You can confirm by checking the process list
ps -ef | pantasks_server
If there are two entries listed -> you have two servers running.
The issue with this is that when running pantasks_client you might end up connecting to either one of the servers, and if you are in the more recently created one then 'status' might not work and you can't see jobs that could be running on the other one. The best way to get out of this situation is to shut them both down:
$> pantasks_client pantasks: stop pantasks: shutdown
Between the stop and shutdown check that the jobs have finished, if any, by querying the 'status' or 'controller status'
For the second part of this task we want a label to be added automatically each time pantasks stdscience starts up. To do this add the following line to the 'input' file in the stdscience directory:
add.label update.OSS
To restart the pantasks_server, do what we did before:
start_server.sh stdscience
Then to load the config and start it running, do the following in pantasks:
pantasks: server input input pantasks: setup pantasks: run
The first loads the modified config, the second activates all the tasks with the loaded config, then run actually starts the server running.
This new label currently has the highest priority, as it defaults to highest if there is no entry... this is not what we want. To add the label to the list of labels, and set the priority, use the label tool! To check if it is present first:
labeltool -dbname gpc1 -listlabel -simple | grep update
Add label and set priority:
labeltool -dbname gpc1 -definelabel -set_label update.OSS -set_priority 300
In this case we want it to be a lower priority than MOPS.2 which was 310, so we chose 300.
Give it a few minutes and ippmonitor should reflect these changes by showing it in the correct place.
Other Things:
Some weird spikes in loads going off at 16:30...not sure why.
Tuesday : 2019.03.05
JRF (Morning): It looks like processing has been slowing down over the evening (drops from around 200 to 150 items per hour).
Can turn off/on separate tasks in pantasks e.g.
chip.revert.off
will turn off the chip.revert task, this is indicated by a minus sign next to the task:
-+ chip.revert
Simply do chip.revert.on to turn it back on. To turn off a whole bunch of tasks at once you can do chip.off, to turn off all chip tasks. Can be useful if you want it to focus on getting some warps through. You can also shutdown pantasks without being on the machine running it (I didn't know this).
ippb22 still having hardware issues. Taken down on nebulous, Haydn will investigate what to do with it.
Looking to see if anything is misbehaving in ganglia, ippdb09 has pretty high load most of the time: Having a look at the processlist in the nebulous database gives the following process that has been running for a long time:
| 26081089 | nebulous | ippc73.ifa.hawaii.edu:47235 | nebulous | Query | 52602 | Sending data | SELECT
storage_object.so_id,
uri,
mountedvol.available,
vol_id,
cab_id
FROM storage_object
JOIN instance
USING (so_id)
JOIN mountedvol
USING(vol_id)
JOIN volume
USING(vol_id)
WHERE ext_id = 'gpc1/OSS.nt/2014/10/28/o6958g0546o.810979/o6958g0546o.810979.ch.1190018.XY50.ptn'
AND mountedvol.available = '1'
Killed the process:
mysql> kill 26081089;
JRF (afternoon ft czarlord Mark):
doing the updates of lots of 2014 data we finally got some faults reported on ippmonitor. Following the link, and checking the updates most issues appear to be related to trying to fetch files from a busy node e.g. ippb04. To fix this you can turn chip.revert off and on again or run:
chiptool -revertprocessedimfile -fault 2 -label update.OSS -dbname gpc1
This will set the chip to revert and try again.
Consolerc and SVN commiting:
ipp129 seems to be working so it should be set to up in nebulous (done). However, if the machine is to become unresponsive then we would need to be able to log into the console to cycle it. Therefore, an entry needs to be made for ipp129 in the dotConsolerc file:
ipp126:
console: itc-i12-con.ifa.hawaii.edu
ipp129:
console: itc-i12-con.ifa.hawaii.edu
ippc17:
console: itc-i10-con.ifa.hawaii.edu
The actual console entry came from the google sheets hardware document that says that ipp129 is in rack i13. "But John, you put the console as i12?!" - True, but that is because i12 actually manages i12 AND i13. There isn't necessarily a separate console for each rack. The i12 rack is fancy too, allows you to exit with 'Ctrl+z' then 'exit'
I changed this file on the ippitc account, and my svn checkout. A couple of useful svn commands:
To view the status i.e. the files that have changes in yuor local copy
svn st
To view the change log for a file:
svn log hardware/dotConsolerc
...which looks like:
------------------------------------------------------------------------ r40635 | fairlamb | 2019-03-05 16:17:19 -1000 (Tue, 05 Mar 2019) | 1 line adding ipp129 to consolerc ------------------------------------------------------------------------ r40455 | mhuber | 2018-06-25 08:48:50 -1000 (Mon, 25 Jun 2018) | 1 line update .Consolerc for added nodes
To see the info on a file, such as last changed date (and by who):
svn info hardware/dotConsolerc
...looks like this:
Path: hardware/dotConsolerc Name: dotConsolerc Working Copy Root Path: /home/john/ipp-clone/trunk URL: https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/trunk/hardware/dotConsolerc Relative URL: ^/trunk/hardware/dotConsolerc Repository Root: https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp Repository UUID: 60eb6cdc-a59c-4636-a4e0-dba66a9721fd Revision: 40635 Node Kind: file Schedule: normal Last Changed Author: fairlamb Last Changed Rev: 40635 Last Changed Date: 2019-03-05 16:17:19 -1000 (Tue, 05 Mar 2019) Text Last Updated: 2019-03-05 16:05:14 -1000 (Tue, 05 Mar 2019) Checksum: 06cf3f1a18c8f5b2af639cacdb0d55231218c0cf
To see the differences between your local file and the repository version:
svn diff https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/trunk/hardware/dotConsolerc hardware/dotConsolerc
Finally to commit a file doing the following:
svn commit hardware/dotConsolerc -m 'adding ipp129 to consolerc'
Other
ipp127, ipp128 -> Haydn says that these machines are fine, and we're just getting fluff emails.
- TdB: Similar to John, I also changed the Consolerc file, this time to add ipp128 to the list of serial consoles. Like 129, ipp128 is also located in rack i13 and controlled by i12. So, the change made was:
ipp126: console: itc-i12-con.ifa.hawaii.edu ipp128: console: itc-i12-con.ifa.hawaii.edu ipp129: console: itc-i12-con.ifa.hawaii.edu ippc17: console: itc-i10-con.ifa.hawaii.edu
I changed this file on the ippitc account, and my svn checkout, which was then committed on grens.
Wednesday : 2019.03.06
- TdB: After the repairs to ipp127-129 by Haydn, I set ipp128 back to neb-host up to try to get some files written there and test it's functionality.
neb-host --state up --host ipp128 --note 'TdB: repair->up: back to up after repairs by Haydn. Getting some new files written to it, to test its functionality'
