| Version 3 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.03.04 - 2019.03.10
(Up to PS1 IPP Czar Logs)
Monday : 2019.03.04
JRF (morning): ippb21 was unreachable this morning. This is after one of the SSDs had been replaced by Haydn. Rebooted it twice, seems to be behaving after the second reboot.
JRF (morning): ippb22 has also started reported similar problems. So we, JRF & EAM, have asked Haydn to have a look or think about what else might be a problem on those machines. (update) - Both b21 and b22 have now have disks replaced and been rebooted. They are behaving well at the moment so they have been set to repair in nebulous so that they get used.
Czar training:
Today we are changing the configuration of PS1 stdscience so that the warp stage will be targeting specific machine and
The file that contains the config for which machines are targeted for which stage is:
ipphosts.mhpcc.config
Located in the src directory directory:
~ippitc/src/ipp-20170121/ippTasks
Best practice is to create a backup of the current version, and another to modify and eventually replace the current one with.
At the moment ipp123-126 should now be targeted for any warp processing. Warps use the skycells, so this is the meta data we should change. We are mapping these skycells to these 16 specified machines. The remaining skycells will repeated with pattern. The new entry:
ipphosts METADATA camera STR skycell count S32 16 sky00 STR ipp123.0 sky01 STR ipp124.0 sky02 STR ipp125.0 sky03 STR ipp126.0 sky04 STR ipp123.1 sky05 STR ipp124.1 sky06 STR ipp125.1 sky07 STR ipp126.1 sky08 STR ipp123.0 sky09 STR ipp124.0 sky10 STR ipp125.0 sky11 STR ipp126.0 sky12 STR ipp123.1 sky13 STR ipp124.1 sky14 STR ipp125.1 sky15 STR ipp126.1 END
Note that there is a similar setup for the current mops tasks:
~ippmops/psconfig/ipp-20170121.lin64/share/pantasks/modules
Now that we have changed the source we would normally have to rebuild...however this is dangerous as it will incorporate any changes that have been checked in (the in production could should not really be modified, or at least this config should be separate...but these are separate issues). We can skip all that by just putting it into the production code:
~ippitc/psconfig/ipp-20170121.lin64/share/pantasks/modules
To incorporate the changes in stdscience pantasks we need to restart it.
Manually restarting a pantasks with a new label added to config (and also how not to):
On purpose, we have gone to the machine where ps1 pantasks stdscience is running and started the server (to know where it is running check the ippitc/stdscience/ptolemy.rc):
start_server.sh stdscience
This means there are two pantasks servers running now, which is not a great state to be in. You can confirm by checking the process list
ps -ef | pantasks_server
If there are two entries listed -> you have two servers running.
The issue with this is that when running pantasks_client you might end up connecting to either one of the servers, and if you are in the more recently created one then 'status' might not work and you can't see jobs that could be running on the other one. The best way to get out of this situation is to shut them both down:
$> pantasks_client pantasks: stop pantasks: shutdown
Between the stop and shutdown check that the jobs have finished, if any, by querying the 'status' or 'controller status'
For the second part of this task we want a label to be added automatically each time pantasks stdscience starts up. To do this add the following line to the 'input' file in the stdscience directory:
add.label update.OSS
To restart the pantasks_server, do what we did before:
start_server.sh stdscience
Then to load the config and start it running, do the following in pantasks:
pantasks: server input input pantasks: setup pantasks: run
The first loads the modified config, the second activates all the tasks with the loaded config, then run actually starts the server running.
This new label currently has the highest priority, as it defaults to highest if there is no entry... this is not what we want. To add the label to the list of labels, and set the priority, use the label tool! To check if it is present first:
labeltool -dbname gpc1 -listlabel -simple | grep update
Add label and set priority:
labeltool -dbname gpc1 -definelabel -set_label update.OSS -set_priority 300
In this case we want it to be a lower priority than MOPS.2 which was 310, so we chose 300.
Give it a few minutes and ippmonitor should reflect these changes by showing it in the correct place.
Other Things:
Some weird spikes in loads going off at 16:30...not sure why.
