PS1 IPP Czar Logs for the week 2019.05.17 - 2019.05.23

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.17

• TdB: 11:00 A set of 620 warps were stuck on IPPmonitor and not processing. Sending them to cleanup and back did not help. Their corresponding chip exps have now been sent to cleanup and are being updated again. Hopefully this will dislodge them.

• TdB: 13:46 The night_report.pl script has been updated to properly parse exposures in the BRIGHT obs_mode, which was causing problems in the report from last night. Testing shows it works well now.

CZAR HANDOVER by TdB -> JFL

updates:

The update beast needs to be fed regularly. The current state of %LAP.PV3% updates still needed is:

  42    LAP.PV3.20140730.2014% ->left over exps from wait state cleaning
  23595  LAP.PV3.20140730.20150% 
  32335  LAP.PV3.20140730.bigmem.20150%

A more detailed breakdown can be obtained using:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,warpRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'

For queuing updates, I was using ym own query besides Mark's one, if there is a need to do selections on e.g. the chip state of the exp corresponding to the maximum warp id exp:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.maxstate,subb.data_group,subb.label,subb.maxchipstate FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.data_group LIKE "LAP.PV3.20140730.20150%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state as maxstate,warpRun.data_group,camRun.label,chipRun.state as maxchipstate FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id) WHERE subb.maxstate LIKE "cleaned" limit 1000;' | grep -v exp_id | awk '{print "chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id  "$3" ; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id  "$3" ;warptool -dbname gpc1  -updaterun -set_label update.LAP.PV3 -warp_id "$2"; warptool -dbname gpc1  -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh

One extra thing to think about: We are only updating warps with state=cleaned. Maybe there are warps with state=goto_cleaned which need doing? or state=update that somehow got stuck in a non-updating label?

wait state cleaning:

There are still plenty of wait state chip exps that need to be cleaned. We need to continue doing this until they are all done, in between other tasks. Get a sense of which data_groups need to be done, using:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait"  GROUP BY data_group) as s ORDER by waitct ASC;'

I have cleaned everything that will be used by the future updates, hopefully. But, you never know....

We can send a data_group to clean up, using:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "wait" and data_group like "LAP.PV3.20140730.20150%" and label not like "update.LAP.PV3";' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$1}' | tcsh 

And to check if the cleanup is working ok, we can query the current states (and numbers) for a data_group:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label,state from (SELECT count(*) as waitct,data_group,label,state FROM chipRun where data_group like "LAP.PV3.20140730.20150%" GROUP BY data_group,state) as s;' 

general cleaning:

Once in a while the chip pixels that have been processed need to eb cleaning. Typically when a good 10k are done. Use the following query:

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' | tcsh

PS2 data incoming:

PS2 should be up soon, and the x nodes are set up for their data processing. When they are in use, you might want to take a periodic look at the power draw in cabinet i.06. There was an email from Haydn about how to do this, but see me if you need more help.

General:

-The pstamp and cleanup pantasks need rebooting once their Njobs reach 100k.

-With all this cleaning going on, some machines might be put back to up once enough disk space is freed. Keep an eye on the red on IPPmonitor and see if you think enough space is freed.

-keep an eye on the night_report script when doing bright Euclid observing. See email from this morning. The script was reporting there were 0 diffs expected, which is clearly not right. update: the script has been adjusted, but keep an eye on it.

Saturday : 2019.05.18

getting warm at the ATRC

Temperatures rose pretty high over on Maui at the ATRC cluster (see this website for details: ​https://ps1wiki.ifa.hawaii.edu/cgi-bin/ps1opsDashboard.py)

I set the following machines to repair, just in case they need to be moved to down to reduce heat: ippb06,08,16-23

Setting these to repair, and it started moving into the evening, meant that the temperature dropped to safer levels

Sunday : 2019.05.19

Monday : 2019.05.20

Tuesday : 2019.05.21

Wednesday : 2019.05.22

Thursday : 2019.05.23