| Version 23 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.04.22 - 2019.04.28
Czar of the week: TdB + EAM (czar-lord), previously CCL
(Up to PS1 IPP Czar Logs)
Monday : 2019.04.22
- CCL: these are full stage on chip, but still show up on red flags
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182888 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182889 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182890 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182891 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182892 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182893 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182894 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182895 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1182896 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182888 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182889 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182890 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182891 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182892 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182893 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182894 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182895 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1182896 --> can't be updated due to ippb nodes are shutdown.
- CCL: most of them have burntool files on ippb node without copies on ITC nodes.
SELECT rawExp.exp_name,chipProcessedImfile.class_id,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipRun.data_group,chipRun.dist_group,chipProcessedImfile.fault FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipProcessedImfile.fault != 0 AND chipRun.label like 'update.LAP.PV3' group by chip_id; +-------------+----------+--------+---------+--------+----------------+-------------------------------+-------------+-------+ | exp_name | class_id | exp_id | chip_id | state | label | data_group | dist_group | fault | +-------------+----------+--------+---------+--------+----------------+-------------------------------+-------------+-------+ | o5464g0207o | XY75 | 230361 | 1179692 | update | update.LAP.PV3 | LAP.PV3.20140730.20141024 | LAP.ThreePi | 2 | | o5444g0007o | XY76 | 219509 | 1179773 | update | update.LAP.PV3 | LAP.PV3.20140730.20141024 | LAP.ThreePi | 2 | | o5444g0024o | XY75 | 219526 | 1179774 | update | update.LAP.PV3 | LAP.PV3.20140730.20141024 | LAP.ThreePi | 2 | | o5444g0009o | XY75 | 219511 | 1181481 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5444g0026o | XY75 | 219528 | 1181482 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5027g0338o | XY75 | 87852 | 1181498 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5462g0101o | XY26 | 229028 | 1181500 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5343g0321o | XY73 | 173486 | 1181536 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5343g0336o | XY76 | 173502 | 1181539 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5487g0040o | XY76 | 242068 | 1181546 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5487g0052o | XY75 | 242080 | 1181547 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5343g0330o | XY74 | 173494 | 1181573 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5343g0333o | XY73 | 173498 | 1181605 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5343g0372o | XY76 | 173537 | 1181945 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5362g0293o | XY75 | 181552 | 1182008 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5390g0030o | XY75 | 191840 | 1182127 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5390g0067o | XY76 | 191878 | 1182376 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5390g0068o | XY74 | 191879 | 1182678 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5365g0588o | XY73 | 183722 | 1182737 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5365g0590o | XY12 | 183730 | 1182738 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5365g0609o | XY74 | 183764 | 1182742 | update | update.LAP.PV3 | LAP.PV3.20140730.20141025 | LAP.ThreePi | 2 | | o5369g0248o | XY75 | 185907 | 1182788 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5369g0251o | XY27 | 185910 | 1182873 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5371g0256o | XY76 | 186536 | 1183057 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5371g0258o | XY75 | 186538 | 1183093 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5405g0451o | XY75 | 198470 | 1183305 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5405g0470o | XY74 | 198489 | 1183308 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5444g0013o | XY76 | 219515 | 1184003 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5444g0016o | XY76 | 219518 | 1184043 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5343g0353o | XY75 | 173517 | 1184369 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5343g0367o | XY74 | 173532 | 1184371 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5343g0369o | XY74 | 173534 | 1184372 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5404g0151o | XY75 | 197586 | 1184373 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5404g0172o | XY76 | 197607 | 1184374 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o6435g0293o | XY30 | 613989 | 1184406 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o6525g0133o | XY74 | 646008 | 1184407 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o6513g0165o | XY04 | 641174 | 1184566 | update | update.LAP.PV3 | LAP.PV3.20140730.ipp.20141026 | LAP.ThreePi | 2 | | o5343g0351o | XY47 | 173516 | 1184668 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | | o5344g0252o | XY75 | 174158 | 1184675 | update | update.LAP.PV3 | LAP.PV3.20140730.20141026 | LAP.ThreePi | 2 | +-------------+----------+--------+---------+--------+----------------+-------------------------------+-------------+-------+
- MEH: adding QUB.2 to pstamp server for bulk QUB requests (similar to MOPS.2, large/long request not needed ASAP), ps_ud_QUB.2 label and updates done in ippqub:stdscience_ws pantask -- lots of faults, czar not need to worry about at this time and due to the ongoing issue of multiple copies of needed files only on the ippb nodes and none on the actual IPP-ITC cluster...
- TdB: Around 21:00 I noticed that data was no longer downloading from the summit. A quick look at the summitcopy pantask stdout log showed that MySQL was down, which was causing the problem. Gene checked the process list for mysql:
ps aux | grep mys
it was not listed, so it had crashed or had been shutdown. Looking at /var/log/mysql/mysqld.err showed the messages about the crash. Gene restarted MySQL on ippdb09 as follows:restarting MySQL: /etc/init.d/mysql start (it complains about the PID file being left behind, but tries to start anyway. The start also takes too long, and the startup script gives up and claims the start fails. NOTE this is not true: you need to check the process list (ps aux | grep mys) and check for mysql. It will probably be there, it is just taking a long time to start up. You can check the log file (tail -f /var/log/mysql/mysqld.err) and wait for it to say "ready for connections.").
Once mysql was back up, nebulous started working again and downloads proceeded OK.
Also, I put ippb10-11 back to up to deal with the storage there. Temperature in the ARTC IT seems to be holding, as processing picked up.
Tuesday : 2019.04.23
- TdB: Following the fix to the AC system in the ATRC IT centre, the b-nodes that were put to down to reduce the temperature (ippb08-23) are now but pack up.
Following the AC fix, not all machines came back up without problems. ippb08 and ipp17 are reporting to be down in ganglia. For ippb08, a restart of gmond was needed:
sudo /etc/init.d/gmond restartippb17 seems not be mounting its datadirs. Doing less /etc/exports show no dirs are being exported at all, and doing sudo exportfs -s shows nothing is exported either. The exports file needs to be fixed, which required Gavin's help to copy the required data to the file. Finally, the ganglia daemon for the ganglia user account was not running, which was the reason for it still showing up as off on ganglia.
Wednesday : 2019.04.24
- TdB: There are two exposures stuck in the full state. send them to clean and then update again:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1185108 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1185115
followed by:chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1185108 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1185115
That cleared them out. There is also a warp error that need rerunning. A fault 4 with memory block corruption.chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5250g0237o chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1532684 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1479726
- MEH: sending more chips finished warp updates to cleanup later tonight --
- MEH: manually restarting ~ippitc/pstamp after a large backlog QUB request finished
Thursday : 2019.04.25
- MEH: queuing up a backlog of OSS.WS for distribution
- TdB: There are some missing burn tables that needed fixing, using:
ipp_apply_burntool_fix.pl --exp_name o6601g0088o --class_id XY61 --verbose --dbname gpc1
and subsequent reversion
There were also some files that cannot load ch.fits files. I sent them to clean and followed that by an update:
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1154188followed by:
warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id 1154188There were also some memory block issues, which were sent to be rerun from scratch:
chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6150g0465o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5803g0117o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5803g0135o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6284g0063o chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1202573 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1218742 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1218743 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1227699 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1151879 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1169382 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1169386 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1183696
Finally, there were also some files with multiple zero size files, for which non-zero copies where copied over.
Friday : 2019.04.26
- TdB: More burntools again:
ipp_apply_burntool_fix.pl --exp_name o5417g0140o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5408g0398o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5407g0515o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5438g0127o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5512g0091o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5518g0177o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5518g0178o --class_id XY10 --verbose --dbname gpc1
There are two files for which the burntool fix fails:
ipp_apply_burntool_fix.pl --exp_name o5391g0113o --class_id XY10 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5391g0123o --class_id XY10 --verbose --dbname gpc1It returns an error:
instance location: file:///data/ipp139.1/nebulous/f0/9d/11703263040.gpc1:20100714:o5391g0113o:o5391g0113o.ota10.burn.tbl Running [neb-mv neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.burn.tbl neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.burn.tbl.bad]... neb-rm --force neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.burn.tbl.bad /home/kiawe/eugene/src/psconfig/ipp-gentoo-ipp070.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 192159 --class_id XY10 --this_uri neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.fits --previous_uri neb://ipp008.0/gpc1/20100714/o5391g0112o/o5391g0112o.ota10.fits --verbose Running [/home/kiawe/eugene/src/psconfig/ipp-gentoo-ipp070.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 192159 --class_id XY10 --this_uri neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.fits --previous_uri neb://ipp008.0/gpc1/20100714/o5391g0112o/o5391g0112o.ota10.fits --verbose]... Argument "'/home/kiawe/eugene/src/psconfig/ipp-gentoo-ipp070.lin64..." isn't numeric in right bitshift (>>) at /home/kiawe/eugene/src/psconfig/ipp-gentoo-ipp070.lin64/bin/ipp_apply_burntool_fix.pl line 122. Unable to perform /home/kiawe/eugene/src/psconfig/ipp-gentoo-ipp070.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 192159 --class_id XY10 --this_uri neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.fits --previous_uri neb://ipp008.0/gpc1/20100714/o5391g0112o/o5391g0112o.ota10.fits --verbose: 4 at /home/kiawe/eugene/src/psconfig/ipp-gentoo-ipp070.lin64/bin/ipp_apply_burntool_fix.pl line 123
Mark suggested a fix, using:
~mhuber/ipp_apply_burntool_fix_force_nebmv_bad.pl --exp_name o5391g0113o --class_id XY10 --verbose --dbname gpc1But, this doesn't work either. After doing some simple math, we see:
mysql -hXXYXX -uXXYXX -pippuser gpc1 -B -e "SELECT * FROM rawExp where exp_name LIKE 'o5391g011%' limit 5;" +--------+-------------+--------+-----------+---------------------+--------------------+----------+-----------+------------------------------+-------+-----------+-------------------------------+----------+-----------+---------+-----------------------------------+----------+-----------+---------+----------+----------+----------+----------------+-------------+-----------+---------------+-----------+------------+----------+------------+------+------+-----------+----------+----------+---------+----------+----------+---------+---------+-----------------+--------------+----------------+--------------+------------+----------------+------------+----------------+---------------+---------------+----------+--------+--------+--------+--------+--------+-------------+-----------+----------+------------+----------+------------+----------+-------+---------------------+----------+ | exp_id | exp_name | camera | telescope | dateobs | exp_tag | exp_type | filelevel | workdir | state | reduction | dvodb | tess_id | end_stage | filter | comment | obs_mode | obs_group | airmass | ra | decl | exp_time | sat_pixel_frac | bg | bg_stdev | bg_mean_stdev | alt | az | ccd_temp | posang | m1_x | m1_y | m1_z | m1_tip | m1_tilt | m2_x | m2_y | m2_z | m2_tip | m2_tilt | env_temperature | env_humidity | env_wind_speed | env_wind_dir | teltemp_m1 | teltemp_m1cell | teltemp_m2 | teltemp_spider | teltemp_truss | teltemp_extra | pon_time | user_1 | user_2 | user_3 | user_4 | user_5 | object | sun_angle | sun_alt | moon_angle | moon_alt | moon_phase | hostname | fault | epoch | magicked | +--------+-------------+--------+-----------+---------------------+--------------------+----------+-----------+------------------------------+-------+-----------+-------------------------------+----------+-----------+---------+-----------------------------------+----------+-----------+---------+----------+----------+----------+----------------+-------------+-----------+---------------+-----------+------------+----------+------------+------+------+-----------+----------+----------+---------+----------+----------+---------+---------+-----------------+--------------+----------------+--------------+------------+----------------+------------+----------------+---------------+---------------+----------+--------+--------+--------+--------+--------+-------------+-----------+----------+------------+----------+------------+----------+-------+---------------------+----------+ | 192157 | o5391g0110o | GPC1 | PS1 | 2010-07-14 08:50:24 | o5391g0110o.192157 | OBJECT | CHIP | neb://@HOST@.0/gpc1/20100714 | full | NULL | NULL | NULL | reg | g.00000 | 129.3PI.g.E+4 ps1_06_2303 visit 1 | 3PI | NULL | 1.219 | 5.287517 | 0.176414 | 43 | 0.000406 | 6955.771484 | 14.335364 | 895.292603 | 55.079364 | 102.276326 | -80.72 | 280.371185 | 0 | 1000 | 0 | -18.9916 | -74.0019 | 471.602 | -508.621 | -1190.25 | 74.3442 | 204.864 | 10.8 | 16.2 | 1.2 | 50 | 10.6 | 10.8 | 9.2 | 8.75 | 10.6 | 8.4 | 362813 | NULL | NULL | NULL | NULL | NULL | ps1_06_2303 | 146.918 | -41.036 | 149.427 | -25.2161 | 0.093804 | ipp026 | 0 | 2010-07-13 22:55:06 | 0 | | 192158 | o5391g0112d | GPC1 | PS1 | 2010-07-14 08:52:48 | o5391g0112d.192158 | DARK | CHIP | neb://@HOST@.0/gpc1/20100714 | lossy | NULL | /data/ipp004.0/isp/catdir.isp | RINGS.V0 | reg | g.00000 | Shutter Error | NULL | NULL | 1.001 | 4.730797 | 0.35602 | 0.5 | 0.000495 | 6938.524414 | 14.544302 | 895.132568 | 88.000005 | 98.443298 | -80.7383 | 299.184387 | 0 | 1000 | -0.008933 | -19.0107 | -74.0066 | 471.57 | -500.008 | -1207.35 | 72.1602 | 204.855 | 10.8 | 16.6 | 2 | 50 | 10.6 | 10.8 | 9.2 | 8.85 | 10.65 | 8.4 | 362912 | NULL | NULL | NULL | NULL | NULL | NULL | 132.525 | -41.3271 | 117.573 | -25.723 | 0.093877 | ipp006 | 0 | 2010-07-13 22:55:42 | 0 | | 192159 | o5391g0113o | GPC1 | PS1 | 2010-07-14 08:54:36 | o5391g0113o.192159 | OBJECT | CHIP | neb://@HOST@.0/gpc1/20100714 | full | NULL | NULL | NULL | reg | g.00000 | 129.3PI.g.E+4 ps1_06_2314 visit 1 | 3PI | NULL | 1.228 | 5.328436 | 0.204925 | 43 | 0.000486 | 6954.918945 | 12.056411 | 892.011841 | 54.465026 | 98.850287 | -80.6633 | 280.685211 | 0 | 1000 | -0.008933 | -18.9944 | -73.9877 | 471.564 | -504.788 | -1201.25 | 73.3966 | 204.819 | 10.8 | 16.1 | 2.6 | 56 | 10.6 | 10.8 | 9.25 | 8.95 | 10.65 | 8.4 | 363064 | NULL | NULL | NULL | NULL | NULL | ps1_06_2314 | 144.666 | -41.5419 | 150.16 | -26.1025 | 0.093932 | ipp033 | 0 | 2010-07-13 22:58:40 | 0 | | 192160 | o5391g0114o | GPC1 | PS1 | 2010-07-14 08:55:32 | o5391g0114o.192160 | OBJECT | CHIP | neb://@HOST@.0/gpc1/20100714 | full | NULL | NULL | NULL | reg | g.00000 | 129.3PI.g.E+4 ps1_06_2214 visit 1 | 3PI | NULL | 1.244 | 5.331604 | 0.153198 | 43 | 0.000404 | 6959.689941 | 12.558017 | 895.28418 | 53.437656 | 103.575835 | -80.6417 | 280.709198 | 0 | 1000 | 0.017867 | -19.0025 | -74.0113 | 471.565 | -498.14 | -1217.05 | 71.5826 | 205.017 | 10.8 | 15.3 | 2.3 | 63 | 10.6 | 10.8 | 9.3 | 9 | 10.7 | 8.4 | 363119 | NULL | NULL | NULL | NULL | NULL | ps1_06_2214 | 147.392 | -41.6521 | 152.184 | -26.2991 | 0.09396 | ipp018 | 0 | 2010-07-13 22:59:48 | 0 | | 192162 | o5391g0115o | GPC1 | PS1 | 2010-07-14 08:56:28 | o5391g0115o.192162 | OBJECT | CHIP | neb://@HOST@.0/gpc1/20100714 | full | NULL | NULL | NULL | reg | g.00000 | 129.3PI.g.E+4 ps1_06_2315 visit 1 | 3PI | NULL | 1.268 | 5.373886 | 0.181498 | 43 | 0.000456 | 6971.907227 | 14.595263 | 895.269653 | 52.035263 | 99.767319 | -80.6267 | 281.252014 | 0 | 1000 | 0.008933 | -18.9889 | -74.0066 | 471.602 | -489.404 | -1238.95 | 69.3174 | 204.719 | 10.8 | 15 | 2.3 | 50 | 10.6 | 10.8 | 9.35 | 9 | 10.65 | 8.4 | 363176 | NULL | NULL | NULL | NULL | NULL | ps1_06_2315 | 144.988 | -41.7614 | 152.941 | -26.4955 | 0.093989 | ipp010 | 0 | 2010-07-13 23:01:14 | 0 | +--------+-------------+--------+-----------+---------------------+--------------------+----------+-----------+------------------------------+-------+-----------+-------------------------------+----------+-----------+---------+-----------------------------------+----------+-----------+---------+----------+----------+----------+----------------+-------------+-----------+---------------+-----------+------------+----------+------------+------+------+-----------+----------+----------+---------+----------+----------+---------+---------+-----------------+--------------+----------------+--------------+------------+----------------+------------+----------------+---------------+---------------+----------+--------+--------+--------+--------+--------+-------------+-----------+----------+------------+----------+------------+----------+-------+---------------------+----------+
We see that the previous file from which the code tries to grab the burn table is in fact a dark image, which coincidentally has been compressed:
neb-ls -p neb://ipp008.0/gpc1/20100714/o5391g0112d/o5391g0112d.ota10.fits /data/ippb09.1/nebulous/17/e7/532164132.gpc1:20100714:o5391g0112d:o5391g0112d.ota10.fits.fzBut, the image before that one looks ok. So, manually call the script using that one instead:
/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 192159 --class_id XY10 --this_uri neb://ipp008.0/gpc1/20100714/o5391g0113o/o5391g0113o.ota10.fits --previous_uri neb://ipp008.0/gpc1/20100714/o5391g0110o/o5391g0110o.ota10.fits --verbose
Similarly, for exposure o5391g0123o:
/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/ipp_apply_burntool_single.pl --dbname gpc1 --camera GPC1 --exp_id 192168 --class_id XY10 --this_uri neb://ipp008.0/gpc1/20100714/o5391g0123o/o5391g0123o.ota10.fits --previous_uri neb://ipp008.0/gpc1/20100714/o5391g0119o/o5391g0119o.ota10.fits --verbose
----------------
There is a whole heap of warp exposure faulting due to missing input mask fits files. Turns out these are non-existent neb keys. re-run from scratch:
chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6549g0194o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5481g0225o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5481g0233o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6535g0419o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6545g0339o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6600g0149o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6557g0297o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6910g0289o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6911g0440o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5887g0014o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5438g0165o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5467g0099o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6276g0139o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5756g0429o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5751g0485o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6538g0365o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5879g0072o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6226g0279o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5466g0210o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5486g0109o chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1224184 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1224312 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1225199 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1225226 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1225229 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1225241 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1226316 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1226323 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1226492 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1248818 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1248821 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1255405 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1259404 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1260099 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1261009 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1263639 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1263666 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1264333 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1264770 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1264860 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1191586 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1191999 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1192321 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1192351 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1192357 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1192366 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1192893 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1192903 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1193060 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1196803 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1196835 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1202458 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1209625 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1209906 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1210332 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1213020 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1213068 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1215632 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1215931 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1216023
Fault 4 memory blocks. re-run as well:
chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5479g0127o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6961g0109o chiptool -dbname gpc1 -definebyquery -set_label ps_ud_WEB -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6224g0154o chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1263620 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1264673 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1264883 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1213010 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1215908 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1216046
Saturday : 2019.04.27
- MEH: queuing night stack and SSdiff once normal nightly processing finished for OSS.nightlyscience
Sunday : 2019.04.28
Note:
See TracWiki
for help on using the wiki.
