| Version 4 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.04.26 - 2019.05.02
(Up to PS1 IPP Czar Logs)
Check out the new date!
Friday : 2019.04.26
JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same.
Saturday : 2019.04.27
Sunday : 2019.04.28
Missing mdc file + burntool problem by JRF
Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning) Will reprocess:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o
Check they're finished:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o");
Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool:
o5444g0110o XY10 219612 2142808 new mopsfix mops.fixbrokenLAP.20190428 NULL 2
Perhaps try:
ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1
Tell it to update?
chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 2142808
Missing mk.fits files: by JRF Leaving jobs stuck in the warp stage
Try to update them first:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id XXXXXX warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX
One of the ones sent to update came back complaining about a missing file, but it is there, so try again:
o6224g0254o 537186 1281637 1295352 1270256 1276893 skycell.1060.063 update ps_ud_WEB chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893
Failing that, reprocess them:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name
have sent them to reprocess, check status soon!
The o6624g0189o exposure is done now, so clean up the remaining errors under the original label: o6624g0189o
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762
As for o6224g0254o, there is still an issue: It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size). Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones.
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893
Then, set to update again:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893
Evening
There are now loads of broken jobs (must have hit a bad patch).
Greater than 600 jobs...but only around 50 unique warp ids
Best action is probably... set to clean, then update?
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id xxxxxx warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx
missing 2 darks by JRF
Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it
Now, this is where I made a mistake. DO NOT DO THIS
check_server.sh start.server summitcopy
Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running...
So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and: {{{stop shutdown now }}} make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up:
pantasks_server & pantasks_client server input input setup run
Note that you need to be in the correct directory, and logged in as the correct user.
The problem was not solved...
Questions? How come there is no cleanup/stack stage for ps2?
Monday : 2019.04.29
Still have the dark issue.
Still a bunch of broken jobs to work through, so here we go!
