PS1 IPP Czar Logs for the week 2019.04.26 - 2019.05.02

(Up to PS1 IPP Czar Logs)

Check out the new date!

Friday : 2019.04.26

JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same.

Saturday : 2019.04.27

Sunday : 2019.04.28

Missing mdc file + burntool problem by JRF

Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning) Will reprocess: chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o Check they're finished: SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o");

Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool: o5444g0110o XY10 219612 2142808 new mopsfix mops.fixbrokenLAP.20190428 NULL 2

Perhaps try: ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1

Tell it to update? chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 2142808

Missing mk.fits files: by JRF Leaving jobs stuck in the warp stage

Try to update them first: {{{chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id XXXXXX warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX}}}

One of the ones sent to update came back complaining about a missing file, but it is there, so try again: {{{o6224g0254o 537186 1281637 1295352 1270256 1276893 skycell.1060.063 update ps_ud_WEB chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893}}}

Failing that, reprocess them: chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name have sent them to reprocess, check status soon!

The o6624g0189o exposure is done now, so clean up the remaining errors under the original label: o6624g0189o

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762

As for o6224g0254o, there is still an issue: It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size). Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones. {{{chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893}}} Then, set to update again: {{{chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893}}}

Evening

There are now loads of broken jobs (must have hit a bad patch).

Greater than 600 jobs...but only around 50 unique warp ids

Best action is probably... set to clean, then update?

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id xxxxxx
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx

missing 2 darks by JRF

Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it

Now, this is where I made a mistake. DO NOT DO THIS {{{check_server.sh start.server summitcopy }}} Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running...

So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and: {{{stop shutdown now}}} make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up:

pantasks_server & pantasks_client
server input input
setup
run}}}
Note that you need to be in the correct directory, and logged in as the correct user.

The problem was not solved...

Questions?
How come there is no cleanup/stack stage for ps2?


=== Monday : 2019.04.29 ===

Still have the dark issue.


Still a bunch of broken jobs to work through, so here we go!

  
=== Tuesday : 2019.04.30 ===

=== Wednesday : 2019.05.01 ===

=== Thursday : 2019.05.02 ===