Changes between Version 16 and Version 17 of PS1_IPP_Czarlog_20190204

Timestamp:

Feb 7, 2019, 4:03:54 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190204

@@ -157 +157 @@
 === Thursday : 2019.02.07 ===
 
+JRF: No data last night, but there were a bunch of errors from some reprocessing that MEH started.
+
+'''Checking error logs:'''
+A typical error from the ippitc/stdscience log:
+{{{
+Starting script /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl on ipp109 at Wed Feb  6 18:40:41 
+*** stderr ***
+
+system failure for: warp_skycell.pl --threads @MAX_THREADS@ --warp_id 440532 --warp_skyfile_id 37161875 --skycell_id skycell.0868.085 --tess_dir RINGS.V3 --camera GPC1 --outroot neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.wrp.440532.skycell.0868.085 --redirect-output --run-state update  --dbname gpc1 --verbose
+job exit status: 2
+job host: ipp109
+job dtime: 3.714778
+job exit date: Wed Feb  6 18:40:45 2019
+*** stdout ***
+stdout 936
+}}}
+The majority of the errors all occurred around 18:40. Check out the error in more detail by following the {{outroot}} provided in the error message using neb-ls
+Then neb-locate the log and have a read (in this case it is the update log as the state is update). Curiously the log file was b-zipped, which they should not be normally and might have perhaps caused the problem.
+
+Looking at the bottom of the file we get the error:
+
+{{{
+neb entry neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.wrp.440532.skycell.0868.085.skycell not found, not created
+ at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 210
+Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/dvoImageExtract -D CATDIR /local/ipp/gpc1/tess/RINGS.V3 skycell.0868.085 -o /data/ipp123.0/nebulous/e6/f8/10999657730.gpc1:ecliptic:20:00hr:o5773g0307o.370390:o5773g0307o.370390.wrp.440532.skycell.0868.085.skycell]...
+matched chips for 0 images, 0 failed (264400 total)
+2019/02/06 18:40:44 | ipp109 | FATAL | Nebulous::Client::find_instances - unhandled fault - database error: no instances available for key: neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.cm.458814.XY76.mk.fits at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 1991, <DATA> line 14.
+Unable to find instance of Nebulous handle neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.cm.458814.XY76.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 237
+Couldn't find input file: neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.cm.458814.XY76.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563.
+Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warptool -updateskyfile -warp_id 440532 -skycell_id skycell.0868.085 -fault 2 -dbname gpc1]...
+}}}
+It appears that it couldn't find the fits file it wanted, so we should check the current state of the exposure.
+
+Query the chipRun and rawExp table:
+{{{
+SELECT exp_name, exp_id, label, data_group, chipRun.state FROM rawExp LEFT JOIN chipRun USING (exp_id) WHERE exp_name="o5773g0307o";
+}}}
+
+The chip state is still stuck in update, so we should sent it to be fixed (details of this are given on the page for PSNSC_MOPS:[http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PSNSC_MOPS]):
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o5773g0307o
+}}}
+
+For the above to be actually processed the label needs to be added to the stdscience pantasks. Use 'add.label PSNSC.wfix.20180525' in the client.
+
+There is labeltool for viewing all the priorities of the labels. Higher number -> higher priority.
+{{{
+labeltool -dbname gpc1 -listlabel -simple
+}}}
+
+
+'''Clearing IPPmonitor errors:'''
+Next, there were some errors on IPP monitor for ps_ud_MOPS. It turns out these are mostly red herrings. The problems were only in parentheses on IPP monitor, so there isn't a proper problem but some errors reported that should be looked at.
+
+Clicking the links there is a problem with a skycell warp? There is an error log reported on the page, so we can find that from nebulous and look at the issue.
+
+For each chip_id do the following to update it:
+{{{
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1059958
+}}}
+
+
+Once the chip stage is done set the warp_ids to update too:
+{{{
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX
+}}}
+
+
+This should clear the errors.
+
+
+'''Crontabs:'''
+These are important for keeping nightly processing flowing smoothly as they correct some faults that crop up.
+
+Under ippuser home there is a folder called 'cron_tab_c30', the name informs us that the cron_tab is on ippc30.
+Going to this machine and viewing the crontabs (crontab -l), there are few in operation. Notably one of them is to correct the fault 4 memory issue.
+
+
+'''Taking nodes down:'''
+There were two nodes that had pretty full disks >97%. It's only one particular volume on each so we used the neb-host command to alter the status and specified that it should just be for that volume:
+{{{
+neb-host --host ipp114.0 --state repair --volume --note "Disk is getting very full, so taking down"
+}}}
+
+
+=== Friday : 2019.02.08 ===
+
 JRF: ME, MEH, and TdB shall be going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area).
 
-=== Friday : 2019.02.08 ===
-
 === Saturday : 2019.02.09 ===