Changes between Version 2 and Version 3 of PS1_IPP_Czarlog_20190204

Timestamp:

Feb 5, 2019, 3:29:24 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190204

@@ -27 +27 @@
   JRF: Will be meeting with Mark later to go through some errors that were reported in the logs (TODO: update this)
 
+JRF: stdscience error log reported too many connections to the database. This had come up previously (in the past month):
+{{{
+ -> psDBAlloc (psDB.c:166): Database error generated by the server
+     Failed to connect to database.  Error: Too many connections
+ -> faketoolConfig (faketoolConfig.c:373): unknown psLib error
+     Can't configure database
+ -> main (faketool.c:70): (null)
+     failed to configure
+Unable to perform faketool -addproces
+
+config error for: fake_imfile.pl --exp_id 1450315 --fake_id 2062988 --class_id XY31 --chiproot=neb://ipp090.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.ch.2127506 --camroot=neb://any/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.cm.2093978 --camera GPC1 --outroot neb://ipp090.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.fk.2062988 --reduction SWEETSPOT --dbname gpc1 --verbose
+job exit status: 3
+job host: ippc69
+job dtime: 3.755007
+job exit date: Mon Feb  4 20:22:07 2019
+*** stdout ***
+stdout 13038
+}}}
+
+and another:
+
+{{{
+ -> psDBAlloc (psDB.c:166): Database error generated by the server
+     Failed to connect to database.  Error: Too many connections
+ -> faketoolConfig (faketoolConfig.c:373): unknown psLib error
+     Can't configure database
+ -> main (faketool.c:70): (null)
+     failed to configure
+Unable to perform faketool -addproces
+
+config error for: fake_imfile.pl --exp_id 1450315 --fake_id 2062988 --class_id XY54 --chiproot=neb://ipp112.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.ch.2127506 --camroot=neb://any/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.cm.2093978 --camera GPC1 --outroot neb://ipp112.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.fk.2062988 --reduction SWEETSPOT --dbname gpc1 --verbose
+job exit status: 3
+job host: ippc79
+job dtime: 3.872324
+job exit date: Mon Feb  4 20:22:07 2019
+*** stdout ***
+stdout 11212
+}}}
+
+JRF: The stdscience log last night on PS1 showed a bunch of errors reported around 20:30. The majority of these were on ippc118. Look at the ganglia logs there was a spike in network activity, up to about 17Mb, but not too large that it should cause an issue. Either way the tasks that failed reverted successfully and continued on after this.
+
+JRF: MEH will look into the reported nightly_science.pl errors. Some are reported for gpc2 in the gpc1 logs!!
+e.g.
+
+{{{
+failure for: nightly_science.pl --queue_diffs --date 2019-02-05 --dbname gpc2 --camera GPC2
+job exit status: 2
+job host: localhost
+job dtime: 8.072818
+job exit date: Mon Feb  4 21:04:02 2019
+}}}
+Commented out the use of gpc2 dates in the 'stdscience/input' file. Believe this is the reason that there are gpc2 errors being reported in the gpc1 logs above.
+
+JRF: killed a 4 day old job in the database that was run as ippuser to query something. WARNING: very dangerous to go around killing jobs.
 
 === Wednesday : 2019.02.06 ===
+
+JRF: ME, MEH, and TdB shall be going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area).
 
 === Thursday : 2019.02.07 ===