dettool runs sometimes fail in a not reproduceable way

[koppenh@…]

Creating master detrend images, I found that sometimes different dettool runs
fail with one of the following two messages:

1)
Running [/home/panstarrs/koppenh/psconfig/ipp-1.1.linux/bin/ppImage -dbname
koppenh -file /data/ipp000.1/koppenh/M33/raw/bias/split/766198b.ccd21.fits
/data/ipp000.1/koppenh/M33/mkbias.2/766198b.ccd21.5/766198b.ccd21.5.detproc.11
-recipe PPIMAGE PPIMAGE_O -stat
/data/ipp000.1/koppenh/M33/mkbias.2/766198b.ccd21.5/766198b.ccd21.5.detproc.11.ccd21.stats]...
* stderr *
Unable to parse camera. -> psFitsOpen (psFits.c:165): I/O error

Could not open

file,'/data/ipp000.1/koppenh/M33/raw/bias/split/766198b.ccd21.fits'. CFITSIO
Error: could not open the named file

-> pmFPAfileDefineFromArgs (pmFPAfileDefine.c:294): I/O error

Failed to open file

/data/ipp000.1/koppenh/M33/raw/bias/split/766198b.ccd21.fits

-> ppImageParseCamera (ppImageParseCamera.c:15): I/O error

Failed to build FPA from PPIMAGE.INPUT

Unable to perform ppImage: 3 at
/home/panstarrs/koppenh/psconfig/ipp-1.1.linux/bin/detrend_process_imfile.pl
line 168.

-> addprocessedimfileMode (dettool.c:2141): unknown error

-uri is required

config error for: detrend_process_imfile.pl --det_id 11 --det_type bias
--exp_tag 766198b.ccd21.5 --class chip --class_id ccd21 --input_uri
file:/data/ipp000.1/koppenh/M33/raw/bias/split/766198b.ccd21.fits --camera
MegaCam --workdir /data/ipp000.1/koppenh/M33/mkbias.2 --dbname koppenh
job exit status: 3
job dtime: 4.42049

2)
Running [/home/panstarrs/koppenh/psconfig/ipp-1.1.linux/bin/ppImage -list
/data/ipp000.1/koppenh/M33/mkbias.1/718964b.ccd21.2/718964b.ccd21.2.detproc.10.b1.list
/data/ipp000.1/koppenh/M33/mkbias.1/718964b.ccd21.2/718964b.ccd21.2.detproc.10
-recipe PPIMAGE PPIMAGE_J1]...
* stderr *
print() on closed filehandle $list1File at
/home/panstarrs/koppenh/psconfig/ipp-1.1.linux/bin/detrend_process_exp.pl line 134.
print() on closed filehandle $list2File at
/home/panstarrs/koppenh/psconfig/ipp-1.1.linux/bin/detrend_process_exp.pl line 135.
Aborting. Error stack:2007-03-09 20:06:01Z|po22 |A|psLib.sys

unable to open specified list file

Unable to perform ppImage: 4 at
/home/panstarrs/koppenh/psconfig/ipp-1.1.linux/bin/detrend_process_exp.pl line 200.

-> addprocessedexpMode (dettool.c:2397): unknown error

-recip is required

programming error for: detrend_process_exp.pl --det_id 10 --exp_tag
718964b.ccd21.2 --camera MegaCam --workdir /data/ipp000.1/koppenh/M33/mkbias.1
--dbname koppenh
job exit status: 4
job dtime: 4.516444

The files are definitely there, and when the scheduler is rerunning the jobs
they finish. I think it could be a problem with nfs and the high loads on ipp000
and po13 where I am runnung pantasks.

[Paul Price]

We've noticed similar problems when reducing large data sets. Two ways forward
are fiddling around with NFS, and debugging the high load on pantasks. The
latter is a high priority item.

In the mean time, to minimise this problem, I suggest:

• do not run pantasks on the machine on which the data is located
• do not use the machine running pantasks as a compute node

i.e., run pantasks on a dedicated machine.

[jhoblitt]

assuming this issue went away...