#1138 closed defect (invalid)
Random psFitsOpen and fitsClose failures during ppImage via NFS
| Reported by: | Owned by: | Paul Price | |
|---|---|---|---|
| Priority: | high | Milestone: | |
| Component: | fits | Version: | unspecified |
| Severity: | minor | Keywords: | |
| Cc: |
Description
I just processed 1688 SDSS pixel files through register, chip and cam. 3 of them fail with errors when opening or closing the fits files. All 3 were read or written via NFS.
The error messages I got were:
2008-09-15 15:34:07Z|aida78 |4|ppImageLoop
Chip 5: 0 1
Unable to loop over input
-> psFitsOpen (psFits.c:174): I/O error
deleted file still exists!
-> pmFPAfileOpen (pmFPAfileIO.c:750): I/O error
error opening file /IPP/data/SDSS/stripe82/coadd/testrun_ipp/1056-0206.15/1056-0206.15.ch.15.smf
-> pmFPAfileWrite (pmFPAfileIO.c:391): I/O error
failed to open /IPP/data/SDSS/stripe82/coadd/testrun_ipp/1056-0206.15/1056-0206.15.ch.15.smf (PSASTRO.OUTPUT)
-> pmFPAfileIOChecks (pmFPAfileIO.c:75): I/O error
failed WRITE in FPA_AFTER block for PSASTRO.OUTPUT
-> ppImageLoop (ppImageLoop.c:140): unknown psLib error
error running astrometry.
Found 0 leaks at ppImage
Unable to perform ppImage: 2 at /IPP/bleed.lin64/bin/chip_imfile.pl line 270
main::my_die('Unable to perform ppImage: 2', 15, 15, 'CHIP1', 2) called at /IPP/bleed.lin64/bin/chip_imfile.pl line 173
2008-09-15 15:52:11Z|aida79 |4|ppImageLoop
Chip 5: 0 1
Unable to loop over input
-> fitsClose (psFits.c:98): file doesn't obey FITS standard
Error while closing psFits object
-> pmFPAfileClose (pmFPAfileIO.c:556): unknown psLib error
failed to close /IPP/data/SDSS/stripe82/coadd/testrun_ipp/1755-0344.35/1755-0344.35.ch.35.smf (PSASTRO.OUTPUT) (-1:-1:-1)
-> pmFPAfileIOChecks (pmFPAfileIO.c:79): I/O error
failed CLOSE in FPA_AFTER block for PSASTRO.OUTPUT
-> ppImageLoop (ppImageLoop.c:140): unknown psLib error
error running astrometry.
Found 0 leaks at ppImage
Unable to perform ppImage: 2 at /IPP/bleed.lin64/bin/chip_imfile.pl line 270
main::my_die('Unable to perform ppImage: 2', 35, 35, 'CHIP4', 2) called at /IPP/bleed.lin64/bin/chip_imfile.pl line 173
The first error happened twice, the second one once (low-number statistics, but I've seen the second one before).
The files process fine if I restart them, i.e. it's nothing to do with the file as such - just random other things, and I'm guessing it's something to do with NFS since it only ever happens for jobs that run on hosts other than the one carrying the disks with data and work directory.
I sent an email about this before when I saw it in 2.6.1, but since it persists and there is no real reason for failure, I thought it would be useful for robustness' sake to track this down.
This is in a version exported from cvs head on 2008-09-12
Yeah, we get similar statistics, and we blame it on NFS too. Typically a few tasks will fail at the start of a run (just after you enter "run" in pantasks) with error code 2, presumably because of NFS timeouts when lots of jobs are requesting data from the same place at the same time.
I recommend striping your data across machines (you may want to look into using Nebulous) and tuning your NFS and disk parameters. You can also just run a revert job explicitly selecting code 2 so that these files end up back in the queue and get processed.