fs: fix close listener leak in FileHandle streams - #64227
Conversation
b433fb3 to
c4a165a
Compare
Fixes: nodejs#64214 Signed-off-by: y1d7ng <[email protected]>
Fixes: nodejs#64214 Signed-off-by: y1d7ng <[email protected]>
| function cleanup() { | ||
| options.fd.removeListener('close', onclose); | ||
| options.fd[kUnref](); | ||
| } | ||
| stream.once('end', cleanup); | ||
| stream.once('finish', cleanup); | ||
| stream.once('error', cleanup); |
There was a problem hiding this comment.
@Y1D7NG should we removeListener for all 3 in cleanup, so we don't leave these hanging as well?
| function cleanup() { | |
| options.fd.removeListener('close', onclose); | |
| options.fd[kUnref](); | |
| } | |
| stream.once('end', cleanup); | |
| stream.once('finish', cleanup); | |
| stream.once('error', cleanup); | |
| function cleanup() { | |
| options.fd.removeListener('close', onclose); | |
| stream.removeListener('end', cleanup); | |
| stream.removeListener('finish', cleanup); | |
| stream.removeListener('error', cleanup); | |
| options.fd[kUnref](); | |
| } | |
| stream.once('end', cleanup); | |
| stream.once('finish', cleanup); | |
| stream.once('error', cleanup); |
There was a problem hiding this comment.
From what I'm aware of for similar functions, it's common practice to (intentionally) leave these listeners dangling on the stream and rely on normal garbage collection to tidy it all up in the end.
There was some discussion on this stuff in #35452. In this case, the duplex stuff doesn't apply so I'd personally say it doesn't need the special handling that was eventually added there.
There was a problem hiding this comment.
Thanks for bringing that, that's interesting. Unsure how to proceed then. @nodejs/streams is the current behavior as expected?
There was a problem hiding this comment.
to be clear: in my comment I'm only talking about the events on the stream (end/finish/error) - removing the close listener from the file descriptor (i.e. the purpose of this PR) is definitely necessary.
There was a problem hiding this comment.
There are 2 approvals and a green CI.
Could this PR be landed, or does it need clarification of the above comments first?
There was a problem hiding this comment.
My comments are non-blocking, although I don't see it hurting waiting extra time to get insights from the right team tho.
There was a problem hiding this comment.
I think the listeners can be left to normal garbage collection
There was a problem hiding this comment.
After waiting 7 days there haven't been any further comments, or any blocking objections, so adding to commit queue.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
This is broken and should be reverted. Will unref the handle multiple times. |
That is very unfortunate 🙁 ! Would you want to initiate the revert? I'm not sure how best to handle this. |
|
I'll submit a revert PR, if the original approvers and @ronag would be so kind as to approve the revert as well. |
I can approve. But for future reference, that's why we should wait for the subject matter experts to also approve. Even if that means the PR taking its due time to be merged. |
|
My mistake for not waiting long enough. Very sorry! |
In all fairness, it waited 7 days as required 🙂 |
This reverts commit 8488e13. It was advised that the fix in question is broken since it will unref the handle multiple times. Signed-off-by: Mike McCready <[email protected]> PR-URL: #65387 Refs: #64227 Refs: #64214 Reviewed-By: Claudio Wunder <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
Yeah, not assigning blame. After all I also approved the PR. But I feel that for such reasons waiting for more time till we get full confirmation could be helpful. |
Fixes: #64214 Signed-off-by: y1d7ng <[email protected]> PR-URL: #64227 Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Claudio Wunder <[email protected]>
This reverts commit 8488e13. It was advised that the fix in question is broken since it will unref the handle multiple times. Signed-off-by: Mike McCready <[email protected]> PR-URL: #65387 Refs: #64227 Refs: #64214 Reviewed-By: Claudio Wunder <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
Fixes: #64214 Signed-off-by: y1d7ng <[email protected]> PR-URL: #64227 Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Claudio Wunder <[email protected]>
This reverts commit 8488e13. It was advised that the fix in question is broken since it will unref the handle multiple times. Signed-off-by: Mike McCready <[email protected]> PR-URL: #65387 Refs: #64227 Refs: #64214 Reviewed-By: Claudio Wunder <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
Fixes: #64214 Signed-off-by: y1d7ng <[email protected]> PR-URL: #64227 Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Claudio Wunder <[email protected]>
This reverts commit 8488e13. It was advised that the fix in question is broken since it will unref the handle multiple times. Signed-off-by: Mike McCready <[email protected]> PR-URL: #65387 Refs: #64227 Refs: #64214 Reviewed-By: Claudio Wunder <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
|
Should be removed from #65873 |
Fixes: #64214 Signed-off-by: y1d7ng <[email protected]> PR-URL: #64227 Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Claudio Wunder <[email protected]>
This reverts commit 8488e13. It was advised that the fix in question is broken since it will unref the handle multiple times. Signed-off-by: Mike McCready <[email protected]> PR-URL: #65387 Refs: #64227 Refs: #64214 Reviewed-By: Claudio Wunder <[email protected]> Reviewed-By: Chemi Atlow <[email protected]> Reviewed-By: Robert Nagy <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
s/node/pull/64229 Signed-off-by: Rasad Regmi <[email protected]> Title: fs: fix FileHandle close listener/ref leak in streams Description: `fileHandle.createReadStream()`/`createWriteStream()` with `autoClose: false` leaks a `'close'` listener (and an internal reference) on the `FileHandle` every time the stream finishes on its own, because `autoClose: false` also disables `autoDestroy`, so the stream never reaches `_destroy()` — the only place that previously released them. Enough such streams on one long-lived handle trips `MaxListenersExceededWarning`. `importFd()` in `lib/internal/fs/streams.js` now builds a single idempotent `releaseHandleRef` shared between `FileHandleOperations .close()` (the normal destroy path, unchanged for the default `autoClose: true` behavior) and a `finished()`-based safety net registered after the stream's readable/writable state is initialized, for streams that finish without ever calling `destroy()`. `finished()` correctly resolves on `'end'`/`'finish'` alone when the stream won't emit `'close'` (see `willEmitClose()` in `internal/streams/utils.js`), and the shared idempotency guard means it's safe even if a stream is also explicitly closed after finishing on its own — the exact scenario that caused a prior fix (nodejs#64227) to fs: fix FileHandle close listener/ref leak in streams createReadStream()/createWriteStream({ autoClose: false }) created from a FileHandle attached a 'close' listener and took a reference on the handle in importFd(), but only released either when the stream went through _destroy(). Since autoClose: false also disables autoDestroy, a stream that finishes on its own never reaches _destroy(), so the listener and reference leaked. Creating enough such streams on a long-lived handle (e.g. repeated ranged reads) eventually tripped MaxListenersExceededWarning. A prior fix (fixed in 64227, reverted in 65387) released the reference again on the stream's 'end'/'finish'/'error' independent of the destroy path, which could unref the handle twice if the stream was later also explicitly closed/destroyed - a normal thing to do after a stream naturally ends. This introduces a single idempotent release function shared by both paths (the normal destroy path and a finished()-based safety net for autoClose: false streams that never reach _destroy()), so the reference and listener are released exactly once regardless of how the stream ends. Fixes: nodejs#64214 Refs: nodejs#64227 Refs: nodejs#65387 Refs: nodejs#64229 Signed-off-by: Rasad Regmi <[email protected]> Manual Git Steps (for you to run) cd "/home/rasadregmi/Desktop/Open Source Contributions Repos/node" git checkout -b fix-filehandle-stream-close-listener-leak main git add lib/internal/fs/streams.js test/parallel/test-fs-promises-file-hand git commit -s git push fork fix-filehandle-stream-close-listener-leak Commit Message fs: fix FileHandle close listener/ref leak in streams createReadStream()/createWriteStream({ autoClose: false }) created from a FileHandle attached a 'close' listener and took a reference on the handle in importFd(), but only released either when the stream went through _destroy(). Since autoClose: false also disables autoDestroy, a stream that finishes on its own never reaches _destroy(), so the listener and reference leaked. Creating enough such streams on a long-lived handle (e.g. repeated ranged reads) eventually tripped MaxListenersExceededWarning. A prior fix (fixed in 64227, reverted in 65387) released the reference again on the stream's 'end'/'finish'/'error' independent of the destroy path, which could unref the handle twice if the stream was later also explicitly closed/destroyed - a normal thing to do after a stream naturally ends. This introduces a single idempotent release function shared by both paths (the normal destroy path and a finished()-based safety net for autoClose: false streams that never reach _destroy()), so the reference and listener are released exactly once regardless of how the stream ends. Fixes: nodejs#64214 Refs: nodejs#64227 Refs: nodejs#65387 Refs: nodejs#64229 Signed-off-by: Rasad Regmi <[email protected]>
createReadStream()/createWriteStream({ autoClose: false }) created
from a FileHandle attached a 'close' listener and took a reference
on the handle in importFd(), but only released either when the
stream went through _destroy(). Since autoClose: false also
disables autoDestroy, a stream that finishes on its own never
reaches _destroy(), so the listener and reference leaked. Creating
enough such streams on a long-lived handle (e.g. repeated ranged
reads) eventually tripped MaxListenersExceededWarning.
A prior fix (fixed in 64227, reverted in 65387) released the
reference again on the stream's 'end'/'finish'/'error' independent
of the destroy path, which could unref the handle twice if the
stream was later also explicitly closed/destroyed - a normal thing
to do after a stream naturally ends.
This introduces a single idempotent release function shared by both
paths (the normal destroy path and a finished()-based safety net for
autoClose: false streams that never reach _destroy()), so the
reference and listener are released exactly once regardless of how
the stream ends.
Fixes: nodejs#64214
Refs: nodejs#64227
Refs: nodejs#65387
Refs: nodejs#64229
Signed-off-by: Rasad Regmi <[email protected]>
fix: #64214
When autoClose is false, FileHandle.createReadStream registers a close listener on the handle. That listener is not removed after the stream finishes normally, so creating streams repeatedly on the same handle accumulates listeners.
In importFd, when autoClose is false, remove that listener and perform the corresponding unref cleanup when the stream ends (on end for read streams, on finish for write streams) or on error.