Skip to content

Connector.close() + pool.end() leaks TLS sockets for in-flight queries #562

Description

@kym6464

Bug Description

Connector.close() followed by pool.end() leaves TLS sockets alive for any mysql2 pool connection that was mid-query when shutdown began. The leak is 1:1 with the number of in-flight queries: with 5 concurrent SELECT SLEEP(2) queries running, whyIsNodeRunning() 500 ms after shutdown reports 5 TLSWRAP handles still pinning the event loop. Their stacks root at:

@google-cloud/cloud-sql-connector/dist/mjs/socket.js:107 — tls.connect(socketOpts)
@google-cloud/cloud-sql-connector/dist/mjs/connector.js:148 — getSocket(...)
mysql2/lib/base/connection.js:64 — this.stream = opts.config.stream(opts)

Once the socket count grows past a few (e.g. a serving process handling concurrent traffic at the moment SIGTERM arrives), the process can fail to exit on graceful shutdown, relying on a forced kill.

Reproduces on both @google-cloud/[email protected] and @google-cloud/[email protected] (latest as of filing).

Suspected cause: CloudSQLInstance.close() destroys sockets in this.sockets, but pool connections that are currently checked out (mid-query) don't appear to be tracked there. Additionally, Connector.close() is synchronous but internally fire-and-forget (instance.promise.then(inst => inst.close())), so caller code that immediately follows close() with pool.end() races the microtask that actually destroys instance sockets.

Example code (or command)

import mysql from 'mysql2/promise';
import { AuthTypes, Connector } from '@google-cloud/cloud-sql-connector';
import whyIsNodeRunning from 'why-is-node-running';

const INSTANCE_CONNECTION_NAME = 'TODO: your-project:region:instance';
const DATABASE = 'TODO: your-database';
const USER = 'TODO: your-iam-user';

async function main() {
  const connector = new Connector();
  const clientOpts = await connector.getOptions({
    instanceConnectionName: INSTANCE_CONNECTION_NAME,
    authType: AuthTypes.IAM,
  });
  const pool = mysql.createPool({
    ...clientOpts,
    user: USER,
    database: DATABASE,
  });

  // Fire concurrent in-flight queries without awaiting — simulates request
  // handlers still running when shutdown begins.
  const queries = Array.from({ length: 5 }, (_, i) =>
    pool.query('SELECT SLEEP(2), ?', [i]).catch(() => {})
  );

  // Let the queries actually open TLS sockets.
  await new Promise((r) => setTimeout(r, 100));

  await pool.end();
  connector.close();

  // Let libuv settle (collect deactivated timers, flush pending writes).
  await new Promise((r) => setTimeout(r, 500));

  whyIsNodeRunning();

  await Promise.allSettled(queries);
}

main();

Stacktrace

Output of whyIsNodeRunning() 500 ms after pool.end() + connector.close():

% node temp.direct.mjs

...

# TLSWRAP
node_modules/.pnpm/@[email protected]/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/socket.js:107    - const tlsSocket = tls.connect(socketOpts);
node_modules/.pnpm/@[email protected]/node_modules/@google-cloud/cloud-sql-connector/dist/mjs/connector.js:148 - const tlsSocket = getSocket({
node_modules/.pnpm/[email protected]/node_modules/mysql2/lib/base/connection.js:64                                                       - this.stream = opts.config.stream(opts);
node_modules/.pnpm/[email protected]/node_modules/mysql2/lib/base/pool_connection.js:7                                                   - super(options);
node_modules/.pnpm/[email protected]/node_modules/mysql2/lib/pool_connection.js:5                                                        - class PoolConnection extends BasePoolConnection {

...

# FILEHANDLE
(unknown stack trace)

# TTYWRAP
temp.direct.mjs:40 - console.log('shutting down while queries are in flight...');

# Timeout
temp.direct.mjs:46 - await new Promise((r) => setTimeout(r, 500));
temp.direct.mjs:46 - await new Promise((r) => setTimeout(r, 500));

How to reproduce

  1. npm init -y && npm pkg set type=module
  2. npm i @google-cloud/[email protected] [email protected] why-is-node-running
  3. Save the example code above as repro.mjs, filling in INSTANCE_CONNECTION_NAME, DATABASE, USER.
  4. Ensure gcloud auth application-default login has been run and the IAM user has access to the instance.
  5. node repro.mjs
  6. Observe whyIsNodeRunning() reports N TLSWRAP handles rooted at @google-cloud/cloud-sql-connector/dist/mjs/socket.js — one per in-flight query that hadn't completed when pool.end() was called.

Environment details

  • OS: MacOS 15.7.4 ARM (M2 Pro)
  • Node.js version: v20.20.0 and v24.13.0
  • pnpm version: 10.22.0
  • @google-cloud/cloud-sql-connector version: 1.10.0
  • mysql2 version: 3.16.3

Steps to reproduce

  1. Create a Connector and a mysql2 pool per the package README.
  2. Issue several concurrent queries without awaiting them.
  3. Before the queries resolve, call await pool.end() and connector.close().
  4. Wait long enough (≥ 500 ms) for libuv to collect any handles that were legitimately cleared.
  5. Call whyIsNodeRunning() (or otherwise inspect the event loop).
  6. Expected: no TLSWRAP handles from the connector. Actual: one leaked TLSWRAP per in-flight query at shutdown time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions