Skip to content

FEAT: Implementing and Integrating AQE API's - execute, executemany, fetch, fetchall and fetchmany - #792

Draft
Subrata (subrata-ms) wants to merge 8 commits into
mainfrom
subrata-ms/AQECursor
Draft

Subrata (subrata-ms) wants to merge 8 commits into
mainfrom
subrata-ms/AQECursor

Conversation

@subrata-ms

@subrata-ms Subrata (subrata-ms) commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Work Item / Issue Reference

AB#47195,47195,47997,47198

GitHub Issue: #<ISSUE_NUMBER>


Summary

This pull request refactors the asynchronous query layer to improve clarity, error handling, and code organization. The main changes include renaming internal variables for clarity, extracting statement execution and result fetching logic into dedicated modules, enhancing exception translation, and updating tests to reflect improved error handling. These updates make the async API more robust and maintainable.

Refactoring and Code Organization:

  • Renamed internal attributes from native_connection/native_cursor to py_core_async_connection/py_core_async_cursor in both AsyncConnection and AsyncCursor for improved clarity and future maintainability. (mssql_python/async_query/async_connection.py, mssql_python/async_query/async_cursor.py) [1] [2]
  • Extracted statement execution and result fetching logic from AsyncCursor into new helper modules: async_execute.py and async_fetch.py, leading to cleaner, more modular code. (mssql_python/async_query/async_execute.py, mssql_python/async_query/async_fetch.py, mssql_python/async_query/async_cursor.py) [1] [2] [3]

Error Handling Improvements:

  • Enhanced exception translation in exception_translator.py to classify and translate lower-level driver errors into appropriate public exceptions (e.g., OperationalError, ProgrammingError), and updated the async connection tests to verify these translations. (mssql_python/async_query/exception_translator.py, tests/AsyncTest/test_002_async_connection.py) [1] [2] [3] [4]

Behavioral and API Improvements:

  • Improved fetch tracking in AsyncCursor, including accurate rowcount reporting after fetch operations and resetting fetch state on nextset. (mssql_python/async_query/async_cursor.py) [1] [2]
  • Updated column name handling in AsyncCursor.description to support automatic lowercasing based on settings, and ensured that row wrapping consistently produces Row objects with proper metadata. (mssql_python/async_query/async_cursor.py, mssql_python/async_query/async_fetch.py) [1] [2]

Testing:

  • Updated and expanded tests to cover the improved exception translation and internal API changes, ensuring correct error propagation and behavior after connection closure. (tests/AsyncTest/test_002_async_connection.py, tests/AsyncTest/test_003_async_exceptions.py) [1] [2] [3]

Copilot AI lite review requested due to automatic review settings September 17, 2026 11:26
@github-actions github-actions Bot added pr-size: large Substantial code update labels Sep 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved moderate and critical findings remain in result-set state, parameter binding, fetch handling, and credential-redaction coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Refactors the async query API by separating execution and fetching helpers, improving error translation and row handling, and expanding test coverage.

Changes:

  • Extracts async execution and fetch operations into dedicated modules.
  • Improves exception translation, logging, row wrapping, and fetch tracking.
  • Expands tests for execution, fetching, errors, cursor behavior, and logging.
File summaries
File Reviewed changes Final review findings
tests/AsyncTest/test_007_async_fetch.py Adds fetch, row, type, and navigation coverage.
tests/AsyncTest/test_006_async_execute.py Adds execute and executemany coverage.
tests/AsyncTest/test_005_async_cursor.py Updates cursor property and lifecycle tests.
tests/AsyncTest/test_004_async_logging.py Adds operation logging coverage. Critical (2 votes): Preserve broad credential-redaction assertions or verify known secrets are absent.
tests/AsyncTest/test_003_async_exceptions.py Expands exception translation coverage.
tests/AsyncTest/test_002_async_connection.py Updates connection error and lifecycle tests.
mssql_python/async_query/exception_translator.py Classifies and translates async errors.
mssql_python/async_query/async_fetch.py Implements fetching, row wrapping, and metadata handling. Moderate (1 vote): Snapshot native_uuid; short-circuit non-positive fetch sizes. Nit (3 votes): Cache row metadata maps instead of rebuilding them per row.
mssql_python/async_query/async_execute.py Implements async execution helpers. Moderate (2 votes): Normalize Row parameter sequences like the synchronous path.
mssql_python/async_query/async_cursor.py Delegates operations and tracks fetch state. Moderate (2 votes): Snapshot lowercase/UUID settings and row metadata per result set.
mssql_python/async_query/async_connection.py Renames connection internals and integrates logging.
Review details

Suppressed comments (2)

mssql_python/async_query/async_fetch.py:62

  • The wrapper passes a non-positive size through to py-core. The synchronous cursor short-circuits size <= 0 before its native call (mssql_python/cursor.py:2842-2843), and this PR's test requires fetchmany(-1) to return []; a Rust binding using an unsigned size can reject -1 instead. Short-circuit requested_size <= 0 before calling py-core.
    with translate_py_core_exceptions():
        if size is None:
            rows = await _get_py_core_async_cursor(cursor).fetchmany()
        else:
            rows = await _get_py_core_async_cursor(cursor).fetchmany(size)

mssql_python/async_query/async_fetch.py:30

  • This consults the mutable global native_uuid at fetch time, so changing it between execute() and fetch*() changes the representation of an already-executed result. The synchronous cursor snapshots UUID conversion indices at execute (mssql_python/cursor.py:1404-1423) and tests cover this (tests/test_004_cursor.py:16785-16815); store the setting/result-set metadata when execute or nextset completes.
    uuid_str_indices = (
        tuple(index for index, column in enumerate(description) if column[1] is uuid.UUID)
        if not get_settings().native_uuid
        else None
  • Files reviewed: 11/11 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/AsyncTest/test_004_async_logging.py
Comment on lines +105 to +107
lowercase = get_settings().lowercase
return [
((column[0].lower() if lowercase else column[0]), *column[1:]) for column in description
Comment on lines +26 to +27
if len(parameters) == 1 and isinstance(parameters[0], (tuple, list)):
parameters = tuple(parameters[0])
Comment on lines +19 to +23
def _wrap_row(cursor: "AsyncCursor", values: tuple[Any, ...]) -> Row:
description = cursor.description or ()
column_map = {column[0]: index for index, column in enumerate(description)}
column_map_lower = (
{name.lower(): index for name, index in column_map.items()}
Copilot AI review requested due to automatic review settings September 17, 2026 11:32
@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

100%


🎯 Overall Coverage

83%


📈 Total Lines Covered: 8534 out of 10188
📁 Project: mssql-python


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

  • mssql_python/async_query/async_connection.py (100%)
  • mssql_python/async_query/async_cursor.py (100%)
  • mssql_python/async_query/async_execute.py (100%)
  • mssql_python/async_query/async_fetch.py (100%)
  • mssql_python/async_query/exception_translator.py (100%)

Summary

  • Total: 141 lines
  • Missing: 0 lines
  • Coverage: 100%

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)
mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 61.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.row.py: 77.6%
mssql_python.pybind.ddbc_bindings.cpp: 77.7%
mssql_python.pybind.connection.connection_pool.cpp: 81.8%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.py_type_cache.hpp: 91.6%

🔗 Quick Links

⚙️ Build Summary 📋 Coverage Details

View Azure DevOps Build

Browse Full Coverage Report

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved compatibility, correctness, and performance findings remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (4)

mssql_python/async_query/async_cursor.py:107

  • This property applies the global lowercase setting on every access, so changing it after execute() changes the metadata and row key casing for an already-established result set. The synchronous cursor snapshots this at execution (tests/test_004_cursor.py:3592-3608); retain the effective casing with the async result metadata instead.
        if description is None:
            return None
        lowercase = get_settings().lowercase
        return [
            ((column[0].lower() if lowercase else column[0]), *column[1:]) for column in description

mssql_python/async_query/async_execute.py:27

  • The new async path only unwraps tuple/list parameters, but the added test_execute_accepts_dbapi_row passes a Row and expects its columns to bind as individual parameters. The synchronous implementation explicitly normalizes Row to a tuple because the downstream binder otherwise treats the whole row as one value (mssql_python/cursor.py:1739-1747); apply the same normalization before forwarding to PyAsyncCursor.execute, or that new async contract will fail.
    cursor._reset_fetch_tracking()  # pyright: ignore[reportPrivateUsage]
    if len(parameters) == 1 and isinstance(parameters[0], (tuple, list)):
        parameters = tuple(parameters[0])

mssql_python/async_query/async_fetch.py:30

  • native_uuid is read while each row is wrapped, so toggling the module setting after execute() but before fetch*() changes the type of rows from one result set. The synchronous cursor snapshots this setting at execution (tests/test_004_cursor.py:16336-16357); cache the UUID conversion indices when async result metadata is established.
    uuid_str_indices = (
        tuple(index for index, column in enumerate(description) if column[1] is uuid.UUID)
        if not get_settings().native_uuid
        else None

mssql_python/async_query/async_fetch.py:23

  • _wrap_row rebuilds the column map and, when enabled, the lowercase map for every returned row. fetchmany() and fetchall() therefore repeat O(column_count) metadata work for every row; the synchronous cursor precomputes these maps once per result set, so cache and reuse them for large async result sets.
def _wrap_row(cursor: "AsyncCursor", values: tuple[Any, ...]) -> Row:
    description = cursor.description or ()
    column_map = {column[0]: index for index, column in enumerate(description)}
    column_map_lower = (
        {name.lower(): index for name, index in column_map.items()}
  • Files reviewed: 11/11 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment on lines +61 to +65
seq_of_parameters: Sequence[Sequence[Any]] | Sequence[Mapping[str, Any]],
) -> None:
await async_execute.executemany(
self,
operation,
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Copilot AI review requested due to automatic review settings September 17, 2026 11:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical test indentation error blocks collection, and fetch behavior, performance, and documentation require fixes.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

mssql_python/async_query/async_fetch.py:24

  • _wrap_row is called for every row by fetchone, fetchmany, and fetchall, but this rebuilds the description-derived maps for every row, including a fresh cursor.description list and dictionaries. That makes large result sets pay O(rows × columns) Python metadata work and can make one result set change representation if lowercase/native_uuid is changed between fetches; the synchronous cursor caches these maps/settings once per result set (mssql_python/cursor.py:1838-1848). Cache the row metadata when the result set is established and reuse it here.
    description = cursor.description or ()
    column_map = {column[0]: index for index, column in enumerate(description)}
    column_map_lower = (
        {name.lower(): index for name, index in column_map.items()}
        if get_settings().lowercase

mssql_python/async_query/async_fetch.py:60

  • Non-positive sizes are still passed to the native cursor. The synchronous cursor returns immediately for size <= 0 (mssql_python/cursor.py:2839-2843), and the new async test expects fetchmany(-1) to return []; passing -1 to a native usize-like parameter can instead raise before the wrapper's tracking guard runs.
    requested_size = cursor.arraysize if size is None else size
    logger.debug("AsyncCursor.fetchmany: starting; requested_size=%s", requested_size)
    with translate_py_core_exceptions():
        if size is None:
            rows = await _get_py_core_async_cursor(cursor).fetchmany()

mssql_python/async_query/exception_translator.py:98

  • The new fallback at this line translates allowlisted built-in RuntimeError and TypeError instances, but the surrounding docstrings still say non-py-core errors are returned unchanged and that the context manager translates only py-core failures. Please update that documented contract so callers are not misled about which native Python errors are wrapped.
    return _translate_known_builtin_error(error)
  • Files reviewed: 11/11 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread tests/AsyncTest/test_004_async_logging.py Outdated
Copilot AI review requested due to automatic review settings September 17, 2026 12:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Five moderate issues remain unresolved.

Review details

Suppressed comments (5)

Previously missed (1) — in code that hasn't changed since the last review.

mssql_python/async_query/async_fetch.py:62

  • The synchronous cursor returns [] immediately for size <= 0 (mssql_python/cursor.py:2839-2843), but this path passes zero and negative sizes into mssql_py_core. That makes the new fetchmany(0/-1) contract depend on backend validation and can raise instead of returning the expected empty list; short-circuit non-positive sizes before the native call.

mssql_python/async_query/async_cursor.py:87

  • After a fetch sets _fetch_rowcount, rowcount bypasses the native cursor on every later read. Because close() does not clear that cache, a fetched cursor still returns its old count after closing instead of following the native closed-cursor behavior used by the other properties and operations. Clear fetch tracking when close succeeds before logging completion.
            await self._py_core_async_cursor.close()

mssql_python/async_query/async_cursor.py:62

  • This removes the existing public use_prepare keyword from AsyncCursor.executemany; callers that used the previous signature now fail with TypeError, and the native helper no longer receives the flag. Retain and forward the keyword (as execute still does), or deliberately version/document this breaking API change.
        seq_of_parameters: Sequence[Sequence[Any]] | Sequence[Mapping[str, Any]],
    ) -> None:

mssql_python/async_query/async_execute.py:27

  • This unwraps only tuples/lists, so a Row passed as the sole parameter is forwarded to PyAsyncCursor.execute as one bound value. The new test_execute_accepts_dbapi_row expects the row's two values to bind to two ? markers; mirror the synchronous cursor's Row-to-tuple normalization (mssql_python/cursor.py:1742-1747) before calling the native API.
    if len(parameters) == 1 and isinstance(parameters[0], (tuple, list)):
        parameters = tuple(parameters[0])

mssql_python/async_query/async_fetch.py:30

  • _wrap_row runs once for every returned row, but each invocation rereads description, rebuilds both column maps, and rescans all columns for UUIDs. Large fetchall()/fetchmany() results therefore repeat O(column_count) metadata work and allocate a map per row; cache these result-set maps on AsyncCursor when execution/nextset() changes and reuse them here.
    description = cursor.description or ()
    column_map = {column[0]: index for index, column in enumerate(description)}
    column_map_lower = (
        {name.lower(): index for name, index in column_map.items()}
        if get_settings().lowercase
        else None
    )
    uuid_str_indices = (
        tuple(index for index, column in enumerate(description) if column[1] is uuid.UUID)
        if not get_settings().native_uuid
        else None
  • Files reviewed: 11/11 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: large Substantial code update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants