Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
19b28fc
CVE-2020-10735: Prevent DoS by very large int()
tiran May 5, 2020
0a96b20
Default to disable, improve tests and docs
tiran Jan 19, 2022
88f6d5d
fix typo
tiran Jan 19, 2022
70c195e
More docs (WIP)
tiran Jan 19, 2022
e17e93b
Basic documentation for sys functions
tiran Jan 19, 2022
fbd14b7
Use ValueError, ignore underscore, scale limit
tiran Jan 20, 2022
dd74d70
Fix CI
tiran Jan 20, 2022
0e01461
Address Greg's review
tiran Aug 1, 2022
0b21e5f
Fix sys.flags len and docs
tiran Aug 1, 2022
3b38abe
Keep the warning, but remove advice about limiting input length in th…
gpshead Aug 2, 2022
37193ed
Renamed the APIs & too many other refactorings.
gpshead Aug 5, 2022
c90b79f
Improve the configuring docs.
gpshead Aug 7, 2022
fea25ea
Stop tying to base10, just use string digits.
gpshead Aug 7, 2022
ac9f22f
Remove the added now-unneeded helper log tbl fn.
gpshead Aug 7, 2022
da72dd1
prevent intdostimeit from emitting errors in test_tools.
gpshead Aug 7, 2022
d7e4d7b
Remove a leftover base 10 reference. clarify.
gpshead Aug 7, 2022
5c7e6d5
versionadded/changed to 3.12
gpshead Aug 7, 2022
61a5bc9
Link to the CVE from the main doc.
gpshead Aug 7, 2022
c15adde
Add a What's New entry.
gpshead Aug 7, 2022
76ae1c2
Add a Misc/NEWS.d entry.
gpshead Aug 7, 2022
1ad88f5
Undo addition to PyConfig to ease backporting.
gpshead Aug 8, 2022
0c83111
Remove the Tools/scripts/ example and timing code.
gpshead Aug 8, 2022
5d39ab6
un-add the <math.h> include (not needed for PR anymore)
gpshead Aug 8, 2022
5b77b3e
Remove added unused imports.
gpshead Aug 8, 2022
de00cdc
Tabs -> Spaces
gpshead Aug 8, 2022
3cc8553
make html and make doctest in Doc pass.
gpshead Aug 8, 2022
da97e65
Raise the default limit and the threshold.
gpshead Aug 10, 2022
ef03a16
Remove xmlrpc.client changes, test-only.
gpshead Aug 12, 2022
e916845
Rearrange the new stdtypes docs, w/limits + caution.
gpshead Aug 13, 2022
101502e
Make a huge int a SyntaxError with lineno when parsing.
gpshead Aug 16, 2022
fa8a58a
Mention the chosen default in the NEWS entry.
gpshead Aug 16, 2022
313ab6d
Properly clear & free the prior exception.
gpshead Aug 16, 2022
614cd02
Add a note to the float.as_integer_ratio() docs.
gpshead Aug 17, 2022
16ad090
Clarify the documentation wording and error msg.
gpshead Aug 17, 2022
4eb72e6
Fix test_idle, it used a long int on a line.
gpshead Aug 17, 2022
da36550
Rename the test.support context manager and document it.
gpshead Aug 19, 2022
f4372cc
Documentation cleanup.
gpshead Aug 19, 2022
c421853
Update attribution in Misc/NEWS.d
gpshead Aug 25, 2022
9f2168a
Regen global strings
tiran Sep 1, 2022
3c8504b
Make the doctest actually run & fix it.
gpshead Sep 1, 2022
1586419
Fix the docs build.
gpshead Sep 2, 2022
94bd3ee
Rename the news file to appease the Bedevere bot.
gpshead Sep 2, 2022
0b91f65
Regen argument clinic after the rebase merge.
gpshead Sep 2, 2022
02776f9
Hexi hexa
tiran Sep 2, 2022
173fa4e
Hexi hexa 2
tiran Sep 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Documentation cleanup.
  • Loading branch information
gpshead committed Sep 2, 2022
commit f4372cc5cd6055436c04ab8c13e5bd0f4975d3cd
48 changes: 24 additions & 24 deletions Doc/library/stdtypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5473,23 +5473,23 @@ Integer string conversion length limitation
===========================================

CPython has a global limit for converting between :class:`int` and :class:`str`
to mitigate denial of service attacks. This limit *only* applies to
non-power-of-two number bases such as decimal. Hexidecimal, octal, and binary
are not limited. The limit can be configured.
to mitigate denial of service attacks. This limit *only* applies to decimal or
other non-power-of-two number bases. Hexidecimal, octal, and binary conversions
Comment thread
gpshead marked this conversation as resolved.
Outdated
Comment thread
tiran marked this conversation as resolved.
Outdated
are unlimited. The limit can be configured.

The limit is necessary because CPython's integer type is an abitrary length
number (commonly known as a bignum) stored in binary form. There exists no
algorithm that can convert a string to a binary integer or a binary integer to
a string in linear time, unless the base is a power of 2. Even the best known
algorithms for base 10 have sub-quadratic complexity. Converting a large value
such as ``int('1' * 500_000)`` can take over a second on a fast CPU.
The :class:`int` type in CPython is an abitrary length number stored in binary
form (commonly known as a "bignum"). There exists no algorithm that can convert
a string to a binary integer or a binary integer to a string in linear time,
*unless* the base is a power of 2. Even the best known algorithms for base 10
have sub-quadratic complexity. Converting a large value such as ``int('1' *
500_000)`` can take over a second on a fast CPU.

Limiting conversion size offers a practical way to avoid `CVE-2020-10735
<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_.

The limit is applied to the number of digit characters in the input or output
string. That means that higher bases can process larger numbers before the
limit triggers. Underscores and the sign are not counted towards the limit.
string when a non-linear conversion algorithm would be involved. Underscores
and the sign are not counted towards the limit.

When an operation would exceed the limit, a :exc:`ValueError` is raised::

Expand All @@ -5509,12 +5509,12 @@ When an operation would exceed the limit, a :exc:`ValueError` is raised::
ValueError: Exceeds the limit (4300) for integer string conversion: value has 8599 digits.
>>> len(hex(i_squared))
7144
>>> assert int(hex(i_squared), base=16) == i
>>> assert int(hex(i_squared), base=16) == i # Hexidecimal is unlimited.

The default limit is 4300 digits as seen in
:data:`sys.int_info.default_max_str_digits <sys.int_info>`. The smallest limit
is 640 digits as seen in :data:`sys.int_info.str_digits_check_threshold
<sys.int_info>`.
The default limit is 4300 digits as provided in
:data:`sys.int_info.default_max_str_digits <sys.int_info>`.
The lowest limit that can be configured is 640 digits as provided in
:data:`sys.int_info.str_digits_check_threshold <sys.int_info>`.

Verification::

Expand All @@ -5532,14 +5532,14 @@ Affected APIs
-------------

The limition only applies to potentially slow conversions between :class:`int`
and :class:`str`:
and :class:`str` or :class:`bytes`:

* ``int(string)`` with default base 10.
* ``int(string, base)`` for all bases that are not a power of 2.
* ``str(integer)``.
* ``repr(integer)``
* any other string conversion to base 10, for example ``f"{integer}"``,
``"{}".format(integer)``, or ``"%d" % integer``.
``"{}".format(integer)``, or ``b"%d" % integer``.

The limitations do not apply to functions with a linear algorithm:

Expand All @@ -5557,7 +5557,7 @@ Before Python starts up you can use an environment variable or an interpreter
command line flag to configure the limit:

* :envvar:`PYTHONINTMAXSTRDIGITS`, e.g.
``PYTHONINTMAXSTRDIGITS=640 python3`` to set the limit to ``640`` or
``PYTHONINTMAXSTRDIGITS=640 python3`` to set the limit to 640 or
``PYTHONINTMAXSTRDIGITS=0 python3`` to disable the limitation.
* :option:`-X int_max_str_digits <-X>`, e.g.
``python3 -X int_max_str_digits=640``
Expand All @@ -5578,8 +5578,8 @@ Information about the default and minimum can be found in :attr:`sys.int_info`:

* :data:`sys.int_info.default_max_str_digits <sys.int_info>` is the compiled-in
default limit.
* :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` is the minimum
accepted value for the limit.
* :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` is the lowest
accepted value for the limit (other than 0 which disables it).

.. versionadded:: 3.12

Expand All @@ -5596,16 +5596,16 @@ Information about the default and minimum can be found in :attr:`sys.int_info`:

Test your application thoroughly if you use a low limit. Ensure your tests
run with the limit set early via the environment or flag so that it applies
during startup and even during any installation step that may precompile
source to ``.pyc`` files.
during startup and even during any installation step that may invoke Python
to precompile ``.py`` sources to ``.pyc`` files.

Recommended configuration
-------------------------

The default :data:`sys.int_info.default_max_str_digits` is expected to be
reasonable for most applications. If your application requires a different
limit, set it from your main entry point using Python version agnostic code as
these APIs were added in patch releases before 3.12.
these APIs were added in security patch releases in versions before 3.12.

Example::

Expand Down
51 changes: 26 additions & 25 deletions Doc/library/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -502,27 +502,27 @@ always available.
The :term:`named tuple` *flags* exposes the status of command line
flags. The attributes are read only.

================================== ======================================================================================================
attribute flag
================================== ======================================================================================================
:const:`debug` :option:`-d`
:const:`inspect` :option:`-i`
:const:`interactive` :option:`-i`
:const:`isolated` :option:`-I`
:const:`optimize` :option:`-O` or :option:`-OO`
:const:`dont_write_bytecode` :option:`-B`
:const:`no_user_site` :option:`-s`
:const:`no_site` :option:`-S`
:const:`ignore_environment` :option:`-E`
:const:`verbose` :option:`-v`
:const:`bytes_warning` :option:`-b`
:const:`quiet` :option:`-q`
:const:`hash_randomization` :option:`-R`
:const:`dev_mode` :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`)
:const:`utf8_mode` :option:`-X utf8 <-X>`
:const:`safe_path` :option:`-P`
:const:`int_max_str_digits` :option:`-X int_max_str_digits <-X>` (:ref:`integer string conversion length limitation <int_max_str_digits>`)
================================== ======================================================================================================
============================= ==============================================================================================================
attribute flag
============================= ==============================================================================================================
:const:`debug` :option:`-d`
:const:`inspect` :option:`-i`
:const:`interactive` :option:`-i`
:const:`isolated` :option:`-I`
:const:`optimize` :option:`-O` or :option:`-OO`
:const:`dont_write_bytecode` :option:`-B`
:const:`no_user_site` :option:`-s`
:const:`no_site` :option:`-S`
:const:`ignore_environment` :option:`-E`
:const:`verbose` :option:`-v`
:const:`bytes_warning` :option:`-b`
:const:`quiet` :option:`-q`
:const:`hash_randomization` :option:`-R`
:const:`dev_mode` :option:`-X dev <-X>` (:ref:`Python Development Mode <devmode>`)
:const:`utf8_mode` :option:`-X utf8 <-X>`
:const:`safe_path` :option:`-P`
:const:`int_max_str_digits` :option:`-X int_max_str_digits <-X>` (:ref:`integer string conversion length limitation <int_max_str_digits>`)
============================= ==============================================================================================================

.. versionchanged:: 3.2
Added ``quiet`` attribute for the new :option:`-q` flag.
Expand Down Expand Up @@ -729,8 +729,8 @@ always available.

.. function:: get_int_max_str_digits()

Return current global value for :ref:`integer string conversion length
limitation <int_max_str_digits>`. See also :func:`set_int_max_str_digits`
Returns the current value for the :ref:`integer string conversion length
limitation <int_max_str_digits>`. See also :func:`set_int_max_str_digits`.

.. versionadded:: 3.12

Expand Down Expand Up @@ -1333,8 +1333,9 @@ always available.

.. function:: set_int_max_str_digits(n)

Set global interpreter limit for :ref:`integer string conversion length
limitation <int_max_str_digits>`. See also :func:`get_int_max_str_digits`
Set the :ref:`integer string conversion length limitation
<int_max_str_digits>` used by this interpreter. See also
:func:`get_int_max_str_digits`.

.. versionadded:: 3.12

Expand Down