Register IPython's built-in magics lazily - #15340
Open
Carreau wants to merge 1 commit into
Open
Conversation
Starting a shell imported all fifteen modules under IPython.core.magics and instantiated every Magics class in them, even though a session typically uses a handful of magics at most. Most of that cost is not the class bodies but what they drag in: urllib and ast for %edit, cProfile, pstats and timeit for %timeit, plus the eighteen @magic_arguments() decorations that each build an argparse parser and format its help at import time. Only the magic *names* are now known up front, from hand-maintained tables in IPython.core.magics._table, and the module implementing a magic is imported the first time that magic is looked up. Registering a name installs a LazyMagic placeholder: listing magics, completing them and membership tests only read the table's keys and stay cheap, while calling a magic -- or touching any attribute of one -- imports the module, registers the real Magics instance in its place and delegates. The same mechanism is available to third parties through MagicsManager.register_lazy_class(), which, unlike the existing lazy_magics config, does not require packaging the magics as an extension. This takes roughly 30 ms off `import IPython` and 20 ms off import-plus-shell-startup. Keeping the tables honest is tests/test_magic_table.py: it imports every magics module, instantiates every Magics subclass, and fails -- printing the corrected table -- if anything is missing, stale or attributed to the wrong class. It also checks that starting a shell still imports no magics module at all. Places that need the whole picture rather than one magic keep working: %config and its completions load everything first, so the list of configurable classes is unchanged, and registry["ExecutionMagics"]-style lookups load the class on the miss. init_magics no longer runs %colors purely to trigger its observer, since that alone would import the basic magics on every startup. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BqGDoRyv11kHsBvvFQD4T4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starting a shell imported all fifteen modules under IPython.core.magics
and instantiated every Magics class in them, even though a session
typically uses a handful of magics at most. Most of that cost is not the
class bodies but what they drag in: urllib and ast for %edit, cProfile,
pstats and timeit for %timeit, plus the eighteen @magic_arguments()
decorations that each build an argparse parser and format its help at
import time.
Only the magic names are now known up front, from hand-maintained
tables in IPython.core.magics._table, and the module implementing a magic
is imported the first time that magic is looked up. Registering a name
installs a LazyMagic placeholder: listing magics, completing them and
membership tests only read the table's keys and stay cheap, while
calling a magic -- or touching any attribute of one -- imports the
module, registers the real Magics instance in its place and delegates.
The same mechanism is available to third parties through
MagicsManager.register_lazy_class(), which, unlike the existing
lazy_magics config, does not require packaging the magics as an
extension.
This takes roughly 30 ms off
import IPythonand 20 ms offimport-plus-shell-startup.
Keeping the tables honest is tests/test_magic_table.py: it imports every
magics module, instantiates every Magics subclass, and fails -- printing
the corrected table -- if anything is missing, stale or attributed to
the wrong class. It also checks that starting a shell still imports no
magics module at all.
Places that need the whole picture rather than one magic keep working:
%config and its completions load everything first, so the list of
configurable classes is unchanged, and registry["ExecutionMagics"]-style
lookups load the class on the miss. init_magics no longer runs %colors
purely to trigger its observer, since that alone would import the basic
magics on every startup.
Co-Authored-By: Claude Opus 5 [email protected]
Claude-Session: https://claude.ai/code/session_01BqGDoRyv11kHsBvvFQD4T4