Infra: Add custom lexers for PEP 823 + 824 - #5125

@hugovk Would you mind taking a look at this change from an infra perspective? Is it fine to add the custom lexers within pep_sphinx_extensions/lexers or should they be moved somewhere else? As a followup it might be great to add the ASDL lexer from the CPython repo here as well. Just wasn't sure about the licensing which is why I didn't include it. https://github.com/python/cpython/blob/main/Doc/tools/extensions/lexers/asdl_lexer.py

Note: The custom lexers themselves don't attempt to be feature complete, they work well enough to enable syntax highlighting in the PEPs though.

Hi! I'll come back to this, but a quick note: might be some overlap with #5084? cc @ZeroIntensity

Yeah, the new keywords look like they can be covered by #5084.

I think it would also be a good idea to make this generic; rather than making a custom lexer for every new syntax PEP, we should add a way to customize code blocks if necessary (my PR lets you do .. code-block:: python+soft-keywords:my_keyword, for example).

While I do like a good general solution, I'm not sure it's the best option here. Syntax changes in itself are difficult to generalize. Sure there are different categories, like adding a new (soft-)keyword or operator, but how would you make sure it works in every case? And if it doesn't work and requires changes, you would either have to go back and check every PEP that nothing broke or add a small test framework as well.

Writing a custom lexer is fairly strait forward and only a few lines of code. This will be even easier once there are a few examples you just need to copy and modify. It's also important to keep in mind that they don't have to be perfect. It's enough if they work for the examples in the PEP and nothing more.

A benefit of #5084 is that it only applies to the PEPs you want it to.

It could be confusing if all the custom syntax started showing up in all PEPs.

A benefit of #5084 is that it only applies to the PEPs you want it to.

It could be confusing if all the custom syntax started showing up in all PEPs.

Unless I'm missing something, that applies to this PR as well. It also requires the author to explicitly request the lexer e.g. .. code-block:: py823, compared to .. code-block:: python+soft-keywords:my_keyword for #5084.

Sorry, you're right, it was me missing something :)

Also, PEP 823/4 don't add new keywords, they add new operators.

Could we perhaps allow PEPs to specify custom PEG grammar and then generate a parser for it at build time?

Could we perhaps allow PEPs to specify custom PEG grammar and then generate a parser for it at build time?

That might be possible, but is it worth the effort? Tbh I only considered custom lexers recently after I saw how easy it was to write one for the examples in the PEPs. I'd like to point out that they are by no means perfect und sure there are likely edge cases which aren't covered but that isn't the point.

They are a quick and dirty solution which work well enough to add some basic syntax highlighting, nothing more. It was so simple to write that I even include highlighting for rejected ideas (maybe for 823 and otherwise for 824).