10000 Discussion of Python 3.6 support · Issue #2415 · micropython/micropython · GitHub
[go: up one dir, main page]

Skip to content

Discussion of Python 3.6 support #2415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
4 of 53 tasks
dpgeorge opened this issue Sep 13, 2016 · 29 comments
Open
4 of 53 tasks

Discussion of Python 3.6 support #2415

dpgeorge opened this issue Sep 13, 2016 · 29 comments
Labels
rfc Request for Comment

Comments

@dpgeorge
Copy link
Member
dpgeorge commented Sep 13, 2016

Python 3.6 beta 1 was released on 12 Sep 2016, and a summary of the new features can be found here: https://docs.python.org/3.6/whatsnew/3.6.html . Being in beta stage means we can provide feedback on any of the new features/changes if we like.

(See #1329 for Python 3.5 support, which is still pending.)

New syntax features:

New built-in features:

  • PEP 468 - Preserving the order of **kwargs in a function
  • PEP 487 - Simpler customization of class creation
  • PEP 520 - Preserving Class Attribute Definition Order

Standard library changes:

  • PEP 495 - Local Time Disambiguation
  • PEP 506 - Adding A Secrets Module To The Standard Library [not relevant to uPy core]
  • PEP 519 - Adding a file system path protocol

CPython internals (not exposed at the Python level so not relevant to uPy):

  • PEP 509 - Add a private version to dict
  • PEP 523 - Adding a frame evaluation API to CPython

Linux/Windows changes:

  • PEP 524 - Make os.urandom() blocking on Linux (during system startup)
  • PEP 528 - Change Windows console encoding to UTF-8 (provisional)
  • PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional)

Other Language Changes

  • A global or nonlocal statement must now textually appear before the first use of the affected name in the same scope. Previously this was a SyntaxWarning.
  • It is now possible to set a special method to None to indicate that the corresponding operation is not available. For example, if a class sets __iter__() to None, the class is not iterable.
  • Long sequences of repeated traceback lines are now abbreviated as "[Pre 8000 vious line repeated {count} more times]"
  • Import now raises the new exception ModuleNotFoundError when it cannot find a module. Code that currently checks for ImportError (in try-except) will still work.
  • Class methods relying on zero-argument super() will now work correctly when called from metaclass methods during class creation.

Changes to MicroPython built-in modules

  • array - Exhausted iterators of array.array will now stay exhausted even if the iterated array is extended.
  • asyncio (many, may need another ticket)
  • binascii - The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline character is appended to the return value.
  • cmath
    • The new cmath.tau (τ) constant has been added.
    • New constants: cmath.inf and cmath.nan to match math.inf and math.nan, and also cmath.infj and cmath.nanj to match the format used by complex repr.
  • collections
    • The new Collection abstract base class has been added to represent sized iterable container classes.
    • The new Reversible abstract base class represents iterable classes that also provide the __reversed__() method.
    • The new AsyncGenerator abstract base class represents asynchronous generators
    • The namedtuple() function now accepts an optional keyword argument module, which, when specified, is used for the __module__ attribute of the returned named tuple class.
    • The verbose and rename arguments for namedtuple() are now keyword-only.
    • Recursive collections.deque instances can now be pickled.
  • hashlib
    • hashlib supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2.
    • BLAKE2 hash functions were added to the module. blake2b() and blake2s() are always available and support the full feature set of BLAKE2.
    • The SHA-3 hash functions sha3_224(), sha3_256(), sha3_384(), sha3_512(), and SHAKE hash functions shake_128() and shake_256() were added.
    • The password-based key derivation function scrypt() is now available with OpenSSL 1.1.0 and newer.
  • json - json.load() and json.loads() now support binary input. Encoded JSON should be represented using either UTF-8, UTF-16, or UTF-32.
  • math - The tau (τ) constant has been added to the math and cmath modules.
  • os
    • See the summary of PEP 519 for details on how the os and os.path modules now support path-like objects.
    • A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol.
    • On Linux, os.urandom() now blocks until the system urandom entropy pool is initialized to increase the security.
    • The Linux getrandom() syscall (get random bytes) is now exposed as the new os.getrandom() function.
  • re
    • Added support of modifier spans in regular expressions. Examples: '(?i:p)ython' matches 'python' and 'Python', but not 'PYTHON'; '(?i)g(?-i:v)r' matches 'GvR' and 'gvr', but not 'GVR'.
    • Match object groups can be accessed by __getitem__, which is equivalent to group(). So mo['name'] is now equivalent to mo.group('name').
    • Match objects now support index-like objects as group indices.
  • socket
    • The ioctl() function now supports the SIO_LOOPBACK_FAST_PATH control code.
    • The getsockopt() constants SO_DOMAIN, SO_PROTOCOL, SO_PEERSEC, and SO_PASSSEC are now supported.
    • The setsockopt() now supports the setsockopt(level, optname, None, optlen: int) form.
    • The socket module now supports the address family AF_ALG to interface with Linux Kernel crypto API. ALG_*, SOL_ALG and sendmsg_afalg() were added.
    • New Linux constants TCP_USER_TIMEOUT and TCP_CONGESTION were added.
  • ssl
    • ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2.
    • 3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added.
    • SSLContext has better default configuration for options and ciphers.
    • SSL session can be copied from one client-side connection to another with the new SSLSession class. TLS session resumption can speed up the initial handshake, reduce latency and improve performance.
    • The new get_ciphers() method can be used to get a list of enabled ciphers in order of cipher priority.
    • All constants and flags have been converted to IntEnum and IntFlags.
    • Server and client-side specific TLS protocols for SSLContext were added.
    • Added SSLContext.post_handshake_auth to enable and ssl.SSLSocket.verify_client_post_handshake() to initiate TLS 1.3 post-handshake authentication.
  • struct - now supports IEEE 754 half-precision floats via the 'e' format specifier.
  • sys - The new getfilesystemencodeerrors() function returns the name of the error mode used to convert between Unicode filenames and bytes filenames.
  • zlib - The compress() and decompress() functions now accept keyword arguments.

(Changes to non-built-in modules will need to be documented elsewhere.)

@dpgeorge dpgeorge added the rfc Request for Comment label Sep 13, 2016
@dpgeorge
Copy link
Member Author

PEP 468, preserving the order of kwargs, is of particular interest to us because it might be hard to implement efficiently. CPython 3.6 will have dict == OrderedDict (ie all dicts are ordered), using the PyPy implementation to make it efficient. Essentially all dicts have 2 tables, one which is the ordered array of (key, value) pairs (ordered by order of insertion), and the other is a (compact) hash table that indexes the ordered array.

@pfalcon
Copy link
Contributor
pfalcon commented Sep 13, 2016

I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented. Easy and helpful. The rest, I'm more and more getting disappointed where CPython goes.

8000

@dpgeorge
Copy link
Member Author

I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented

+10. It would be pretty easy to add, and I agree really nice to then write something like machine.freq(42_000_000).

@dhylands
Copy link
Contributor
dhylands commented Sep 13, 2016

I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented

+10. It would be pretty easy to add, and I agree really nice to then write something like machine.freq(42_000_000).

+1 here too. I've been using that while coding in rust and while a bit odd at first, it makes reading the larger integers easier.

@dbc
Copy link
Contributor
dbc commented Sep 16, 2016

I'd love to see "PEP 515 - Underscores in Numeric Literals" implemented. Easy and helpful.

Oh yes, oh yes. This is one my favourite language features in any language. I love it for hex and binary literals. Interestingly, at PyCon 2016 in Portland a few months ago, when the BDFL was reading over the list of PEP's, I cheered and applauded loudly for this one.... and about 2 thousand other people in the same room with me were dead silent. :/ So I'm thinking us embedded programmer types have different priorities.

As to the rest, I'm thinking that preserving order in kwargs is hardly worth the expense of memory in Micropython, for as often as it is used. OTOH, it may be a forced march if it becomes important for code portability. OTOOH, since we have 'micro' versions of the important libraries anyway, perhaps porting is a given in any case.

PEP530 just tickles me. Not sure it's of great value to Micropython, but I love it.

(Yes, I'm back! Been away from Micropython since about last December. Sometimes, life happens, and good stuff falls below the cut-off line on the to-do list. Hopefully, Micropython can crawl back above the line for a while.)

@peterhinch
Copy link
Contributor

@dbc Welcome back! I don't follow your point about PEP530 though - improved async support is surely welcome. Many embedded applications rely on asynchronous code.

@dbc
Copy link
Contributor
dbc commented Sep 16, 2016

@peterhinch re: PEP530 -- I should have been more clear about my fear of RAM bloat. Totally agree that asynch is great for embedded, I'm a big fan. I'd use async comprehensions if they were available, certainly -- like a cat on catnip. And of course we want Micropython to be as close to 100% source compatible with CPython as is practical. My fear is the "practical" part in Micropython -- if asynch comprehensions cause the SRAM footprint to grow significantly for programs using non-async list comprehensions or non-async generator expressions, then I'd be wary of it.

Thinking about it... the semantics of an async comprehension are satisfied by just doing a normal comprehension. The completion order is arbitrary, after all. So maybe the syntax can be accepted but the functionality could be a normal comprehension.

@peterhinch
Copy link
Contributor

@dbc Will the availability of the async keyword affect the bytecode emitted when the keyword is absent? This surely depends on the cleverness of the compiler. But I'll leave this to people who actually understand compilers :)

@pohmelie
Copy link
Contributor

PEP 498 - Literal String Formatting. It's step forward and "replace both existing % and str.format" looks like best solution for me. Definitely I want it in µpy.
PEP 525 - Asynchronous Generators (provisional). This is just awesome, since this can replace pretty ugly async for context class. It is just same as generator and __enter__/__iter__ methods of object. But, this requires too much changes, since this can't be done just as sugar for async/await in µpy right now. And probably no one need this in µpy.

@dbc
Copy link
Contributor
dbc commented Sep 20, 2016

PEP 498: I read that in detail now. So.. PEP 498 looks like the future. I never use %-formatting in new code. When PEP 498 strings happen in CPython, I will probably go exclusively to PEP498 format strings for new code and never look back, they are much more readable.

@dpgeorge says " likely too much bloat "

Ignoring PEP 498 means sacrificing CPython compatibility in a very visible way. Removing .format() in favor of literal string formatting is a very visible breaking change. Both of those options hit every user every day.

My naive assumption would be that much of the low-level formatting code could be factored out of .format() and literal string formatting. Pre-processing the PEP 498 strings would add to the compiler, certainly. Or is it RAM bloat and not code bloat that is the concern? An f-string becomes a byte-code expression in RAM instead of a simple string. Is this the concern?

I guess if you twisted my arm, I'd choose dropping %-formatting and .format() in favor of f-strings if I had to choose between f-strings and .format().

@pohmelie
Copy link
Contributor
pohmelie commented Sep 20, 2016

@dbc, one thing is pretty annoying. If you have only f-strings and got template from somewhere as string you should do something like

>>> s = "{x}"
>>> x = 1
>>> eval(f"f'{s}'")
'1'

@deshipu
Copy link
Contributor
deshipu commented Sep 20, 2016

You can still do:

>>> s = "{x}"
>>> x = 1
>>> s.format(x=x)

@pohmelie
Copy link
Contributor

@deshipu, @dbc and I talking about «replace both existing % and str.format», so there is not str.format/format/%

@deshipu
Copy link
Contributor
deshipu commented Sep 20, 2016

I see, I didn't catch that. In any case, they are not getting removed, so more power to you.

Argh, and now I re-read the whole conversation, and see how stupid my comment is, sorry.

@pohmelie
Copy link
Contributor

@deshipu, as @dpgeorge said in first post this is possible in theory.

@dbc
Copy link
Contributor
dbc commented Sep 20, 2016

If you have only f-strings and got template from somewhere as string you should do something like ...

Yes, that is somewhat klunky.

@peterhinch
Copy link
Contributor

@pohmelie @dbc said

Removing .format() in favor of literal string formatting is a very visible breaking change. Both of those options hit every user every day.

I agree that removing .format() would be very bad news. Likewise the idea of removing the string modulus operator. In my view such changes would break far too much code and would be a worse departure from cPython compatibility than opting not to support f-strings.

@ARF1
Copy link
ARF1 commented Mar 2, 2017

+1 For PEP 487 - Simpler customization of class creation

This is deceptively humbly named. It is a MUCH saner and light-weight replacement for metaclasses in many cases. Also, it should be fairly simple to implement given that the machinery will be fairly similar to __new__.

@s-celles
Copy link

f-string is something I'd love to see in MicroPython
PEP 498 -- Literal String Interpolation

@kamikaze
Copy link
Contributor
kamikaze commented Dec 8, 2018

Kill the .format() , voting for f-strings ! broken code is an abandoned code in 99% cases. I don't see any reason to not update the code for this change.

@mattytrentini
Copy link
Contributor

I believe PEP-15 is implemented in at least 1.10. I think we can strike it through!

@dpgeorge
Copy link
Member Author
dpgeorge commented Apr 2, 2019

PEP 515 (underscores in numeric literals) was implemented in 6a445b6

@asottile
Copy link
asottile commented May 2, 2019

for those looking for another way to get f-strings, I've added better support to future-fstrings for use as a preprocessor: https://github.com/asottile/future-fstrings#transform-source-for-micropython

It has a PEP-498 compliant source transformer that produces equivalent '...'.format(...) calls

@pmp-p
Copy link
Contributor
pmp-p commented Sep 20, 2019

thank you @asottile but sadly it seems, we won't benefit of bpo-36817 since asottile-archive/future-fstrings#40 was closed.

Where is the "Discussion of Python 3 support" status ? i may have missed it. I hardly see any 3.x future support with notimpl on major language features ( including some that exist since 2.x )

i only found 3.5 #1329

@asottile
Copy link

that issue is still doable, I just don't really feel motivated to work on it -- if someone wants to pick it up then by all means do so :)

@jimmo
Copy link
Member
jimmo commented Sep 20, 2019

For f-strings, did you see #4998 referenced above?

@pmp-p
Copy link
Contributor
pmp-p commented Sep 22, 2019

@jimmo no i'm not searching for 3.6/3.7/3.8 totallly optionnal features from #4998

Apart from the warm thanks addressed to the person deserving them for adaptation ( after all he did not have to take the time to add that ) of his usefull tool for use with micropython in replacement of clumsy https://github.com/pmp-p/fstrings_helper my question was :

Where is the "Discussion of Python 3 support" status ? i may have missed it. I hardly see any 3.x future support with notimpl on major language features ( including some that exist since 2.x )

i only found 3.5 #1329, i'm more interested for discussion upon completion of 3.4 and reaching stability.

@kevinkk525
Copy link
Contributor

Any chance to implement PEP 526 (Syntax for Variable Annotations (provisional))?

I understand that it doesn't have any impact beyond the parser but would be hugely beneficial in development (at least to me it would be). I guess if it is too much work then using strip-hints is an option. Running it on each script before uploading/freezing would be an acceptable workaround if it works reliable.

@dpgeorge
Copy link
Member Author

Any chance to implement PEP 526 (Syntax for Variable Annotations (provisional))?

Yes I would like to see this implemented soon. It would then allow more sophisticated type hinting, and can also get use in viper annotations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rfc Request for Comment
Projects
None yet
Development

No branches or pull requests

0