Updating rackerlabs master with upstream master changes #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
btorch
changed the title
If you have a pool with no free items, one greenthread blocked in pool.get(), and then you call pool.put(item), sometimes the put will block. This happens when the greenthread blocked in pool.get() has a pending timeout. The timeout's timer has fired, the call to throw() has been scheduled, but throw() has not actually run yet. In pool.put(), we see a waiting getter, so we do a blocking self.channel.put()... but when the getter runs, it unwinds its stack and does not take the item, leaving the caller of pool.put() blocked despite there being enough free space. This commit fixes that by (a) making LightQueue.put() and .get() work with 0-length queues, even with timeouts, and (b) checking for queue.Full in Pool.put() and handling it correctly. #495
An application that wants to accept UTF-8 header values on Python 2 can
do so fairly easily: since str is bytes, it can decode from UTF-8
directly and return appropriate errors if that decoding fails.
On Python 3, it gets a little more interesting. Since str is unicode,
the bytes-on-the-wire are decoded as Latin-1 before being put in the
WSGI environment, so the application must encode as Latin-1 then decode
as UTF-8. For the most part, this Just Works as the transformation
should be lossless.
Sometimes, however, headers get truncated on py3. This is the result
of a difference in how .strip() behaves for bytes vs unicode -- in
particular, there are some unicode characters that are considered
whitespace whose Latin-1 encoded byte string is not:
>>> [x for i in range(256) for x in (chr(i),)
... if x.isspace() and not x.encode('latin1').isspace()]
['\x1c', '\x1d', '\x1e', '\x1f', '\x85', '\xa0']
Looking at RFC 7230, it defines header fields as
a case-insensitive field name followed by a colon (":"), optional
leading whitespace, the field value, and optional trailing whitespace
where "whitespace" is limited to SP or HTAB. So, let's just strip *those*
from header values.
Note that (some versions of?) py2's mimetools.Message includes the
trailing CRLF, so strip that, too.
#504
Closes #488; for some additional context, see #468 and #467. PEP-0333 says [1] > all strings referred to in this specification **must** be of type ``str`` but muddied that message earlier with a bunch of talk about > all strings passed to or from the server must be standard Python byte > strings, not Unicode objects PEP-3333 sought to clarify [2] with > WSGI therefore defines two kinds of "string": > > * "Native" strings (which are always implemented using the type > named ``str``) that are used for request/response headers and > metadata > > * "Bytestrings" (which are implemented using the ``bytes`` type > in Python 3, and ``str`` elsewhere), that are used for the bodies > of requests and responses (e.g. POST/PUT input data and HTML page > outputs). > > Do not be confused however: even if Python's ``str`` type is actually > Unicode "under the hood", the *content* of native strings must > still be translatable to bytes via the Latin-1 encoding! And later, in "Unicode Issues" [3], it adds > For values referred to in this specification as "bytestrings" > (i.e., values read from ``wsgi.input``, passed to ``write()`` > or yielded by the application), the value **must** be of type > ``bytes`` under Python 3, and ``str`` in earlier versions of > Python. So the upshot seems to be - All request and response bodies must be bytes, regardless of Python version. - All headers and other WSGI environment keys need to be native strings; i.e. bytes strings on Python 2 and unicode strings on Python 3. [1] https://www.python.org/dev/peps/pep-0333/#unicode-issues [2] https://www.python.org/dev/peps/pep-3333/#a-note-on-string-types [3] https://www.python.org/dev/peps/pep-3333/#unicode-issues
… of installation and development
When performing DNS lookups, the source address from resolv.conf may have stray zeroes in it, or the address string returned from socket.recvfrom may. Purge them before comparing tuples. Resolves: rhbz#1607967 Signed-off-by: Lon Hohberger <[email protected]>
If the source address for a packet did not match where we sent, the udp() function would spin in an infinite loop and the timer would never expire, causing the process to hang. Signed-off-by: Lon Hohberger <[email protected]>
Tests: - normal operation - no reply (timeout) - unexpected source address (w/ timeout & ignore_unexpected set) - number of zeroes in ipv6 address string different - unexpected address Signed-off-by: Lon Hohberger <[email protected]>
…ts file getaddrinfo() behaves differently from the standard implementation as it will try to contact nameservers if only one (IPv4 or IPv6) entry is returned from /etc/hosts file. This patch avoids getaddrinfo() querying nameservers if at least one entry is fetched through the hosts file to match the behavior of the original socket.getaddrinfo() implementation. Closes #515 Signed-off-by: Daniel Alvarez <[email protected]>
Signed-off-by: Lon Hohberger <[email protected]>
Newer OpenSSL requires RSA key at least 2048 bits
Output compatibe with golang.org/x/tools/cmd/benchcmp which allows easy automated performance comparison. Usage: - git checkout -b newbranch - change code - bin/bench-compare -python ./venv-27 - bin/bench-compare -python ./venv-36 - copy benchmark results - git commit, include benchmark results
…ashirin Origin: #517 Benchmarks performed on **some old laptop** running many other programs, OSX (no way to set CPU affinity) so don't look at exact values, only comparison. Baseline: 96fccf3 CPython 2.7: Benchmark_hub_timers 5867319 5733979 -2.27% Benchmark_pool_spawn 20666 20319 -1.68% Benchmark_pool_spawn_n 12697 12022 -5.32% Benchmark_sleep 21982 20385 -7.27% Benchmark_pool_spawn 20878 20025 -4.09% Benchmark_spawn 53915 52598 -2.44% Benchmark_spawn_link1 59215 56062 -5.32% Benchmark_spawn_link5 69550 67660 -2.72% Benchmark_spawn_link5_unlink3 72435 68624 -5.26% Benchmark_pool_spawn_n 12223 12058 -1.35% Benchmark_spawn_n 7394 7585 +2.58% Benchmark_spawn_n_kw 7798 7473 -4.17% Benchmark_spawn_nowait 11309 11510 +1.78% Again, benchmark environment is not clean, machine is used by many more processes. I invite seeing these numbers as "no significant change in performance" while explicit attribute initialisation is better than getattr/del games.
…ls even when it is unsupported on current platform Solution: eager import all built-in hubs, explicitly check support later #466
* Fix for Python 3.7 * Remove redundant piece of code. * Put back do_handshake_on_connect kwarg * Use Python 3.7 instead of 3.7-dev * Fix buildbot failing permissions with 3.7 * tests: env_tpool_zero assert details * setup: Python 3.7 classificator
* Issue #535: use Python 2 compatible syntax for keyword-only args. * Validate that encode_chunked is the *only* keyword argument passed.
* [bug] reimport submodule as well in patcher.inject * [dev] add unit-test * [dev] move unit test to isolated tests * improve unit test
The pyevent hub was removed one year ago, however the doc still contains a reference to it. This reference is empty and can let think that this hub is still present. Lets remove this reference.
… (#949) Co-authored-by: Hervé Beraud <[email protected]>
Improve the general index by adding a contribution guide, changelog, etc. Refactor sections to host various topics and to generate a more sexy table of context which will be more user friendly. Refactor formatting heading levels. Make Asyncio warning message reusable and centralized. Give more visibility to the Asyncio topic.
* drop header keys with underscores * use a dedicated formalize_key_naming function for header normalisation * adjust tests to comply with the new header security checks
* Upgrade RLocks as last thing we do * Fix style --------- Co-authored-by: Itamar Turner-Trauring <[email protected]>
* os.read() waits until file descriptor is ready before reading. * os.write() waits on file descriptor before attempting to write.
* shorten the lines * Document alternatives and methodology * adding a glossary * why abandonning eventlet This is the beginning of the implementation of the migration guide proposed here: https://review.opendev.org/c/openstack/governance/+/902585 The goal is to centralize all the materials we have to help users to safely migrate off of Eventlet.
…#985) Previously, if an application tried to read some set length from such a request (because, say, it always tried to read every request's body), the server could hang waiting on the client to send more bytes. The spec, however, is clear -- even going back to HTTP/1.0 -- if Content-Length (nor Transfer-Encoding, in HTTP/1.1) is missing, the request has no body, which is equivalent to having a body of length 0. See: - https://www.rfc-editor.org/rfc/rfc1945#section-7.2 - https://www.rfc-editor.org/rfc/rfc2616#section-4.3 - https://www.rfc-editor.org/rfc/rfc7230#section-3.3 - https://www.rfc-editor.org/rfc/rfc9112#section-6.3-2.7
… response with a status code of 1xx, 204 or (2xx to CONNECT request) (#747) * wsgi: server MUST NOT send Content-Length/Transfer-Encoding header in response with a status code of 1xx, 204 or (2xx to CONNECT request) https://httpwg.org/specs/rfc7230.html#rfc.section.3.3.1 #746 * wsgi: Stop requiring that bodyless responses close conn * wsgi: 304 and HEAD responses must also not have bodies * Update wsgi_test.py --------- Co-authored-by: Itamar Turner-Trauring <[email protected]> Co-authored-by: Tim Burke <[email protected]> Co-authored-by: Hervé Beraud <[email protected]>
* reorder python jobs by python versions * Supporting Python 3.13 * Fix YAML syntax * Explicit Python 3.13 version until it's out * fix typo * Use latest 3.13 prerelease * Python 3.13 support Emulate Python 3.13's start_joinable_thread API using greenthreads. We cut some corners, of course: * We aren't maintaining a table of green thread idents to threads, so we can't wait for all threads on shutdown. * Our _make_thread_handle() can only make a handle for the current thread (as we don't have a way to look up green threads by ident). * .join() on a non-GreenThread (e.g. the main thread) just returns immediately. Fixes: #964 * _tstate_lock was removed in Python 3.13 In python/cpython#114271, _tstate_lock was replaced with an event on PyThreadState. * Add Python 3.13 to tox --------- Co-authored-by: Hervé Beraud <[email protected]> Co-authored-by: Itamar Turner-Trauring <[email protected]>
* Test for OS threads using eventlet. * Support OS threads using eventlet too. * Explain why we don't import at module level. --------- Co-authored-by: Itamar Turner-Trauring <[email protected]>
Signed-off-by: tinyboxvk <[email protected]> Co-authored-by: Hervé Beraud <[email protected]>
Co-authored-by: Itamar Turner-Trauring <[email protected]>
) * Catch bad monkey patching. * Handle fork() * Make this test pass again on asyncio hub. * Pacify pep8 * Add new test file --------- Co-authored-by: Itamar Turner-Trauring <[email protected]>
…syncio hub (#1011) * Another test * Replace EVENTLET_MONKEYPATCH with a more modest and less dangerous alternative. * Change import order to work with asyncio hub * Load hub early in asyncio tests. * Correct syntax * Make sure hub is initialized before doing anything else. * Do affect subprocesses. * Revert another change * Fix lock upgrading on asyncio hub. * Failing fork() tests on macOS is not something we should try to fix. * Switch to a better mechanism for unmonkey-patching for asyncio. * More testing * Not necessary apparently. * Make sure selectors has the correct select * Pacify pep8 * Unused. --------- Co-authored-by: Itamar Turner-Trauring <[email protected]>
Related to #1007 Signed-off-by: Hervé Beraud <[email protected]>
) * Calling eventlet.sleep(0) isn't really blocking, so don't blow up. * Fix style * Drop 3.7, it's breaking CI. --------- Co-authored-by: Itamar Turner-Trauring <[email protected]>
* Drop support of Python 3.7 Python 3.7 is not supported since almost 1 year, Debian Buster LTS is ending on 30th June 2024, and github actions only support versions of Python which are not EOL [3]. For all these reasons, I think we can drop the support of Python 3.7. [1] https://devguide.python.org/versions/ [2] https://www.debian.org/releases/buster/ [3] https://github.com/actions/python-versions?tab=readme-ov-file#support-policy * Drop support of Python 3.7 Python 3.7 is not supported since almost 1 year, Debian Buster LTS is ending on 30th June 2024, and github actions only support versions of Python which are not EOL [3]. For all these reasons, I think we can drop the support of Python 3.7. [1] https://devguide.python.org/versions/ [2] https://www.debian.org/releases/buster/ [3] https://github.com/actions/python-versions?tab=readme-ov-file#support-policy * Remove duplicates. * 3.13 is supported. --------- Co-authored-by: Itamar Turner-Trauring <[email protected]>
* Spew: Correct line lookup from inspect.getsourcelines() I can't write a test for this, as I don't know how to trigger is part of Spew, without also falling into the OSError branch (which test_line_nofile tests). However, it has been observed to hit this code when running the tests under pytest-xdist (execnext), and the code is obviously wrong. This corrects it. * Don't expect inspect.getsourcelines() to fail under pytest-xdist
Inserting ourselves between protocol.__new__ and protocol.__init__ seems unnecessary, especially since we've replaced (and no longer call) stdlib's socketserver.BaseRequestHandler.__init__ Also removes a py2-ism: types.InstanceType is no longer a thing in py3. Co-authored-by: Hervé Beraud <[email protected]>
... because it is required only in Python <= 3.5 which has no overlap with the python versions currently supported. Co-authored-by: Hervé Beraud <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.