diff --git a/CHANGELOG.md b/CHANGELOG.md index 47ac88c8..18ded9d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## Unreleased +## Added + +* Support for `zstd` content decoding using the python `zstandard` package is added. Installable using `httpx[zstd]`. (#3139) + +### Fixed + +* Fix `app` type signature in `ASGITransport`. (#3109) + +## 0.27.0 (21st February, 2024) + +### Deprecated + +* The `app=...` shortcut has been deprecated. Use the explicit style of `transport=httpx.WSGITransport()` or `transport=httpx.ASGITransport()` instead. + ### Fixed * Respect the `http1` argument while configuring proxy transports. (#3023) @@ -88,7 +102,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). * The logging behaviour has been changed to be more in-line with other standard Python logging usages. We no longer have a custom `TRACE` log level, and we no longer use the `HTTPX_LOG_LEVEL` environment variable to auto-configure logging. We now have a significant amount of `DEBUG` logging available at the network level. Full documentation is available at https://www.python-httpx.org/logging/ (#2547, encode/httpcore#648) * The `Response.iter_lines()` method now matches the stdlib behaviour and does not include the newline characters. It also resolves a performance issue. (#2423) * Query parameter encoding switches from using + for spaces and %2F for forward slash, to instead using %20 for spaces and treating forward slash as a safe, unescaped character. This differs from `requests`, but is in line with browser behavior in Chrome, Safari, and Firefox. Both options are RFC valid. (#2543) -* NetRC authentication is no longer automatically handled, but is instead supported by an explicit `httpx.NetRCAuth()` authentication class. See the documentation at https://www.python-httpx.org/advanced/#netrc-support (#2525) +* NetRC authentication is no longer automatically handled, but is instead supported by an explicit `httpx.NetRCAuth()` authentication class. See the documentation at https://www.python-httpx.org/advanced/authentication/#netrc-authentication (#2525) ### Removed @@ -141,7 +155,7 @@ See the "Removed" section of these release notes for details. ### Changed * Drop support for Python 3.6. (#2097) -* Use `utf-8` as the default character set, instead of falling back to `charset-normalizer` for auto-detection. To enable automatic character set detection, see [the documentation](https://www.python-httpx.org/advanced/#character-set-encodings-and-auto-detection). (#2165) +* Use `utf-8` as the default character set, instead of falling back to `charset-normalizer` for auto-detection. To enable automatic character set detection, see [the documentation](https://www.python-httpx.org/advanced/text-encodings/#using-auto-detection). (#2165) ### Fixed @@ -160,7 +174,7 @@ See the "Removed" section of these release notes for details. ### Added -* Support for [the SOCKS5 proxy protocol](https://www.python-httpx.org/advanced/#socks) via [the `socksio` package](https://github.com/sethmlarson/socksio). (#2034) +* Support for [the SOCKS5 proxy protocol](https://www.python-httpx.org/advanced/proxies/#socks) via [the `socksio` package](https://github.com/sethmlarson/socksio). (#2034) * Support for custom headers in multipart/form-data requests (#1936) ### Fixed @@ -315,7 +329,7 @@ finally: The 0.18.x release series formalises our low-level Transport API, introducing the base classes `httpx.BaseTransport` and `httpx.AsyncBaseTransport`. -See the "[Writing custom transports](https://www.python-httpx.org/advanced/#writing-custom-transports)" documentation and the [`httpx.BaseTransport.handle_request()`](https://github.com/encode/httpx/blob/397aad98fdc8b7580a5fc3e88f1578b4302c6382/httpx/_transports/base.py#L77-L147) docstring for more complete details on implementing custom transports. +See the "[Custom transports](https://www.python-httpx.org/advanced/transports/#custom-transports)" documentation and the [`httpx.BaseTransport.handle_request()`](https://github.com/encode/httpx/blob/397aad98fdc8b7580a5fc3e88f1578b4302c6382/httpx/_transports/base.py#L77-L147) docstring for more complete details on implementing custom transports. Pull request #1522 includes a checklist of differences from the previous `httpcore` transport API, for developers implementing custom transports. @@ -632,7 +646,7 @@ This release switches to `httpcore` for all the internal networking, which means It also means we've had to remove our UDS support, since maintaining that would have meant having to push back our work towards a 1.0 release, which isn't a trade-off we wanted to make. -We also now have [a public "Transport API"](https://www.python-httpx.org/advanced/#custom-transports), which you can use to implement custom transport implementations against. This formalises and replaces our previously private "Dispatch API". +We also now have [a public "Transport API"](https://www.python-httpx.org/advanced/transports/#custom-transports), which you can use to implement custom transport implementations against. This formalises and replaces our previously private "Dispatch API". ### Changed diff --git a/README.md b/README.md index 62fb295d..5e459a28 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ and async APIs**. Install HTTPX using pip: ```shell -$ pip install httpx +pip install httpx ``` Now, let's get started: @@ -43,7 +43,7 @@ Now, let's get started: Or, using the command-line client. ```shell -$ pip install 'httpx[cli]' # The command line client is an optional dependency. +pip install 'httpx[cli]' # The command line client is an optional dependency. ``` Which now allows us to use HTTPX directly from the command-line... @@ -66,7 +66,7 @@ HTTPX builds on the well-established usability of `requests`, and gives you: * An integrated command-line client. * HTTP/1.1 [and HTTP/2 support](https://www.python-httpx.org/http2/). * Standard synchronous interface, but with [async support if you need it](https://www.python-httpx.org/async/). -* Ability to make requests directly to [WSGI applications](https://www.python-httpx.org/advanced/#calling-into-python-web-apps) or [ASGI applications](https://www.python-httpx.org/async/#calling-into-python-web-apps). +* Ability to make requests directly to [WSGI applications](https://www.python-httpx.org/advanced/transports/#wsgi-transport) or [ASGI applications](https://www.python-httpx.org/advanced/transports/#asgi-transport). * Strict timeouts everywhere. * Fully type annotated. * 100% test coverage. @@ -94,13 +94,13 @@ Plus all the standard features of `requests`... Install with pip: ```shell -$ pip install httpx +pip install httpx ``` Or, to include the optional HTTP/2 support, use: ```shell -$ pip install httpx[http2] +pip install httpx[http2] ``` HTTPX requires Python 3.8+. @@ -138,6 +138,7 @@ As well as these optional installs: * `rich` - Rich terminal support. *(Optional, with `httpx[cli]`)* * `click` - Command line client support. *(Optional, with `httpx[cli]`)* * `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx[brotli]`)* +* `zstandard` - Decoding for "zstd" compressed responses. *(Optional, with `httpx[zstd]`)* A huge amount of credit is due to `requests` for the API layout that much of this work follows, as well as to `urllib3` for plenty of design diff --git a/README_chinese.md b/README_chinese.md deleted file mode 100644 index ad20c5a1..00000000 --- a/README_chinese.md +++ /dev/null @@ -1,144 +0,0 @@ -

- HTTPX -

- -

HTTPX - 适用于 Python 的下一代 HTTP 客户端

- -

- - Test Suite - - - Package version - -

- -HTTPX 是适用于 Python3 的功能齐全的 HTTP 客户端。 它集成了 **一个命令行客户端**，同时支持 **HTTP/1.1 和 HTTP/2**，并提供了 **同步和异步 API**。 - ---- - -通过 pip 安装 HTTPX： - -```shell -$ pip install httpx -``` - -使用 httpx： - -```pycon ->>> import httpx ->>> r = httpx.get('https://www.example.org/') ->>> r - ->>> r.status_code -200 ->>> r.headers['content-type'] -'text/html; charset=UTF-8' ->>> r.text -'\n\n\nExample Domain...' -``` - -或者使用命令行客户端。 - -```shell -$ pip install 'httpx[cli]' # 命令行功能是可选的。 -``` - -它允许我们直接通过命令行来使用 HTTPX... - -

- httpx --help -

- -发送一个请求... - -

- httpx http://httpbin.org/json -

- -## 特性 - -HTTPX 建立在成熟的 requests 可用性基础上，为您提供以下功能： - -* 广泛的 [requests 兼容 API](https://www.python-httpx.org/compatibility/)。 -* 内置的命令行客户端功能。 -* HTTP/1.1 [和 HTTP/2 支持](https://www.python-httpx.org/http2/)。 -* 标准同步接口，也支持 [异步](https://www.python-httpx.org/async/)。 -* 能够直接向 [WSGI 应用发送请求](https://www.python-httpx.org/advanced/#calling-into-python-web-apps) 或向 [ASGI 应用发送请求](https://www.python-httpx.org/async/#calling-into-python-web-apps)。 -* 每一处严格的超时控制。 -* 完整的类型注解。 -* 100% 测试。 - -加上这些应该具备的标准功能... - -* 国际化域名与 URL -* Keep-Alive & 连接池 -* Cookie 持久性会话 -* 浏览器风格的 SSL 验证 -* 基础或摘要身份验证 -* 优雅的键值 Cookies -* 自动解压缩 -* 内容自动解码 -* Unicode 响应正文 -* 分段文件上传 -* HTTP(S)代理支持 -* 可配置的连接超时 -* 流式下载 -* .netrc 支持 -* 分块请求 - -## 安装 - -使用 pip 安装： - -```shell -$ pip install httpx -``` - -或者，安装可选的 HTTP/2 支持： - -```shell -$ pip install httpx[http2] -``` - -HTTPX 要求 Python 3.8+ 版本。 - -## 文档 - -项目文档现已就绪，请访问 [https://www.python-httpx.org/](https://www.python-httpx.org/) 来阅读。 - -要浏览所有基础知识，请访问 [快速开始](https://www.python-httpx.org/quickstart/)。 - -更高级的主题，可参阅 [高级用法](https://www.python-httpx.org/advanced/) 章节, [异步支持](https://www.python-httpx.org/async/) 或者 [HTTP/2](https://www.python-httpx.org/http2/) 章节。 - -[Developer Interface](https://www.python-httpx.org/api/) 提供了全面的 API 参考。 - -要了解与 HTTPX 集成的工具, 请访问 [第三方包](https://www.python-httpx.org/third_party_packages/)。 - -## 贡献 - -如果您想对本项目做出贡献，请访问 [贡献者指南](https://www.python-httpx.org/contributing/) 来了解如何开始。 - -## 依赖 - -HTTPX 项目依赖于这些优秀的库： - -* `httpcore` - `httpx` 基础传输接口实现。 - * `h11` - HTTP/1.1 支持。 -* `certifi` - SSL 证书。 -* `idna` - 国际化域名支持。 -* `sniffio` - 异步库自动检测。 - -以及这些可选的安装： - -* `h2` - HTTP/2 支持。 *(可选的，通过 `httpx[http2]`)* -* `socksio` - SOCKS 代理支持。 *(可选的， 通过 `httpx[socks]`)* -* `rich` - 丰富的终端支持。 *(可选的，通过 `httpx[cli]`)* -* `click` - 命令行客户端支持。 *(可选的，通过 `httpx[cli]`)* -* `brotli` 或者 `brotlicffi` - 对 “brotli” 压缩响应的解码。*(可选的，通过 `httpx[brotli]`)* - -这项工作的大量功劳都归功于参考了 `requests` 所遵循的 API 结构，以及 `urllib3` 中众多围绕底层网络细节的设计灵感。 - ---- - -

HTTPX 使用 BSD 开源协议 code。
精心设计和制作。
— 🦋 —

diff --git a/docs/advanced/authentication.md b/docs/advanced/authentication.md index edcc15f8..63d26e5f 100644 --- a/docs/advanced/authentication.md +++ b/docs/advanced/authentication.md @@ -1,7 +1,7 @@ Authentication can either be included on a per-request basis... ```pycon ->>> auth = httpx.BasicAuthentication(username="username", password="secret") +>>> auth = httpx.BasicAuth(username="username", password="secret") >>> client = httpx.Client() >>> response = client.get("https://www.example.com/", auth=auth) ``` @@ -9,7 +9,7 @@ Authentication can either be included on a per-request basis... Or configured on the client instance, ensuring that all outgoing requests will include authentication credentials... ```pycon ->>> auth = httpx.BasicAuthentication(username="username", password="secret") +>>> auth = httpx.BasicAuth(username="username", password="secret") >>> client = httpx.Client(auth=auth) >>> response = client.get("https://www.example.com/") ``` @@ -19,7 +19,7 @@ Or configured on the client instance, ensuring that all outgoing requests will i HTTP basic authentication is an unencrypted authentication scheme that uses a simple encoding of the username and password in the request `Authorization` header. Since it is unencrypted it should typically only be used over `https`, although this is not strictly enforced. ```pycon ->>> auth = httpx.BasicAuthentication(username="finley", password="secret") +>>> auth = httpx.BasicAuth(username="finley", password="secret") >>> client = httpx.Client(auth=auth) >>> response = client.get("https://httpbin.org/basic-auth/finley/secret") >>> response diff --git a/docs/advanced/extensions.md b/docs/advanced/extensions.md new file mode 100644 index 00000000..d9208ccd --- /dev/null +++ b/docs/advanced/extensions.md @@ -0,0 +1,242 @@ +# Extensions + +Request and response extensions provide a untyped space where additional information may be added. + +Extensions should be used for features that may not be available on all transports, and that do not fit neatly into [the simplified request/response model](https://www.encode.io/httpcore/extensions/) that the underlying `httpcore` package uses as its API. + +Several extensions are supported on the request: + +```python +# Request timeouts actually implemented as an extension on +# the request, ensuring that they are passed throughout the +# entire call stack. +client = httpx.Client() +response = client.get( + "https://www.example.com", + extensions={"timeout": {"connect": 5.0}} +) +response.request.extensions["timeout"] +{"connect": 5.0} +``` + +And on the response: + +```python +client = httpx.Client() +response = client.get("https://www.example.com") +print(response.extensions["http_version"]) # b"HTTP/1.1" +# Other server responses could have been +# b"HTTP/0.9", b"HTTP/1.0", or b"HTTP/1.1" +``` + +## Request Extensions + +### `"trace"` + +The trace extension allows a callback handler to be installed to monitor the internal +flow of events within the underlying `httpcore` transport. + +The simplest way to explain this is with an example: + +```python +import httpx + +def log(event_name, info): + print(event_name, info) + +client = httpx.Client() +response = client.get("https://www.example.com/", extensions={"trace": log}) +# connection.connect_tcp.started {'host': 'www.example.com', 'port': 443, 'local_address': None, 'timeout': None} +# connection.connect_tcp.complete {'return_value': } +# connection.start_tls.started {'ssl_context': , 'server_hostname': b'www.example.com', 'timeout': None} +# connection.start_tls.complete {'return_value': } +# http11.send_request_headers.started {'request': } +# http11.send_request_headers.complete {'return_value': None} +# http11.send_request_body.started {'request': } +# http11.send_request_body.complete {'return_value': None} +# http11.receive_response_headers.started {'request': } +# http11.receive_response_headers.complete {'return_value': (b'HTTP/1.1', 200, b'OK', [(b'Age', b'553715'), (b'Cache-Control', b'max-age=604800'), (b'Content-Type', b'text/html; charset=UTF-8'), (b'Date', b'Thu, 21 Oct 2021 17:08:42 GMT'), (b'Etag', b'"3147526947+ident"'), (b'Expires', b'Thu, 28 Oct 2021 17:08:42 GMT'), (b'Last-Modified', b'Thu, 17 Oct 2019 07:18:26 GMT'), (b'Server', b'ECS (nyb/1DCD)'), (b'Vary', b'Accept-Encoding'), (b'X-Cache', b'HIT'), (b'Content-Length', b'1256')])} +# http11.receive_response_body.started {'request': } +# http11.receive_response_body.complete {'return_value': None} +# http11.response_closed.started {} +# http11.response_closed.complete {'return_value': None} +``` + +The `event_name` and `info` arguments here will be one of the following: + +* `{event_type}.{event_name}.started`, `` +* `{event_type}.{event_name}.complete`, `{"return_value": <...>}` +* `{event_type}.{event_name}.failed`, `{"exception": <...>}` + +Note that when using async code the handler function passed to `"trace"` must be an `async def ...` function. + +The following event types are currently exposed... + +**Establishing the connection** + +* `"connection.connect_tcp"` +* `"connection.connect_unix_socket"` +* `"connection.start_tls"` + +**HTTP/1.1 events** + +* `"http11.send_request_headers"` +* `"http11.send_request_body"` +* `"http11.receive_response"` +* `"http11.receive_response_body"` +* `"http11.response_closed"` + +**HTTP/2 events** + +* `"http2.send_connection_init"` +* `"http2.send_request_headers"` +* `"http2.send_request_body"` +* `"http2.receive_response_headers"` +* `"http2.receive_response_body"` +* `"http2.response_closed"` + +The exact set of trace events may be subject to change across different versions of `httpcore`. If you need to rely on a particular set of events it is recommended that you pin installation of the package to a fixed version. + +### `"sni_hostname"` + +The server's hostname, which is used to confirm the hostname supplied by the SSL certificate. + +If you want to connect to an explicit IP address rather than using the standard DNS hostname lookup, then you'll need to use this request extension. + +For example: + +``` python +# Connect to '185.199.108.153' but use 'www.encode.io' in the Host header, +# and use 'www.encode.io' when SSL verifying the server hostname. +client = httpx.Client() +headers = {"Host": "www.encode.io"} +extensions = {"sni_hostname": "www.encode.io"} +response = client.get( + "https://185.199.108.153/path", + headers=headers, + extensions=extensions +) +``` + +### `"timeout"` + +A dictionary of `str: Optional[float]` timeout values. + +May include values for `'connect'`, `'read'`, `'write'`, or `'pool'`. + +For example: + +```python +# Timeout if a connection takes more than 5 seconds to established, or if +# we are blocked waiting on the connection pool for more than 10 seconds. +client = httpx.Client() +response = client.get( + "https://www.example.com", + extensions={"timeout": {"connect": 5.0, "pool": 10.0}} +) +``` + +This extension is how the `httpx` timeouts are implemented, ensuring that the timeout values are associated with the request instance and passed throughout the stack. You shouldn't typically be working with this extension directly, but use the higher level `timeout` API instead. + +### `"target"` + +The target that is used as [the HTTP target instead of the URL path](https://datatracker.ietf.org/doc/html/rfc2616#section-5.1.2). + +This enables support constructing requests that would otherwise be unsupported. + +* URL paths with non-standard escaping applied. +* Forward proxy requests using an absolute URI. +* Tunneling proxy requests using `CONNECT` with hostname as the target. +* Server-wide `OPTIONS *` requests. + +Some examples: + +Using the 'target' extension to send requests without the standard path escaping rules... + +```python +# Typically a request to "https://www.example.com/test^path" would +# connect to "www.example.com" and send an HTTP/1.1 request like... +# +# GET /test%5Epath HTTP/1.1 +# +# Using the target extension we can include the literal '^'... +# +# GET /test^path HTTP/1.1 +# +# Note that requests must still be valid HTTP requests. +# For example including whitespace in the target will raise a `LocalProtocolError`. +extensions = {"target": b"/test^path"} +response = httpx.get("https://www.example.com", extensions=extensions) +``` + +The `target` extension also allows server-wide `OPTIONS *` requests to be constructed... + +```python +# This will send the following request... +# +# CONNECT * HTTP/1.1 +extensions = {"target": b"*"} +response = httpx.request("CONNECT", "https://www.example.com", extensions=extensions) +``` + +## Response Extensions + +### `"http_version"` + +The HTTP version, as bytes. Eg. `b"HTTP/1.1"`. + +When using HTTP/1.1 the response line includes an explicit version, and the value of this key could feasibly be one of `b"HTTP/0.9"`, `b"HTTP/1.0"`, or `b"HTTP/1.1"`. + +When using HTTP/2 there is no further response versioning included in the protocol, and the value of this key will always be `b"HTTP/2"`. + +### `"reason_phrase"` + +The reason-phrase of the HTTP response, as bytes. For example `b"OK"`. Some servers may include a custom reason phrase, although this is not recommended. + +HTTP/2 onwards does not include a reason phrase on the wire. + +When no key is included, a default based on the status code may be used. + +### `"stream_id"` + +When HTTP/2 is being used the `"stream_id"` response extension can be accessed to determine the ID of the data stream that the response was sent on. + +### `"network_stream"` + +The `"network_stream"` extension allows developers to handle HTTP `CONNECT` and `Upgrade` requests, by providing an API that steps outside the standard request/response model, and can directly read or write to the network. + +The interface provided by the network stream: + +* `read(max_bytes, timeout = None) -> bytes` +* `write(buffer, timeout = None)` +* `close()` +* `start_tls(ssl_context, server_hostname = None, timeout = None) -> NetworkStream` +* `get_extra_info(info) -> Any` + +This API can be used as the foundation for working with HTTP proxies, WebSocket upgrades, and other advanced use-cases. + +See the [network backends documentation](https://www.encode.io/httpcore/network-backends/) for more information on working directly with network streams. + +**Extra network information** + +The network stream abstraction also allows access to various low-level information that may be exposed by the underlying socket: + +```python +response = httpx.get("https://www.example.com") +network_stream = response.extensions["network_stream"] + +client_addr = network_stream.get_extra_info("client_addr") +server_addr = network_stream.get_extra_info("server_addr") +print("Client address", client_addr) +print("Server address", server_addr) +``` + +The socket SSL information is also available through this interface, although you need to ensure that the underlying connection is still open, in order to access it... + +```python +with httpx.stream("GET", "https://www.example.com") as response: + network_stream = response.extensions["network_stream"] + + ssl_object = network_stream.get_extra_info("ssl_object") + print("TLS version", ssl_object.version()) +``` diff --git a/docs/advanced/proxies.md b/docs/advanced/proxies.md index 2a6b7d5f..f1ee3ec8 100644 --- a/docs/advanced/proxies.md +++ b/docs/advanced/proxies.md @@ -73,7 +73,7 @@ This is an optional feature that requires an additional third-party library be i You can install SOCKS support using `pip`: ```shell -$ pip install httpx[socks] +pip install httpx[socks] ``` You can now configure a client to make requests via a proxy using the SOCKS protocol: diff --git a/docs/advanced/transports.md b/docs/advanced/transports.md index fdc58b08..d4e7615d 100644 --- a/docs/advanced/transports.md +++ b/docs/advanced/transports.md @@ -2,7 +2,7 @@ HTTPX's `Client` also accepts a `transport` argument. This argument allows you to provide a custom Transport object that will be used to perform the actual sending of the requests. -## HTTPTransport +## HTTP Transport For some advanced configuration you might need to instantiate a transport class directly, and pass it to the client instance. One example is the @@ -42,7 +42,9 @@ You can configure an `httpx` client to call directly into a Python web applicati This is particularly useful for two main use-cases: * Using `httpx` as a client inside test cases. -* Mocking out external services during tests or in dev/staging environments. +* Mocking out external services during tests or in dev or staging environments. + +### Example Here's an example of integrating against a Flask application: @@ -57,12 +59,15 @@ app = Flask(__name__) def hello(): return "Hello World!" -with httpx.Client(app=app, base_url="http://testserver") as client: +transport = httpx.WSGITransport(app=app) +with httpx.Client(transport=transport, base_url="http://testserver") as client: r = client.get("/") assert r.status_code == 200 assert r.text == "Hello World!" ``` +### Configuration + For some more complex cases you might need to customize the WSGI transport. This allows you to: * Inspect 500 error responses rather than raise exceptions by setting `raise_app_exceptions=False`. @@ -78,9 +83,72 @@ with httpx.Client(transport=transport, base_url="http://testserver") as client: ... ``` +## ASGI Transport + +You can configure an `httpx` client to call directly into an async Python web application using the ASGI protocol. + +This is particularly useful for two main use-cases: + +* Using `httpx` as a client inside test cases. +* Mocking out external services during tests or in dev or staging environments. + +### Example + +Let's take this Starlette application as an example: + +```python +from starlette.applications import Starlette +from starlette.responses import HTMLResponse +from starlette.routing import Route + + +async def hello(request): + return HTMLResponse("Hello World!") + + +app = Starlette(routes=[Route("/", hello)]) +``` + +We can make requests directly against the application, like so: + +```python +transport = httpx.ASGITransport(app=app) + +async with httpx.AsyncClient(transport=transport, base_url="http://testserver") as client: + r = await client.get("/") + assert r.status_code == 200 + assert r.text == "Hello World!" +``` + +### Configuration + +For some more complex cases you might need to customise the ASGI transport. This allows you to: + +* Inspect 500 error responses rather than raise exceptions by setting `raise_app_exceptions=False`. +* Mount the ASGI application at a subpath by setting `root_path`. +* Use a given client address for requests by setting `client`. + +For example: + +```python +# Instantiate a client that makes ASGI requests with a client IP of "1.2.3.4", +# on port 123. +transport = httpx.ASGITransport(app=app, client=("1.2.3.4", 123)) +async with httpx.AsyncClient(transport=transport, base_url="http://testserver") as client: + ... +``` + +See [the ASGI documentation](https://asgi.readthedocs.io/en/latest/specs/www.html#connection-scope) for more details on the `client` and `root_path` keys. + +### ASGI startup and shutdown + +It is not in the scope of HTTPX to trigger ASGI lifespan events of your app. + +However it is suggested to use `LifespanManager` from [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan#usage) in pair with `AsyncClient`. + ## Custom transports -A transport instance must implement the low-level Transport API, which deals +A transport instance must implement the low-level Transport API which deals with sending a single request, and returning a response. You should either subclass `httpx.BaseTransport` to implement a transport to use with `Client`, or subclass `httpx.AsyncBaseTransport` to implement a transport to @@ -98,28 +166,81 @@ A complete example of a custom transport implementation would be: import json import httpx - class HelloWorldTransport(httpx.BaseTransport): """ A mock transport that always returns a JSON "Hello, world!" response. """ def handle_request(self, request): - message = {"text": "Hello, world!"} - content = json.dumps(message).encode("utf-8") - stream = httpx.ByteStream(content) - headers = [(b"content-type", b"application/json")] - return httpx.Response(200, headers=headers, stream=stream) + return httpx.Response(200, json={"text": "Hello, world!"}) ``` -Which we can use in the same way: +Or this example, which uses a custom transport and `httpx.Mounts` to always redirect `http://` requests. -```pycon ->>> import httpx ->>> client = httpx.Client(transport=HelloWorldTransport()) ->>> response = client.get("https://example.org/") ->>> response.json() -{"text": "Hello, world!"} +```python +class HTTPSRedirect(httpx.BaseTransport): + """ + A transport that always redirects to HTTPS. + """ + def handle_request(self, request): + url = request.url.copy_with(scheme="https") + return httpx.Response(303, headers={"Location": str(url)}) + +# A client where any `http` requests are always redirected to `https` +transport = httpx.Mounts({ + 'http://': HTTPSRedirect() + 'https://': httpx.HTTPTransport() +}) +client = httpx.Client(transport=transport) +``` + +A useful pattern here is custom transport classes that wrap the default HTTP implementation. For example... + +```python +class DebuggingTransport(httpx.BaseTransport): + def __init__(self, **kwargs): + self._wrapper = httpx.HTTPTransport(**kwargs) + + def handle_request(self, request): + print(f">>> {request}") + response = self._wrapper.handle_request(request) + print(f"<<< {response}") + return response + + def close(self): + self._wrapper.close() + +transport = DebuggingTransport() +client = httpx.Client(transport=transport) +``` + +Here's another case, where we're using a round-robin across a number of different proxies... + +```python +class ProxyRoundRobin(httpx.BaseTransport): + def __init__(self, proxies, **kwargs): + self._transports = [ + httpx.HTTPTransport(proxy=proxy, **kwargs) + for proxy in proxies + ] + self._idx = 0 + + def handle_request(self, request): + transport = self._transports[self._idx] + self._idx = (self._idx + 1) % len(self._transports) + return transport.handle_request(request) + + def close(self): + for transport in self._transports: + transport.close() + +proxies = [ + httpx.Proxy("http://127.0.0.1:8081"), + httpx.Proxy("http://127.0.0.1:8082"), + httpx.Proxy("http://127.0.0.1:8083"), +] +transport = ProxyRoundRobin(proxies=proxies) +client = httpx.Client(transport=transport) ``` ## Mock transports @@ -329,4 +450,5 @@ mounts = { There are also environment variables that can be used to control the dictionary of the client mounts. They can be used to configure HTTP proxying for clients. -See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](../environment_variables.md#http_proxy-https_proxy-all_proxy) for more information. +See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](../environment_variables.md#http_proxy-https_proxy-all_proxy) +and [`NO_PROXY`](../environment_variables.md#no_proxy) for more information. diff --git a/docs/api.md b/docs/api.md index 3f9878c7..d01cc649 100644 --- a/docs/api.md +++ b/docs/api.md @@ -114,7 +114,7 @@ what gets sent over the wire.* 'example.org' ``` -* `def __init__(url, allow_relative=False, params=None)` +* `def __init__(url, **kwargs)` * `.scheme` - **str** * `.authority` - **str** * `.host` - **str** diff --git a/docs/async.md b/docs/async.md index d54a353d..089d7831 100644 --- a/docs/async.md +++ b/docs/async.md @@ -191,54 +191,4 @@ anyio.run(main, backend='trio') ## Calling into Python Web Apps -Just as `httpx.Client` allows you to call directly into WSGI web applications, -the `httpx.AsyncClient` class allows you to call directly into ASGI web applications. - -Let's take this Starlette application as an example: - -```python -from starlette.applications import Starlette -from starlette.responses import HTMLResponse -from starlette.routing import Route - - -async def hello(request): - return HTMLResponse("Hello World!") - - -app = Starlette(routes=[Route("/", hello)]) -``` - -We can make requests directly against the application, like so: - -```pycon ->>> import httpx ->>> async with httpx.AsyncClient(app=app, base_url="http://testserver") as client: -... r = await client.get("/") -... assert r.status_code == 200 -... assert r.text == "Hello World!" -``` - -For some more complex cases you might need to customise the ASGI transport. This allows you to: - -* Inspect 500 error responses rather than raise exceptions by setting `raise_app_exceptions=False`. -* Mount the ASGI application at a subpath by setting `root_path`. -* Use a given client address for requests by setting `client`. - -For example: - -```python -# Instantiate a client that makes ASGI requests with a client IP of "1.2.3.4", -# on port 123. -transport = httpx.ASGITransport(app=app, client=("1.2.3.4", 123)) -async with httpx.AsyncClient(transport=transport, base_url="http://testserver") as client: - ... -``` - -See [the ASGI documentation](https://asgi.readthedocs.io/en/latest/specs/www.html#connection-scope) for more details on the `client` and `root_path` keys. - -## Startup/shutdown of ASGI apps - -It is not in the scope of HTTPX to trigger lifespan events of your app. - -However it is suggested to use `LifespanManager` from [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan#usage) in pair with `AsyncClient`. +For details on calling directly into ASGI applications, see [the `ASGITransport` docs](../advanced/transports#asgitransport). \ No newline at end of file diff --git a/docs/contributing.md b/docs/contributing.md index 47dd9dc5..110a127c 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -46,14 +46,14 @@ Then clone your fork with the following command replacing `YOUR-USERNAME` with your GitHub username: ```shell -$ git clone https://github.com/YOUR-USERNAME/httpx +git clone https://github.com/YOUR-USERNAME/httpx ``` You can now install the project and its dependencies using: ```shell -$ cd httpx -$ scripts/install +cd httpx +scripts/install ``` ## Testing and Linting @@ -64,7 +64,7 @@ and documentation building workflow. To run the tests, use: ```shell -$ scripts/test +scripts/test ``` !!! warning @@ -76,19 +76,19 @@ Any additional arguments will be passed to `pytest`. See the [pytest documentati For example, to run a single test script: ```shell -$ scripts/test tests/test_multipart.py +scripts/test tests/test_multipart.py ``` To run the code auto-formatting: ```shell -$ scripts/lint +scripts/lint ``` Lastly, to run code checks separately (they are also run as part of `scripts/test`), run: ```shell -$ scripts/check +scripts/check ``` ## Documenting @@ -98,7 +98,7 @@ Documentation pages are located under the `docs/` folder. To run the documentation site locally (useful for previewing changes), use: ```shell -$ scripts/docs +scripts/docs ``` ## Resolving Build / CI Failures @@ -122,7 +122,7 @@ This job failing means there is either a code formatting issue or type-annotatio You can look at the job output to figure out why it's failed or within a shell run: ```shell -$ scripts/check +scripts/check ``` It may be worth it to run `$ scripts/lint` to attempt auto-formatting the code @@ -206,8 +206,8 @@ UI options. At this point the server is ready to start serving requests, you'll need to configure HTTPX as described in the -[proxy section](https://www.python-httpx.org/advanced/#http-proxying) and -the [SSL certificates section](https://www.python-httpx.org/advanced/#ssl-certificates), +[proxy section](https://www.python-httpx.org/advanced/proxies/#http-proxies) and +the [SSL certificates section](https://www.python-httpx.org/advanced/ssl/), this is where our previously generated `client.pem` comes in: ``` diff --git a/docs/http2.md b/docs/http2.md index 3cab09d9..434606c4 100644 --- a/docs/http2.md +++ b/docs/http2.md @@ -28,7 +28,7 @@ trying out our HTTP/2 support. You can do so by first making sure to install the optional HTTP/2 dependencies... ```shell -$ pip install httpx[http2] +pip install httpx[http2] ``` And then instantiating a client with HTTP/2 support enabled: diff --git a/docs/index.md b/docs/index.md index 86b6d1cb..98bf0fd6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -28,7 +28,7 @@ HTTPX is a fully featured HTTP client for Python 3, which provides sync and asyn Install HTTPX using pip: ```shell -$ pip install httpx +pip install httpx ``` Now, let's get started: @@ -50,7 +50,7 @@ Or, using the command-line client. ```shell # The command line client is an optional dependency. -$ pip install 'httpx[cli]' +pip install 'httpx[cli]' ``` Which now allows us to use HTTPX directly from the command-line... @@ -68,7 +68,7 @@ HTTPX builds on the well-established usability of `requests`, and gives you: * A broadly [requests-compatible API](compatibility.md). * Standard synchronous interface, but with [async support if you need it](async.md). * HTTP/1.1 [and HTTP/2 support](http2.md). -* Ability to make requests directly to [WSGI applications](async.md#calling-into-python-web-apps) or [ASGI applications](async.md#calling-into-python-web-apps). +* Ability to make requests directly to [WSGI applications](advanced/transports.md#wsgi-transport) or [ASGI applications](advanced/transports.md#asgi-transport). * Strict timeouts everywhere. * Fully type annotated. * 100% test coverage. @@ -119,6 +119,7 @@ As well as these optional installs: * `rich` - Rich terminal support. *(Optional, with `httpx[cli]`)* * `click` - Command line client support. *(Optional, with `httpx[cli]`)* * `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx[brotli]`)* +* `zstandard` - Decoding for "zstd" compressed responses. *(Optional, with `httpx[zstd]`)* A huge amount of credit is due to `requests` for the API layout that much of this work follows, as well as to `urllib3` for plenty of design @@ -129,19 +130,19 @@ inspiration around the lower-level networking details. Install with pip: ```shell -$ pip install httpx +pip install httpx ``` Or, to include the optional HTTP/2 support, use: ```shell -$ pip install httpx[http2] +pip install httpx[http2] ``` -To include the optional brotli decoder support, use: +To include the optional brotli and zstandard decoders support, use: ```shell -$ pip install httpx[brotli] +pip install httpx[brotli,zstd] ``` HTTPX requires Python 3.8+ diff --git a/docs/quickstart.md b/docs/quickstart.md index 974119f7..aa203a83 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -100,7 +100,8 @@ b'\n\n\nExample Domain...' Any `gzip` and `deflate` HTTP response encodings will automatically be decoded for you. If `brotlipy` is installed, then the `brotli` response -encoding will also be supported. +encoding will be supported. If `zstandard` is installed, then `zstd` +response encodings will also be supported. For example, to create an image from binary data returned by a request, you can use the following code: @@ -362,7 +363,8 @@ Or stream the text, on a line-by-line basis... HTTPX will use universal line endings, normalising all cases to `\n`. -In some cases you might want to access the raw bytes on the response without applying any HTTP content decoding. In this case any content encoding that the web server has applied such as `gzip`, `deflate`, or `brotli` will not be automatically decoded. +In some cases you might want to access the raw bytes on the response without applying any HTTP content decoding. In this case any content encoding that the web server has applied such as `gzip`, `deflate`, `brotli`, or `zstd` will +not be automatically decoded. ```pycon >>> with httpx.stream("GET", "https://www.example.com") as r: diff --git a/httpx/__init__.py b/httpx/__init__.py index 1665b35a..7a603da6 100644 --- a/httpx/__init__.py +++ b/httpx/__init__.py @@ -1,48 +1,15 @@ from .__version__ import __description__, __title__, __version__ -from ._api import delete, get, head, options, patch, post, put, request, stream -from ._auth import Auth, BasicAuth, DigestAuth, NetRCAuth -from ._client import USE_CLIENT_DEFAULT, AsyncClient, Client -from ._config import Limits, NetworkOptions, Proxy, Timeout, create_ssl_context -from ._content import ByteStream -from ._exceptions import ( - CloseError, - ConnectError, - ConnectTimeout, - CookieConflict, - DecodingError, - HTTPError, - HTTPStatusError, - InvalidURL, - LocalProtocolError, - NetworkError, - PoolTimeout, - ProtocolError, - ProxyError, - ReadError, - ReadTimeout, - RemoteProtocolError, - RequestError, - RequestNotRead, - ResponseNotRead, - StreamClosed, - StreamConsumed, - StreamError, - TimeoutException, - TooManyRedirects, - TransportError, - UnsupportedProtocol, - WriteError, - WriteTimeout, -) -from ._models import Cookies, Headers, Request, Response -from ._status_codes import codes -from ._transports.asgi import ASGITransport -from ._transports.base import AsyncBaseTransport, BaseTransport -from ._transports.default import AsyncHTTPTransport, HTTPTransport -from ._transports.mock import MockTransport -from ._transports.wsgi import WSGITransport -from ._types import AsyncByteStream, SyncByteStream -from ._urls import URL, QueryParams +from ._api import * +from ._auth import * +from ._client import * +from ._config import * +from ._content import * +from ._exceptions import * +from ._models import * +from ._status_codes import * +from ._transports import * +from ._types import * +from ._urls import * try: from ._main import main diff --git a/httpx/__version__.py b/httpx/__version__.py index 3edc842c..c121a898 100644 --- a/httpx/__version__.py +++ b/httpx/__version__.py @@ -1,3 +1,3 @@ __title__ = "httpx" __description__ = "A next generation HTTP client, for Python 3." -__version__ = "0.26.0" +__version__ = "0.27.0" diff --git a/httpx/_api.py b/httpx/_api.py index c7af9472..3dd943b3 100644 --- a/httpx/_api.py +++ b/httpx/_api.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing from contextlib import contextmanager @@ -20,25 +22,37 @@ from ._types import ( VerifyTypes, ) +__all__ = [ + "delete", + "get", + "head", + "options", + "patch", + "post", + "put", + "request", + "stream", +] + def request( method: str, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + params: QueryParamTypes | None = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, follow_redirects: bool = False, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, trust_env: bool = True, ) -> Response: """ @@ -120,20 +134,20 @@ def stream( method: str, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + params: QueryParamTypes | None = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, follow_redirects: bool = False, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, trust_env: bool = True, ) -> typing.Iterator[Response]: """ @@ -173,14 +187,14 @@ def stream( def get( url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, @@ -213,14 +227,14 @@ def get( def options( url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, @@ -253,14 +267,14 @@ def options( def head( url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, @@ -293,18 +307,18 @@ def head( def post( url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, @@ -338,18 +352,18 @@ def post( def put( url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, @@ -383,18 +397,18 @@ def put( def patch( url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, @@ -428,14 +442,14 @@ def patch( def delete( url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Optional[AuthTypes] = None, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | None = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, follow_redirects: bool = False, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, trust_env: bool = True, diff --git a/httpx/_auth.py b/httpx/_auth.py index e8bc0cd9..b03971ab 100644 --- a/httpx/_auth.py +++ b/httpx/_auth.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import hashlib import os import re @@ -14,6 +16,9 @@ if typing.TYPE_CHECKING: # pragma: no cover from hashlib import _Hash +__all__ = ["Auth", "BasicAuth", "DigestAuth", "NetRCAuth"] + + class Auth: """ Base class for all authentication schemes. @@ -124,18 +129,14 @@ class BasicAuth(Auth): and uses HTTP Basic authentication. """ - def __init__( - self, username: typing.Union[str, bytes], password: typing.Union[str, bytes] - ) -> None: + def __init__(self, username: str | bytes, password: str | bytes) -> None: self._auth_header = self._build_auth_header(username, password) def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: request.headers["Authorization"] = self._auth_header yield request - def _build_auth_header( - self, username: typing.Union[str, bytes], password: typing.Union[str, bytes] - ) -> str: + def _build_auth_header(self, username: str | bytes, password: str | bytes) -> str: userpass = b":".join((to_bytes(username), to_bytes(password))) token = b64encode(userpass).decode() return f"Basic {token}" @@ -146,7 +147,7 @@ class NetRCAuth(Auth): Use a 'netrc' file to lookup basic auth credentials based on the url host. """ - def __init__(self, file: typing.Optional[str] = None) -> None: + def __init__(self, file: str | None = None) -> None: # Lazily import 'netrc'. # There's no need for us to load this module unless 'NetRCAuth' is being used. import netrc @@ -165,16 +166,14 @@ class NetRCAuth(Auth): ) yield request - def _build_auth_header( - self, username: typing.Union[str, bytes], password: typing.Union[str, bytes] - ) -> str: + def _build_auth_header(self, username: str | bytes, password: str | bytes) -> str: userpass = b":".join((to_bytes(username), to_bytes(password))) token = b64encode(userpass).decode() return f"Basic {token}" class DigestAuth(Auth): - _ALGORITHM_TO_HASH_FUNCTION: typing.Dict[str, typing.Callable[[bytes], "_Hash"]] = { + _ALGORITHM_TO_HASH_FUNCTION: dict[str, typing.Callable[[bytes], _Hash]] = { "MD5": hashlib.md5, "MD5-SESS": hashlib.md5, "SHA": hashlib.sha1, @@ -185,12 +184,10 @@ class DigestAuth(Auth): "SHA-512-SESS": hashlib.sha512, } - def __init__( - self, username: typing.Union[str, bytes], password: typing.Union[str, bytes] - ) -> None: + def __init__(self, username: str | bytes, password: str | bytes) -> None: self._username = to_bytes(username) self._password = to_bytes(password) - self._last_challenge: typing.Optional[_DigestAuthChallenge] = None + self._last_challenge: _DigestAuthChallenge | None = None self._nonce_count = 1 def auth_flow(self, request: Request) -> typing.Generator[Request, Response, None]: @@ -226,7 +223,7 @@ class DigestAuth(Auth): def _parse_challenge( self, request: Request, response: Response, auth_header: str - ) -> "_DigestAuthChallenge": + ) -> _DigestAuthChallenge: """ Returns a challenge from a Digest WWW-Authenticate header. These take the form of: @@ -237,7 +234,7 @@ class DigestAuth(Auth): # This method should only ever have been called with a Digest auth header. assert scheme.lower() == "digest" - header_dict: typing.Dict[str, str] = {} + header_dict: dict[str, str] = {} for field in parse_http_list(fields): key, value = field.strip().split("=", 1) header_dict[key] = unquote(value) @@ -256,7 +253,7 @@ class DigestAuth(Auth): raise ProtocolError(message, request=request) from exc def _build_auth_header( - self, request: Request, challenge: "_DigestAuthChallenge" + self, request: Request, challenge: _DigestAuthChallenge ) -> str: hash_func = self._ALGORITHM_TO_HASH_FUNCTION[challenge.algorithm.upper()] @@ -311,7 +308,7 @@ class DigestAuth(Auth): return hashlib.sha1(s).hexdigest()[:16].encode() - def _get_header_value(self, header_fields: typing.Dict[str, bytes]) -> str: + def _get_header_value(self, header_fields: dict[str, bytes]) -> str: NON_QUOTED_FIELDS = ("algorithm", "qop", "nc") QUOTED_TEMPLATE = '{}="{}"' NON_QUOTED_TEMPLATE = "{}={}" @@ -329,9 +326,7 @@ class DigestAuth(Auth): return header_value - def _resolve_qop( - self, qop: typing.Optional[bytes], request: Request - ) -> typing.Optional[bytes]: + def _resolve_qop(self, qop: bytes | None, request: Request) -> bytes | None: if qop is None: return None qops = re.split(b", ?", qop) @@ -349,5 +344,5 @@ class _DigestAuthChallenge(typing.NamedTuple): realm: bytes nonce: bytes algorithm: str - opaque: typing.Optional[bytes] - qop: typing.Optional[bytes] + opaque: bytes | None + qop: bytes | None diff --git a/httpx/_client.py b/httpx/_client.py index a0b4209c..d95877e8 100644 --- a/httpx/_client.py +++ b/httpx/_client.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import datetime import enum import logging @@ -56,6 +58,8 @@ from ._utils import ( same_origin, ) +__all__ = ["USE_CLIENT_DEFAULT", "AsyncClient", "Client"] + # The type annotation for @classmethod and context managers here follows PEP 484 # https://www.python.org/dev/peps/pep-0484/#annotating-instance-and-class-methods T = typing.TypeVar("T", bound="Client") @@ -160,19 +164,17 @@ class BaseClient: def __init__( self, *, - auth: typing.Optional[AuthTypes] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, + auth: AuthTypes | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, follow_redirects: bool = False, max_redirects: int = DEFAULT_MAX_REDIRECTS, - event_hooks: typing.Optional[ - typing.Mapping[str, typing.List[EventHook]] - ] = None, + event_hooks: None | (typing.Mapping[str, list[EventHook]]) = None, base_url: URLTypes = "", trust_env: bool = True, - default_encoding: typing.Union[str, typing.Callable[[bytes], str]] = "utf-8", + default_encoding: str | typing.Callable[[bytes], str] = "utf-8", ) -> None: event_hooks = {} if event_hooks is None else event_hooks @@ -210,8 +212,8 @@ class BaseClient: return url.copy_with(raw_path=url.raw_path + b"/") def _get_proxy_map( - self, proxies: typing.Optional[ProxiesTypes], allow_env_proxies: bool - ) -> typing.Dict[str, typing.Optional[Proxy]]: + self, proxies: ProxiesTypes | None, allow_env_proxies: bool + ) -> dict[str, Proxy | None]: if proxies is None: if allow_env_proxies: return { @@ -238,20 +240,18 @@ class BaseClient: self._timeout = Timeout(timeout) @property - def event_hooks(self) -> typing.Dict[str, typing.List[EventHook]]: + def event_hooks(self) -> dict[str, list[EventHook]]: return self._event_hooks @event_hooks.setter - def event_hooks( - self, event_hooks: typing.Dict[str, typing.List[EventHook]] - ) -> None: + def event_hooks(self, event_hooks: dict[str, list[EventHook]]) -> None: self._event_hooks = { "request": list(event_hooks.get("request", [])), "response": list(event_hooks.get("response", [])), } @property - def auth(self) -> typing.Optional[Auth]: + def auth(self) -> Auth | None: """ Authentication class used when none is passed at the request-level. @@ -323,15 +323,15 @@ class BaseClient: method: str, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Request: """ Build and return a request instance. @@ -342,7 +342,7 @@ class BaseClient: See also: [Request instances][0] - [0]: /advanced/#request-instances + [0]: /advanced/clients/#request-instances """ url = self._merge_url(url) headers = self._merge_headers(headers) @@ -391,9 +391,7 @@ class BaseClient: return self.base_url.copy_with(raw_path=merge_raw_path) return merge_url - def _merge_cookies( - self, cookies: typing.Optional[CookieTypes] = None - ) -> typing.Optional[CookieTypes]: + def _merge_cookies(self, cookies: CookieTypes | None = None) -> CookieTypes | None: """ Merge a cookies argument together with any cookies on the client, to create the cookies used for the outgoing request. @@ -404,9 +402,7 @@ class BaseClient: return merged_cookies return cookies - def _merge_headers( - self, headers: typing.Optional[HeaderTypes] = None - ) -> typing.Optional[HeaderTypes]: + def _merge_headers(self, headers: HeaderTypes | None = None) -> HeaderTypes | None: """ Merge a headers argument together with any headers on the client, to create the headers used for the outgoing request. @@ -416,8 +412,8 @@ class BaseClient: return merged_headers def _merge_queryparams( - self, params: typing.Optional[QueryParamTypes] = None - ) -> typing.Optional[QueryParamTypes]: + self, params: QueryParamTypes | None = None + ) -> QueryParamTypes | None: """ Merge a queryparams argument together with any queryparams on the client, to create the queryparams used for the outgoing request. @@ -427,7 +423,7 @@ class BaseClient: return merged_queryparams.merge(params) return params - def _build_auth(self, auth: typing.Optional[AuthTypes]) -> typing.Optional[Auth]: + def _build_auth(self, auth: AuthTypes | None) -> Auth | None: if auth is None: return None elif isinstance(auth, tuple): @@ -442,7 +438,7 @@ class BaseClient: def _build_request_auth( self, request: Request, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, ) -> Auth: auth = ( self._auth if isinstance(auth, UseClientDefault) else self._build_auth(auth) @@ -557,7 +553,7 @@ class BaseClient: def _redirect_stream( self, request: Request, method: str - ) -> typing.Optional[typing.Union[SyncByteStream, AsyncByteStream]]: + ) -> SyncByteStream | AsyncByteStream | None: """ Return the body that should be used for the redirect request. """ @@ -566,6 +562,15 @@ class BaseClient: return request.stream + def _set_timeout(self, request: Request) -> None: + if "timeout" not in request.extensions: + timeout = ( + self.timeout + if isinstance(self.timeout, UseClientDefault) + else Timeout(self.timeout) + ) + request.extensions = dict(**request.extensions, timeout=timeout.as_dict()) + class Client(BaseClient): """ @@ -624,31 +629,27 @@ class Client(BaseClient): def __init__( self, *, - auth: typing.Optional[AuthTypes] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, + auth: AuthTypes | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, - mounts: typing.Optional[ - typing.Mapping[str, typing.Optional[BaseTransport]] - ] = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, + mounts: None | (typing.Mapping[str, BaseTransport | None]) = None, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, follow_redirects: bool = False, limits: Limits = DEFAULT_LIMITS, max_redirects: int = DEFAULT_MAX_REDIRECTS, - event_hooks: typing.Optional[ - typing.Mapping[str, typing.List[EventHook]] - ] = None, + event_hooks: None | (typing.Mapping[str, list[EventHook]]) = None, base_url: URLTypes = "", - transport: typing.Optional[BaseTransport] = None, - app: typing.Optional[typing.Callable[..., typing.Any]] = None, + transport: BaseTransport | None = None, + app: typing.Callable[..., typing.Any] | None = None, trust_env: bool = True, - default_encoding: typing.Union[str, typing.Callable[[bytes], str]] = "utf-8", + default_encoding: str | typing.Callable[[bytes], str] = "utf-8", ) -> None: super().__init__( auth=auth, @@ -682,6 +683,13 @@ class Client(BaseClient): if proxy: raise RuntimeError("Use either `proxy` or 'proxies', not both.") + if app: + message = ( + "The 'app' shortcut is now deprecated." + " Use the explicit style 'transport=WSGITransport(app=...)' instead." + ) + warnings.warn(message, DeprecationWarning) + allow_env_proxies = trust_env and app is None and transport is None proxy_map = self._get_proxy_map(proxies or proxy, allow_env_proxies) @@ -695,7 +703,7 @@ class Client(BaseClient): app=app, trust_env=trust_env, ) - self._mounts: typing.Dict[URLPattern, typing.Optional[BaseTransport]] = { + self._mounts: dict[URLPattern, BaseTransport | None] = { URLPattern(key): None if proxy is None else self._init_proxy_transport( @@ -719,12 +727,12 @@ class Client(BaseClient): def _init_transport( self, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, limits: Limits = DEFAULT_LIMITS, - transport: typing.Optional[BaseTransport] = None, - app: typing.Optional[typing.Callable[..., typing.Any]] = None, + transport: BaseTransport | None = None, + app: typing.Callable[..., typing.Any] | None = None, trust_env: bool = True, ) -> BaseTransport: if transport is not None: @@ -746,7 +754,7 @@ class Client(BaseClient): self, proxy: Proxy, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, limits: Limits = DEFAULT_LIMITS, @@ -778,17 +786,17 @@ class Client(BaseClient): method: str, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Build and send a request. @@ -804,7 +812,7 @@ class Client(BaseClient): [Merging of configuration][0] for how the various parameters are merged with client-level configuration. - [0]: /advanced/#merging-of-configuration + [0]: /advanced/clients/#merging-of-configuration """ if cookies is not None: message = ( @@ -835,17 +843,17 @@ class Client(BaseClient): method: str, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> typing.Iterator[Response]: """ Alternative to `httpx.request()` that streams the response body @@ -886,8 +894,8 @@ class Client(BaseClient): request: Request, *, stream: bool = False, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, ) -> Response: """ Send a request. @@ -900,7 +908,7 @@ class Client(BaseClient): See also: [Request instances][0] - [0]: /advanced/#request-instances + [0]: /advanced/clients/#request-instances """ if self._state == ClientState.CLOSED: raise RuntimeError("Cannot send a request, as the client has been closed.") @@ -912,6 +920,8 @@ class Client(BaseClient): else follow_redirects ) + self._set_timeout(request) + auth = self._build_request_auth(request, auth) response = self._send_handling_auth( @@ -935,7 +945,7 @@ class Client(BaseClient): request: Request, auth: Auth, follow_redirects: bool, - history: typing.List[Response], + history: list[Response], ) -> Response: auth_flow = auth.sync_auth_flow(request) try: @@ -968,7 +978,7 @@ class Client(BaseClient): self, request: Request, follow_redirects: bool, - history: typing.List[Response], + history: list[Response], ) -> Response: while True: if len(history) > self.max_redirects: @@ -1041,13 +1051,13 @@ class Client(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `GET` request. @@ -1070,13 +1080,13 @@ class Client(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send an `OPTIONS` request. @@ -1099,13 +1109,13 @@ class Client(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `HEAD` request. @@ -1128,17 +1138,17 @@ class Client(BaseClient): self, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `POST` request. @@ -1165,17 +1175,17 @@ class Client(BaseClient): self, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `PUT` request. @@ -1202,17 +1212,17 @@ class Client(BaseClient): self, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `PATCH` request. @@ -1239,13 +1249,13 @@ class Client(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `DELETE` request. @@ -1296,9 +1306,9 @@ class Client(BaseClient): def __exit__( self, - exc_type: typing.Optional[typing.Type[BaseException]] = None, - exc_value: typing.Optional[BaseException] = None, - traceback: typing.Optional[TracebackType] = None, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: TracebackType | None = None, ) -> None: self._state = ClientState.CLOSED @@ -1366,31 +1376,27 @@ class AsyncClient(BaseClient): def __init__( self, *, - auth: typing.Optional[AuthTypes] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, + auth: AuthTypes | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, - proxy: typing.Optional[ProxyTypes] = None, - proxies: typing.Optional[ProxiesTypes] = None, - mounts: typing.Optional[ - typing.Mapping[str, typing.Optional[AsyncBaseTransport]] - ] = None, + proxy: ProxyTypes | None = None, + proxies: ProxiesTypes | None = None, + mounts: None | (typing.Mapping[str, AsyncBaseTransport | None]) = None, timeout: TimeoutTypes = DEFAULT_TIMEOUT_CONFIG, follow_redirects: bool = False, limits: Limits = DEFAULT_LIMITS, max_redirects: int = DEFAULT_MAX_REDIRECTS, - event_hooks: typing.Optional[ - typing.Mapping[str, typing.List[typing.Callable[..., typing.Any]]] - ] = None, + event_hooks: None | (typing.Mapping[str, list[EventHook]]) = None, base_url: URLTypes = "", - transport: typing.Optional[AsyncBaseTransport] = None, - app: typing.Optional[typing.Callable[..., typing.Any]] = None, + transport: AsyncBaseTransport | None = None, + app: typing.Callable[..., typing.Any] | None = None, trust_env: bool = True, - default_encoding: typing.Union[str, typing.Callable[[bytes], str]] = "utf-8", + default_encoding: str | typing.Callable[[bytes], str] = "utf-8", ) -> None: super().__init__( auth=auth, @@ -1424,6 +1430,13 @@ class AsyncClient(BaseClient): if proxy: raise RuntimeError("Use either `proxy` or 'proxies', not both.") + if app: + message = ( + "The 'app' shortcut is now deprecated." + " Use the explicit style 'transport=ASGITransport(app=...)' instead." + ) + warnings.warn(message, DeprecationWarning) + allow_env_proxies = trust_env and app is None and transport is None proxy_map = self._get_proxy_map(proxies or proxy, allow_env_proxies) @@ -1438,7 +1451,7 @@ class AsyncClient(BaseClient): trust_env=trust_env, ) - self._mounts: typing.Dict[URLPattern, typing.Optional[AsyncBaseTransport]] = { + self._mounts: dict[URLPattern, AsyncBaseTransport | None] = { URLPattern(key): None if proxy is None else self._init_proxy_transport( @@ -1461,12 +1474,12 @@ class AsyncClient(BaseClient): def _init_transport( self, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, limits: Limits = DEFAULT_LIMITS, - transport: typing.Optional[AsyncBaseTransport] = None, - app: typing.Optional[typing.Callable[..., typing.Any]] = None, + transport: AsyncBaseTransport | None = None, + app: typing.Callable[..., typing.Any] | None = None, trust_env: bool = True, ) -> AsyncBaseTransport: if transport is not None: @@ -1488,7 +1501,7 @@ class AsyncClient(BaseClient): self, proxy: Proxy, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, limits: Limits = DEFAULT_LIMITS, @@ -1520,17 +1533,17 @@ class AsyncClient(BaseClient): method: str, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Build and send a request. @@ -1546,7 +1559,7 @@ class AsyncClient(BaseClient): and [Merging of configuration][0] for how the various parameters are merged with client-level configuration. - [0]: /advanced/#merging-of-configuration + [0]: /advanced/clients/#merging-of-configuration """ if cookies is not None: # pragma: no cover @@ -1578,17 +1591,17 @@ class AsyncClient(BaseClient): method: str, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> typing.AsyncIterator[Response]: """ Alternative to `httpx.request()` that streams the response body @@ -1629,8 +1642,8 @@ class AsyncClient(BaseClient): request: Request, *, stream: bool = False, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, ) -> Response: """ Send a request. @@ -1643,7 +1656,7 @@ class AsyncClient(BaseClient): See also: [Request instances][0] - [0]: /advanced/#request-instances + [0]: /advanced/clients/#request-instances """ if self._state == ClientState.CLOSED: raise RuntimeError("Cannot send a request, as the client has been closed.") @@ -1655,6 +1668,8 @@ class AsyncClient(BaseClient): else follow_redirects ) + self._set_timeout(request) + auth = self._build_request_auth(request, auth) response = await self._send_handling_auth( @@ -1678,7 +1693,7 @@ class AsyncClient(BaseClient): request: Request, auth: Auth, follow_redirects: bool, - history: typing.List[Response], + history: list[Response], ) -> Response: auth_flow = auth.async_auth_flow(request) try: @@ -1711,7 +1726,7 @@ class AsyncClient(BaseClient): self, request: Request, follow_redirects: bool, - history: typing.List[Response], + history: list[Response], ) -> Response: while True: if len(history) > self.max_redirects: @@ -1784,13 +1799,13 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault, None] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `GET` request. @@ -1813,13 +1828,13 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send an `OPTIONS` request. @@ -1842,13 +1857,13 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `HEAD` request. @@ -1871,17 +1886,17 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `POST` request. @@ -1908,17 +1923,17 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `PUT` request. @@ -1945,17 +1960,17 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `PATCH` request. @@ -1982,13 +1997,13 @@ class AsyncClient(BaseClient): self, url: URLTypes, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - auth: typing.Union[AuthTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - follow_redirects: typing.Union[bool, UseClientDefault] = USE_CLIENT_DEFAULT, - timeout: typing.Union[TimeoutTypes, UseClientDefault] = USE_CLIENT_DEFAULT, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT, + follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT, + timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT, + extensions: RequestExtensions | None = None, ) -> Response: """ Send a `DELETE` request. @@ -2039,9 +2054,9 @@ class AsyncClient(BaseClient): async def __aexit__( self, - exc_type: typing.Optional[typing.Type[BaseException]] = None, - exc_value: typing.Optional[BaseException] = None, - traceback: typing.Optional[TracebackType] = None, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: TracebackType | None = None, ) -> None: self._state = ClientState.CLOSED diff --git a/httpx/_compat.py b/httpx/_compat.py index 493e6210..7d86dced 100644 --- a/httpx/_compat.py +++ b/httpx/_compat.py @@ -2,8 +2,12 @@ The _compat module is used for code which requires branching between different Python environments. It is excluded from the code coverage checks. """ + +import re import ssl import sys +from types import ModuleType +from typing import Optional # Brotli support is optional # The C bindings in `brotli` are recommended for CPython. @@ -16,6 +20,24 @@ except ImportError: # pragma: no cover except ImportError: brotli = None +# Zstandard support is optional +zstd: Optional[ModuleType] = None +try: + import zstandard as zstd +except (AttributeError, ImportError, ValueError): # Defensive: + zstd = None +else: + # The package 'zstandard' added the 'eof' property starting + # in v0.18.0 which we require to ensure a complete and + # valid zstd stream was fed into the ZstdDecoder. + # See: https://github.com/urllib3/urllib3/pull/2624 + _zstd_version = tuple( + map(int, re.search(r"^([0-9]+)\.([0-9]+)", zstd.__version__).groups()) # type: ignore[union-attr] + ) + if _zstd_version < (0, 18): # Defensive: + zstd = None + + if sys.version_info >= (3, 10) or ssl.OPENSSL_VERSION_INFO >= (1, 1, 0, 7): def set_minimum_tls_version_1_2(context: ssl.SSLContext) -> None: diff --git a/httpx/_config.py b/httpx/_config.py index 1af8a456..736cfdf3 100644 --- a/httpx/_config.py +++ b/httpx/_config.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import logging import os import ssl @@ -12,6 +14,8 @@ from ._types import CertTypes, HeaderTypes, TimeoutTypes, URLTypes, VerifyTypes from ._urls import URL from ._utils import get_ca_bundle_from_env +__all__ = ["Limits", "Proxy", "Timeout", "create_ssl_context"] + SOCKET_OPTION = typing.Union[ typing.Tuple[int, int, int], typing.Tuple[int, int, typing.Union[bytes, bytearray]], @@ -49,7 +53,7 @@ UNSET = UnsetType() def create_ssl_context( - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, trust_env: bool = True, http2: bool = False, @@ -69,7 +73,7 @@ class SSLConfig: def __init__( self, *, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, verify: VerifyTypes = True, trust_env: bool = True, http2: bool = False, @@ -211,12 +215,12 @@ class Timeout: def __init__( self, - timeout: typing.Union[TimeoutTypes, UnsetType] = UNSET, + timeout: TimeoutTypes | UnsetType = UNSET, *, - connect: typing.Union[None, float, UnsetType] = UNSET, - read: typing.Union[None, float, UnsetType] = UNSET, - write: typing.Union[None, float, UnsetType] = UNSET, - pool: typing.Union[None, float, UnsetType] = UNSET, + connect: None | float | UnsetType = UNSET, + read: None | float | UnsetType = UNSET, + write: None | float | UnsetType = UNSET, + pool: None | float | UnsetType = UNSET, ) -> None: if isinstance(timeout, Timeout): # Passed as a single explicit Timeout. @@ -255,7 +259,7 @@ class Timeout: self.write = timeout if isinstance(write, UnsetType) else write self.pool = timeout if isinstance(pool, UnsetType) else pool - def as_dict(self) -> typing.Dict[str, typing.Optional[float]]: + def as_dict(self) -> dict[str, float | None]: return { "connect": self.connect, "read": self.read, @@ -299,9 +303,9 @@ class Limits: def __init__( self, *, - max_connections: typing.Optional[int] = None, - max_keepalive_connections: typing.Optional[int] = None, - keepalive_expiry: typing.Optional[float] = 5.0, + max_connections: int | None = None, + max_keepalive_connections: int | None = None, + keepalive_expiry: float | None = 5.0, ) -> None: self.max_connections = max_connections self.max_keepalive_connections = max_keepalive_connections @@ -329,9 +333,9 @@ class Proxy: self, url: URLTypes, *, - ssl_context: typing.Optional[ssl.SSLContext] = None, - auth: typing.Optional[typing.Tuple[str, str]] = None, - headers: typing.Optional[HeaderTypes] = None, + ssl_context: ssl.SSLContext | None = None, + auth: tuple[str, str] | None = None, + headers: HeaderTypes | None = None, ) -> None: url = URL(url) headers = Headers(headers) @@ -350,7 +354,7 @@ class Proxy: self.ssl_context = ssl_context @property - def raw_auth(self) -> typing.Optional[typing.Tuple[bytes, bytes]]: + def raw_auth(self) -> tuple[bytes, bytes] | None: # The proxy authentication as raw bytes. return ( None diff --git a/httpx/_content.py b/httpx/_content.py index cd0d17f1..786699f3 100644 --- a/httpx/_content.py +++ b/httpx/_content.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import inspect import warnings from json import dumps as json_dumps @@ -5,13 +7,9 @@ from typing import ( Any, AsyncIterable, AsyncIterator, - Dict, Iterable, Iterator, Mapping, - Optional, - Tuple, - Union, ) from urllib.parse import urlencode @@ -27,6 +25,8 @@ from ._types import ( ) from ._utils import peek_filelike_length, primitive_value_to_str +__all__ = ["ByteStream"] + class ByteStream(AsyncByteStream, SyncByteStream): def __init__(self, stream: bytes) -> None: @@ -105,8 +105,8 @@ class UnattachedStream(AsyncByteStream, SyncByteStream): def encode_content( - content: Union[str, bytes, Iterable[bytes], AsyncIterable[bytes]], -) -> Tuple[Dict[str, str], Union[SyncByteStream, AsyncByteStream]]: + content: str | bytes | Iterable[bytes] | AsyncIterable[bytes], +) -> tuple[dict[str, str], SyncByteStream | AsyncByteStream]: if isinstance(content, (bytes, str)): body = content.encode("utf-8") if isinstance(content, str) else content content_length = len(body) @@ -135,7 +135,7 @@ def encode_content( def encode_urlencoded_data( data: RequestData, -) -> Tuple[Dict[str, str], ByteStream]: +) -> tuple[dict[str, str], ByteStream]: plain_data = [] for key, value in data.items(): if isinstance(value, (list, tuple)): @@ -150,14 +150,14 @@ def encode_urlencoded_data( def encode_multipart_data( - data: RequestData, files: RequestFiles, boundary: Optional[bytes] -) -> Tuple[Dict[str, str], MultipartStream]: + data: RequestData, files: RequestFiles, boundary: bytes | None +) -> tuple[dict[str, str], MultipartStream]: multipart = MultipartStream(data=data, files=files, boundary=boundary) headers = multipart.get_headers() return headers, multipart -def encode_text(text: str) -> Tuple[Dict[str, str], ByteStream]: +def encode_text(text: str) -> tuple[dict[str, str], ByteStream]: body = text.encode("utf-8") content_length = str(len(body)) content_type = "text/plain; charset=utf-8" @@ -165,7 +165,7 @@ def encode_text(text: str) -> Tuple[Dict[str, str], ByteStream]: return headers, ByteStream(body) -def encode_html(html: str) -> Tuple[Dict[str, str], ByteStream]: +def encode_html(html: str) -> tuple[dict[str, str], ByteStream]: body = html.encode("utf-8") content_length = str(len(body)) content_type = "text/html; charset=utf-8" @@ -173,7 +173,7 @@ def encode_html(html: str) -> Tuple[Dict[str, str], ByteStream]: return headers, ByteStream(body) -def encode_json(json: Any) -> Tuple[Dict[str, str], ByteStream]: +def encode_json(json: Any) -> tuple[dict[str, str], ByteStream]: body = json_dumps(json).encode("utf-8") content_length = str(len(body)) content_type = "application/json" @@ -182,12 +182,12 @@ def encode_json(json: Any) -> Tuple[Dict[str, str], ByteStream]: def encode_request( - content: Optional[RequestContent] = None, - data: Optional[RequestData] = None, - files: Optional[RequestFiles] = None, - json: Optional[Any] = None, - boundary: Optional[bytes] = None, -) -> Tuple[Dict[str, str], Union[SyncByteStream, AsyncByteStream]]: + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: Any | None = None, + boundary: bytes | None = None, +) -> tuple[dict[str, str], SyncByteStream | AsyncByteStream]: """ Handles encoding the given `content`, `data`, `files`, and `json`, returning a two-tuple of (, ). @@ -217,11 +217,11 @@ def encode_request( def encode_response( - content: Optional[ResponseContent] = None, - text: Optional[str] = None, - html: Optional[str] = None, - json: Optional[Any] = None, -) -> Tuple[Dict[str, str], Union[SyncByteStream, AsyncByteStream]]: + content: ResponseContent | None = None, + text: str | None = None, + html: str | None = None, + json: Any | None = None, +) -> tuple[dict[str, str], SyncByteStream | AsyncByteStream]: """ Handles encoding the given `content`, returning a two-tuple of (, ). diff --git a/httpx/_decoders.py b/httpx/_decoders.py index 3f507c8e..62f2c0b9 100644 --- a/httpx/_decoders.py +++ b/httpx/_decoders.py @@ -3,12 +3,15 @@ Handlers for Content-Encoding. See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding """ + +from __future__ import annotations + import codecs import io import typing import zlib -from ._compat import brotli +from ._compat import brotli, zstd from ._exceptions import DecodingError @@ -137,6 +140,44 @@ class BrotliDecoder(ContentDecoder): raise DecodingError(str(exc)) from exc +class ZStandardDecoder(ContentDecoder): + """ + Handle 'zstd' RFC 8878 decoding. + + Requires `pip install zstandard`. + Can be installed as a dependency of httpx using `pip install httpx[zstd]`. + """ + + # inspired by the ZstdDecoder implementation in urllib3 + def __init__(self) -> None: + if zstd is None: # pragma: no cover + raise ImportError( + "Using 'ZStandardDecoder', ..." + "Make sure to install httpx using `pip install httpx[zstd]`." + ) from None + + self.decompressor = zstd.ZstdDecompressor().decompressobj() + + def decode(self, data: bytes) -> bytes: + assert zstd is not None + output = io.BytesIO() + try: + output.write(self.decompressor.decompress(data)) + while self.decompressor.eof and self.decompressor.unused_data: + unused_data = self.decompressor.unused_data + self.decompressor = zstd.ZstdDecompressor().decompressobj() + output.write(self.decompressor.decompress(unused_data)) + except zstd.ZstdError as exc: + raise DecodingError(str(exc)) from exc + return output.getvalue() + + def flush(self) -> bytes: + ret = self.decompressor.flush() # note: this is a no-op + if not self.decompressor.eof: + raise DecodingError("Zstandard data is incomplete") # pragma: no cover + return bytes(ret) + + class MultiDecoder(ContentDecoder): """ Handle the case where multiple encodings have been applied. @@ -167,11 +208,11 @@ class ByteChunker: Handles returning byte content in fixed-size chunks. """ - def __init__(self, chunk_size: typing.Optional[int] = None) -> None: + def __init__(self, chunk_size: int | None = None) -> None: self._buffer = io.BytesIO() self._chunk_size = chunk_size - def decode(self, content: bytes) -> typing.List[bytes]: + def decode(self, content: bytes) -> list[bytes]: if self._chunk_size is None: return [content] if content else [] @@ -194,7 +235,7 @@ class ByteChunker: else: return [] - def flush(self) -> typing.List[bytes]: + def flush(self) -> list[bytes]: value = self._buffer.getvalue() self._buffer.seek(0) self._buffer.truncate() @@ -206,11 +247,11 @@ class TextChunker: Handles returning text content in fixed-size chunks. """ - def __init__(self, chunk_size: typing.Optional[int] = None) -> None: + def __init__(self, chunk_size: int | None = None) -> None: self._buffer = io.StringIO() self._chunk_size = chunk_size - def decode(self, content: str) -> typing.List[str]: + def decode(self, content: str) -> list[str]: if self._chunk_size is None: return [content] if content else [] @@ -233,7 +274,7 @@ class TextChunker: else: return [] - def flush(self) -> typing.List[str]: + def flush(self) -> list[str]: value = self._buffer.getvalue() self._buffer.seek(0) self._buffer.truncate() @@ -264,10 +305,10 @@ class LineDecoder: """ def __init__(self) -> None: - self.buffer: typing.List[str] = [] + self.buffer: list[str] = [] self.trailing_cr: bool = False - def decode(self, text: str) -> typing.List[str]: + def decode(self, text: str) -> list[str]: # See https://docs.python.org/3/library/stdtypes.html#str.splitlines NEWLINE_CHARS = "\n\r\x0b\x0c\x1c\x1d\x1e\x85\u2028\u2029" @@ -305,7 +346,7 @@ class LineDecoder: return lines - def flush(self) -> typing.List[str]: + def flush(self) -> list[str]: if not self.buffer and not self.trailing_cr: return [] @@ -320,8 +361,11 @@ SUPPORTED_DECODERS = { "gzip": GZipDecoder, "deflate": DeflateDecoder, "br": BrotliDecoder, + "zstd": ZStandardDecoder, } if brotli is None: SUPPORTED_DECODERS.pop("br") # pragma: no cover +if zstd is None: + SUPPORTED_DECODERS.pop("zstd") # pragma: no cover diff --git a/httpx/_exceptions.py b/httpx/_exceptions.py index 12369295..77f45a6d 100644 --- a/httpx/_exceptions.py +++ b/httpx/_exceptions.py @@ -30,12 +30,46 @@ Our exception hierarchy: x ResponseNotRead x RequestNotRead """ + +from __future__ import annotations + import contextlib import typing if typing.TYPE_CHECKING: from ._models import Request, Response # pragma: no cover +__all__ = [ + "CloseError", + "ConnectError", + "ConnectTimeout", + "CookieConflict", + "DecodingError", + "HTTPError", + "HTTPStatusError", + "InvalidURL", + "LocalProtocolError", + "NetworkError", + "PoolTimeout", + "ProtocolError", + "ProxyError", + "ReadError", + "ReadTimeout", + "RemoteProtocolError", + "RequestError", + "RequestNotRead", + "ResponseNotRead", + "StreamClosed", + "StreamConsumed", + "StreamError", + "TimeoutException", + "TooManyRedirects", + "TransportError", + "UnsupportedProtocol", + "WriteError", + "WriteTimeout", +] + class HTTPError(Exception): """ @@ -57,16 +91,16 @@ class HTTPError(Exception): def __init__(self, message: str) -> None: super().__init__(message) - self._request: typing.Optional["Request"] = None + self._request: Request | None = None @property - def request(self) -> "Request": + def request(self) -> Request: if self._request is None: raise RuntimeError("The .request property has not been set.") return self._request @request.setter - def request(self, request: "Request") -> None: + def request(self, request: Request) -> None: self._request = request @@ -75,9 +109,7 @@ class RequestError(HTTPError): Base class for all exceptions that may occur when issuing a `.request()`. """ - def __init__( - self, message: str, *, request: typing.Optional["Request"] = None - ) -> None: + def __init__(self, message: str, *, request: Request | None = None) -> None: super().__init__(message) # At the point an exception is raised we won't typically have a request # instance to associate it with. @@ -230,9 +262,7 @@ class HTTPStatusError(HTTPError): May be raised when calling `response.raise_for_status()` """ - def __init__( - self, message: str, *, request: "Request", response: "Response" - ) -> None: + def __init__(self, message: str, *, request: Request, response: Response) -> None: super().__init__(message) self.request = request self.response = response @@ -335,7 +365,7 @@ class RequestNotRead(StreamError): @contextlib.contextmanager def request_context( - request: typing.Optional["Request"] = None, + request: Request | None = None, ) -> typing.Iterator[None]: """ A context manager that can be used to attach the given request context diff --git a/httpx/_main.py b/httpx/_main.py index adb57d5f..72657f8c 100644 --- a/httpx/_main.py +++ b/httpx/_main.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import functools import json import sys @@ -125,8 +127,8 @@ def format_request_headers(request: httpcore.Request, http2: bool = False) -> st def format_response_headers( http_version: bytes, status: int, - reason_phrase: typing.Optional[bytes], - headers: typing.List[typing.Tuple[bytes, bytes]], + reason_phrase: bytes | None, + headers: list[tuple[bytes, bytes]], ) -> str: version = http_version.decode("ascii") reason = ( @@ -152,8 +154,8 @@ def print_request_headers(request: httpcore.Request, http2: bool = False) -> Non def print_response_headers( http_version: bytes, status: int, - reason_phrase: typing.Optional[bytes], - headers: typing.List[typing.Tuple[bytes, bytes]], + reason_phrase: bytes | None, + headers: list[tuple[bytes, bytes]], ) -> None: console = rich.console.Console() http_text = format_response_headers(http_version, status, reason_phrase, headers) @@ -268,7 +270,7 @@ def download_response(response: Response, download: typing.BinaryIO) -> None: def validate_json( ctx: click.Context, - param: typing.Union[click.Option, click.Parameter], + param: click.Option | click.Parameter, value: typing.Any, ) -> typing.Any: if value is None: @@ -282,7 +284,7 @@ def validate_json( def validate_auth( ctx: click.Context, - param: typing.Union[click.Option, click.Parameter], + param: click.Option | click.Parameter, value: typing.Any, ) -> typing.Any: if value == (None, None): @@ -296,7 +298,7 @@ def validate_auth( def handle_help( ctx: click.Context, - param: typing.Union[click.Option, click.Parameter], + param: click.Option | click.Parameter, value: typing.Any, ) -> None: if not value or ctx.resilient_parsing: @@ -448,20 +450,20 @@ def handle_help( def main( url: str, method: str, - params: typing.List[typing.Tuple[str, str]], + params: list[tuple[str, str]], content: str, - data: typing.List[typing.Tuple[str, str]], - files: typing.List[typing.Tuple[str, click.File]], + data: list[tuple[str, str]], + files: list[tuple[str, click.File]], json: str, - headers: typing.List[typing.Tuple[str, str]], - cookies: typing.List[typing.Tuple[str, str]], - auth: typing.Optional[typing.Tuple[str, str]], + headers: list[tuple[str, str]], + cookies: list[tuple[str, str]], + auth: tuple[str, str] | None, proxy: str, timeout: float, follow_redirects: bool, verify: bool, http2: bool, - download: typing.Optional[typing.BinaryIO], + download: typing.BinaryIO | None, verbose: bool, ) -> None: """ diff --git a/httpx/_models.py b/httpx/_models.py index e08248a8..01d9583b 100644 --- a/httpx/_models.py +++ b/httpx/_models.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import datetime import email.message import json as jsonlib @@ -51,6 +53,8 @@ from ._utils import ( parse_header_links, ) +__all__ = ["Cookies", "Headers", "Request", "Response"] + class Headers(typing.MutableMapping[str, str]): """ @@ -59,8 +63,8 @@ class Headers(typing.MutableMapping[str, str]): def __init__( self, - headers: typing.Optional[HeaderTypes] = None, - encoding: typing.Optional[str] = None, + headers: HeaderTypes | None = None, + encoding: str | None = None, ) -> None: if headers is None: self._list = [] # type: typing.List[typing.Tuple[bytes, bytes, bytes]] @@ -117,7 +121,7 @@ class Headers(typing.MutableMapping[str, str]): self._encoding = value @property - def raw(self) -> typing.List[typing.Tuple[bytes, bytes]]: + def raw(self) -> list[tuple[bytes, bytes]]: """ Returns a list of the raw header items, as byte pairs. """ @@ -127,7 +131,7 @@ class Headers(typing.MutableMapping[str, str]): return {key.decode(self.encoding): None for _, key, value in self._list}.keys() def values(self) -> typing.ValuesView[str]: - values_dict: typing.Dict[str, str] = {} + values_dict: dict[str, str] = {} for _, key, value in self._list: str_key = key.decode(self.encoding) str_value = value.decode(self.encoding) @@ -142,7 +146,7 @@ class Headers(typing.MutableMapping[str, str]): Return `(key, value)` items of headers. Concatenate headers into a single comma separated value when a key occurs multiple times. """ - values_dict: typing.Dict[str, str] = {} + values_dict: dict[str, str] = {} for _, key, value in self._list: str_key = key.decode(self.encoding) str_value = value.decode(self.encoding) @@ -152,7 +156,7 @@ class Headers(typing.MutableMapping[str, str]): values_dict[str_key] = str_value return values_dict.items() - def multi_items(self) -> typing.List[typing.Tuple[str, str]]: + def multi_items(self) -> list[tuple[str, str]]: """ Return a list of `(key, value)` pairs of headers. Allow multiple occurrences of the same key without concatenating into a single @@ -173,7 +177,7 @@ class Headers(typing.MutableMapping[str, str]): except KeyError: return default - def get_list(self, key: str, split_commas: bool = False) -> typing.List[str]: + def get_list(self, key: str, split_commas: bool = False) -> list[str]: """ Return a list of all header values for a given key. If `split_commas=True` is passed, then any comma separated header @@ -195,14 +199,14 @@ class Headers(typing.MutableMapping[str, str]): split_values.extend([item.strip() for item in value.split(",")]) return split_values - def update(self, headers: typing.Optional[HeaderTypes] = None) -> None: # type: ignore + def update(self, headers: HeaderTypes | None = None) -> None: # type: ignore headers = Headers(headers) for key in headers.keys(): if key in self: self.pop(key) self._list.extend(headers._list) - def copy(self) -> "Headers": + def copy(self) -> Headers: return Headers(self, encoding=self.encoding) def __getitem__(self, key: str) -> str: @@ -306,18 +310,18 @@ class Headers(typing.MutableMapping[str, str]): class Request: def __init__( self, - method: typing.Union[str, bytes], - url: typing.Union["URL", str], + method: str | bytes, + url: URL | str, *, - params: typing.Optional[QueryParamTypes] = None, - headers: typing.Optional[HeaderTypes] = None, - cookies: typing.Optional[CookieTypes] = None, - content: typing.Optional[RequestContent] = None, - data: typing.Optional[RequestData] = None, - files: typing.Optional[RequestFiles] = None, - json: typing.Optional[typing.Any] = None, - stream: typing.Union[SyncByteStream, AsyncByteStream, None] = None, - extensions: typing.Optional[RequestExtensions] = None, + params: QueryParamTypes | None = None, + headers: HeaderTypes | None = None, + cookies: CookieTypes | None = None, + content: RequestContent | None = None, + data: RequestData | None = None, + files: RequestFiles | None = None, + json: typing.Any | None = None, + stream: SyncByteStream | AsyncByteStream | None = None, + extensions: RequestExtensions | None = None, ) -> None: self.method = ( method.decode("ascii").upper() @@ -334,7 +338,7 @@ class Request: Cookies(cookies).set_cookie_header(self) if stream is None: - content_type: typing.Optional[str] = self.headers.get("content-type") + content_type: str | None = self.headers.get("content-type") headers, stream = encode_request( content=content, data=data, @@ -368,14 +372,14 @@ class Request: # * Creating request instances on the *server-side* of the transport API. self.stream = stream - def _prepare(self, default_headers: typing.Dict[str, str]) -> None: + def _prepare(self, default_headers: dict[str, str]) -> None: for key, value in default_headers.items(): # Ignore Transfer-Encoding if the Content-Length has been set explicitly. if key.lower() == "transfer-encoding" and "Content-Length" in self.headers: continue self.headers.setdefault(key, value) - auto_headers: typing.List[typing.Tuple[bytes, bytes]] = [] + auto_headers: list[tuple[bytes, bytes]] = [] has_host = "Host" in self.headers has_content_length = ( @@ -428,14 +432,14 @@ class Request: url = str(self.url) return f"<{class_name}({self.method!r}, {url!r})>" - def __getstate__(self) -> typing.Dict[str, typing.Any]: + def __getstate__(self) -> dict[str, typing.Any]: return { name: value for name, value in self.__dict__.items() if name not in ["extensions", "stream"] } - def __setstate__(self, state: typing.Dict[str, typing.Any]) -> None: + def __setstate__(self, state: dict[str, typing.Any]) -> None: for name, value in state.items(): setattr(self, name, value) self.extensions = {} @@ -447,25 +451,25 @@ class Response: self, status_code: int, *, - headers: typing.Optional[HeaderTypes] = None, - content: typing.Optional[ResponseContent] = None, - text: typing.Optional[str] = None, - html: typing.Optional[str] = None, + headers: HeaderTypes | None = None, + content: ResponseContent | None = None, + text: str | None = None, + html: str | None = None, json: typing.Any = None, - stream: typing.Union[SyncByteStream, AsyncByteStream, None] = None, - request: typing.Optional[Request] = None, - extensions: typing.Optional[ResponseExtensions] = None, - history: typing.Optional[typing.List["Response"]] = None, - default_encoding: typing.Union[str, typing.Callable[[bytes], str]] = "utf-8", + stream: SyncByteStream | AsyncByteStream | None = None, + request: Request | None = None, + extensions: ResponseExtensions | None = None, + history: list[Response] | None = None, + default_encoding: str | typing.Callable[[bytes], str] = "utf-8", ) -> None: self.status_code = status_code self.headers = Headers(headers) - self._request: typing.Optional[Request] = request + self._request: Request | None = request # When follow_redirects=False and a redirect is received, # the client will set `response.next_request`. - self.next_request: typing.Optional[Request] = None + self.next_request: Request | None = None self.extensions: ResponseExtensions = {} if extensions is None else extensions self.history = [] if history is None else list(history) @@ -498,7 +502,7 @@ class Response: self._num_bytes_downloaded = 0 - def _prepare(self, default_headers: typing.Dict[str, str]) -> None: + def _prepare(self, default_headers: dict[str, str]) -> None: for key, value in default_headers.items(): # Ignore Transfer-Encoding if the Content-Length has been set explicitly. if key.lower() == "transfer-encoding" and "content-length" in self.headers: @@ -580,7 +584,7 @@ class Response: return self._text @property - def encoding(self) -> typing.Optional[str]: + def encoding(self) -> str | None: """ Return an encoding to use for decoding the byte content into text. The priority for determining this is given by... @@ -616,7 +620,7 @@ class Response: self._encoding = value @property - def charset_encoding(self) -> typing.Optional[str]: + def charset_encoding(self) -> str | None: """ Return the encoding, as specified by the Content-Type header. """ @@ -632,7 +636,7 @@ class Response: content, depending on the Content-Encoding used in the response. """ if not hasattr(self, "_decoder"): - decoders: typing.List[ContentDecoder] = [] + decoders: list[ContentDecoder] = [] values = self.headers.get_list("content-encoding", split_commas=True) for value in values: value = value.strip().lower() @@ -721,7 +725,7 @@ class Response: and "Location" in self.headers ) - def raise_for_status(self) -> "Response": + def raise_for_status(self) -> Response: """ Raise the `HTTPStatusError` if one occurred. """ @@ -762,14 +766,14 @@ class Response: return jsonlib.loads(self.content, **kwargs) @property - def cookies(self) -> "Cookies": + def cookies(self) -> Cookies: if not hasattr(self, "_cookies"): self._cookies = Cookies() self._cookies.extract_cookies(self) return self._cookies @property - def links(self) -> typing.Dict[typing.Optional[str], typing.Dict[str, str]]: + def links(self) -> dict[str | None, dict[str, str]]: """ Returns the parsed header links of the response, if any """ @@ -789,14 +793,14 @@ class Response: def __repr__(self) -> str: return f"" - def __getstate__(self) -> typing.Dict[str, typing.Any]: + def __getstate__(self) -> dict[str, typing.Any]: return { name: value for name, value in self.__dict__.items() if name not in ["extensions", "stream", "is_closed", "_decoder"] } - def __setstate__(self, state: typing.Dict[str, typing.Any]) -> None: + def __setstate__(self, state: dict[str, typing.Any]) -> None: for name, value in state.items(): setattr(self, name, value) self.is_closed = True @@ -811,12 +815,10 @@ class Response: self._content = b"".join(self.iter_bytes()) return self._content - def iter_bytes( - self, chunk_size: typing.Optional[int] = None - ) -> typing.Iterator[bytes]: + def iter_bytes(self, chunk_size: int | None = None) -> typing.Iterator[bytes]: """ A byte-iterator over the decoded response content. - This allows us to handle gzip, deflate, and brotli encoded responses. + This allows us to handle gzip, deflate, brotli, and zstd encoded responses. """ if hasattr(self, "_content"): chunk_size = len(self._content) if chunk_size is None else chunk_size @@ -836,9 +838,7 @@ class Response: for chunk in chunker.flush(): yield chunk - def iter_text( - self, chunk_size: typing.Optional[int] = None - ) -> typing.Iterator[str]: + def iter_text(self, chunk_size: int | None = None) -> typing.Iterator[str]: """ A str-iterator over the decoded response content that handles both gzip, deflate, etc but also detects the content's @@ -866,9 +866,7 @@ class Response: for line in decoder.flush(): yield line - def iter_raw( - self, chunk_size: typing.Optional[int] = None - ) -> typing.Iterator[bytes]: + def iter_raw(self, chunk_size: int | None = None) -> typing.Iterator[bytes]: """ A byte-iterator over the raw response content. """ @@ -916,11 +914,11 @@ class Response: return self._content async def aiter_bytes( - self, chunk_size: typing.Optional[int] = None + self, chunk_size: int | None = None ) -> typing.AsyncIterator[bytes]: """ A byte-iterator over the decoded response content. - This allows us to handle gzip, deflate, and brotli encoded responses. + This allows us to handle gzip, deflate, brotli, and zstd encoded responses. """ if hasattr(self, "_content"): chunk_size = len(self._content) if chunk_size is None else chunk_size @@ -941,7 +939,7 @@ class Response: yield chunk async def aiter_text( - self, chunk_size: typing.Optional[int] = None + self, chunk_size: int | None = None ) -> typing.AsyncIterator[str]: """ A str-iterator over the decoded response content @@ -971,7 +969,7 @@ class Response: yield line async def aiter_raw( - self, chunk_size: typing.Optional[int] = None + self, chunk_size: int | None = None ) -> typing.AsyncIterator[bytes]: """ A byte-iterator over the raw response content. @@ -1017,7 +1015,7 @@ class Cookies(typing.MutableMapping[str, str]): HTTP Cookies, as a mutable mapping. """ - def __init__(self, cookies: typing.Optional[CookieTypes] = None) -> None: + def __init__(self, cookies: CookieTypes | None = None) -> None: if cookies is None or isinstance(cookies, dict): self.jar = CookieJar() if isinstance(cookies, dict): @@ -1079,10 +1077,10 @@ class Cookies(typing.MutableMapping[str, str]): def get( # type: ignore self, name: str, - default: typing.Optional[str] = None, - domain: typing.Optional[str] = None, - path: typing.Optional[str] = None, - ) -> typing.Optional[str]: + default: str | None = None, + domain: str | None = None, + path: str | None = None, + ) -> str | None: """ Get a cookie by name. May optionally include domain and path in order to specify exactly which cookie to retrieve. @@ -1104,8 +1102,8 @@ class Cookies(typing.MutableMapping[str, str]): def delete( self, name: str, - domain: typing.Optional[str] = None, - path: typing.Optional[str] = None, + domain: str | None = None, + path: str | None = None, ) -> None: """ Delete a cookie by name. May optionally include domain and path @@ -1125,9 +1123,7 @@ class Cookies(typing.MutableMapping[str, str]): for cookie in remove: self.jar.clear(cookie.domain, cookie.path, cookie.name) - def clear( - self, domain: typing.Optional[str] = None, path: typing.Optional[str] = None - ) -> None: + def clear(self, domain: str | None = None, path: str | None = None) -> None: """ Delete all cookies. Optionally include a domain and path in order to only delete a subset of all the cookies. @@ -1140,7 +1136,7 @@ class Cookies(typing.MutableMapping[str, str]): args.append(path) self.jar.clear(*args) - def update(self, cookies: typing.Optional[CookieTypes] = None) -> None: # type: ignore + def update(self, cookies: CookieTypes | None = None) -> None: # type: ignore cookies = Cookies(cookies) for cookie in cookies.jar: self.jar.set_cookie(cookie) diff --git a/httpx/_multipart.py b/httpx/_multipart.py index 1d451c38..8edb6227 100644 --- a/httpx/_multipart.py +++ b/httpx/_multipart.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import io import os import typing @@ -21,8 +23,8 @@ from ._utils import ( def get_multipart_boundary_from_content_type( - content_type: typing.Optional[bytes], -) -> typing.Optional[bytes]: + content_type: bytes | None, +) -> bytes | None: if not content_type or not content_type.startswith(b"multipart/form-data"): return None # parse boundary according to @@ -39,9 +41,7 @@ class DataField: A single form field item, within a multipart form field. """ - def __init__( - self, name: str, value: typing.Union[str, bytes, int, float, None] - ) -> None: + def __init__(self, name: str, value: str | bytes | int | float | None) -> None: if not isinstance(name, str): raise TypeError( f"Invalid type for name. Expected str, got {type(name)}: {name!r}" @@ -52,7 +52,7 @@ class DataField: f" got {type(value)}: {value!r}" ) self.name = name - self.value: typing.Union[str, bytes] = ( + self.value: str | bytes = ( value if isinstance(value, bytes) else primitive_value_to_str(value) ) @@ -93,8 +93,8 @@ class FileField: fileobj: FileContent - headers: typing.Dict[str, str] = {} - content_type: typing.Optional[str] = None + headers: dict[str, str] = {} + content_type: str | None = None # This large tuple based API largely mirror's requests' API # It would be good to think of better APIs for this that we could @@ -137,7 +137,7 @@ class FileField: self.file = fileobj self.headers = headers - def get_length(self) -> typing.Optional[int]: + def get_length(self) -> int | None: headers = self.render_headers() if isinstance(self.file, (str, bytes)): @@ -199,7 +199,7 @@ class MultipartStream(SyncByteStream, AsyncByteStream): self, data: RequestData, files: RequestFiles, - boundary: typing.Optional[bytes] = None, + boundary: bytes | None = None, ) -> None: if boundary is None: boundary = os.urandom(16).hex().encode("ascii") @@ -212,7 +212,7 @@ class MultipartStream(SyncByteStream, AsyncByteStream): def _iter_fields( self, data: RequestData, files: RequestFiles - ) -> typing.Iterator[typing.Union[FileField, DataField]]: + ) -> typing.Iterator[FileField | DataField]: for name, value in data.items(): if isinstance(value, (tuple, list)): for item in value: @@ -231,7 +231,7 @@ class MultipartStream(SyncByteStream, AsyncByteStream): yield b"\r\n" yield b"--%s--\r\n" % self.boundary - def get_content_length(self) -> typing.Optional[int]: + def get_content_length(self) -> int | None: """ Return the length of the multipart encoded content, or `None` if any of the files have a length that cannot be determined upfront. @@ -253,7 +253,7 @@ class MultipartStream(SyncByteStream, AsyncByteStream): # Content stream interface. - def get_headers(self) -> typing.Dict[str, str]: + def get_headers(self) -> dict[str, str]: content_length = self.get_content_length() content_type = self.content_type if content_length is None: diff --git a/httpx/_status_codes.py b/httpx/_status_codes.py index 671c30e1..133a6231 100644 --- a/httpx/_status_codes.py +++ b/httpx/_status_codes.py @@ -1,5 +1,9 @@ +from __future__ import annotations + from enum import IntEnum +__all__ = ["codes"] + class codes(IntEnum): """HTTP status codes and reason phrases @@ -21,7 +25,7 @@ class codes(IntEnum): * RFC 8470: Using Early Data in HTTP """ - def __new__(cls, value: int, phrase: str = "") -> "codes": + def __new__(cls, value: int, phrase: str = "") -> codes: obj = int.__new__(cls, value) obj._value_ = value diff --git a/httpx/_transports/__init__.py b/httpx/_transports/__init__.py index e69de29b..7a321053 100644 --- a/httpx/_transports/__init__.py +++ b/httpx/_transports/__init__.py @@ -0,0 +1,15 @@ +from .asgi import * +from .base import * +from .default import * +from .mock import * +from .wsgi import * + +__all__ = [ + "ASGITransport", + "AsyncBaseTransport", + "BaseTransport", + "AsyncHTTPTransport", + "HTTPTransport", + "MockTransport", + "WSGITransport", +] diff --git a/httpx/_transports/asgi.py b/httpx/_transports/asgi.py index 08cd392f..8578d4ae 100644 --- a/httpx/_transports/asgi.py +++ b/httpx/_transports/asgi.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing import sniffio @@ -14,17 +16,19 @@ if typing.TYPE_CHECKING: # pragma: no cover Event = typing.Union[asyncio.Event, trio.Event] -_Message = typing.Dict[str, typing.Any] +_Message = typing.MutableMapping[str, typing.Any] _Receive = typing.Callable[[], typing.Awaitable[_Message]] _Send = typing.Callable[ - [typing.Dict[str, typing.Any]], typing.Coroutine[None, None, None] + [typing.MutableMapping[str, typing.Any]], typing.Awaitable[None] ] _ASGIApp = typing.Callable[ - [typing.Dict[str, typing.Any], _Receive, _Send], typing.Coroutine[None, None, None] + [typing.MutableMapping[str, typing.Any], _Receive, _Send], typing.Awaitable[None] ] +__all__ = ["ASGITransport"] -def create_event() -> "Event": + +def create_event() -> Event: if sniffio.current_async_library() == "trio": import trio @@ -36,7 +40,7 @@ def create_event() -> "Event": class ASGIResponseStream(AsyncByteStream): - def __init__(self, body: typing.List[bytes]) -> None: + def __init__(self, body: list[bytes]) -> None: self._body = body async def __aiter__(self) -> typing.AsyncIterator[bytes]: @@ -46,17 +50,8 @@ class ASGIResponseStream(AsyncByteStream): class ASGITransport(AsyncBaseTransport): """ A custom AsyncTransport that handles sending requests directly to an ASGI app. - The simplest way to use this functionality is to use the `app` argument. - ``` - client = httpx.AsyncClient(app=app) - ``` - - Alternatively, you can setup the transport instance explicitly. - This allows you to include any additional configuration arguments specific - to the ASGITransport class: - - ``` + ```python transport = httpx.ASGITransport( app=app, root_path="/submount", @@ -81,7 +76,7 @@ class ASGITransport(AsyncBaseTransport): app: _ASGIApp, raise_app_exceptions: bool = True, root_path: str = "", - client: typing.Tuple[str, int] = ("127.0.0.1", 123), + client: tuple[str, int] = ("127.0.0.1", 123), ) -> None: self.app = app self.raise_app_exceptions = raise_app_exceptions @@ -123,7 +118,7 @@ class ASGITransport(AsyncBaseTransport): # ASGI callables. - async def receive() -> typing.Dict[str, typing.Any]: + async def receive() -> dict[str, typing.Any]: nonlocal request_complete if request_complete: @@ -137,7 +132,7 @@ class ASGITransport(AsyncBaseTransport): return {"type": "http.request", "body": b"", "more_body": False} return {"type": "http.request", "body": body, "more_body": True} - async def send(message: typing.Dict[str, typing.Any]) -> None: + async def send(message: typing.MutableMapping[str, typing.Any]) -> None: nonlocal status_code, response_headers, response_started if message["type"] == "http.response.start": diff --git a/httpx/_transports/base.py b/httpx/_transports/base.py index f6fdfe69..66fd99d7 100644 --- a/httpx/_transports/base.py +++ b/httpx/_transports/base.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing from types import TracebackType @@ -6,6 +8,8 @@ from .._models import Request, Response T = typing.TypeVar("T", bound="BaseTransport") A = typing.TypeVar("A", bound="AsyncBaseTransport") +__all__ = ["AsyncBaseTransport", "BaseTransport"] + class BaseTransport: def __enter__(self: T) -> T: @@ -13,9 +17,9 @@ class BaseTransport: def __exit__( self, - exc_type: typing.Optional[typing.Type[BaseException]] = None, - exc_value: typing.Optional[BaseException] = None, - traceback: typing.Optional[TracebackType] = None, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: TracebackType | None = None, ) -> None: self.close() @@ -64,9 +68,9 @@ class AsyncBaseTransport: async def __aexit__( self, - exc_type: typing.Optional[typing.Type[BaseException]] = None, - exc_value: typing.Optional[BaseException] = None, - traceback: typing.Optional[TracebackType] = None, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: TracebackType | None = None, ) -> None: await self.aclose() diff --git a/httpx/_transports/default.py b/httpx/_transports/default.py index 0829b570..1896059e 100644 --- a/httpx/_transports/default.py +++ b/httpx/_transports/default.py @@ -23,6 +23,9 @@ client = httpx.Client(transport=transport) transport = httpx.HTTPTransport(uds="socket.uds") client = httpx.Client(transport=transport) """ + +from __future__ import annotations + import contextlib import typing from types import TracebackType @@ -62,6 +65,16 @@ T = typing.TypeVar("T", bound="HTTPTransport") A = typing.TypeVar("A", bound="AsyncHTTPTransport") +__all__ = ["AsyncHTTPTransport", "HTTPTransport"] + + +SOCKET_OPTION = typing.Union[ + typing.Tuple[int, int, int], + typing.Tuple[int, int, typing.Union[bytes, bytearray]], + typing.Tuple[int, int, None, int], +] + + @contextlib.contextmanager def map_httpcore_exceptions() -> typing.Iterator[None]: try: @@ -121,7 +134,7 @@ class HTTPTransport(BaseTransport): def __init__( self, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, limits: Limits = DEFAULT_LIMITS, @@ -203,9 +216,9 @@ class HTTPTransport(BaseTransport): def __exit__( self, - exc_type: typing.Optional[typing.Type[BaseException]] = None, - exc_value: typing.Optional[BaseException] = None, - traceback: typing.Optional[TracebackType] = None, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: TracebackType | None = None, ) -> None: with map_httpcore_exceptions(): self._pool.__exit__(exc_type, exc_value, traceback) @@ -262,7 +275,7 @@ class AsyncHTTPTransport(AsyncBaseTransport): def __init__( self, verify: VerifyTypes = True, - cert: typing.Optional[CertTypes] = None, + cert: CertTypes | None = None, http1: bool = True, http2: bool = False, limits: Limits = DEFAULT_LIMITS, @@ -296,6 +309,7 @@ class AsyncHTTPTransport(AsyncBaseTransport): ), proxy_auth=proxy.raw_auth, proxy_headers=proxy.headers.raw, + proxy_ssl_context=proxy.ssl_context, ssl_context=ssl_context, max_connections=limits.max_connections, max_keepalive_connections=limits.max_keepalive_connections, @@ -343,9 +357,9 @@ class AsyncHTTPTransport(AsyncBaseTransport): async def __aexit__( self, - exc_type: typing.Optional[typing.Type[BaseException]] = None, - exc_value: typing.Optional[BaseException] = None, - traceback: typing.Optional[TracebackType] = None, + exc_type: type[BaseException] | None = None, + exc_value: BaseException | None = None, + traceback: TracebackType | None = None, ) -> None: with map_httpcore_exceptions(): await self._pool.__aexit__(exc_type, exc_value, traceback) diff --git a/httpx/_transports/mock.py b/httpx/_transports/mock.py index 82043da2..8c418f59 100644 --- a/httpx/_transports/mock.py +++ b/httpx/_transports/mock.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing from .._models import Request, Response @@ -7,8 +9,11 @@ SyncHandler = typing.Callable[[Request], Response] AsyncHandler = typing.Callable[[Request], typing.Coroutine[None, None, Response]] +__all__ = ["MockTransport"] + + class MockTransport(AsyncBaseTransport, BaseTransport): - def __init__(self, handler: typing.Union[SyncHandler, AsyncHandler]) -> None: + def __init__(self, handler: SyncHandler | AsyncHandler) -> None: self.handler = handler def handle_request( diff --git a/httpx/_transports/wsgi.py b/httpx/_transports/wsgi.py index a23d42c4..8592ffe0 100644 --- a/httpx/_transports/wsgi.py +++ b/httpx/_transports/wsgi.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import io import itertools import sys @@ -14,6 +16,9 @@ if typing.TYPE_CHECKING: _T = typing.TypeVar("_T") +__all__ = ["WSGITransport"] + + def _skip_leading_empty_chunks(body: typing.Iterable[_T]) -> typing.Iterable[_T]: body = iter(body) for chunk in body: @@ -71,11 +76,11 @@ class WSGITransport(BaseTransport): def __init__( self, - app: "WSGIApplication", + app: WSGIApplication, raise_app_exceptions: bool = True, script_name: str = "", remote_addr: str = "127.0.0.1", - wsgi_errors: typing.Optional[typing.TextIO] = None, + wsgi_errors: typing.TextIO | None = None, ) -> None: self.app = app self.raise_app_exceptions = raise_app_exceptions @@ -117,8 +122,8 @@ class WSGITransport(BaseTransport): def start_response( status: str, - response_headers: typing.List[typing.Tuple[str, str]], - exc_info: typing.Optional["OptExcInfo"] = None, + response_headers: list[tuple[str, str]], + exc_info: OptExcInfo | None = None, ) -> typing.Callable[[bytes], typing.Any]: nonlocal seen_status, seen_response_headers, seen_exc_info seen_status = status diff --git a/httpx/_types.py b/httpx/_types.py index 649d101d..b7b0518c 100644 --- a/httpx/_types.py +++ b/httpx/_types.py @@ -108,6 +108,8 @@ RequestFiles = Union[Mapping[str, FileTypes], Sequence[Tuple[str, FileTypes]]] RequestExtensions = MutableMapping[str, Any] +__all__ = ["AsyncByteStream", "SyncByteStream"] + class SyncByteStream: def __iter__(self) -> Iterator[bytes]: diff --git a/httpx/_urlparse.py b/httpx/_urlparse.py index 07bbea90..575e6d56 100644 --- a/httpx/_urlparse.py +++ b/httpx/_urlparse.py @@ -15,6 +15,9 @@ Previously we relied on the excellent `rfc3986` package to handle URL parsing an validation, but this module provides a simpler alternative, with less indirection required. """ + +from __future__ import annotations + import ipaddress import re import typing @@ -95,10 +98,10 @@ class ParseResult(typing.NamedTuple): scheme: str userinfo: str host: str - port: typing.Optional[int] + port: int | None path: str - query: typing.Optional[str] - fragment: typing.Optional[str] + query: str | None + fragment: str | None @property def authority(self) -> str: @@ -119,7 +122,7 @@ class ParseResult(typing.NamedTuple): ] ) - def copy_with(self, **kwargs: typing.Optional[str]) -> "ParseResult": + def copy_with(self, **kwargs: str | None) -> ParseResult: if not kwargs: return self @@ -146,7 +149,7 @@ class ParseResult(typing.NamedTuple): ) -def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult: +def urlparse(url: str = "", **kwargs: str | None) -> ParseResult: # Initial basic checks on allowable URLs. # --------------------------------------- @@ -243,7 +246,7 @@ def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult: parsed_scheme: str = scheme.lower() parsed_userinfo: str = quote(userinfo, safe=SUB_DELIMS + ":") parsed_host: str = encode_host(host) - parsed_port: typing.Optional[int] = normalize_port(port, scheme) + parsed_port: int | None = normalize_port(port, scheme) has_scheme = parsed_scheme != "" has_authority = ( @@ -260,11 +263,11 @@ def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult: # For 'path' we need to drop ? and # from the GEN_DELIMS set. parsed_path: str = quote(path, safe=SUB_DELIMS + ":/[]@") # For 'query' we need to drop '#' from the GEN_DELIMS set. - parsed_query: typing.Optional[str] = ( + parsed_query: str | None = ( None if query is None else quote(query, safe=SUB_DELIMS + ":/?[]@") ) # For 'fragment' we can include all of the GEN_DELIMS set. - parsed_fragment: typing.Optional[str] = ( + parsed_fragment: str | None = ( None if fragment is None else quote(fragment, safe=SUB_DELIMS + ":/?#[]@") ) @@ -327,9 +330,7 @@ def encode_host(host: str) -> str: raise InvalidURL(f"Invalid IDNA hostname: {host!r}") -def normalize_port( - port: typing.Optional[typing.Union[str, int]], scheme: str -) -> typing.Optional[int]: +def normalize_port(port: str | int | None, scheme: str) -> int | None: # From https://tools.ietf.org/html/rfc3986#section-3.2.3 # # "A scheme may define a default port. For example, the "http" scheme @@ -391,9 +392,18 @@ def normalize_path(path: str) -> str: normalize_path("/path/./to/somewhere/..") == "/path/to" """ - # https://datatracker.ietf.org/doc/html/rfc3986#section-5.2.4 + # Fast return when no '.' characters in the path. + if "." not in path: + return path + components = path.split("/") - output: typing.List[str] = [] + + # Fast return when no '.' or '..' components in the path. + if "." not in components and ".." not in components: + return path + + # https://datatracker.ietf.org/doc/html/rfc3986#section-5.2.4 + output: list[str] = [] for component in components: if component == ".": pass @@ -405,44 +415,22 @@ def normalize_path(path: str) -> str: return "/".join(output) -def percent_encode(char: str) -> str: - """ - Replace a single character with the percent-encoded representation. - - Characters outside the ASCII range are represented with their a percent-encoded - representation of their UTF-8 byte sequence. - - For example: - - percent_encode(" ") == "%20" - """ - return "".join([f"%{byte:02x}" for byte in char.encode("utf-8")]).upper() - - -def is_safe(string: str, safe: str = "/") -> bool: - """ - Determine if a given string is already quote-safe. - """ - NON_ESCAPED_CHARS = UNRESERVED_CHARACTERS + safe + "%" - - # All characters must already be non-escaping or '%' - for char in string: - if char not in NON_ESCAPED_CHARS: - return False - - return True +def PERCENT(string: str) -> str: + return "".join([f"%{byte:02X}" for byte in string.encode("utf-8")]) def percent_encoded(string: str, safe: str = "/") -> str: """ Use percent-encoding to quote a string. """ - if is_safe(string, safe=safe): + NON_ESCAPED_CHARS = UNRESERVED_CHARACTERS + safe + + # Fast path for strings that don't need escaping. + if not string.rstrip(NON_ESCAPED_CHARS): return string - NON_ESCAPED_CHARS = UNRESERVED_CHARACTERS + safe return "".join( - [char if char in NON_ESCAPED_CHARS else percent_encode(char) for char in string] + [char if char in NON_ESCAPED_CHARS else PERCENT(char) for char in string] ) @@ -479,7 +467,7 @@ def quote(string: str, safe: str = "/") -> str: return "".join(parts) -def urlencode(items: typing.List[typing.Tuple[str, str]]) -> str: +def urlencode(items: list[tuple[str, str]]) -> str: """ We can use a much simpler version of the stdlib urlencode here because we don't need to handle a bunch of different typing cases, such as bytes vs str. diff --git a/httpx/_urls.py b/httpx/_urls.py index 26202e95..f9f68a99 100644 --- a/httpx/_urls.py +++ b/httpx/_urls.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing from urllib.parse import parse_qs, unquote @@ -7,6 +9,8 @@ from ._types import QueryParamTypes, RawURL, URLTypes from ._urlparse import urlencode, urlparse from ._utils import primitive_value_to_str +__all__ = ["URL", "QueryParams"] + class URL: """ @@ -70,9 +74,7 @@ class URL: themselves. """ - def __init__( - self, url: typing.Union["URL", str] = "", **kwargs: typing.Any - ) -> None: + def __init__(self, url: URL | str = "", **kwargs: typing.Any) -> None: if kwargs: allowed = { "scheme": str, @@ -213,7 +215,7 @@ class URL: return self._uri_reference.host.encode("ascii") @property - def port(self) -> typing.Optional[int]: + def port(self) -> int | None: """ The URL port as an integer. @@ -270,7 +272,7 @@ class URL: return query.encode("ascii") @property - def params(self) -> "QueryParams": + def params(self) -> QueryParams: """ The URL query parameters, neatly parsed and packaged into an immutable multidict representation. @@ -338,7 +340,7 @@ class URL: """ return not self.is_absolute_url - def copy_with(self, **kwargs: typing.Any) -> "URL": + def copy_with(self, **kwargs: typing.Any) -> URL: """ Copy this URL, returning a new URL with some components altered. Accepts the same set of parameters as the components that are made @@ -353,19 +355,19 @@ class URL: """ return URL(self, **kwargs) - def copy_set_param(self, key: str, value: typing.Any = None) -> "URL": + def copy_set_param(self, key: str, value: typing.Any = None) -> URL: return self.copy_with(params=self.params.set(key, value)) - def copy_add_param(self, key: str, value: typing.Any = None) -> "URL": + def copy_add_param(self, key: str, value: typing.Any = None) -> URL: return self.copy_with(params=self.params.add(key, value)) - def copy_remove_param(self, key: str) -> "URL": + def copy_remove_param(self, key: str) -> URL: return self.copy_with(params=self.params.remove(key)) - def copy_merge_params(self, params: QueryParamTypes) -> "URL": + def copy_merge_params(self, params: QueryParamTypes) -> URL: return self.copy_with(params=self.params.merge(params)) - def join(self, url: URLTypes) -> "URL": + def join(self, url: URLTypes) -> URL: """ Return an absolute URL, using this URL as the base. @@ -420,9 +422,7 @@ class QueryParams(typing.Mapping[str, str]): URL query parameters, as a multi-dict. """ - def __init__( - self, *args: typing.Optional[QueryParamTypes], **kwargs: typing.Any - ) -> None: + def __init__(self, *args: QueryParamTypes | None, **kwargs: typing.Any) -> None: assert len(args) < 2, "Too many arguments." assert not (args and kwargs), "Cannot mix named and unnamed arguments." @@ -434,7 +434,7 @@ class QueryParams(typing.Mapping[str, str]): elif isinstance(value, QueryParams): self._dict = {k: list(v) for k, v in value._dict.items()} else: - dict_value: typing.Dict[typing.Any, typing.List[typing.Any]] = {} + dict_value: dict[typing.Any, list[typing.Any]] = {} if isinstance(value, (list, tuple)): # Convert list inputs like: # [("a", "123"), ("a", "456"), ("b", "789")] @@ -495,7 +495,7 @@ class QueryParams(typing.Mapping[str, str]): """ return {k: v[0] for k, v in self._dict.items()}.items() - def multi_items(self) -> typing.List[typing.Tuple[str, str]]: + def multi_items(self) -> list[tuple[str, str]]: """ Return all items in the query params. Allow duplicate keys to occur. @@ -504,7 +504,7 @@ class QueryParams(typing.Mapping[str, str]): q = httpx.QueryParams("a=123&a=456&b=789") assert list(q.multi_items()) == [("a", "123"), ("a", "456"), ("b", "789")] """ - multi_items: typing.List[typing.Tuple[str, str]] = [] + multi_items: list[tuple[str, str]] = [] for k, v in self._dict.items(): multi_items.extend([(k, i) for i in v]) return multi_items @@ -523,7 +523,7 @@ class QueryParams(typing.Mapping[str, str]): return self._dict[str(key)][0] return default - def get_list(self, key: str) -> typing.List[str]: + def get_list(self, key: str) -> list[str]: """ Get all values from the query param for a given key. @@ -534,7 +534,7 @@ class QueryParams(typing.Mapping[str, str]): """ return list(self._dict.get(str(key), [])) - def set(self, key: str, value: typing.Any = None) -> "QueryParams": + def set(self, key: str, value: typing.Any = None) -> QueryParams: """ Return a new QueryParams instance, setting the value of a key. @@ -549,7 +549,7 @@ class QueryParams(typing.Mapping[str, str]): q._dict[str(key)] = [primitive_value_to_str(value)] return q - def add(self, key: str, value: typing.Any = None) -> "QueryParams": + def add(self, key: str, value: typing.Any = None) -> QueryParams: """ Return a new QueryParams instance, setting or appending the value of a key. @@ -564,7 +564,7 @@ class QueryParams(typing.Mapping[str, str]): q._dict[str(key)] = q.get_list(key) + [primitive_value_to_str(value)] return q - def remove(self, key: str) -> "QueryParams": + def remove(self, key: str) -> QueryParams: """ Return a new QueryParams instance, removing the value of a key. @@ -579,7 +579,7 @@ class QueryParams(typing.Mapping[str, str]): q._dict.pop(str(key), None) return q - def merge(self, params: typing.Optional[QueryParamTypes] = None) -> "QueryParams": + def merge(self, params: QueryParamTypes | None = None) -> QueryParams: """ Return a new QueryParams instance, updated with. @@ -635,7 +635,7 @@ class QueryParams(typing.Mapping[str, str]): query_string = str(self) return f"{class_name}({query_string!r})" - def update(self, params: typing.Optional[QueryParamTypes] = None) -> None: + def update(self, params: QueryParamTypes | None = None) -> None: raise RuntimeError( "QueryParams are immutable since 0.18.0. " "Use `q = q.merge(...)` to create an updated copy." diff --git a/httpx/_utils.py b/httpx/_utils.py index bc3cb001..a9ece194 100644 --- a/httpx/_utils.py +++ b/httpx/_utils.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import codecs import email.message import ipaddress @@ -27,9 +29,9 @@ _HTML5_FORM_ENCODING_RE = re.compile( def normalize_header_key( - value: typing.Union[str, bytes], + value: str | bytes, lower: bool, - encoding: typing.Optional[str] = None, + encoding: str | None = None, ) -> bytes: """ Coerce str/bytes into a strictly byte-wise HTTP header key. @@ -42,9 +44,7 @@ def normalize_header_key( return bytes_value.lower() if lower else bytes_value -def normalize_header_value( - value: typing.Union[str, bytes], encoding: typing.Optional[str] = None -) -> bytes: +def normalize_header_value(value: str | bytes, encoding: str | None = None) -> bytes: """ Coerce str/bytes into a strictly byte-wise HTTP header value. """ @@ -53,7 +53,7 @@ def normalize_header_value( return value.encode(encoding or "ascii") -def primitive_value_to_str(value: "PrimitiveData") -> str: +def primitive_value_to_str(value: PrimitiveData) -> str: """ Coerce a primitive data type into a string value. @@ -91,7 +91,7 @@ def format_form_param(name: str, value: str) -> bytes: return f'{name}="{value}"'.encode() -def get_ca_bundle_from_env() -> typing.Optional[str]: +def get_ca_bundle_from_env() -> str | None: if "SSL_CERT_FILE" in os.environ: ssl_file = Path(os.environ["SSL_CERT_FILE"]) if ssl_file.is_file(): @@ -103,7 +103,7 @@ def get_ca_bundle_from_env() -> typing.Optional[str]: return None -def parse_header_links(value: str) -> typing.List[typing.Dict[str, str]]: +def parse_header_links(value: str) -> list[dict[str, str]]: """ Returns a list of parsed link headers, for more info see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link @@ -119,7 +119,7 @@ def parse_header_links(value: str) -> typing.List[typing.Dict[str, str]]: :param value: HTTP Link entity-header field :return: list of parsed link headers """ - links: typing.List[typing.Dict[str, str]] = [] + links: list[dict[str, str]] = [] replace_chars = " '\"" value = value.strip(replace_chars) if not value: @@ -140,7 +140,7 @@ def parse_header_links(value: str) -> typing.List[typing.Dict[str, str]]: return links -def parse_content_type_charset(content_type: str) -> typing.Optional[str]: +def parse_content_type_charset(content_type: str) -> str | None: # We used to use `cgi.parse_header()` here, but `cgi` became a dead battery. # See: https://peps.python.org/pep-0594/#cgi msg = email.message.Message() @@ -152,21 +152,21 @@ SENSITIVE_HEADERS = {"authorization", "proxy-authorization"} def obfuscate_sensitive_headers( - items: typing.Iterable[typing.Tuple[typing.AnyStr, typing.AnyStr]], -) -> typing.Iterator[typing.Tuple[typing.AnyStr, typing.AnyStr]]: + items: typing.Iterable[tuple[typing.AnyStr, typing.AnyStr]], +) -> typing.Iterator[tuple[typing.AnyStr, typing.AnyStr]]: for k, v in items: if to_str(k.lower()) in SENSITIVE_HEADERS: v = to_bytes_or_str("[secure]", match_type_of=v) yield k, v -def port_or_default(url: "URL") -> typing.Optional[int]: +def port_or_default(url: URL) -> int | None: if url.port is not None: return url.port return {"http": 80, "https": 443}.get(url.scheme) -def same_origin(url: "URL", other: "URL") -> bool: +def same_origin(url: URL, other: URL) -> bool: """ Return 'True' if the given URLs share the same origin. """ @@ -177,7 +177,7 @@ def same_origin(url: "URL", other: "URL") -> bool: ) -def is_https_redirect(url: "URL", location: "URL") -> bool: +def is_https_redirect(url: URL, location: URL) -> bool: """ Return 'True' if 'location' is a HTTPS upgrade of 'url' """ @@ -192,7 +192,7 @@ def is_https_redirect(url: "URL", location: "URL") -> bool: ) -def get_environment_proxies() -> typing.Dict[str, typing.Optional[str]]: +def get_environment_proxies() -> dict[str, str | None]: """Gets proxy information from the environment""" # urllib.request.getproxies() falls back on System @@ -200,7 +200,7 @@ def get_environment_proxies() -> typing.Dict[str, typing.Optional[str]]: # We don't want to propagate non-HTTP proxies into # our configuration such as 'TRAVIS_APT_PROXY'. proxy_info = getproxies() - mounts: typing.Dict[str, typing.Optional[str]] = {} + mounts: dict[str, str | None] = {} for scheme in ("http", "https", "all"): if proxy_info.get(scheme): @@ -241,11 +241,11 @@ def get_environment_proxies() -> typing.Dict[str, typing.Optional[str]]: return mounts -def to_bytes(value: typing.Union[str, bytes], encoding: str = "utf-8") -> bytes: +def to_bytes(value: str | bytes, encoding: str = "utf-8") -> bytes: return value.encode(encoding) if isinstance(value, str) else value -def to_str(value: typing.Union[str, bytes], encoding: str = "utf-8") -> str: +def to_str(value: str | bytes, encoding: str = "utf-8") -> str: return value if isinstance(value, str) else value.decode(encoding) @@ -257,13 +257,13 @@ def unquote(value: str) -> str: return value[1:-1] if value[0] == value[-1] == '"' else value -def guess_content_type(filename: typing.Optional[str]) -> typing.Optional[str]: +def guess_content_type(filename: str | None) -> str | None: if filename: return mimetypes.guess_type(filename)[0] or "application/octet-stream" return None -def peek_filelike_length(stream: typing.Any) -> typing.Optional[int]: +def peek_filelike_length(stream: typing.Any) -> int | None: """ Given a file-like stream object, return its length in number of bytes without reading it into memory. @@ -373,7 +373,7 @@ class URLPattern: self.host = "" if url.host == "*" else url.host self.port = url.port if not url.host or url.host == "*": - self.host_regex: typing.Optional[typing.Pattern[str]] = None + self.host_regex: typing.Pattern[str] | None = None elif url.host.startswith("*."): # *.example.com should match "www.example.com", but not "example.com" domain = re.escape(url.host[2:]) @@ -387,7 +387,7 @@ class URLPattern: domain = re.escape(url.host) self.host_regex = re.compile(f"^{domain}$") - def matches(self, other: "URL") -> bool: + def matches(self, other: URL) -> bool: if self.scheme and self.scheme != other.scheme: return False if ( @@ -401,7 +401,7 @@ class URLPattern: return True @property - def priority(self) -> typing.Tuple[int, int, int]: + def priority(self) -> tuple[int, int, int]: """ The priority allows URLPattern instances to be sortable, so that we can match from most specific to least specific. @@ -417,7 +417,7 @@ class URLPattern: def __hash__(self) -> int: return hash(self.pattern) - def __lt__(self, other: "URLPattern") -> bool: + def __lt__(self, other: URLPattern) -> bool: return self.priority < other.priority def __eq__(self, other: typing.Any) -> bool: diff --git a/mkdocs.yml b/mkdocs.yml index 7c6fcbd5..f6e4dfde 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -34,6 +34,7 @@ nav: - Event Hooks: 'advanced/event-hooks.md' - Transports: 'advanced/transports.md' - Text Encodings: 'advanced/text-encodings.md' + - Extensions: 'advanced/extensions.md' - Guides: - Async Support: 'async.md' - HTTP/2 Support: 'http2.md' diff --git a/pyproject.toml b/pyproject.toml index 4f7a848f..c4c18805 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -52,6 +52,9 @@ http2 = [ socks = [ "socksio==1.*", ] +zstd = [ + "zstandard>=0.18.0", +] [project.scripts] httpx = "httpx:main" @@ -93,14 +96,16 @@ text = "\n---\n\n[Full changelog](https://github.com/encode/httpx/blob/master/CH pattern = 'src="(docs/img/.*?)"' replacement = 'src="https://raw.githubusercontent.com/encode/httpx/master/\1"' -# https://beta.ruff.rs/docs/configuration/#using-rufftoml -[tool.ruff] +[tool.ruff.lint] select = ["E", "F", "I", "B", "PIE"] ignore = ["B904", "B028"] -[tool.ruff.isort] +[tool.ruff.lint.isort] combine-as-imports = true +[tool.ruff.lint.per-file-ignores] +"__init__.py" = ["F403", "F405"] + [tool.mypy] ignore_missing_imports = true strict = true diff --git a/requirements.txt b/requirements.txt index 5582acda..0a430eae 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,28 +2,28 @@ # On the other hand, we're not pinning package dependencies, because our tests # needs to pass with the latest version of the packages. # Reference: https://github.com/encode/httpx/pull/1721#discussion_r661241588 --e .[brotli,cli,http2,socks] +-e .[brotli,cli,http2,socks,zstd] # Optional charset auto-detection # Used in our test cases chardet==5.2.0 # Documentation -mkdocs==1.5.3 +mkdocs==1.6.0 mkautodoc==0.2.0 -mkdocs-material==9.5.3 +mkdocs-material==9.5.25 # Packaging -build==1.0.3 -twine==4.0.2 +build==1.2.1 +twine==5.1.0 # Tests & Linting -coverage[toml]==7.4.0 -cryptography==41.0.7 -mypy==1.8.0 -pytest==7.4.4 -ruff==0.1.13 -trio==0.24.0 +coverage[toml]==7.5.3 +cryptography==42.0.7 +mypy==1.10.0 +pytest==8.2.1 +ruff==0.4.7 +trio==0.25.1 trio-typing==0.10.0 trustme==1.1.0 -uvicorn==0.25.0 +uvicorn==0.30.0 diff --git a/tests/client/test_async_client.py b/tests/client/test_async_client.py index 49664df5..8d7eaa3c 100644 --- a/tests/client/test_async_client.py +++ b/tests/client/test_async_client.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing from datetime import timedelta @@ -181,7 +183,7 @@ async def test_100_continue(server): async def test_context_managed_transport(): class Transport(httpx.AsyncBaseTransport): def __init__(self) -> None: - self.events: typing.List[str] = [] + self.events: list[str] = [] async def aclose(self): # The base implementation of httpx.AsyncBaseTransport just @@ -214,7 +216,7 @@ async def test_context_managed_transport_and_mount(): class Transport(httpx.AsyncBaseTransport): def __init__(self, name: str) -> None: self.name: str = name - self.events: typing.List[str] = [] + self.events: list[str] = [] async def aclose(self): # The base implementation of httpx.AsyncBaseTransport just diff --git a/tests/client/test_auth.py b/tests/client/test_auth.py index e6bac23d..5776fc33 100644 --- a/tests/client/test_auth.py +++ b/tests/client/test_auth.py @@ -3,6 +3,7 @@ Integration tests for authentication. Unit tests for auth classes also exist in tests/test_auth.py """ + import hashlib import netrc import os diff --git a/tests/client/test_client.py b/tests/client/test_client.py index fcc6ec6a..65783901 100644 --- a/tests/client/test_client.py +++ b/tests/client/test_client.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing from datetime import timedelta @@ -230,7 +232,7 @@ def test_merge_relative_url_with_encoded_slashes(): def test_context_managed_transport(): class Transport(httpx.BaseTransport): def __init__(self) -> None: - self.events: typing.List[str] = [] + self.events: list[str] = [] def close(self): # The base implementation of httpx.BaseTransport just @@ -262,7 +264,7 @@ def test_context_managed_transport_and_mount(): class Transport(httpx.BaseTransport): def __init__(self, name: str) -> None: self.name: str = name - self.events: typing.List[str] = [] + self.events: list[str] = [] def close(self): # The base implementation of httpx.BaseTransport just @@ -355,7 +357,7 @@ def test_raw_client_header(): assert response.json() == [ ["Host", "example.org"], ["Accept", "*/*"], - ["Accept-Encoding", "gzip, deflate, br"], + ["Accept-Encoding", "gzip, deflate, br, zstd"], ["Connection", "keep-alive"], ["User-Agent", f"python-httpx/{httpx.__version__}"], ["Example-Header", "example-value"], diff --git a/tests/client/test_event_hooks.py b/tests/client/test_event_hooks.py index 6604dd31..78fb0484 100644 --- a/tests/client/test_event_hooks.py +++ b/tests/client/test_event_hooks.py @@ -36,7 +36,7 @@ def test_event_hooks(): "host": "127.0.0.1:8000", "user-agent": f"python-httpx/{httpx.__version__}", "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", }, @@ -87,7 +87,7 @@ async def test_async_event_hooks(): "host": "127.0.0.1:8000", "user-agent": f"python-httpx/{httpx.__version__}", "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", }, @@ -144,7 +144,7 @@ def test_event_hooks_with_redirect(): "host": "127.0.0.1:8000", "user-agent": f"python-httpx/{httpx.__version__}", "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", }, @@ -159,7 +159,7 @@ def test_event_hooks_with_redirect(): "host": "127.0.0.1:8000", "user-agent": f"python-httpx/{httpx.__version__}", "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", }, @@ -201,7 +201,7 @@ async def test_async_event_hooks_with_redirect(): "host": "127.0.0.1:8000", "user-agent": f"python-httpx/{httpx.__version__}", "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", }, @@ -216,7 +216,7 @@ async def test_async_event_hooks_with_redirect(): "host": "127.0.0.1:8000", "user-agent": f"python-httpx/{httpx.__version__}", "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ=", }, diff --git a/tests/client/test_headers.py b/tests/client/test_headers.py index 264ca0bd..c51e40c3 100755 --- a/tests/client/test_headers.py +++ b/tests/client/test_headers.py @@ -34,7 +34,7 @@ def test_client_header(): assert response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "example-header": "example-value", "host": "example.org", @@ -56,7 +56,7 @@ def test_header_merge(): assert response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "host": "example.org", "user-agent": "python-myclient/0.2.1", @@ -78,7 +78,7 @@ def test_header_merge_conflicting_headers(): assert response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "host": "example.org", "user-agent": f"python-httpx/{httpx.__version__}", @@ -100,7 +100,7 @@ def test_header_update(): assert first_response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "host": "example.org", "user-agent": f"python-httpx/{httpx.__version__}", @@ -111,7 +111,7 @@ def test_header_update(): assert second_response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "another-header": "AThing", "connection": "keep-alive", "host": "example.org", @@ -164,7 +164,7 @@ def test_remove_default_header(): assert response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "host": "example.org", } @@ -192,7 +192,7 @@ def test_host_with_auth_and_port_in_url(): assert response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "host": "example.org", "user-agent": f"python-httpx/{httpx.__version__}", @@ -215,7 +215,7 @@ def test_host_with_non_default_port_in_url(): assert response.json() == { "headers": { "accept": "*/*", - "accept-encoding": "gzip, deflate, br", + "accept-encoding": "gzip, deflate, br, zstd", "connection": "keep-alive", "host": "example.org:123", "user-agent": f"python-httpx/{httpx.__version__}", diff --git a/tests/conftest.py b/tests/conftest.py index 1bcb6a42..5c4a6ae5 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -236,7 +236,7 @@ class TestServer(Server): def install_signal_handlers(self) -> None: # Disable the default installation of handlers for signals such as SIGTERM, # because it can only be done in the main thread. - pass + pass # pragma: nocover async def serve(self, sockets=None): self.restart_requested = asyncio.Event() diff --git a/tests/models/test_url.py b/tests/models/test_url.py index 79e1605a..32571238 100644 --- a/tests/models/test_url.py +++ b/tests/models/test_url.py @@ -229,6 +229,11 @@ def test_url_normalized_host(): assert url.host == "example.com" +def test_url_percent_escape_host(): + url = httpx.URL("https://exam%le.com/") + assert url.host == "exam%25le.com" + + def test_url_ipv4_like_host(): """rare host names used to quality as IPv4""" url = httpx.URL("https://023b76x43144/") @@ -278,24 +283,64 @@ def test_url_leading_dot_prefix_on_relative_url(): assert url.path == "../abc" -# Tests for optional percent encoding +# Tests for query parameter percent encoding. +# +# Percent-encoding in `params={}` should match browser form behavior. -def test_param_requires_encoding(): +def test_param_with_space(): + # Params passed as form key-value pairs should be escaped. url = httpx.URL("http://webservice", params={"u": "with spaces"}) assert str(url) == "http://webservice?u=with%20spaces" def test_param_does_not_require_encoding(): + # Params passed as form key-value pairs should be escaped. + url = httpx.URL("http://webservice", params={"u": "%"}) + assert str(url) == "http://webservice?u=%25" + + +def test_param_with_percent_encoded(): + # Params passed as form key-value pairs should always be escaped, + # even if they include a valid escape sequence. + # We want to match browser form behaviour here. url = httpx.URL("http://webservice", params={"u": "with%20spaces"}) - assert str(url) == "http://webservice?u=with%20spaces" + assert str(url) == "http://webservice?u=with%2520spaces" def test_param_with_existing_escape_requires_encoding(): + # Params passed as form key-value pairs should always be escaped, + # even if they include a valid escape sequence. + # We want to match browser form behaviour here. url = httpx.URL("http://webservice", params={"u": "http://example.com?q=foo%2Fa"}) assert str(url) == "http://webservice?u=http%3A%2F%2Fexample.com%3Fq%3Dfoo%252Fa" +# Tests for query parameter percent encoding. +# +# Percent-encoding in `url={}` should match browser URL bar behavior. + + +def test_query_with_existing_percent_encoding(): + # Valid percent encoded sequences should not be double encoded. + url = httpx.URL("http://webservice?u=phrase%20with%20spaces") + assert str(url) == "http://webservice?u=phrase%20with%20spaces" + + +def test_query_requiring_percent_encoding(): + # Characters that require percent encoding should be encoded. + url = httpx.URL("http://webservice?u=phrase with spaces") + assert str(url) == "http://webservice?u=phrase%20with%20spaces" + + +def test_query_with_mixed_percent_encoding(): + # When a mix of encoded and unencoded characters are present, + # characters that require percent encoding should be encoded, + # while existing sequences should not be double encoded. + url = httpx.URL("http://webservice?u=phrase%20with spaces") + assert str(url) == "http://webservice?u=phrase%20with%20spaces" + + # Tests for invalid URLs diff --git a/tests/test_asgi.py b/tests/test_asgi.py index 29715060..8b817891 100644 --- a/tests/test_asgi.py +++ b/tests/test_asgi.py @@ -92,7 +92,8 @@ async def test_asgi_transport_no_body(): @pytest.mark.anyio async def test_asgi(): - async with httpx.AsyncClient(app=hello_world) as client: + transport = httpx.ASGITransport(app=hello_world) + async with httpx.AsyncClient(transport=transport) as client: response = await client.get("http://www.example.org/") assert response.status_code == 200 @@ -101,7 +102,8 @@ async def test_asgi(): @pytest.mark.anyio async def test_asgi_urlencoded_path(): - async with httpx.AsyncClient(app=echo_path) as client: + transport = httpx.ASGITransport(app=echo_path) + async with httpx.AsyncClient(transport=transport) as client: url = httpx.URL("http://www.example.org/").copy_with(path="/user@example.org") response = await client.get(url) @@ -111,7 +113,8 @@ async def test_asgi_urlencoded_path(): @pytest.mark.anyio async def test_asgi_raw_path(): - async with httpx.AsyncClient(app=echo_raw_path) as client: + transport = httpx.ASGITransport(app=echo_raw_path) + async with httpx.AsyncClient(transport=transport) as client: url = httpx.URL("http://www.example.org/").copy_with(path="/user@example.org") response = await client.get(url) @@ -124,7 +127,8 @@ async def test_asgi_raw_path_should_not_include_querystring_portion(): """ See https://github.com/encode/httpx/issues/2810 """ - async with httpx.AsyncClient(app=echo_raw_path) as client: + transport = httpx.ASGITransport(app=echo_raw_path) + async with httpx.AsyncClient(transport=transport) as client: url = httpx.URL("http://www.example.org/path?query") response = await client.get(url) @@ -134,7 +138,8 @@ async def test_asgi_raw_path_should_not_include_querystring_portion(): @pytest.mark.anyio async def test_asgi_upload(): - async with httpx.AsyncClient(app=echo_body) as client: + transport = httpx.ASGITransport(app=echo_body) + async with httpx.AsyncClient(transport=transport) as client: response = await client.post("http://www.example.org/", content=b"example") assert response.status_code == 200 @@ -143,7 +148,8 @@ async def test_asgi_upload(): @pytest.mark.anyio async def test_asgi_headers(): - async with httpx.AsyncClient(app=echo_headers) as client: + transport = httpx.ASGITransport(app=echo_headers) + async with httpx.AsyncClient(transport=transport) as client: response = await client.get("http://www.example.org/") assert response.status_code == 200 @@ -151,7 +157,7 @@ async def test_asgi_headers(): "headers": [ ["host", "www.example.org"], ["accept", "*/*"], - ["accept-encoding", "gzip, deflate, br"], + ["accept-encoding", "gzip, deflate, br, zstd"], ["connection", "keep-alive"], ["user-agent", f"python-httpx/{httpx.__version__}"], ] @@ -160,14 +166,16 @@ async def test_asgi_headers(): @pytest.mark.anyio async def test_asgi_exc(): - async with httpx.AsyncClient(app=raise_exc) as client: + transport = httpx.ASGITransport(app=raise_exc) + async with httpx.AsyncClient(transport=transport) as client: with pytest.raises(RuntimeError): await client.get("http://www.example.org/") @pytest.mark.anyio async def test_asgi_exc_after_response(): - async with httpx.AsyncClient(app=raise_exc_after_response) as client: + transport = httpx.ASGITransport(app=raise_exc_after_response) + async with httpx.AsyncClient(transport=transport) as client: with pytest.raises(RuntimeError): await client.get("http://www.example.org/") @@ -199,7 +207,8 @@ async def test_asgi_disconnect_after_response_complete(): message = await receive() disconnect = message.get("type") == "http.disconnect" - async with httpx.AsyncClient(app=read_body) as client: + transport = httpx.ASGITransport(app=read_body) + async with httpx.AsyncClient(transport=transport) as client: response = await client.post("http://www.example.org/", content=b"example") assert response.status_code == 200 @@ -213,3 +222,13 @@ async def test_asgi_exc_no_raise(): response = await client.get("http://www.example.org/") assert response.status_code == 500 + + +@pytest.mark.anyio +async def test_deprecated_shortcut(): + """ + The `app=...` shortcut is now deprecated. + Use the explicit transport style instead. + """ + with pytest.warns(DeprecationWarning): + httpx.AsyncClient(app=hello_world) diff --git a/tests/test_auth.py b/tests/test_auth.py index 7bb45de5..6b6df922 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -3,6 +3,7 @@ Unit tests for auth classes. Integration tests also exist in tests/client/test_auth.py """ + from urllib.request import parse_keqv_list import pytest diff --git a/tests/test_decoders.py b/tests/test_decoders.py index 170a9345..bcbb18bb 100644 --- a/tests/test_decoders.py +++ b/tests/test_decoders.py @@ -1,8 +1,12 @@ +from __future__ import annotations + +import io import typing import zlib import chardet import pytest +import zstandard as zstd import httpx @@ -71,6 +75,53 @@ def test_brotli(): assert response.content == body +def test_zstd(): + body = b"test 123" + compressed_body = zstd.compress(body) + + headers = [(b"Content-Encoding", b"zstd")] + response = httpx.Response( + 200, + headers=headers, + content=compressed_body, + ) + assert response.content == body + + +def test_zstd_decoding_error(): + compressed_body = "this_is_not_zstd_compressed_data" + + headers = [(b"Content-Encoding", b"zstd")] + with pytest.raises(httpx.DecodingError): + httpx.Response( + 200, + headers=headers, + content=compressed_body, + ) + + +def test_zstd_multiframe(): + # test inspired by urllib3 test suite + data = ( + # Zstandard frame + zstd.compress(b"foo") + # skippable frame (must be ignored) + + bytes.fromhex( + "50 2A 4D 18" # Magic_Number (little-endian) + "07 00 00 00" # Frame_Size (little-endian) + "00 00 00 00 00 00 00" # User_Data + ) + # Zstandard frame + + zstd.compress(b"bar") + ) + compressed_body = io.BytesIO(data) + + headers = [(b"Content-Encoding", b"zstd")] + response = httpx.Response(200, headers=headers, content=compressed_body) + response.read() + assert response.content == b"foobar" + + def test_multi(): body = b"test 123" @@ -224,7 +275,7 @@ def test_text_decoder_empty_cases(): [((b"Hello,", b" world!"), ["Hello,", " world!"])], ) def test_streaming_text_decoder( - data: typing.Iterable[bytes], expected: typing.List[str] + data: typing.Iterable[bytes], expected: list[str] ) -> None: response = httpx.Response(200, content=iter(data)) assert list(response.iter_text()) == expected diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index 6547ab37..60c8721c 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import typing import httpcore @@ -34,7 +36,7 @@ def test_httpcore_all_exceptions_mapped() -> None: pytest.fail(f"Unmapped httpcore exceptions: {unmapped_exceptions}") -def test_httpcore_exception_mapping(server: "TestServer") -> None: +def test_httpcore_exception_mapping(server: TestServer) -> None: """ HTTPCore exception mapping works as expected. """ diff --git a/tests/test_main.py b/tests/test_main.py index 67eeb0d2..feb796e1 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -129,7 +129,7 @@ def test_verbose(server): "GET / HTTP/1.1", f"Host: {server.url.netloc.decode('ascii')}", "Accept: */*", - "Accept-Encoding: gzip, deflate, br", + "Accept-Encoding: gzip, deflate, br, zstd", "Connection: keep-alive", f"User-Agent: python-httpx/{httpx.__version__}", "", @@ -154,7 +154,7 @@ def test_auth(server): "GET / HTTP/1.1", f"Host: {server.url.netloc.decode('ascii')}", "Accept: */*", - "Accept-Encoding: gzip, deflate, br", + "Accept-Encoding: gzip, deflate, br, zstd", "Connection: keep-alive", f"User-Agent: python-httpx/{httpx.__version__}", "Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=", diff --git a/tests/test_multipart.py b/tests/test_multipart.py index fc283c9c..764f85a2 100644 --- a/tests/test_multipart.py +++ b/tests/test_multipart.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import io import tempfile import typing @@ -148,7 +150,7 @@ def test_multipart_file_tuple(): @pytest.mark.parametrize("file_content_type", [None, "text/plain"]) -def test_multipart_file_tuple_headers(file_content_type: typing.Optional[str]) -> None: +def test_multipart_file_tuple_headers(file_content_type: str | None) -> None: file_name = "test.txt" file_content = io.BytesIO(b"") file_headers = {"Expires": "0"} @@ -460,8 +462,8 @@ class TestHeaderParamHTML5Formatting: assert expected in request.read() def test_unicode_with_control_character(self): - filename = "hello\x1A\x1B\x1C" - expected = b'filename="hello%1A\x1B%1C"' + filename = "hello\x1a\x1b\x1c" + expected = b'filename="hello%1A\x1b%1C"' files = {"upload": (filename, b"")} request = httpx.Request("GET", "https://www.example.com", files=files) assert expected in request.read() diff --git a/tests/test_timeouts.py b/tests/test_timeouts.py index 09b25160..666cc8e3 100644 --- a/tests/test_timeouts.py +++ b/tests/test_timeouts.py @@ -42,3 +42,14 @@ async def test_pool_timeout(server): with pytest.raises(httpx.PoolTimeout): async with client.stream("GET", server.url): await client.get(server.url) + + +@pytest.mark.anyio +async def test_async_client_new_request_send_timeout(server): + timeout = httpx.Timeout(1e-6) + + async with httpx.AsyncClient(timeout=timeout) as client: + with pytest.raises(httpx.TimeoutException): + await client.send( + httpx.Request("GET", server.url.copy_with(path="/slow_response")) + ) diff --git a/tests/test_utils.py b/tests/test_utils.py index 0ef87d18..f98a18f2 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -11,8 +11,6 @@ from httpx._utils import ( URLPattern, get_ca_bundle_from_env, get_environment_proxies, - is_https_redirect, - same_origin, ) from .common import TESTS_DIR @@ -225,33 +223,59 @@ def test_obfuscate_sensitive_headers(headers, output): def test_same_origin(): - origin1 = httpx.URL("https://example.com") - origin2 = httpx.URL("HTTPS://EXAMPLE.COM:443") - assert same_origin(origin1, origin2) + origin = httpx.URL("https://example.com") + request = httpx.Request("GET", "HTTPS://EXAMPLE.COM:443") + + client = httpx.Client() + headers = client._redirect_headers(request, origin, "GET") + + assert headers["Host"] == request.url.netloc.decode("ascii") def test_not_same_origin(): - origin1 = httpx.URL("https://example.com") - origin2 = httpx.URL("HTTP://EXAMPLE.COM") - assert not same_origin(origin1, origin2) + origin = httpx.URL("https://example.com") + request = httpx.Request("GET", "HTTP://EXAMPLE.COM:80") + + client = httpx.Client() + headers = client._redirect_headers(request, origin, "GET") + + assert headers["Host"] == origin.netloc.decode("ascii") def test_is_https_redirect(): - url = httpx.URL("http://example.com") - location = httpx.URL("https://example.com") - assert is_https_redirect(url, location) + url = httpx.URL("https://example.com") + request = httpx.Request( + "GET", "http://example.com", headers={"Authorization": "empty"} + ) + + client = httpx.Client() + headers = client._redirect_headers(request, url, "GET") + + assert "Authorization" in headers def test_is_not_https_redirect(): - url = httpx.URL("http://example.com") - location = httpx.URL("https://www.example.com") - assert not is_https_redirect(url, location) + url = httpx.URL("https://www.example.com") + request = httpx.Request( + "GET", "http://example.com", headers={"Authorization": "empty"} + ) + + client = httpx.Client() + headers = client._redirect_headers(request, url, "GET") + + assert "Authorization" not in headers def test_is_not_https_redirect_if_not_default_ports(): - url = httpx.URL("http://example.com:9999") - location = httpx.URL("https://example.com:1337") - assert not is_https_redirect(url, location) + url = httpx.URL("https://example.com:1337") + request = httpx.Request( + "GET", "http://example.com:9999", headers={"Authorization": "empty"} + ) + + client = httpx.Client() + headers = client._redirect_headers(request, url, "GET") + + assert "Authorization" not in headers @pytest.mark.parametrize( diff --git a/tests/test_wsgi.py b/tests/test_wsgi.py index a952da6a..0134bee8 100644 --- a/tests/test_wsgi.py +++ b/tests/test_wsgi.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import sys import typing import wsgiref.validate @@ -12,7 +14,7 @@ if typing.TYPE_CHECKING: # pragma: no cover from _typeshed.wsgi import StartResponse, WSGIApplication, WSGIEnvironment -def application_factory(output: typing.Iterable[bytes]) -> "WSGIApplication": +def application_factory(output: typing.Iterable[bytes]) -> WSGIApplication: def application(environ, start_response): status = "200 OK" @@ -29,7 +31,7 @@ def application_factory(output: typing.Iterable[bytes]) -> "WSGIApplication": def echo_body( - environ: "WSGIEnvironment", start_response: "StartResponse" + environ: WSGIEnvironment, start_response: StartResponse ) -> typing.Iterable[bytes]: status = "200 OK" output = environ["wsgi.input"].read() @@ -44,7 +46,7 @@ def echo_body( def echo_body_with_response_stream( - environ: "WSGIEnvironment", start_response: "StartResponse" + environ: WSGIEnvironment, start_response: StartResponse ) -> typing.Iterable[bytes]: status = "200 OK" @@ -63,9 +65,9 @@ def echo_body_with_response_stream( def raise_exc( - environ: "WSGIEnvironment", - start_response: "StartResponse", - exc: typing.Type[Exception] = ValueError, + environ: WSGIEnvironment, + start_response: StartResponse, + exc: type[Exception] = ValueError, ) -> typing.Iterable[bytes]: status = "500 Server Error" output = b"Nope!" @@ -90,41 +92,47 @@ def log_to_wsgi_log_buffer(environ, start_response): def test_wsgi(): - client = httpx.Client(app=application_factory([b"Hello, World!"])) + transport = httpx.WSGITransport(app=application_factory([b"Hello, World!"])) + client = httpx.Client(transport=transport) response = client.get("http://www.example.org/") assert response.status_code == 200 assert response.text == "Hello, World!" def test_wsgi_upload(): - client = httpx.Client(app=echo_body) + transport = httpx.WSGITransport(app=echo_body) + client = httpx.Client(transport=transport) response = client.post("http://www.example.org/", content=b"example") assert response.status_code == 200 assert response.text == "example" def test_wsgi_upload_with_response_stream(): - client = httpx.Client(app=echo_body_with_response_stream) + transport = httpx.WSGITransport(app=echo_body_with_response_stream) + client = httpx.Client(transport=transport) response = client.post("http://www.example.org/", content=b"example") assert response.status_code == 200 assert response.text == "example" def test_wsgi_exc(): - client = httpx.Client(app=raise_exc) + transport = httpx.WSGITransport(app=raise_exc) + client = httpx.Client(transport=transport) with pytest.raises(ValueError): client.get("http://www.example.org/") def test_wsgi_http_error(): - client = httpx.Client(app=partial(raise_exc, exc=RuntimeError)) + transport = httpx.WSGITransport(app=partial(raise_exc, exc=RuntimeError)) + client = httpx.Client(transport=transport) with pytest.raises(RuntimeError): client.get("http://www.example.org/") def test_wsgi_generator(): output = [b"", b"", b"Some content", b" and more content"] - client = httpx.Client(app=application_factory(output)) + transport = httpx.WSGITransport(app=application_factory(output)) + client = httpx.Client(transport=transport) response = client.get("http://www.example.org/") assert response.status_code == 200 assert response.text == "Some content and more content" @@ -132,7 +140,8 @@ def test_wsgi_generator(): def test_wsgi_generator_empty(): output = [b"", b"", b"", b""] - client = httpx.Client(app=application_factory(output)) + transport = httpx.WSGITransport(app=application_factory(output)) + client = httpx.Client(transport=transport) response = client.get("http://www.example.org/") assert response.status_code == 200 assert response.text == "" @@ -161,14 +170,15 @@ def test_wsgi_server_port(url: str, expected_server_port: str) -> None: SERVER_PORT is populated correctly from the requested URL. """ hello_world_app = application_factory([b"Hello, World!"]) - server_port: typing.Optional[str] = None + server_port: str | None = None def app(environ, start_response): nonlocal server_port server_port = environ["SERVER_PORT"] return hello_world_app(environ, start_response) - client = httpx.Client(app=app) + transport = httpx.WSGITransport(app=app) + client = httpx.Client(transport=transport) response = client.get(url) assert response.status_code == 200 assert response.text == "Hello, World!" @@ -184,9 +194,19 @@ def test_wsgi_server_protocol(): start_response("200 OK", [("Content-Type", "text/plain")]) return [b"success"] - with httpx.Client(app=app, base_url="http://testserver") as client: + transport = httpx.WSGITransport(app=app) + with httpx.Client(transport=transport, base_url="http://testserver") as client: response = client.get("/") assert response.status_code == 200 assert response.text == "success" assert server_protocol == "HTTP/1.1" + + +def test_deprecated_shortcut(): + """ + The `app=...` shortcut is now deprecated. + Use the explicit transport style instead. + """ + with pytest.warns(DeprecationWarning): + httpx.Client(app=application_factory([b"Hello, World!"]))