Stuff

README.md

@@ -0,0 +1,82 @@
+# HTTPCore
+
+I started to dive into implementation and API design here.
+
+I know this isn't what you were suggesting with `requests-core`, but it'd be
+worth you taking a slow look at this and seeing if there's anything that you
+think is a no-go.
+
+`httpcore` provides the same proposed *functionality* as requests-core, but at a slightly
+lower abstraction level.
+
+Rather than returning `Response` models, it returns the minimal possible
+interface. There's no `response.text` or any other cleverness, `response.headers`
+are plain byte-pair lists, rather than a headers datastructure etc...
+
+**The proposal here is that `httpcore` would be a silent-partner dependency of `requests3`,
+taking the place of the existing `urllib3` dependency.**
+
+---
+
+The benefits to my mind of this level of abstraction are that it is as
+agnostic as possible to whatever request/response models are built on top
+of it, and exposes only plain datastructures that reflect the network response.
+
+* An `encode/httpcore` package would be something I'd gladly maintain. The naming
+  makes sense to me, as there's no strictly implied relationship to `requests`,
+  although it would fulfil all the requirements for `requests3` to build on,
+  and would have a strict semver policy.
+* An `encode/httpcore` package is something that would play in well to the
+  collaboratively sponsored OSS story that Encode is pitching. It'd provide what
+  you need for `requests3` without encroaching on the `requests` brand.
+  We'd position it similarly to how `urllib3` is positioned to `requests` now.
+  A focused, low-level networking library, that `requests` then builds the
+  developer-focused API on top of.
+* The current implementation includes all the async API points.
+  The `PoolManger.request()` and `PoolManager.close()` methods are currently
+  stubbed-out. All the remaining implementation hangs off of those two points.
+* Take a quick look over the test cases or the package itself to get a feel
+  for it. It's all type annotated, and should be easy to find your way around.
+* I've not yet added corresponding sync API points to the implementation, but
+  they will come.
+* There's [a chunk of work towards connection pooling here](https://github.com/encode/requests-async/blob/5ec2aa80bd4499997fa744f3be19a0bdeccbaeed/requests_async/connections.py). I've not had enough time to nail it yet, but it's got the broad brush-strokes.
+* We would absolutely want to implement HTTP/2 support.
+* Trio support is something that could *potentially* come later, but it needs to
+  be a secondary consideration.
+* I think all the functionality required is stubbed out in the API, with two exceptions.
+  1. I've not yet added any proxy configuration API. Haven't looked into that enough
+  yet. 2. I've not yet added any retry configuration API, since I havn't really
+  looked enough into which side of requests vs. urllib3 that sits on, or exactly how
+  urllib3 tackles retries, etc.
+* I'd be planning to prioritize working on this from Mon 15th April. I don't think
+  it'd take too long to get it to a feature complete and API stable state.
+  (With the exception of the later HTTP/2 work, which I can't really assess yet.)
+  I probably don't have any time left before then - need to focus on what I'm
+  delivering to DjangoCon Europe over the rest of this week.
+* To my mind the killer app for `requests3`/`httpcore` is a high-performance
+  proxy server / gateway service in Python. Pitching the growing ASGI ecosystem
+  is an important part of that story.
+* I think there's enough headroom before PyCon to have something ready to pitch by then.
+  I could be involved in sprints remotely if there's areas we still need to fill in,
+  anyplace.
+
+```python
+import httpcore
+
+response = await httpcore.request('GET', 'http://example.com')
+assert response.status_code == 200
+assert response.body == b'Hello, world'
+```
+
+API...
+
+```python
+response = await httpcore.request(method, url, [headers], [body], [stream])
+```
+
+Explicit PoolManager...
+
+```python
+async with httpcore.PoolManager([ssl], [timeout], [limits]) as pool:
+    response = await pool.request(method, url, [headers], [body], [stream])
+```

httpcore/__init__.py

@@ -1 +1,5 @@
+from .api import PoolManager, Response, request
+from .config import PoolLimits, SSLConfig, TimeoutConfig
+from .exceptions import ResponseClosed, StreamConsumed
+
 __version__ = "0.0.1"

httpcore/api.py

@@ -0,0 +1,132 @@
+import typing
+from types import TracebackType
+
+from .config import (
+    DEFAULT_POOL_LIMITS,
+    DEFAULT_SSL_CONFIG,
+    DEFAULT_TIMEOUT_CONFIG,
+    PoolLimits,
+    SSLConfig,
+    TimeoutConfig,
+)
+from .exceptions import ResponseClosed, StreamConsumed
+
+
+async def request(
+    method: str,
+    url: str,
+    *,
+    headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (),
+    body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"",
+    stream: bool = False,
+    ssl: SSLConfig = DEFAULT_SSL_CONFIG,
+    timeout: TimeoutConfig = DEFAULT_TIMEOUT_CONFIG,
+) -> "Response":
+    async with PoolManager(ssl=ssl, timeout=timeout) as pool:
+       return await pool.request(
+           method=method, url=url, headers=headers, body=body, stream=stream
+       )
+
+
+class PoolManager:
+    def __init__(
+        self,
+        *,
+        ssl: SSLConfig = DEFAULT_SSL_CONFIG,
+        timeout: TimeoutConfig = DEFAULT_TIMEOUT_CONFIG,
+        limits: PoolLimits = DEFAULT_POOL_LIMITS,
+    ):
+        self.ssl = ssl
+        self.timeout = timeout
+        self.limits = limits
+        self.is_closed = False
+
+    async def request(
+        self,
+        method: str,
+        url: str,
+        *,
+        headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (),
+        body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"",
+        stream: bool = False,
+    ) -> "Response":
+        if stream:
+            async def streaming_body():
+                yield b"Hello, "
+                yield b"world!"
+            return Response(200, body=streaming_body)
+        return Response(200, body=b"Hello, world!")
+
+    async def close(self) -> None:
+        self.is_closed = True
+
+    async def __aenter__(self) -> "PoolManager":
+        return self
+
+    async def __aexit__(
+        self,
+        exc_type: typing.Type[BaseException] = None,
+        exc_value: BaseException = None,
+        traceback: TracebackType = None,
+    ) -> None:
+        await self.close()
+
+
+class Response:
+    def __init__(
+        self,
+        status_code: int,
+        *,
+        headers: typing.Sequence[typing.Tuple[bytes, bytes]] = (),
+        body: typing.Union[bytes, typing.AsyncIterator[bytes]] = b"",
+        on_close: typing.Callable = None,
+    ):
+        self.status_code = status_code
+        self.headers = list(headers)
+        self.on_close = on_close
+        self.is_closed = False
+        self.is_streamed = False
+        if isinstance(body, bytes):
+            self.is_closed = True
+            self.body = body
+        else:
+            self.body_aiter = body
+
+    async def read(self) -> bytes:
+        if not hasattr(self, "body"):
+            body = b""
+            async for part in self.stream():
+                body += part
+            self.body = body
+        return self.body
+
+    async def stream(self) -> typing.AsyncIterator[bytes]:
+        if hasattr(self, "body"):
+            yield self.body
+        else:
+            if self.is_streamed:
+                raise StreamConsumed()
+            if self.is_closed:
+                raise ResponseClosed()
+            self.is_streamed = True
+            async for part in self.body_aiter():
+                yield part
+            await self.close()
+
+    async def close(self) -> None:
+        if not self.is_closed:
+            self.is_closed = True
+            if self.on_close is not None:
+                await self.on_close()
+
+    async def __aenter__(self) -> "Response":
+        return self
+
+    async def __aexit__(
+        self,
+        exc_type: typing.Type[BaseException] = None,
+        exc_value: BaseException = None,
+        traceback: TracebackType = None,
+    ) -> None:
+        if not self.is_closed:
+            await self.close()

httpcore/config.py

@@ -0,0 +1,54 @@
+import typing
+
+
+class SSLConfig:
+    """
+    SSL Configuration.
+    """
+
+    def __init__(self, *, cert: typing.Optional[str], verify: typing.Union[str, bool]):
+        self.cert = cert
+        self.verify = verify
+
+
+class TimeoutConfig:
+    """
+    Timeout values.
+    """
+
+    def __init__(
+        self,
+        timeout: float = None,
+        *,
+        connect_timeout: float = None,
+        read_timeout: float = None,
+        pool_timeout: float = None
+    ):
+        if timeout is not None:
+            # Specified as a single timeout value
+            assert connect_timeout is None
+            assert read_timeout is None
+            assert pool_timeout is None
+            connect_timeout = timeout
+            read_timeout = timeout
+            pool_timeout = timeout
+
+        self.connect_timeout = connect_timeout
+        self.read_timeout = read_timeout
+        self.pool_timeout = pool_timeout
+
+
+class PoolLimits:
+    """
+    Limits on the number of connections in a connection pool.
+    """
+
+    def __init__(self, *, max_hosts: int, conns_per_host: int, hard_limit: bool):
+        self.max_hosts = max_hosts
+        self.conns_per_host = conns_per_host
+        self.hard_limit = hard_limit
+
+
+DEFAULT_SSL_CONFIG = SSLConfig(cert=None, verify=True)
+DEFAULT_TIMEOUT_CONFIG = TimeoutConfig(timeout=5.0)
+DEFAULT_POOL_LIMITS = PoolLimits(max_hosts=10, conns_per_host=10, hard_limit=False)

httpcore/exceptions.py

@@ -0,0 +1,42 @@
+class Timeout(Exception):
+    """
+    A base class for all timeouts.
+    """
+
+
+class ConnectTimeout(Timeout):
+    """
+    Timeout while establishing a connection.
+    """
+
+
+class ReadTimeout(Timeout):
+    """
+    Timeout while reading response data.
+    """
+
+
+class PoolTimeout(Timeout):
+    """
+    Timeout while waiting to acquire a connection from the pool.
+    """
+
+
+class BadResponse(Exception):
+    """
+    A malformed HTTP response.
+    """
+
+
+class StreamConsumed(Exception):
+    """
+    Attempted to read or stream response content, but the content has already
+    been streamed.
+    """
+
+
+class ResponseClosed(Exception):
+    """
+    Attempted to read or stream response content, but the request has been
+    closed without loading the body.
+    """

requirements.txt

@@ -0,0 +1,11 @@
+h11
+
+# Testing
+autoflake
+black
+codecov
+isort
+mypy
+pytest
+pytest-asyncio
+pytest-cov

scripts/install

@@ -0,0 +1,24 @@
+#!/bin/sh -e
+
+# Use the Python executable provided from the `-p` option, or a default.
+[[ $1 = "-p" ]] && PYTHON=$2 || PYTHON="python3"
+
+MIN_VERSION="(3, 6)"
+VERSION_OK=`"$PYTHON" -c "import sys; print(sys.version_info[0:2] >= $MIN_VERSION and '1' or '');"`
+
+if [[ -z "$VERSION_OK" ]] ; then
+    PYTHON_VERSION=`"$PYTHON" -c "import sys; print('%s.%s' % sys.version_info[0:2]);"`
+    DISP_MIN_VERSION=`"$PYTHON" -c "print('%s.%s' % $MIN_VERSION)"`
+    echo "ERROR: Python $PYTHON_VERSION detected, but $DISP_MIN_VERSION+ is required."
+    echo "Please upgrade your Python distribution to install Databases."
+    exit 1
+fi
+
+REQUIREMENTS="requirements.txt"
+VENV="venv"
+PIP="$VENV/bin/pip"
+
+set -x
+"$PYTHON" -m venv "$VENV"
+"$PIP" install -r "$REQUIREMENTS"
+"$PIP" install -e .

scripts/lint

@@ -0,0 +1,15 @@
+#!/bin/sh -e
+
+export PREFIX=""
+if [ -d 'venv' ] ; then
+    export PREFIX="venv/bin/"
+fi
+
+set -x
+
+${PREFIX}autoflake --in-place --recursive httpcore tests
+${PREFIX}black httpcore tests
+${PREFIX}isort --multi-line=3 --trailing-comma --force-grid-wrap=0 --combine-as --line-width 88 --recursive --apply httpcore tests
+${PREFIX}mypy httpcore --ignore-missing-imports --disallow-untyped-defs
+
+scripts/clean

scripts/test

@@ -0,0 +1,12 @@
+#!/bin/sh -e
+
+export PACKAGE="httpcore"
+export PREFIX=""
+if [ -d 'venv' ] ; then
+    export PREFIX="venv/bin/"
+fi
+
+set -x
+
+PYTHONPATH=. ${PREFIX}pytest --ignore venv --cov tests --cov ${PACKAGE} --cov-report= ${@}
+${PREFIX}coverage report

tests/test_api.py

@@ -0,0 +1,96 @@
+import pytest
+
+import httpcore
+
+
+@pytest.mark.asyncio
+async def test_request():
+    response = await httpcore.request("GET", "http://example.com")
+    assert response.status_code == 200
+    assert response.body == b"Hello, world!"
+    assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_read_response():
+    response = await httpcore.request("GET", "http://example.com")
+
+    assert response.status_code == 200
+    assert response.body == b"Hello, world!"
+    assert response.is_closed
+
+    body = await response.read()
+
+    assert body == b"Hello, world!"
+    assert response.body == b"Hello, world!"
+    assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_stream_response():
+    response = await httpcore.request("GET", "http://example.com")
+
+    assert response.status_code == 200
+    assert response.body == b"Hello, world!"
+    assert response.is_closed
+
+    body = b''
+    async for part in response.stream():
+        body += part
+
+    assert body == b"Hello, world!"
+    assert response.body == b"Hello, world!"
+    assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_read_streaming_response():
+    response = await httpcore.request("GET", "http://example.com", stream=True)
+
+    assert response.status_code == 200
+    assert not hasattr(response, 'body')
+    assert not response.is_closed
+
+    body = await response.read()
+
+    assert body == b"Hello, world!"
+    assert response.body == b"Hello, world!"
+    assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_stream_streaming_response():
+    response = await httpcore.request("GET", "http://example.com", stream=True)
+
+    assert response.status_code == 200
+    assert not hasattr(response, 'body')
+    assert not response.is_closed
+
+    body = b''
+    async for part in response.stream():
+        body += part
+
+    assert body == b"Hello, world!"
+    assert not hasattr(response, 'body')
+    assert response.is_closed
+
+
+@pytest.mark.asyncio
+async def test_cannot_read_after_stream_consumed():
+    response = await httpcore.request("GET", "http://example.com", stream=True)
+
+    body = b''
+    async for part in response.stream():
+        body += part
+
+    with pytest.raises(httpcore.StreamConsumed):
+        await response.read()
+
+@pytest.mark.asyncio
+async def test_cannot_read_after_response_closed():
+    response = await httpcore.request("GET", "http://example.com", stream=True)
+
+    await response.close()
+
+    with pytest.raises(httpcore.ResponseClosed):
+        await response.read()