[Klavionik]

Hey! I'm trying to add support for the httpx2 package, following the guidelines from the authors.

This PR allows users of the package to seamlessly migrate their application and test code from httpx, just by updating the package. Users who have no intentions to migrate will be fine as well: the added _compat.py module tries to use httpx2 first, but falls back to httpx.

This is a highly anticipated change I assume, so I'm looking forward for a review and any comments/suggestions are very welcome.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[bagowix]

perhaps it's worth doing this as an extra?

[Colin-b]

I would rather remove both httpx and httpx2 as dependencies. Make them optional (as the suggestion above, an extra dependency for each)

[bagowix]

I would leave this choice up to the user and not talk about the need, because migration does not happen so quickly.

"Neither `httpx2` nor the legacy `httpx` package is installed.\n"
"Install `httpx2` with:\n"
"    $ pip install httpx2\n"

[bagowix]

@Colin-b Hi! Could we prioritize this, if possible, please?

[Klavionik]

@Colin-b Kind reminder. 🙏🏻

[Colin-b]

I feel like it needs more work, I envision the following in term of support:

1. only httpx2 installed -> valid
2. only httpx installed -> valid
3. both installed -> use httpx2

This would need to be documented, and ofc the test suite will need to be updated, same for the CI that will need to run in 3 different dependency setup.

I am however not asking you to do it, I can review this proposal a bit more and add those changes.

[Klavionik]

I could work on these changes if it helps.

So, the plan would be:

1. Turn httpx and httpx2 into extra dependencies. pip install pytest-httpx will then install just this package.
2. Update the docs to explain the new ways to install pytest-httpx.
3. Update the test suite and the CI config to run in three environments: only httpx, only httpx2, httpx + httpx2. Perhaps add a test case to cover the compatibility layer itself?

Is that correct?

I envision the following in term of support:
...

I believe it works exactly this way right now.

[Colin-b]

If you feel like it, be my guest!
Note that I made a mistake in my suggestion when both are installed, and I think it makes it slightly less straightforward to handle.
When both dependencies are installed, we should mock both, I did not put thoughts into the actual implementation but I would like to avoid as much code duplication as possible.

[Klavionik]

I'm struggling to understand what would be the use case of this. I mean, why would anyone install two basically identical libraries under different names, and use both at the same time?

Oh, nevermind, I got this. 😌 Anyway, yeah, I gotta try, but this is indeed trickier than I expected.

[Colin-b]

Hi @Klavionik , just letting you know that I am also looking at introducing support for both httpx and httpx2 on my end. I am far from done, and I might not invest too much time into it right now so feel free to continue pushing for this.

It requires quite an extensive amount of changes to support all 3 combination (httpx alone, httpx2 alone, both httpx and httpx2 installed and mocked) so I don't really have a good view on the direction I want to take. If you want to start with only support for one or the other it is fine as well as it is obviously way easier.

[Klavionik]

@Colin-b So, it’s a lot of change, and I feel like I need to provide some kind of an overview.

The approach I’ve tried here is to collect httpx modules as first-class citizens and expose them as “backends” via a helper function get_backends. Then at fixture setup, I apply mocking for each backend, capturing it within a mock closure. When handling requests, HTTPXMock receives the corresponding backend and sets it as a private attribute of the request, to retrieve and use later when we need to compare or construct httpx-specific objects (Response, URL, etc.). Basically, that is the whole idea.

To make it work with any combination of httpx and httpx2, I have decoupled the library and the tests from any global runtime dependency on httpx. A module-level httpx is only used for type hints. Except for the tests that have to run with both backends installed, of course.

I’ve tried to be as subtle as possible, keeping method signatures the same, hence the usages of get_backend all over the place. It’s probably not that neat as injecting the backend, but I would argue that in this case it doesn't matter that much. And I think this way it will be easier to remove them when support for httpx is dropped in the future.

To run the tests in all three environments I’ve used tox, both locally and in CI. It runs ok on my machine, but I could not verify the CI (for obvious reasons).

P.S. And as for the quality gate failing because of code duplication - it's in the tests, and I'm not sure if there's a better way.

[Klavionik]

I've found out that there was a missing piece in the solution: public API requires type hints that allow for both httpx and httpx2. So I've added a new commit updating some of the the type hints.

Upd.: Not sure I'm happy about the return type of get_request(s) though. The current signature may complicate things for the client code, requiring to use the other library to correctly type hint functions that receive the request (or use type guards).

def process_request(request: httpx.Request | httpx2.Request):
    ...

request = httpx_mock.get_request()

assert request is not None
process_request(request)

On the other hand, right now I can't think of a different way to handle this. Maybe I'll try having a "generic httpx request" type that replicates the interface of a true request, but that's a lot of the API surface to cover. Dunno.

[sonarqubecloud]

Quality Gate failed

Failed conditions
4.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud