Incorrect handling of overloads with `defaultdict` and `MutableMapping`

[Dr-Irv]

Bug Report
This came from a PR with pandas-stubs. The overloads with defaultdict and MutableMapping aren't working correctly.

Requires a stub file and a test file.

To Reproduce

File defdictover.pyi :

from collections import defaultdict
from typing import TypeVar, MutableMapping, overload, Literal, Hashable, Any

_T_MUTABLE_MAPPING = TypeVar("_T_MUTABLE_MAPPING", bound=MutableMapping, covariant=True)

@overload
def to_dict(
    orient: Literal["records"],
    *,
    into: type[dict] = ...,
) -> list[dict[Hashable, Any]]: ...
@overload
def to_dict(
    orient: Literal["index"],
    *,
    into: defaultdict,
) -> defaultdict[Hashable, dict[Hashable, Any]]: ...
@overload
def to_dict(
    orient: Literal["index"],
    *,
    into: MutableMapping | type[MutableMapping],
) -> MutableMapping[Hashable, dict[Hashable, Any]]: ...

File tstdefdictover.py :

from collections import defaultdict
from typing import Any
from defdictover import to_dict

target: defaultdict[Any, list] = defaultdict(list)

reveal_type(to_dict("index", into=target))

Then mypy tstdefdictover.py produces:

tstdefdictover.py:7: note: Revealed type is "Any"

Expected Behavior

With pyright, we get:

tstdefdictover.py:7:13 - information: Type of "to_dict("index", into=target)" is "defaultdict[Hashable, dict[Hashable, Any]]"

Actual Behavior

See above - it sees the type as Any

Your Environment

• Mypy version used: 1.17
• Mypy command-line flags: None
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: 3.12

[brianschubert]

Smaller reproducer:

from typing import overload, Any

class A[T]: ...
class B[T](A[T]): ...

@overload
def foo(x: B[Any]) -> B[Any]: ...
@overload
def foo(x: A[Any]) -> A[Any]: ...
def foo(x: A[Any]) -> A[Any]: raise NotImplementedError

b: B[Any]
reveal_type(foo(b))  # N: Revealed type is "Any"

edit: only the Any at the callsite appears to be relevant:

from typing import overload, Any

class A[T]: ...
class B[T](A[T]): ...

@overload
def foo(x: B[int]) -> B[int]: ...
@overload
def foo(x: A[int]) -> A[int]: ...
def foo(x: A[int]) -> A[int]: raise NotImplementedError

b: B[Any]
reveal_type(foo(b))  # N: Revealed type is "Any"

[brianschubert]

(nevermind about the crash - that was due to an unrelated issue caused by a typo in the reproducer. I'll open a separate issue for it. edit: #19526)

[sterliakov]

Is this really a bug? While this specific example seems unambiguous, consider the following:

from typing import overload, Any

class A[T]: ...
class B[T](A[T]): ...

@overload
def foo(x: B[int]) -> B[int]: ...
@overload
def foo(x: A[str]) -> A[str]: ...
def foo(x: A[int] | A[str]) -> A[int] | A[str]: raise NotImplementedError

b: B[Any]
reveal_type(foo(b))  # N: Revealed type is "???"

What should ??? be? In my mind it should be Any as "idk, have your any" fallback. There is no way I could expect B[int] there. I can also understand inferring union (A[str] | B[int]), join (A[int | str]) or A[Any] (the latter with a bit of a stretch), but picking the first overload in such case (where it's Any what decides which overload to pick) is unreasonable IMO.

I don't have enough time to dig the overloads spec now (and I still remember how controversial that page was), but I prefer mypy behavior (Any) over pyright behavior (B[int]) even if that means nonconformance.

Any in the last reproducer by @brianschubert "causes overload ambiguity", we look for such conditions explicitly:

mypy/mypy/checkexpr.py

Lines 6598 to 6653 in af92439

	def any_causes_overload_ambiguity(
	items: list[CallableType],
	return_types: list[Type],
	arg_types: list[Type],
	arg_kinds: list[ArgKind],
	arg_names: Sequence[str | None] | None,
	) -> bool:
	"""May an argument containing 'Any' cause ambiguous result type on call to overloaded function?
	Note that this sometimes returns True even if there is no ambiguity, since a correct
	implementation would be complex (and the call would be imprecisely typed due to Any
	types anyway).
	Args:
	items: Overload items matching the actual arguments
	arg_types: Actual argument types
	arg_kinds: Actual argument kinds
	arg_names: Actual argument names
	"""
	if all_same_types(return_types):
	return False
	actual_to_formal = [
	map_formals_to_actuals(
	arg_kinds, arg_names, item.arg_kinds, item.arg_names, lambda i: arg_types[i]
	)
	for item in items
	]
	for arg_idx, arg_type in enumerate(arg_types):
	# We ignore Anys in type object callables as ambiguity
	# creators, since that can lead to falsely claiming ambiguity
	# for overloads between Type and Callable.
	if has_any_type(arg_type, ignore_in_type_obj=True):
	matching_formals_unfiltered = [
	(item_idx, lookup[arg_idx])
	for item_idx, lookup in enumerate(actual_to_formal)
	if lookup[arg_idx]
	]
	matching_returns = []
	matching_formals = []
	for item_idx, formals in matching_formals_unfiltered:
	matched_callable = items[item_idx]
	matching_returns.append(matched_callable.ret_type)
	# Note: if an actual maps to multiple formals of differing types within
	# a single callable, then we know at least one of those formals must be
	# a different type then the formal(s) in some other callable.
	# So it's safe to just append everything to the same list.
	for formal in formals:
	matching_formals.append(matched_callable.arg_types[formal])
	if not all_same_types(matching_formals) and not all_same_types(matching_returns):
	# Any maps to multiple different types, and the return types of these items differ.
	return True
	return False

[erictraut]

Is this really a bug?

I think mypy is working correctly here and is in compliance with the overload call evaluation algorithm defined in the typing spec. Pyright is nonconformant in this case. This is a known issue in pyright.

[Dr-Irv]

Is this really a bug?

I think mypy is working correctly here and is in compliance with the overload call evaluation algorithm defined in the typing spec. Pyright is nonconformant in this case. This is a known issue in pyright.

@erictraut which example are you referring to? The original one I posted, or the second example: #19525 (comment)

[erictraut]

which example are you referring to

Both. The second example is a simplified version of the first.

[cmp0xff]

Both. The second example is a simplified version of the first.

Could you elaborate why the first example is correct? In my opinion, inserting b: B[Any] into foo should be evaluated by the first overload

@overload
def foo(x: B[Any]) -> B[Any]: ...

and should give the type B[Any].

[JelleZijlstra]

Any should be seen as a standin for some unknown static type (a "materialization"; see the spec: https://typing.python.org/en/latest/spec/concepts.html#materialization ). This example uses a number of distinct Anys, and in general we have to infer a type that works for all possible materializations. In this case, there are many possible materializations that would make it so the call with B[Any] as an argument hits the second overload. For example, the Any in the argument and in the parameter type of the second overload could materialize to one type (e.g., int), while the Any in the parameter type of the first overload could materialize to another type (e.g., str).

This means that we hit step 5 of the overload evaluation spec and there are multiple matching overloads with distinct return types. Therefore, we return Any.

[Dr-Irv]

Any should be seen as a standin for some unknown static type (a "materialization"; see the spec: https://typing.python.org/en/latest/spec/concepts.html#materialization ). This example uses a number of distinct Anys, and in general we have to infer a type that works for all possible materializations. In this case, there are many possible materializations that would make it so the call with B[Any] as an argument hits the second overload. For example, the Any in the argument and in the parameter type of the second overload could materialize to one type (e.g., int), while the Any in the parameter type of the first overload could materialize to another type (e.g., str).

This means that we hit step 5 of the overload evaluation spec and there are multiple matching overloads with distinct return types. Therefore, we return Any.

Can you make the same analysis with my original example involving defaultdict and MutableMapping, because I'm having a hard time seeing how your analysis applies in that case.