Describe the bug, including details regarding any error messages, version, and platform.
pa.concat_tables accepts **kwargs but only ever inspects promote. Every other keyword is dropped without a warning or an error:
>>> import pyarrow as pa
>>> t1 = pa.Table.from_pydict({'a': [1.0]})
>>> t2 = pa.Table.from_pydict({'a': [2.0]})
>>> pa.concat_tables([t1, t2], totally_bogus_kwarg=123).num_rows
2
https://github.com/apache/arrow/blob/515410b2a1/python/pyarrow/table.pxi#L6324-L6338
Two ways this bites:
- A typo in a keyword is silently ignored rather than raising
TypeError, which is what a reader would expect from a normal Python signature.
unify_schemas in particular looks like it should work. It is a real field on the underlying ConcatenateTablesOptions, R's concat_tables() takes a unify_schemas argument, and the C++ option is what promote_options sets on the caller's behalf. So this reads as if it silently does the opposite of what was asked:
>>> pa.concat_tables(
... [pa.Table.from_pydict({'a': [1]}), pa.Table.from_pydict({'a': [1.0]})],
... unify_schemas=False, promote_options="permissive").schema.field('a').type
DataType(double)
Schemas were unified despite unify_schemas=False.
This surfaced in GH-38809, where the reporter passed unify_schemas=True and reasonably assumed it had an effect. The type-promotion half of that issue is fixed; this half is not.
A fix could be either to raise TypeError on unrecognised keywords, keeping **kwargs only for the deprecated promote, or to retire promote altogether (it has warned FutureWarning since 14.0.0) and give concat_tables an explicit signature with no **kwargs. The second is a breaking change and would need its own deprecation window; the first is not.
Reproduced on pyarrow 25.0.1, macOS/arm64, and the code path is unchanged on main.
Component(s)
Python
Describe the bug, including details regarding any error messages, version, and platform.
pa.concat_tablesaccepts**kwargsbut only ever inspectspromote. Every other keyword is dropped without a warning or an error:https://github.com/apache/arrow/blob/515410b2a1/python/pyarrow/table.pxi#L6324-L6338
Two ways this bites:
TypeError, which is what a reader would expect from a normal Python signature.unify_schemasin particular looks like it should work. It is a real field on the underlyingConcatenateTablesOptions, R'sconcat_tables()takes aunify_schemasargument, and the C++ option is whatpromote_optionssets on the caller's behalf. So this reads as if it silently does the opposite of what was asked:Schemas were unified despite
unify_schemas=False.This surfaced in GH-38809, where the reporter passed
unify_schemas=Trueand reasonably assumed it had an effect. The type-promotion half of that issue is fixed; this half is not.A fix could be either to raise
TypeErroron unrecognised keywords, keeping**kwargsonly for the deprecatedpromote, or to retirepromotealtogether (it has warnedFutureWarningsince 14.0.0) and giveconcat_tablesan explicit signature with no**kwargs. The second is a breaking change and would need its own deprecation window; the first is not.Reproduced on pyarrow 25.0.1, macOS/arm64, and the code path is unchanged on
main.Component(s)
Python