[SPARK-52785][PYTHON] Simplifying super() syntax in PySpark

[simolanayak]

What changes were proposed in this pull request?

9 instances of super(Class, cls) syntax removed and simplified to super()
These are present in the files:

• _globals.py
• sql/connect/plan.py
• sql/pandas/serializers.py
• sql/tests/test_udf.py
• streaming/tests/test_listener.py

Why are the changes needed?

Issue SPARK-52785
Remove Python 2 support

Does this PR introduce any user-facing change?

No behavior changes

How was this patch tested?

github actions

Was this patch authored or co-authored using generative AI tooling?

No

[huaxingao]

@simolanayak Can we keep the PR title short (e.g., SPARK-52785: Simplify super() usage in PySpark) and move the file list / ‘N instances’ details into the What changes were proposed section? The current title is a bit long.

[HyukjinKwon]

LG @gaogaotiantian

[huaxingao]

LGTM! Thanks @simolanayak for your contribution!

[gaogaotiantian]

Most of the changes in this PR are incorrect (semantics equivalent wise). I did this once and fixed most of the super() usages. The ones I left out were intentional.

Basically

class A(B):
    def f(self):
        # equivalent to
        # super(A, self).f()
        # but NOT
        # super(B, self).f()
        super().f()

So we can only change the ones that explicitly use the classes the methods are defined in (not their base class). I believe at least a few of them are super intentional to skip the methods of one of the parent classes.

That being said, there are a few of the changes are correct. So we should keep those.

[gaogaotiantian]

I believe the ones in serializers should be reverted too.

[HyukjinKwon]

Let's also keep the PR description updated:

_globals.py
sql/connect/plan.py
sql/pandas/serializers.py
sql/tests/test_udf.py
streaming/tests/test_listener.py