feat: [BE] Add pagination to published components endpoint

[Mbeaulne]

Description

The GET /api/published_components/ endpoint currently returns all components with no pagination. There are existing TODO comments in the code acknowledging this:

# TODO: Implement paging
# TODO: Implement filtering/search
# TODO: Implement visibility/access control

As the component library grows and a Components tab is added to the homepage, this needs pagination support.

Part of TangleML/tangle-ui#1963

Implementation

Follow the same pagination pattern used by GET /api/pipeline_runs/:

• Add page_size query param (default 20)
• Add page_token query param for cursor-based pagination (base64-encoded offset)
• Add next_page_token to response
• Add total_count to response (component count is much smaller than runs, so a full count is fine here)

Response change

@dataclasses.dataclass(kw_only=True)
class ListPublishedComponentsResponse:
    published_components: list[PublishedComponentResponse]
    next_page_token: str | None = None  # NEW
    total_count: int | None = None      # NEW

Relevant file

• component_library_api_server.py — list() method

[yuechao-qin]

Just trying to get a sense of priority / urgency:

• how many components are being returned now without pagination?
• do we expect the growth rate to be N times in the upcoming weeks?
• can the UI mitigate this in other ways while waiting for this feature?

[Ark-kun]

this needs pagination support

Agreed.

Add total_count to response (component count is much smaller than runs, so a full count is fine here)

I'm a bit skeptical here.

[Mbeaulne]

Good questions — here's where things stand:

• How many components are being returned now? Under 100 currently.
• Do we expect N times growth in the upcoming weeks? Honestly not sure — it's possible. Pinging @Ark-kun for his take on expected growth.
• Can the UI mitigate this in the meantime? Likely yes — we could implement front-end pagination over the full returned list as a stopgap. That said, if the component count does grow significantly it won't scale well, which is part of the motivation for doing this properly on the backend.

[Mbeaulne]

Curious what's driving the skepticism — is it the assumption that component counts will stay small enough to make a full total_count cheap? How many components are we realistically expecting over time?

[yuechao-qin]

@Mbeaulne i think this is a good candidate for my goal of creating an experimental API in oasis-backend. I will need to 1) create a design doc and review it with the team and then 2) implement it. Curious how urgent are you wanting this feature. My thought is might have an release by mid April if everything goes smoothly.