Add support for query tokenization parameters by matt-bernhardt · Pull Request #435 · MITLibraries/timdex-ui

matt-bernhardt · 2026-07-22T16:45:46Z

Developer

Accessibility

ANDI or WAVE has been run in accordance to our guide.
This PR contains no changes to the view layer.
New issues flagged by ANDI or WAVE have been resolved.
New issues flagged by ANDI or WAVE have been ticketed (link in the Pull Request details above).
No new accessibility issues have been flagged.

New ENV

All new ENV is documented in README.
All new ENV has been added to Heroku Pipeline, Staging and Prod.
ENV has not changed.

Approval beyond code review

UXWS/stakeholder approval has been confirmed.
UXWS/stakeholder review will be completed retroactively.
UXWS/stakeholder review is not needed.

Additional context needed to review

E.g., if the PR includes updated dependencies and/or data
migration, or how to confirm the feature is working.

Code Reviewer

Code

I have confirmed that the code works as intended.
Any CodeClimate issues have been fixed or confirmed as
added technical debt.

Documentation

The commit message is clear and follows our guidelines
(not just this pull request message).
The documentation has been updated or is unnecessary.
New dependencies are appropriate or there were no changes.

Testing

There are appropriate tests covering any new functionality.
No additional test coverage is required.

qltysh · 2026-07-22T16:47:31Z

❌ 21 blocking issues (22 total)

Tool	Category	Rule	Count
rubocop	Lint	Method has too many lines. [18/10]	5	❌
rubocop	Style	Prefer single-quoted strings when you don't need string interpolation or special symbols.	4	❌
rubocop	Lint	Assignment Branch Condition size for `query\_timdex` is too high. [<9, 18, 16> 25.71/17]	3	❌
rubocop	Style	Line is too long. [130/120]	2	❌
rubocop	Lint	Cyclomatic complexity for `query\_timdex` is too high. [10/7]	1	❌
rubocop	Lint	Perceived complexity for `query\_timdex` is too high. [11/8]	1	❌
rubocop	Style	`%w`-literals should be delimited by `\[` and `\]`.	1	❌
rubocop	Style	Use `tuning\_params\.intersect?\(query\.keys\)` instead of `tuning_params.any? do \|term\|

  query\.keys\.include?\(term\)
end`\.|1|❌|

|rubocop|Lint|Class has too many lines. [149/100]|1|❌|
|rubocop|Style|Freeze mutable objects assigned to constants.|1|❌|
|rubocop|Lint|Use SCREAMING_SNAKE_CASE for constants.|1|❌|
|qlty|Structure|Function with high complexity (count = 6): process_timdex_response|1||

coveralls · 2026-07-22T18:55:17Z

Coverage Report for CI Build 30507589899

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage decreased (-0.7%) to 97.651%

Details

Coverage decreased (-0.7%) from the base build.
Patch coverage: 12 uncovered changes across 1 file (20 of 32 lines covered, 62.5%).
No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
app/controllers/search_controller.rb	23	11	47.83%
Total (5 files)	32	20	62.5%

Coverage Regressions

No coverage regressions found.

Coverage Stats


Relevant Lines:	1575
Covered Lines:	1538
Line Coverage:	97.65%
Coverage Strength:	85.13 hits per line

💛 - Coveralls

qltysh · 2026-07-22T20:18:00Z

+    extract_tokenization_params(enhanced_query)
    @query['index'] = ENV.fetch('TIMDEX_INDEX', nil)
    @query['booleanType'] = enhanced_query[:booleanType]
    @query.compact!


Found 2 issues:

1. Assignment Branch Condition size for initialize is too high. [<7, 20, 5> 21.77/17] [rubocop:Metrics/AbcSize]

2. Method has too many lines. [15/10] [rubocop:Metrics/MethodLength]

This is necessary to bypass the schema validation that comes for free with the graphql-client gem. No tests yet because this is super rough.

qltysh · 2026-07-30T02:09:22Z

+      #   errors: raw.errors.details.to_h
+      # }
+      process_timdex_response(raw, query)
    end


Found 4 issues:

1. Assignment Branch Condition size for query_timdex is too high. [<9, 18, 16> 25.71/17] [rubocop:Metrics/AbcSize]

2. Cyclomatic complexity for query_timdex is too high. [10/7] [rubocop:Metrics/CyclomaticComplexity]

3. Method has too many lines. [18/10] [rubocop:Metrics/MethodLength]

4. Perceived complexity for query_timdex is too high. [11/8] [rubocop:Metrics/PerceivedComplexity]

qltysh · 2026-07-30T02:09:22Z

@@ -238,6 +242,29 @@ def execute_geospatial_query(query)
    end


Method has too many lines. [11/10] [rubocop:Metrics/MethodLength]

qltysh · 2026-07-30T02:09:22Z

+      http.request(req)
+    end
+
+    JSON.parse(res.body)


Found 2 issues:

1. Assignment Branch Condition size for execute_tuning_query is too high. [<8, 15, 1> 17.03/17] [rubocop:Metrics/AbcSize]

2. Method has too many lines. [13/10] [rubocop:Metrics/MethodLength]

qltysh · 2026-07-30T02:09:22Z

  end
+
+  def tuning_request?(query)
+    tuning_params = %w(semanticMustBoostThreshold semanticDropBoostThreshold semanticShortQueryMaxTokens)


%w-literals should be delimited by [ and ]. [rubocop:Style/PercentLiteralDelimiters]

qltysh · 2026-07-30T02:09:22Z

+    tuning_params = %w(semanticMustBoostThreshold semanticDropBoostThreshold semanticShortQueryMaxTokens)
+
+    tuning_params.any? do |term|
+      query.keys.include?(term)


Use tuning_params.intersect?(query.keys) instead of tuning_params.any? do |term| query.keys.include?(term) end. [rubocop:Style/ArrayIntersect]

qltysh · 2026-07-30T02:09:22Z

+    if tuning_request?(query)
+      {
+        data: (raw["data"].nil? ? {} : raw["data"]),
+        errors: (raw["errors"].nil? ? {} : raw["errors"])


Found 2 issues:

1. Prefer single-quoted strings when you don't need string interpolation or special symbols. [rubocop:Style/StringLiterals]

2. Prefer single-quoted strings when you don't need string interpolation or special symbols. [rubocop:Style/StringLiterals]

qltysh · 2026-07-30T02:09:22Z

+        }
+      }
+    }
+  GRAPHQL


Class has too many lines. [149/100] [rubocop:Metrics/ClassLength]

qltysh · 2026-07-30T02:09:22Z

@@ -0,0 +1,151 @@
+class TimdexTuning
+  TuningQuery = <<-GRAPHQL


Found 2 issues:

1. Freeze mutable objects assigned to constants. [rubocop:Style/MutableConstant]

2. Use SCREAMING_SNAKE_CASE for constants. [rubocop:Naming/ConstantName]

qltysh · 2026-07-30T02:09:23Z

+  #   drop_alt = 0.4
+
+  #   VCR.use_cassette('default tuning for stock query') do
+  #     get "/results?q=#{query}&semanticMustBoostThreshold=#{must_default}&semanticDropBoostThreshold=#{drop_default}&tab=timdex"


Line is too long. [130/120] [rubocop:Layout/LineLength]

qltysh · 2026-07-30T02:09:23Z

+  #   end
+
+  #   VCR.use_cassette('alternate tuning for stock query') do
+  #     get "/results?q=#{query}&semanticMustBoostThreshold=#{must_alt}&semanticDropBoostThreshold=#{drop_alt}&tab=timdex"


Line is too long. [122/120] [rubocop:Layout/LineLength]

First work towards supporting tokenization parameters

ac699fe

mitlib temporarily deployed to timdex-ui-pi-use-636-1nr9riqsh July 22, 2026 16:48 Inactive

Remove tokenizer tuning parameters

b3adbae

mitlib temporarily deployed to timdex-ui-pi-use-636-1nr9riqsh July 22, 2026 18:53 Inactive

Move tokenization parameters to their own extraction method

85b9120

mitlib temporarily deployed to timdex-ui-pi-use-636-1nr9riqsh July 22, 2026 20:16 Inactive

qltysh Bot reviewed Jul 22, 2026

View reviewed changes

mitlib temporarily deployed to timdex-ui-pi-use-636-06a3xafm9 July 28, 2026 20:37 Inactive

Build separate pathway for tuning queries

8fa35ef

This is necessary to bypass the schema validation that comes for free with the graphql-client gem. No tests yet because this is super rough.

mitlib deployed to timdex-ui-pi-use-636-06a3xafm9 July 30, 2026 02:07 View deployment

qltysh Bot reviewed Jul 30, 2026

View reviewed changes

Uh oh!

Conversation

matt-bernhardt commented Jul 22, 2026

Developer

Accessibility

New ENV

Approval beyond code review

Additional context needed to review

Code Reviewer

Code

Documentation

Testing

Uh oh!

qltysh Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 21 blocking issues (22 total)

Uh oh!

coveralls commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for CI Build 30507589899

Coverage decreased (-0.7%) to 97.651%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

qltysh Bot Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

qltysh Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qltysh Bot commented Jul 22, 2026 •

edited

Loading

coveralls commented Jul 22, 2026 •

edited

Loading