Autorefresh for Credit Recovery by Megha18jain · Pull Request #1710 · datacommonsorg/data

Megha18jain · 2025-11-11T10:24:26Z

No description provided.

gemini-code-assist · 2025-11-11T10:24:48Z

Summary of Changes

Hello @Megha18jain, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new, fully automated data import pipeline for credit recovery statistics sourced from the Civil Rights Data Collection (CRDC). The pipeline handles the entire process from data acquisition and preprocessing to mapping the cleaned data to Data Commons statistical variables. This ensures that relevant educational data is regularly updated and made available for analysis, enhancing the comprehensiveness of the dataset.

Highlights

New Data Import Pipeline: A complete pipeline for importing credit recovery data from the Civil Rights Data Collection (CRDC) has been added, establishing a new data source within the system.
Automated Data Acquisition: A new download.py script automates fetching CRDC data archives for years 2012-2025 directly from the official ED.gov website and organizes relevant files into a dedicated directory.
Data Preprocessing and Standardization: The creditprocess.py script cleans raw CRDC data by standardizing 'Yes'/'No' values to 1/0 in key columns and extracts academic years from filenames, preparing the data for further processing.
Statistical Variable Mapping and Configuration: New pvmap.csv and metadata.csv files define how raw data columns map to Data Commons statistical variables, including counts for schools offering/not offering credit recovery and student enrollment figures.
Automated Weekly Scheduling: A manifest.json file has been added to configure the entire credit recovery data import process to run automatically on a weekly cron schedule, ensuring regular updates.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the autorefresh functionality for Credit Recovery data, including scripts for downloading and processing data, along with necessary configuration files. The changes involve adding new Python scripts (download.py, creditprocess.py), a manifest.json for orchestration, and several CSV files for data mapping and metadata. Overall, the structure is clear, but there are several inconsistencies in file paths and naming between the README.md, manifest.json, and the Python scripts, which need to be addressed for correct operation. Additionally, a critical typo in pvmap.csv will prevent proper data processing, and some Python logic needs minor adjustments for correctness and maintainability. All new files are missing a trailing newline, which is a minor formatting issue.

gemini-code-assist · 2025-11-11T10:26:25Z

@@ -0,0 +1,8 @@
+key,,,,,,,,,,,,
+LEA_STATE,observationAbout,{Data},,,,,,,,,,
+SCH_CRED-ITRECOVERY_IND:1,statType,measuredValue,measuredProperty,count,populationType,School,schoolSubject,CreditRecoveryProgram,#Aggregate,sum,value,1,subjectOfferStatus,SubjectOffered


There is a typo in the key SCH_CRED-ITRECOVERY_IND:1. It should likely be SCH_CREDITRECOVERY_IND:1 to match the column name used in creditprocess.py. This typo will prevent the correct mapping of this property value.

SCH_CREDITRECOVERY_IND:1,statType,measuredValue,measuredProperty,count,populationType,School,schoolSubject,CreditRecoveryProgram,#Aggregate,sum,value,1,subjectOfferStatus,SubjectOffered

gemini-code-assist · 2025-11-11T10:26:25Z

+            "scripts": [
+                "download.py",
+                "creditprocess.py",
+                "../../tools/statvar_importer/stat_var_processor.py --input_data=input_credit/*.csv --pv_map=pvmap.csv --config_file=metadata.csv --places_resolved_csv=places_resolved.csv --output_path=output/credit_output"


The output_path output/credit_output implies that the output files will be named credit_output.csv, credit_output.tmcf, etc. This is inconsistent with the README.md which refers to output/output_credit.csv and output_pro/ directory. Please ensure consistency across all documentation and configuration.

gemini-code-assist · 2025-11-11T10:26:25Z

+            "import_inputs": [
+                {
+                    "template_mcf": "output/credit_output.tmcf",
+                    "cleaned_csv": "output_files/credit_output.csv"


The cleaned_csv path output_files/credit_output.csv is inconsistent with the output_path specified in the scripts section (output/credit_output). This will likely lead to the importer not finding the generated CSV file. It should probably be output/credit_output.csv.

Suggested change

"cleaned_csv": "output_files/credit_output.csv"

"cleaned_csv": "output/credit_output.csv"

gemini-code-assist · 2025-11-11T10:26:25Z

+            year_match = re.search(r'(\d{4})-(\d{2})', filename)
+            if not year_match:
+                print(f"Skipping '{filename}': Year not found in filename.")
+                continue
+            year = f"20{year_match.group(2)}"


The year extraction logic might be inconsistent with how download.py names the files. download.py uses year_str (e.g., "2012-13") as the year prefix in the filename. The regex re.search(r'(\d{4})-(\d{2})', filename) will match "2012-13", where group(1) is "2012" and group(2) is "13". The current year = f"20{year_match.group(2)}" would result in "2013". If the intention is to use the start year of the academic period (e.g., 2012 for 2012-13 data), then year_match.group(1) should be used instead.

Suggested change

year_match = re.search(r'(\d{4})-(\d{2})', filename)

if not year_match:

print(f"Skipping '{filename}': Year not found in filename.")

continue

year = f"20{year_match.group(2)}"

year_match = re.search(r'(\d{4})-(\d{2})', filename)

if not year_match:

print(f"Skipping '{filename}': Year not found in filename.")

continue

year = year_match.group(1)

gemini-code-assist · 2025-11-11T10:26:25Z

+
+if __name__ == "__main__":
+    os.chdir(SCRIPT_DIRECTORY)
+    main()


This file is missing a trailing newline, which is a common best practice for Python files.

Suggested change

main()

os.chdir(SCRIPT_DIRECTORY)

main()

gemini-code-assist · 2025-11-11T10:26:26Z

+    -   `output.csv`: The main data file with aggregated statistics.
+    -   `output.tmcf`: Template MCF file for the import.
+    -   `output_stat_vars.mcf`: Statistical variable definitions.
+    -   `output_stat_vars_schema.mcf`: Schema for the statistical variables.


The output filenames listed here (output.csv, output.tmcf, output_stat_vars.mcf, output_stat_vars_schema.mcf) are inconsistent with the output_path specified in manifest.json (output/credit_output). The manifest will generate files like output/credit_output.csv, output/credit_output.tmcf, etc. Please update these entries in the README for accuracy.

gemini-code-assist · 2025-11-11T10:26:26Z

@@ -0,0 +1,3 @@
+parameter,value
+output_columns,"observationAbout, observationDate, value, variableMeasured"
+header_rows,1


This file is missing a trailing newline, which is a common best practice for CSV files.

output_columns,"observationAbout, observationDate, value, variableMeasured" header_rows,1

gemini-code-assist · 2025-11-11T10:26:26Z

+MP,geoId/69
+PR,geoId/72
+UM,geoId/74
+VI,geoId/78


This file is missing a trailing newline, which is a common best practice for CSV files.

PR,geoId/72 UM,geoId/74 VI,geoId/78

gemini-code-assist · 2025-11-11T10:26:26Z

+-   `creditprocess.py`: Script to process the raw data and prepare it for the final import.
+-   `files/credit/`: Directory containing the raw, downloaded credit recovery data files.
+-   `input_credit/`: Directory containing the processed, cleaned CSV files.
+-   `output_pro/`: Directory containing the final, aggregated output files for import.


The directory name output_pro/ is inconsistent with the output_path specified in manifest.json (output/credit_output). Please update the README to reflect the correct output directory.

gemini-code-assist · 2025-11-11T10:26:26Z

+SCH_CREDITRECOVERYENR,value,{Number},statType,measuredValue,measuredProperty,count,populationType,Student,schoolSubject,CreditRecoveryProgram,enrollmentStatus,Enrolled,,
+JJ:0,juvenileJusticeFacilityStatus,NotJuvenileJusticeFacility,,,,,,,,,,
+JJ:1,juvenileJusticeFacilityStatus,JuvenileJusticeFacility,,,,,,,,,,
+year,observationDate,{Data},,,,,,,,,,


This file is missing a trailing newline, which is a common best practice for CSV files.

JJ:0,juvenileJusticeFacilityStatus,NotJuvenileJusticeFacility,,,,,,,,,, JJ:1,juvenileJusticeFacilityStatus,JuvenileJusticeFacility,,,,,,,,,, year,observationDate,{Data},,,,,,,,,,

ajaits · 2025-12-30T10:12:36Z

pls check if the 15k lines are produced as output for the 48 lines in input.

ajaits · 2025-12-30T10:16:54Z

+                "../../tools/statvar_importer/stat_var_processor.py --input_data=input_credit/*.csv --pv_map=pvmap.csv --config_file=metadata.csv  --output_path=output/credit_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf "
+            ],
+            "source_files": [
+                "input_credit/*.csv"


pls copy the .zip downloaded form source to GCS as well.

Megha18jain added 2 commits November 11, 2025 10:23

Autorefresh for Credit Recovery

51ebabf

Merge branch 'master' into credit

013c22e

gemini-code-assist Bot reviewed Nov 11, 2025

View reviewed changes

Megha18jain added 2 commits November 12, 2025 12:19

update pvmap

5905361

Merge branch 'credit' of github.com:Megha18jain/data into credit

5f91043

Megha18jain requested a review from krishnaswamypradeep November 12, 2025 12:24

krishnaswamypradeep reviewed Nov 13, 2025

View reviewed changes

Megha18jain added 2 commits November 19, 2025 15:29

Updated files

660d67b

Updated files

352e478

krishnaswamypradeep approved these changes Nov 20, 2025

View reviewed changes

Megha18jain added 2 commits November 24, 2025 05:04

updated download script

d6dc57d

Changed import name

49948d5

ajaits reviewed Dec 30, 2025

View reviewed changes

	"cleaned_csv": "output_files/credit_output.csv"
	"cleaned_csv": "output/credit_output.csv"

Conversation

Megha18jain commented Nov 11, 2025

Uh oh!

gemini-code-assist Bot commented Nov 11, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajaits Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

ajaits Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants