Conversation
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
mbani01
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
services/apps/snowflake_connectors/src/integrations/cvent/transformer.ts
Outdated
Show resolved
Hide resolved
| redis: { | ||
| enabled: true, | ||
| }, | ||
| } |
There was a problem hiding this comment.
Temporal worker disabled in configuration
High Severity
ServiceWorker is configured with temporal.enabled: false, but this service’s purpose is to run Temporal workflows/activities. If the archetype honors that flag, the worker won’t connect/register activities, so cron-triggered workflows on the snowflakeConnectors task queue never get processed.
| verified: true, | ||
| sourceId, | ||
| }) | ||
| } |
There was a problem hiding this comment.
Cvent identity mapping uses wrong fields
Medium Severity
The identity-building branch is keyed off userName, but it always adds an email identity with value: email (which can be undefined), and the else branch sets MemberIdentityType.USERNAME while still using value: email. This can emit invalid/incorrect member identities and misattribute users.
1 participant
This pull request introduces a new "snowflake connectors" service for exporting data to Snowflake and managing export jobs. It adds the necessary database schema, service configuration, Docker setup, and initial implementation for export activities and scheduling. The main themes are the creation of the Snowflake export infrastructure, integration with the job scheduler, and updates to the monorepo's dependency management.
Snowflake Export Infrastructure
integration."snowflakeExportJobs"to track Snowflake export jobs, including fields for platform, S3 path, timestamps, and error handling, along with relevant indexes.s3_pathcolumn to prevent duplicate export entries.exportStartedAttimestamp and ametricsJSONB column, while removing thetotalRowsandtotalBytescolumns to support more flexible export metrics. [1] [2]exportActivity.ts, which performs the Snowflake export and writes job metadata. [1] [2]Service and Job Scheduling
@crowd/snowflake-connectorswith its own dependencies, scripts, and Docker configuration. [1] [2] [3] [4]Monorepo Dependency and Lockfile Updates
pnpm-lock.yamlto include the newsnowflake_connectorsapp and its dependencies, and adjusted AWS SDK dependency resolutions for compatibility. [1] [2] [3] [4] [5] [6] [7] [8] [9]Other Platform/Activity Updates
activityTypestable for event registration and attendance.@joanagmaia you can help reviewing the export SQL and transformation
Note
Medium Risk
Adds new data pipeline components (Temporal workflows, S3 access, and DB job-claiming) and introduces new schema/migrations, which could impact export correctness and operational stability if misconfigured or if job claiming/transforming misbehaves.
Overview
Adds end-to-end Snowflake export infrastructure: a new
integration."snowflakeExportJobs"table (with indexes/uniques3_path,exportStartedAt, and JSONBmetrics) to track export batches, processing state, and errors.Introduces a new
services/apps/snowflake_connectorsworker that runs Temporal workflows toCOPY INTOS3 (batched) and a DB-polling consumer that claims pending jobs, reads Parquet from S3, resolves/creates integrations via Redis-backed lookups, and emits activity results; includes initial Cvent export SQL + transformer plus new CventactivityTypes.Wires the export into operations via a daily cron trigger in
cron_service, adds Docker compose/build files for the new service, updates the monorepo lockfile, and extends@crowd/snowflakewithSnowflakeClient.destroy()plus new organization source/attribute enums/priorities forcvent.Written by Cursor Bugbot for commit 64d1639. This will update automatically on new commits. Configure here.