[elixir] feat: fetch dmap worker by grejdi-mbta · Pull Request #49 · mbta/data_platform

grejdi-mbta · 2022-05-26T20:27:13Z

This PR adds a worker for fetching DMAP data from Cubic's cloud storage and uploading them to our 'Incoming' bucket.

grejdi-mbta · 2022-05-26T21:39:30Z

@paulswartz Feel free to peek in. Still working through the upload to S3 and some small Credo refactors, but overall should be in good shape.

paulswartz

should the sample files have data?

.env.template

ex_cubic_ingestion/lib/ex_cubic_ingestion/schema/cubic_dmap_dataset.ex

ex_cubic_ingestion/lib/ex_cubic_ingestion/utility.ex

ex_cubic_ingestion/lib/ex_cubic_ingestion/workers/fetch_dmap.ex

ex_cubic_ingestion/lib/ex_cubic_ingestion/downloader.ex

github-actions · 2022-06-04T02:15:18Z

Coverage of commit `56613b8`

Summary coverage rate:
  lines......: 89.8% (368 of 410 lines)
  functions..: 69.1% (152 of 220 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_dmap_dataset.ex      | 100%      7| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_dmap_feed.ex         | 100%      6| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_table.ex             |84.6%     13|77.8%     9|    -      0
  lib/ex_cubic_ingestion/validators.ex                     | 100%      5| 100%     4|    -      0
  lib/ex_cubic_ingestion/workers/fetch_dmap.ex             |94.1%     34| 100%     6|    -      0
  lib/ex_cubic_ingestion/workers/schedule_dmap.ex          | 0.0%      1| 0.0%     1|    -      0
  test/support/ex_aws.ex                                   |88.2%     34| 100%     4|    -      0
  test/support/httpoison.ex                                |75.0%      8| 100%     1|    -      0

Download coverage report

github-actions · 2022-06-04T02:25:22Z

Coverage of commit `68672eb`

Summary coverage rate:
  lines......: 90.0% (368 of 409 lines)
  functions..: 69.4% (152 of 219 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_dmap_dataset.ex      | 100%      7| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_dmap_feed.ex         | 100%      6| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_table.ex             |84.6%     13|77.8%     9|    -      0
  lib/ex_cubic_ingestion/validators.ex                     | 100%      5| 100%     4|    -      0
  lib/ex_cubic_ingestion/workers/fetch_dmap.ex             |94.1%     34| 100%     6|    -      0
  test/support/ex_aws.ex                                   |88.2%     34| 100%     4|    -      0
  test/support/httpoison.ex                                |75.0%      8| 100%     1|    -      0

Download coverage report

grejdi-mbta · 2022-06-04T02:25:29Z

should the sample files have data?

Just wanted to have placeholders for now. I will add some in soon.

ex_cubic_ingestion/lib/ex_cubic_ingestion/schema/cubic_dmap_feed.ex

ex_cubic_ingestion/lib/ex_cubic_ingestion/validators.ex

ex_cubic_ingestion/lib/ex_cubic_ingestion/workers/fetch_dmap.ex

ex_cubic_ingestion/test/ex_cubic_ingestion/workers/fetch_dmap_test.exs

…updates.

github-actions · 2022-06-06T19:45:06Z

Coverage of commit `4d2c2a2`

Summary coverage rate:
  lines......: 90.0% (368 of 409 lines)
  functions..: 69.4% (152 of 219 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_dmap_dataset.ex      | 100%      7| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_dmap_feed.ex         | 100%      6| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_table.ex             |84.6%     13|77.8%     9|    -      0
  lib/ex_cubic_ingestion/validators.ex                     | 100%      5| 100%     4|    -      0
  lib/ex_cubic_ingestion/workers/fetch_dmap.ex             |94.1%     34| 100%     6|    -      0
  test/support/ex_aws.ex                                   |88.2%     34| 100%     4|    -      0
  test/support/httpoison.ex                                |75.0%      8| 100%     1|    -      0

Download coverage report

github-actions · 2022-06-06T21:10:00Z

Coverage of commit `e138d8a`

Summary coverage rate:
  lines......: 90.0% (371 of 412 lines)
  functions..: 69.4% (152 of 219 functions)
  branches...: no data found

Files changed coverage rate:
                                                           |Lines       |Functions  |Branches    
  Filename                                                 |Rate     Num|Rate    Num|Rate     Num
  ===============================================================================================
  lib/ex_cubic_ingestion/schema/cubic_dmap_dataset.ex      | 100%     12| 100%     9|    -      0
  lib/ex_cubic_ingestion/schema/cubic_dmap_feed.ex         | 100%      6| 100%     8|    -      0
  lib/ex_cubic_ingestion/schema/cubic_table.ex             |84.6%     13|77.8%     9|    -      0
  lib/ex_cubic_ingestion/validators.ex                     | 100%      5| 100%     4|    -      0
  lib/ex_cubic_ingestion/workers/fetch_dmap.ex             |93.8%     32| 100%     5|    -      0
  test/support/ex_aws.ex                                   |88.2%     34| 100%     4|    -      0
  test/support/httpoison.ex                                |75.0%      8| 100%     1|    -      0

Download coverage report

paulswartz

🍰

grejdi-mbta force-pushed the gg-1202207406750018-worker-for-fetch-dmap branch from 00e51b0 to 2eabb35 Compare May 26, 2022 22:04

paulswartz reviewed May 28, 2022

View reviewed changes

grejdi-mbta commented Jun 2, 2022

View reviewed changes

ex_cubic_ingestion/lib/ex_cubic_ingestion/downloader.ex Outdated Show resolved Hide resolved

grejdi-mbta mentioned this pull request Jun 4, 2022

[elixir] feat: use stream for downloading and uploading #51

Open

grejdi-mbta requested a review from paulswartz June 4, 2022 02:25

grejdi-mbta changed the title ~~[elixir] fetch dmap worker~~ [elixir] feat: fetch dmap worker Jun 4, 2022

paulswartz reviewed Jun 6, 2022

View reviewed changes

grejdi-mbta added 7 commits June 6, 2022 15:40

wip: fetch dmap progess

eabb979

fix: rename utility to validators. removed extra api key. small test …

414d8d1

…updates.

feat: add fetch and upload with stream.

ec315c5

fix: simpler download/upload. updates to tests.

11616b8

fix: remove logger, and override dependecy check for fetch dmap.

aafade8

fix: dialyzer issues

b517dfd

fix: remove empty schedule worker.

4d2c2a2

grejdi-mbta force-pushed the gg-1202207406750018-worker-for-fetch-dmap branch from 68672eb to 4d2c2a2 Compare June 6, 2022 19:40

fix: code review issues

e138d8a

grejdi-mbta requested a review from paulswartz June 6, 2022 21:22

paulswartz approved these changes Jun 6, 2022

View reviewed changes

grejdi-mbta merged commit c47b372 into main Jun 7, 2022

grejdi-mbta deleted the gg-1202207406750018-worker-for-fetch-dmap branch June 7, 2022 13:13

Conversation

grejdi-mbta commented May 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grejdi-mbta commented May 26, 2022

Uh oh!

paulswartz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 4, 2022

Coverage of commit 56613b8

Uh oh!

github-actions bot commented Jun 4, 2022

Coverage of commit 68672eb

Uh oh!

grejdi-mbta commented Jun 4, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 6, 2022

Coverage of commit 4d2c2a2

Uh oh!

github-actions bot commented Jun 6, 2022

Coverage of commit e138d8a

Uh oh!

paulswartz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

grejdi-mbta commented May 26, 2022 •

edited

Loading

Coverage of commit `56613b8`

Coverage of commit `68672eb`

Coverage of commit `4d2c2a2`

Coverage of commit `e138d8a`