Pipeline file access within code

[crooktec-tannercrook]

When editing code in the pipeline there is a place where you can create files. Is there anyway to reference these files in the python script? I have a package that requires a json keyfile and it is the one thing that is blocking me from building that pipeline.

[saul-data]

Hi @crooktec-tannercrook, if the json file is small, you could save the json file in code editor by simply creating a file. If it is a large json file, I'd recommend storing that file in a database like Postgresql or S3 and then pulling it from that storage.

Here is an example:

You should then reference that file using a relative path in your python code.

import json

# Get scripts directory
import pathlib
jsonpath = pathlib.Path(__file__).parent.resolve()
print(jsonpath)
  
# Opening JSON file
f = open(str(jsonpath)+'/myjsonfile.json')
  
# returns JSON object as 
# a dictionary
data = json.load(f)

print(data)

[crooktec-tannercrook]

This is exactly what I needed. Thank you! I will try to put some documentation together so I can help contribute as I think this is gold when migrating from an airflow solution.

[saul-data]

Thank you. 😊 I’m glad this helped you. Agreed we need more recipes and documentation, any assistance with contribution will be greatly appreciated by us and the data community. There are some exciting new features coming in soon including a package manager so you can build your own Python packages to share code or functions across the pipelines.