Related: #1720
Is your feature request related to a problem? If so, Please describe.
We are handling preparing video for streaming ourselves to avoid bottlenecks in the DIVE backend related to video conversion. Our DIVE server is lightweight, and our datasets are extremely large (petabytes).
We can avoid transcoding by getting video into the appropriate format - however, there still seem to be other backend tasks that are resource intensive. One of these is a backend task that ffprobes the videos to get video metadata. While the actual ffprobe is lightweight in terms of flops on the server, it is extremely heavy on the network. Viz. the DIVE server has to download the entire video out of our buckets onto the server to run this ffprobe command. For a single video, this is not super problematic, but for large datasets, this is a significant resource constraint.
Describe the solution you'd like
Our current setup consists of a lightweight DIVE server, backed by a GIRDER that ties into our google cloud buckets. The google cloud buckets hold all of the video, and it is extensive. The DIVE server is not equipped to handle these quantities of video, not even for the smallest of tasks (like ffprobe). It would be nice to have a solution where the DIVE server is completely pass-through for the video: the backend never downloads the video or reads it, it just streams directly from the bucket onto the user's machine, completely bypassing the DIVE server in every way.
Related: #1720
Is your feature request related to a problem? If so, Please describe.
We are handling preparing video for streaming ourselves to avoid bottlenecks in the DIVE backend related to video conversion. Our DIVE server is lightweight, and our datasets are extremely large (petabytes).
We can avoid transcoding by getting video into the appropriate format - however, there still seem to be other backend tasks that are resource intensive. One of these is a backend task that
ffprobes the videos to get video metadata. While the actualffprobeis lightweight in terms of flops on the server, it is extremely heavy on the network. Viz. the DIVE server has to download the entire video out of our buckets onto the server to run thisffprobecommand. For a single video, this is not super problematic, but for large datasets, this is a significant resource constraint.Describe the solution you'd like
Our current setup consists of a lightweight DIVE server, backed by a GIRDER that ties into our google cloud buckets. The google cloud buckets hold all of the video, and it is extensive. The DIVE server is not equipped to handle these quantities of video, not even for the smallest of tasks (like
ffprobe). It would be nice to have a solution where the DIVE server is completely pass-through for the video: the backend never downloads the video or reads it, it just streams directly from the bucket onto the user's machine, completely bypassing the DIVE server in every way.