[BUG] Interrupted uploads do not resume

[JakeSmarter]

Basic information

• Release version: 0.9.5
• System: Linux, probably any
• Capture Device: any

Steps to reproduce behavior

1. Start uploading a bunch of images, preferably like gigabytes per sequence.
2. Wait for the server to choke or deliberately throttle your network bandwidth, potentially down to 0. Pressing Ctrl+C and then resuming (while the upload session is still alive on the server) with exactly the same command should have the same effect.
3. Observe that either the server or the client resumes at 'offset': 0.

It is unclear whether this is a client or server bug. My gut feeling tells me that it is a server bug but I may be wrong. The server may change the chunk size during upload depending on load. The client should adapt to this. The server should also always respond with the correct offset, independent of the current chunk size.

Expected behavior

Per sequence uploads should resume on the offset of the first incomplete chunk.

Actual behavior

2022-10-13 14:11:23,512 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 0, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0}
2022-10-13 14:11:37,891 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 3792867, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:11:53,101 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 7585734, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:12:08,341 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 11378601, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:12:22,579 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 0, 'offset': 15171468, 'retries': 0, 'upload_last_restart_time': 1665663083.5125833, 'upload_first_offset': 0, 'chunk_size': 3792867}
2022-10-13 14:12:25,712 - WARNING - Error uploading chunk_size 3792867 at offset 0: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
2022-10-13 14:12:25,712 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 14:12:28,367 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 62.19941592216492, 'offset': 0, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665663148.3675609}

Note that 'offset': 0, while it should have been 'offset': 15171468 because this is the offset of the first incomplete chunk.

All of the above means that it is currently extremely difficult to upload large sequences on low bandwidth connections because as soon as the load on the server changes (which is very likely on long uploads), the chunk size changes and the client has to restart uploading from offset 0. This is a huge waste of resources!

[JakeSmarter]

🤦‍♂️ I have just spent over 4 hours uploading a sequence of 163 images worth 618,237,384 bytes. mapillary_tools is still not done because it is constantly resetting to either 0 or an arbitrary number. This is something that should have been completed in about 5 minutes. And, before you ask: no, my connection is stable.

You see, this is exactly what I mean by “Mapillary being dysfunctional”; every upload tool from Mapillary is completely useless and unusable. Or, a horror to use and nobody seems to care. It is almost offending if you look at the amount of hoops I have to jump through in order to just contribute some meager amount of images voluntarily. Just because I am geek I have the patience and skill to deal with it. But, how do you expect any non‑technical people to be willing to contribute via this mess?

[ptpt]

Thanks for reporting and sorry for the inconvenience.

This looks suspicious indeed. It seems the server returned offset=0 regardless it had data chunk uploaded there.

Could you check if it work without a proxy?

I can't reproduce it locally:

2022-10-13 10:43:20,649 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0}
2022-10-13 10:43:21,049 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0, 'offset': 20340672, 'retries': 0, 'upload_last_restart_time': 1665683001.049864, 'upload_first_offset': 20340672}
2022-10-13 10:43:28,298 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0, 'offset': 37117888, 'retries': 0, 'upload_last_restart_time': 1665683001.049864, 'upload_first_offset': 20340672, 'chunk_size': 16777216}
2022-10-13 10:43:44,106 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 0, 'offset': 53895104, 'retries': 0, 'upload_last_restart_time': 1665683001.049864, 'upload_first_offset': 20340672, 'chunk_size': 16777216}


2022-10-13 10:44:05,663 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: ('Connection aborted.', OSError(65, 'No route to host'))
2022-10-13 10:44:05,664 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 10:44:07,682 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_1e13989bbf76d8cab1eed147c7193fe3.mp4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10af3f550>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
2022-10-13 10:44:07,682 - INFO    - Retrying in 4 seconds (2/200)
2022-10-13 10:44:11,691 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_1e13989bbf76d8cab1eed147c7193fe3.mp4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10ac28b50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
2022-10-13 10:44:11,691 - INFO    - Retrying in 8 seconds (3/200)
2022-10-13 10:44:19,704 - WARNING - Error uploading chunk_size 16777216 at offset 20340672: ConnectionError: HTTPSConnectionPool(host='rupload.facebook.com', port=443): Max retries exceeded with url: /mapillary_public_uploads/mly_tools_1e13989bbf76d8cab1eed147c7193fe3.mp4 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10ac28b80>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
2022-10-13 10:44:19,704 - INFO    - Retrying in 16 seconds (4/200)
2022-10-13 10:44:36,045 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 44.612921953201294, 'offset': 58990528, 'retries': 4, 'upload_first_offset': 20340672, 'chunk_size': 16777216, 'upload_last_restart_time': 1665683076.0457752}
2022-10-13 10:44:43,132 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 4, 'sequence_idx': 0, 'file_type': 'gopro', 'entity_size': 3307582003, 'import_path': 'GoProMax/100GOPRO/GH013770.MP4', 'md5sum': '1e13989bbf76d8cab1eed147c7193fe3', 'upload_start_time': 1665683000.649157, 'upload_total_time': 44.612921953201294, 'offset': 75767744, 'retries': 4, 'upload_first_offset': 20340672, 'chunk_size': 16777216, 'upload_last_restart_time': 1665683076.0457752}

[bob3bob3]

If either of you need a reference point (debug logs) to work with I have been using the tools to upload BlackVue videos 24/7 for 2 weeks now. Something like 870GBytes, 180,000 mp4's so far that seems to be working okay. I have not been watching the logs closely, but only have maybe 20 of the ConnectionError similar to above.

No doubt that I am uploading mp4's rather than jpg's has a lesser chance of failure,

Drop me a note if you would like any analysis or dump,

[JakeSmarter]

More logs…

2022-10-13 20:31:10,269 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22126.903530597687, 'offset': 428593971, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685613.4696445}
2022-10-13 20:31:11,338 - WARNING - Error uploading chunk_size 3792867 at offset 360322365: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-10-13 20:31:11,339 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 20:31:16,334 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 371700966, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:31:31,591 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 375493833, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:31:46,339 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 379286700, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:32:01,102 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 383079567, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:32:26,992 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22384.77271795273, 'offset': 386872434, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685876.3342628}
2022-10-13 20:32:37,634 - WARNING - Error uploading chunk_size 3792867 at offset 371700966: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
2022-10-13 20:32:37,634 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 20:32:44,441 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 375493833, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:33:11,953 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 379286700, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:33:28,227 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 383079567, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:33:46,208 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 386872434, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:03,132 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 390665301, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:17,073 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 394458168, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:30,741 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 398251035, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:45,439 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 402043902, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:34:59,471 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 405836769, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:14,861 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 409629636, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:31,725 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 413422503, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:46,345 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 417215370, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:35:59,818 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 421008237, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:36:18,185 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 424801104, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:36:32,820 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 428593971, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:36:47,286 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 432386838, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:37:04,628 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 436179705, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:37:26,269 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 439972572, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:37:47,220 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 443765439, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:38:11,899 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 447558306, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:38:31,960 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 451351173, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:38:47,581 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 455144040, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:03,287 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 458936907, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:26,043 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 462729774, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:44,205 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 466522641, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:39:57,682 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 470315508, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:11,354 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 474108375, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:24,670 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 477901242, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:39,238 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22466.072811365128, 'offset': 481694109, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665685964.4413586}
2022-10-13 20:40:40,743 - WARNING - Error uploading chunk_size 3792867 at offset 375493833: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip
2022-10-13 20:40:40,743 - INFO    - Retrying in 2 seconds (1/200)
2022-10-13 20:40:44,878 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22942.374705791473, 'offset': 364115232, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665686444.8789496}
2022-10-13 20:41:17,740 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22942.374705791473, 'offset': 367908099, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665686444.8789496}
2022-10-13 20:42:09,556 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 0, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip', 'sequence_image_count': 163, 'entity_size': 618237384, 'md5sum': 'c043f21e7f5d2f4ab25c1ba5099bf5a7', 'upload_start_time': 1665663081.9819736, 'upload_total_time': 22942.374705791473, 'offset': 371700966, 'retries': 1, 'upload_first_offset': 0, 'chunk_size': 3792867, 'upload_last_restart_time': 1665686444.8789496}

Could you check if it work without a proxy?

Sure, this may be proxy related. However, I highly doubt it. Too me, the server looks like the most probable culprit because the offset is computed by the server. Unfortunately, I cannot bypass the proxy. Anyhow, proxy or not it should work because I cannot imagine a scenario where the server should respond a wrong offset over a proxy or the client to modify the offset. It is the server that keeps track of the upload session state. Btw, I have never had any issues with other online services accepting large (interrupted) uploads.

I am not sure how you could reproduce this. But, it seems like these recurring messages are usually causing trouble:

WARNING - Error uploading chunk_size 3792867 at offset 360322365: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
WARNING - Error uploading chunk_size 3792867 at offset 371700966: HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_c043f21e7f5d2f4ab25c1ba5099bf5a7.zip

[ptpt]

@gitne I suspect it's related to HTTP status 412 error, curious to see the payload.

I'm trying to add a check on retires: if the offset fetched from the server does not move as expected, exit with full HTTP response printed out.

I will make a new alpha release and let you know soon.

[JakeSmarter]

I will make a new alpha release and let you know soon.

Great, thank you! 😄

Just FYI: I have observed this behavior ever since I have started using version 0.8.0.

[ptpt]

Just a quick check @gitne: were you uploading this sequence in multiple processes/machines simultaneously? That might cause the offset to be inconsistent AFICS.

[JakeSmarter]

Just a quick check @gitne: were you uploading this sequence in multiple processes/machines simultaneously?

No, single instance on one machine only.

[ptpt]

The fix is released here https://github.com/mapillary/mapillary_tools/releases/tag/v0.9.5a1

BTW it would be great if you could setup a local env here https://github.com/mapillary/mapillary_tools#development so we can test on branches without releasing binaries (faster iteration).

[JakeSmarter]

This is what I got on my first attempt with v0.9.5a1:

2022-10-14 12:25:44,228 - DEBUG   - CLI param: verbose: True
2022-10-14 12:25:44,228 - DEBUG   - CLI param: import_path: [PosixPath('mapillary_public_uploads')]
2022-10-14 12:25:44,228 - DEBUG   - CLI param: dry_run: False
2022-10-14 12:25:44,272 - INFO    - File mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip has been uploaded already. Check the upload history at /home/GITNE/.local/share/mapillary_tools/upload_history/c0/43f21e7f5d2f4ab25c1ba5099bf5a7.json
2022-10-14 12:25:44,273 - DEBUG   - Uploaded to cluster: None
2022-10-14 12:25:44,307 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0}
2022-10-14 12:25:46,299 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 39330460, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460}
2022-10-14 12:26:05,397 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 42355880, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:24,231 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 45381300, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:35,169 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 48406720, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:46,149 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 51432140, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:26:57,397 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 54457560, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:27:07,733 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 57482980, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:27:28,181 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 60508400, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:27:47,437 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 63533820, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:28:06,483 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 66559240, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:28:24,166 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 69584660, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:28:41,148 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 72610080, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:00,671 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 75635500, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:11,399 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 78660920, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:22,306 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 81686340, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:32,839 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 84711760, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:29:45,819 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 87737180, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:01,497 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 90762600, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:12,107 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 93788020, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:22,642 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 96813440, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:32,824 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 99838860, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:42,983 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 102864280, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:30:53,270 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 105889700, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:04,135 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 108915120, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:14,425 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 111940540, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:26,219 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 114965960, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:37,200 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 117991380, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:48,100 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 121016800, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:31:58,424 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 124042220, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:09,483 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 127067640, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:20,301 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 130093060, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:30,732 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 133118480, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:41,610 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 136143900, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:32:52,026 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 139169320, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:03,374 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 142194740, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:14,322 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 145220160, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:30,316 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 148245580, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:45,847 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 151271000, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:33:56,210 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 154296420, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:06,474 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 157321840, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:18,293 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 160347260, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:30,022 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 163372680, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:40,938 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 166398100, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:34:51,999 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 169423520, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:02,358 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 172448940, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:15,365 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 175474360, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:26,466 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 178499780, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:38,630 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 181525200, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:35:55,036 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665743144.3074815, 'upload_total_time': 0, 'offset': 184550620, 'retries': 0, 'upload_last_restart_time': 1665743146.2998044, 'upload_first_offset': 39330460, 'chunk_size': 3025420}
2022-10-14 12:36:02,285 - CRITICAL - Upload server offset 69584660 does not match expected offset 184550620
2022-10-14 12:36:02,286 - DEBUG   - API Logging for action upload_failed_upload: {'images': 0, 'sequences': 0, 'size': 0.0, 'uploaded_size': 0.0, 'speed': 0, 'time': 0}
Traceback (most recent call last):
  File "mapillary_tools/uploader.py", line 488, in _upload_fp
  File "mapillary_tools/upload_api_v4.py", line 128, in upload
  File "requests/models.py", line 1021, in raise_for_status
requests.exceptions.HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "mapillary_tools/upload.py", line 785, in _upload_zipfiles
  File "mapillary_tools/uploader.py", line 140, in upload_zipfile
  File "mapillary_tools/uploader.py", line 362, in _upload_zipfile_fp
  File "mapillary_tools/uploader.py", line 511, in _upload_fp
mapillary_tools.upload_api_v4.UploadHTTPError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
  File "mapillary_tools/commands/__main__.py", line 141, in main
  File "mapillary_tools/commands/upload_zip.py", line 32, in run
  File "mapillary_tools/upload.py", line 623, in upload
  File "mapillary_tools/upload.py", line 618, in upload
  File "mapillary_tools/upload.py", line 789, in _upload_zipfiles
mapillary_tools.upload.UploadError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}
[837964] Failed to execute script 'main' due to unhandled exception!

Next, I am simply going to retry and see how far it will go.

BTW it would be great if you could setup a local env here https://github.com/mapillary/mapillary_tools#development so we can test on branches without releasing binaries (faster iteration).

I will see what I can do. My best bet is to build a Flatpak because it provides a defined set of dependencies. This is going to help get reproducible results. 🤞

[JakeSmarter]

After retry I got this:

2022-10-14 12:48:07,229 - DEBUG   - CLI param: verbose: True
2022-10-14 12:48:07,229 - DEBUG   - CLI param: import_path: [PosixPath('mapillary_public_uploads')]
2022-10-14 12:48:07,229 - DEBUG   - CLI param: dry_run: False
2022-10-14 12:48:07,263 - INFO    - File mly_tools_00c8a153ca702f5b1d714bd83d0e8362.zip has been uploaded already. Check the upload history at /home/GITNE/.local/share/mapillary_tools/upload_history/c0/43f21e7f5d2f4ab25c1ba5099bf5a7.json
2022-10-14 12:48:07,264 - DEBUG   - Uploaded to cluster: None
2022-10-14 12:48:07,291 - DEBUG   - Sending upload_start via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0}
2022-10-14 12:48:10,780 - DEBUG   - Sending upload_fetch_offset via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 18152520, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520}
2022-10-14 12:49:30,090 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 21177940, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:50:18,054 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 24203360, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:50:48,944 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 27228780, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:51:20,380 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 30254200, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:51:34,717 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 33279620, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:51:45,598 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 36305040, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:52:00,109 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 39330460, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:52:15,236 - DEBUG   - Sending upload_progress via IPC: {'total_sequence_count': 2, 'sequence_idx': 1, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip', 'sequence_image_count': 539, 'entity_size': 1630701559, 'md5sum': '3a59c68825c203b5bbf22154e4802238', 'upload_start_time': 1665744487.2919557, 'upload_total_time': 0, 'offset': 42355880, 'retries': 0, 'upload_last_restart_time': 1665744490.7808063, 'upload_first_offset': 18152520, 'chunk_size': 3025420}
2022-10-14 12:52:28,474 - CRITICAL - Upload server offset 39330460 does not match expected offset 42355880
2022-10-14 12:52:28,474 - DEBUG   - API Logging for action upload_failed_upload: {'images': 0, 'sequences': 0, 'size': 0.0, 'uploaded_size': 0.0, 'speed': 0, 'time': 0}
Traceback (most recent call last):
  File "mapillary_tools/uploader.py", line 488, in _upload_fp
  File "mapillary_tools/upload_api_v4.py", line 128, in upload
  File "requests/models.py", line 1021, in raise_for_status
requests.exceptions.HTTPError: 412 Client Error: Precondition Failed for url: https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "mapillary_tools/upload.py", line 785, in _upload_zipfiles
  File "mapillary_tools/uploader.py", line 140, in upload_zipfile
  File "mapillary_tools/uploader.py", line 362, in _upload_zipfile_fp
  File "mapillary_tools/uploader.py", line 511, in _upload_fp
mapillary_tools.upload_api_v4.UploadHTTPError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
  File "mapillary_tools/commands/__main__.py", line 141, in main
  File "mapillary_tools/commands/upload_zip.py", line 32, in run
  File "mapillary_tools/upload.py", line 623, in upload
  File "mapillary_tools/upload.py", line 618, in upload
  File "mapillary_tools/upload.py", line 789, in _upload_zipfiles
mapillary_tools.upload.UploadError: POST https://rupload.facebook.com/mapillary_public_uploads/mly_tools_3a59c68825c203b5bbf22154e4802238.zip
> HTTP Status: 412
{"debug_info":{"retriable":true,"type":"OffsetInvalidError","message":"Request starting offset is invalid"}}
[838207] Failed to execute script 'main' due to unhandled exception!

The offset is apparently still incorrect on retry. The last offset was 184,550,620 and on retry it was 18,152,520. 😕 That’s like a magnitude lower! 😲

[ptpt]

It seems that you were uploading zipfiles from mapillary_public_uploads which is the default location for the dry run upload testing. I think it's fine.

What looks really strange though, is that the embedded md5sum in the zip filename mapillary_public_uploads/mly_tools_2c198fbcf23dfe64e2aa6b5c5418f758.zip is not same as the calculated md5sum 3a59c68825c203b5bbf22154e4802238 in the log. This makes me suspect if there is anything wrong with the md5sum calculation, so multiple zipfiles (with different content) got the same md5sum and uploaded to the same endpoint, therefore the HTTP 412 error returned.

I can't reproduce this issue neither unfortunately:

# generate zipfiles in mapillary_public_uploads
python3 -m mapillary_tools.commands --verbose process_and_upload ~/ImageTestData/myimages/ --dry_run

# upload these zipfiles
python3 -m mapillary_tools.commands --verbose upload_zip ./mapillary_public_uploads

# Here is one of the logs:
# as you can see the md5sum in the filename and the log match
2022-10-14 10:30:25,780 - DEBUG   - Sending upload_end via IPC: {'total_sequence_count': 3, 'sequence_idx': 2, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_6bfe61d03070c89f5d6fccdced76f0c7.zip', 'sequence_image_count': 2, 'entity_size': 6746255, 'md5sum': '6bfe61d03070c89f5d6fccdced76f0c7', 'upload_start_time': 1665768624.169763, 'upload_total_time': 1.478506088256836, 'offset': 6746255, 'retries': 0, 'upload_last_restart_time': 1665768624.301506, 'upload_first_offset': 0, 'chunk_size': 0, 'upload_end_time': 1665768625.7800121}

@gitne would you mind share us one of these zipfile for debugging via support _at_ mapillary.zendesk.com?