Implement review suggestions 2

.github/workflows/awsfulltest.yml

@@ -24,8 +24,6 @@ jobs:
 
       - name: Launch workflow via Seqera Platform
         uses: seqeralabs/action-tower-launch@51565b514bff1827cf34620de25d0055759f1fc9 # v2
-        # Add full size test data (but still relatively small datasets for few samples)
-        # on the `test_full.config` test runs with only one set of parameters
         with:
           workspace_id: ${{ vars.TOWER_WORKSPACE_ID }}
           access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}

CHANGELOG.md

@@ -9,28 +9,9 @@ Initial release of nf-core/datasync, created with the [nf-core](https://nf-co.re
 
 ### `Added`
 
-- [[#87](https://github.com/nf-core/datasync/pull/87)] - Add local test profile ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#79](https://github.com/nf-core/datasync/pull/79)] - Add exit status for `RCLONE` modules in multiqc report ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#74](https://github.com/nf-core/datasync/pull/74)] - Implement warning and `--download` parameter when working with remote directories and SHA checksums ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila), [@apeltzer](https://github.com/apeltzer) and [@antoniasaracco](https://github.com/antoniasaracco)).
-- [[#67](https://github.com/nf-core/datasync/pull/67)] - Copy only files which are successfully validated ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)).
-- [[#57](https://github.com/nf-core/datasync/pull/57)] - Add nf-tests with edge cases([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas)).
-- [[#55](https://github.com/nf-core/datasync/pull/55)] - Add subdirectories with sample id to results. Improvements to MultiQC report (width of columns, sections and sub-section headers)colors to multiqc results and display first errors in tables ([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas)).
-- [[#54](https://github.com/nf-core/datasync/pull/54)] - Add colors to multiqc results and display first errors in tables ([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas)).
-- [[#31](https://github.com/nf-core/datasync/pull/31)] - Compute checksum for each input file ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#35](https://github.com/nf-core/datasync/pull/35)] - Compare generated checksum to given checksum in input and update test profiles ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#46](https://github.com/nf-core/datasync/pull/46)] - Import Rclone module from nf-core([@antoniasaracco](https://github.com/antoniasaracco), review by [@delfiterradas](https://github.com/delfiterradas)).
-- [[#41](https://github.com/nf-core/datasync/pull/41)] - Generate MultiQC Report with comparechecksum tables and input samplesheet ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#49](https://github.com/nf-core/datasync/pull/49)] - Install `RCLONE_CHECK` and `RCLONE_CHECKSUM` modules from nf-core ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)).
-- [[#59](https://github.com/nf-core/datasync/pull/59)] - Create pipeline documentation ([@antoniasaracco](https://github.com/antoniasaracco), review by [@atrigila](https://github.com/atrigila) and [@delfiterradas](https://github.com/delfiterradas)).
-
-### `Fixed`
-
-- [[#84](https://github.com/nf-core/datasync/pull/84)] - Update `RCLONE_` modules, enhance source and destination path handling, add local `create_filter_list` module and other small fixes ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#83](https://github.com/nf-core/datasync/pull/83)] - Add apptainer version to avoid error in CI test ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
-- [[#78](https://github.com/nf-core/datasync/pull/78)] - Fixed linting issues reported by nf-core and Nextflow, and updated `rclone` modules and nf-tests to sort generated report files for deterministic snapshots ([@atrigila](https://github.com/atrigila), review by [@delfiterradas](https://github.com/delfiterradas) and [@antoniasaracco](https://github.com/antoniasaracco)).
-- [[#76](https://github.com/nf-core/datasync/pull/76)] - Remove `--one-way` from rclone/checksum confi ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)).
-- [[#73](https://github.com/nf-core/datasync/pull/73)] - Create tmp `files_to_copy.tx` avoiding error with write permissions in work directory ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)).
-
-### `Dependencies`
-
-### `Deprecated`
+- Samplesheet-driven copying of files and directories between local paths and rclone-supported object storage locations.
+- Validation of source data against supplied MD5 and SHA-256 checksum manifests before transfer.
+- The ability to copy only files that pass checksum validation, and to download remote files when SHA-256 verification is required.
+- Post-transfer comparison of copied data against the source, with detailed rclone status files for each sample.
+- A MultiQC report covering the input samplesheet, validation summary, checksum validation, and post-transfer checks.
+- A local test profile to explore the pipeline and its outputs.

README.md

@@ -9,14 +9,12 @@
 [![GitHub Actions CI Status](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/nf-test.yml)
 [![GitHub Actions Linting Status](https://github.com/nf-core/datasync/actions/workflows/linting.yml/badge.svg)](https://github.com/nf-core/datasync/actions/workflows/linting.yml)[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/datasync/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX)
 [![nf-test](https://img.shields.io/badge/unit_tests-nf--test-337ab7.svg)](https://www.nf-test.com)
-
 [![Nextflow](https://img.shields.io/badge/version-%E2%89%A525.10.4-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)
 [![nf-core template version](https://img.shields.io/badge/nf--core_template-4.1.0-green?style=flat&logo=nfcore&logoColor=white&color=%2324B064&link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/4.1.0)
 [![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)
 [![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)
 [![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)
 [![Launch on Seqera Platform](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Seqera%20Platform-%234256e7)](https://cloud.seqera.io/launch?pipeline=https://github.com/nf-core/datasync)
-
 [![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23datasync-4A154B?labelColor=000000&logo=slack)](https://nfcore.slack.com/channels/datasync)[![Follow on Bluesky](https://img.shields.io/badge/bluesky-%40nf__core-1185fe?labelColor=000000&logo=bluesky)](https://bsky.app/profile/nf-co.re)[![Follow on Mastodon](https://img.shields.io/badge/mastodon-nf__core-6364ff?labelColor=FFFFFF&logo=mastodon)](https://mstdn.science/@nf_core)[![Watch on YouTube](http://img.shields.io/badge/youtube-nf--core-FF0000?labelColor=000000&logo=youtube)](https://www.youtube.com/c/nf-core)
 
 ## Introduction
@@ -34,7 +32,7 @@ The current tested use case for this pipeline is transfer between S3 buckets.
 
 Pass an `rclone` configuration with `--rclone_config` whenever a source or destination URI needs credentials or provider settings. Samplesheet paths use local paths or standard URIs such as `s3://bucket/path`, not rclone's `remote:path` syntax. For non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/).
 
-![nf-core/datasync metro map](docs/images/datasync-metromap.png)
+![nf-core/datasync metro map](docs/images/datasync_nf-metro.svg)
 
 ## Quick start
 

assets/samplesheet.csv

@@ -1,2 +1,2 @@
 sample,input,output_path,checksum_md5
-sample,path/to/file,output_path,path/to/file
+sample,s3://foo/path/to/files/,output_path/data,path/to/checksum.md5

docs/images/datasync-nf-metro.mmd

@@ -0,0 +1,33 @@
+%%metro title: nf-core/datasync
+%%metro mode: dark
+%%metro style: nfcore
+%%metro diamond_style: symmetric
+%%metro line: main | Pipeline | #4CAF50
+
+%%metro file: ref_in | CSV | Input\nSamplesheet | banner
+%%metro file: reads_in | Config | Rclone\nConfig | banner
+%%metro file: report_out | HTML | Report | banner
+%%metro grid: datasync | 0,0
+
+graph LR
+	subgraph datasync [Datasync pipeline]
+        ref_in[Input]
+        reads_in[ ]
+        node1[Rclone\nChecksum]
+        node2[Create\nFilter List]
+        node3[Rclone\nCheck]
+        node4[Rclone\nCopy]
+        node5[MultiQC]
+        Rclone\nChecksum[Rclone\nChecksum]
+        ref_in -->|main| Rclone\nChecksum
+        reads_in -->|main| Rclone\nChecksum
+        report_out[Output]
+        ref_in -->|main| node1
+        reads_in -->|main| node1
+        node1 -->|main| node2
+        node2 -->|main| node4
+        node4 -->|main| node3
+        node3 -->|main| node5
+        node1 -->|main| node4
+        node5 -->|main| report_out
+    end

modules/local/create_filter_list/main.nf

@@ -7,12 +7,7 @@ process CREATE_FILTER_LIST {
     output:
     tuple val(meta), path('files_to_copy.txt')
 
-    script:
-    def content = common.join('\n')
-
-    """
-    cat > files_to_copy.txt <<'EOF'
-${content}
-EOF
-    """
+    exec:
+    def outFile = task.workDir.resolve('files_to_copy.txt')
+    outFile.text = common.join('\n') + '\n'
 }

nextflow_schema.json

@@ -47,24 +47,24 @@
                     "exists": true,
                     "description": "Path to the rclone config file used for cloud storage authentication.",
                     "help_text": "Provide an rclone config file to support cloud providers such as AWS S3 or Azure Blob Storage. This file is loaded by rclone when copying data.",
-                    "fa_icon": "fas fa-file"
+                    "fa_icon": "fas fa-cog"
                 },
                 "rclone_dry_run": {
                     "type": "boolean",
                     "description": "Perform a dry run of the rclone copy command.",
-                    "fa_icon": "fas fa-eye",
+                    "fa_icon": "fas fa-forward",
                     "help_text": "If set, the pipeline will not actually copy any files to the destination. Instead, it will print out what would have been copied. This is useful for testing and debugging."
                 },
                 "copy_matching_only": {
                     "type": "boolean",
                     "description": "Only copy files that matched their provided input checksums.",
-                    "fa_icon": "fas fa-eye",
+                    "fa_icon": "fas fa-copy",
                     "help_text": "If set, the pipeline will only copy files that were correctly validated and will skip any file that did not match their input checksum."
                 },
                 "download": {
                     "type": "boolean",
                     "description": "Download remote files for sha256 checksum verification in `RCLONE_CHECKSUM`.",
-                    "fa_icon": "fas fa-eye",
+                    "fa_icon": "fas fa-cloud-download-alt",
                     "help_text": "If set, `RCLONE_CHECKSUM` will download remote files for any sample for which a SHA256 file was provided."
                 }
             }