[DOCKTESTERS] Thanks!

Wed Mar 22 10:00:10 EDT 2017

Hi,

I have a local copy of the file on my desktop. It's a bit ironic, but if GNOS is currently down, we could setup a local transfer or use a secure OICR USB key. The file is 22 GB in size.

Denis Yuen
Senior Software Developer

Ontario Institute for Cancer Research
MaRS Centre
661 University Avenue
Suite 510
Toronto, Ontario, Canada M5G 0A3

Toll-free: 1-866-678-6427
Twitter: @OICR_news
www.oicr.on.ca<http://www.oicr.on.ca/>

This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.

________________________________
From: docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org <docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org> on behalf of George Mihaiescu <George.Mihaiescu at oicr.on.ca>
Sent: March 22, 2017 8:56:46 AM
To: Jonas Demeulemeester
Cc: docktesters at lists.icgc.org
Subject: Re: [DOCKTESTERS] Thanks!

I received a new GNOS token and installed the gtdownload client (not easy because I'm running on a Ubuntu 16.04 VM).

Now, when I run the "bin/get_dkfz_resources.sh" script, it stays at zero:
Status: 0 bytes downloaded (0.000% complete) current rate:        /s
Child 1 downloading          (        )
Child 2 downloading          (        )
Child 3 downloading          (        )
Child 4 downloading          (        )
Child 5 downloading          (        )
Child 6 downloading          (        )
Child 7 downloading          (        )
Child 8 downloading          (        )
Status: 0 bytes downloaded (0.000% complete) current rate:        /s
Child 1 downloading          (        )
Child 2 downloading          (        )
Child 3 downloading          (        )
Child 4 downloading          (        )
Child 5 downloading          (        )
Child 6 downloading          (        )
Child 7 downloading          (        )
Child 8 downloading          (        )
Status: 0 bytes downloaded (0.000% complete) current rate:        /s

Is there another way I can download that file?

Also, I saw on https://wiki.oicr.on.ca/display/PANCANCER/Workflow+Testing+Data that you already ran the Sanger workflow against D0218695, do you remember how long it took?
I couldn't find the original run time for that donor looking through github (https://github.com/ICGC-TCGA-PanCancer/ceph_transfer_ops).

The initial VM running this donor has been running for more than 10 days, and I don't remember Sanger taking so long.

Thank you,
George

From: Jonas Demeulemeester <Jonas.Demeulemeester at crick.ac.uk<mailto:Jonas.Demeulemeester at crick.ac.uk>>
Date: Monday, March 20, 2017 at 3:13 PM
To: George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>>
Cc: Miguel Vazquez <mikisvaz at gmail.com<mailto:mikisvaz at gmail.com>>, Junjun Zhang <Junjun.Zhang at oicr.on.ca<mailto:Junjun.Zhang at oicr.on.ca>>, "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: Re: [DOCKTESTERS] Thanks!

Hi George,

Do you have the DKFZ workflow dependencies tarball in place (and named correctly)?
That's the file it's clearly not finding:

17:06:08.641 [main] ERROR io.dockstore.common.FileProvisioning - Could not copy /root/PCAWG-Docker-Test/resources//dkfz-workflow-dependencies_150318_0951.tar.gz to /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/ee5564fe-ba17-4383-afd8-e785394e365f/dkfz-workflow-dependencies_150318_0951.tar.gz
java.nio.file.NoSuchFileException: /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/ee5564fe-ba17-4383-afd8-e785394e365f/dkfz-workflow-dependencies_150318_0951.tar.gz

You can find the link to this reference tarball on the DKFZ pipeline github page (https://github.com/ICGC-TCGA-PanCancer/dkfz_dockered_workflows)

Hope this helps,
Jonas

On 20 Mar 2017, at 17:19, George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>> wrote:

Hi,

How do I run the DKFZ workflow?
I first ran the DELLY which ended with the following output:
Uploading: #somatic_sv_vcf from /root/PCAWG-Docker-Test/tests/Delly/DO218695/./datastore/launcher-0ce3d535-bd87-4234-a5c0-a3df48d7c5a5/outputs/run_id.embl-delly_1-3-0-preFilter.20150318.somatic.sv.vcf.gz to : /root/PCAWG-Docker-Test/tests/Delly/DO218695//output//DO218695.delly.somatic.sv.vcf.gz
[##################################################] 100%
Uploading: #cov_plots from /root/PCAWG-Docker-Test/tests/Delly/DO218695/./datastore/launcher-0ce3d535-bd87-4234-a5c0-a3df48d7c5a5/outputs/run_id.embl-delly_1-3-0-preFilter.20150318.sv.cov.plots.tar.gz to : /root/PCAWG-Docker-Test/tests/Delly/DO218695//output//DO218695.delly.sv.cov.plots.tar.gz

After that, I tried to run the DKFZ but it errors as below:

root at dockstore4-dkfz:~/PCAWG-Docker-Test# bin/run_test.sh DKFZ DO218695
Running:
cd /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/ && dockstore tool launch --script --entry quay.io/pancancer/pcawg-dkfz-workflow:2.0.1_cwl1.0<http://quay.io/pancancer/pcawg-dkfz-workflow:2.0.1_cwl1.0>quay.io/jwerner_dkfz/DKFZBiasFilter:1.2.2<http://quay.io/jwerner_dkfz/DKFZBiasFilter:1.2.2>   --json Dockstore.json
WARNING: You're currently running as root; probably by accident.
Press control-C to abort or Enter to continue as root.
Set DOCKSTORE_ROOT to disable this warning.

Creating directories for run of Dockstore launcher at: ./datastore//launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2
Provisioning your input files to your local machine
Downloading: #delly-bedpe from /root/PCAWG-Docker-Test/tests/Delly/DO218695/output//DO218695.delly.somatic.sv.bedpe.txt into directory: /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/9c1f2887-bce0-41dd-a4d2-52f000d79e65
Downloading: #normal-bam from /root/PCAWG-Docker-Test/data/DO218695/normal.bam into directory: /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/0a43e408-0cdf-4d99-99a3-e9860161a246
Downloading: #reference-gz from /root/PCAWG-Docker-Test/resources//dkfz-workflow-dependencies_150318_0951.tar.gz into directory: /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/ee5564fe-ba17-4383-afd8-e785394e365f
17:06:08.641 [main] ERROR io.dockstore.common.FileProvisioning - Could not copy /root/PCAWG-Docker-Test/resources//dkfz-workflow-dependencies_150318_0951.tar.gz to /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/ee5564fe-ba17-4383-afd8-e785394e365f/dkfz-workflow-dependencies_150318_0951.tar.gz
java.nio.file.NoSuchFileException: /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/ee5564fe-ba17-4383-afd8-e785394e365f/dkfz-workflow-dependencies_150318_0951.tar.gz -> /root/PCAWG-Docker-Test/resources/dkfz-workflow-dependencies_150318_0951.tar.gz
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
        at java.nio.file.Files.createLink(Files.java:1086)
        at io.dockstore.common.FileProvisioning.provisionInputFile(FileProvisioning.java:273)
        at io.github.collaboratory.LauncherCWL.copyIndividualFile(LauncherCWL.java:726)
        at io.github.collaboratory.LauncherCWL.doProcessFile(LauncherCWL.java:688)
        at io.github.collaboratory.LauncherCWL.pullFilesHelper(LauncherCWL.java:659)
        at io.github.collaboratory.LauncherCWL.pullFiles(LauncherCWL.java:586)
        at io.github.collaboratory.LauncherCWL.run(LauncherCWL.java:185)
        at io.dockstore.client.cli.nested.AbstractEntryClient.handleCWLLaunch(AbstractEntryClient.java:1028)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launchCwl(AbstractEntryClient.java:968)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launchCwl(AbstractEntryClient.java:951)
        at io.dockstore.client.cli.nested.AbstractEntryClient.launch(AbstractEntryClient.java:935)
        at io.dockstore.client.cli.nested.AbstractEntryClient.processEntryCommands(AbstractEntryClient.java:247)
        at io.dockstore.client.cli.Client.run(Client.java:704)
        at io.dockstore.client.cli.Client.main(Client.java:796)
java.lang.RuntimeException: Could not copy /root/PCAWG-Docker-Test/resources//dkfz-workflow-dependencies_150318_0951.tar.gz to /root/PCAWG-Docker-Test/tests/DKFZ/DO218695/./datastore/launcher-81c42580-21ad-4480-9ee9-a0d2a7a14bf2/inputs/ee5564fe-ba17-4383-afd8-e785394e365f/dkfz-workflow-dependencies_150318_0951.tar.gz

P.S. I have three other Sanger tests running that were started at different intervals (and on VMs with different CPU/memory/disk), but none of them has completed yet.

Thank you,
George

From: Miguel Vazquez <mikisvaz at gmail.com<mailto:mikisvaz at gmail.com>>
Date: Monday, March 13, 2017 at 8:52 AM
To: George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>>
Cc: Junjun Zhang <Junjun.Zhang at oicr.on.ca<mailto:Junjun.Zhang at oicr.on.ca>>, Jonas Demeulemeester <Jonas.Demeulemeester at crick.ac.uk<mailto:Jonas.Demeulemeester at crick.ac.uk>>, "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: Re: [DOCKTESTERS] Thanks!

Hi George,

Answers inline

On Mon, Mar 13, 2017 at 2:43 PM, George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>> wrote:
Hi Miguel,

I've started the test by running "bin/run_test.sh Sanger DO50398", so I guess with just one workflow running it should complete faster than two weeks.

I think it still should take a long time. My scripts will run one workflow after another.

Because I'm running in Collaboratory I've changed the "get_icgc_donor.sh" script to use a docker container that has the icgc client inside and pull data from Collaboratory. There is no "bam.bas" file downloaded, just a ".bam" and a ".bam.bai" files, not sure if this is an issue.

I wondered the same thing first time I did this, but this file is produced by the pipeline. There was some problem with this that was dealt with by the developers and updated in the docker. So I think you won't have a problem

By looking at the "bin/compare_result_type.sh" it looks like it's using the gnos client to pull down the existing VCF files for comparison reasons, but I think we store those files in Collaboratory as well, so I'll work with Junjun to adapt the script for this.

Let me know if you need any help

I think I initially tried to run the DKFZ workflow, but it complained about having to run Delly first, so I abandoned this for now.

Yes, if you look at the run_batch.sh you will see that when using DKFZ it will always run Delly first. Delly prepares some files the the  DKFZ file needs, namely related to copy number I believe.

I'll set up a new VM and run the "run_batch.sh" on the DO52140 donor.

Remember that you will need to add the relevant has-keys for the different files in the etc/donor_files.csv. Its a bit tedious right now. You need to go to the ICGC DCC and find these codes manually for the files you need. Ask me if you need help. Once you have all you can run all the workflows for that donor and evaluate results.

https://github.com/mikisvaz/PCAWG-Docker-Test/blob/master/etc/donor_files.csv

Regards

Miguel

George

From: Miguel Vazquez <mikisvaz at gmail.com<mailto:mikisvaz at gmail.com>>
Date: Monday, March 13, 2017 at 6:53 AM
To: George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>>
Cc: Junjun Zhang <Junjun.Zhang at oicr.on.ca<mailto:Junjun.Zhang at oicr.on.ca>>, Jonas Demeulemeester <Jonas.Demeulemeester at crick.ac.uk<mailto:Jonas.Demeulemeester at crick.ac.uk>>, "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: Re: [DOCKTESTERS] Thanks!

Hi George,

The Sanger workflow is very lengthy, it takes about two weeks in my tests.

About correctness, my scripts also cover that part, if you are not using them they might still help you to clarify how we do it. The idea is to take each of the output files produced: SNV_MNV, Indel, SV, and CNV, for both germline and somatic and compare it with the result uploaded to GNOS (not all pipelines produce all files). This is the relevant part in the run_batch.sh script:

https://github.com/mikisvaz/PCAWG-Docker-Test/blob/master/bin/run_batch.sh#L42-L46

The bin/compare_result_type.sh script will take care of downloading the correct file from GNOS and running the comparison. The comparison itself is simple since all files are VCFs, it consists in taking out the variants in terms of chromosome, position, reference and alternative allele and measuring the overlaps.

https://github.com/mikisvaz/PCAWG-Docker-Test/blob/master/bin/compare_result_type.sh

About which donors to test, DO52140 is one Jonas and I have both tested and could be interesting to get a third opinion. Also, any other donor could be interesting to see if something new comes up. I'm not sure which options is best.

Miguel

On Mon, Mar 13, 2017 at 5:12 AM, George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>> wrote:
Hi,

I've started Sanger on DO50398 and it's been running for more than 24 hours, currently at "Workflow step succeeded: s58_bbAllele_merge_59"

I just started a second run on a different VM on same donor, just to compare run times.
The VM used has 8 cores, 48 GB of RAM and 1.1 TB disk and I'll send some monitoring graphs when it finishes the workflow, but I have no idea how to check its correctness.

Give me a list of donors and what workflows you want me to run and I'll try to schedule them tomorrow.

George

From: Junjun Zhang <Junjun.Zhang at oicr.on.ca<mailto:Junjun.Zhang at oicr.on.ca>>
Date: Sunday, March 12, 2017 at 10:45 PM
To: Jonas Demeulemeester <Jonas.Demeulemeester at crick.ac.uk<mailto:Jonas.Demeulemeester at crick.ac.uk>>, George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>>
Cc: Miguel Vazquez <miguel.vazquez at cnio.es<mailto:miguel.vazquez at cnio.es>>, Denis Yuen <Denis.Yuen at oicr.on.ca<mailto:Denis.Yuen at oicr.on.ca>>, "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: Re: [DOCKTESTERS] Thanks!

Thanks Miguel and Jonas for your help here!

Do you have any update on the latest testing? Please feel free updating the wiki with any update: https://wiki.oicr.on.ca/display/PANCANCER/2017-03-13+PCAWG-TECH+Teleconference

Regards,
Junjun

From: Jonas Demeulemeester <Jonas.Demeulemeester at crick.ac.uk<mailto:Jonas.Demeulemeester at crick.ac.uk>>
Date: Saturday, March 11, 2017 at 7:15 PM
To: George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>>
Cc: Miguel Vazquez <miguel.vazquez at cnio.es<mailto:miguel.vazquez at cnio.es>>, Junjun Zhang <junjun.zhang at oicr.on.ca<mailto:junjun.zhang at oicr.on.ca>>, Denis Yuen <Denis.Yuen at oicr.on.ca<mailto:Denis.Yuen at oicr.on.ca>>, "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: Re: [DOCKTESTERS] Thanks!

Hi George,

Yup, I've been running the PCAWG dockers mainly using Miguel's set of scripts.
Give them a go and if you run into issues, just let us know!

Cheers,
Jonas

On 11 Mar 2017, at 17:00, George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>> wrote:

Sure, I'll give it a try and report later.

Thank you,
George Mihaiescu
Senior Cloud Architect

Ontario Institute for Cancer Research
MaRS Centre
661 University Avenue
Suite 510
Toronto, Ontario
Canada M5G 0A3

Email: George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>
Toll-free: 1-866-678-6427
Twitter: @OICR_news

www.oicr.on.ca<http://www.oicr.on.ca>

This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.

From: Miguel Vazquez <miguel.vazquez at cnio.es<mailto:miguel.vazquez at cnio.es>>
Date: Saturday, March 11, 2017 at 10:57 AM
To: Junjun Zhang <Junjun.Zhang at oicr.on.ca<mailto:Junjun.Zhang at oicr.on.ca>>
Cc: Denis Yuen <Denis.Yuen at oicr.on.ca<mailto:Denis.Yuen at oicr.on.ca>>, Jonas Demeulemeester <jonas.demeulemeester at crick.ac.uk<mailto:jonas.demeulemeester at crick.ac.uk>>, George Mihaiescu <George.Mihaiescu at oicr.on.ca<mailto:George.Mihaiescu at oicr.on.ca>>, "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: Re: [DOCKTESTERS] Thanks!

Hi Junjun,

I think Jonas has been using my scripts to run some of the tests, maybe George could try them as well, it should be very easy for him to try the Sanger, Delly+DKFZ, BWA-Mem, and the BiasFilter.

https://github.com/mikisvaz/PCAWG-Docker-Test

He would just need to update the tokens for DACO access and the scripts will take care of downloading the BAM files, running the workflows and evaluating the result.

The documentation there is reasonably updated, but if this sounds good then perhaps he could contact me and I could walk him through the details.

Best regards

Miguel

On Fri, Mar 10, 2017 at 9:51 PM, Junjun Zhang <Junjun.Zhang at oicr.on.ca<mailto:Junjun.Zhang at oicr.on.ca>> wrote:
Dear Docktesters,

George Mihaiescu, cloud architect, of the Collaboratory at OICR plans to run some bioinformatics workflows to test Collab environment.

Just thought this is a good opportunity to use as extra help for testing out the PCAWG dockerized workflows.

Miguel, Denis and others, what workflows / datasets do you think would be good for George to run?

Thanks,
Junjun

From: <docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org<mailto:docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org>> on behalf of Denis Yuen <Denis.Yuen at oicr.on.ca<mailto:Denis.Yuen at oicr.on.ca>>
Date: Wednesday, March 1, 2017 at 10:26 AM
To: "docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>" <docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>>
Subject: [DOCKTESTERS] Thanks!

Hi,

Just wanted to say thanks to Miguel and Jonas for keeping the workflow testing data page up-to-date.

https://wiki.oicr.on.ca/display/PANCANCER/Workflow+Testing+Data

As we work on new versions or debugging, it is invaluable to know what versions of the workflows have worked outside OICR, thanks!

Denis Yuen
Senior Software Developer

OntarioInstituteforCancerResearch
MaRSCentre
661 University Avenue
Suite510
Toronto, Ontario,Canada M5G0A3

Toll-free: 1-866-678-6427
Twitter: @OICR_news
www.oicr.on.ca<http://www.oicr.on.ca/>

This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.

_______________________________________________
docktesters mailing list
docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>
https://lists.icgc.org/mailman/listinfo/docktesters

The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 1 Midland Road London NW1 1AT

_______________________________________________
docktesters mailing list
docktesters at lists.icgc.org<mailto:docktesters at lists.icgc.org>
https://lists.icgc.org/mailman/listinfo/docktesters

The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 1 Midland Road London NW1 1AT
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.icgc.org/mailman/private/docktesters/attachments/20170322/4f39bdc2/attachment-0001.html>