[DOCKTESTERS] Fwd: [PAWG-TECH] Draft agenda for PCAWG-TECH teleconference
Miguel Vazquez
miguel.vazquez at cnio.es
Wed Oct 5 06:44:39 EDT 2016
Hi all,
There was en Error running the test data for the DKFZ Workflow. It appears
to have some path hardcoded somewhere:
Here are the logs:
8<---------------
+ [[ 1 == 0 ]]
+ for logfile in '${jobstateFiles[@]}'
++ cat
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_071931526_roddy_snvCalling/jobStateLogfile.txt
++ grep -v null:
++ wc -l
++ grep :STARTED:
+ cntStarted=4
++ cat
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_071931526_roddy_snvCalling/jobStateLogfile.txt
++ grep -v null:
++ grep :0:
++ wc -l
+ cntSuccessful=4
++ expr 4 - 4
+ cntErrornous=0
+ [[ 0 -gt 0 ]]
+ [[ 0 == 0 ]]
+ echo 'No errors found for
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_071931526_roddy_snvCalling/jobStateLogfile.txt'
No errors found for
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_071931526_roddy_snvCalling/jobStateLogfile.txt
+ for logfile in '${jobstateFiles[@]}'
++ cat
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_072436413_roddy_indelCalling/jobStateLogfile.txt
++ grep -v null:
++ grep :STARTED:
++ wc -l
+ cntStarted=2
++ cat
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_072436413_roddy_indelCalling/jobStateLogfile.txt
++ grep -v null:
++ wc -l
++ grep :0:
+ cntSuccessful=2
++ expr 2 - 2
+ cntErrornous=0
+ [[ 0 -gt 0 ]]
+ [[ 0 == 0 ]]
+ echo 'No errors found for
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_072436413_roddy_indelCalling/jobStateLogfile.txt'
No errors found for
/mnt/datastore/testdata/run_id/roddyExecutionStore/exec_161005_072436413_roddy_indelCalling/jobStateLogfile.txt
There was at least one error in a job status logfile. Will exit now!
+ [[ true == true ]]
+ echo 'There was at least one error in a job status logfile. Will exit
now!'
+ exit 5
*mv: cannot stat `/mnt/datastore/resultdata/*': No such file or directory*
Result directory listing is:
Error while running job: Error collecting output for parameter
'germline_indel_vcf_gz': Did not find output file with glob pattern:
'['*.germline.indel.vcf.gz']'
[job temp5368999563481290609.cwl] completed permanentFail
Final process status is permanentFail
Workflow error, try again with --debug for more information:
Process status is ['permanentFail']
stdout :
java.lang.RuntimeException: problems running command: cwltool --enable-dev
--non-strict --enable-net --outdir
/mnt/1TB/work/DockerTest-Miguel/tests/DKFZ/HCC1143/./datastore/launcher-7bb3a7ef-64e2-43ac-81c9-dd42e1b35f1b/outputs/
--tmpdir-prefix
/mnt/1TB/work/DockerTest-Miguel/tests/DKFZ/HCC1143/./datastore/launcher-7bb3a7ef-64e2-43ac-81c9-dd42e1b35f1b/working/
/tmp/1475650852550-0/temp5368999563481290609.cwl
/mnt/1TB/work/DockerTest-Miguel/tests/DKFZ/HCC1143/./datastore/launcher-7bb3a7ef-64e2-43ac-81c9-dd42e1b35f1b/workflow_params.json
8<---------------
The Dockerstore.json I used is as follows:
{
"run-id": "run_id",
"tumor-bam":
{
"path":"/home/ubuntu/DockerTest-Miguel/data/HCC1143/tumor.bam",
"class":"File"
},
"normal-bam":
{
"path":"/home/ubuntu/DockerTest-Miguel/data/HCC1143/normal.bam",
"class":"File"
},
"reference-gz":
{
"path":
"/home/ubuntu/DockerTest-Miguel/resources//dkfz-workflow-dependencies_150318_0951.tar.gz",
"class": "File"
},
"delly-bedpe":
{
"path":"/home/ubuntu/DockerTest-Miguel/resources//run_id.embl-delly_1-3-0-preFilter.20150318.somatic.sv.bedpe.txt",
"class":"File"
},
"germline_indel_vcf_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.germline.indel.vcf.gz",
"class": "File"
},
"somatic_snv_mnv_vcf_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.somatic.snv.mnv.vcf.gz",
"class": "File"
},
"germline_snv_mnv_vcf_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.germline.snv.mnv.vcf.gz",
"class": "File"
},
"somatic_cnv_tar_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.somatic.cnv.tar.gz",
"class": "File"
},
"somatic_cnv_vcf_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.somatic.cnv.vcf.gz",
"class": "File"
},
"somatic_indel_tar_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.somatic.indel.tar.gz",
"class": "File"
},
"somatic_snv_mnv_tar_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.somatic.snv.mnv.tar.gz",
"class": "File"
},
"somatic_indel_vcf_gz": {
"path":
"/home/ubuntu/DockerTest-Miguel/tests/DKFZ/HCC1143//output//HCC1143.somatic.indel.vcf.gz",
"class": "File"
}
}
On Wed, Oct 5, 2016 at 10:44 AM, Miguel Vazquez <miguel.vazquez at cnio.es>
wrote:
> Hello all,
>
> I've created a repository in
>
> https://github.com/mikisvaz/PCAWG-Docker-Test
>
> With scripts that can help you run the tests. They can help download
> samples, prepare the Dockerstore.json files and run them. I've added an
> entry in the google doc.
>
> I've ran the Sanger on the HCC1143 (Test) data and it seemed to have
> completed correctly (I updated the table). I'm in the process now of
> running the same test data in the DKFZ and sample DO50311 on the Delly
> workflow.
>
> I have two standing questions:
>
> - What are the other workflows I need to test?
> - What is the bedpe file used by DKFZ and how to produce it. Denis has
> pointed me to some forum threads that I have to study a bit, but it seems
> that we need to run something to produce it before we can run the workflow,
> and I wonder how this fits into our testing scheme. I'll read more about it
> when I have a chance.
>
> Best regards
>
>
>
> On Fri, Sep 30, 2016 at 3:52 PM, Francis Ouellette <francis at oicr.on.ca>
> wrote:
>
>> Dear Miguel and Junjun,
>>
>> Any more attempts on testing the PCAWG sanger docker container?
>>
>> If you reproduce the same error, we will need to involve Bryan and
>> Keiran Raine <kr2 at sanger.ac.uk> (author of the container).
>>
>> Let’s get this one figured out.
>>
>> I am going to assume that the making of the docker container is what
>> needs resolving.
>>
>> Brian: We may need your input here.
>>
>> Details of our current experiment should continue to be posted here:
>>
>> https://goo.gl/XX5BG9
>>
>> Thank you all,
>>
>> francis
>>
>> PS would be good for others on list to follow directions on above google
>> doc and also see if they can succeed on this workflow.
>>
>> Junjun and Miguel have tried different clouds, but used the sanger
>> workflow, on the same data set.
>>
>> Thank you for trying to do this.
>>
>> Would be good if I heard back from anybody before Monday AM (tech call).
>>
>> @bffo
>>
>> PS I CCed Keiran, but waiting to hear back from Brian before we need to
>> involve him some more.
>> PPS Junjun/Miguel: maybe you can try the DKFZ docker as well? (on the
>> same data set).
>>
>>
>> --
>> B.F. Francis Ouellette http://oicr.on.ca/person/fran
>> cis-ouellette
>>
>>
>>
>> Begin forwarded message:
>>
>> *From: *Christina Yung <Christina.Yung at oicr.on.ca>
>> *Subject: **[PAWG-TECH] Draft agenda for PCAWG-TECH teleconference*
>> *Date: *September 30, 2016 at 9:19:39 AM EDT
>> *To: *"pawg-tech (pawg-tech at lists.icgc.org)" <pawg-tech at lists.icgc.org>
>>
>> Hi Everyone,
>>
>>
>>
>> Below is a draft agenda for Monday’s tech call. Please let me know if
>> you have any agenda items for discussion.
>>
>> https://wiki.oicr.on.ca/display/PANCANCER/2016-10-03+PCAWG-
>> TECH+Teleconference
>>
>>
>>
>> Have a great weekend!
>>
>> Christina
>>
>>
>> Call Info
>>
>> *Usual Time 9 AM Eastern Time, Mondays*
>>
>> *UK 0208 322 2500*
>>
>> *Canada 1-866-220-6419*
>>
>> *United States **1-877-420-0272*
>>
>> *All Others Please see attached PDF file
>> <https://wiki.oicr.on.ca/download/attachments/57776990/PAWG-TECH-Telecon-International-Numbers.pdf?version=1&modificationDate=1401456003000&api=v2> with
>> a list of numbers for other countries.*
>>
>> *Participant Code 5910819#*
>> Agenda
>>
>>
>>
>> *Time*
>>
>> *Item*
>>
>> *Who*
>>
>> *Attachments/Links*
>>
>> 5min
>>
>> Welcome. Wait for group members to log on
>>
>> Christina Yung <https://wiki.oicr.on.ca/display/~cyung>, OICR
>>
>> 10min
>>
>> Overall status
>> Christina Yung <https://wiki.oicr.on.ca/display/~cyung>, OICR
>>
>> · Linkouts to Most Current PCAWG Data
>> <https://wiki.oicr.on.ca/display/PANCANCER/Linkouts+to+Most+Current+PCAWG+Data>
>>
>> · Report data issues to pcawg-data at icgc.org, GNOS issue to:
>> Help at annaisystems.com
>>
>> · From Boston F2F: PCAWG datasets & dependencies
>> <https://docs.google.com/spreadsheets/d/1VltMMLL5XDIFipM3zzLqZOGj5z-UxIMKLDbsuDiD2Q8/edit?usp=sharing>
>>
>> *Action Items*
>>
>> 1. [Joachim] Consensus SV - final?
>>
>> 2. [Jakob] Consensus SNVs - changes to "SNV near indels"
>> annotation?
>>
>> 3. [Junjun] Specimen ID mapping for miRNA and methylation
>>
>> 4. [Jonathan & Joachim] Consensus calls for cell lines, followed
>> by filtering
>>
>> 5. [Matthias & Gordon] Docker containers for filtering methods
>>
>> 6. [Christina] Run alignment & variant workflows on
>> medulloblastoma sample (tumor 40x, normal 30x) from ICGC benchmark to
>> estimate false negative rate
>>
>> 7. [Christina] Follow up with institutes interested in hosting
>> PCAWG data long-term
>>
>> 8. [All] As per Jennifer's email on Sept 16, please provide
>> authorship information again or for the first time using PCAWG Author Form (
>> http://goo.gl/forms/5Wq5x5X1DK). Save the link "Edit your response" so
>> you can go back later to provide updates, for example about your evolving
>> role in writing specific papers.
>>
>> 9. [All] Contribute to the manuscripts on
>>
>> a. infrastructure: https://docs.google.com/docume
>> nt/d/10alAxrWLdLSyhci-rfNuVH13rFXCJkaY_rzf1KJn7nc/edit
>>
>> b. variants: Paper ( https://goo.gl/g9CLsu ), Supplement (
>> https://goo.gl/EWYh7e )
>>
>> c. Rogue's Gallery of Cancer Genome Sequencing Artifacts ( outline
>> <https://wiki.oicr.on.ca/download/attachments/66958609/Rogues%E2%80%99%20Gallery%20of%20Cancer%20Genome%20Sequencing%20Artefacts.pdf?version=1&modificationDate=1469802643000&api=v2>
>> )
>>
>> 10. [Junjun] Discuss PCAWG vs DCC glossary terms at next PCAWG-10/13
>> conference
>>
>> 10min
>>
>> Status of dockerizing workflows
>> Brian O'Connor <https://wiki.oicr.on.ca/display/~boconnor>, UCSC
>>
>> Gordon Saksena
>> <https://wiki.oicr.on.ca/display/~gsaksena@broadinstitute.org>, Broad
>>
>> Francis Ouellette <https://wiki.oicr.on.ca/display/~francis>, OICR
>> *Status of PCAWG Workflow ports to Dockstore*:
>>
>> Denis has been porting the Dockstore entries to CWL version 1.0 which is
>> part of our effort to publish Dockstore (this doesn't affect the content of
>> the pipelines, simply their "descriptors" which allow them to be runnable
>> via Dockstore). Denis has also worked on testing BWA-Mem, Sanger, EMBL
>> (which all work with CWL 1.0 and Kerian's test dataset) and has fixed
>> issues with DKFZ and is testing the latter with a real sample shortly.
>>
>> 1. BWA-Mem - Ready for testing by Francis' team
>>
>> 2. Sanger - Ready for testing by Francis' team.
>>
>> 3. EMBL - Ready for testing by Francis' team.
>>
>> 4. DKFZ - Ready for testing by Francis' team. I've exchanged
>> emails with Manuel Ballesteros who has been testing this pipeline.
>>
>> 5. Broad - Variant calling (MuTect, dRanger, snowman), need some
>> work, Gordan sent details previously
>>
>> 6. OxoG - Waiting for Dimitri to provide OxoG docker
>>
>> 7. Variantbam
>>
>> 8. Consensus algorithm
>>
>> PCAWG Docker (Dockstore) Testing Working Group
>> <https://wiki.oicr.on.ca/display/PANCANCER/PCAWG+Docker+%28Dockstore%29+Testing+Working+Group>
>>
>> 5min
>>
>> Other business?
>>
>> Group
>>
>>
>>
>>
>>
>>
>>
>> *Christina K. Yung, PhD*
>> Project Manager, Cancer Genome Collaboratory
>>
>> *Ontario Institute for Cancer Research*
>> MaRS Centre
>>
>> 661 University Avenue, Suite 510
>> Toronto, Ontario, Canada M5G 0A3
>> Tel: 416-673-8578
>>
>> www.oicr.on.ca
>>
>>
>>
>> This message and any attachments may contain confidential and/or
>> privileged information for the sole use of the intended recipient. Any
>> review or distribution by anyone other than the person for whom it was
>> originally intended is strictly prohibited. If you have received this
>> message in error, please contact the sender and delete all copies.
>> Opinions, conclusions or other information contained in this message may
>> not be that of the organization.
>>
>>
>> _______________________________________________
>> PAWG-TECH mailing list
>> PAWG-TECH at lists.icgc.org
>> https://lists.icgc.org/mailman/listinfo/pawg-tech
>>
>>
>>
>> _______________________________________________
>> docktesters mailing list
>> docktesters at lists.icgc.org
>> https://lists.icgc.org/mailman/listinfo/docktesters
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.icgc.org/mailman/private/docktesters/attachments/20161005/66cffd3c/attachment-0001.html>
More information about the docktesters
mailing list