[DOCKTESTERS] Amazon account

Tue Sep 20 10:21:47 EDT 2016

I was only thinking about making this docker image for the testing group so it's more efficient to do our work.

Junjun

> On Sep 20, 2016, at 9:44 AM, Oliver Hofmann <ohofmann72 at gmail.com> wrote:
> 
> I'll second that. A lot of this is specific to the ICGC samples as
> well (data access, etc.) whereas I expect many end users will want to
> just run this on their own WGS data.
> 
> Oliver
> 
> On 20 September 2016 at 09:43:02, Brian O'Connor
> (brian.oconnor at oicr.on.ca) wrote:
>> Hi Junjun,
>> 
>> Great to hear this!
>> 
>> I would recommend against trying to make this a docker in docker. It’s something to be
>> avoided because it really causes problems running in various environments.
>> 
>> Maybe a well commented bash script? I think the instructions are long but really we expect
>> these workflows to run in a lot of different environments and it’s better to explain how
>> it works so users can customize for their environment.
>> 
>> What do you all think?
>> 
>> Brian
>> 
>>> On Sep 19, 2016, at 11:02 PM, Junjun Zhang wrote:
>>> 
>>> Good news, both Miguel and myself have Sanger pipeline running on AWS and Collab respectively.
>>> 
>>> Here is the documentation on all steps we went through to get things set up: https://docs.google.com/document/d/1EPo2Wgh-WJz75GdykgTI1fpm89yIdoGGHAyVlJ9PbcA/edit
>>> 
>>> As you can see there are quite some steps to go through, does it make sense to build a docker
>> image for setting up testing environment?
>>> 
>>> It's kind of like docker in docker, is that OK?
>>> 
>>> Junjun
>>> 
>>> 
>>> From: docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org [docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org]
>> on behalf of Denis Yuen [Denis.Yuen at oicr.on.ca]
>>> Sent: Monday, September 19, 2016 3:39 PM
>>> To: Miguel Vazquez
>>> Cc: docktesters at lists.icgc.org
>>> Subject: Re: [DOCKTESTERS] Amazon account
>>> 
>>> Hi,
>>> 
>>> Quoting myself from an email since it is applicable here too
>>> 
>>> Hi,
>>> Sounds reasonable, the hardware requirements listed at https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker#hardware-requirements
>> match my recollection and a r3.4xlarge matches that handily in terms of RAM and CPU
>>> 
>>> The only thing I would check would be to make sure that the working directory (where you
>> run Dockstore) is in a large 1TB volume. The workflow should be able to overwhelm 320 GB
>> if that's all you have, causing it to crash.
>>> 
>>> When we were running for Pan-cancer, we sometimes used lvm to merge all ephemeral drives
>> on an AWS instance into one larger drive.
>>> But for testing, it would probably be simpler just to use one large EBS volume.
>>> 
>>> Denis Yuen
>>> Bioinformatics Software Developer
>>> 
>>> Ontario Institute for Cancer Research
>>> MaRS Centre
>>> 661 University Avenue
>>> Suite 510
>>> Toronto, Ontario, Canada M5G 0A3
>>> Toll-free: 1-866-678-6427
>>> Twitter: @OICR_news
>>> www.oicr.on.ca
>>> This message and any attachments may contain confidential and/or privileged information
>> for the sole use of the intended recipient. Any review or distribution by anyone other
>> than the person for whom it was originally intended is strictly prohibited. If you have
>> received this message in error, please contact the sender and delete all copies. Opinions,
>> conclusions or other information contained in this message may not be that of the organization.
>>> From: docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org [docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org]
>> on behalf of Miguel Vazquez [mikisvaz at gmail.com]
>>> Sent: September 19, 2016 3:26 PM
>>> To: Francis Ouellette
>>> Cc: docktesters at lists.icgc.org; Zhibin
>>> Subject: Re: [DOCKTESTERS] Amazon account
>>> 
>>> Thanks Francis.
>>> 
>>> BTW Brian and Junjun, I think I might not have enough disk space in the instance you got
>> for me, gtdownload croaks:
>>> 
>>> Error: The system *might* run out of disk space before all downloads are complete, Downloading
>> will continue until less than 1.00 GB is available.
>>> 
>>> and does not seem to download anything.
>>> 
>>> I took the liberty to make a directory in /mn/, which has 95GB avai. but it does not seem
>> to change things. Excuse my ignorance, but how big are these files?
>>> 
>>> Miguel
>>> 
>>> On Mon, Sep 19, 2016 at 9:22 PM, Francis Ouellette wrote:
>>> Hi Miguel,
>>> 
>>> I’m CCing docktester .. I think Junjun or Brian will be best to answer this …
>>> I think a bit more RAM would be OK, so could do: m4.10xlarge (10 CPU and 160 GB RAM).
>>> 
>>> Would that be good Brian?
>>> 
>>> @bffo
>>> 
>>> 
>>> --
>>> B.F. Francis Ouellette http://oicr.on.ca/person/francis-ouellette
>>> 
>>> 
>>> 
>>>> On Sep 19, 2016, at 3:09 PM, Zhibin wrote:
>>>> 
>>>> Jo Miguel,
>>>> 
>>>> I am not familiar with Sanger pipeline. You should launch instances based on the number
>> of CPUs and memory you need.
>>>> 
>>>> Best,
>>>> 
>>>> Zhibin
>>>> 
>>>> On Mon, Sep 19, 2016 at 3:05 PM, Miguel Vazquez wrote:
>>>> Hello again,
>>>> 
>>>> What image will you suggest to run the Sanger pipeline? I'm not very used to AWS and I
>> wouldn't want to burn through your credit accidentally. I was thinking of r3.4xlarge
>> that has 16 cores, 122GB memmory and 1x320 (SSD), would that be a good choice?
>>>> 
>>>> Best
>>>> 
>>>> Miguel
>>>> 
>>>> On Mon, Sep 19, 2016 at 8:42 PM, Miguel Vazquez wrote:
>>>> Thanks Zhibin
>>>> Best regards
>>>> Miguel
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> docktesters mailing list
>>> docktesters at lists.icgc.org
>>> https://lists.icgc.org/mailman/listinfo/docktesters
>> 
>> _______________________________________________
>> docktesters mailing list
>> docktesters at lists.icgc.org
>> https://lists.icgc.org/mailman/listinfo/docktesters
> 
> --
> @fiamh
> Skype: ohofmann
> +44 795 621 5540