[DOCKTESTERS] Amazon account

[Brian O'Connor]

Hi Junjun,

Great to hear this! 

I would recommend against trying to make this a docker in docker.  It’s something to be avoided because it really causes problems running in various environments.

Maybe a well commented bash script?  I think the instructions are long but really we expect these workflows to run in a lot of different environments and it’s better to explain how it works so users can customize for their environment.

What do you all think?

Brian

> On Sep 19, 2016, at 11:02 PM, Junjun Zhang <junjun.zhang at oicr.on.ca> wrote:
> 
> Good news, both Miguel and myself have Sanger pipeline running on AWS and Collab respectively.
> 
> Here is the documentation on all steps we went through to get things set up: https://docs.google.com/document/d/1EPo2Wgh-WJz75GdykgTI1fpm89yIdoGGHAyVlJ9PbcA/edit
> 
> As you can see there are quite some steps to go through, does it make sense to build a docker image for setting up testing environment?
> 
> It's kind of like docker in docker, is that OK?
> 
> Junjun
> 
> 
> From: docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org [docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org] on behalf of Denis Yuen [Denis.Yuen at oicr.on.ca]
> Sent: Monday, September 19, 2016 3:39 PM
> To: Miguel Vazquez
> Cc: docktesters at lists.icgc.org
> Subject: Re: [DOCKTESTERS] Amazon account
> 
> Hi,
> 
> Quoting myself from an email since it is applicable here too
> 
> Hi,
> Sounds reasonable, the hardware requirements listed at https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker#hardware-requirements match my recollection and a r3.4xlarge matches that handily in terms of RAM and CPU
> 
> The only thing I would check would be to make sure that the working directory (where you run Dockstore) is in a large 1TB volume. The workflow should be able to overwhelm 320 GB if that's all you have, causing it to crash.  
> 
> When we were running for Pan-cancer, we sometimes used lvm to merge all ephemeral drives on an AWS instance into one larger drive. 
> But for testing, it would probably be simpler just to use one large EBS volume. 
> 
> Denis Yuen
> Bioinformatics Software Developer
> 
> Ontario Institute for Cancer Research 
> MaRS Centre
> 661 University Avenue
> Suite 510
> Toronto, Ontario, Canada M5G 0A3
> Toll-free: 1-866-678-6427
> Twitter: @OICR_news
> www.oicr.on.ca
> This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.
> From: docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org [docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org] on behalf of Miguel Vazquez [mikisvaz at gmail.com]
> Sent: September 19, 2016 3:26 PM
> To: Francis Ouellette
> Cc: docktesters at lists.icgc.org; Zhibin
> Subject: Re: [DOCKTESTERS] Amazon account
> 
> Thanks Francis.
> 
> BTW Brian and Junjun, I think I might not have enough disk space in the instance you got for me, gtdownload croaks:
> 
> Error:  The system *might* run out of disk space before all downloads are complete, Downloading will continue until less than 1.00 GB is available.
> 
> and does not seem to download anything.
> 
> I took the liberty to make a directory in /mn/, which has 95GB avai. but it does not seem to change things. Excuse my ignorance, but how big are these files?
> 
> Miguel
> 
> On Mon, Sep 19, 2016 at 9:22 PM, Francis Ouellette <francis at oicr.on.ca> wrote:
> Hi Miguel,
> 
> I’m CCing docktester .. I think Junjun or Brian will be best to answer this …
>  I think a bit more RAM would be OK, so could do: m4.10xlarge (10 CPU and 160 GB RAM).
> 
> Would that be good Brian?
> 
> @bffo
> 
> 
> -- 
> B.F. Francis Ouellette          http://oicr.on.ca/person/francis-ouellette    
> 
> 
> 
>> On Sep 19, 2016, at 3:09 PM, Zhibin <zhibin at gmail.com> wrote:
>> 
>> Jo Miguel,
>> 
>> I am not familiar with Sanger pipeline. You should launch instances based on the number of CPUs and memory you need.
>> 
>> Best,
>> 
>> Zhibin
>> 
>> On Mon, Sep 19, 2016 at 3:05 PM, Miguel Vazquez <mikisvaz at gmail.com> wrote:
>> Hello again,
>> 
>> What image will you suggest to run the Sanger pipeline? I'm not very used to AWS and I wouldn't want to burn through your credit accidentally. I was thinking of r3.4xlarge that has 16 cores, 122GB memmory and 1x320 (SSD), would that be a good choice?
>> 
>> Best
>> 
>> Miguel
>> 
>> On Mon, Sep 19, 2016 at 8:42 PM, Miguel Vazquez <mikisvaz at gmail.com> wrote:
>> Thanks Zhibin
>> Best regards 
>> Miguel 
> 
> 
> 
> _______________________________________________
> docktesters mailing list
> docktesters at lists.icgc.org
> https://lists.icgc.org/mailman/listinfo/docktesters