[DOCKTESTERS] Amazon account

Tue Sep 20 09:44:28 EDT 2016

I'll second that. A lot of this is specific to the ICGC samples as
well (data access, etc.) whereas I expect many end users will want to
just run this on their own WGS data.

Oliver

On 20 September 2016 at 09:43:02, Brian O'Connor
(brian.oconnor at oicr.on.ca) wrote:
> Hi Junjun,
>
> Great to hear this!
>
> I would recommend against trying to make this a docker in docker. It’s something to be
> avoided because it really causes problems running in various environments.
>
> Maybe a well commented bash script? I think the instructions are long but really we expect
> these workflows to run in a lot of different environments and it’s better to explain how
> it works so users can customize for their environment.
>
> What do you all think?
>
> Brian
>
> > On Sep 19, 2016, at 11:02 PM, Junjun Zhang wrote:
> >
> > Good news, both Miguel and myself have Sanger pipeline running on AWS and Collab respectively.
> >
> > Here is the documentation on all steps we went through to get things set up: https://docs.google.com/document/d/1EPo2Wgh-WJz75GdykgTI1fpm89yIdoGGHAyVlJ9PbcA/edit
> >
> > As you can see there are quite some steps to go through, does it make sense to build a docker
> image for setting up testing environment?
> >
> > It's kind of like docker in docker, is that OK?
> >
> > Junjun
> >
> >
> > From: docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org [docktesters-bounces+junjun.zhang=oicr.on.ca at lists.icgc.org]
> on behalf of Denis Yuen [Denis.Yuen at oicr.on.ca]
> > Sent: Monday, September 19, 2016 3:39 PM
> > To: Miguel Vazquez
> > Cc: docktesters at lists.icgc.org
> > Subject: Re: [DOCKTESTERS] Amazon account
> >
> > Hi,
> >
> > Quoting myself from an email since it is applicable here too
> >
> > Hi,
> > Sounds reasonable, the hardware requirements listed at https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker#hardware-requirements
> match my recollection and a r3.4xlarge matches that handily in terms of RAM and CPU
> >
> > The only thing I would check would be to make sure that the working directory (where you
> run Dockstore) is in a large 1TB volume. The workflow should be able to overwhelm 320 GB
> if that's all you have, causing it to crash.
> >
> > When we were running for Pan-cancer, we sometimes used lvm to merge all ephemeral drives
> on an AWS instance into one larger drive.
> > But for testing, it would probably be simpler just to use one large EBS volume.
> >
> > Denis Yuen
> > Bioinformatics Software Developer
> >
> > Ontario Institute for Cancer Research
> > MaRS Centre
> > 661 University Avenue
> > Suite 510
> > Toronto, Ontario, Canada M5G 0A3
> > Toll-free: 1-866-678-6427
> > Twitter: @OICR_news
> > www.oicr.on.ca
> > This message and any attachments may contain confidential and/or privileged information
> for the sole use of the intended recipient. Any review or distribution by anyone other
> than the person for whom it was originally intended is strictly prohibited. If you have
> received this message in error, please contact the sender and delete all copies. Opinions,
> conclusions or other information contained in this message may not be that of the organization.
> > From: docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org [docktesters-bounces+denis.yuen=oicr.on.ca at lists.icgc.org]
> on behalf of Miguel Vazquez [mikisvaz at gmail.com]
> > Sent: September 19, 2016 3:26 PM
> > To: Francis Ouellette
> > Cc: docktesters at lists.icgc.org; Zhibin
> > Subject: Re: [DOCKTESTERS] Amazon account
> >
> > Thanks Francis.
> >
> > BTW Brian and Junjun, I think I might not have enough disk space in the instance you got
> for me, gtdownload croaks:
> >
> > Error: The system *might* run out of disk space before all downloads are complete, Downloading
> will continue until less than 1.00 GB is available.
> >
> > and does not seem to download anything.
> >
> > I took the liberty to make a directory in /mn/, which has 95GB avai. but it does not seem
> to change things. Excuse my ignorance, but how big are these files?
> >
> > Miguel
> >
> > On Mon, Sep 19, 2016 at 9:22 PM, Francis Ouellette wrote:
> > Hi Miguel,
> >
> > I’m CCing docktester .. I think Junjun or Brian will be best to answer this …
> > I think a bit more RAM would be OK, so could do: m4.10xlarge (10 CPU and 160 GB RAM).
> >
> > Would that be good Brian?
> >
> > @bffo
> >
> >
> > --
> > B.F. Francis Ouellette http://oicr.on.ca/person/francis-ouellette
> >
> >
> >
> >> On Sep 19, 2016, at 3:09 PM, Zhibin wrote:
> >>
> >> Jo Miguel,
> >>
> >> I am not familiar with Sanger pipeline. You should launch instances based on the number
> of CPUs and memory you need.
> >>
> >> Best,
> >>
> >> Zhibin
> >>
> >> On Mon, Sep 19, 2016 at 3:05 PM, Miguel Vazquez wrote:
> >> Hello again,
> >>
> >> What image will you suggest to run the Sanger pipeline? I'm not very used to AWS and I
> wouldn't want to burn through your credit accidentally. I was thinking of r3.4xlarge
> that has 16 cores, 122GB memmory and 1x320 (SSD), would that be a good choice?
> >>
> >> Best
> >>
> >> Miguel
> >>
> >> On Mon, Sep 19, 2016 at 8:42 PM, Miguel Vazquez wrote:
> >> Thanks Zhibin
> >> Best regards
> >> Miguel
> >
> >
> >
> > _______________________________________________
> > docktesters mailing list
> > docktesters at lists.icgc.org
> > https://lists.icgc.org/mailman/listinfo/docktesters
>
> _______________________________________________
> docktesters mailing list
> docktesters at lists.icgc.org
> https://lists.icgc.org/mailman/listinfo/docktesters
>

--
@fiamh
Skype: ohofmann
+44 795 621 5540