SOP - Metagenome submission, publication and submission to INSDC via MG-RAST

Author(s): Andreas Wilke, andreas@mg-rast.org and the MG-RAST team
Version: 1.0
Introduction:
As explained in our user manual, MG-RAST can be used to host data for public access. There are three interfaces for uploading and publishing data, the Web interface, intended for most users, command line scripts, intended for programmers, and the native RESTful API, recommended for experienced programmers.
When data is published in MG-RAST, it can also be released to the INSDC databases. This tutorial covers both use cases.
We note that MG-RAST provides temporary IDs and permanent public identifiers. The permanent identifiers are assigned at the time data is made public. Permanent MG-RAST identifiers begin with “mgm” (e.g. “mgm4449249.3”) for data sets and mgp (e.g.”mgp128”) for projects/studies.
The following data types are supported:

• Shotgun metagenomes (“raw” and assembled)
• Metatranscriptome data (“raw” and assembled)
• Ribosomal amplicon data (16s, 18s, ITS amplicons)
• Metabarcoding data (e.g. cytochrome C amplicons; basically all non ribosomal amplicons)

PLEASE NOTE: We strongly prefer raw data over assembled data, if you submit assembled data, please submit the raw reads in parallel. If you perform local optimization e.g. adapter removal or quality clipping, please submit the raw data as well.
Audience:
This document is intended for experienced to very experienced users and programmers. We recommend that most users not use the RESTful API. There is also a document describing data publication and INSDC submission via the web UI.
Requirements:
An access token for the MG-RAST API, this can be obtained from the MG-RAST web page (http://mg-rast.org) in the user section.
You will need a working python interpreter and the command line scripts and example data can be found in https://github.com/MG-RAST/MG-RAST-Tools:

• Scripts: MG-RAST-Tools/tools/bin
• Data: MG-RAST-Tools/examples/sop/data

Change into MG-RAST-Tools/examples/sop/data and call:

sh get_test_data.sh

to add additional example data.
Either download the repository as a zipped archive from https://github.com/MG-RAST/MG-RAST-Tools/archive/master.zip or use the git command line tool:

git clone http://github.com/MG-RAST/MG-RAST-Tools.git

We have tested this for:

• max. size per file: 10GB
• max. project size: 200 metagenomes

There is no reason to assume it will not work for larger data sets.

SOP:
Upload and submit sequence data and metadata to MG-RAST using the command mg-submit.py Note: This is an asynchronous process that may take some time depending on the size and number of datasets. (Note: We recommend that novice users try the web frontend; the cmd-line is primarily intended for programmers) The metadata in this example is in Microsoft Excel format, there is also an option of using JSON formatted data. Please note: We have observed multiple problems with spreadsheets that were converted from older version of Excel or “compatible” tools e.g. OpenOffice.
Example:

mg-submit.py submit simple   … --metadata 

Verify the results and obtain a temporary identifier E.g. by using the WebUI at http://mg-rast.org -- you can also use that to publish the data and trigger submission to INSDC.
Publish your project in MG-RAST and obtain a stable and public MG-RAST project identifier
Note: once the data is made public the data is read only, but metadata can be improved
Example:

mg-project make-public $temporary_ID

Trigger release to INSDC/ submit to EBI
Note: Metadata updates are automatically synced with INSDC databases within 48 hours.
Example:

mg-project submit-ebi $PROJECT_ID

Check status of release to INSDC/ submission to EBI
Note: This is an asynchronous process that may take some time depending on the size and number of datasets.
Example:

mg-project status-ebi $PROJECT_ID

We include a sample submission below:
From within the MG-RAST-Tool repository directory

# Retrieve repository and setup environment
git clone http://github.com/MG-RAST/MG-RAST-Tools.git
cd MG-RAST-Tools

# Path to scripts for this example
PATH=$PATH:`pwd`/tools/bin

# set environment variables
source set_env.sh

# Set credentials, obtain token from your user preferences in the UI
mg-submit.py login --token 

# Create metadata spreadsheet. Make sure you map your samples to your 
# sequence files
# Upload metagenomes and metadata to MG-RAST

mg-submit.py submit simple \
           examples/sop/data/sample_1.fasta.gz \
           examples/sop/data/sample_2.fasta.gz \
           --metadata examples/sop/data/metadata.xlsx

# Output
> Temp Project ID: ed2102aa666d676d343735323836382e33
> Submission ID: 77a1a1a5-4cbd-4673-86bf-f87c9096c3e1

# Remember IDs for later use
SUBMISSION_ID=77a1a1a5-4cbd-4673-86bf-f87c9096c3e1
TEMP_ID=mgp128             

# Check if project is finished
mg-submit.py status $SUBMISSION_ID

# Output
> Submission: 77a1a1a5-4cbd-4673-86bf-f87c9096c3e1 Status: in-progress


# Make project public in MG-RAST
mg-project.py make-public $TEMP_ID

# Output
> # Your project is public. 
> Project ID: mgp128
> URL: metagenomics.anl.gov/linkin.cgi?project=mgp128
PROJECT_ID=mgp128

# Release project to INSDC archives
mg-project.py submit-ebi $PROJECT_ID

# Output
> # Your Project mgp128 has been submitted
> Submission ID: 0cf7d811-1d43-4554-ab97-3cb1f5ceb6aa

# Check if project is finished
mg-project.py status-ebi $PROJECT_ID

# Output
> Completed
> ENA Study Accession: ERP104408