**Data Release Notes for the workspace**

Update: November 28, 2023

* On November 28, 2023, directory 'temp_QC' was removed as all the data therein were unusable. 

Update: October 31, 2023

* On October 27, 2023, a top directory cleanup was performed that removed 38,233 unusable files which were a product of aborted/failed file transfers. Please reach out to AnVIL Support for more information. 

Update: May 23, 2023

* From March 30, 2023 to May 9, 2023, the following files below were temporarily unavailable. As of May 9, 2023, these files have been restored. 

assembly_verkko_v1.2_trio.tar
m54329U_220829_095708-bc2050.5mc.hifi_reads.bam
m54329U_220107_233847-bc1016.5mc.hifi_reads.bam
m54329U_211230_014258-bc1010.5mc.hifi_reads.bam
m54329U_220117_123334-bc1012.5mc.hifi_reads.bam
m64076_220516_221911-bc2018.5mc.hifi_reads.bam
m64076_220518_191414-bc2018.5mc.hifi_reads.bam
m64076_220826_143529-bc2049.5mc.hifi_reads.bam
r54329U_20220901_220229_C01.reads.bam
r54329U_20220901_220229_D01.reads.bam
HG00621.GRCh38_no_alt.bam
HG01106.GRCh38_no_alt.bam
HG01109.CHM13Y_EBV.bam
HG01109.GRCh38_no_alt.bam
HG01978.CHM13Y_EBV.bam
HG02080.GRCh38_no_alt.bam
HG02486.GRCh38_no_alt.bam
HG02559.GRCh38_no_alt.bam
NA20129.GRCh38_no_alt.bam
NA21309.GRCh38_no_alt.bam
01_09_20_R941_GM24631_2.fast5.tar
01_09_20_R941_GM24631_3.fast5.tar
HG00733_1.fast5.tar.gz
HG00733_2.fast5.tar.gz
HG00733_3.fast5.tar.gz
m54329U_200615_084313.ccs.bam


# Human Pangenome Reference Consortium's AnVIL_HPRC Workspace

![White Logo](https://s3-us-west-2.amazonaws.com/human-pangenomics/backup/logo-proof-full.png)

This workspace holds sequencing, assembly, and pangenome data submitted to the [Human Pangenome Reference Consortium](https://humanpangenome.org/). Data is stored in this workspace to allow immediate use by researchers participating in the Human Pangenome Project. Data in this workspace is constantly being added and updated and the workspace is under active development as our production pipeline continues. Note that the HPRC Year 1 data is pending publication, currently scheduled for march of 2022. If you would like to publish using this data please contact the HPRC Coordinating Center at hprcadmin@gowustl.onmicrosoft.com.

### Data Table Organization
**Sequencing Data:** Sample Table 
**Participant Information:** Participant Table 
**Assemblies:** Assembly_Sample, Assembly_Annotation Tables 
**Pangenomes:** Minigraph, Minigraph_CACTUS, PGGB Tables

------------------
# Sequencing Data
Production information about the HPRC Y1 samples is in the sections below. Note that samples from the HPRC_PLUS cohort had data assembled from other consortia, and the information in the sections below does not apply.

## PacBio HiFi Data From UW and WashU
PacBio HiFi sequencing was performed on the Sequel2 instrument using v2 chemistry. Samples were sequenced across 3-5 flowcells in order to generate 35-fold coverage or greater. The average coverage for the year1 samples is >40X with a median insert size of 20kb. The following file types are listed in the table:

* {flowcell_id}.ccs.bam - consensus called unaligned bam file

Consensus sequences were generated with v4.0.0 of the CCS algorithm

## Nanopore PromethION Data From UC Santa Cruz

The UCSC Nanopore PromethION HPP data were generated using the a new long read protocol (details to be published on protocols.io shortly). Briefly, this protocol uses Circulomics discs-based UHMW DNA isolation, followed by library preparation using ONT SQK-RAD004 kit and library cleanup using Circulomics discs. Typically, we used 3 RAD004-based sequencing libraries per PromethION flow cell (and 3 flow cells per individual). The observed read N50s (average) are ~80 kb, with 11x coverage in 100kb+ readsa dn 3x coverage in 200kb+ reads for each individual. The following file types are listed in the table:

* sample_Guppy_4.0.11_prom.fastq.gz

## Hi-C/Omni-C Data From UCSC
OmniC is a restriction enzyme-free HiC protocol. DNA is fragmented using an endonuclease that cuts DNA randomly, i.e., not at specific restriction sites. It uses a biotinylated linker to facilitate proximity ligation. This short linker sequence may be present within the read data. The architecture of an OmniC library molecule is:

P5adapter:GenomicRegion:Linker:GenomicRegion:P7adapter

Depending on the read length, some or all of the Linker sequence may be present in R1, R2, or both. When present, the Linker sequence is: AGGTTCGTCCATCGATCGATGGACGAACCT Note that the linker sequence is a DNA palindrome. Thus, the Linker sequence will be the same whether present in R1 or R2 (forward or reverse read).

For each HPRC sample, two OmniC libraries are generated and sequenced. Duplicates can be identified and removed within each library, but not between libraries as independent sample material goes into each. Libraries are sequenced across several Novaseq lanes, depending on lane availability, with 2x150 paired-end sequencing.

The raw fastq OmniC data can be aligned using bwa mem with flags -5SP In practice, bwa mem (local alignment) will not omit the Linker sequence from the alignment in cases when it is present in the read.

The following file types are available:

* {library_name}.R1_001.fastq.gz - forward reads
* {library_name}.R2_001.fastq.gz - reverse reads
 
## Bionano Data From Rockefeller
The following file types are available:

* {sample_name}_Saphyr_DLE1_3680591.bnx.gz - raw data view of molecule and label information and quality scores
* {sample_name}_Saphyr_DLE1_3680616.cmap - location information for label sites

For information about Bionano file specifications or utilization of Bionano files see Bionano's website.

## Strand-seq Data From EMBL
Strand-seq is a single-cell sequencing technology that resolves individual homologs within a cell by restricting sequence analysis to the DNA template strands used during DNA replication (see Sanders, et al.). Each sample has 192 fastq.gz files available -- from paired-end sequencing of 96 single cells (nuclei).

## Trio Illumina Data From NYGC
When available, we have included high coverage (30X) Illumina datasets produced by NYGC for select 1000G samples. Information about this dataset can be found at [IGSR](https://www.internationalgenome.org/data-portal/data-collection/30x-grch38).

Note that the data is stored in CRAM format (aligned to GRCHh38), but is placed in the raw_data area for each sample as the consortium uses the raw reads from the CRAMs. When used in assembly pipelines, the [extract_reads.wdl](https://github.com/human-pangenomics/hpp_production_workflows/blob/master/QC/wdl/tasks/extract_reads.wdl) is used to call samtools fastq to convert the reads to fastq format.


------------------
# Assemblies

The **assembly_sample** holds assemblies from year 1 data. Assemblies from 47 samples have been uploaded to Genbank and released. Genbank accessions can be found [here](https://github.com/human-pangenomics/HPP_Year1_Assemblies/blob/main/genbank_changes/y1_genbank_assembly_accession_ids.txt). The table has columns for maternal and paternal fastas alongside columns for each assembly aligned to both CHM13 and GRCh38. The **assembly_annotation** table holds annotation files for the assemblies found in the **assembly_sample** table.

For more information about the assembly process see the [Y1 Assembly GitHub repo](https://github.com/human-pangenomics/HPP_Year1_Assemblies)
## Assembly Metrics
**Select metrics from the [automated assembly QC]() are shown below**

### QV
![QV](https://github.com/human-pangenomics/HPP_Year1_Assemblies/blob/main/automated_qc_results/Y1_assemblies_v2_genbank_QV.png?raw=true)

### N50
![N50](https://github.com/human-pangenomics/HPP_Year1_Assemblies/blob/main/automated_qc_results/Y1_assemblies_v2_genbank_N50.png?raw=true)

### Hamming Error Rate
![Hamming](https://github.com/human-pangenomics/HPP_Year1_Assemblies/blob/main/automated_qc_results/Y1_assemblies_v2_genbank_hamming.png?raw=true)

### Switch Error Rate
![Switch](https://github.com/human-pangenomics/HPP_Year1_Assemblies/blob/main/automated_qc_results/Y1_assemblies_v2_genbank_switch.png?raw=true)

------------------

# Pangenomes
Pangenomes are available from three different strategies summarized in the table (and relevant sections) below:

| | **Minigraph** | **Minigraph-Cactus** | **PGGB** |
| :-------- | :-------- | :------ | :------ |
| sequence comparison | reference-based, progressive | reference-based, progressive | symmetric, all-vs-all |
| resolution | SV only | base-level (via abPOA) | base-level (via abPOA) |
| scope | full assemblies | Non-centromeric | full assemblies |
| cyclic paths | no | non-reference | all |
| short read mapping | untested | yes (fast) | untested |
| long read mapping | yes (fastest) | yes | yes (slowest) |
| Assembly mapping | yes (direct) | untested | yes (via injection) |


### Assembly Inputs

The assemblies in the **assembly_sample** table were used as inputs to graph building pipelines. Of the 47 samples assembled (94 assemblies) in year 1, all but three samples were included in graph constructions (HG002, HG005 and NA19240 were excluded for evaluation purposes). GRCh38 and CHM13 were added to make the total number of haplotypes included 90.


## Graphs
### Minigraph

[Minigraph](https://github.com/lh3/minigraph) is a generalization of minimap2 (very fast) which builds the graph with iterative construction. Minigraph aligns with approximate locations and can be used to call structural variants (>50nt). Graphs were built with both GRCh38 and CHM13+Y (found [here](https://s3-us-west-2.amazonaws.com/human-pangenomics/pangenomes/freeze/freeze1/minigraph/CHM13v11Y.fa.gz)) used as reference sequences.

Graphs and associated files are available in the **minigraph** table.

### Minigraph/CACTUS

[The CACTUS pangenome pipeline](https://github.com/ComparativeGenomicsToolkit/cactus/blob/master/doc/pangenome.md) adds base-level alignments to the minigraph graphs above (so both GRCh38- and CHM13-based graphs are available). Centromeric regions are removed during Minigraph/CACTUS graph creation.

Graphs and associated files are available in the **minigraph_cactus** table.

The graphs are available in gfa format alongside other graph and index files. Information about the associated file formats can be found:
* graph formats: [xg/gg](https://github.com/vgteam/vg/wiki/Index-Types)
* index formats: [gbwt/dist/min](https://github.com/vgteam/vg/wiki/Index-Types)
* snarls format: [snarls](https://github.com/vgteam/vg/wiki/Index-Construction)

#### Filtered Graphs
The [Giraffe](https://www.biorxiv.org/content/10.1101/2020.12.04.412486v2.abstract) short read mapper relies on the graph's snarl decomposition. The versions of the Cactus/Minigraph graphs released here contain some spurious large deletion edges that make this decomposition less efficient, which impacts Giraffe *runtime*. Furthermore, we have found that for calling small variants with the Giraffe-[DeepVariant](https://doi.org/10.1038/nbt.4235) pipeline, *accuracy* is improved if all alleles with frequency < 10% are removed from the graph before indexing. Two filtered versions of each of the two Minigraph/Cactus graphs are available. The graphs with `maxdel.10mb` in the name (recommended to speed up general mapping experiments) were created by removing edges that imply deletions > 10mb, and the graphs with `minaf.0.1` in the name (recommended when using with DeepVariant) were created by removing, in addition to the deletions, nodes that are covered by fewer than 9 haplotypes.

### PGGB

The Pangenome Graph Builder pipeline ([PGGB](https://github.com/pangenome/pggb)) creates and all-vs-all graph with base-level alignments and no clipping of mitochondrial or centromeric regions.

The graph files can be found in the **pggb** table








To view currently available sequencing data, navigate to the DATA tab of this workspace to view data stored in the "sample" data table. In-progress assembiles can also be found in the "sample_assembly" table. These assemblies were generated with Hifiasm v0.14.

Data can also be accessed from our [S3 Bucket](https://s3-us-west-2.amazonaws.com/human-pangenomics/index.html?prefix=working/) without egress fees. Information about the currently available data can be found on our [GitHub page](https://github.com/human-pangenomics/HPP_Year1_Data_Freeze_v1.0).

------------------

### Data Use Policy

Publication Protocol âÂ¢Â¬ÃÂ¬Ã¶âÃÂ¬ÃâÃÂ¬Â¨ V1.0 | April 7, 2020

The goal of the HPRC publication policy is to encourage collaboration and coordination among investigators while ensuring the timely release of research to the scientific community. The document provides general guidelines and a description of the process for disclosure of planned publications within the consortium through submission. This policy will be drafted and updated by the HPRC Administration and Coordination Working Group and approved by the HPRC steering committee.

**General Guidelines**
* This policy applies to all manuscripts resulting from HPRC funded activities.
* It is the responsibility of the HPRC members to take steps necessary to protect any intellectual property rights.
* Any publication must comply with the NIH Public Access policy and authors should make efforts to publish their work using open access policies when possible.
* Individual investigators funded under the HPRC may separately or collaboratively publish the results of their own work.
* Investigators within HPRC who have similar work are encouraged to work together on publications.
* A manuscript tracking sheet will be used to share information about planned manuscripts.
 * Entries in the manuscript tracking sheet are the responsibility of the corresponding author.
 * The Information requested in the manuscript tracking sheet will include the following: Authors, Title, Status, Potential overlap with other HPRC Publications, Submission Date, Journal, Publication Date, Preprint Submission Link, PMCI Number
 * The Manuscript Tracking Sheet is available through the HPRC WIKI.

**Internal Sharing of Planned Manuscripts**
* Investigators shall notify the Steering Committee of any publication using HPRC data or analysis that may impinge on planned integrative data analysis by the consortium, so that the potential impact of the pending publication can be discussed.
* Investigators are encouraged to enter information into the tracking system as early in the process as possible to enable collaboration and minimize overlap
* Information about manuscripts shared internally within the HPRC will be treated as confidential and may not be shared. 
* HPRC members are encouraged to sharing information about publications related to the Human Reference even if not directly funded by the HPRC.

**External Sharing of Submitted Manuscripts**
 * When applicable pre-prints of submitted manuscripts must be shared via the appropriate pre-print server prior to or concurrently with the time of submission to a journal.
 * Pre-prints are considered publications and should be cited like papers. 

**Resolution of Disagreements**

If questions or disagreements arise between investigators they should work to resolve them together. If a mutually agreeable solution cannot be attained then the investigators may petition the Steering Committee to help resolve the situation. The decision of the Steering Committee will be final and all parties agree to abide by their decision.

------------------

### More Materials

General Reference:
* Human Pangenome Reference Consortium Homepage: [link](https://humanpangenome.org/).
* Wiki (Login Required): [link](http://wiki.humanpangenome.org/index.php/Main_Page)

Data Storage (Subject To Change):
* This Workspace: [link](https://app.terra.bio/#workspaces/anvil-datastorage/AnVIL_HPRC)
* S3 Bucket: [link](https://s3-us-west-2.amazonaws.com/human-pangenomics/index.html?prefix=submissions/).

GitHub Repositories:
* Production Workflows: [link](https://github.com/human-pangenomics/hpp_production_workflows).
* Dockstore Site: [link](https://dockstore.org/organizations/HumanPangenome).
* Y1 Data Freeze: [link](https://github.com/human-pangenomics/HPP_Year1_Data_Freeze_v1.0)
* Y1 Assemblies: [link](https://github.com/human-pangenomics/HPP_Year1_Assemblies)
* Pangenome Resources: [link](https://github.com/human-pangenomics/hpp_pangenome_resources)

------------------

Workspace contact email: juklucas@ucsc.edu

Project Name: Human Pangenome Reference 

Data permissions: Open Access
 Platform: AnVIL

ANVIL_HPRC

Description

Dataset Details