Already Have FASTQ, BAM, or VCF Files? How to Prepare Data for Population Genomics Analysis
For Research Use Only. Not for use in diagnostic procedures or clinical decision-making.
You have sequencing data. Your collaborator's lab ran the samples two years ago. Your institution's core facility delivered a hard drive of FASTQ files. Your previous project generated VCFs that were analyzed once and archived. Now you want to repurpose that data for population genomics — population structure, diversity metrics, selection scans, GWAS — and you need to know whether the files you already have are sufficient, or whether you need to go back to an earlier stage of the pipeline.
This guide explains what population genomics analyses each file format supports, what metadata and reference information you need to supply alongside the files, and when re-calling variants from raw data is necessary. If your goal is to outsource the analysis rather than run it in-house, this checklist tells you exactly what to package before contacting a provider.
Figure 1: The three common starting formats for population genomics analysis — each format supports different downstream analyses and carries different metadata requirements.
What Your Files Can Do
Not every file format supports every analysis. The pipeline stage at which your data stopped determines what questions you can answer without going back.
Table 1: File Format Capabilities for Population Genomics Analysis
| File Format | What It Contains | Analyses Supported Directly | Analyses That Require Re-Processing |
| FASTQ | Raw sequencing reads with base quality scores | Everything — alignment, variant calling, all downstream analyses | Nothing — FASTQ is the most flexible starting point |
| BAM (aligned) | Reads mapped to a reference genome | Variant calling, coverage analysis, all downstream population genomics | None, if alignment quality is acceptable and reference genome is appropriate |
| BAM (unaligned) | Reads not yet mapped | Same as FASTQ after alignment | Must be aligned before any analysis |
| gVCF | Per-sample variant and non-variant site information | Joint genotyping, population variant calling, imputation | Single-sample gVCFs must be jointly genotyped before GWAS or selection scans |
| VCF (multi-sample, unfiltered) | Variant calls across all samples | Filtering, population structure, diversity, selection scans, GWAS | May need re-filtering if original QC thresholds are inappropriate for your question |
| VCF (filtered, analysis-ready) | High-confidence variant calls with QC annotations | Most population genomics analyses | Limited flexibility — if filtering is too aggressive, rare or low-quality variants of interest may be missing |
The most common situation researchers face is having a multi-sample VCF from a previous study and wanting to run new analyses — perhaps GWAS or selective sweep scans — that were not part of the original analysis plan. Whether this is possible depends on what the VCF contains, how it was filtered, and whether the original samples are appropriate for the new question.
Preparing FASTQ for Analysis
FASTQ is the most flexible starting format — if you have it, keep it, back it up, and never delete it. From FASTQ, any analysis is possible. The key preparation steps before sending FASTQ files to a provider are:
- Verify read structure. Are the reads paired-end or single-end? What is the read length? Are there multiple FASTQ files per sample (multiple lanes) that need to be merged, or can they be analyzed separately and combined after alignment? Provide a manifest file listing every FASTQ file, its paired mate if applicable, and its sample identifier.
- Check base quality. Run FastQC or a similar tool on a subset of files and review the per-base quality scores, GC content distribution, adapter content, and overrepresented sequences. If adapter contamination is present, specify whether you want the provider to trim adapters or whether trimming has already been performed.
- Confirm the reference genome. If you have a reference genome you want the provider to use, provide the FASTA file and its associated annotation (GFF/GTF). If you do not have a preferred reference, specify your species and let the provider propose the best available assembly. The reference genome choice affects alignability, variant calling sensitivity, and downstream interpretation — using a different reference than the one from your previous analysis will produce different results.
- Document what happened to the samples. Which lab generated the data? On which instrument? Using which library preparation kit? When? The answers affect batch effect diagnosis, coverage expectation, and whether the data can be merged with samples sequenced elsewhere.
Preparing BAM Files for Reuse
BAM files — aligned reads — are one step downstream of FASTQ. They are smaller, more structured, and closer to analysis-ready, but they introduce a dependency that FASTQ files do not: the reference genome used for alignment.
Before submitting BAM files for population genomics analysis, verify:
- Which reference genome was used for alignment? Provide the exact assembly accession (e.g., GRCh38.p14, TAIR10, IRGSP-1.0) or the FASTA file itself. If the provider must use a different reference — because a newer assembly is available, or because your reference is not compatible with the variant calling pipeline — the BAM files must be converted back to FASTQ and realigned. This is technically straightforward but adds cost and time, and the mapping quality of the new alignment depends on the compatibility between the original read lengths and the new reference.
- Were duplicates marked? Duplicate marking (Picard MarkDuplicates) is standard in GATK best practices. If duplicates were not marked, the provider must do it before variant calling. If duplicates were removed rather than marked, coverage depth calculations will be affected — marked duplicates are ignored for variant calling but counted for coverage; removed duplicates affect both.
- Were base quality scores recalibrated (BQSR)? BQSR corrects systematic sequencing errors and improves variant calling accuracy. If BQSR was not performed — common in older datasets or data from non-Illumina platforms — the provider should run it before variant calling.
- Are the BAM files coordinate-sorted and indexed? Unsorted or queryname-sorted BAM files cannot be used directly for variant calling. The provider will need to sort and index them. This is fast but should be specified in the scope of work.
Preparing VCF for Downstream Work
A multi-sample VCF is the most common starting point for population genomics analysis — and the format most likely to contain hidden limitations. A VCF that was perfectly adequate for the original study's aims may be inadequate for yours.
Table 2: VCF Suitability Checklist for Population Genomics
| Check | Question | Red Flag |
| Variant types included | Are SNPs and indels both present, or SNPs only? | SNPs-only VCFs miss functionally important indels |
| Multi-allelic handling | Are multi-allelic sites split or left as complex alleles? | Unsplit multi-allelic sites break PLINK and many GWAS tools |
| Filtering annotation | Are QUAL, QD, MQ, FS, SOR, and ReadPosRankSum annotations present? | Missing filtering annotations mean you cannot reproduce or adjust QC thresholds |
| Missingness pattern | What is the per-sample and per-variant missingness rate? | >10% missingness at either level requires investigation — batch effect or systematic dropout |
| Sample inclusion | Does the VCF include all samples, or were some removed during QC? | If samples were removed, you need the removal criteria and the original sample list |
| Population labels | Are population or group labels embedded in the VCF header or sample names? | Without population labels, ADMIXTURE and FST cannot be run correctly |
| Reference genome | Which reference assembly were variants called against? | Same as BAM — must match the provider's pipeline reference |
For population structure analysis including PCA, ADMIXTURE, and phylogenetic trees, a VCF is usually sufficient as-is after LD pruning. For selection scans, additional filtering by MAF and missingness is standard. For GWAS, imputation to a reference panel may be required if the VCF was called from low-coverage data or a sparse SNP array.
Figure 2: VCF suitability inspection — verifying that a VCF contains the necessary metadata fields and annotations before downstream population genomics analysis.
Metadata Your Files Need
Data files alone are not enough. Metadata — the structured information that links each sample to its biological context — is what transforms a collection of genotypes into a population genomics study.
At minimum, provide a metadata table (CSV or Excel) with one row per sample and the following columns:
- Sample ID — must match the sample names in the FASTQ, BAM, or VCF files exactly, including case and any underscores or hyphens.
- Population or group label — the primary grouping variable for PCA, ADMIXTURE, FST, and GWAS. This can be geographic origin, breed, ecotype, treatment group, or case/control status.
- Phenotype data (if applicable) — quantitative traits as numeric values, binary traits as 0/1 or 1/2 coding, with a separate column explaining the coding scheme. Include a column for any covariates that should be included in association models (sex, age, principal components).
- Sequencing batch or run identifier — the batch in which each sample was sequenced. Even if you think there were no batch effects, recording batch information allows the provider to test for them.
- Relatedness or pedigree information (if known) — family structure, known siblings, or parent-offspring relationships. This is essential for GWAS and for interpreting PCA and ADMIXTURE results.
- Sample QC status — if any samples failed QC in a previous analysis, note which ones and why.
If you are uncertain whether your VCF or metadata is sufficient for the analyses you want, our population genomics data rescue guide covers common data-quality problems and how to address them.
When Re-Calling Is Necessary
Re-calling variants from BAM or FASTQ files is additional work — but in several situations, it is unavoidable if you want reliable results.
Re-calling is strongly recommended when:
- A newer, higher-quality reference genome has been released since your original variant calling was performed. Upgrading from a scaffold-level to a chromosome-level assembly, or from an older assembly with known misassemblies, can recover thousands of variants that were previously unmappable or mis-called.
- Your original VCF was called with tools or parameters that are now considered suboptimal — for example, a VCF called with an older version of GATK (pre-4.0) without BQSR, or called per-sample without joint genotyping.
- You are adding new samples to an existing dataset. Calling variants jointly across old and new samples produces more consistent genotypes and reduces reference bias compared to calling new samples separately and merging VCFs.
- You are switching from single-nucleotide variant (SNV)-only calling to joint SNV and indel calling, and your original VCF contains only SNVs.
- Your original VCF was generated from low-coverage data without genotype likelihood-based methods (ANGSD, bcftools mpileup with GL model), and you now need more sensitive rare variant detection.
Re-calling may not be necessary when:
- Your VCF was called with a recent, well-documented GATK best-practices pipeline, all filtering annotations are present, and the reference genome is still the current standard for your species.
- Your planned analyses are limited to common-variant GWAS or coarse population structure (PCA at K=2–5), where moderate false negative rates at low MAF have minimal impact.
When in doubt, provide your VCF to the provider and ask them to assess whether re-calling is warranted. A credible provider will evaluate the VCF metadata — variant caller version, filtering annotations, reference genome, joint calling status — and make a recommendation rather than defaulting to re-calling everything.
Sending Files to a Provider
Once your files and metadata are prepared, package them for transfer.
- Organize files by format. One directory for FASTQ files, one for BAM files, one for VCF files, one for metadata spreadsheets, and one for reference genome files. Include a README that lists every file, its format, and what it contains.
- Use checksums. Generate MD5 or SHA256 checksums for every file and include them in the README. This allows the provider to verify data integrity after transfer.
- Choose a transfer method. For datasets under 100 GB, cloud storage (AWS S3, Google Drive, Dropbox) is usually sufficient. For larger datasets, discuss with the provider whether they accept physical hard drives, provide an SFTP server, or support direct cloud-to-cloud transfer.
- Specify data use and retention. State whether the provider may retain a copy of your data after analysis for pipeline validation, or whether all data must be deleted after project completion. This is particularly important if your IRB protocol or data-use agreement imposes specific restrictions.
- Plan for reanalysis. If you anticipate reanalyzing the data in the future — with an improved reference genome, new analysis methods, or additional samples — arrange for long-term storage of the raw FASTQ files before the project closes. A complete population genomics report includes not just results but the data provenance that makes reanalysis possible.
Figure 3: File packaging and transfer guide — organize by format, generate checksums, and choose the right transfer method for your dataset size.
Frequently Asked Questions
Yes, for many analyses. A well-formed multi-sample VCF with variant QC annotations supports PCA, ADMIXTURE, FST, phylogenetic tree construction, genetic diversity indices, and GWAS after appropriate LD pruning and MAF filtering. What you cannot do with a VCF alone is re-call variants with improved methods, call structural variants that were not in the original pipeline, recover variants filtered out by the original QC thresholds, or re-align to a different reference genome. If your VCF was generated with a rigorous pipeline and your planned analyses are standard population genomics workflows, the VCF is likely sufficient.
Platform differences matter less than you might expect for downstream analysis — a BAM file aligned to GRCh38 from MGI DNBSEQ data is processed the same way as a BAM from Illumina NovaSeq data. The critical concern is batch effects: if your existing data and any new data the provider generates come from different platforms, library preparations, or sequencing centers, the provider must test for and report platform-associated batch effects before combining the datasets. This is standard practice and should be part of the analysis scope.
A reference genome is good enough if it has a contig N50 above 1 Mb for mammalian genomes (above 100 kb for plant or invertebrate genomes), a BUSCO completeness score above 90% for the appropriate lineage, and has not been superseded by a major new release from a recognized genome consortium. If your reference is more than five years old, check whether a newer assembly exists. The difference between a scaffold-level and chromosome-level assembly can be the difference between detecting a selective sweep and missing it entirely.
The provider needs to know: the reference genome assembly used (with accession), the variant caller and version (if providing a VCF), whether duplicates were marked or removed (if providing BAM files), whether base quality recalibration was performed, the read length and paired-end status (for FASTQ), and the original sequencing platform and library preparation kit. Without these, the provider is working blind — they can process the files, but they cannot interpret QC metrics in context or troubleshoot unexpected results.
Yes, but with important caveats. Data from different sequencing platforms, library preparations, coverage depths, or variant calling pipelines must be tested for batch effects before combining. The standard approach is to process all samples through the same bioinformatics pipeline from the earliest common starting point — ideally from FASTQ. If only VCFs are available from different sources, cross-reference them against a common set of high-confidence variants to assess concordance, and include batch as a covariate in downstream statistical models.
References:
- Bolger AM, Lohse M, Usadel B. Trimmomatic: a flexible trimmer for Illumina sequence data. Bioinformatics. 2014;30(15):2114-2120. doi:10.1093/bioinformatics/btu170
- McKenna A, Hanna M, Banks E, et al. The Genome Analysis Toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data. Genome Research. 2010;20(9):1297-1303. doi:10.1101/gr.107524.110
- Danecek P, Auton A, Abecasis G, et al. The variant call format and VCFtools. Bioinformatics. 2011;27(15):2156-2158. doi:10.1093/bioinformatics/btr330
- Purcell S, Neale B, Todd-Brown K, et al. PLINK: a tool set for whole-genome association and population-based linkage analyses. American Journal of Human Genetics. 2007;81(3):559-575. doi:10.1086/519795
- Korneliussen TS, Albrechtsen A, Nielsen R. ANGSD: analysis of next generation sequencing data. BMC Bioinformatics. 2014;15:356. doi:10.1186/s12859-014-0356-4
For Research Use Only. Not for use in diagnostic procedures or clinical decision-making.
Related
Services
Download
