You're staring at a phylogenetic tree. Maybe you built it yourself. Maybe you're reviewing someone else's. Either way, there's a branch sitting off to the side — separate from the main group you care about — labeled "outgroup.
And you're wondering: does it actually matter which one I pick? Can I just grab whatever's convenient? What happens if I get it wrong?
Short answer: yes, it matters. A lot. And no, you can't just pick whatever's handy The details matter here..
What Is an Outgroup
An outgroup is a taxon — species, population, sequence — that you know sits outside the group you're actually studying. And the outgroup's job is to root the tree. Consider this: that's your ingroup. To tell you which direction evolution flowed. So that group? To polarize character states: ancestral versus derived.
Think of it like a compass. Think about it: you're hiking through trait space. The outgroup tells you which way is "back toward the ancestor.
But here's the thing most textbooks gloss over: an outgroup isn't just "something outside." It's a hypothesis. Plus, you're asserting — based on prior evidence — that this taxon diverged before your ingroup's most recent common ancestor. Practically speaking, if that assertion is wrong, your root is wrong. And if your root is wrong, every downstream inference about character evolution, ancestral states, and divergence timing gets shaky And it works..
And yeah — that's actually more nuanced than it sounds.
The Rooting Problem
Unrooted trees don't have a time axis. They show relationships — who's closer to whom — but not who came first. So rooting adds that arrow of time. The outgroup provides the reference point: "this lineage split off earlier, so the root goes here.
Maximum likelihood, Bayesian inference, parsimony — they all need a root to make sense of ancestral state reconstruction. Because of that, even distance methods like neighbor-joining produce unrooted trees by default. You must root them afterward, and that's where the outgroup enters Simple, but easy to overlook..
Why It Matters / Why People Care
Get the outgroup wrong, and you don't just get a slightly off tree. You can flip the entire topology Simple, but easy to overlook..
Imagine you're studying placental mammals. Long-branch attraction kicks in. Rodents look basal. Now, the monotreme gets pulled toward the fastest-evolving placental — say, rodents — and suddenly your root sits inside Placentalia. Even so, your tree roots cleanly. Now imagine you accidentally used a monotreme (platypus) but your alignment has saturation issues at deep timescales. You use a marsupial as outgroup — solid choice, diverged ~160 mya. Everything else falls out from there.
Quick note before moving on.
That's not a hypothetical. It happened in early mammalian phylogenetics. Repeatedly.
The outgroup choice affects:
- Tree topology — especially deep nodes
- Ancestral state reconstruction — what the ancestor looked like
- Divergence time estimation — calibration points propagate from the root
- Tests of monophyly — if your "outgroup" is actually nested inside, your ingroup isn't monophyletic
And here's what most people miss: the outgroup doesn't just root the tree. Even so, if your outgroup falls within what you thought was your ingroup, congratulations — your ingroup was paraphyletic all along. It defines the ingroup. That's not a rooting error. That's a discovery.
How It Works (or How to Choose One)
There's no universal "best" outgroup. But there are principles. Follow them and you'll avoid the worst mistakes.
1. Phylogenetic Distance: The Goldilocks Zone
Too close, and you lack power to polarize deep characters. Too distant, and you hit saturation, alignment ambiguity, and long-branch attraction Nothing fancy..
The sweet spot? The closest well-established outgroup. The sister group to your ingroup, if you know it. Think about it: if you're studying birds, use crocodilians — not lizards, not mammals, not amphibians. Practically speaking, crocodilians are the extant sister clade. Plus, they share the most recent common ancestor with birds outside Aves. That minimizes branch length while maximizing phylogenetic signal.
But — and this is crucial — "closest" means phylogenetically closest, not genetically closest in terms of raw sequence similarity. Convergent evolution, GC bias, and rate heterogeneity can make a distant taxon look closer in a BLAST search. Don't trust BLAST for outgroup selection. Trust established phylogenies Worth keeping that in mind..
2. Multiple Outgroups Beat One
Single outgroups are fragile. If that one taxon has a weird rate acceleration, a hidden paralogy, or contamination — your root is compromised.
Two or more outgroups let you:
- Check consistency: do they root the tree in the same place?
- Detect rogue taxa: if one outgroup pulls the root somewhere weird, that's a red flag
- Break up long branches: two moderately distant outgroups often outperform one very distant one
Ideally, pick outgroups that bracket your ingroup — one on each side of the root, if the phylogeny allows. They diverged at different depths. Also, for a study of core eudicots, you might use magnoliids and monocots. If both place the root identically, you've got confidence It's one of those things that adds up..
3. Data Type Matters
Morphology? Molecules? Both?
Molecular outgroups need alignable sequences. Day to day, if your ingroup is a protein-coding gene family, your outgroup needs orthologs — not paralogs. This sounds obvious. Worth adding: it's not. On the flip side, gene duplication predating the ingroup/outgroup split is a classic trap. Because of that, you align what you think are orthologs, but you've actually mixed paralogous lineages. Think about it: the tree roots wrong. The duplication looks like a speciation event.
Morphological outgroups need scorable characters. Also, fossil taxa can be amazing outgroups — they break long branches, preserve ancestral morphologies, and provide direct temporal calibration. But they come with missing data. Lots of it. That's not fatal, but it requires models that handle missingness properly (Mk models with ascertainment bias correction, for instance).
4. Test Before You Commit
Run a quick analysis with candidate outgroups. Compare:
- Root position stability
- Branch lengths on the outgroup branches
- Support values at the ingroup root node
- Likelihood scores (if using model-based methods)
If adding a second outgroup shifts the root, investigate. Which means don't just pick the one that gives you the topology you expected. That's confirmation bias wearing a lab coat Nothing fancy..
Common Mistakes / What Most People Get Wrong
Mistake 1: "Any Outgroup Will Do"
No. A bad outgroup is worse than no outgroup — at least an unrooted tree admits uncertainty. A confidently rooted wrong tree misleads everyone who cites it That's the whole idea..
Mistake 2: Using a Taxon With Uncertain Phylogenetic Position
"Let's use Taxon X as outgroup — it's probably outside the group.If Taxon X turns out to be inside, your ingroup isn't monophyletic. Your root is inside the tree. Now, "Probably" isn't good enough. " Stop. Your ancestral state reconstructions are garbage.
Only use outgroups with strong, independent evidence for their position. Consensus across methods. Multiple loci. On the flip side, if the literature debates whether Taxon X is sister to your ingroup or nested within it, don't use it as an outgroup. Use it as an ingroup. Published phylogenomics. Let the analysis tell you where it falls That alone is useful..
Mistake 3: Ignoring Saturation
You're studying a deep divergence — say, animal phyla. You pick a choanoflagellate outgroup. The branch leading to it
The branch leading to it is long. Which means really long. Practically speaking, hundreds of millions of years of independent evolution. Your ingroup sequences have changed; the outgroup sequences have changed more. And multiple substitutions at the same site erase phylogenetic signal. Saturation turns homology into noise And that's really what it comes down to..
You align the sequences. The model (GTR+Γ, LG, whatever) tries to correct for multiple hits. But models assume the process is homogeneous and the saturation is estimable. At deep timescales, it often isn’t. The outgroup branch attracts other long branches in your ingroup — or it wanders randomly, dragging the root with it. You get a root position with 100% bootstrap support that is entirely artifactual.
Fix it: Test for saturation explicitly. Plot transitions/transversions vs. genetic distance. Use Xia’s test in DAMBE. If the outgroup sequences are saturated relative to the ingroup, discard them. Use a closer outgroup. Use conserved amino acids instead of nucleotides. Use site-heterogeneous models (CAT, LG4X) that resist long-branch attraction better. Or — radical thought — root the tree using a non-substitution method: gene order, intron positions, rare genomic changes, or a carefully vetted morphological matrix. Molecular sequences aren't the only characters that exist.
Mistake 4: The "Single Gene" Outgroup Trap
You have a phylogenomic dataset — 500 loci, 100 taxa. But for the outgroup, you only have a transcriptome for one species. So you use that one species, represented by 500 genes, to root the tree.
Congratulations. You have just given one terminal taxon 500x the voting power of your ingroup taxa for the purpose of rooting. If that one outgroup species has a weird substitution rate, a contaminated assembly, a hidden paralog in 50 loci, or just an idiosyncratic evolutionary history, it dictates your root. The sheer volume of data from a single lineage creates false precision.
Fix it: Outgroup sampling needs taxonomic breadth, not just genomic depth. Multiple outgroup species, spanning the diversity of the sister clade. If you only have genomic data for one outgroup, supplement with Sanger-sequenced loci for two or three more. Or use the "gene jackknifing" approach: root each gene tree separately with the available outgroup data, then summarize root positions across genes (e.g., with ASTRAL or a consensus method). If 450 genes root the tree one way and 50 root it another, that is your result — not the concatenated analysis that swamps the signal But it adds up..
Mistake 5: Forgetting the Ingroup Root Is a Hypothesis, Not a Fact
You ran the analysis. Here's the thing — the root is placed. You publish the rooted tree. You treat the basal split in your ingroup as the "first divergence That's the part that actually makes a difference..
But the root is the least reliable node in the tree. Think about it: it depends entirely on the outgroup — its sampling, its data quality, its model fit, its branch length. Change the outgroup, change the root. Now, change the model, change the root. Remove the fastest-evolving sites, change the root.
Fix it: Report root stability. Show the rooted tree and the unrooted network. Explicitly state: "The position of the root is sensitive to outgroup choice / model / data partitioning." If your downstream conclusions (ancestral state reconstruction, diversification rate shifts, biogeographic scenarios) collapse when the root moves one node, say so. Don't build a narrative on a node that has 65% bootstrap support and shifts when you swap the outgroup.
Conclusion
Outgroup selection is not a checkbox on a pipeline. It is a phylogenetic hypothesis in its own right — one that requires the same rigor you apply to your ingroup. The perfect outgroup does not exist. There is only the appropriate outgroup: close enough to align and model, distant enough to be unequivocally external, sampled deeply enough to average over lineage-specific artifacts, and vetted thoroughly enough that its position is beyond reasonable doubt Easy to understand, harder to ignore..
Counterintuitive, but true.
Treat your outgroups as data, not tools. In practice, a rooted tree is only as trustworthy as the branch that holds the root. Test their influence. Analyze their properties. Report their instability. If that branch is rotten — saturated, undersampled, misidentified, or model-violating — the whole tree falls over, no matter how beautiful the ingroup topology looks.
Honestly, this part trips people up more than it should.
Root wisely. Now, or don't root at all. An unrooted network honestly displayed is infinitely more valuable than a rooted tree confidently wrong.
