You're staring at a phylogenetic tree. Because of that, maybe you built it yourself. Maybe you're reviewing someone else's. Either way, there's a branch sitting off to the side — separate from the main group you care about — labeled "outgroup Nothing fancy..
And you're wondering: does it actually matter which one I pick? In real terms, can I just grab whatever's convenient? What happens if I get it wrong?
Short answer: yes, it matters. A lot. And no, you can't just pick whatever's handy Nothing fancy..
What Is an Outgroup
An outgroup is a taxon — species, population, sequence — that you know sits outside the group you're actually studying. Which means that group? That's your ingroup. The outgroup's job is to root the tree. To tell you which direction evolution flowed. To polarize character states: ancestral versus derived Not complicated — just consistent. Nothing fancy..
Some disagree here. Fair enough.
Think of it like a compass. You're hiking through trait space. The outgroup tells you which way is "back toward the ancestor And that's really what it comes down to. Still holds up..
But here's the thing most textbooks gloss over: an outgroup isn't just "something outside.In real terms, " It's a hypothesis. On the flip side, you're asserting — based on prior evidence — that this taxon diverged before your ingroup's most recent common ancestor. If that assertion is wrong, your root is wrong. And if your root is wrong, every downstream inference about character evolution, ancestral states, and divergence timing gets shaky That's the part that actually makes a difference. Nothing fancy..
Quick note before moving on Most people skip this — try not to..
The Rooting Problem
Unrooted trees don't have a time axis. They show relationships — who's closer to whom — but not who came first. Rooting adds that arrow of time. The outgroup provides the reference point: "this lineage split off earlier, so the root goes here Worth keeping that in mind..
Maximum likelihood, Bayesian inference, parsimony — they all need a root to make sense of ancestral state reconstruction. Day to day, even distance methods like neighbor-joining produce unrooted trees by default. You must root them afterward, and that's where the outgroup enters.
Why It Matters / Why People Care
Get the outgroup wrong, and you don't just get a slightly off tree. You can flip the entire topology Most people skip this — try not to..
Imagine you're studying placental mammals. Your tree roots cleanly. Now imagine you accidentally used a monotreme (platypus) but your alignment has saturation issues at deep timescales. Still, long-branch attraction kicks in. But rodents look basal. You use a marsupial as outgroup — solid choice, diverged ~160 mya. The monotreme gets pulled toward the fastest-evolving placental — say, rodents — and suddenly your root sits inside Placentalia. Everything else falls out from there Less friction, more output..
That's not a hypothetical. It happened in early mammalian phylogenetics. Repeatedly.
The outgroup choice affects:
- Tree topology — especially deep nodes
- Ancestral state reconstruction — what the ancestor looked like
- Divergence time estimation — calibration points propagate from the root
- Tests of monophyly — if your "outgroup" is actually nested inside, your ingroup isn't monophyletic
And here's what most people miss: the outgroup doesn't just root the tree. Also, if your outgroup falls within what you thought was your ingroup, congratulations — your ingroup was paraphyletic all along. That's not a rooting error. It defines the ingroup. That's a discovery.
How It Works (or How to Choose One)
There's no universal "best" outgroup. But there are principles. Follow them and you'll avoid the worst mistakes.
1. Phylogenetic Distance: The Goldilocks Zone
Too close, and you lack power to polarize deep characters. Too distant, and you hit saturation, alignment ambiguity, and long-branch attraction Easy to understand, harder to ignore..
The sweet spot? Now, the closest well-established outgroup. But the sister group to your ingroup, if you know it. If you're studying birds, use crocodilians — not lizards, not mammals, not amphibians. And crocodilians are the extant sister clade. They share the most recent common ancestor with birds outside Aves. That minimizes branch length while maximizing phylogenetic signal.
But — and this is crucial — "closest" means phylogenetically closest, not genetically closest in terms of raw sequence similarity. Even so, don't trust BLAST for outgroup selection. Convergent evolution, GC bias, and rate heterogeneity can make a distant taxon look closer in a BLAST search. Trust established phylogenies.
2. Multiple Outgroups Beat One
Single outgroups are fragile. If that one taxon has a weird rate acceleration, a hidden paralogy, or contamination — your root is compromised.
Two or more outgroups let you:
- Check consistency: do they root the tree in the same place?
- Detect rogue taxa: if one outgroup pulls the root somewhere weird, that's a red flag
- Break up long branches: two moderately distant outgroups often outperform one very distant one
Ideally, pick outgroups that bracket your ingroup — one on each side of the root, if the phylogeny allows. They diverged at different depths. For a study of core eudicots, you might use magnoliids and monocots. If both place the root identically, you've got confidence.
3. Data Type Matters
Morphology? Molecules? Both?
Molecular outgroups need alignable sequences. Think about it: this sounds obvious. Even so, the tree roots wrong. Also, it's not. This leads to if your ingroup is a protein-coding gene family, your outgroup needs orthologs — not paralogs. You align what you think are orthologs, but you've actually mixed paralogous lineages. And gene duplication predating the ingroup/outgroup split is a classic trap. The duplication looks like a speciation event Nothing fancy..
Morphological outgroups need scorable characters. Lots of it. Here's the thing — fossil taxa can be amazing outgroups — they break long branches, preserve ancestral morphologies, and provide direct temporal calibration. But they come with missing data. That's not fatal, but it requires models that handle missingness properly (Mk models with ascertainment bias correction, for instance).
4. Test Before You Commit
Run a quick analysis with candidate outgroups. Compare:
- Root position stability
- Branch lengths on the outgroup branches
- Support values at the ingroup root node
- Likelihood scores (if using model-based methods)
If adding a second outgroup shifts the root, investigate. Don't just pick the one that gives you the topology you expected. That's confirmation bias wearing a lab coat Small thing, real impact..
Common Mistakes / What Most People Get Wrong
Mistake 1: "Any Outgroup Will Do"
No. A bad outgroup is worse than no outgroup — at least an unrooted tree admits uncertainty. A confidently rooted wrong tree misleads everyone who cites it.
Mistake 2: Using a Taxon With Uncertain Phylogenetic Position
"Let's use Taxon X as outgroup — it's probably outside the group.Even so, " Stop. "Probably" isn't good enough. That's why if Taxon X turns out to be inside, your ingroup isn't monophyletic. Your root is inside the tree. Your ancestral state reconstructions are garbage.
Only use outgroups with strong, independent evidence for their position. Use it as an ingroup. Multiple loci. Even so, if the literature debates whether Taxon X is sister to your ingroup or nested within it, don't use it as an outgroup. Consensus across methods. Published phylogenomics. Let the analysis tell you where it falls.
Mistake 3: Ignoring Saturation
You're studying a deep divergence — say, animal phyla. You pick a choanoflagellate outgroup. The branch leading to it
The branch leading to it is long. Here's the thing — really long. Hundreds of millions of years of independent evolution. Your ingroup sequences have changed; the outgroup sequences have changed more. Think about it: multiple substitutions at the same site erase phylogenetic signal. Saturation turns homology into noise Not complicated — just consistent. Practical, not theoretical..
You align the sequences. The model (GTR+Γ, LG, whatever) tries to correct for multiple hits. But models assume the process is homogeneous and the saturation is estimable. At deep timescales, it often isn’t. The outgroup branch attracts other long branches in your ingroup — or it wanders randomly, dragging the root with it. You get a root position with 100% bootstrap support that is entirely artifactual.
Fix it: Test for saturation explicitly. Plot transitions/transversions vs. genetic distance. Use Xia’s test in DAMBE. If the outgroup sequences are saturated relative to the ingroup, discard them. Use a closer outgroup. Use conserved amino acids instead of nucleotides. Use site-heterogeneous models (CAT, LG4X) that resist long-branch attraction better. Or — radical thought — root the tree using a non-substitution method: gene order, intron positions, rare genomic changes, or a carefully vetted morphological matrix. Molecular sequences aren't the only characters that exist.
Mistake 4: The "Single Gene" Outgroup Trap
You have a phylogenomic dataset — 500 loci, 100 taxa. But for the outgroup, you only have a transcriptome for one species. So you use that one species, represented by 500 genes, to root the tree Easy to understand, harder to ignore..
Congratulations. You have just given one terminal taxon 500x the voting power of your ingroup taxa for the purpose of rooting. Even so, if that one outgroup species has a weird substitution rate, a contaminated assembly, a hidden paralog in 50 loci, or just an idiosyncratic evolutionary history, it dictates your root. The sheer volume of data from a single lineage creates false precision.
Fix it: Outgroup sampling needs taxonomic breadth, not just genomic depth. Multiple outgroup species, spanning the diversity of the sister clade. If you only have genomic data for one outgroup, supplement with Sanger-sequenced loci for two or three more. Or use the "gene jackknifing" approach: root each gene tree separately with the available outgroup data, then summarize root positions across genes (e.g., with ASTRAL or a consensus method). If 450 genes root the tree one way and 50 root it another, that is your result — not the concatenated analysis that swamps the signal Not complicated — just consistent. No workaround needed..
Mistake 5: Forgetting the Ingroup Root Is a Hypothesis, Not a Fact
You ran the analysis. The root is placed. You publish the rooted tree. You treat the basal split in your ingroup as the "first divergence.
But the root is the least reliable node in the tree. It depends entirely on the outgroup — its sampling, its data quality, its model fit, its branch length. Consider this: change the model, change the root. In real terms, change the outgroup, change the root. Remove the fastest-evolving sites, change the root And that's really what it comes down to..
Fix it: Report root stability. Show the rooted tree and the unrooted network. Explicitly state: "The position of the root is sensitive to outgroup choice / model / data partitioning." If your downstream conclusions (ancestral state reconstruction, diversification rate shifts, biogeographic scenarios) collapse when the root moves one node, say so. Don't build a narrative on a node that has 65% bootstrap support and shifts when you swap the outgroup But it adds up..
Conclusion
Outgroup selection is not a checkbox on a pipeline. It is a phylogenetic hypothesis in its own right — one that requires the same rigor you apply to your ingroup. Because of that, the perfect outgroup does not exist. There is only the appropriate outgroup: close enough to align and model, distant enough to be unequivocally external, sampled deeply enough to average over lineage-specific artifacts, and vetted thoroughly enough that its position is beyond reasonable doubt But it adds up..
Treat your outgroups as data, not tools. Analyze their properties. Which means test their influence. Report their instability. A rooted tree is only as trustworthy as the branch that holds the root. If that branch is rotten — saturated, undersampled, misidentified, or model-violating — the whole tree falls over, no matter how beautiful the ingroup topology looks That's the part that actually makes a difference..
Root wisely. Or don't root at all. An unrooted network honestly displayed is infinitely more valuable than a rooted tree confidently wrong.
