One of our mining clients brought in a consultant to survey erosion across their rehab. The report came back saying erosion wasn’t too bad. The environmental team didn’t believe it, and they asked what I thought.
Their site is about 1,000 hectares, or around 10 km2. If I chose some parts of the site to visit, I too could walk around and report they don’t have an erosion problem. But if I chose different areas, I could tell them they have a very big problem. Both answers would be correct based on the areas visited, and both would be backed by careful fieldwork, but neither sampled the area at a scale or coverage that would detect the erosion. The reality is that you can stand quite close to a massive gully and never see it.
The client called the survey “ground-truthing”. I’ll leave that phrase alone for now… but it says a lot about what we treat as ‘truth’. I told this story to Professor Stuart Phinn from the University of Queensland when we caught up recently. Stuart has been working in remote sensing for close to four decades, so you could say he’s pretty clued up. Within a few minutes we were both on the same soapbox, which tells you how long this problem has been around.
Precise, but only here
Stuart put the trade-off better than I usually do:
“You can go out to this spot and do a 10-by-10-metre area survey, get it as precise (at the level of detail) and accurate (close to a reference value) as possible. And you get an answer, but you only sampled 0.001% of your whole area. Is it representative – is it sufficient for the required assessment of erosion for the client? If you take drone, crewed aircraft, or satellite data, you can cover the whole 1000 ha area. Let’s say you’re going to get 85% erosion detection accuracy with those data. There’s a trade-off there. The whole-area map is right in some areas, but not in others. So which bit do you want to accept?”
To be fair, field survey would rarely rely on a single plot or quadrat. But twenty plots for example, still only add up to 0.02% – and that would be considered a generous sampling (not to mention labour intensive). That’s not a criticism of plots. A plot tells you something precise and accurate about the plot itself. A map from aerial or satellite data tells you something approximate about everywhere in the site. They answer fundamentally different questions.
What gets me is that we only seem to apply the doubt in one direction. For example, Stuart sees reviewers ask how anyone knows terrestrial laser scanning data for analysing vegetation structure is valid, when as he put it, “this is the most detailed level of measurement we can do on the ground.” I had a coral bleaching paper criticized because reviewers wanted snorkel transects or quadrats evaluating the site. I had millimetre-resolution imagery where you could see exactly which coral was bleached and which was dead. But the shadow of doubt remains over remote sensing derived information.
Wait, aren’t our eyes remote sensors? hmmm.
Every map is “likely to be”
Maps have a problem that field reports don’t. Once something is published as a map, it carries a sense of authority, and people believe that’s what’s on the ground. I’ve looked at global habitat maps that show seagrass in places where I know seagrass doesn’t grow. But the map says it’s there, so that’s what people take away. Yet if the satellite map is wrong in one or more areas that someone knows in detail on the ground – then that is often assumed to be the case for the rest of the map! Stuart’s response was simple: “You and I know that a remote sensing derived map is saying that’s what it’s likely to be, not necessarily what it is.” Often with satellite image mapping it takes several iterations of the mapping process, working on areas where users identify problems, and adding new data points to the models behind them, to get them right.
He recently spent 20 minutes of a first-year class walking students through one of those global maps. “This was designed to do this. Here’s how it was set up. Here’s where it’s right, here’s why it’s really wrong here, and here are the numbers on accuracy.” Understanding the caveats and limitations is a skill we want every map user to have, yet almost nobody gets taught it.
The opposite reflex is just as common. Someone spots an error in the part of the site they know well, and the whole product goes in the bin. Stuart sees it constantly:
“There’s still this balance towards, ‘Well, it’s wrong here in the area I know, so the whole thing, just throw it out.'”
So the same map gets believed too much by one person and dismissed entirely by the next. Meanwhile the plot survey covering 0.001% of the site is treated as ‘truth’ for all of it. As Stuart put it, “that communication is still absolutely the biggest challenge.”
A starting point isn’t a verdict
Stuart’s best example of the throw-it-out reflex were national and global projects to assess type of environmental change year to year, across whole continents or the planet, from Sentinel 2 imagery. Average accuracy was around 50 to 55%. The response from some of the groups funding this was to question if the results should be published at all.
“It was just a start and it would be made better over time. In some areas it works really well. In some areas it doesn’t, because we don’t have training data there. But this is a starting point. Every state-wide, national and global satellite mapping project goes through the same stages”
Stuart’s view is that the project ultimately failed, not because the first version was imperfect, but because nobody got to see how far it could go. He’s also realistic about why that happens. Spending $10,000’s on a mapping project feels very different to committing a team for two years and $1–2 million, and then committing to keep it going.
I think about this a lot with rehab monitoring. A first-year accuracy figure tells you where the method stands today. It doesn’t tell you where it’ll be after three seasons of better reference data, and it definitely doesn’t tell you the plots were more accurate.
So which bit of wrong do you want?
Every method gets something wrong. Plots cannot provide information where they didn’t sample. Maps are wrong in patches, and a good one tells you roughly where and by how much. The honest question for a rehab team isn’t “is this accurate?” but “which errors can I live with, given what I need to report?”
For progressive rehabilitation milestones, that matters. Erosion, slope, and cover are whole-site criteria. If the evidence only covers a fraction of a percent of the site, you’re reporting on the fraction and hoping the rest behaves. The strongest programs I see don’t pick one method. They use the map to find where the problems are, and they send people on the ground to the places the map says are worth a closer look. They also critically assess their results and methods and refine and improve them over time.
Next time someone hands you a report or a map, it’s worth asking three things:
- How much of the site does this actually cover?
- Where is it most likely to be wrong, and has anyone said so?
- Would I still trust it if it disagreed with what I expected?
That last one is the uncomfortable one. Our client’s environmental team didn’t trust their survey because it didn’t match what they were seeing across the site. They were right to ask the question… and I suspect most teams don’t.
