Field note · decision quality

“Give me a directional answer. It doesn’t have to be perfect.”

A rough answer may be enough. Agree on how much error the decision can tolerate before choosing how much analysis to do.

1.0  The request

The deadline matters too.

Someone needs to make a decision this week and wants to know whether the available data can help. A request for a directional answer is often a reasonable place to start.

An answer that arrives after the budget is allocated cannot inform that decision. If the call is cheap to reverse, a quick estimate may be more useful than a detailed analysis delivered too late.

The difficulty is agreeing on what “rough” allows. Without that conversation, the analyst and the requester may have different expectations of the same estimate.

2.0  Term one

Agree on what “directional” means.

I use directional to mean which way, not how much: up or down, better or worse. Check that the requester means the same thing.

Even that narrower claim needs evidence. An estimate above zero may still be too uncertain to tell you the direction.

Two estimates with uncertainty intervals. The first interval crosses zero, so its direction is unknown. The second sits entirely above zero, so its direction is up. no effect sign unknown sign known: up
The first interval includes both increases and decreases. The second sits entirely above zero, providing stronger evidence of an increase at the chosen uncertainty level.

For a time series, check whether an apparent trend could reflect noise or seasonality. A fitted line alone is not enough to establish a meaningful change.

3.0  Term two

How much error is acceptable?

“It doesn’t have to be perfect” can be meant as reassurance. It still leaves you to decide whether being off by 2% or 20% is acceptable.

An engineering drawing makes this explicit with a tolerance: 12.0 mm, ±0.2. That tells the machinist what variation is acceptable before anyone cuts metal. An estimate needs a similar conversation about acceptable error.

Two dimension lines. The first has a narrow tolerance band around its nominal value. The second is labeled plus or minus question mark and its band covers the whole line. 12.0 ±0.2 12.0 ±?
The first drawing states an acceptable range. The second leaves it unspecified, so two people could read the same request differently.

4.0  The follow-up

Ask about the decision.

“What decision will this estimate help you make?”

Choosing whether to try a small campaign is different from setting next year’s headcount. Once the decision is clear, ask:

“How wrong can this be before you’d call it the other way?”

Write down the answer so you can both refer to it. If the requester cannot give a tolerance yet, work through a few possible estimates and ask what they would do with each.

5.0  The ladder

What can the evidence support?

These are useful distinctions when explaining an answer. They are not a universal ranking: the evidence you need depends on the decision and the cost of being wrong.

  1. R0 An unresolved direction

    The interval includes both increases and decreases. Direction remains uncertain. The range may still help rule out effects large enough to matter.

  2. R1 A correlation

    Two things move together. May help with prediction. Higher churn among accounts that never opened the mobile app could help prioritize outreach. It does not establish that pushing them onto the app would keep them.

  3. R2 A trend that beats noise

    The analysis accounts for noise and seasonality. Supports a claim about direction. That alone does not explain the cause or establish whether the change is large enough to matter.

  4. R3 A signed effect with an interval

    The whole interval sits on one side of the agreed threshold. Supports a go/no-go if the analysis answers the decision question and its assumptions hold.

  5. R4 A magnitude with an interval

    How big, how sure, against a stated threshold. Carries sizing: how much to spend, how many to hire, what to promise the board.

Be explicit when the analysis supports a narrower claim than the decision requires. A correlation useful for outreach may tell you very little about the return on a proposed intervention.

6.0  The spec

Four questions before starting.

  1. Which way, or how much? Do you need to know whether the effect is positive, or whether it is large enough to justify the cost? The required precision depends on that distinction.
  2. What number would change your mind? This gives the estimate a threshold to compare against. If no result would change the decision, clarify what the analysis is intended to inform.
  3. How wrong can it be before you’d call it differently? Agree on the tolerance and record it in the request.
  4. If we’re wrong, when do we find out and what does the unwind cost? A cheap, reversible call can justify a rougher estimate. Discuss how you would notice an error and what reversing the decision would involve.

If a quick estimate is enough for the decision, deliver it with its limitations. There is no need to make the analysis more elaborate than the question requires.

7.0  The delivery

Hand back the number with its tolerance attached.

Include the estimate, its range, and the threshold you agreed on. Explain what the remaining uncertainty means for the decision.

The whole answer

“The estimate is +6%, with a range of +1% to +11%. That supports an increase, but the range crosses your +2% threshold. We cannot yet say the gain clears your bar. A small, reversible pilot may be reasonable; I would want more evidence before committing the full budget.”

The direction is clearer than the decision: the entire interval is positive, but part of it falls below the required gain.

Send the estimate and range together in writing. That gives people something to refer to when they share the result after the meeting.

The job

Trade precision for speed, and put the trade in writing.

Before spending another day refining an estimate, check whether the extra precision would change the decision. Sometimes the rough answer is enough; sometimes the remaining uncertainty is exactly what matters.