“Our detection rate is better than the competition’s.”
It is a familiar line when evaluating email security solutions. Every vendor has its figures, comparisons and examples of threats that others missed.
Fine. But before showing me another dashboard, I want to check something: are orders getting through? Are customer replies leaving the CRM? Are transactional emails working? What happens to the applications and printers that need to send messages?
A company’s email service does far more than carry conversations between people.
When one of those flows stops working, the first warning may come from the business asking why a customer has not received their order confirmation. Meanwhile, the security dashboard may still be showing excellent results.
First, understand which email flows need to work
For me, an email security proof of concept starts with understanding the service.
Which processes depend on it, who sends messages, from where, on behalf of which domain and for what purpose. Also who receives them, and what happens if they never arrive, arrive late or end up in a quarantine nobody is watching.
Not every message has the same impact.
A marketing message may be able to wait. An order confirmation, a notification needed to complete a transaction or a pending customer reply may have different requirements.
Without identifying those flows first, we will be evaluating part of the email service and assuming it represents the whole organisation.
The least visible dependencies tend to reveal themselves on the day we change something.
Applying Zero Trust also means understanding legitimate traffic
A process needing to send email does not mean we should accept any message claiming to come from it.
I need to understand its origin, identity, authentication and the conditions under which it should work. We can then define how to protect it and what happens when those conditions are not met.
If a supplier sends email on the company’s behalf, we need to review how. If an application uses a relay, we need to understand that path. If a device needs to send notifications, we must define its permissions and limits.
“It is a business application” explains why we need to resolve the issue. It does not replace the security review.
That is what I want the PoC to demonstrate: that the solution can apply the agreed policies and maintain the necessary flows, with enough traceability to understand what is happening.
An allowlist cannot be the operating model
The quick answer often appears here: “We will add it to the allowlist.”
Then the next supplier. Then the next application. Then the platform with an urgent message to send.
When you handle hundreds of thousands of emails a day, that approach does not hold up. Each new flow needs intervention, exceptions accumulate, and maintaining them becomes another service someone has to operate.
We also need to know exactly what “allowed” means. Are we identifying a specific flow, or bypassing controls for anything using a domain or address?
Those are different decisions.
Before creating an exception, I want to understand the reason for the block. A sender may be misconfigured, an integration may be wrong, or a policy may not cover a legitimate need.
The solution needs to help us distinguish those cases. If every issue during evaluation is resolved by allowing senders, the PoC is revealing a limitation we need to consider.
One PoC. Three tests that must support the decision.
Business flows, protection and operations are assessed through defined cases, not just a detection counter.
- Orders and CRM
- Transactional email
- Applications and devices
- Threat samples
Document what the solution sees, which actions it can take, and how long they take.
Necessary email works
- Verify the path and receipt.
- Assess the impact of delays and blocks.
Evidence: critical flows verified.
The action is demonstrated
- Validate samples and their results.
- Distinguish observation, blocking and removal.
Evidence: treatment and timing checked.
The team can sustain it
- Investigate and resolve realistic test cases.
- Measure adjustments, escalations and exceptions.
Evidence: effort and ownership understood.
- Proceed
- Proceed with conditions
- Reject
A higher detection rate alone does not compensate for a broken critical flow or unsustainable operations.
Email that needs to get through needs proper test cases
Attack examples have their place. But I also want a representative sample of the email that keeps the operation running.
Transactional emails, orders, replies from the CRM, application notifications and messages from authorised devices. The internal, external and cross-platform flows the organisation actually uses.
For each one, we need to know the expected result and how we will verify it.
An application reporting “sent” may not be enough to demonstrate that the recipient received the message. A single successful delivery is not enough either if the process needs to work at volume or at specific times.
During the PoC, we need to follow the path and understand where a message is accepted, rejected, delayed or held.
If something fails, we need an explanation that helps resolve the cause. “The engine considered it suspicious” leaves quite a lot of work unfinished.
Threat detection is still a requirement
Of course we need to evaluate phishing, malware, impersonation and the other scenarios within scope.
What I question is a decision based almost entirely on a detection comparison.
First, review the samples. Confirm what they contained, what each tool could see and what would have happened to those messages in production.
If a solution observes email after another tool has filtered it, that position affects the comparison. If it is in audit mode, we need to distinguish what it detects from what it actually prevented.
I also want to know how long it takes to apply an action and what the user can do during that interval.
The detection figure needs an explanation. It must sit alongside the result for legitimate email and the effort required to operate the solution.
How much work does it need to work well?
A PoC where the vendor constantly reviews the configuration can be useful for learning. But we need to know how much of that work will remain necessary afterwards.
How many adjustments did we make? Why? How many tickets would the team have received? Which cases could they resolve themselves, and which needed escalation?
If maintaining legitimate email requires a manual review of every new flow, that cost belongs in the evaluation.
The same applies to investigation. The team must be able to find a message, understand its treatment and determine what needs fixing. They also need to know who can release email, change a policy or authorise an exception.
To define the operating model and its RACI, we need to understand the tool and test how people actually work with it.
I would not wait until production to discover that a routine task needs permissions nobody anticipated, or intervention that is not included in the supplier’s service.
What should the PoC demonstrate?
I would agree the criteria before seeing the results.
Each requirement should connect to tests and evidence that let us decide whether it is covered, outstanding or outside scope.
Not everything can be reduced to a percentage.
If a solution blocks very few legitimate messages but those include order confirmations, we need to assess that impact. If it detects more threats, I want to know which ones and how that improvement was validated.
If the result depends on more manual work than the team can sustain, we cannot ignore that either.
| What we evaluate | What we need to demonstrate |
|---|---|
| Critical business flows | They work under the agreed security conditions, and we can verify their path. |
| Protection | The defined threat scenarios receive the expected treatment. |
| False positives | Which legitimate email was affected, why, and how the cause is resolved. |
| Operational scale | The effort needed for adjustments, investigations and onboarding new flows. |
| Exceptions | What they authorise, which controls remain, who owns them, and how they are reviewed. |
| Production rollout | Dependencies, validation steps and the rollback procedure are understood. |
The decision must support implementation and operations
By the end of the PoC, we should understand how the service will work and what still needs resolving before we change it.
Which policies we will apply, which integrations we need, who will do each task and which signals will help us identify problems during rollout.
There will be adjustments afterwards. There always are. But critical business and security requirements must either be verified or identified as conditions that still need resolving.
For me, that is the test that matters.
We can spend hours arguing about which vendor detects more. Before approving one, I want to explain how we will protect email, how the company will keep working and who will take responsibility when something does not work as expected.
Further reading
Official sources for checking and exploring these concepts. This article presents a practical approach; it does not reproduce the references’ full requirements.
- Microsoft Learn · Try and evaluate Defender for Office 365 (audit and blocking modes)
- Microsoft Learn · Troubleshoot email authentication
Examples are illustrative. Implementation decisions should be adapted to each organization’s requirements and dependencies.