A hoax to make money or a valid disinfection solution – how to distinguish them?
A brief summary on various ways of spoofing disinfection test results.
I can´t remember a time seeing this many disinfection solution being advertised at once. Disinfection solution providers are putting up huge signs to let the public know that they offer “safe zones”. Healthcare workers are being approached like never before by salespeople offering all kinds of disinfection solutions that are going to solve all their problems.
It is heart-warming to see everyone out there actively trying to do what they can to help fight this latest outbreak. However, there are also some wolves in disguise: for some, this pandemic seems to be a golden opportunity to make money. I feel sad to see how some companies are trying to fool people with their misleading presentations of the effectiveness of their disinfection solution.
How can you distinguish between good and science? How to separate actual disinfection from nonsense?
Early on during my science studies at the University of Helsinki we were trained to read, analyse, and understand science data. Separating reliable science from nonsense is a skill I use almost daily. Now, 20 years later and with 10 years of being a researcher, I want to share with you a few basic things that you should know when looking at disinfection results.
When doing microbiological experiments, there are two basic ways to prove that a disinfection method is working:
Method 1: Show that microbial levels on average are being significantly lower after introducing the disinfection solution. For this you gather data on the microbial levels before disinfection and compare it to the microbial levels after disinfection. In this case it is crucial to take enough before-samples, and enough after-samples to get reliable results. In addition, if the before-samples already show a very low microbial level, it is difficult to show a decrease that you can call disinfection, since the change in microbial levels is negligible.
Microbial levels on a surface naturally vary to some extent from day to day (see figure 1, graph to the right), hour to hour, and even minute to minute. During summertime there are generally higher microbial levels in our surroundings due to higher temperatures and humidity offering a more suitable living environment. Also, pollen are crucial carriers of microbes during spring and summer months. You don´t want to be shown a few hand-picked results that are part of a natural variation and be told that this is disinfection. You want to see a long-term difference in microbial levels, that are ON AVERAGE being significantly lower than before.
Figure 1: Examples to highlight the differences between unreliable data and a more convincing set of data. The error bars in the graph to the right tells you that the data point consist of an average of at least three samples, which is much more reliable than showing results from just one sample as in the graph to the left. Please note also that the scales on the y-axis are different in the two graphs. It is very easy to make disinfection seem more dramatic by changing the scales on the y-axis.
Method 2: Compare your disinfection solution to a control. Again, taking enough samples and doing repetitions is key. You can have two identical objects
, that have identical levels of microbes on them. Disinfect one, but not the other. Take samples from both in the same way, treat and analyse the samples in an identical manner. You end up with results from the disinfected object that you can compare to the non-disinfected (which is usually called control), to get a sense of how effective the disinfection is. Then you repeat this several times to get a convincing average.
Why is it so important to have control samples to compare to? Microbes are living organisms with a very short but active lifespan. When microbes end up on a dry surface part of them start to naturally die. The environmental conditions are not optimal for living, there is no nutrients to survive on, and some of them will die quite fast. This is not what we call disinfection. This is part of their natural behaviour. But if you only show the results from the disinfected object and not the results from the control, you will be shown the results from the natural die-off + disinfection, which will make you think that the disinfection method is much more effective than it really is.
Figure 2: How leaving out information about the control can change the whole story. In the graph to the left, information about the control has been left out, which can make the disinfection method seem very effective. When comparing this to the information in the graph to the right, there is not much difference between the solution claiming to do disinfection and the control. The control shows you how much the microbes naturally die on the surface without any disinfection.
Errors bars explained
An error bar in a graph represents the uncertainty or variation of the data shown. The error bar is drawn as a line through a point on the graph, parallel to one of the axes. In life sciences, the error bars most often represent the standard deviation (SD) of a data set. Biological measurements are notoriously variable, so just because you have a larger SD it does not automatically indicate your data is not valid. However, error bars give you an indication of reliability, and if they are absent from the graph you should be very suspicious.
Error bars may also show confidence intervals, standard errors, or other quantities. Different types of error bars give quite different information, so figure legends must inform what the error bars represent.
Error bars can communicate the following information about the data:
1. The reliability of the mean value as a representative number for the data set. In other words, how accurately the mean value represents the data (small SD bar = more reliable, larger SD bar = less reliable).
2. How spread the data are around the mean value (small SD bar = low spread, data are clumped around the mean; larger SD bar = larger spread, data are more variable from the mean).
3. The likelihood of there being a significant difference between data sets.
How to misuse ATP-measurements
I don´t want to give ATP meters and methods a bad name, however, I do want to warn you of how they are being misused to fool you. They can be such handy and easy tools to use in certain occasions, but because they are so easy to use, they are being misused and “abused” by self-interested companies that want to trick you into buying their “effective” disinfection solutions.
ATP meters are intended to be used as a quick check to see that the surface you have disinfected is “clean enough” and there is no organic dirt left on the surface. It does not recognize microbes on the surface, instead it measures the amount of ATP on the surface.
Adenosine Triphosphate (ATP) is a molecule that is found inside microbial cells, but also inside human cells, and there is currently no way of knowing if the measured ATP levels on a surface is from microbial or human cells. ATP meters can be used as indicators if there is “organic dirt” on the surface, but there is no guarantee that a positive result is due to microbes on the surface. Detected ATP on a surface could be from microbial origin, but is could be human origin as well. There is no way of knowing.
Also, there is also currently no way of knowing if the ATP found on the surface is from a dead or living cell. Dead microbes are no longer pathogenic, meaning that they will not infect you. High amount of ATP might just indicate that there is organic debris on the surface, but still there might be nothing harmful to you.
Yet another way to misuse this tool is to not pay attention to the surface area from which you swab the sample. How will the area affect the results? Let´s say you have a surface that contains 10 000 microbes per cm2. If you swab a 1 cm2 area you will get a result that corresponds to amount of ATP in 10 000 microbes. If you swab an area of 10 cm2 you will get a result that corresponds to the amount of ATP in 100 000 microbes. See how easy it is to manipulate the results? Crucial for a somewhat reliable result is to make sure that the area you swab the sample from is always the same for every sample. When you compare before-samples to after-samples they need to have been swabbed from an identically large area to be comparable to one another. Otherwise the results are just rubbish and tell you nothing.
I could continue, but I think the message is quite clear already. There are many ways to manipulate disinfection results. In the end it comes down to ethics, moral, and conscience of the disinfection solution provider. When there are people´s lives at stake, there must be some sense of responsibility and awareness. We all need to make a living, sure, but it is frightening to see how some are ready to tell you anything to go just to close a deal. The responsibility of doing the fact- and reliability checking is always on the buyer. That is YOU.
1. Always check the sources. Be very concerned, if there are claims made without any references.
2. Demand to see the actual tests behind the statements. Results can be heavily interpreted by an aggressive marketing department or just misunderstood by someone not skilled enough.
3. Who performed the tests? Make sure they were done at an accredited laboratory (ISO/EN) or a trustworthy institution. There are multiple sources that offer nice looking certificates, but they come from unreliable and by the international scientific community unrecognized laboratories that should not be trusted.
4. Check the values of the x- and y-axis. It is easy to manipulate the scales on the axis to make a decrease seem very dramatic.
5. Make sure there are information on the controls included somehow.
6. When talking about microbiology, a change from 48 cfu to 32 cfu is not considered a reduction. It is just a natural variation that could be due to human factor when performing the tests. A change from 1000 cfu to 32 cfu is a reduction. (cfu = colony forming unit)
7. Understand that in a logarithmic scale an increase of one means a tenfold (10x) increase. 1 log10 reduction means a 90% reduction, which is still not very large in the world of microbiology. Most disinfection standards require at least a 4 log10 reduction for bacteria and 3 log10 for viruses.
8. Pay attention to the term ”up to”, as it normally refers to a theoretical maximum. “Up to 99.999% of viruses dead in just 2 hours” may refer to a theoretical scenario that is not applicable to real life situations.
9. Lastly, there is an old saying: “If something sounds too good to be true, it probably is”. If the disinfection solution is so good, why did it only appear on the market simultaneously with the current pandemic?