This document presents a detailed, albeit methodologically flawed, test of 41 mobile antivirus applications conducted by an independent researcher. The author’s initiative to test not only against a standard set of malware but also against a personally obfuscated subset demonstrates a sophisticated approach to security evaluation. However, several critical weaknesses in the testing framework undermine the reliability and generalizability of the conclusions.
Analysis of the Test’s Foundation
The researcher established a two-pronged testing methodology. The first part measured detection against a collection of 560 mobile malware samples that were approximately six to twelve months old. The second, more novel part, tested detection against 310 of those same samples after the researcher applied obfuscation techniques to them. This dual approach correctly identifies that a good antivirus must do more than match known signatures; it must also possess heuristic or behavioral analysis capabilities to identify disguised threats.
The author also shows a keen awareness of the mobile security market by identifying shared antivirus engines. For example, the report notes that Emsisoft and e-Scan use the Bitdefender engine, while at least ten other lesser-known applications rely on the Trustlook engine. This correctly points to a market consolidation where numerous application front-ends are powered by a few core detection technologies, leading to identical performance results among them.
Critical Methodological Weaknesses
Despite the thoughtful design, the test’s execution contains significant flaws that limit the validity of its findings.
First, the use of an emulator instead of physical hardware is a primary concern. Modern malware often incorporates anti-emulation techniques, causing it to behave differently or not execute at all in a virtualized environment. This means the test may not accurately reflect how the malware or the antivirus would perform on an actual user’s device. Professional testing labs like AV-TEST and AV-Comparatives use physical devices precisely to avoid this variable.
Second, the malware sample set is problematic. The samples are described as being six to twelve months old. While this can test the breadth of an antivirus’s signature database, it fails to assess its ability to defend against current, in-the-wild, and zero-day threats, which is a critical function of modern security software. The origin and specific types of malware within the 560 samples remain undefined, making it impossible to know if the collection was diverse or skewed toward a few malware families.
Third, the timing of the tests introduces a critical inconsistency. The author admits that some products, like Surfshark and G-Data, were tested later than the products whose engines they use (Avira and Bitdefender, respectively). Antivirus vendors update their detection signatures multiple times a day. A test conducted even a few days later can produce a different result, giving the later-tested product an unfair advantage and invalidating any direct comparison.
Finally, the obfuscation method is a black box. The author states the samples were processed with “obfuscators” to hide them, but the specific techniques and their level of sophistication are not described. The term “very primitively… obfuscated” suggests the methods may have been basic. A simple packer or string manipulation might fool a low-tier product but would be trivial for a top-tier one to analyze. Without this information, the “obfuscated threats” test loses its objective meaning.
Interpretation of Results and Conclusions
The report’s ranking system, from ‘S’ (excellent) to ‘F’ (absolute failure), provides a clear but arbitrary hierarchy. The author’s placement of Sophos, Eset, and Ahnlab V3 in the top tier for both general and obfuscated detection aligns with results from some professional tests. However, the methodological issues mean these specific findings should be viewed as an interesting data point rather than a definitive verdict.
The test effectively highlights a significant issue in the mobile app marketplace: the proliferation of low-quality or entirely fake security applications. The author found that 21 of the 41 products failed to detect a single obfuscated threat, and three detected zero threats of any kind. This finding serves as a powerful warning to users who might download an antivirus based solely on its name or Play Store rating.
In conclusion, the author’s effort is a commendable piece of independent research that correctly identifies key challenges in mobile security. The analysis correctly exposes the weakness of many lesser-known antivirus apps and the deceptive nature of engine re-branding. Yet, the foundational flaws in the methodology—particularly the use of an emulator, an outdated sample set, and inconsistent test timing—prevent the results from being considered a reliable benchmark for antivirus performance. The report is best understood as an insightful case study rather than an authoritative consumer guide.
