What is the branch of Computer Science that studies how Anti Virus programs work?

Question

It is a trivial exercise in finite automata to show that there is no algorithm that can detect all the viruses, yet there are many software companies selling Anti Virus Software.

Is there any part of CS that deals with Viruses and Anti Viruses ?

PS : I am not asking about non CS related justification of to have AV or not, but only what category/subject within CS they come under, if any. If AV is not a subject within CS then that is also an acceptable answer, are there any refrences within CS context to Viruses and AV's?

Answer 1

There is a subarea of computer security called Computer Virology. The Journal of Computer Virology is devoted to the topic. Studying how anti-virus software works only scratches the surface of what the area is about.

For instance, there is even some work applying logic to malware: A General Definition of Malware by S. Kramer and J.C. Bradfield. Journal of Computer Virology (6) 2010.

Answer 2

Viruses and antivirus technology have a lot to do with CS. Your question reminded me of something I recently read. Here's an excerpt from the book by Williamson & Shmoys, The Design of Approximation Algorithms, page 6. It's justifying the (practical) importance of approximation algorithms, and uses the well-known set cover problem as an example in the context of viruses:

The set cover problem was used in the development of an antivirus product, which detects computer viruses. In this case it was desired to find salient features that occur in viruses designed for the boot sector of a computer, such that the features do not occur in typical computer applications. These features were then incorporated into another heuristic for detecting these boot sector viruses, a neural network. The elements of the set cover problem were the known boot sector viruses (about 150 at the time). Each set corresponded to some three-byte sequence occurring in these viruses but not in typical computer programs; there were about 21,000 such sequences. Each set contained all the boot sector viruses that had the corresponding three-byte sequence somewhere in it. The goal was to find a small number of such sequences (much smaller than 150) that would be useful for the neural network. By using an approximation algorithm to solve the problem, a small set of sequences was found, and the neural network was able to detect many previously unanalyzed boot sector viruses.

Surprising or not, this example shows that at least techniques in AI and combinatorial optimization are useful. After reading this, one could easily believe that many areas of CS have things that can be applied in the context of viruses and their detection. To answer your question more directly, many things in CS deal with viruses, at least indirectly.

Answer 3

You should be careful when using theoretical result to argue something cannot be done in practice. There are several dangers that one can fall in:

1. the theoretical result has assumptions that doesn't apply,

2. the problem in practice is not modeled well by the theoretical model,

3. in practice a solution doesn't need to be perfect to be useful.

You haven't given a formal definition of what is a virus so expanding on your claim about the trivial exercise can be helpful in understanding what you really mean.

A large part of what an antivirus software do is detecting known viruses (and their variants), and this is done by comparing strings (in files, memory, etc) with a finite list of strings (virus signatures). That is why we need to update the database of the antivirus software regularly.

There are additional capabilities on detecting potential viruses based on their "behavior", but these methods are neither complete nor sound (and they don't need to be sound/complete to be useful). Designing a antivirus software seems to fall more in computer engineering than computer science so far (though computer science and computer engineering are closely related).

Generally these studies would fall in more applied part of computer-security (and more generally system areas: computer networks, operating systems, etc.) of computer science, but using ideas from other parts of computer science (machine learning, etc.) is typical.

You may want to check The Art of Computer Virus Research and Defense.