Suppose I want to build an operating system based on a very small native lower kernel that acts as a managed code interpreter/runtime and a larger upper kernel compiled to a non-native machine language (Java bytecode, CIL, etc.). Examples of similar operating systems would be Singularity and Cosmos.
What pitfalls and development challenges exist writing an OS with this sort of infrastructure in contrast to a purely-native solution?
Depending on the language, there can be many development challenges:
Pointers: If a language doesn't have pointers, it will be a challenge to do relatively-easy tasks. For example, you can use pointers to write to VGA memory for printing to the screen. However, in a managed language, you will need some kind of "plug" (from C/C++) to do the same.
Assembly: An OS always needs some assembly. Languages like C#, Java, etc. don't work so well with it, unlike C/C++. In C or C++ you can also have inline assembly which is very, very useful for many tasks. There are MANY cases where this is needed (examples in x86): loading a GDT, loading an IDT, enabling paging, setting up IRQs, etc.
Control: If you are using something like Cosmos, you aren't having full control. Cosmos is a micro-kernel and essentially "bootstraps" your "kernel". You can implement something like Cosmos from scratch if you really wanted to, however, that can take a long, long time.
Overhead: With managed languages, there is A LOT of overhead compared to C or even C++. Things like Cosmos need to implement a lot of things before even a C# hello world kernel can be run. In C, you are ready to go, no real setup needed. In C++, there are just a few things that need to be implemented to use some of C++'s features.
Structures: In C/C++ there are structs which many managed languages do not have, and so, you would need to implement some way of having something like a struct. For example, if you want to load a IDT (Interrupt Descriptor Table), in C/C++, you can create a struct (with a packed attribute), and load it using the x86 ASM instruction lidt. In a managed language, this is much harder to do...
Managed languages, syntax-wise, are easier, however, for many OS related things are many times not very-well suited. That doesn't mean they can't be used, however, something like C/C++ are often recommended.
I have heard it claimed (by a researcher working on a competing microkernel technique) that very little is known about how to evaluate security of systems that are extensible through managed code.
The problem is that the kinds of bugs that might cause a security hole are very different than security researchers are used to. In a traditional microkernel all the drivers and other subparts of the kernel are isolated from one another by running them in different address spaces. In a microkernel where the isolation is implemented through type checking managed code you avoid the enormous overheads of switching address spaces every time you need to use a sub-service, but the tradeoff is that now evaluating the isolation mechanism is more difficult.
Any particular part of the kernel (say a device driver) written in the managed language is safe if and only if the type checker says the driver is safe and the type checker has no bugs. So the type checker is part of the kernel core. In practice it seems that type checkers are considerably larger and more complicated than are traditional microkernel cores. That means that the attack surface is potentially larger.
I don't know whether traditional micro-kernel isolation techniques or managed-code based isolation techniques are really more or less reliable. There's a bootstrapping problem here: until managed-code isolation techniques are widely used, we're not going to know how often they are insecure. But without knowing how insecure they are, it is difficult to deploy them in situations that are security critical.