- Drivers
- Program compatibility
- Untested security
The last is inevitable. However, a new OS would allow you to use modern security architecture that over time could actually help make things more secure. One recent OS that has decidedly caught on: android.
The other two are harder.
Android decided to use the linux kernel, probably to help with the driver issues. It kept a java-esce programming language to ease developers in and help make programs SOMEWHAT compatible-- it blew away the archaic and ugly java view/model/event layer with its custom GUI layer (now the cause of a lawsuit).
The other issue is this: Existing operating systems are usually good enough. Exceptions in the past: DOS (hardware changes allowed better security), Windows XP (security architecture problems-- not really solved in vista/7, but the new one is better).
Well why would you even want to make one?
- Fun
- Serve a new need (embedded, touch screens, 3D, frequently adding/vanishing resources, reads your mind, injects thoughts into your mind, needs to be distributed, fault tolerant, etc)
- Security through obscurity (or failsafe/stopgap in case of a compromised OS)
- Performance
- Easier for programmers and/or users
So android started for all of the above. Same with iOS.
I'd love to see a distributed one-- the same "computer" runs at work, home, the cloud, from any PC and resources can be randomly connected/disconnected with no problems. Maybe a smart gui could carry the same program on my phone to my desktop with minimal loss of functionality. This would be ridiculously hard though.
Also if you could actually grow computers (which are crazy inexpensive, can be made locally, but they fail a lot), a fault tolerant distributed system would be awesome. Ya know, like our brains.