OS/2 1.x was a clean-sweep, largely legacy-free OS with only limited backwards compatibility with DOS.
OS/2 2.x and later used VMs to do the hard stuff of DOS emulation, because they ran on a chip with hardware-assisted DOS VMs: the 80386’s Virtual86 mode.
NeXTstep was a Unix. It predated FreeBSD, but it was based off the same codebase: BSD 4 Unix. It “only” contained a new display layer, and that itself was based off existing code — Adobe PostScript — and the then-relatively-new technique of object-oriented development. Still substantial achievements, but again, built on existing code, and with no requirement for backwards compatibility.
BeOS was a ground-up new OS which wasn’t backwards or sideways compatible with anything else at all.
NT is based on OS/2 3.x, the planned CPU-independent portable version, with a lot of design concepts from DEC VMS incorporated, because it had the same lead architect, Dave Cutler. Again, the core NT OS isn’t compatible with anything else. This is rarely understood. NT is not a Win32-compatible kernel. NT isn’t compatible with anything else, including VMS. It’s something new. But NT supports personalities, which are like emulation layers running on top of the kernel. When NT shipped, it included 3: OS/2, POSIX and Win32. OS/2 is deprecated now, POSIX has developed into the Linux subsystem, and Win32 is still there, now in 64-bit form.
The point is, none of these OSes were enhanced versions of anything else, and none were constrained by compatibility with existing drivers, extensions, applications, or anything else.
Apple tried to do something much, much harder. It tried to create a successor OS to a single-user, single-tasking (later cooperatively-multitasking, and not very well), OS for the 68000 (not something with hardware memory protection, like the 68030 or 68040), which would introduce those new features: pre-emptive multitasking, virtual memory, memory protection, integrated standards-based networking, etc.
All while retaining the existing base of applications, which weren’t written or designed or planned for any of this. No apps == no market == no use.
Apple took on a far harder project than anyone else, and arguably, with less experience. And the base hardware wasn’t ready for the notion of virtual machines yet.
It’s a great shame it failed, and the company came relatively close — it did have a working prototype.
It’s often said that Apple didn’t take over NeXT, nor did it merge with NeXT — in many important ways, NeXT took over Apple. Most Apple OS developers and project managers left, and were replaced by the NeXT team.
The NeXT management discarded Copland, most Apple technologies — OpenDoc, OpenTransport, GameSprockets, basically everything except QuickTime. It took some very brave, sweeping moves. It took the existing MacOS classic APIs, which weren’t really planned or designed, they just evolved over nearly 1½ decades — and cut out everything that wouldn’t work on a clean, modern, memory-managed, multitasking OS. The resulting cut-down, cleaned-up API was called “Carbon”. This was presented to developers as what they had to target if they wanted their apps to run on the new OS.
Alternatively, they could target the existing, far cleaner and richer NeXT API, now called “Cocoa”.
The NeXT team made no real attempt to be compatible with classic MacOS. Instead, it just ran all of classic MacOS inside a VM — by the timeframe that the new OS was targeting, machines would be high-enough spec to support a complete classic MacOS environment in a window on top of the Unix-based NeXTstep, now rebadged as “Mac OS X”. If you wanted your app to run outside the VM, you had to rebuild for “Carbon”. Carbon apps could run on both late versions of classic MacOS and on OS X.
This is comparable to what NT did: it offered a safe subset of the Win32 APIs inside a “personality” on top of NT, and DOS VMs with most of Win16.
It was a brave move. It’s impressive that it worked so well. It was a fairly desperate, last-ditch attempt to save the company and the platform, and it’s easier to make big, brave decisions when your back is against the wall and there are no alternatives... especially if the mistakes that got you into that corner were made by somebody else.
A lot of old Apple developers left in disgust. People who had put years of work into entire subsystems and APIs that had been thrown in the trash. Some 3rd party developers weren’t very happy, either — but at least there was a good path forwards now.
In hindsight, it’s clear that Apple did have an alternative. It had a rich, relatively modern OS, upon the basis of which it could have moved forwards: A/UX. This was Apple’s Unix for 680x0, basically done as a side project to satisfy a tick-box for US military procurement, which required Unix compatibility. A/UX was very impressive for its time — 1988, before Windows 3.0. It could run both Unix apps and classic MacOS ones, and put a friendly face on Unix, which was pretty ugly in the late 1980s and early 1990s.
But A/UX was never ported to the newer PowerPC Macs.
On the other hand, the NeXT deal got back Steve Jobs. NeXTstep also had world-beating developer tools, which A/UX did not. Nor did BeOS, the other external alternative that Gil Amelio-era Apple considered.
No Jobs, no NeXT dev tools, and no Apple today.