I had problems with this a few years back, but the fix has changed now. Merely installing the Virtualbox Additions does not seem to be enough to get hardware OpenGL working. Also, the instructions I've found only mention Fedora.
To see if you are running with hardware or software rendering, use the command ``glxinfo''. You'll need to install the ``mesa-utils'' package; the info you're after is on the first page of output from glxinfo, so pipe it through ``less'' like so:
glxinfo | less
If it mentions LLVM, you're using software 3D rendering.
Ignore DRISWRAST in the /var/messages/Xorg.0.log file - it's a red herring, it seems to always be there.
So, here are the steps:
 With your VM shut down, go into its Settings in the VirtualBox Manager. On the Display tab, tick the box to ``Enable 3D Acceleration''. I suggest you give it at least 32MB or so of VRAM. The default is 12MB, which is very little. Leave 2D acceleration off - it doesn't work with Linux guests (only Windows ones using the Vbox display driver).
 Boot your VM. Ensure it's up to date. From a shell prompt, type
sudo apt-get install virtualbox-guest-utils
This should also pull in virtualbox-guest-x11, virtualbox-guest-dkms and some other components. You may also wish to install build-essential.
N.B. You could also install the guest additions from within VirtualBox, but if so, it does not add the critical line ``vboxvideo'' to /etc/modules and 3D doesn't work. Adding this manually works, but N.B. that you will have to manually update the extensions in future.
 Reboot the VM.
Use ``glxinfo'' to check. You should see that rendering is now handled by the renderer ``Chromium'' from vendor ``Humper''. This means that you now have hardware-assisted 3D rendering inside your VM.
Interestingly, the frame rate shown with ``glxgears'' has gone way down,
from ~175 to ~85, but actual output is much smoother.
N.B. I have found with GNOME Shell that I got significant screen corruption unless running in fullscreen mode. KDE 4 (in Kubuntu) does not seem to be affected.
I used the hints here:
... to get this working.