Go to about:config and type groups, it's being actively developed.
There's also another API that you can actually use if you're a developer in Browser Toolbox, enable Browser Toolbox
and type gBrowser.addTabGroup, you can add tabs to group using this API, and if you find a way to append it to Tab Tray you should be able to create something like this image:
