Progress Update: March 2019


#1

Hello! Another monthly summary incoming.

Last month we’ve made major strides in three different areas:

  • A ton of editor fixes and tweaks - It’s already looking a lot more stable and pretty much all of the bigger known issues were resolved. There’s still various minor issues and quality of life improvements, as well as general testing to do. But the release is getting closer and we might have all major issues resolved in April.
  • Initial working networking system for v1.2 - Headless mode is now fully supported and initial networking functionality has been added, with the ability to host, connect, send and receive packets.
  • C#/Editor documentation - We’re moved to a brand new documentation generation system which will allow us to write cross-API documentation for both C++ and C#. This is also a prelude to the editor documentation.

As a side project, I’ve also made an important enhancement to the shader cross-compiler, by adding support for a HLSL backend, which is an important step towards BSL 2.0.

Read down below for more detailed information.

Major features

New documentation system

I’ve upgraded the manual documentation system from the previous Doxygen one to daux.io. This upgrade was required primarily because cross-API documentation was not possible with the old system. My goal is to be have a single set of manuals that can reference either the C++ or the C# API (and later, any other language we might add). This ensures the documentation is easy to maintain and makes the framework more malleable to API changes.

daux.io generates nice documentation out of the box, but most importantly it is very simple and easy to customize. For now I have added a link-resolver plugin that can resolve API referenced links to either the C++ and the C# API. The new link syntax is also much less cumbersome to write (e.g. @ref bs::Application::startUp "Application::startUp" vs @Application::startUp).

I’m also planning on adding another plugin that will render class documentation inline in the manual. This will be particularily useful when writing documentation for various Component and Resource types displayed in the editor Inspector. This way a good part of the editor documentation intended for normal users (i.e. non-programmers) can also be generated straight from the API documentation. Again reducing redundancy and cost of maintaining things.

The new system also comes with a variety of other niceties, such as a table of contents visible on every page, previous/next links and “Edit on GitHub” links which all make the manuals nicer to use and were sorely lacking in the old system.

You can find the new docs at the nightly doc page here. It’s functional but visuals are not done yet.

In short, this was an important step towards C# and editor documentation.

Full headless mode

Null backends were added for physics, audio and the renderer systems. This is on top of the null backend added for render API last month. To enable them you can specify “Null” to PHYSICS_MODULE, RENDERER_MODULE and/or AUDIO_MODULE during CMake generation.

This means the framework can now run fully headless! Perfect for servers and other console-only apps. I still need an easy way to enable headless mode, as manually setting different modules to “Null” and requiring the user to rebuild the framework probably isn’t the best way of doing it. That’ll come sometime before v1.2, but for now this is perfectly functional.

Initial networking

Basic networking support has been added into the framework! You can now establish connections, listen to connections, send and receive packets. It’s basic but ultimately it’s all you need. To enable this functionality turn on EXPERIMENTAL_ENABLE_NETWORKING
in CMake. Note that I don’t yet provide pre-built RakNet dependency for Linux & macOS, so if you want to test on those systems you need to compile RakNet yourself.

The new API is in BsNetwork.h, with NetworkPeer being the primary networking class. A simple example would look like this:

// Start up server
NETWORK_PEER_DESC serverDesc;
serverDesc.listenAddresses.add(NetworkAddress(nullptr, 50000)); // Listen on port 50000
serverDesc.maxNumConnections = serverDesc.maxNumIncomingConnections = 32; // Max 32 connected players

SPtr<NetworkPeer> serverPeer = bs_shared_ptr_new<NetworkPeer>(serverDesc);

// Start up client and connect to server
SPtr<NetworkPeer> clientPeer = bs_shared_ptr_new<NetworkPeer>();
clientPeer->connect("127.0.0.1", 50000);

// Send packet from client to server
char dummyData[256] = {0}; // Data to send, just zeroes

// First byte of a message must be the message ID. Unique ID's starting 
// with NETWORK_USER_MESSAGE_ID (lower ID's are reserved)
dummyData[0] = NETWORK_USER_MESSAGE_ID;

PacketData sendData;
sendData.bytes = dummyData;
sendData.length = sizeof(dummyData);

clientPeer->send(sendData, NetworkAddress("127.0.0.1", 50000));

// Receive packets on the server
NetworkId serverId;
while(true)
{
	NetworkEvent* event = serverPeer->receive();
	if(!event)
		break;
	
	switch(event->type)
	{
		case NetworkEventType::ConnectingDone:
			LOGDBG("Connected");
			serverId = event->senderId;
		break;
		case NetworkEventType::Data:
		{
			const PacketData& data = event->data;
			// Do something with the data
		}
		
		break;
		
	}
		
	serverPeer->free(event);
};

Note that higher level forms of networking are also coming in v1.2, most importantly replication and RPC’s. They will allow simpler networking, ideally without requiring the user to write any networking code, unless for specific circumstances.

The API above is also subject to change.

HLSL backend for the shader cross-compiler

As part of the ongoing upgrade to BSL 2.0, the shader cross-compiler can now generate HLSL code. Previously the cross-compiler was only able to generate GLSL code, while we were using tricks to re-use the input BSL and only preserve the HLSL bits. This makes the shader compilation more complex and limits how much can we extend BSL with new features (it can never be too different from HLSL).

By having a separate HLSL backend we will be able to add bigger and complex new features to BSL, reshaping the language so it can fit our needs as required. It will also allow us to unify the shader compilation process, not requiring us to have separate lexers/parsers for BSL and HLSL parts. Instead we will do everything in one place, making the code cleaner and most importantly compilation faster.

Work on BSL 2.0 is still on going, but this was one of the more important parts still remaining. As time goes on I’ll start wrapping up the remaining features and eventually move us to the new system completely. Perhaps even in time for v1.2.

Minor features

Framework

  • DenseMap implementation by paolopaoletto. It’s an alternative hash-map implementation that stores values densely and uses quadratic probing (compared to chaining used by standard library hash maps).
  • Bitwise helper class added to C# by paolopaoletto
  • SBGen tool can now output an optional .xml with parsed C++ types and their C# counterparts, their members and their documentation. This can be used for the cross-API documentation generation, as well as for potentially other purposes.

Editor

  • Build system has been updated so it now has an install step. Meaning you easily get a distributable editor executable after a custom build.
  • Continous integration was set up, meaning less broken builds and (soon) nightly executable builds
  • Added a LoadOnAssign C# attribute that forces a RRef to be loaded when assigned from the inspector
  • ProjectLibrary can now report import progress, and in-progress imports can be cancelled
  • Added an in-progress spinner for resource re-imports in the Inspector
  • A warning window is now shown when attempting to exit the editor while import is in progress (instead of just hanging)
  • Added a way to directly open a modal window with specific width/height, instead of requiring a resize after
  • Material inspector now accepts sprite textures, animation curves and color gradients, meaning you can animate material properties from the inspector
  • Double-click on BSL and plain text files will now open them in OS-specified editor
  • If managed assemblies are missing, the editor will now detect it and try to rebuild
  • If the script IDE solution is missing, the editor will now detect it and try to rebuild

Bug fixes

Framework

  • GUITexture ScaleToFit mode now behaves properly
  • GUITexture optimal size now properly accounts for animated sprites
  • Fix SpriteTexture not being properly initialized on the core thread, when deserialized
  • Fix Euler angle <-> matrix ordering to match the convention
  • Animation import now properly applies global scale to bind poses
  • Animation import will now properly import bones from files containing multiple meshes
  • Changed animation import euler angle order to avoid artifacts in rare cases
  • Fix text rendering not properly accounting for a two consecutive newlines
  • Transform change notifications are no longer sent out to uninitialized scene objects when deserializing
  • Fixed a crash when deserializing mesh colliders
  • Fixed a divide by zero issue in GUI, happening when a GUI element has zero width or height
  • Fixed import tool not properly receiving its arguments
  • Zero sized textures will no longer crash on DirectX
  • Fix an issue where a window could lose focus and then be unable to gain it back by clicking on it
  • Fix a crash when attempting to retrieve a texture from a material, if a sprite texture with an unloaded atlas is used
  • Resolving resources from RRef will no longer accidentaly start the load asynchonously
  • Relevant particle shader materials marked as internal to make for a nicer inspector display
  • Fixed a particle system crash if more particles than maxParticles are spawned on a single frame
  • C# ApplyOnDirty and PassByValue attributes now properly propagate to child fields
  • Assigning a null evolver to the particle system will no longer cause a crash
  • Wireframe hemisphere debug drawing now draws the relevant arcs at full length
  • Make sure particle system is made aware of changes to the assigned material after it has been assigned
  • Particle system rendering layer is now properly taken into account when culling
  • Resource manifest save/load with no relative path will no longer cause a crash

Editor

  • Fix ColorPicker window sliders being slightly out of place when initially opened
  • Fix editor not properly updating all script managers
  • Fix GUI update lagging one frame
  • Fix texture not being assignable to a material in the Inspector
  • Fix an issue where built-in meshes would fail to load
  • Sprite texture drag and drop to GUITextureField now works
  • GUITextureField will resolve the preview image properly if a preview icon doesn’t exist
  • Material inspector will avoid fully loading a texture resource, just to check its type
  • Assigned sprite texture will no longer be immediately overriden by a null normal texture
  • Keep native modified resources alive in editor until the project is saved
  • Saving a project will now properly save modified sprite textures
  • Default empty texture field will display the “(None)” label and without the clear button
  • Fix an assert during as assembly refresh due to unreleased GC
  • Import options retrieve from the project library will now properly reflect the native import options type
  • Fixed a ProjectLibrary crash when attempting to reimport a file asynchronously when the previous import for the same file hasn’t yet finished
  • Fix editor & project settings incorrectly deleting properties
  • Scene camera position/rotation now persists application resets and assembly reloads
  • Updated C# script template with the new bs namespace

What’s next?

I’m continuing work on editor bug fixing and polish, as the editor release is still my priority. But this is slowly getting wrapped up and I should soon be free to invest more time in v1.2 features. This means continuing with higher level forms of networking, including RPCs and replication.

I might also do more work on multi-scene, BSL 2.0 or the new documentation generation. I’ve been also wanting to add support for dynamic fonts, as the current font import process is proving cumbersome, so I might squeeze that in somewhere if I have the time.

That’s it for now, stay tuned, and special thanks for all the Patrons supporting the project!
Cheers!