Progress Update: April 2019


#1

Hi everyone. Another month, another progress update!

The main focus throughout April was on the following areas (in order of priority)

  • Editor fixes & polish to get it ready for release
  • Complete overhaul of the editor undo/redo system
  • More work on v1.2 networking feature
  • More work on the C#/editor documentation

The editor is still taking priority as I am aiming for a release in June, and that date is getting closer. The good news is that the undo/redo system improvement was the last big task left for it, and the majority of minor issues has been resolved as well. There are still known issues, some critical, but for the most part we’re on track to get something pretty stable by June.

I was hoping to do a bit more work on networking this month, and start on the replication system implementation but sadly work on the editor took a lot of my time. Regardless we’re made some nice additions to the networking foundation and I managed to do a lot of research, getting ready for the initial implementation which may come in May.

Continue below for a more detailed list.

Major features

Networking improvements

Var-ints
Bitwise class has been extended with support for var-ints. Var-ints allow you to have extremely space efficient storage for integers, which is particularily useful for networking where size is critical. They are better than normal integers as they will take up only as much space as they need. So an integer value of ‘10’ will only require 1 byte to store, while a value of ‘300’ will require 2, and so on.

You can use this through encodeVarInt and decodeVarInt methods in the Bitwise class, and we will also be using it internally for our replication purposes, to ensure the packet sizes for networking are minimized.

UINT32 v1 = 55;
UINT32 v2 = 367;
INT32 v3 = -1;
		
UINT8 output[50];

// Encode the values to var-ints
UINT32 writeIdx = 0;
writeIdx = Bitwise::encodeVarInt(v1, output);
writeIdx += Bitwise::encodeVarInt(v2, output + writeIdx);
writeIdx += Bitwise::encodeVarInt(v3, output + writeIdx);
		
// Decode the values back into normal integers
UINT32 readIdx = 0;
readIdx += Bitwise::decodeVarInt(v1, output + readIdx, writeIdx - readIdx);
readIdx += Bitwise::decodeVarInt(v2, output + readIdx, writeIdx - readIdx);
readIdx += Bitwise::decodeVarInt(v3, output + readIdx, writeIdx - readIdx);

Bitstream
Brand new Bitstream class has been added, allowing you to write data as a stream of bits, in an extremely packed and space efficient way. You would also use this primarily for networking, where space requirements are extremely tight.

The class provides a variety of methods of encoding data into the stream. One of the methods are var-ints as shown above, but there are others such as:

  • Conditional data - If data is unchanged, encode just one bit, otherwise encode the data itself + one header bit
  • Number ranges - If a number if known to be in a range, encode it relative to the range, reducing the number of bits required to encode it.
  • Normalized values - If a number is normalized, lesser number of bits might be needed to encode it.
  • Raw bits - Write a specific number of bits manually
Bitstream bs;

// Write a normal integer, taking up 32 bits
bs.write(12345); 

// Write only the first 10 bits of 'v'
UINT32 v = 987;
bs.writeBits((uint8_t*)&v, 10); 

// Write a negative var-int encoded as 2 bytes
bs.writeVarInt(-100); 

// Write a var-int if value is different from 'v', otherwise just write one bit
bs.writeVarIntDelta(987, v); 

// Write a float in [0,1] range, encoded into user-defined number of bits (16 by default)
bs.writeNorm(0.333f);

// Write a float in an arbitrary range, encoded into user-defined number of bits (16 by default)
bs.writeRange(7.5f, 5.0f, 15.0f);

// Write a integer in an arbitrary range, using only as many bits needed to represent it
bs.writeRange(750U, 500U, 1000U);

// ... use the equivalent read* methods to read the data back

Replication
Replication will ultimately allow scene objects and components to synchronize their data across a network without requiring the user to write any networking code.

I don’t have any code to show you yet, but I’ve done considerable research into existing implementations in order to design something that works best for bsf/Banshee.

The API on how is this to look on the user side is:

  • User will mark component fields in the RTTI as replicable. Only those fields will be replicated.
  • User will then manually enable replication on components and scene object instances. Such objects and their fields will be synced over the network according to the rules set in the RTTI class. User will be also able to send messages to such objects directly, acting like normal method calls (RPCs).
  • Prefabs would be intimately involved in the replication process. You will be able to spawn prefabs across the network easily. Prefabs would avoid large initial-state updates as all clients would be guaranteed to have the prefab resource and therefore the same initial state.
  • User would be able to define whether an app instance is a client, server, or both on application start-up

Eventually the plan is to add a variety of conditions and states for the field in the RTTI type. This can include frequency of updates, extrapolation rules, compression, priority and more. Scene objects will be able to define relevancy (e.g. don’t sync with far away players) and ideally I want to transparently handle things such as client side prediction and interpolation/extrapolation.

It will take some time to work out the full design, but we can start working on the foundation right away and work out the full design along the way.

C# documentation improvements

C# API documentation has been generated! It is available on http://docs.bsframework.io/nightly/csharp/api/.

The script binding generator has been improved so it generates an .xml file with all the data it has parsed. This generates a file with all the exported C++ types and their C# counterparts, as well as their documentation. We can use this file for many things, such as adding inline API documentation in the manuals themselves or cross-referencing types across both C++ and C# API.

The Daux generator we use for generation of manuals has been updated to support the latter. Any API references in the manual can now be resolved to either C++ or the C# API. This is a big step towards both the bsf C# and editor documentation, as it allows us to easily re-use the manuals for both C++ and C#.

Final step that needs to be done is add preprocessing for parts of the manual that are either C+±only or C#-only (such as the example snippets). This is something that is still to be resolved, but I ran successful experiments using the gpp preprocessor and might start integrating it into the documentation pipeline soon.

Once this is done, I can finally start on actually updating the documentation for C# purposes, as well as start on editor specific documenation.

Undo/Redo system

While undo/redo probably seems like a fairly standard and simple feature to a normal user, it is actually quite tricky to implement. My initial implementation for Banshee was very simplistic and inadequate, never properly tested, nor has it every properly worked.

In the newest implementation I decided on a complete refactor, building a much more intricate system that can support all our needs. In particular the enhancements are:

  • Changes to individual inspector fields are now tracked. Previously you could undo/redo changes only on the entire scene object, while now you can do it on a per-field basis, as it should be.
  • Scene objects and fields will now auto-focus as undo or redo operation is being done, making it more clear what has changed
  • Operations with scene handles such as move/rotate/scale are now undoable
  • Adding & removing of components to a scene object is now undoable
  • Creation of scene objects with some initial state can now be recorded and is undoable
  • When writing custom inspector code, undo/redo just works

This is still something that’s being worked on, but bulk of the work is done. In particular certain inspectors such as Camera or Light will not yet work with the undo-redo system as they haven’t been updated to the new inspector drawing API.

Talking about inspector drawing API, this has also been significantly improved. New drawer class has been added that allows you to very easily add new inspector fields, with transparent support for undo/redo and all other inspector functionality. These custom fields can also be incorporated with the auto-generated inspector ensuring you only need to write code for special cases.

I’ve started refactoring old inspector code and in most cases the old inspector can be completely removed (100% code removed, relying fully on the auto-generated inspector), or reduced to a minimal 20-30 lines of code for the mentioned special cases (compared to normal 200-300). Ultimately this makes everything easier to maintain, ensures all inspectors have uniform functionality and most importantly, it makes it trivially easy to add types with little to no extra work for the inspector GUI.

Minor features

Framework

  • All game objects now have a persistent unique UUID making it easier to consistently identify objects
  • Added an unsafe SPtr (USPtr) implementation that is faster than normal SPtr - by @paolopaoletto
  • AsyncOp refactored so it has a template type parameter and can now be auto-exported to C#
  • Importer and Resources modules are now exported to the C# API
  • ScriptAssemblyManager::getReflectableFromManagedObject now supports components and resources as well
  • SerializableObject can now return information about its base type
  • 'Open in Visual Studio` now works to immediately open and initialize the project without additional steps - by @ReDucTor

Editor

  • Expand/collapse state of hierarcy window is now persisted across script compilation
  • New, better scene loading approach that is more intuitive and differentialbe from prefab instantiation
  • GUIVector*Field types are now auto-exported to C#, removing a lot of boilerplate code
  • When creating a new library asset, assume user is going to rename it and immediately start the rename op - by @jayrulez
  • SerializedObject and SerializedDiff now work with native-wrapped types as well, allowing them to be used on built-in components and resources
  • Added SerializableSceneObject for serializing the state of a scene object and restoring it on demand
  • Collider, Joint, GUIWidget components as well as RenderSettings now use the new automated inspector, removing a ton of boilerplate code
  • Added a better inspector drawer that makes it much easier to write custom inspectors but also utilize the auto-inspector generation

Fixes

Framework

  • C# UUID class is now serializable
  • C# assembly reload will no longer attempt to load the same assembly twice
  • Significantly improved GUI rebuild performance if GUI elements are children to a fixed-size GUI panel
  • GUI no longer lags one frame after window resize, causing visual artifacts
  • Maximizing a window will use the correct monitor size in multi-monitor setups
  • Fix built-in sphere mesh bounds being wrong due to some uninitialized verices
  • C# GUIWidget will no longer cause a crash during assembly refresh
  • When a component is deserialized its transform is now immediately transfered to the corresponding actor
  • Keyboard button pressed together with alt or ctrl keys no longer register as text input events, as these are generally used as shortcuts
  • GUIToggle will no longer trigger events if its value is changed programatically, in order to be consistent with the rest of the GUI
  • Container GUI elements can now forward focus requests to child GUI elements
  • Animation events now trigger using local clip time, rather than global animation time
  • Animation events now correctly use animation frame delta, rather than simulation frame delta
  • Updated docs so they clearly reflect custom components require a RTTI type implementation
  • Rendering a reflection probe no longer causes a crash during distance culling
  • Fixed profiler warnings when executing renderer tasks
  • Renderer tasks will no longer try to execute before the dependant frame data is ready
  • Fix D6 joint using uninitialized axis motion values
  • GUI focus lost events are now triggered before focus gain events
  • Core objects are now guaranteed to be kept alive until core sync finishes
  • Creation of a material with no shader will now properly initialize the material
  • Implicit conversion from Resource to RRef<T> will no longer cause an exception if the resource is null

Editor

  • Library window entries are now uniformly spaced out
  • Auto-select objects dropped in the Hierarchy window
  • Pinging a resource will now auto-scroll the Library scroll view to the pinged resource
  • Fixed a crash that was causing the release version of the editor not to run
  • Significantly improved library window refresh speeds by avoiding unnecessary GUI updates
  • GUI will no longer appear cut-off for a frame during modal window resize
  • Modal window now reports content and window width/height separately, fixing an issue where the color picker window was cut off
  • Nightly build fixed so the builds are now actually runnable
  • Automated inspector now properly displays base class fields as well
  • Built-in shader code updated to match the current API so we don’t get a compile error on creation
  • Native async resource import will no longer incorrectly destroy the resource before the import gets finalized

What’s next?

In May I’m hoping to get started on the implementation of the replication system for v1.2. A lot of the focus will also remain on the editor, fixing any bigger issues, as well as doing testing. I need to get the editor back and running on Linux and macOS, which might be larger tasks depending in what state those implementations are.

I’ve started a Discord server to keep in closer touch with the community, you can join here: https://discord.gg/8Xyf5gF

Special thanks goes to all the contributors and Patrons! It’s always nice to see people care about the project enough to invest their time and money into it. :slight_smile:

That’s it for now, stay tuned!