Progress Update: October 2018


#1

Emissive

Hello everyone, another month passed, a ton of work done and time for a progress update!

In the last month I’ve focused on wrapping up the particle system with a bunch of new features, added support for emissive materials, worked on new unified BSL compiler and added a few important enhancements to the core.

With the particle system nearly 100% feature complete, the only remaining feature for v1.1 is the decal system which will be coming soon, and then it’s just a matter of polishing everything and wrapping it up for release, which is still planned for near the end of the year.

For those wanting more details here is a complete breakdown of what was added:

Emissive materials

All standard materials that support lighting can now have parts of their surfaces defined as emissive through an emissive mask texture and an emissive color and intensity. Emission is useful for surfaces representing light emitters (screens, monitors, neon lights, etc.), it interacts nicely with effects such as bloom, and can be even be used for indirect lighting through the use of light probes.

To set up an emissive material assign the gEmissiveMaskTex texture parameter to the material. Red channel of this texture will control which parts of the surface are considered emissive, where 1 means emissive and 0 non-emissive. Actual color and intensity of the emitted light can be controlled through gEmissiveColor property.

HShader shader = gBuiltinResources().getBuiltinShader(BuiltinShader::Standard);
HMaterial material = Material::create(shader);
...
// Mark entire surface as emissive
material->setTexture("gEmissiveMaskTex", gBuiltinResources().getTexture(BuiltinTexture::White));

// Make the color a bright red
material->setColor("gEmissiveColor", Color::Red* 10.0f);

The header of this post contains a gif of an emissive material being applied to a particle system.

Emission bursts

EmissionBurst
So far the particle system only supported continuous particle emission - meaning the particles were either being emitted (if active), or not (if inactive). Emission bursts give you more control and allow you to spawn a specific number of particles at a specific point the particle system lifetime. This can be useful in many situations, for example fireworks or explosions.

To set up bursts, assign an array of one or multiple bursts to ParticleEmitter::setEmissionBursts.

SPtr<ParticleEmitter> emitter = ...;
emitter->setEmissionBursts(
{
    ParticleBurst(
     1.0f, // Time at which to trigger
     200.0f, // How many particles to spawn
     0, // How many times to repeat (0 = infinite)
     1.0f), // Interval between bursts
});

Non-randomized shape emission

SequentialEmission

More control has been added to allow particles to spawn in non-random ways on a surface of an emission shape.

Sequential mesh emission

Mesh emission shape now supports a sequential mode where particles will be emitted in the order the mesh vertices are defined. This makes the mesh act like an exact list of positions at which you want particles to spawn, and at what times. You can use this to create very specialized particle system where you need very fine control over the system.

PARTICLE_STATIC_MESH_SHAPE_DESC meshShape;
meshShape.mesh = ...;
meshShape.type = ParticleEmitterMeshType::Vertex;
meshShape.sequential = true;

emitter->setShape(ParticleEmitterStaticMeshShape::create(meshShape));

New emission modes

Line, Circle and Cone shapes support multiple new emission modes. Use the new mode field (of ParticleEmissionMode type) available on those emission shape descriptor structs to define it.

ParticleEmissionMode::type controls which of the new emission modes to use. Use ParticleEmissionMode::Type::Loop and ParticleEmissionMode::Type::PingPong to spawn particles sequentially on the shape. Use ParticleEmissionMode::interval to control how far apart should the spawned particles be, and ParticleEmissionMode::speed to control how fast should the particles move around the shape.

Use ParticleEmissionMode::Type::Random to spawn the particles randomly, which is the old behaviour. You also get extra control to limit the random spawns to a specific interval through ParticleEmissionMode::interval, which means the particles will only spawn on certain positions on the shape.

Finally ParticleEmissionMode::Type::Spread will spread out all particles spawned during a single burst over the entire shape, optionally using ParticleEmissionMode::interval to limit the spawn positions.

// Spawn particles, moving around the circle
PARTICLE_CIRCLE_SHAPE_DESC circleShape;
circleShape.radius = 0.7f;
circleShape.mode.type = ParticleEmissionMode::Type::Loop;
circleShape.mode.speed = 90.0f; // 90 degrees per second
circleShape.mode.interval = 10.0f; // At 10 degree intervals

emitter->setShape(ParticleEmitterCircleShape::create(circleShape));

See the gif above for an example of this mode in action.

BurstAndSequential
Emission bursts spawning on a circle shape using the Spread emission type.

More particle evolvers

ForceOverTime
New evolvers have been added to the CPU particle system, covering some basic functionality that was still missing:

  • ParticleColor - Changes the particle color over its lifetime.
  • ParticleSize - Changes the particle size over its lifetime.
  • ParticleRotation - Rotates the particle around it’s Z axis (for 2D particles) or separately on each axis (for 3D particles)
  • ParticleForce - Applies an arbitrary force to to particles.
// Fades from white to black over the particle's lifetime
PARTICLE_COLOR_DESC colorDesc;
colorDesc.color = ColorGradient(
{
    ColorGradientKey(Color::White, 0.0f),
    ColorGradientKey(Color::Black, 1.0f)
});

SPtr<ParticleEvolver> colorEvolver = bs_shared_ptr_new<ParticleColor>(colorDesc);
particleSystem->getEvolvers().add(std::move(colorEvolver));

// Scales the particle by 4 over the particle's lifetime
PARTICLE_SIZE_DESC sizeDesc;
sizeDesc.size = TAnimationCurve<float>(
{
    TKeyframe<float>{1.0f, 0.0f, 1.0f, 0.0f},
    TKeyframe<float>{4.0f, 1.0f, 0.0f, 1.0f},
});

SPtr<ParticleEvolver> sizeEvolver = bs_shared_ptr_new<ParticleSize>(sizeDesc);
particleSystem->getEvolvers().add(std::move(sizeEvolver));

// Rotates the particle by 180 degrees over its lifetime
PARTICLE_ROTATION_DESC rotationDesc;
rotationDesc.rotation = TAnimationCurve<float>(
{
     TKeyframe<float>{0.0f, 0.0f, 1.0f, 0.0f},
     TKeyframe<float>{180.0f, 1.0f, 0.0f, 1.0f},
});

SPtr<ParticleEvolver> rotationEvolver = bs_shared_ptr_new<ParticleRotation>(rotationDesc);
particleSystem->getEvolvers().add(std::move(rotationEvolver));

// Applies a force to particles, pushing them towards the positive X axis
PARTICLE_FORCE_DESC forceDesc;
forceDesc.force = TAnimationCurve<Vector3>(
{
    TKeyframe<Vector3>{Vector3::ZERO, Vector3::ZERO, Vector3::ONE, 0.0f},
    TKeyframe<Vector3>{Vector3(100.0f, 0.0f, 0.0f), -Vector3::ONE, Vector3::ZERO, 0.5f},
});
forceDesc.worldSpace = true;

SPtr<ParticleEvolver> forceEvolver = bs_shared_ptr_new<ParticleForce>(forceDesc);
particleSystem->getEvolvers().add(std::move(forceEvolver));

Soft particles

SoftParticles

Soft particles is a special rendering mode added to the particle shader that allows the particle to be softly faded out when it intersects scene geometry. In the gif above notice the particle system intersecting the ground on the left image (no soft particles) and being faded in on the right image (soft particles).

To enable soft particle rendering enable the SOFT shader variation on the material (variations are a new feature I’ll talk more about below). Then use the material as normal.

HShader particleShader = gBuiltinResources().getBuiltinShader(BuiltinShader::ParticlesUnlit);
HMaterial particleMaterial = Material::create(particleShader);
particleMaterial->setVariation(ShaderVariation(
{
    ShaderVariation::Param("SOFT", true)
}));

Other additions

And there is a variety of other smaller additions…

User-controllable material variations

As shown in the soft particle example above, users can now enable/disable shader variations on a Material. Shader variations are explained in the BSL manual, but have so far only usable by the renderer on internal materials, not on materials created/used by users.

Users can now define different versions of the same shader, where each variation has some minor (or major) difference.

// Example shader with a single variation parameter SOME_PARAM. A version of
// this shader will be compiled with every value of SOME_PARAM, as well as combinations
// with any other variation parameters if present.
shader Surface
{
	variations
	{
		SOME_PARAM = { false, true };
	};

	code
	{
		float4 fsmain(in VStoFS input) : SV_Target0
		{
			float4 output = input.color;
		
			#if SOME_PARAM
				// Do something
			#else
			    // Do something else
			#endif
		
			return output;
		}
	};
};

These variations can then be enabled by calling Material::setVariation with a list of enabled variation parameters to use.

HShader shader = ...;
HMaterial material = Material::create(shader);
material->setVariation(ShaderVariation(
{
    ShaderVariation::Param("SOME_PARAM", true)
}));

Variations can be used as an alternative to runtime conditionals (e.g. if based on some parameter) as they are resolved at compile time so you don’t need to pay the cost of the conditional or unused features in the shader.

Be aware of the combinatorial explosion that can result in having too many variation parameters, as each parameter needs to combined with every other. Each variation is in reality its own separate shader so they can end up taking a lot of space, or take a long time to compile.

Import tool

Import process for bsf built-in resources was so far handled on framework start-up - it would check all the raw data assets and reimport them as needed. This process ended up being problematic for multiple reasons:

  • It would often trigger when not needed. This resulted in files in the Data folder being modified just by users running their applications. And even worse, in would result in long wait times during framework start up (esp. for those running the framework for the first time, making a bad first impression)
  • It would import the resources using whatever configuration you were compiling the framework as. This resulted in extra slow import times if the configuration didn’t have optimizations turned on (i.e. Debug configuration)
  • Shader compile errors would often get unnoticed as they would be logged during application start-up, rather than the build step

The whole import process has been now moved to a separate import tool executable. It’s part of the framework as bsfImportTool target. If user enables asset import (BUILD_ASSETS flag in CMake), the import tool will check the source assets for changes and import them as needed, whenever CMake is ran. Meaning the whole process is now ran at build time, yielding a much nicer workflow - and better yet, it can be completely avoided by the majority of users.

Note:

  • If you have so far relied on the framework to automatically process your changes, it won’t anymore. This includes modifications to the shader source code. You will now need to run CMake after every modification, with BUILD_ASSETS enabled.
  • If you turn on BUILD_ASSETS you need to compile and install the import tool to Dependencies/tools/bsfImportTool for the CMake to be able to find it.
  • If you aren’t modifying any source data assets (files from Data/Raw folder), then keep BUILD_ASSETS disabled and don’t worry about any of this

BSL 2.0 (Work-in-progress)

Last time I mentioned I wanted to make a slew of enhancements to the bsf shading language (BSL). These changes would make the language more expressive, safer to type, significantly faster to compile, reduce size overhead of shader variations and open up room for developing an IDE and a debugger.

The first step is unifying the BSL compiler in the XShaderCompiler library, which is currently only responsible for parsing HLSL parts of the code. I’ll be slowly working on this with no planned ETA, as this is not part of any release milestone yet.

So far I’ve moved parsing for blend, stencil, depth and rasterizer states to XShaderCompiler, as well as improved the syntax for sampler state definitions. Next up on the agenda is to add parsing for shader/mixin/pass blocks, and support parsing of shaders with multiple entry points, variations and sub-shaders, all in a single parsing pass. At which point I’ll be able to drop the existing lexer/parser code and rely on XShaderCompiler fully, and start working on some nice new language additions (like templates!) and even IDE integration (I’m thinking of an extension for VS Code as they seem to be easy to write).

I’ll be working in the bsl branch of my fork of XShaderCompiler if you want to follow the progress (https://github.com/BearishSun/XShaderCompiler/tree/bsl)

SmallVector container

One of our contributors @paolopaoletto has added support for the SmallVector container. This container accepts an extra template parameter N, and will not allocate dynamic memory if its size stays under N. It’s a phenomenal container if you want to avoid making dynamic memory allocations.

What’s next?

Rough rundown of my plans for the next month or two:

  • Decal rendering support
  • Particles continuous emission, drag/dampening evolver
  • Particle documentation
  • More examples:
    • Light & Shadow example
    • Particles example
  • Clean up bugs
  • Finally sort out Linux rendering with Mesa drivers
  • Continued work on BSL 2.0
  • Editor
    • Integrate particle system
    • Async scene loading

And more… as I always tend to throw in a few unplanned features, plus we have a few contributors working on interesting new additions :slight_smile:

In the longterm we can look forward to some nice features in v1.2 (networking, pathfinding, more shading models), C# scripting support and new stable release of the editor - exciting stuff on the horizon!

Finally and most importantly, huge thanks to my Patrons, your support really means a lot in keeping this project moving forward.

Until next time!