Custom code for Nanite on Snapdragon

How to Access Snapdragon Game Studios Unreal Engine Branches

To access Snapdragon Game Studios Unreal Engine Branches go to this website and follow the instructions from Unreal Engine on GitHub 

Now to get Nanite to work on Snapdragon, here are the engine modifications we used (note that in order to access the links below, you need to be logged in through the UnrealEngine account linking steps described above):

https://github.com/SnapdragonGameStudios-UnrealEngine/UnrealEngine/tree/nanite_5.4

https://github.com/SnapdragonGameStudios-UnrealEngine/UnrealEngine/tree/nanite_5.5

https://github.com/SnapdragonGameStudios-UnrealEngine/UnrealEngine/tree/nanite_5.6

No legacy HLOD system

We completely avoided the old Hierarchical Level of Detail system. Nanite’s LOD functionality seems strictly superior.

No editor-cooked content

We’ll likely update the project to a more recent version of Unreal, so we don’t want to rely on any *.uasset file specific to the engine. Those *.uassets could change with engine versions, and we’d prefer to explicitly manage project *.uassets across each update ourselves. We fail builds that try to package *.uassets from the Editor’s Content directory so that no one accidentally introduces such a reference.

Non-standard aspect ratio

The default value was yielding a sort of asymmetric vignette on Snapdragon 8 Elite Gen 5:

So, we changed the value for maximum aspect ratio...

…eliminated those black bars:

Package with Android (ASTC) only

Compared to other texture formats, ASTC was most efficient on Snapdragon.

Profiling Tools

stat fps as a coarse measure of performance

Development builds performed very similarly to our Shipping builds, so the moving average frame counter in Unreal Engine’s stat fps told us when frame time was clearly too long.

UnrealInsights with GPU events on Android

UnrealInsights told us where our GPU time and CPU time were going.

Even after all of our optimization work, we found that camera cuts often produced a single long frame (33.4ms-66.6ms – but no longer). That was due to Virtual Shadowmaps, Lumen, Nanite and DistanceFields processing the newly in-view assets.

We verified this with UnrealInsights events that we manually placed in Sequencer after every camera cut:

We could have aggressively optimized our content to never exceed 33.3ms, even after a camera cut. However, since our GPU frame time was otherwise always below 28ms (and often as low as 22ms), and CPU frame time never exceeded 33.3ms (and was typically closer to 8ms), we chose to accept a single missed vsync after many camera cuts.

Because a camera cut is already discontinuous, it’s difficult to notice a single missed vsync. We suspected that optimizing even more aggressively would significantly compromise our artists’ vision, with practically no gain in perceived motion smoothness.

GPU events in UnrealInsights

To make GPU events appear in UnrealInsights, we packaged our Development apks with

 +CVars=r.Vulkan.SupportsTimestampQueries=1

Next we created a UECommandLine.txt that contained the following

../../../QCOM1004/QCOM1004.uproject -tracehost=127.0.0.1 -filetrace -loadtimetrace  -statnamedevents -trace=Bookmark,Frame,CPU,GPU,LoadTime,File

After installing the app, we pushed the command line arguments so the Development build would pick them up:

adb push C:\UECommandLine.txt /data/media/0/Android/data/com.SnapdragonStudios.ProjectOne/files/UnrealGame/QCOM1004/UECommandLine.txt

Before launching the UnrealInsights app, we needed to invoke the following:

adb reverse tcp:1981 tcp:1981

…only once until the Windows x64 machine is restarted.

With that, launching the Development build while UnrealInsights is open automatically performs an event capture full of useful GPU events.

Snapdragon Profiler for profiling pGMEM memory metrics

We used Snapdragon Profiler to measure differences between storing NaniteVisBuffer data in system memory (the default behavior) and storing it in pGMEM explicitly. We discovered significant reductions in the following Snapdragon Profiler metrics:

• Bytes Data Actually Written
• Global Buffer Data Read BW
• Global Buffer Data Read Request BW
• Global Memory Load Instructions
• SP Memory

UnrealInsights showed that the long GPU frames after camera cuts were consistently and measurably slower with explicit pGMEM usage – but not slow enough to miss more than the one (allowable) vsync.

Internal tools showed:

• power consumption reduced by a factor of 0.96x
• a slight increase in frame pacing stability (29.76Hz -> 29.86Hz)

DataLayers for streaming and high-level occlusion

We saw significantly improved framerates when we manually made large sets of assets visible and invisible as the cinematic played. So, we batched assets into DataLayers:

We then activated and deactivated them in Sequencer:

Switching DataLayers on and off primarily helped us maintain our 33.3ms frame time – the whole cinematic could easily fit in the memory of the Snapdragon 8 Elite Gen 5 memory, with many gigabytes to spare.  Reducing the high-water mark of the app’s memory usage was a side benefit.

However, despite those wins, a DataLayer’s Activate event (even with preloading) could cause a GPU spike when processing ShadowDepths, LumenScreenProbeGather, GlobalDistanceFieldUpdate, NaniteVisBuffer and NaniteBassPass. Thus, sometimes we paired those events with a camera cut, which often exceeded 33.3ms itself.

So that we would not miss more than one vsync, we ensured that no frame that contained both a camera cut and an Activate event exceeded 66.6ms. 

Of course, we could further subdivide these DataLayers into smaller DataLayers to reduce the spike-on-Activate.

Preloading DataLayers in Sequencer at least ensured that some of the processing involved in making assets camera-ready (getting the assets “Loaded”, but not yet “Activated” and visible) was amortized over many frames, reducing the spike-on-Activate somewhat.

When you manage the DataLayer manually, the “Streaming” system has nothing to do. We avoided streaming entirely:

Asset guidelines

Nanite meshes

In general, most meshes in the scene should be Nanite meshes:

Meshes we make non-Nanite, such as the sky sphere, are rare.  Such exceptional meshes should generally:

• have large triangles
• not occlude anything
• not be instanced (e.g. there’s only one of them)

Particles

Particles run on the CPU, since CPU usage is so light (typically less than 9ms, with occasional spikes) and GPU usage so heavy during the cinematic.

Avoid Desired Age: When particle artists want an effect to play predictably in Sequencer, NiagaraComponent’s System Life Cycle must never use Desired Age, but instead use Desired Age No Seek.  The latter does not explicitly simulate ~36ms (or a 1-vsync miss) worth of frames, but instead "fast-forwards" to the desired "Age" (time in the particle simulation). 

 For our content the visual results looked identical:

Shaders

We prohibit the following nodes:

* World Position Offset: This is expensive on Snapdragon in general, but is particularly expensive with Nanite and Virtual Shadowmaps

* Vertex Interpolator

* Custom UVs

All of those nodes get evaluated multiple times per invocation. Consequently, they are much more expensive on mobile devices than on the desktop.

Lumen

Set Lights to Movable (not Static or Stationary) to play best with Lumen:

Max Draw Distance set small as feasible:

Avoid Soft Source Radius, as its cost didn’t meaningfully contribute to this content:

Minimize Lumen card count (less than 12) wherever it didn’t hurt visuals:

Minimize shadow casting lights wherever it didn’t hurt visuals:

Textures

Minimize texture sizes whenever it didn't hurt visuals (we still used many 4K textures).

Maximize opaque textures; minimize masked textures.

Transparency:

·         Transparent materials were only applied to non-Nanite meshes

·         Apply no more than 1-2 transparent planes over any given pixel, since alpha-blending can get expensive.

Fixing meshes drawing on Windows x64 but not Android

Bounds Scale sometimes needed to be 2 to make a mesh draw on Android (Windows x64 would always draw all meshes as expected):

We ran with the following CVars to avoid some meshes popping in several frames after a camera cut on Android:

r.AllowOcclusionQueries=0;Hardware occlusion culling off
r.HZBOcclusion=0

Fixing Nanite’s very occasional, bad choice for Level of Detail (LOD)

Nanite is amazing, but in one case its LOD change noticeably popped in a few frames after a camera cut:

The mesh remained in the Nanite pipeline, but we switched off LOD, then loaded and rendered the full-detail mesh immediately with the following settings:

• Fallback Target = Relative Error instead of Auto
• Fallback Relative Error = 0.0 instead of 1.0
• Minimum Residency (Root Geometry) = Full instead of 32kb

That had no measurable impact on performance, and we still had many gigabytes of free memory on the Snapdragon 8 Elite Gen 5.

CVar Settings

We shipped ProjectOne with the CVar settings shown below.

Pipeline State Object (PSO) Compilation
We tried to front-load as much PSO compilation as possible behind the cinematic’s splash screen logos. Nonetheless, the first time the cinematic is played, there are tens of long frames (many of which miss multiple vsync’s) while the rest of the PSOs are built during the playthrough. 

Subsequent playthroughs of the cinematic load all PSOs from cache and miss zero vsync’s due to building PSOs:

r.ShaderPipelineCache.AlwaysGenerateOSCache=0
r.ShaderPipelineCache.Enabled=1
r.ShaderPipelineCache.ExcludePrecachePSO=0
r.ShaderPipelineCache.GameFileMaskEnabled=0
r.ShaderPipelineCache.LazyLoadShadersWhenPSOCacheIsPresent=0
r.ShaderPipelineCache.LogPSO=1
r.ShaderPipelineCache.ReportPSO=1
r.ShaderPipelineCache.SaveBoundPSOLog=1
r.ShaderPipelineCache.SaveUserCache=1
r.ShaderPipelineCache.StartupMode=1
r.ShaderPipelineCacheTools.IncludeComputePSODuringCook=1

Save up to 0.91x in power consumption by avoiding hardware mesh shaders
By using r.Nanite.MeshShaderRasterization=0 you can reduce Lumen and Nanite power consumption.

Performance Optimizations

These settings are what kept our long frames after camera cuts under 66.6ms and the rest of our frames under 33.3ms, without meaningfully compromising our artists’ vision.

When Unreal reported increasing the size of a cache (an expensive reallocation event like GrowPoolAllocation typically misses 1 or more vsyncs), we increased the size of the cache until dynamic allocations were no longer needed.  (That includes CVars like r.Nanite.Streaming.StreamingPoolSize and r.Shadow.Virtual.MaxPhysicalPages).  Even using these large caches left us with many gigabytes of free memory on the Snapdragon 8 Elite Gen 5.

We also reduced computation anytime our artists felt the reduction in fidelity was negligible.  (This includes values like r.LumenScene.Radiosity.ProbeSpacing and r.LumenScene.DirectLighting.MaxLightsPerTile):

landscape.SupportGPUCulling:1
r.AllowGlobalClipPlane:0
r.AllowStaticLighting:0
r.AmbientOcclusionLevels:-1
r.AmbientOcclusionMaxQuality:100
r.AmbientOcclusionMipLevelFactor:0.4
r.AmbientOcclusionRadiusScale:1.0
r.AntiAliasingMethod:2
r.AOGlobalDistanceField.MinMeshSDFRadius:40
r.AOQuality:1
r.CapsuleShadows:1
r.ClearCoatNormal:0
r.ContactShadows:0
r.DefaultBackBufferPixelFormat:4
r.DefaultFeature.AmbientOcclusion:0
r.DistanceFieldAO:1
r.DistanceFields.MaxObjectBoundingRadius:25000
r.DistanceFields.MaxPerMeshResolution:64
r.DistanceFields.ParallelUpdate:1
r.DistanceFields:1
r.DistanceFieldShadowing:0
r.DynamicGlobalIlluminationMethod:0
r.GBufferFormat:1
r.GenerateMeshDistanceFields:1
r.HighResScreenshotDelay:8
r.LightMaxDrawDistanceScale:0.5
r.LightShaftQuality:1
r.Lumen.DiffuseIndirect.Allow:1
r.Lumen.HardwareRayTracing:1
r.Lumen.Reflections.Allow:1
r.Lumen.Reflections.DownsampleFactor:16
r.Lumen.Reflections.HierarchicalScreenTraces.MaxIterations:15
r.Lumen.Reflections.MaxRoughnessToTraceForFoliage:0.2
r.Lumen.Reflections.RadianceCache:1
r.Lumen.Reflections.ScreenSpaceReconstruction.TonemapStrength:1
r.Lumen.Reflections.Temporal.MaxFramesAccumulated:16
r.Lumen.Reflections.Temporal.MaxRayDirections:4
r.Lumen.ScreenProbeGather.DownsampleFactor:64
r.Lumen.ScreenProbeGather.FullResolutionJitterWidth:1
r.Lumen.ScreenProbeGather.IntegrationTileClassification:1
r.Lumen.ScreenProbeGather.IrradianceFormat:1
r.Lumen.ScreenProbeGather.RadianceCache.ProbeResolution:8
r.Lumen.ScreenProbeGather.ScreenTraces.HZBTraversal.FullResDepth:0
r.Lumen.ScreenProbeGather.ShortRangeAO.HardwareRayTracing:0
r.Lumen.ScreenProbeGather.SpatialFilterNumPasses:2
r.Lumen.ScreenProbeGather.StochasticInterpolation:1
r.Lumen.ScreenProbeGather.Temporal.MaxFramesAccumulated:10
r.Lumen.ScreenProbeGather.TwoSidedFoliageBackfaceDiffuse:0
r.Lumen.TraceMeshSDFs:0
r.Lumen.TranslucencyReflections.FrontLayer.Allow:0
r.Lumen.TranslucencyReflections.FrontLayer.Enable:0
r.Lumen.TranslucencyReflections.FrontLayer.EnableForProject:0
r.Lumen.TranslucencyVolume.GridPixelSize:128
r.Lumen.TranslucencyVolume.RadianceCache.NumProbesToTraceBudget:7
r.Lumen.TranslucencyVolume.RadianceCache.ProbeResolution:2
r.Lumen.TranslucencyVolume.TraceFromVolume:0
r.Lumen.TranslucencyVolume.TracingOctahedronResolution:3
r.LumenScene.DirectLighting.MaxLightsPerTile:2
r.LumenScene.DirectLighting.UpdateFactor:64
r.LumenScene.Radiosity.HemisphereProbeResolution:2
r.LumenScene.Radiosity.ProbeSpacing:64
r.LumenScene.Radiosity.Temporal.MaxFramesAccumulated:4
r.LumenScene.Radiosity.UpdateFactor:512
r.Mobile.AntiAliasing:2
r.Mobile.ShadingPath:1
r.Mobile.VirtualTextures:0
r.MobileHDR:1
r.MotionBlur.HalfResGather:0
r.MotionBlurQuality:4
r.Nanite.Streaming.StreamingPoolSize:1024
r.NeverOcclusionTestDistance:500
r.NormalMapsForStaticLighting:0
r.PostProcessing.PropagateAlpha:1
r.PSOPrecache.CustomDepth:0
r.PSOPrecaching:1
r.RayTracing.Shadows:1
r.RayTracing:0
r.ReflectionCaptureResolution:2048
r.ReflectionMethod:2
r.SeparateTranslucency:0
r.ShaderPipelineCache.ExcludePrecachePSO:0
r.Shadow.CachedShadowsCastFromMovablePrimitives:0
r.Shadow.CSM.MaxCascades:2
r.Shadow.CSM.TransitionScale:0.25
r.Shadow.CSM.TransitionScale:1.0
r.Shadow.CSMCaching:1
r.Shadow.CSMShadowDistanceFadeoutMultiplier:2.5
r.Shadow.DistanceScale:1.0
r.Shadow.MaxCSMResolution:2048
r.Shadow.MaxNumPointShadowCacheUpdatesPerFrame:1
r.Shadow.MaxNumSpotShadowCacheUpdatesPerFrame:1
r.Shadow.MaxResolution:2048
r.Shadow.PreShadowResolutionFactor:0.5
r.Shadow.RadiusThreshold:0.06
r.Shadow.Virtual.Enable:1
r.Shadow.Virtual.MaxPhysicalPages:8192
r.Shadow.Virtual.ResolutionLodBiasDirectional:0.0
r.Shadow.Virtual.ResolutionLodBiasDirectionalMoving:0.0
r.Shadow.Virtual.ResolutionLodBiasLocal:0.0
r.Shadow.Virtual.ResolutionLodBiasLocalMoving:2.0
r.Shadow.Virtual.SMRT.RayCountDirectional:4
r.Shadow.Virtual.SMRT.RayCountLocal:2
r.Shadow.Virtual.SMRT.SamplesPerRayDirectional:2
r.Shadow.Virtual.SMRT.SamplesPerRayLocal:2
r.Shadow.Virtual.SMRT.TexelDitherScaleLocal:4
r.Shadow.WholeSceneShadowCacheMb:40
r.ShadowQuality:2
r.SkinCache.CompileShaders:1
r.SkyLight.RealTimeReflectionCapture:1
r.SkylightIntensityMultiplier:1.0
r.SSR.HalfResSceneColor:0
r.SSR.Quality:0
r.SSS.Scale:0
r.SupportSkyAtmosphereAffectsHeightFog:1
r.TemporalAA.Mobile.UseCompute:0
r.ViewDistanceScale:0.70
r.VirtualTexturedLightmaps:0
r.VirtualTextures:1
r.Visibility.DynamicMeshElements.Parallel:0
r.VolumetricFog.GridPixelSize:16
r.VolumetricFog.GridSizeZ:64
r.VolumetricFog.HistoryMissSupersampleCount:4
r.VolumetricFog:0
r.VSync:1
r.VT.EnableAutoImport:0
r.VT.PoolAutoGrow:1
r.Vulkan.Depth24Bit:0
sg.ViewDistanceQuality:0

Note that while the above CVar’s switch off Lumen, we use a PostProcessVolume to give the artists control of which features to enable:

Summary

To summarize, here are the practices we recommend if you’re developing with Unreal Engine 5.4 and running on Snapdragon 8 Elite Gen 5:

Project settings

• Custom code for Nanite
• Avoid HLOD
• Don’t use editor-cooked content
• Change maximum supported aspect ratio
• Package with ASTC

Profiling tools

• stat fps
• UnrealInsights with GPU events on Android
• Snapdragon Profiler

DataLayers

• Batch assets
• Activate/de-activate them in Sequencer

Asset guidelines

• Enable Nanite meshes
• Avoid Desired Age for particles
• Prohibit World Position Offset, Vertex Interpolator and custom UVs for shaders
• Change Lumen settings
• Minimize texture sizes and masked textures, maximize opaque textures
• Adjust transparencies

Fix meshes drawing (Windows x64, not Android)

Change Nanite settings for level of detail

CVar settings

• Change PSO compilation settings
• Avoid hardware mesh shaders
• Optimize settings for performance, as specified above

Next steps

We spent a couple of person-weeks optimizing ProjectOne. That’s a big chunk of time you’ll save when you apply the same project settings, profiling tools and asset guidelines to your own Unreal Engine 5.4 projects.

Join our Developer Discord and let us know what you think!