editor: Improve filesystem dock and import performance

[stuartcarnie]

Summary

Helps #123948

In the MPR with 80,000 PNG files, importing the assets kept the main thread busy and made the editor unresponsive. When the import finished, the editor then generated a thumbnail for every PNG file and redrew the FileSystem dock many times per frame. This PR is the culmination of several paper cuts in a few areas found by profiling with Instruments.

The main hotspots, as identified in Instruments, were the following:

• DirAccessUnix constructor: repeated the same work twice.
• EditorFileSystem::_find_file: created a DirAccess object and allocated strings on every call.
• EditorFileSystem::reimport_files: the main thread spun in a tight loop while worker threads imported files.
• FileSystem dock: generated thumbnails for files in collapsed folders, and redrew the whole tree for each thumbnail, generating 10s of 1,000s of redraw requests
• DirAccessMacOS::is_hidden: used NSURL for every directory entry, which was slow.

Here is a video showing opening the MRP from #123948 with 80,000 PNG files for the first time, such that it imports all the .png files, creating associated metadata, .md5 hash files and .ctex compressed texture equivalents.

4.8 dev 7 official	This PR
FullImport4.8-dev7.mp4	FullImport-optimised.mp4

Comment	4.8 dev 7 official	This PR
Import complete from startup	43s	25s
Expand sprites node in File System dock	Hangs for 34s	Instant
Worker Threads total CPU time (import processing)	6.22 minutes	1.22 minutes
Main thread CPU time	1.24 minutes	2.46 seconds

The last two CPU usages are particularly important, as they show how much more efficient the code is. My machine has 16 cores. Those with fewer cores will see even greater improvements to wall times.

Speeding Up Large Imports

By default, Godot enables safe backup / rename when writing files:

It's possible on slower drives that this might introduce additional overhead during a large initial import. I would say for most SSDs today, it's probably not much of an issue.

This feature is enabled by default, which means writing a file will

• create a temporary file, and perform a number of additional kernel calls to prepare the temporary file.
• writes data to the file
• Close the file, which in turn closes the temporary file and then renames it to the real path.

When importing 80,000 PNGs, 3 files are written for every .png:

path	comment
<png_file_name>.import	contains all the import properties
.godot/imports/<png_file_name>.ctex	the compressed texture if the imported image file
.godot/imports/<png_file_name>.md5	two MD5 hashes of the source and dest images

DirAccessUnix constructor

DirAccess::create() is called constantly during scans and imports, and in the original profile the DirAccessUnix constructor used 16.9 s of CPU time. The constructor read the current directory with getcwd(), then called change_dir() with that same directory. change_dir() called getcwd() again and then called chdir() twice, ending with current_dir holding the value it already had. I wonder if the call was left over from #11149, which in 2017 changed change_dir() to stop resolving the path with getcwd() and moved that step into the constructor. After that change the call no longer did anything.

The fix was to remove the redundant call to change_dir() call from the constructor.

Result

The constructor's CPU time dropped from 16.9 s to 6.1 s. After the _find_file change, which stops creating a DirAccess on every call (as it wasn't necessary), the constructor used 0.01 s.

EditorFileSystem::_find_file

This function finds a file in the editor's in-memory file tree, and the import threads call it frequently. In the original profile it used 46.6 s of CPU time. It had three primary hot spots:

• It created a DirAccess on every call, which paid the constructor cost above. The object was only needed when a folder was missing from the tree, which I'll assume is rare. Also, it checked a res:// path with an ACCESS_FILESYSTEM object, which cannot read res:// paths, so that check always failed. The check was added in Fix removing a folder that contains a file is not removed from the FileSystem Dock #94435.
• to_lower() is called on every folder and file name. Each call allocated a new string.
• It split the path into a Vector<String>, allocating more
• always called localize_path(), which unconditionally allocates several strings, yet the localized path was almost always the same as the original, wasting CPU and memory allocations

Fixes

• Create a DirAccess only when a folder is missing from the tree, through DirAccess::dir_exists_absolute(). This picks the right access type for res:// paths, so the check now works.
• Compare names character by character, with no allocation. A new helper, _span_equals, compares two Span<char32_t> values. When case matters it uses the existing Span comparison. When case does not matter it converts only the characters that differ, with String::char_lowercase().
• Make String::char_lowercase() an inline function that handles ASCII characters directly. Other characters still use the lookup table in ustring.cpp, through the new private function _char_lowercase_table(). The result is the same as before for every character. Other callers of char_lowercase() also get the faster ASCII path.
• Walk the path in place using Span<char32_t>, and skip localize_path() when the path is already a simple res:// path.

Result

_find_file CPU time dropped from 46.6 s to 0.35 s, about 130 times less.

Caution

This function has a data race under specific conditions, as it is called from multiple threads. Normally _find_file doesn't mutate anything, but if the directory doesn't exist, it mutates the file system tree without protecting it with a synchronisation primitive like an RWLock.

EditorFileSystem::reimport_files

While worker threads import files, the main thread spins in a loop that keeps the progress dialog updated. On every pass it called get_file(), which allocates a new string, and ProgressDialog::task_step(). task_step() returns early unless 200 ms have passed since the last update, so almost every iteration did nothing useful. The loop was just spinning and consuming the main thread. The loop also showed the wrong file name for batches that did not start at index 0, because it used reimport_files[imported_count] instead of reimport_files[from + imported_count]. That has been the case since #98385.

Fixes

• gets the file name once per imported file, not once per pass,
• sleeps for 1 ms when no file has finished.
• uses reimport_files[from + imported_count], so it shows the right file name.

Result

The main-thread CPU time during the import dropped from 42.0 s to 6.8 s, and ProgressDialog::task_step() from 7.4 s to 0.2 s. The import finished in 29.4 s instead of 42.1 s

FileSystem dock thumbnails

Once the import has completed, the editor stayed slow / unresponsive while it generated thumbnails, and most main-thread time went to redrawing the FileSystem dock tree. In tree-only mode, the dock creates a tree item for every file in the project, including files in collapsed folders, and requested a thumbnail for each one. When each thumbnail arrived, the dock called set_icon(), which makes the tree redraw. The editor runs the deferred call queue once per physics step and twice per frame, so the tree was redrawing / invalidating many times per frame.

Fix

• only requests thumbnails for files whose folders are all expanded. When a folder is expanded, it requests thumbnails for the files that are now visible.
• stores finished thumbnails in a list and applies them all at once, once per frame, in NOTIFICATION_INTERNAL_PROCESS. The tree redraws at most once per frame because of thumbnails.

The following shows navigating the tree right after the import has completed:

before	after
expand-tree-before.mp4	expand-tree-after.mp4

DirAccessMacOS::is_hidden

is_hidden() is called for every entry in every directory the editor lists. On macOS it built a full path, created an NSURL and asked it for NSURLIsHiddenKey. This was added in #42381 so the file dialog hides the same folders as Finder, such as /usr and ~/Library. This is a really expensive call:

Fix

Use getattrlistat(). An item is hidden if its name starts with a dot or it has the UF_HIDDEN flag. This is the same rule NSURLIsHiddenKey uses, and the names come from the file system, so . and .. give the same result as before. I verified this by writing a small program to compare 1,462 directory entries, and it gave the same answer as NSURLIsHiddenKey.

Result

is_hidden() main-thread CPU time dropped from 1.25 s to 0.33 s, about 3.8 times less, and directory scanning (_scan_new_dir()) from 1.39 s to 0.48 s.

Note

This doesn't account for the additional CPU time spent releasing the temporary objects (NSURL, NSString, etc) that are handled by the NSAutoReleasePool after each event loop iteration

[Maran23]

I think you mixed up the two expanded-tree videos:

By default, Godot enables safe backup / rename when writing files:

Thinking out loud, does this make sense for the .import files as well?

[stuartcarnie]

@Maran23 yes, thanks for spotting that 🤦🏻 – I've fixed it

[Ivorforce]

Core changes are fine - I actually used the same optimization for #99971.
Can't say anything about the rest of the code.

[NoNormalDev]

I've seen this PR: #116464, which seems to slightly conflict with this one? It may be good to resolve that one first, since it seems simpler. But it's a draft, so I'm not sure that anything can be done.

Not saying that PR replaces this one, it addresses something different.