Some Suggestions to Help the Delphi Compiler

[DP News-Robot]

Over the last couple of years, Embarcadero has put a significant effort into optimizing the Delphi Object Pascal compilers' performance and reducing its memory usage, so that our customers with large applications can have a better experience working with Delphi. The primary focus has been on the Win32 compiler, but other compilers have been improved as well.

We made significant progress in many use cases scenarios, but at the same time by looking at many projects from our customers we discovered a number of cases in which the compiler isn't behaving as expected. We plan addressing most of those problems in the near future, even if some of the scenarios are bordering on being invalid (like passing non-existing folders to the compiler). At the same time we are getting requests from customers if there is anything they can do in their project configuration or source code to make things better for their developer today.

This is the reason for this blog post: Offering some tips and suggestions you can consider (if they are applicable to your code base) to help the compiler be faster and consume less memory using the current version of Delphi.

Notice that most of the ideas mentioned below help both with compilation and when using Code Insight powered by Delphi LSP, given it uses the same compiler behind the scenes. By helping the compiler, you are also helping Code Insight.

Invalid Search Path or Library Path Entries

The compiler looks for any unit referenced in the code by uses statements in the project folder, the project search path, the library path, and some other locations. We have noticed that having invalid folders in these paths causes the compiler to repeatedly look into those locations, an operation that is much slower, at the operating system level, for non-existing paths.

You can see an example of a report we received, titled 'Delphi compiler works extremely slowly when several invalid paths are present in LIB PATH' at https://quality.embarcadero.com/browse/RSP-3931, as an example.

We made some tests on small applications and for the following for a test case:

• Compilation time with invalid path: 40.3 seconds
  
  
• Compilation time without invalid path: 1.7 seconds
  
  

As you can see, the difference is huge. The plan is to change the compiler to verify and exclude invalid paths upfront, but it shouldn't be too difficult for developers to clean up that list themselves.

Source Code Files on UNC drives

A related issue is that some file access operations (specifically looking for missing files) are significantly more expensive when compiling a project entirely or with units on a network mapped drive, compared to using a local drive.

Actually, we fully recommend keeping your source code on an local SSD drive: Delphi compilations are quite disk intensive and a fast hard disk can make a huge difference.

Unit Aliases

Unit aliases offer the option to map a unit name in a uses statement to a different real unit name. This can be handy when migrating old code and was used by Delphi over the years, when some of the core units were merged or renamed.

For example, in some early versions of Delphi you'd find the following unit aliases:

WinTypes=Windows;WinProcs=Windows;DbiProcs=BDE;Dbi Types=BDE;DbiErrs=BDE Today, there is no default unit alias defined for new projects, but you might have existing projects with unit aliases. The recommendation is to remove them (possibly one by one) and update the source code of your projects with references to the correct units.

A large use of unit aliases adds some extra work to the compiler. While to our knowledge this causes only a very little extra overhead, it is still a bit of extra time and the effort also helps you keep your code cleaner and more readable.



Unit Scope Names and Unqualified Unit Names

Over ten years ago, Delphi introduced unit scope names, that is a prefix in front of all units that are part of the default libraries, like Vcl.Forms or System.SysUtils. This is covered in https://docwiki.embarcadero.com/RADStudio/Alexandria/en/Unit_Scope_Names

If you have existing projects with short unit names, a fast migration path is to define unit scope names as part of the project options, as you can see here:



As you can see, the Delphi IDE still adds a large number of unit scope names by default even to new projects (this is something we should consider changing, but that's a separate point). Now unqualified unit names in your code cause a file look up for each of the possible prefixes, in all of the folders of your search or library path. Consider having 20 folders and 20 prefixes, that's potentially 400 file locations to look for.

No surprise we have reports like this one, titled 'Unqualified unit names cause massive amounts of file lookups' and logged as https://quality.embarcadero.com/browse/RSP-18130. The compiler has some caching logic to reduce the work, but again, this is only a feature to help code migration, but developers should eventually clean up their uses statement, move to use fully qualified unit names for all units in Embarcadero libraries and remove all unit scope names configurations at the project settings level. This is another code change that can have a significant benefit on the compiler.

Repeated and Duplicate Unit Names

Speaking of unit names, there is another scenario that is known to cause trouble. In this case it is less of a performance issue, but a scenario that can cause failures, particularly in Code Insight and some of our IDE tooling.

The scenario is the following: you have a project group open in the IDE with multiple projects including different units (in different folders) with the same name. In other words, one project has MyUnit.pas in a folder and the other project has a different and unrelated unit also called MyUnit.pas in a different folder.

This is a scenario the compiler generally handles fine, but not always as you see in the report 'Compiler confuses unit names in project group' at https://quality.embarcadero.com/browse/RSP-39293. We did specific work a couple of years ago so that the debugger won't get confused in similar scenarios. However, DelphiLSP can get confused by units with the same name.

Whenever possible, it would be much better to avoid using the same name for different units.

Circular Unit References

Getting back to scenarios that affect the compiler performance, one of the most critical ones we know of is the excessive use of circular unit references in the implementation section of the units. As you know, the interface section of the units does not allow mutual or circular unit references. However, in the implementation section of a unit, you can theoretically refer to all other units in the entire project.

When Delphi compiles a unit, it has to look at all used units and navigate the entire graph, making sure of source to check the cycles -- to avoid an infinite loop while searching. In the case of a complex graph with hundreds of units the cost of this navigation process is significant.

Recently our support team made an interesting experiment. They created a code generator that creates an application in which every unit uses every other unit. This is clearly an extreme use case -- I really hope you don't have projects like this! In such a scenario, adding one more unit almost doubles the compilation time. In other words, there is an exponential degradation of the time to compile when one more unit is added that uses all other units in the project and is used by all other units (it is exponential as every additional unit adds n*2 unit references where n is the number of existing units).

Again, this is a scenario the compiler was already optimized for, moving searches and name lookup to used hash dictionaries and similar optimized data structures, and an area we are actively looking right now to improve to compiler. See for example this report, resolved in Delphi 10.4.2: https://quality.embarcadero.com/browse/RSP-28811

However, the use of circular uses statements is also a use case scenario that goes against any good programming practice, as you end up with a spaghetti graph of used units. Reducing the dependency across units and across classes is a good practice. Having some linear dependencies (low level units used by high level units, UI units using data access units and not vice versa) is acceptable, but having a complex graph is not healthy for the maintainability of your code over time. In addition, it does put some significant stress on the compiler performance and memory usage.

We do have actual reports from Delphi developers with fairly large applications that reorganizing their uses statements to avoid circular unit references cut up to 90% of their compilation time, like ten minutes getting down to one minute and also helped Code Insight become responsive.

In case you want to address this issue, I recommend considering one of the tools below, which can help you determine the current uses units graph of your application and detect cycles:

• GExpert Project Dependencies (see image below): https://gexperts.org/tour/index.html...endencies.html
  
  
• Peganza Pascal Analyzer: https://www.peganza.com/#PAL
  
  
• Delphi Unit Dependency Scanner: https://github.com/norgepaul/DUDS*
  
  

As an aside, notice that the compiler also dislikes when the same units are used in a different sequence (as this affects the dependency graph of units using those units). In other words, if two units both use unit A and unit B, writing "uses A,B;" in both cases is better than having "uses B, A;" in the second one. This is fairly minor, compared to avoiding circular unit references, but still noticeable in a very large application.

Finally, you could also consider removing unnecessary uses statements, one tool offering specific help on this is Delphi Parser:*

https://delphiparser.com/product/rem...ation-edition/

Repeated Generic Types Instances

The last suggestion we want to offer in terms of helping the Delphi compiler is a bit more contentious and something the compiler should be better at, after a few years. However, we know this can make a dramatic difference, in this case both in terms of compilation time and memory usages, avoiding the risk of out-of-memory compiler errors.

Delphi type system (like the classic Turbo Pascal one) is based on type declaration equivalence. In other words, even if two types have the same name and the same structure, they are different if declared in different units. If you assign a variable of type MyUnit1.TMyType to a variable of type MyUnit2.TMyType, you get a compiler error, regardless of the fact the two types have the same identical structure.

The only exception to this rule is for generic types. If you declare a variable of type TList you are also creating an implicit type, TList, based on the generic type TList. You can have multiple type instances declared in different units and all matching the TList signature and the compiler considers them compatible and allows you to assign them one to the other.*

What happens behind the scenes, though, is that the compiler generates a new temporary type of each of the occurrences in different units. So you'll have, say, TList in MyUnt1 (with a temporary type name) and TList in MyUnit2 (with a different temporary name). For each of these types, the compiler creates instances for the generic type but also for each of the generic methods of the generic type. With a large amount of complex generic types (inherited and nested), this can take a significant effort by the compiler.

The solution to this issue is not to stop using generics, but to create shared instances of generic types whenever possible. In other words, rather than using TList is dozens of different units, you can declare*

type TListOfStrings = TList; and use the TListOfStrings type all over the places where you'd use TList. In simple examples, the difference is barely noticeable. In a large application with significant use of generic types, it can make a very significant difference.

Again, we all know that the compiler could handle similar scenarios better, and we have been working to improve this use case -- and we did improve it quite a bit over recent years. Still, you can help the compiler be faster and use less memory by refactoring some of your code in that direction -- that is writing code like the generic type equivalence didn't exist.

Is There More? Possibly, Yes

These are some of the scenarios we have found when examining the Delphi compiler performance that can be partially addressed by changes to the code. I'd particularly recommend removing unit aliases and unqualified units, along with the detection and removal of circular unit references. If you are making significant use of generics, consider also the last suggestion above.

This is recommended in particular if you have a large application and you are experiencing slowness in the compiler: Most of the Delphi developers are quite happy with the compiler performance and won't need to do anything in this respect.

I'm also quite certain I didn't list all of the scenarios that cause a compiler slow down that can be partially addressed by code or configuration changes. If you have additional suggestions, based on your experience, they are more than welcome!

PS. Have a nice holidays season and all the best for 2023.
</br>
</br>*



Weiterlesen...