14 Apr Discussing alternative memory management strategy for .NET
Discussing alternative memory management strategy for .NET
Abstract: In this article, we are discussing problems that relying on Garbage Collector as a memory management strategy brings. Then we discuss possible alternative solutions, including Microsoft’s own research of introducing the Unmanaged Heap to .NET framework.
Academic discussion on abstract language features
First of all, it is all right to discuss possible changes or enhancements to any programming language or framework, including C# and .NET. While C# and .NET are products of Microsoft and it is under the responsibility and authority of some Product Managers there to decide in which direction C# and .NET will go, we are free to discuss it from a Computer Science perspective, looking at it as only one realization of abstract programing language concepts. So, when we are talking here about C# we are thinking of some “abstract C#-like language” and we do not care if discussed features will or will not be included in a particular product.
When I talk about “.NET framework” in the text below, I mean framework in a generic sense, including all family members like “.NET framework”, “.NET Core”, “.NET”, and “Mono” etc.
Software Engineering is an engineering discipline, that gets inspiration and drive from real-world engineering problems, discusses them, and tries to improve and solve them on an abstract level, to be resolved in some new generation of products. The principles discussed here mostly will apply to Java language or can be applied to some future, still non-existing computer language or framework. So, please see this as an academic discussion on the topic.
Garbage Collector skepticism
Even when I first learned about Garbage Collector technology, which was in Java language, I was already a “GC skeptic”.
It is true, it resolved in a start a huge problem that we had in some C/C++ products with memory leaks, which I was hunting for days, with all possible tools like then Purify  and BoundChecker .
But, as a trained mathematician, I immediately sensed that in Garbage Collection there is a similarity with the “Graph traversal problem” , and the complexity of the traversal can be significant. If you have one industrious thread creating graphs of objects (that need to be traversed) and another thread that is traversing created graphs of objects (for the purpose of Garbage Collection), it can easily happen that if the first thread needs 1 unit of time to create some complex graph, that the second thread might need 10 units of time to traverse it, for the purpose of Garbage Collection. So, it was obvious from the beginning that Garbage Collection technology will have some limitations.
But the truth is threads are not creating new object graphs all the time, they are doing other things too. So, Garbage Collection technology is here, it is usable, it is solving a huge problem with memory leaks, but let us see how practically usable is for a wide range of applications.
Garbage Collector in .NET – mature phase
.NET Framework/.NET Core is now 20 years old technology. From its first release, it included Garbage Collector technology. I think we can freely say that GC technology is in its zenith. Over years, I am sure, many scientists and engineers were working on developing and improving GC technology. It is highly unlikely that in the years ahead, in near future there will be some fundamental break-thru that will significantly improve GC technology. There might be some tweaking of heuristics rules, to accommodate ever-improving hardware, more CPU power, and more RAM, but what we have now regarding GC technology is what we are locked with for years to come.
Garbage Collector in .NET – generational and heuristic
In  it has been outlined the architecture of .NET Garbage Collector. Here we will just mention that the best solution engineers were able to come over the years is to make it “generational”, which by itself is already a heuristic approach.
The reason why they made GC generational, is because “full garbage collection is simply too expensive to do every time” (). There is only a “common sense” argument that objects “that survive one GC cycle” need to be less frequently checked cleanup and can be moved to one Generation above. There is no scientific/mathematical algorithm to support that. It would be so easy to create a program that does exactly the opposite. Also, dividing “objects up into small and large objects” () with 85KB limit and keeping them in a large object heap (LOH) is again just another heuristic. The reason they are doing that is hopefully to faster compact the memory or compact memory less often.
The reason why I emphasize that many elements of GC architecture are heuristics, that is not algorithmically based/provable is because of consequences. Consequences are that the heuristics GC strategy might and will fail in some cases, meaning for some C#/.NET programs GC will have poor performance.
GC Pauses and Latency
The latest version of GC has 2 types of collection: 1) Foreground; and 2) background (, ). Foreground GC is applied to Gen0 and Gen1, while Background is applied only to Gen2. When foreground garbage collections occur, all managed threads are suspended. Background GC enables managed threads to continue operations during a garbage collection. For many applications it is bad news that periodically all its threads will be frozen, so GC can do its work. The question is of course, how often and for how long application threads will be suspended. The practical result of the usage of GC in your application is Latency, which is unavoidable.
It is possible to “tweak” GC, to adopt it to your application. You can choose the “Low latency” mode (), where a collection of Gen2 is suppressed/limited, while a collection of Gen0 and Gen1 still continues. Also, there is a number of configuration options () that enable you to influence how GC will work. But I think it is unrealistically to expect that programmers will spend time tweaking for each application GC parameters. A serious approach to tweaking/configuring GC would, of course, include some test-trial-correction work using some advanced memory monitoring tools, if you have time for that.
Having said all that, the .NET GC technology is the best many engineers were able to come up with, after 20 years of research.
Writing “GC friendly” programs
With the arrival of Garbage Collector technology, there has been a huge movement requesting programmers to change their style and write their programs, so they “cooperate” with GC, or better to say are “GC friendly”. I was a bit skeptical because GC technology was advertised as making things easier for programmers and working transparently in the background. Now, you suddenly ask programmers to put additional effort to write programs in a style that will “accommodate” GC. On one hand, you freed programmers from the need to think about the deallocation of memory, but on other hand but you are giving them another task, supposedly easier, to apply patterns that will accommodate Garbage Collection.
Let’s have a look at some of the recommendations.
Microsoft’s GC Performance Hints – unrealistic
In  there is a list of Microsoft’s own recommendations on how to program in presence of GC. They list, to quote them: “sorts of things we should try to avoid to get the best performance out of the collector”
- (avoid) Too Many Allocations – That is almost funny advice. Nobody is allocating objects just for fun. Programmers allocate variables/objects because program logic needs it, and in a serious programs, there will be a lot of them. It is almost, as they say, “do not use memory too much, so our nice GC doesn’t need to work hard”.
- (avoid) Too-Large Allocations – Again, ridiculous advice. I have a laptop with 32GB RAM, and you are telling me not to create big objects. I guess the problem for GC is compacting Heap and moving around big objects is time-consuming. But what is the point of all that memory if I do not use it intensively?
- (avoid) Too Many Pointers – I see that many pointers/references mean complex graph objects, and traversing complex graphs is time-consuming. But if your business logic is scientific/engineering your object graphs might get very complex. It is unrealistic advice.
- (avoid) Too Many Object Writes – Again, they are implying what your program logic/ business domain implied algorithm should look like. If you need to change object states often, that is it. Or the program will not do its primary purpose.
- (avoid) Too Many Almost-Long-Life Objects – Again, I have 32GB of memory and don’t want to be told not to keep many objects in my memory, if my program logic needs that.
- (avoid) Finalization – That is a language feature and is there for a good purpose. If it is there, why limit its use.
To look at the above advice from a funny side, it is almost as they say: “Do not create too complex, too complicated, and too big programs, so our nice GC can work fast”. But seriously, programmers have many other things on their minds when solving problems, so additional requests “to be nice to GC” that would influence their program logic/algorithms are not welcome at all.
Advice to reuse objects
There have been a lot of recommendations to “reuse” allocated objects, in order to avoid re-allocation. There are even patterns for that, like the “Object pool pattern” . First of all, it again puts the burden on programmers to change the logic of their programs and create additional work for them, like resetting the object’s state, pooling objects, etc. Sometimes, it is not realistic, you just need to create new objects and that is it.
Roslyn Coding Conventions
In  are some recommendations for the Rosalyn project: “Generally the performance gains within Roslyn come down to one thing: Ensuring the garbage collector does the least possible amount of work”. So, it implicitly recognizes that GC latency is the major problem related to performance.
Listed recommendations are:
- Avoid LINQ.
- Avoid using foreach over collections that do not have a struct enumerator.
- Consider using an object pool. There are many usages of object pools in the compiler to see an example.
Again, it puts limitations on the programmer which techniques/technologies to use, and all that just to accommodate GC and minimize its latency. Also, avoiding objects/structures on Heap and using only objects/structures on Stack suggests there is something wrong with the solution of Managed Heap.
Practical problems with GC in the .NET framework
Here we will list some examples of problems related to the usage of GC technology
Latency in high scale .NET applications
In  there is one nice example of as they say, “battle with .NET GC”. I will just list here author’s conclusions:
- The remaining delays these days are mostly GC-related, however, they are way shorter than they used to be.
- If you face similar problems in a high scale .NET application ….. you could rewrite components in C++
The last one advice, using C++ language to rewrite parts of code, a bit questions the suitability of .NET GC for large scale applications.
GC Pauses and Latency
It has been widely recognized and measured () that GC brings significant latency to applications caused by periods when application threads are suspended so GC can do its work.
The present state of GC in .NET framework
Here we will be critical of GC technology as part of .NET framework. For years GC technology was a “holy cow” of .NET framework and nobody would say a bad thing about it. This is not really a “Down with GC” kind of text, but more like “GC is fine, but not always”. Here is my humble opinion.
- GC technology worked fine and adequately for many C# applications, especially small and middle-size complexity/size projects
- But GC technology for many C# applications, especially high complexity/large size created many problems with latency and failed to be an effective memory management strategy
- By requesting programmers to write “GC friendly code” responsibility and burden of work were pushed from .NET framework to individual programmers. In a way, it was unfair because now programmers were made responsible for the limitations of GC technology and the choice to use it in .NET framework, which was imposed on them. Even worse, even if they follow all those recommendations to cooperate with GC, there are still no guarantees that will succeed in avoiding GC-induced latency, etc. (see ). Also, let us remember, that GC main purpose was to simplify and make easier work for programmers, not to create additional tasks that now replace original tasks of simply cleaning/deallocating memory of used variables/objects. Many programmers are still capable of deallocating memory by themselves, given the appropriate API in the language of their choice.
- Do not get hypnotized by talk about GC technology, like “Generation 0, Large-object-heap, etc”. It is definitely a great and clever technical solution, but after 20 years of development it has reached its zenith and we can freely say it has serious limitations and is not a good solution for every case and every application scenario. I personally like to use it and value it but would like to have sometimes a choice of using something else.
- Problem is that .NET framework is offering GC as the ultimate and only memory management solution. Programmers using language as C# do not have the choice to decide for example to use Unmanaged Heap where they would take over responsibility for deallocating memory on themselves, for which they might be completely competent. (here we are excluding usage of APIs like AllocHGlobal and similar since they are not integrated into C# language in a sense that you can easily allocate object with something like NewUnmanagedHeap(..type..) ).
- The choice that .NET framework made by locking itself to only GC as a memory management solution, which has serious limitations, made it inherit those limitations. The practical result is that .NET framework, because of its limitations, might not be the best language/framework of choice for every kind of application scenario. That is a serious problem for .NET framework, which has the ambition to be “the best and fastest programming framework … in the world?”
- The big hurdle is that many libraries and application frameworks for C#/.NET developed over years are based on GC technology. Even if we would add to C# language something like NewUnmanagedHeap(..type..) / DeleteUnmangedHeap(..object..), which you would use in your main app body, you are still going to be affected by GC pauses created by allocated objects used in your libraries.
- C#/.NET framework would benefit if it would be able to offer programmers alternative memory management solutions, that would enable them to overcome the limitations of GC.
- Programmers are a very clever group of people. GC technology makes them lazy, but, if need be, they will be able to clean/deallocate memory properly by themselves, given the appropriate API. They were doing it before GC technology, and they are doing that now in other languages that are not GC-based.
Alternative memory management strategy for .NET framework
Roughly, alternative approaches can be divided into 2 groups:
- One Heap solutions – having some kind of Managed-Heap-Enhanced solution
- Two Heap solutions – Having separate Managed and Unmanaged Heap
In 1) the idea is to have one GC Managed Heap that is enhanced with some additional API that would enable developers to directly communicate that certain object/allocated memory is no longer needed and in that way would make deallocation more efficient and hopefully GC much faster, so pauses would be rare and shorter. Below discussed GCD (*) is an attempt in that direction.
In 2) the idea is to have two heaps, Managed and Unmanaged Heap and programmers can choose between allocating objects in the garbage collected heap or the manual heap. Typically, the design would allow references to and from both heaps. Below described Snowflake project (***) is such a kind of solution.
NOTE: GCD is found to be unfinished research
Initially, I made a sketch of an algorithm for a solution called GCD listed below at (*) and put it on public discussion. In the comments below  user “Davin On Life” in his review showed that, in the form it is presented, is not as effective as it appeared. So, I am withdrawing it as unfinished research.
If you are not interested in reading about “attempts that have problems”, please jump to (***).
I decided to keep the original text of the proposed GCD solution for two reasons.
- It is good to keep a record of attempts and approaches that were proposed, debated, and found inadequate so that we do not discuss them again. And also, to remember why they were inadequate, so we can improve on new attempts.
- While the first iteration of the proposed GCD solution might have problems, maybe in the next iterations, after more research, we will be able to improve the solution and overcome the problems.
Garbage Collection with Delete (GCD) (*)
Here I will propose one alternative memory management strategy, that after short thinking on the subject come to my mind. Again, it is an abstract concept that can be applied to any language, including Java and .NET framework. But since I am Software Engineer who is lately doing work mostly in .NET framework environment, and am driving my inspiration from that environment, I will be mostly talking about how it might be applied there, in that specific C# language environment. Again, that is speculative abstract idea elaboration applied to a concrete engineering problem, and I see it as an intellectual exercise, without implying that I have any authority over the future of .NET framework.
So, basically, I am suggesting adding C++ style delete and destructors in addition to GC to C#. Let us call it in further text “Garbage Collection with Delete” (GCD) solution. Here are the main proposed principles.
- Managed Heap and GC, in their full strength, are applied in the GCD framework. Objects are allocated with “new” on the managed heap.
- Classes of interest are added C++ style destructors, let’s say for class XYZ name it ~~XYZ(). Destructors can be virtual.
- C# language is enhanced with “delete” operator. If applied to the object reference of class XYZ, it will invoke virtual destructor ~~XYZ() (if it exists) and deallocate the object from the Managed Heap. And here I mean momentarily, synchronous call, no “lazy evaluation” or “deferred execution” of any kind.
- Programmers are free to call the delete operator on any object at any time. If an object has a destructor, the destructor is invoked first. If delete on some object reference is never called, the object is subject to a regular GC cycle.
- The big problem is “dangling references”. If there are 2 references to object Obj1, and delete is invoked over one reference of it, what needs to happen to the other reference? I see at this time two approaches: 1) usage of some technology like “Smart pointers” () to not immediately delete the object, or 2) delete the object and report null-reference by another reference, which I know might result in “null-reference exception”. I have doubts here, and do not know the internals of GC well enough to know if it can by itself easily support approach 1), which would be a “safer” solution.
- I will not go here into other details of how destructors work etc., which is well known by every C++ programmer. Inside destructor, you are free to delete other objects that were part of the original object allocation, or you are free not to care and leave them to GC to take care of them.
That is basically it.
Benefits of GCD solution
Here are what I see as benefits of GCD solution.
- First of all, there are no memory leaks, same as in GC solution. Programmers are free to delete the objects if they care, but if not, deallocation is taken care of by regular GC, since all objects are on the Managed Heap as before.
- Programmers have the opportunity, but no obligation to take care of the deallocation of objects. If they intervene, objects will be deallocated immediately, if they choose not to intervene, GC will take care of the objects.
- Programmers have the opportunity to notice and take care of critical allocations. If some loop is allocating the same object 1000 times, one small line of delete will fix the problem and GC will have 1000 pointers less to analyze, so it is expected to be much faster in its work. If a programmer notices some big objects being no longer needed, a couple of deletes will make life easier for GC which will still need to compact the heap.
- If some objects create a complex graph of allocated contained objects, which creates timely work for GC, a smart destructor can disassemble that object and free GC from analyzing complex graphs, making it much faster and GC pauses will hopefully be rare and shorter.
- Programmers have no obligation to write “full” destructors, they can disassemble what is most important, like the biggest objects or most complicated graphs, and leave it to GC to resolve the “small potatoes”.
- With the intervention of programmers in the delete/deallocation process, the list of objects that GC needs to analyze can be shorter and less complex, making GC pauses rare and shorter.
- For demanding, high-performing applications, programmers have the opportunity to practically completely take responsibility for the deallocation of objects with delete/destructors, leaving GC just as a backup strategy, which would make GC cycles and pauses not-needed and non-existing. With appropriate API GC can be put into a “do-not-work-until-emergency” state, making GC pauses and latency non-existing.
- In GCD solution, all benefits of GC are preserved, and in addition, performance can be improved by programmers if they decide to intervene in code.
Programming style in GCD framework
I see 3 main styles of developing programs inside GCD memory management framework:
- Complete reliance on GC. That is the same as all the programmers were doing so far in environments like Java or .NET framework. It might be the appropriate strategy for many applications. It is great for beginner programmers. It is great for prototyping.
- Partial manual deallocation of critical objects. Programmers might decide to intervene in hot spots they notice, manually deallocating critical objects, like big objects, or numerous allocations of small objects. That can with minimal work have a significant impact on performance. They can decide not to invest too much work and leave most of the work to GC.
- Total manual control of deallocation of objects. In high-performance applications, big applications, complex applications, and when a sufficient qualified workforce is available, programmers might decide to take full or near-full control of all deallocations in the application in order to maximize performance. That can be a journey, not a big-bang event, since there is no penalty if they can not solve it all at once, since GC as a backup is always there.
The transition from GC to GCD framework
Here we will as an intellectual exercise discuss what the transition from GC to GCD framework would look like. For specificity, we will be discussing it on the example of the .NET framework. Assuming that GCD is accepted as a memory management strategy.
- First of all, all programs developed in GC framework will continue to work in GCD framework. All libraries will continue to work too. So, we have backward compatibility with the existing code base or C#/.NET.
- For a class XYZ, destructors ~~XYZ() look a lot like finalizes ~XYZ(), but are different in nature. First, destructors need to be virtual. Second, finalizers were tasked with freeing unmanaged resources, while destructors address managed and unmanaged resources. Typically, a destructor would need to invoke a finalizer for the appropriate class and delete explicitly managed resources (objects) it cares to clean manually.
- Regarding IDisposable interface in C#, it looks to me that it was really intended to be a destructor for the object. For backward compatibility, it can be left the way it is, but there will be probably a pattern how using directive/ IDisposable interface can be replaced with delete/destructor.
- Over time, programmers can use new features of GCD and apply delete/destructors to their app’s main body code to improve performance. Also, C#/.NET libraries publishers over time can review their libraries and improve their code with some manual deallocation tweaking. New versions of the library would be usable only in the new .NET framework that supports GCD memory management.
Tools needed to support GCD framework
There are some tools that I think would be needed to support efficient GCD programming. At least tools I would like to see for myself.
I recall that I was intensively using tools in C++ environment to locate memory leaks and trace memory allocation. That were Purify  and BoundChecker , which had some technology of “instrumenting” code and were producing a really nice list of your memory leaks and locations in code where allocations were done. I was using several years ago some C#/.NET memory profiling tools but am not familiar with the latest versions.
What I would like to see are tools that would produce a similar list for GCD environment. The purpose is to support programmers to be able to fine-tweak their application regarding memory allocation/deallocation as fine as they want. What I would like to see is:
- Of course, a list of all objects on Managed Heap that are hanging around and location in code where they were allocated
- But also, I want to see a list of all the objects that were Garbage Collected and the location in the code where they were allocated. I want to see a report like this: the object of class XVZ was allocated 1000 times at line code 1234 and is Garbage Collected 1000 times. That is really in C++ terminology “a memory leak” handled by GC and is a good candidate to be deallocated manually in GCD framework, so GC does not need to take care of it, which will free GC of extra work and make GC pauses shorter.
So, basically, I want to have easy insight into GC work, so I can manually, if I need to, resolve/deallocate objects so GC is not working hard and blocking/pausing applications for a long time. Those familiar with GC architecture will surely be able to point to more useful information that GC can give to programmers, maybe like traversing which graphs took a lot of time, so a clever destructor can be written and applied. The point is to optimize programmers’ work, so they do not write destructors for all classes, but only for those causing problems to GC. I see no other way to speed up GC work, except to leave it less and simpler work to do.
Microsoft experimenting with Unmanaged Heap in .NET (***)
An anonymous reader pointed me to articles describing Microsoft’s own experimentation with the introduction of Unmanaged Heap to the .NET framework, called project “Snowflake” (, . ). A group of researchers even created a fork from .NET code and implemented a solution and tested it. Project Snowflake was serious research project, involving 3 researchers from the world’s top universities and 5 researchers from Microsoft Research. I would estimate that it included maybe a dozen of support staff (IT, HR, project administration, etc.) and lasted 3+ months, with a big budget. That means, if you count all the workhours, that it costs maybe a million dollars or more to get a research paper like .
Here is an outline of that project.
- Authors agree that “GC greatly improves programmer productivity and ensures memory safety.” (), but “…But GC comes with a performance cost …. can be problematic for some.” ()
- In their design, programmers can choose between allocating objects in the garbage collected heap or the manual heap.
- A key design goal is to maintain full GC interoperability, allowing pointers to and from the two heaps.
- In order to deal with the problem of “dangling references”, they developed their own technology/pattern called “Owner and Shield” (Snowflake API). That technology/pattern is similar to hazard-pointers .
- How does a programmer know which objects should be allocated on the manual heap? The methodology is to use a heap profiling tool, to determine if the GC has a significant cost, then identify objects that have survived and have been collected in the older generations and look for candidates.
- Experimental results from their .NET CoreCLR implementation show “up to 3x savings in peak working sets and 2x improvements in runtime” (). That means that some applications will use 3 times less memory and are 2 times faster. They say performance benefits are due to “This is possible because for very large pools of objects the GC spends considerable time walking the object graph to free up memory.” ().
- The research article is dated 2017 and nothing is said about potential integration in the .NET product.
Is it realistic to change such a big framework .NET so significantly?
Well, with the arrival of C#8 and “Nullable reference types” () which are now becoming standard I was surprised with the courage and determination to declare non-natural what we were living with during the years that passed. All this hype was about to change millions of lines of code from expressions like “string s=null” to “string? s=null”. I learned to program on expressions like “void * p=0” so really didn’t see a problem with “A billion-dollar mistake” . But effort spent to change that paradigm regardless of breaking so many lines of existing code showed that big changes to .NET framework are possible. If and when they decide that a new .NET memory management strategy is needed, they showed they can push such a change.
In this article, we were first analyzing the current state of GC technology and the problems it brings. Then we were discussing the problems it brings in the real world of .NET framework (meaning whole family “.NET framework”, “.NET Core”, “.NET”, and “Mono”) applications due to the fact that is the only possible memory management strategy in that framework.
Then, we discussed an alternative solution, which is the usage of Garbage Collection complemented with delete/destructors, which is still unfinished research.
Then we talked about some research Microsoft itself is doing in that field, called the project “Snowflake”.
We believe that rallying on GC as the only memory management strategy is a serious problem for the .NET framework. Some alternatives and improvements would be beneficial, be it something similar to what was proposed here as the GCD framework, Microsoft “Snowflake” project, or some completely different approach. Time will show us what direction the .NET framework will go.