^{Author: Andres Koch, April 2024}

Abstract

Over the last three decades big code bases have been accumulated. After 5 to 10 years maintenance of these code bases they become more and more a challenge for every CTO and director of development. Code complexity and clarity do not get better, pessimistically speaking it becomes worse and the code bases are exploding in size and complexity. To improve or at least understand these code bases this cannot conducted by a manual process executed by humans. Automation of code analysis and reengineering becomes more significant than ever, even though it has been omnipresent for long. This essay tries to focus on the methods and technology to analyse code bases and what option one has to reengineer these using semi-automated mechanisms. With new AI-based technologies there is also a new vision to combine structured and static analysis with machine learning methods, which can be very promising.

Introduction

The problem of highly complex and huge code bases is mainly home-made. Although CI/CD is implemented in almost every significant development group, despite this code does not get better. Due to sprint-based development, design and architecture become underrated and the code first philosophy becomes more significant. The result: code complexity and clarity do not get better, pessimistically speaking it becomes worse. With armies of development personal with good integrated development environments in action, the resulting code bases are exploding in size and complexity. To improve or at least understand these in order to rectify it, this cannot be achieved anymore by manual processes executed by humans. Automation of code analysis and reengineering becomes more significant than ever, even though it has been relevant for the last two or more decades.

The following aspects of software analysis can be considered:

• Code structure
• Dependencies of code, libraries and other artefacts
• Code quality
• Error detection and prevention
• Business rules evaluation
• Performance optimization
• Module boundary detection
• Analysis of security issues
• Analysis of system interprocess communication

Source Code

“The truth is in the code” is often stated to indicate that documentation is not really needed or cannot be 100% accurate. It occurs that the design has lesser value in these days, but that "design by coding" prevails. Independent of any code centric notion it is true that the code bases in general contain a lot of valuable information about the software running on a system. The software contains the instructions which after compilation command the underlying computer to react on input data, processing these and generating new or modified data. And this has lots of truth in it. The truth is also called meta data.

Code as the Pool of Knowledge

The quality and construction of the code very much depends on the culture of the developers who are designing and developing the code. It also depends on the policies of the development department or any standardization body followed on how to construct and organize code. Thirty to forty years ago computer resources available were limited and thus some strange design or coding decision were introduced. Nowadays the computer resources are almost unbound but moderate skills and discipline of writing good code can often result in code anomalies. To "repair" these anomalies or to refactor (restructure) the code to newer coding standards, actions are needed on a periodical basis. One of the purposes is, fitting the code in such a way that it can be expanded, changed and maintained for the next period of time. In order to do this, there are these choices:

• to refactor the code
• to redevelop parts or the whole of the existing code in the same or another programming language
• to modularize the code into logical components

If a system is intended to run for several decades, the replacement of the whole code is rather uneconomic and bears tremendous risks. To replace just part of it is usually possible and it can be done at a rather low risk. Refactoring the code to adapt it to newer language features, newer API of libraries or new coding standards is a constant maintenance process. This should be a periodic action either in parallel or along with the current development or maintenance process. It can also be sensible to refactor the code before a total re-design and re-engineering based on existing code is done.

Code Analysis

Existing code, which was constructed one or more decades ago, often needs refactoring but also a redesign and frequently reengineering. In such code-bases it is very likely that the knowledge about the code has faded away, documentation is not up to date or missing and human memory is either lost by time or by relevant people who left the organization. In such cases an in-depth analysis of the code helps to recover these details. Such an analysis is usually done in various ways:

• Static code analysis: This is usually just analysing the code without knowledge about the runtime behaviour. Code metrics, dependencies and code anomalies like clones, endless loops, dead code and other violations of coding rules and best practices (often given by local development policies or coming along with analysis tools e.g., SonarQube) can be retrieved by static code analysis. The predominant integrated development environments (IDE) usually support instant static code analysis built into the tool, which informs the developer immediately about the anomaly just coded. This can also be done in the build-pipelines of the software by reporting it in similar ways like failed tests. But one should not forget that other artefact e.g. configuration-artefacts can be relevant too.
• Dynamic code analysis: By injecting technical code artefacts into the existing code or by built-in logging, or by extracting runtime information from a virtual container (e.g., Java) and other ways, the runtime paths and lots of interesting statistics can be retrieved. In distributed environments the communication traffic between the applications components is a good fund of behaviour data too. Dynamic code analysis is often used by performance measuring tools to show in which part of the program most of the cpu time is getting lost. These analysis data of such tools can be used for code analysis too, e.g., for detecting "unused" or seldom used code.
• Multilevel Environment Analysis: The sole truth of knowledge about the behaviour of larger systems is not restricted to code but it has to be retrieved from other artefacts as well. These are configuration data, meta data in data bases (e.g., 4GL, SAP systems) and central repositories with a remote access, just to list some. This information has to be inspected and handled on various level of abstractions to get the full picture. It has often to be repeated in multiple runs, where the analysis tool has to be enhanced with other aspects. One of the most important aspects should be considered, that the business domain can be retrieved in such a way, that the code analysed can be assigned to a specific business domains or subdomains. Only this allows to conduct reengineering in an appropriate way that it is of value for the overall business.

Code Analysis in More Detail

The practical application of code analysis is usually not a one-time action but more an iterative evaluation process. In this process the knowledge gained, influence the methods and tools to evaluate further. In most of the cases, tools cover just a limited number of aspects of a complete analysis process. On the other hand, one wants to analyse only as long until enough information is gathered in order to do the reengineering work. Independent of the method used, it should be considered that the more precise the analysis results should be the more effort has to be invested. But often the Pareto-approach is sufficient to see the major aspects of the system. The methods used are listed below:

• Abstract Syntax Trees (AST): Mainly relevant to source code but only possible if a parser for the language is available. It results in a technical structure of the programs.
• Lexical Analysis and Parsing: Full or partial parsing of the code base is not always possible as sometimes older language versions or dialects were used. This applies for programming languages as well as for SQL-Definitions of databases. When DSLs (Domain specific language) were used, parsers are often missing or not usable for the analysis tools. Then these parsers have to be written. When parsers become to complex (e.g., PL/1) or languages are irregular and not context-free it can make sense to extract the desired data by lexical analysers only using regular expressions to scan code. Note that this approach can be tedious too which let one fall back onto using a parser.
• Rule-Based Analysis: This approach defines rules to evaluate certain meta data from the code. This is often done by using lexical text analysis to evaluate certain programming patterns.
• Data Flow Analysis: This is focused on analysing the access of databases and calls to or from other systems. As both are involving interfaces this can be achieved with moderate effort.
• Control Flow Analysis: This method is focused on code details and control within the program. Here existing tools such as SonarQube and others can be useful.
• Pattern Matching: Pattern matching usually depends either on a specific lexical analysis or full parsing of programming code. This is best done by scanning or parsing the code into an appropriate model (e.g., graph database) and analysing it in this model. This approach usually needs to be adapted on the individual system. For this approach AI and ML tools can be supportive even though this is a matter of preparing the data in a way that it can be processed by the tools.
• Integration with IDE and with CI/CD Pipelines: continuous code analysis to be integrated in a CI/CD pipeline can be beneficial to keep code at a good quality standard. This integration should not slow down the whole pipeline and has not necessarily to run at each CI/CD cycle but can be run before staging a production release.
• Custom Rules and Configuration: This part of analysis is beyond code analysis but also includes configuration properties. In this analysis larger dependency spans can be evaluated such as which service communicates with what other service. This can be done automatically after an individual inspection of parts of the code to find which information is held in configuration or program files. Certain pattern of coding recognized and known can influence the adaption of the automatic code analysis.
• Program interaction patterns: A specific field of interest in todays distributed architecture is the interdependency by communication of various (micro-)services. These dependencies can not only be found by analysing the code but also with the properties in configuration files (as mentioned previously). New architecture patterns are often used without bounds and with lots of self-interpretation. As the components are developed by more than one team, rules can be different. As a consequence, the chaotic result is often discovered too late and documentation is often missing. Thus, an analysis beyond code can bring clarity into missing component dependency.
• Modular Boundary Analysis: The intend to separate a monolithic application into a multi-component program is prevalent in these days of the microservice-trend. As a good practise, one can take certain artefacts such database schemas or user interface pages as the starting point. These artefacts are manually categorized to a specific business or logical domain similar to domain driven design. The automated analysis process then finds all the dependencies (also on chains) and detecting the "islands". In this way, it can propose the cutting points. Special graph algorithms can be helpful too. The cutting points or lines should be determined manually.
• Language-Specific Analysers: There are specific code analysers almost for every programming language. Theses in general sniff for bad code, violations of certain coding rules and also track dependencies of other artefacts. Usually, these tools are based on the full parser of the language and they are integrated either in IDEs (integrated development environments) or in specific tools. For a software module or project the IDE typically is sufficient to detect code smells and find and show dependencies in code. On the other hand, nowadays software modules depend on hundreds of libraries, which are pulled during building the software. These dependencies are even more crucial in analysis than the dependencies within the project, although the latter is not unimportant at all. Especially keeping track of the library versions and specific security deficiencies are very crucial.
• Code Metrics: These were very much in the focus of software built in the 80s and 90s. Today software metrics are of interest, when acquiring companies with a software asset to get a rough picture of the value and complexity of the acquired software. Due to modern programming languages and features the value of code metrics is not that prevalent anymore but not to be neglected at all. The problem with code metrics is to get a comparison with normative values within the software industry.
• Machine Learning and Artificial Intelligence: ML and AI is a promising approach for the future. Code is text of a specific form and structure. When AI programs can write code, why should they not be able to analyse code. The downside of this approach are the machine power and memory resources needed to do this well (e.g., neural networks). Expert knowledge is also desperately needed to feed the machines. The combination of one or more methods above can lead to incredible results, one could not have imagined before. Another aspect is, that developer pattern in a specific project or team could very well be evaluated by a LLM.

As a precaution: None of the analysis methods mentioned above can fully replace design models and documentation which are supposed to be done by software engineers. As soon as code bases exceeds 50’000-100’000 lines of code it appropriate that one should utilize automation of tools to do specific analysis.

Although automated analysis can give very detailed information about a system, it will only be a part of the whole picture. Semantic and dynamic information is much harder to retrieve and to classify. Human intelligence is needed to evaluate the data assembled and to draw conclusions. AI is for sure of great assistance to faster find the aspects looked for.

In our opinion it is always a combination of automation and human sense which can bring optimal results. Automation can point out hotspots of concerns and these can be inspected by a human who then can aim the automation to focus on more specific aspects of interest.

Representation of the analysis results is also a challenge in itself. Large bodies of knowledge are usually very detailed. A graphical visualization helps on viewing hotspots of interest but it does not give a sufficient view for the details. Tools which allow to drill down from the gross view into the details are more practical. The resulting repositories of the analysis can be walked through by computer programs to satisfy specific queries, which is much more efficient than doing manually. To analyse computer artefacts in deep is very much related with big data analysis, actually the same techniques can be used. A good and necessary approach is to use a appropriate programming language (e.g., a scripting language) to analyse the data once imported in an appropriate da base, similar to analyse big data.

It can be expected that in the near future specifically tailored machine learning (ML) tools can detect code and design anomalies much better. The effort for a substantial analysis of a system should not be underestimated. It is a matter of precision but an 80/20 approach can be sufficient for many cases.

Migration of Legacy Code and Design

Situations when code transformation can be feasible, are:

• Migration of legacy code to another programming language
• Generating code from existing meta data
• Generating code due to redesign or reengineering but the same programming language domain

Migrating of old code into code of a new programming language has the liability, that the old design persists and that the code is just transformed into another programming language. E.g., code of a 4GL-language is transformed into an object-oriented language, i.e. a procedural design is transformed into the object oriented paradigm, which implies that the procedural design will stay.

Most if today’s refactoring is done in the field of Java and C# and systems developed during the last two decades. Code which is older than 25-30 years is not probable to be transformed anymore. But there are always exceptions.

A best practice process could look like:

• Retrieving meta data from existing code and other artefacts
• Carefully analyse these artefacts (possibly automated) and as a conclusion draw a target architecture and migration plan from this
• Creating a target framework in the same way as one would proceed in a normal development project
• With a manual proof of concept verify the migration plan.
• A good test concept for the generated artefacts should not be neglected.
• When proofed valid one takes the earlier retrieved metadata and transforms it into the target code of the framework. If any errors are detected it usually is just a matter of correcting the generator and do a regeneration.
• Code generation can be part of a building process for cases where meta data e.g., meta data are available from databases or from design tools. This can be decided very individually.

Reengineering is a more advanced and individual process than analysis. This is because generating new artefacts ask for precision. The goal of reengineering should always be to reduce the complexity of the design or better the modularity of a system. This can have various levels starting with simple refactoring such as changing code to comply with new API of newer versions of a library. If the replacement of a framework is needed, the change of existing code is not restricted to changing API but also to change of the design. This will imply a more complex reengineering process.

Do not try to achieve 100% generation of target code but stay pragmatic and consider the effort to write the generator program, which should not be higher than doing it manually. But if efforts are about equal it still can be preferable to go with the generator, since the possibility of regeneration, the resulting consistency and quality can be worth the work.

There is another drawback when generating code. Developers are often objecting to generated code. To offend such aversions one could state, that if a good developer can write good code, he is also able to write an appropriate generator, which generates good code.

Tooling

As soon as one has a number of digital resources in the form of program code and meta data like configuration files and database content (not the productive data) the question arises if there are tools which help to analyse and refactor the code. The tool that covers everything is quite an illusion. There are tools for specific purposes such as the analysis of the structure and dependency of programming code and database code. These are specific use cases with well-defined limits. When it comes to analyse how code was written by the authors, following (or not following) the coding policy and using specific coding patterns it becomes individual. Every development group has its certain view and behaviour, which almost always is reflected in the code of the programs, scripts and databases. To process this type of artefacts it needs also individual or very much parameterizable tools. If there are a large number of similar artefacts, one can automatize analysing, refactoring or even reengineering of structure and code. Then additional fitting of existing tools or even developing scripts and programs can be worth the effort, when the effort to write it is much less than executing it manually.

Future of Code Analysis

Considering increased machine power available today creates the opportunity to use more artificial intelligence and machine learning tools to support code analysis. As code is text written and influenced by human minds this implies that the developer pattern could be learned and recognized by such tools. It will just be a matter of training these algorithms. The optimal workbench will include the combination of static analysis and machine learning tools. We want to bread robots which do the tedious work and humans who direct it.

Summary

Automated code analysis and code reengineering of larger code bases (>50'000 LOCs) is necessary since it cannot be handled manually anymore in an efficient way. A meaningful combination of various tools and individual programming can bring massive progress even though an 100% automation is not sensible trying to achieve. But experience shows that savings of 30-60% of effort is a considerable result.

Literature

[1] Martin Fowler, Kent Beck, Refactoring (Improving the Design of Existing Code), Addison-Wesley Signature Series, ISBN-13 978-0134757599

[2] Chris Birchall, Re-Engineering Legacy Software, April 2016, Manning Publications, ISBN 9781617292507

[3] Andres Koch, Remo Koch, Vom Monolithen zur Service-Architektur mit Hilfe von Graphen, Vortrag am 22. Workshop Software-Reengineering & Evolution der Fachgruppe Software-Reengineering der Gesellschaft für Informatik in Paderborn am 17. September 2020

[4] Dr. R. Thurner, Andres Koch, Füsse im Beton, Kopf in der Cloud, OBJEKTspektrum 06/2019

[5] Andres Koch, Remo Koch, Automatisierte Code-Refaktorierung in der Praxis, Vortrag am 21. Workshop Software-Reengineering & Evolution der Fachgruppe Software-Reengineering der Gesellschaft für Informatik in Bad-Honnef am 6. Mai 2019

[6] Kai-Uwe Herrmann, Bison (Schweiz) AG, Teilautomatisiertes Architektur-Reengineering in einem JavaEE Monolithen, Vortrag am 21. Workshop Software-Reengineering & Evolution der Fachgruppe Software-Reengineering der Gesellschaft für Informatik in Bad-Honnef am 6. Mai 2019

[7] Andres Koch, Remo Koch, Metadaten Basiertes Reeingineering, Vortrag am 20. Workshop Software-Reengineering & Evolution der Fachgruppe Software-Reengineering der Gesellschaft für Informatik in Bad-Honnef am 4. Mai 2018

[8] Dr. Reinhold Thurner, Andres Koch, Remo Koch, Teilautomatisiert Migration eines GUI in eine Web-Umgebung, Artikel im Objekt Spektrum Ausgabe 6/2014