GitOps Repository Structures and Patterns Part 3: Repository Patterns

---

---

In this series of articles, I will introduce you to different structures and patterns that you can use to design your GitOps process. This third part is about Repository Patterns, which answer the question about the number of GitOps repos. There are a few patterns to choose from in this category. However, the choice does not have to be exclusive. Some patterns can also be combined well. For an introduction into GitOps-repository patterns and structures, take a look at the first part of this series. The second part explains operator-deployment patterns, the fourth part promotion patterns and the fifth part wiring patterns. In the sixth part I show different implementations of the structures and patterns using example repositories.

Monorepo

When the entire configuration is stored in a single repo, it is called “monorepo”. It is questionable whether the typical advantages of monorepo in software development, as used by some big tech companies, for example easier refactoring and dependency management, also apply to GitOps. Disadvantages of monorepos are often more difficult to configure per-folder authorization and poor performance due to the large repo, due to the many commits. The opposite of monorepo is sometimes referred to as “polyrepo”. However, the term polyrepo does not refer to a single pattern.

Repo per Team

When multiple repos are created, most people first think of the “repo per team” pattern. More generalized, the term “tenant” is sometimes used instead of team. This offers the advantage that repo-level authorization is typically easy to handle in source code management (SCM). In addition, it often feels natural to mimic the structures of the organization, as mentioned at the outset (Conway’s Law). In addition, each team is also only shown what is relevant to them, which reduces complexity and “mental load.”

Figure 1: GitOps-Repository vs. App-Repository

Repo per App

The “Repo per App” pattern offers an alternative. This pattern is often used for apps that are developed by the team itself. This is meant in contrast to off-the-shelf/3rd-party apps that are operated but not self-developed. The pattern has the advantage that everything that belongs to an application is in the same repo. This includes the application’s code, documentation and config. Config means Kubernetes resources, for example; in some cases, this is referred to as “Infra As Code”. Storing source code and config in the same repo is very popular, especially among developers. However, it is recommended (for example, according to Argo CD) to separate the source code of the application from the config. At this point, it makes sense to distinguish the terms app repo (contains source code) and GitOps repo.

Figure 1 shows a comparison and further synonyms. The separation of app repo and GitOps repo has, among other things, the advantage that the entire config (for example, of a team with multiple applications or of an entire cluster) is in a central location, where it can be better audited and searched. This separation also has advantages for automating the update of newly built image versions with the CI server, as it avoids endless loops of build jobs and Git pushes. On the other hand, this very thing is also a disadvantage, since there is no CI job to perform static code analysis, for example. What now?

Config Replication

Figure 2: Implementation of Repo per App via Config Replication

A compromise can be to implement the “Repo per App” pattern by “Config Replication”. In this case, the config remains in the app repo and is then pushed from the CI server to the GitOps repo. In this process, the CI server can also be used to implement Shift Left: It can perform static code analysis. Enclosed are some concrete tool suggestions: By means of yamllint simple syntax errors can be found early. Kubeconform prevents Kubernetes resources from using fields that are not present in the API server’s schema. Helm Lint can be used to find errors in Helm Charts.

Tip: Schemas can also be generated and validated for Helm Charts. With conftest, OpenPolicyAgent policies can be validated before deployment to the cluster. There are also a large number of security scanners that ensure that no insecure configs end up in the cluster. For example, kubescape is being developed as part of the CNCF.

Figure 3: Example Pull Request

The CI server can also provide further automation, for example for promotion (more on this in the next part of this series) or to provide commits and pull requests with further information. Figure 3 shows an example here. In it, the commit contains a prefix for the environment (here “production”), a link to the issue tracker (connects the config to the user story) and a link to the associated commit in the app repo. The git author is inherited from the app repo commit, but the CI server is set as committer. This makes sure that this commit is recognizable as a generated commit.

Of course, Config Replication does not only have advantages. One disadvantage is the complexity of the resulting pipelines. To avoid having to duplicate the logic for every application, it makes sense to develop something reusable here. Examples of this are a GitHub Action or a Jenkins Shared Library. Experience shows that quite a bit of effort is required before this all works resiliently. For example, concurrency in interaction with Git and automatic merges can be risky (retry strategies and risk of inconsistency) and the advanced features mentioned, such as static code analysis and information in commits can cause additional effort. So it is recommended to use something existing here instead of building it yourself. An example is the GitOps-Build-Lib for Jenkins. It was also used to build the commit shown in Figure 3. Another disadvantage is the redundancy of the config. It exists once in the app repo and once in the GitOps repo.

Repo Pointer

Figure 4: Implementation of Repo per App via Repo Pointer

If the redundancy bothers you, you can alternatively implement the “repo per app” pattern with a “repo pointer” as shown schematically in Figure 4. In this case, the code is not replicated, but referenced from the GitOps repo to the app repo. This can be implemented, for example, with an Application Custom Resource (CR) in Argo CD or a Kustomization CR in Flux. The GitOps operator then pulls the config directly from the app repo. The Git-native way via submodules would also be possible. However, experience has shown that the use of CRs is more usable. This approach also has its disadvantages. For example, the GitOps controller must be authorized on many repos and the GitOps repo is no longer the central location for Config, but only contains links.

Repo per Environment

A last pattern that is feasible with regard to repositories is “Repo per Environment” (also “Environment per Repo” or with the term “Stage” instead of “Environment”). Here, a repository is created for each environment (for example, development, staging, production). This pattern is rarely chosen because it leads to a large number of repositories and a less automated, more tedious GitOps process. Reasons for choosing this are mostly organizational requirements. For example, when developers are not allowed to access production or authorization on folders within a Git repo is not possible or isolated enough. Another use case is when releases need to be approved by a security team.

Conclusion

The different repository patterns offer the possibility to adapt the GitOps process to your own requirements. Further possibilities are offered by the patterns presented in the other parts of this article series: operator deployment patterns, promotion patterns, wiring patterns and real-life examples.

---

---