“Clang” CFE Internals Manual

Introduction

This document describes some of the more important APIs and internal design decisions made in the Clang C front-end. The purpose of this document is to both capture some of this high-level information and also describe some of the design decisions behind it. This is meant for people interested in hacking on Clang, not for end-users. The description below is categorized by libraries, and does not describe any of the clients of the libraries.

LLVM Support Library

The LLVM libSupport library provides many underlying libraries and data-structures, including command line option processing, various containers, and a system abstraction layer, which is used for file system access.

The Clang “Basic” Library

This library certainly needs a better name. The “basic” library contains a number of low-level utilities for tracking and manipulating source buffers, locations within the source buffers, diagnostics, tokens, target abstraction, and information about the subset of the language being compiled for.

Part of this infrastructure is specific to C (such as the TargetInfo class), other parts could be reused for other non-C-based languages (SourceLocation, SourceManager, Diagnostics, FileManager). When and if there is future demand, we can figure out if it makes sense to introduce a new library, move the general classes somewhere else, or introduce some other solution.

We describe the roles of these classes in order of their dependencies.

The Diagnostics Subsystem

The Clang Diagnostics subsystem is an important part of how the compiler communicates with the human. Diagnostics are the warnings and errors produced when the code is incorrect or dubious. In Clang, each diagnostic produced has (at the minimum) a unique ID, an English translation associated with it, a SourceLocation to “put the caret”, and a severity (e.g., WARNING or ERROR). They can also optionally include a number of arguments to the diagnostic (which fill in “%0“‘s in the string) as well as a number of source ranges that related to the diagnostic.

In this section, we’ll be giving examples produced by the Clang command line driver, but diagnostics can be rendered in many different ways depending on how the DiagnosticConsumer interface is implemented. A representative example of a diagnostic is:

t.c:38:15: error: invalid operands to binary expression ('int *' and '_Complex float')
P = (P-42) + Gamma*4;
    ~~~~~~ ^ ~~~~~~~

In this example, you can see the English translation, the severity (error), you can see the source location (the caret (”^”) and file/line/column info), the source ranges “~~~~”, arguments to the diagnostic (”int*” and “_Complex float”). You’ll have to believe me that there is a unique ID backing the diagnostic :).

Getting all of this to happen has several steps and involves many moving pieces, this section describes them and talks about best practices when adding a new diagnostic.

The Diagnostic*Kinds.td files

Diagnostics are created by adding an entry to one of the clang/Basic/Diagnostic*Kinds.td files, depending on what library will be using it. From this file, tblgen generates the unique ID of the diagnostic, the severity of the diagnostic and the English translation + format string.

There is little sanity with the naming of the unique ID’s right now. Some start with err_, warn_, ext_ to encode the severity into the name. Since the enum is referenced in the C++ code that produces the diagnostic, it is somewhat useful for it to be reasonably short.

The severity of the diagnostic comes from the set {NOTE, REMARK, WARNING, EXTENSION, EXTWARN, ERROR}. The ERROR severity is used for diagnostics indicating the program is never acceptable under any circumstances. When an error is emitted, the AST for the input code may not be fully built. The EXTENSION and EXTWARN severities are used for extensions to the language that Clang accepts. This means that Clang fully understands and can represent them in the AST, but we produce diagnostics to tell the user their code is non-portable. The difference is that the former are ignored by default, and the latter warn by default. The WARNING severity is used for constructs that are valid in the currently selected source language but that are dubious in some way. The REMARK severity provides generic information about the compilation that is not necessarily related to any dubious code. The NOTE level is used to staple more information onto previous diagnostics.

These severities are mapped into a smaller set (the Diagnostic::Level enum, {Ignored, Note, Remark, Warning, Error, Fatal}) of output levels by the diagnostics subsystem based on various configuration options. Clang internally supports a fully fine-grained mapping mechanism that allows you to map almost any diagnostic to the output level that you want. The only diagnostics that cannot be mapped are NOTEs, which always follow the severity of the previously emitted diagnostic and ERRORs, which can only be mapped to Fatal (it is not possible to turn an error into a warning, for example).

Diagnostic mappings are used in many ways. For example, if the user specifies -pedantic, EXTENSION maps to Warning, if they specify -pedantic-errors, it turns into Error. This is used to implement options like -Wunused_macros, -Wundef, etc.

Mapping to Fatal should only be used for diagnostics that are considered so severe that error recovery won’t be able to recover sensibly from them (thus spewing a ton of bogus errors). One example of this class of error is failure to #include a file.

Diagnostic Wording

The wording used for a diagnostic is critical because it is the only way for a user to know how to correct their code. Use the following suggestions when wording a diagnostic:

  • Diagnostics in Clang do not start with a capital letter and do not end with punctuation.

    • This does not apply to proper nouns like Clang or OpenMP, to acronyms like GCC or ARC, or to language standards like C23 or C++17.

    • A trailing question mark is allowed. e.g., unknown identifier %0; did you mean %1?.

  • Appropriately capitalize proper nouns like Clang, OpenCL, GCC, Objective-C, etc. and language standard versions like C11 or C++11.

  • The wording should be succinct. If necessary, use a semicolon to combine sentence fragments instead of using complete sentences. e.g., prefer wording like '%0' is deprecated; it will be removed in a future release of Clang over wording like '%0' is deprecated. It will be removed in a future release of Clang.

  • The wording should be actionable and avoid using standards terms or grammar productions that a new user would not be familiar with. e.g., prefer wording like missing semicolon over wording like syntax error (which is not actionable) or expected unqualified-id (which uses standards terminology).

  • The wording should clearly explain what is wrong with the code rather than restating what the code does. e.g., prefer wording like type %0 requires a value in the range %1 to %2 over wording like %0 is invalid.

  • The wording should have enough contextual information to help the user identify the issue in a complex expression. e.g., prefer wording like both sides of the %0 binary operator are identical over wording like identical operands to binary operator.

  • Use single quotes to denote syntactic constructs or command line arguments named in a diagnostic message. e.g., prefer wording like 'this' pointer cannot be null in well-defined C++ code over wording like this pointer cannot be null in well-defined C++ code.

  • Prefer diagnostic wording without contractions whenever possible. The single quote in a contraction can be visually distracting due to its use with syntactic constructs, and contractions can be harder to understand for non- native English speakers.

The Format String

The format string for the diagnostic is very simple, but it has some power. It takes the form of a string in English with markers that indicate where and how arguments to the diagnostic are inserted and formatted. For example, here are some simple format strings:

"binary integer literals are an extension"
"format string contains '\\0' within the string body"
"more '%%' conversions than data arguments"
"invalid operands to binary expression (%0 and %1)"
"overloaded '%0' must be a %select{unary|binary|unary or binary}2 operator"
     " (has %1 parameter%s1)"

These examples show some important points of format strings. You can use any plain ASCII character in the diagnostic string except “%” without a problem, but these are C strings, so you have to use and be aware of all the C escape sequences (as in the second example). If you want to produce a “%” in the output, use the “%%” escape sequence, like the third diagnostic. Finally, Clang uses the “%...[digit]” sequences to specify where and how arguments to the diagnostic are formatted.

Arguments to the diagnostic are numbered according to how they are specified by the C++ code that produces them, and are referenced by %0 .. %9. If you have more than 10 arguments to your diagnostic, you are doing something wrong :). Unlike printf, there is no requirement that arguments to the diagnostic end up in the output in the same order as they are specified; you could have a format string with “%1 %0” that swaps them, for example. The text in between the percent and digit are formatting instructions. If there are no instructions, the argument is just turned into a string and substituted in.

Here are some “best practices” for writing the English format string:

  • Keep the string short. It should ideally fit in the 80-column limit of the DiagnosticKinds.td file. This avoids the diagnostic wrapping when printed, and forces you to think about the important point you are conveying with the diagnostic.

  • Take advantage of location information. The user will be able to see the line and location of the caret, so you don’t need to tell them that the problem is with the 4th argument to the function: just point to it.

  • Do not capitalize the diagnostic string, and do not end it with a period.

  • If you need to quote something in the diagnostic string, use single quotes.

Diagnostics should never take random English strings as arguments: you shouldn’t use “you have a problem with %0” and pass in things like “your argument” or “your return value” as arguments. Doing this prevents translating the Clang diagnostics to other languages (because they’ll get random English words in their otherwise localized diagnostic). The exceptions to this are C/C++ language keywords (e.g., auto, const, mutable, etc) and C/C++ operators (/=). Note that things like “pointer” and “reference” are not keywords. On the other hand, you can include anything that comes from the user’s source code, including variable names, types, labels, etc. The “select” format can be used to achieve this sort of thing in a localizable way, see below.

Formatting a Diagnostic Argument

Arguments to diagnostics are fully typed internally and come from a couple of different classes: integers, types, names, and random strings. Depending on the class of the argument, it can be optionally formatted in different ways. This gives the DiagnosticConsumer information about what the argument means without requiring it to use a specific presentation (consider this MVC for Clang :).

It is really easy to add format specifiers to the Clang diagnostics system, but they should be discussed before they are added. If you are creating a lot of repetitive diagnostics and/or have an idea for a useful formatter, please bring it up on the cfe-dev mailing list.

Here are the different diagnostic argument formats currently supported by Clang:

“s” format

Example:

"requires %0 parameter%s0"

Class:

Integers

Description:

This is a simple formatter for integers that is useful when producing English diagnostics. When the integer is 1, it prints as nothing. When the integer is not 1, it prints as “s”. This allows some simple grammatical forms to be to be handled correctly, and eliminates the need to use gross things like "requires %1 parameter(s)". Note, this only handles adding a simple “s” character, it will not handle situations where pluralization is more complicated such as turning fancy into fancies or mouse into mice. You can use the “plural” format specifier to handle such situations.

“select” format

Example:

"must be a %select{unary|binary|unary or binary}0 operator"

Class:

Integers

Description:

This format specifier is used to merge multiple related diagnostics together into one common one, without requiring the difference to be specified as an English string argument. Instead of specifying the string, the diagnostic gets an integer argument, and the format string selects the numbered option. In this case, the “%0” value must be an integer in the range [0..2]. If it is 0, it prints “unary”, if it is 1 it prints “binary” if it is 2, it prints “unary or binary”. This allows other language translations to substitute reasonable words (or entire phrases) based on the semantics of the diagnostic instead of having to do things textually. The selected string does undergo formatting.

“enum_select” format

Example:

unknown frobbling of a %enum_select<FrobbleKind>{%VarDecl{variable declaration}|%FuncDecl{function declaration}}0 when blarging

Class:

Integers

Description:

This format specifier is used exactly like a select specifier, except it additionally generates a namespace, enumeration, and enumerator list based on the format string given. In the above case, a namespace is generated named FrobbleKind that has an unscoped enumeration with the enumerators VarDecl and FuncDecl, which correspond to the values 0 and 1. This permits a clearer use of the Diag in source code, as the above could be called as: Diag(Loc, diag::frobble) << diag::FrobbleKind::VarDecl.

“plural” format

Example:

"you have %0 %plural{1:mouse|:mice}0 connected to your computer"

Class:

Integers

Description:

This is a formatter for complex plural forms. It is designed to handle even the requirements of languages with very complex plural forms, as many Baltic languages have. The argument consists of a series of expression/form pairs, separated by “:”, where the first form whose expression evaluates to true is the result of the modifier.

An expression can be empty, in which case it is always true. See the example at the top. Otherwise, it is a series of one or more numeric conditions, separated by “,”. If any condition matches, the expression matches. Each numeric condition can take one of three forms.

  • number: A simple decimal number matches if the argument is the same as the number. Example: "%plural{1:mouse|:mice}0"

  • range: A range in square brackets matches if the argument is within the range. The range is inclusive on both ends. Example: "%plural{0:none|1:one|[2,5]:some|:many}0"

  • modulo: A modulo operator is followed by a number, and equals sign and either a number or a range. The tests are the same as for plain numbers and ranges, but the argument is taken modulo the number first. Example: "%plural{%100=0:even hundred|%100=[1,50]:lower half|:everything else}1"

The parser is very unforgiving. A syntax error, even whitespace, will abort, as will a failure to match the argument against any expression.

“ordinal” format

Example:

"ambiguity in %ordinal0 argument"

Class:

Integers

Description:

This is a formatter which represents the argument number as an ordinal: the value 1 becomes 1st, 3 becomes 3rd, and so on. Values less than 1 are not supported. This formatter is currently hard-coded to use English ordinals.

“human” format

Example:

"total size is %human0 bytes"

Class:

Integers

Description:

This is a formatter which represents the argument number in a human-readable format: the value 123 stays 123, 12345 becomes 12.34k, 6666666 becomes 6.67M, and so on for ‘G’ and ‘T’.

“objcclass” format

Example:

"method %objcclass0 not found"

Class:

DeclarationName

Description:

This is a simple formatter that indicates the DeclarationName corresponds to an Objective-C class method selector. As such, it prints the selector with a leading “+”.

“objcinstance” format

Example:

"method %objcinstance0 not found"

Class:

DeclarationName

Description:

This is a simple formatter that indicates the DeclarationName corresponds to an Objective-C instance method selector. As such, it prints the selector with a leading “-“.

“q” format

Example:

"candidate found by name lookup is %q0"

Class:

NamedDecl *

Description:

This formatter indicates that the fully-qualified name of the declaration should be printed, e.g., “std::vector” rather than “vector”.

“diff” format

Example:

"no known conversion %diff{from $ to $|from argument type to parameter type}1,2"

Class:

QualType

Description:

This formatter takes two QualTypes and attempts to print a template difference between the two. If tree printing is off, the text inside the braces before the pipe is printed, with the formatted text replacing the $. If tree printing is on, the text after the pipe is printed and a type tree is printed after the diagnostic message.

“sub” format

Example:

Given the following record definition of type TextSubstitution:

<