This post is about techniques useful for validating iOS builds for people familiar with iOS development.

Background

An iOS app is the product of running multiple compilers, code generators, and scripts. The complexity of this process adds fragility to development, causing late nights and hot fixes for developers of top apps in the App Store. Through build validation, hard to catch regressions and bugs can be squashed ahead of time.

During a complex migration of an iOS app from Xcode to Bazel, techniques mentioned at the Bazel Conference were applied to ensure the iOS Bazel build was the same as the Xcode one.

“Build validation” can be useful to assert correctness in other contexts, like build rule changes, Xcode updates/changes, validating XCHammer generated Xcode projects, comparing CI builds, or large refactors.

Build Validation and Techniques

Validating the product of an iOS build is not achievable by naively introspecting the output .app. If 2 builds produced identical output, then it’d be possible to compare the raw bits on disk. In practice, this won’t work: some tools can’t produce an identical output every time. Additionally, bit level comparisons alone wouldn’t provide enough info to identify root causes of deltas.

By intelligently validating the build steps, inputs, and outputs, it’s possible to produce better insights. The remainder of the post is about validating a .app and validating the steps to produce a .app.

Output validation

A .app is produced from a build to be later loaded on a users device. The application directory contains several bits e.g. the main binary of the app, dynamically loadable code ( frameworks ), application extensions, nib files, compiled asset catalogs, machine learning models, arbitrary text files, images. The Apple developer website includes a complete description of this package.

Output validation at the .app level can find differences regardless of how each was produced. This process requires processing each file in the .app uniquely. Here’s a few non exhaustive examples of things worth checking between 2 built apps:

Executables and binaries

Binaries can be checked by reading segments to textual output. For example:

What’s in the symbol table?

nm -an Some.app/Some

What libraries are linked?

otool -l Some.app/Some

Other information, like size is worth checking

What’s the size of the app?

du -hs Some.app/

Exhaustive techniques like binary size profiling could be applied too.

App Capabilities

Capabilities can be checked by introspecting plists, entitlements, and provisioning.

What entitlements are used?

codesign -d --entitlements :-  Some.app/Some

Whats in the provisioning profile?

security -D -i Some.app/Some/embedded.mobileprovision

Build command validation

Today, iOS build systems run many commands: an invocation to some program with some arguments and inputs. By validating the commands ran during the build, differences in tooling and inputs can be uncovered.

First, there needs to be a way to extract the commands. A tool like XcodeCompilationDatabase can extract most of the compiler invocations from an Xcode build. After extracting the actual commands, it’s possible to compare the inputs e.g. a checksum of clang, a hash of a source file, the compiler arguments.

Clang compilation

By using a CompilationDatabase, it’s possible to read in the actual invocations used for source files:

{ "Some.c": "/path/to/clang -Wall -c Some.c -o Some.o" }

By reading this file, we can tell that Some.c is part of the build. Now, it’s possible to compute a checksum of the source, Some.c, the compiler, /path/to/clang and the corresponding arguments. In practice, comparing 2 clang invocations is not trivial and requires knowing about how the compiler will use each argument. e.g.: ordering matters, there are special rules in clang to determine how arguments are parsed, header files are determined by searching of include paths.

A few data structures and utilities exist in clang that can help normalize arguments, like the diagtool. In some scenarios, it’s insightful to validate at the object file level by either raw bit comparison or with tools like nm.

Conclusion

Build validation can unveil insights by analyzing the build process and it’s outputs. By breaking down a build into it’s elemental functions, abstractions like Bazel Rules or Xcode Build Settings are easy to validate and reason about. This process is no silver bullet by any means, but is useful in addition to standard practices like testing or QA.

A reuseable system to apply these techniques would be beneficial for the community at large. A Swift library could be implemented to write build validation programs for many use cases.