Documentation Code Testing
Hi folks,
I'm working on a project called swift-doc-testing, a bundle of a Swift CLI and a VSCode extension for building and testing code snippets in documentation comments (for swift packages). I want to take this post as an opportunity to share the purpose and uses of this project, and would love to hear your feedback: how likely you'd use this? what features you'd like to have for such too?
Problems to solve:
Code snippets in doc comments alongside implementation code is useful and convenient. They provide context and usage, itself serving as an intuitive, irreplaceable documentation.
However, they often suffer from the following issues:
- code snippets are not compiled/checked by the compiler, thus prone to errors and difficulty in maintenance
- code snippets can be hard to format automatically, which also makes maintenance harder
- code snippets that are complex may not be readable for doc writers, given the absence of syntax highlight
This Project Offers
- a CLI that can
- extract code snippets in doc comments
- apply formatting automatically to doc comments
- compile and run test snippets as tests given a doc test template
- initialize a doc test template and provide convenient edit functionality
- a VSCode plugin that uses the CLI and
- gathers discovered code snippets in doc comments
- compiles and runs the code snippets as tests (with swift-testing), integrates test results over swift-testing's event stream to native VSCode test system
This project targets documentation code testing in swift packages, not xcode projects.
How It Works
Discovery
In a Swift doc comment like the following
/// Returns `true` if the list contains no elements.
///
/// Example with empty and non-empty lists:
///
/// ```swift
/// let empty = LinkedList<Int>()
/// #expect(empty.isEmpty)
///
/// var nonEmpty = LinkedList<String>()
/// nonEmpty.append("item")
/// #expect(!nonEmpty.isEmpty)
/// ```
public var isEmpty: Bool {
head == nil
}
The CLI picks up the code in code fence that has a swift language tag, and has the option to interpret code fences without a language tag as swift code.
```swift
```
Additionally, it currently supports test options, inspired by rustdoc test:
@no-check: skips checking the code fence@compile-fail: asserts such code snippet fails to compile
and several on my todo list:
@compile-only: only compiles@assert-error: test throws error
The options can be used like the following, similar to the code block annotation capability introduced in swift 6.3
```swift, @no-check
```
Test harness Generation
The CLI helps generate a DocTest folder under the client Swift package (swift-doc-testing edit init), which is a Swift package of the following structure.
❯ tree DocTest
DocTest
├── Package.swift
└── Tests
└── DocTestTemplate
└── DocTestTemplate.swift
You can setup the testing environment like you normally would for Swift packages, for instance
// swift-tools-version: 6.3
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "DocTestTemplate",
dependencies: [
.package(name: "client-package", path: ".."),
.package(url: "https://pending.com/swift-doc-testing-utils", from: "0.0.1"),
],
targets: [
.target(
name: "TestUtils",
path: "Tests/Utils",
),
.testTarget(
name: "DocTestTemplate",
dependencies: [
.product(name: "Product", package: "client-package"),
.product(name: "SwiftDocTestingUtils", package: "swift-doc-testing-utils"),
"TestUtils",
],
),
],
swiftLanguageModes: [.v6]
)
The test template is a file named DocTestTemplate.swift under the DocTestTemplate target, that will be used to scaffold test targets using doc code with the help of swift syntax.
import SwiftDocTestUtils
import Testing
#importSourcePackage()
@Test
func `#docTestIdentity`() {
#injectDocTests()
}
For instance, a scaffolded test is
// file: docTest_LinkedList_isEmpty_0.swift
import SwiftDocTestUtils
import Testing
import DocTestExample
@Test
func `docTest_LinkedList_isEmpty_0`() {
let empty = LinkedList<Int>()
#expect(empty.isEmpty)
var nonEmpty = LinkedList<String>()
nonEmpty.append("item")
#expect(!nonEmpty.isEmpty)
}
Test Runner
The test runner runs tests using swift test with --experimental-event-stream-output to obtain test execution information. It runs different categories (compile only, compile fail, etc.) separately.
VSCode Integration
VSCode extension
The extension adds a collection of discovered tests, just like the official swift plugin. You can run the tests just like the official swift plugin.
Edit Doc Test Template
The CLI provides a command that composes a temporary directory, hosts the doc test template package, and opens it in VSCode. This is to reduce friction of editing doc test template.
Current Status
I'm close to having a MVP that supports what's described in the previous section, and hopefully can open-source it in the upcoming week. A big chunk of the VSCode extension was built with Claude's help, and I want to make sure things are tested and work before released.
I'm also sorting out how to provide the best friction-less and feature-rich user experience. If you can think of anything you'd like to use, please let me know. I greatly appreciate any feedback.
TODO List
- My initial vision is to provide syntax highlight and LSP for the code snippets in doc comments. Syntax highlight seems possible, but LSP may require more work, as currently doc tests are run in one-time temporary directories.
- Optimizations to doc test compilation, such as organizing caching more efficiently, since users are likely to reuse the same packages across groups of doc tests for a given package.
- Support different templates for different options, such as allowing
compile-failto have a dedicated test template.
Thanks for reading! Would love to hear your thoughts!!