twocentstudios

Print Design Mindset to App Design Mindset

Thu, 29 Jan 2026 14:26:08 -0600

The print design era

There was a time when iOS app design wasn’t all that far off from print design.

For the first couple years of the iPhone SDK (2008-2011):

All iPhones had 320 width by 480 height screens.
All iPhone screens were @1x non-retina.
The platform design language was highly skeuomorphic and rewarded delicately crafted Photoshop assets.
iOS did support localization but not for RTL languages and the launch country list was relatively English-speaking (and it was never required for App Store listing).
Accessibility features that dynamically altered app content visually did not exist.

iPhone, iPhone 3G, iPhone 4, and iPhone 5 — all 320 points wide

Sure, there were dynamic elements like buttons with highlighted, disabled, and selected states. Sure, there were scrolling screens. But the overall workflow of app design to app development was taking a Photoshop mockup and recreating it piece by piece in the SDK.

Until Sketch gained some prominence in the 2013-2015 era, Photoshop was still the most popular tool for less technical designers to work in before artifact hand off.

Sketch app showing iOS device artboard presets including iPhone 6, iPhone 5, and iPad sizes

The first cracks in the print design workflow surprisingly started with the sudden differentiation between points and pixels with the iPhone 4 in 2010. Even then, that didn’t necessitate a mindset change away from print design, just a few configuration changes in Photoshop and some extra work come asset export time.

It wasn’t until the iPhone 6 in 2014 that designers (and developers) could no longer assume devices had the same 320 point width. As an iOS developer, you could get away with hard-coding dimensions for your UIViews for a very long time! (Assuming you supported only landscape OR portrait orientation).

iPhone 6 (375pt), iPhone X (375pt), iPhone 13 (390pt), and iPad Air (768pt)

And if varying screen widths weren’t enough, the iPhone X in 2017 broke the assumption that the screen was even a simple rectangle. The notch, rounded corners, and the home indicator meant designers could no longer treat the canvas as a predictable bordered frame.

The complexity catches up

I’ll skip to present day to say that we’re long past the point that most app designers can, with a good conscience, keep a print design mindset when doing visual design for mobile.

The complexity grew, and luckily for iOS developers, the tooling provided by Apple to rein in that complexity has slowly caught up:

For view layout, we have stacks, content margins, size classes, line limits, leading/trailing instead of left/right.
For Dynamic Type (user adjustable font sizes), we have semantic fonts like .body, .headline, .largeTitle. We have the @Scaled property wrapper in SwiftUI. We have UIFontMetrics for scaling non-system fonts.
For Dark Mode, we have semantic colors like .label, .background, .secondary, .blue, .tint, and asset catalogs.
For localization, we have auto-updating xcstrings files, Double-Length pseudolanguage, RTL pseudolanguage, asset catalog localization, plural strings, and localization aware formatters.

HIG semantic label colors adapt automatically across appearances

HIG system colors vary across light mode, dark mode, and increased contrast

HIG Dynamic Type defines seven non-accessibility size categories with semantic text styles

However, there’s a similar time lag (measured in years) between SDK support and true developer tooling support that exists between developer tooling support and designer tooling support for all these features. There probably always will be. Users will adopt them and demand them faster than our tools will support them.

The mindset shift

But going back to print design.

I love (well, maybe love/hate) designing apps, but what I love more is working with talented designers. There’s something about the design field that draws individuals with a tenacious eye for detail and an equally fierce demand for seeing their work reflected with pixel perfection. And most of the time, I would interpret this desire as driven by empathy for the user and not selfishness. They know users recognize and deserve beauty. In other words, their intentions are good.

The trap I’ve seen some of these designers fall into is assuming it’s enough to have empathy for a user and not all users of their designs. In the early iPhone era, 100% of users would see the same screen pixel-for-pixel that the designer created in their Photoshop document. In 2026, the percent of users that see a screen exactly as it was designed is ballpark closer to single digit percentages, if that.

There are simply too many variables that affect the final visual design a user sees when they open your app. Too many to optimize for pixel accuracy with a print design mindset. Dozens of screen sizes. Dozens of languages. Dark Mode. Near a dozen Dynamic Type settings. Bold text. Button shapes. Virtual home buttons. A virtual cornucopia of more esoteric accessibility options. And on iPad, not even fixed screen sizes thanks to Slide Over, Split View, and Stage Manager.

And that’s not even digging into the impedance mismatches baked into the design tools themselves: system fonts displaying a little bit smaller on device than in Figma; color spaces not matching; static assets losing fidelity during export.

I’ve worked alongside many incredibly talented designers over the years. Almost all of them sooner or later made the jump from print design mindset to app design mindset. It took a lot of frustrating (on both sides!) conversations.

At the end of the day, as a developer, I knew the device was the final arbiter of truth. The painstakingly crafted mockups only reflected reality for the brief period of time before the app went live on App Store, and after that they were vestigial colored rectangles, decaying with a half life in the order of days not weeks. The same way that multi-page product spec lovingly crafted by the product manager is lost to the abyss of Google Docs at the end of the sprint.

I knew the signs of the mindset shift, and they all came like manna from heaven. I’d start to receive mockups with loading states leading to the happy path. I’d get error messages with HIG-respectful copywriting. I’d get sidecar notes in the margins pointing out fixed and flexible dimensions. I’d get line limits for text. I’d see table cells with long and short user names. With empty and populated avatar images. The apps were starting to show signs of being apps before I’d actually turned them into living & breathing apps.

More importantly, I would stop receiving design review feedback about marketing copy not line breaking at the exact same words as the mockup. About using a custom alert view instead of the system one.

The designers were now channeling their intrinsic motivation and eye for detail that previously went into pixel perfection into ensuring the maximum number of users with any combination of device settings would still be able to use the damn app to get their task accomplished. And hopefully be able to appreciate the fact the app looked and felt nice while they were getting things done.

The inter-department friction would melt away. We’d ship faster even though we were supporting more design variables than ever before. After all, designers were taking on more of the work I’d previously been doing turning screen after lifeless screen into constraints and states and conditionals. I had more brain bandwidth to double check their assumptions, cover more edge cases, explore architectures that would keep the code from rotting, experiment with animations, and still have time to pair with them on polish. And even more-so, develop abstractions to make this work faster and easier in the future.

When you make the foundational work easy, you have more time left for whimsy. Which is more fun for both the app makers and the app users.

Making the shift

So how can a designer with a print mindset adopt an app design mindset?

Half the battle is simply recognizing the differences between the two. Giving each a name. Recognizing the signs and fighting your instincts for pixel perfection.

From there, I can offer a few suggestions:

Pairing with developers who care about accessibility, design systems, the human interface guidelines, and most of all, users.
Immersing in Apple-sanctioned resources like the HIG, WWDC videos, and learning how to search, read, and understand the developer documentation.
Immediately learning the avant-garde features of design tools that lag behind the native developer tools.
Following the unicorn designer/developer hybrid folks that work in and share wisdom from both sides of the aisle.
Gaining enough proficiency with developer (and AI?) tools to become dangerous (or just be able to update a color value or knock out a quick interactive prototype).
Visualizing apps not just as a collection of snapshots of happy path screens, but a rich journey of pushes, pops, loads, errors, mistaps, drags, interruptions, and everything in between.
Do the painful work of watching over a user’s shoulder as they tap through your app on their 8 year old iPhone with their settings and their cracked screen.

Cheers to many more happy and healthy designer 🤝 developer partnerships in the years to come.

Closing the Loop on iOS with Claude Code

Sat, 27 Dec 2025 15:37:01 -0600

Closing the loop means giving Claude Code a way to view the output of its work. I’ll be focusing on iOS app development workflows.

Step 1 of closing the loop: building a target so that Claude Code can see the errors and warnings. And doing so in a way that preserves the build cache (clean builds take a long time). This allows Claude Code to see its syntax errors and fix them before you review its work.

Step 2 of closing the loop: installing & launching on the simulator. This saves you the step of opening Xcode and hitting build & run, letting you test each proposed code change right away.

Step 3 of closing the loop: reading the console & log output. This allows Claude Code to proactively verify codepaths and reactively do debugging.

Step 4 of closing the loop: controlling & viewing the iOS simulator. This allows Claude Code to step through entire flows, evaluate visual designs, and generate its own logs.

Step 5 of closing the loop: building, installing, launching, and logging on device. This allows you and Claude Code to test Apple Frameworks that are absent or broken on the simulator.

Building, installing, launching on the simulator from Claude Code

Disclaimers before we start

Agentic tooling is changing rapidly with model and agent versions. I’ll cover each step as thoroughly as I can. The strategies in this post cover about a month of work in December 2025 with Claude Opus 4.5 inside Claude Code v2.0.76 (and several versions below). I used Xcode 26.1 and 26.2 on macOS 15.7.3 mostly developing for iOS 26.

This post is written for humans but can easily be adapted to a Skill or added to your CLAUDE.md file. The command structure will change based on how your project and schemes are set up. I outline a few different strategies that are useful in different situations, but you may only want to use one workflow as your default, or completely ignore certain steps altogether.

If you’ve always used a manual Xcode-based flow, trying to both understand and incorporate these steps into your workflow can be intimidating. If you’ve never working with Xcode via the command line before, start with just the first step for a while. The best part about this workflow is you can seamlessly dip in and out of using Xcode and there’s no switching cost (not even needing to do clean builds).

The below CLI commands also share a lot of coverage with XcodeBuildMCP, a more full-service MCP-based solution. I won’t get into the pros and cons of MCPs vs CLIs (its author has already written about that).

I’m specifically targeting this post to Claude Code and Opus 4.5 based on my first-hand knowledge of their combined capabilities. Other harness and model pairs will work with most of the commands in this post. The command backgrounding feature in step 3 is the only potential snag for some harnesses.

Step 1: Building

Allowing Claude Code to build after every proposed change is a requirement for agentic workflows. Like it does for human developers, the compiler catches dumb syntax errors and, with Swift concurrency, even data races. The alternative is tabbing back over to Xcode, hitting cmd+b, waiting, copying and pasting error messages into the terminal; a massive waste of human time.

Prerequisites

Move DerivedData location to your project folder (optional)

Moving DerivedData to a location inside your project folder is perhaps an unusual suggestion, but it has several benefits for an agentic workflow:

Permissions: you’ll encounter fewer permissions dialogs when Claude is reading inside the project folder that you’re presumably running it in. Most devs expect DerivedData to be cleared regularly so it’s safe.
Git worktrees: An advanced technique is to use Git worktrees to have independent copies of your repo. Colocating DerivedData ensures the separate repos don’t interfere with each others build artifacts.
Docs: The DerivedData has a full copy of your Swift Packages, including any documentation. Claude Code can do fast greps to verify syntax or find examples. In my CLAUDE.md I have a direct link for each important package:

- `DerivedData/train-timetable/SourcePackages/checkouts/swift-composable-architecture/Sources/ComposableArchitecture/Documentation.docc`
- When using Search/Grep/etc. tools, ignore anything in the /DerivedData folder by default unless specifically looking for build artifacts or code/docs for Swift Packages used by this project

Before making this change, add DerivedData/ as a line in your .gitignore file if it’s not already there.

Find the setting in Xcode Settings -> Locations. Set Derived Data to “Relative” and Build Location to “Unique”. It will report /DerivedData as the location.

DerivedData settings in Xcode

Document project file & scheme

Build commands use your project/workspace file location and scheme name. Claude Code can find these pretty easily with tools but it’s faster to document them in CLAUDE.md.

Finding scheme names from Xcode

Get simulators

I’ll assume you’ve already downloaded the iOS simulators and iOS runtime versions you’d like to use in the Xcode interface.

The usual command you’ll use to find the simulator you want produces a very long list, so it’s reasonable to cache your favorite simulator’s UDID so each new Claude Code session doesn’t need do this from scratch each time. “Cache” meaning note it in your CLAUDE.md file, add it as an environment variable, etc.

# Get all available simulators
xcrun simctl list devices available

     -- iOS 26.1 --
         iPhone 17 Pro (89F6D0BC-E855-4BF7-A400-9C19ED7A7350) (Shutdown)
         iPhone 17 Pro Max (F1FA81FA-ED32-40C4-BD78-753254D685AC) (Shutdown)
         iPhone Air (77702E5F-85F5-4997-BA14-BC8D8F639B84) (Shutdown)
		 ...
     -- iOS 26.2 --
         iPhone 17 Pro (DB0531E0-B47E-42AC-9AAB-FEB76D3D563A) (Booted)
         iPhone 17 Pro Max (0C54CF4B-8A45-450E-AB93-B800B97BD4DA) (Shutdown)
         iPhone Air (83BECA5F-7894-4705-B198-3DCAE0C4778E) (Shutdown)
         ...
     -- ...

At first, you’ll probably start with a single threaded workflow, having one preferred simulator booted and in use at a time. The below command will output just the UDID for the latest iPhone Pro with the latest installed iOS version.

# Get the UDID of the latest iPhone Pro (non-max) model with the latest available iOS version 
xcrun simctl list devices available | grep "iPhone.*Pro (" | tail -1 | grep -Eo '[A-F0-9]{8}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{12}'

Once you have more Claudes running in parallel, you’ll can ask each to find its own UDID by looking for a non-booted simulator before it starts building.

Note that even though some commands can be run with more vague identifiers like name, os, or “booted”, it’s much more reliable to select a UDID and use it across commands. For example: specifying "platform=iOS Simulator,name=iPhone 17 Pro" in the build command will pick any simulator that matches, which could be any iOS version. For the build command it’s not as big of an issue, but to ensure predictable runs I recommend using UDIDs only.

Install xcsift

xcsift is a companion parsing library for build output. You’ll be building a lot, and you don’t want to fill up your context with hundreds of lines of “file.swift built”. xcsift solves this by producing just the actionable errors and warnings in json.

brew install xcsift

I add the -w flag to also include warnings in the output. I recommend browsing the xcsift docs to find other flags you might find useful for your project.

Building

OK, after all that setup, we should have all the info we need to assemble the actual build command.

Claude Code will be able to derive and customize the exact build command you need. I recommend doing a quick session with Claude - the goal being to produce a single, always-working command you can document somewhere and use automatically in each future session.

# Example: working build command for a specific simulator
xcodebuild -project train-timetable.xcodeproj -scheme "train-timetable" -destination "platform=iphonesimulator,id=DB0531E0-B47E-42AC-9AAB-FEB76D3D563A" -derivedDataPath DerivedData -configuration Debug build 2>&1 | xcsift -w

-project: path to your xcodeproj file. Use -workspace if you use an xcworkspace file.
-scheme: scheme name we found above.
-destination: for simulator, we use "platform=iphonesimulator,id=$UDID" where id is the UDID of our favorite simulator instance. Note platform=iOS Simulator has the same meaning and also works.
-derivedDataPath: this is super important if you’ve moved the DerivedData to the project folder. Without this, the Xcode instance will be using a different directory and you’ll have super slow (clean) builds each time.
-configuration: Debug is the default, so you don’t usually need this flag. It’s better to be explicit though because this affects the folder where your app binary will be copied to (see step 2).
build: the actual build command
2>&1 | xcsift -w: combines stdout and stderr and pipes them both into xcsift so it has access to all output. -w tells xcsift to also show build warnings, not just errors.

It’s important to test your ideal build command to confirm:

It doesn’t force a clean build each time.
It produces a concise set of errors and warnings.
It doesn’t interfere with builds via Xcode; you should be able to build/install/run from Xcode, use other SourceKit features, etc. and not clear the build cache.

In my understanding for builds, the simulator UDID (or at least simulator name) does not affect the build artifacts or app binary. However, there’s a lot going on behind the scenes so for simplicity I recommend using the same simulator UDID across all build, install, & launch steps. You need to be careful if running multiple simulators from the same project folder (i.e. without git worktrees) because each build command will overwrite the app binary regardless of whether you run the build command with different simulator UDIDs.

Clearing DerivedData

When left to its own devices (literally), sometimes Claude will get frustrated when a build is failing continuously and it can’t figure out how to fix things. It will sometimes try to remove the entire DerivedData folder. This is a bad idea because 1. clearing DerivedData usually doesn’t fix the underlying problem and 2. it will temporarily break Xcode’s ability to read your Swift packages and you’ll need to restart Xcode to get everything working again.

After I got my build commands more streamlined, Claude stopped doing this as much. But I still have decently strict permissions, so when it does happen, the session will usually block on any rm command and I’ll get a chance to step in and reprimand it. If you run into this problem, you can dig deeper into a configuration-based solution and modify your permissions, add hooks, or add more to your CLAUDE.md. Just something to look out for.

Step 2: Installing & Launching

Building should streamline a lot of your Claude Code workflow. But I slept on the automated install & launch step for too long.

When properly set up with step 1, you should be able to wait for Claude Code to build its changes and return control to you. Then you can tab over to Xcode and hit “run without building” to handle the install & launch.

But when you’re doing build->install->launch dozens of times a day, it’s way more streamlined to check Claude’s session output then tab over to the simulator and tap through screens to test out the changes.

Prerequisites

Document the app binary location for your scheme

Ask Claude to find the location of the app binary produced by the build command.

Since my setup has DerivedData in the project folder and a build directory inside it, that’s where my app binary is: DerivedData/Build/Products/Debug-iphonesimulator/Eki Bright.app.

You’ll notice the folder: Debug-iphonesimulator, which corresponds to our build configuration of Debug from earlier and the platform iphonesimulator. If you go off the beaten path and want to try out a Release build for example, make sure to understand this relationship.

Also be cautious because you want to make sure the most recent build is what you’re installing and launching and looking at. And there’s nothing in the file name that will indicate that.

Find your bundle identifier

This will be in your Target’s general settings pane beside Bundle Identifier: com.twocentstudios.train-timetable

Bundle identifier in Xcode target settings

Installing the app on the simulator

Installing is copying over the app binary into a specific simulator’s storage. This step depends on the build step having produced an app binary.

The parameters in the install command are:

UDID: for least headaches this should be the same simulator UDID you specified in the build command.
path/to/My App.app: the app binary location specified by your scheme.

# Example: install a previously built app binary on the simulator by UDID
xcrun simctl install DB0531E0-B47E-42AC-9AAB-FEB76D3D563A "DerivedData/Build/Products/Debug-iphonesimulator/Eki Bright.app"

Launching the app on the simulator

Launching the equivalent of tapping your app’s icon in Springboard. It of course depends on the install step having copied over the app binary.

The parameters in the install command are:

UDID: the simulator UDID you specified in the build & install commands.
bundle id: apps are uniquely identified by bundle id after installation.

# Example: launch a previously installed app binary by bundle id
xcrun simctl launch DB0531E0-B47E-42AC-9AAB-FEB76D3D563A com.twocentstudios.train-timetable

Navigating by URL (bonus)

Depending on your app’s navigation structure and pre-existing support for universal links or App Intents, you can save yourself even more time by having Claude automatically navigate the app to the tab, sheet, or navigation destination you’re currently testing.

The full set of caveats is beyond the scope of this post. In my experience, adding Universal Links support without some caution can lead to giving Claude access to data or flows that are impossible for normal app users to see. It may also add maintenance burden for initializers that are only used during debug. Regardless, jumping through a dozen screens automatically can save you hours of unnecessary manual screen-clicking labor.

The parameters for the openurl command are:

UDID: the simulator UDID you specified in the build & install commands.
URL: the deep link URL your app knows how to process.

xcrun simctl openurl DB0531E0-B47E-42AC-9AAB-FEB76D3D563A "train-timetable://tab?name=search"

In my testing, it’s safe to have Claude to run the openurl command immediately after the install command (or even in the same line) without needing a sleep or otherwise waiting for the launch to complete.

Step 3: Reading Console & Log Output

With Step 1, Claude has access to the compiler’s evaluation of its code changes. We can also give Claude access to the console and log outputs so it can evaluate the runtime results.

There are two strategies: console output via print statements and log output via OSLog/Logger. I use both depending on the situation.

Depending on the strategy, we’ll either amend the launch command from step 2 or prepend a CLI command.

Blocking vs. non-blocking

Claude can do blocking and non-blocking for the console variant, and non-blocking-only for the log output.

Blocking means that the prompt input and Claude’s thinking will be suspended until you explicitly stop it or the default timeout (currently 10 minutes) triggers. It will print output from the command inline, but usually truncate portions.

Non-blocking means Claude will use its background capability to keep the command alive and retain access to its output but immediately move the command to the background so that the prompt input is available.

I recommend the blocking flow for when you want to add a few quick print statements to verify a limited (maybe less than 15 seconds) code execution flow that you, the human, are driving in the simulator and have Claude immediately evaluate the results inline. The amount of lines generated should be small, within 10s of lines.

I recommend non-blocking for all other scenarios, including:

when you want Claude to drive the simulator (discussed in step 4) while monitoring the output.
when you want to use Logger instead of print logging, probably for more permanent logging code in your codebase.
when you’re expecting to generate dozens or hundreds of lines of logs in a single run. In order to be smart about preserving the session context, you’ll want to write to a file and allow either a subagent to extract meaning from it, or have the primary agent use parsing tools to read only the relevant portions.

`--terminate-running-process`

Adding the --terminate-running-process flag to launch ensures idempotency by terminating any existing instance of your app and ensuring the app is always cold launched with the console output available.

Adding the --terminate-running-process is super important to the logging flow since you may not be rebuilding and reinstalling between launches.

When you don’t terminate an existing process, the app instance will stay in memory on the simulator. By default, the launch command will not relaunch the app if it’s already launched. This will happen silently. Critically, it will also not read any console output and Claude will get very confused about why nothing is being logged and it will start thrashing and making very dumb changes, ranging from adding more print commands to clearing DerivedData.

(This was my biggest roadblock in getting a reliable and robust debugging flow with Claude; please learn from my mistakes).

Launching the app on simulator and reading the output

Replace the launch commands from step 2 with any of the below variants depending on your use case.

Blocking console/print direct (when you know output volume is reasonable)

Relevant flags and parameters:

--console-pty: produce console print output.
--terminate-running-process: as discussed above, ensure the command actually runs.
UDID - the simulator UDID you specified in the build & install commands.
bundle id - bundle id of the target that produces the app binary.

# Example with simulator UDID and bundle id
xcrun simctl launch --console-pty --terminate-running-process DB0531E0-B47E-42AC-9AAB-FEB76D3D563A com.twocentstudios.train-timetable

Note: the flags --stdout --stderr do not work. Don’t use them. Use --console-pty instead.

Blocking console/print to file (safer for unknown or expected heavy output)

Relevant flags and parameters:

--console-pty: produce console print output.
--terminate-running-process: as discussed above, ensure the command actually runs.
UDID: the simulator UDID you specified in the build & install commands.
bundle id: bundle id of the target that produces the app binary.
output file path: the plain text file console output will be written to. Note: I write to a tmp folder within DerivedData to ensure Claude has access to the result without triggering unnecessary permissions dialogs.
2>&1: ensure stdout & stderr both end up in the file.

xcrun simctl launch --console-pty --terminate-running-process DB0531E0-B47E-42AC-9AAB-FEB76D3D563A com.twocentstudios.train-timetable > DerivedData/tmp/console.log 2>&1

Non-blocking console/print

Non-blocking requires using Claude Code’s run_in_background parameter on the Bash tool. This will produce a task_id that Claude can later use to get the output (from an implicitly created text file) and kill the task.

After running the Bash tool, the prompt will be unblocked and you can ask Claude to monitor the output or ask it do anything else you want.

The non-blocking flow requires a bit more ceremony; you’ll need to tell Claude when you’re done working with the simulator and it should analyze the results. It usually leaves the background task running (potentially writing log data to the output), so you’ll need to specifically tell it to stop.

The command itself is the same as the one from Blocking console/print direct.

Bash(
  command: "xcrun simctl launch --console-pty --terminate-running-process DB0531E0-B47E-42AC-9AAB-FEB76D3D563A com.twocentstudios.train-timetable",
  run_in_background: true
)

Command running in background with ID: b8e2ca5.

# *wait for next user prompt*

TaskOutput(task_id: "b8e2ca5")
KillShell(shell_id: "b8e2ca5")

Non-blocking Logger/OSLog

With only its training data, Claude knows how to use Logging by importing the OSLog framework. OSLog has strengths and weaknesses compared to console/print logging. You may already be using it in your app. I consider it more of a long term solution you’d add to your codebase alongside each feature and keep it up to date with any changes.

Giving Claude access to these logs is different from the print/console flow we just discussed.

The root command is xcrun simctl spawn.

spawn log stream only captures logs emitted while it’s running (not before). If you want logs starting from launch, always run it before the launch command.

Blocking on spawn log stream doesn’t make sense because you still need to launch the app. You should dispatch it directly to the background as non-blocking.

Relevant flags and parameters (Claude knows how to adjust these freely):

UDID - the simulator UDID you specified in the build & install commands.
--level: matches the log level in your code; debug, info, warning, error, etc.
--predicate: filters the firehose output the messages you’re interested in. Lots of options here depending on how you’ve defined Loggers and added log statements in your codebase.

After spawn log stream is dispatched to the background, you’ll need to launch the app with the launch command. You can choose a blocking or non-blocking launch command.

Note that the raw spawn log stream command is not actually monitoring the specific app process. You can start this early in your session, cast a wide net, and keep this running through your whole session, asking Claude to filter the relevant time periods from the output. I personally haven’t needed this flow though.

Bash(
  command: "xcrun simctl spawn DB0531E0-B47E-42AC-9AAB-FEB76D3D563A log stream --level=debug --predicate 'subsystem == "com.twocentstudios.train-timetable"')"
  run_in_background: true
)

Command running in background with ID: b8e2ca5.

Bash(xcrun simctl launch --terminate-running-process DB0531E0-B47E-42AC-9AAB-FEB76D3D563A com.twocentstudios.train-timetable)

# *blocking prompt until user escapes*

TaskOutput(task_id: "b8e2ca5")
KillShell(shell_id: "b8e2ca5")

Non-blocking console/print & Logger/OSLog

You can combine everything above and give Claude access to both console/print output and Logger/OSLog output. The launch command can be blocking or non-blocking, but the below example is non-blocking.

Bash(
  command: "xcrun simctl spawn DB0531E0-B47E-42AC-9AAB-FEB76D3D563A log stream --level=debug --predicate 'subsystem == "com.twocentstudios.train-timetable"')"
  run_in_background: true
) 
Command running in background with ID: b8e2ca5.

Bash(
  command: "xcrun simctl launch --console-pty --terminate-running-process DB0531E0-B47E-42AC-9AAB-FEB76D3D563A com.twocentstudios.train-timetable",
  run_in_background: true
)
Command running in background with ID: a792db1.

# *wait for next user prompt*

Step 4: Controlling & Viewing the iOS simulator

Giving Claude eyes and virtual fingers to see and control the iOS simulator is where we start to reach the avant-garde. At the current (end of 2025) model & harness capabilities things start to go off the rails pretty quickly. I wouldn’t expect great results from Claude at tasks related to manipulating the simulator like a human, but in certain scenarios, the benefits outweigh the costs.

Prerequisites

AXe

AXe is a comprehensive CLI tool for interacting with iOS Simulators using Apple’s Accessibility APIs and HID (Human Interface Device) functionality.

Claude can use AXe to manipulate the simulator through taps, swipes, button presses, and keyboard typing.

Under the hood, AXe uses Facebook’s idb CLI.

Install AXe with Homebrew.

Image Magick (optional)

ImageMagick® is a free and open-source software suite, used for editing and manipulating digital images.

Claude can use ImageMagick to do some post-processing on screenshots from the simulator.

Install ImageMagick with Homebrew.

FFmpeg (optional)

Claude can use the venerable FFmpeg CLI for advanced video manipulation use cases. You may not need it but there’s a good chance you already have it.

Reading from the simulator

In order to navigate the simulator beyond the universal links openurl use case we detailed above, Claude needs to be able to see the current state of the simulator.

There are 3 options for this:

Accessibility info - Claude can read a hierarchical text description of the current screen using accessibility info.
Screenshots - Claude can take a screenshot of the simulator and use the Read tool to access its multimodal capabilities.
Video - Claude can record a short video capture of the simulator, slice it up into frames, and read a few to assess an animation.

Accessibility info via `describe-ui`

The AXe command for getting the accessibility trace is:

axe describe-ui --udid SIMULATOR_UDID

  ...
      {
        "frame": {"y": 82, "x": 346, "width": 36, "height": 36},
        "AXLabel": "閉じる",
        "type": "Button"
      }
  ...

The output is a big JSON array.

I thought Claude would be better at understanding and navigation with text information than image information, but in practice it almost always ignored my instructions in CLAUDE.md to use describe-ui before the screenshot flow. Perhaps there’s something in the system prompt or it’s less efficient to hunt through all the text.

I also immediately ran into a reported issue in AXe and idb where describe-ui does not print tab or toolbar info, perhaps only from iOS 26. This makes it very difficult to deterministically do any sort of navigation from the root in many apps.

All this is to say that, at the moment, it’s more reliable to use screenshots.

Screenshots

Claude can use simctl to get screenshots.

Like the other commands, I prefer to write to a tmp folder within DerivedData.

Screenshots for most simulators are taken at 3x scale, but input taps and swipes are at 1x. For the dual purposes of 1. reducing the amount of calculation required to translate screen position to next tap position and 2. reducing the amount of image data that needs to be sent to and processed by Claude, I automatically resize all screenshots to 1x via magick.

xcrun simctl io DB0531E0-B47E-42AC-9AAB-FEB76D3D563A screenshot DerivedData/tmp/screen.png && magick DerivedData/tmp/screen.png -resize 33.333% DerivedData/tmp/screen_1x.png

Video

Reading live or even recorded video is currently beyond Opus 4.5’s capabilities. I’m guessing this will be a supported flow sometime in 2026, but until then analyzing video output of the simulator is still at proof-of-concept maturity.

While I was debugging a tricky animation, I gave Claude some leash to test whether it could:

start recording a short clip immediately before a tap.
stop the recording after 2 seconds.
use FFmpeg to grab 5 or 6 frames spaced out across the video.
read the frames and analyze the motion.

It sort of worked? Not really? If you have a use case, you can try experimenting more with this flow. For now I’d consider the actual animation analysis a human-only endeavor. But Claude can still help get the simulator staged up to the start screen.

I believe I used this AXe command as a Claude Code background task:

axe record-video --udid DB0531E0-B47E-42AC-9AAB-FEB76D3D563A --fps 30 --output DerivedData/tmp/recording.mp4

Manipulating the simulator with taps and swipes

AXe has a variety of tap and gesture commands. Claude can tap on points or accessibility labels.

# Tap at logical coordinates (use frame center from describe-ui)
axe tap -x 201 -y 297 --udid DB0531E0-B47E-42AC-9AAB-FEB76D3D563A --post-delay 0.5

Without additional guidance, Claude gets confused about which scroll command maps to what logical direction.

# Scroll (named by finger direction, not content direction)
# scroll-up = finger UP = content UP = see content BELOW = triggers .onScrollDown
# scroll-down = finger DOWN = content DOWN = see content ABOVE
axe gesture scroll-down --udid DB0531E0-B47E-42AC-9AAB-FEB76D3D563A --post-delay 0.5
axe gesture scroll-up --udid DB0531E0-B47E-42AC-9AAB-FEB76D3D563A --post-delay 0.5

It’s useful to note the swipe-from-left-edge gesture because it’s the quickest way for Claude to pop back a level in a NavigationStack.

# Edge swipes (for back navigation, etc.)
axe gesture swipe-from-left-edge --udid DB0531E0-B47E-42AC-9AAB-FEB76D3D563A --post-delay 0.5

Strategies for increasing tap accuracy

The most significant source of indeterministic behavior is in Claude’s ability to accurately measure of coordinates on screen. In other words, it can’t read an image and always find the center point of a button. This means there is plenty of opportunity for situations like:

Claude reads a screen and wants to tap a button.
Claude makes a bad guess and taps above the button.
Claude reads the screen again. There was no change.
Claude makes another bad guess and taps below the button.

Claude’s only feedback about whether its tap was successful is based on its next screenshot. This can lead to situations where it gets irrecoverably lost while navigating your app:

Claude reads a screen and wants to tap a button.
Claude makes a bad guess and taps above the button, hitting a completely different button.
Claude reads the screen again.
Claude sees it’s on a different screen than expected and becomes confused.

I came up with an experimental flow to try to improve Claude’s accuracy, but:

It slows down the entire process by 2x.
By the time Claude realizes it needs to use the experimental flow, it’s already too far lost to recover.

Regardless, my flow, also using ImageMagick, is to make Claude draw a red circle on a screenshot in its targeted tap location before actually performing the tap:

Take screenshot and resize to 1x (so pixels = points):

xcrun simctl io DB0531E0-B47E-42AC-9AAB-FEB76D3D563A screenshot DerivedData/tmp/screen.png && magick DerivedData/tmp/screen.png -resize 33.333% DerivedData/tmp/screen_1x.png

Read the 1x image and estimate target element center in points

Verify guess by drawing a red box at those coordinates:

magick DerivedData/tmp/screen_1x.png -fill none -stroke red -strokewidth 2 -draw "rectangle $((X-30)),$((Y-30)) $((X+30)),$((Y+30))" DerivedData/tmp/screen_marked.png

Read marked image to check if box is on target
If missed, adjust coordinates and repeat from step 3
If correct, tap at the verified coordinates

Tap accuracy verification flow - original screenshot, marked target, and result after tap

Putting it all together

So far in practice, I’ve used this capability alongside universal links to fix and verify simple visual bugs. I have Claude craft input to reproduce the error and find the screen where it occurs, take a “before” screenshot, implement the fix, build/install/launch, then find the same screen, verify the fix, and take an “after” screenshot.

It takes way longer for Claude to do than me, but it’s mostly tedious work, and I’m usually in another tab working on a plan with another Claude. When I come back and see the before and after screenshots alongside the code change, I can feel confident in Claude’s work.

Step 5: Building, Installing, Launching, Reading Output on a Physical Device

Finally, for those Apple SDKs that only work on device, for ensuring observed buggy behavior isn’t just a simulator quirk, or just to get a more realistic look at our apps in context, we can implement steps 1, 2, and 3 on a physical device. Unfortunately, as far as I can tell, there’s no way to control a physical device via CLI tool, so step 4 is out reach for now.

However, building, installing, launching, and logging can still save some time and annoyance during iterative debugging sessions. It’s especially useful to have Claude help analyze logs for (underdocumented) frameworks like Core Location that behave wildly different on a real device than on the simulator.

Below is a collection of tested CLI commands for doing all the above tasks on a physical device.

Note that in my testing all the relevant commands below work equally for devices on the same network and devices connected directly to your Mac via USB.

Prerequisites

Get devices

The device Name and Identifier are both important for on-device debugging. State will be available when Wi-Fi debugging is available, and connected when directly connect via USB.

# Get all connected devices
xcrun devicectl list devices

Name          Hostname                      Identifier                             State                Model                                
-----------   ---------------------------   ------------------------------------   ------------------   -------------------------------------
CT's iPhone   CTs-iPhone.coredevice.local   ABCDEF01-1111-5555-AAAA-F7D81A900001   connected (no DDI)   iPhone 14 Pro (iPhone15,2)           
CT’s iPad     CTs-iPad.coredevice.local     ABCDEF01-2222-6666-BBBB-F44A19F00002   available            iPad Pro (11-inch) (iPad8,1)         
CT’s iPad     CTs-iPad-1.coredevice.local   ABCDEF01-3333-7777-CCCC-D33889000003   unavailable          iPad mini (5th generation) (iPad11,1)

Some example use cases for parsing out values in one go:

# Get the Identifier of the first accessible iPhone (WiFi or USB)
xcrun devicectl list devices | grep "iPhone" | grep -E "(available|connected)" | head -1 | grep -Eo '[A-F0-9]{8}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{12}'

ABCDEF01-1111-5555-AAAA-F7D81A900001

# Get the name of the first accessible iPhone (WiFi or USB)
xcrun devicectl list devices | grep "iPhone" | grep -E "(available|connected)" | head -1 | awk -F'  +' '{print $1}'

CT's iPhone

Build for device

Build commands are the same as those for the simulator, except platform=iOS instead of platform=iphonesimulator/platform=iOS Simulator.

Use the name of your target device from the list devices command.

# Using device name
xcodebuild -project train-timetable.xcodeproj -scheme "train-timetable" -destination "platform=iOS,name=CT's iPhone" -derivedDataPath DerivedData build 2>&1 | xcsift -w

Note: platform=iphoneos does not work. id instead of name does not work.

Install on device

Install commands use devicectl but are the similar to those for the simulator, except --device should use the device ID, and the build product directory should use Debug-iphoneos instead of Debug-iphonesimulator.

xcrun devicectl device install app --device E7E3E660-9E7A-5814-8BBB-F7D81A965CEB "DerivedData/Build/Products/Debug-iphoneos/Eki Bright.app"

Launch on device

The vanilla launch command is below. I again recommend using --terminate-existing, the device equivalent of the simulator’s --terminate-running-process.

xcrun devicectl device process launch --device E7E3E660-9E7A-5814-8BBB-F7D81A965CEB --console --terminate-existing com.twocentstudios.train-timetable

Blocking console/print capture on device

For console/print capture, use the launch command above with the --console flag. It works over USB and Wi-Fi.

xcrun devicectl device process launch --device E7E3E660-9E7A-5814-8BBB-F7D81A965CEB --console --terminate-existing com.twocentstudios.train-timetable

Non-blocking console/print capture on device

# Use run_in_background: true on Bash tool
# Works over USB and WiFi
Bash(
  command: "xcrun devicectl device process launch --device E7E3E660-9E7A-5814-8BBB-F7D81A965CEB --console --terminate-existing com.twocentstudios.train-timetable",
  run_in_background: true
)

Command running in background with ID: b8e2ca5.

# *wait for next user prompt*

TaskOutput(task_id: "b8e2ca5")

Logger/OSLog capture on device (requires manual sudo)

A downside of OSLog is that it requires sudo and Claude Code can’t use sudo commands directly. There are presumably some ways to give Claude this capability in more a dangerous fashion. But a safer workaround for now is for you, the human, to run the below commands in another terminal tab. Claude can give you the full command to copy/paste into the other terminal.

Another downside is that the process is slow and produces lots of logs.

These commands will produce groups of files that Claude can read.

Note that the log collect is of everything on the device in the past, and can quickly balloon to gigabytes of storage. The command to collect even the last 2 minutes of logs can take about 30 seconds to complete on Wi-Fi.

Content filtering with --predicate is not supported: Warning: --predicate is ignored when collecting from attached device.

Claude will handle filtering while reading/analyzing with log show --predicate ....

The most logical way to use this is to:

have Claude build, install, and launch the app on your device (ensure it’s unlocked), and have it note the start time.
tap around and do the testing you need in order to generate the logs you want.
have Claude give you the sudo log collect command with a start time a little before the launch time.
run the sudo log collect command in a separate terminal window, enter your password, wait ~1m for it to finish.
ask Claude to analyze the log archive.

# Human user must run these commands in another terminal tab (Claude Code can't provide sudo password)

# `--last` collects from N minutes before the command was run
sudo log collect --device-name "CT's iPhone" --last 2m --output DerivedData/tmp/device-logs.logarchive

# `--start` collects from the specified start time until the command was run
sudo log collect --device-name "CT's iPhone" --start "2025-12-30 16:11:00" --output DerivedData/tmp/device-logs.logarchive

# Then analyze with log show (Claude can do this)
log show DerivedData/tmp/device-logs.logarchive --predicate 'subsystem == "com.twocentstudios.train-timetable"'

Note --device-udid does not work, use --device-name instead - log: failed to create archive: Device not configured (6).

Note --size does not (seem to) work either.

Final thoughts

How to parameterize names, ids, etc. for these commands

So far, I’ve just been hardcoding these commands with my favorite simulator UDID and project path into my CLAUDE.md. When a new version of Xcode comes out I ask Claude to update all mentions of the UDID to the most recent simulator version and it only takes a minute. Hardcoding these values leaves the least room for hallucination. When running these commands dozens of times a day, you really want consistency.

Other ways to handle this would be:

set environment variables at some level.
add a start hook to have Claude fill in the environment variables fresh for each session.
set up another layer of orchestration that handles the pool of simulators and dispatches an ID to each new Claude instance that requests one.

These are all beyond the scope of this post. If you work primarily on one project with others, you probably already have some tooling for specifying the Xcode version, etc.

Why not include traditional testing?

Arguably, TDD was the original “closing the loop” in software development. TDD has never caught on in the iOS world.

I dabbled with an actual Swift Testing-based testing flow for another recent project, and even wrote about another experimental system a few months ago in Giving Claude Code Eyes to See Your SwiftUI Views that used snapshot testing. What I found was although tests are great for verifying correct behavior over the long term, in the short term they are super slow on iOS:

Installing and launching requires instantiating a brand new simulator for each run (e.g. Clone 1), which takes a long time.
All builds are clean builds (this could have just been a fluke in my setup at the time though).
Swift Testing does not output failures in a way that Claude can read and iterate on (again, potentially solvable).

Admittedly, I didn’t spend as much time debugging these flows. Hopefully someone else will fill in the blanks for testing and write this guide.

Don’t sleep on `simctl`

The simctl CLI we’ve used throughout this post has a ton of other abilities that Claude can use to make our lives easier. This includes adding images to Photos, changing the system time, changing the system language, resetting privacy, resetting the keychain, and many more.

Ask Claude to configure your simulator on the fly instead of clicking through settings with your mouse.

What does the future hold?

I’m honestly not sure how long the hard-won knowledge in this post will be relevant, given the pace of model & harness capabilities. Peter Steinberger already says Codex is good enough and doesn’t need any additional guidance about build commands or working with the simulator.

I can definitely see a world where Claude Code has a live feed of the simulator output it can process and react to at 60 fps, tapping and swiping with full accuracy. This is probably what’s missing in fully closing the development loop on iOS. Doing the same for a real device hopefully isn’t close behind.

At that point though, I’m not sure what else about development will have changed.

Going forward

Most of the material in this guide has been slowly compiled over the month in my various CLAUDE.md files. It was great getting a chance to formalize it even if I can’t make a quickly installable Plugin or Skill to share (hopefully you understand why after reading the post). I’m looking forward to seeing how far I can take each of these steps in the near future.

Corrections

Please reach out if you find any corrections or can contribute any additional knowledge or edge cases.

Shinkansen Live - Developing the App for iOS

Thu, 25 Dec 2025 07:41:42 -0600

In my last post I introduced the motivation and feature set of Shinkansen Live, my latest iOS app. I encourage you to read that one first to learn about what the app does.

Shinkansen Live app icon

In this post, I’ll discuss a few of the interesting development challenges I faced during its week of development from concept to App Store release.

Overall development strategy
OCR and parsing the ticket image
Live Activities
Animations
AlarmKit
Localization
Dynamic Type
VisionKit Camera

Overall development strategy

I created an Xcode project myself with Xcode 26.1 (later switching to Xcode 26.2), then added the TCA package.

Then, I set off to work using Claude Code with Opus 4.5. I started by having it lay out the SwiftUI View and TCA Feature without any logic. Then I built out the rest of the infrastructure around getting the input image, doing OCR, parsing the output, and displaying the results. I’ll go through more of the history later on in the post.

Early view layouts during initial development

OCR and parsing the ticket image

The most difficult part of getting this app to production was the ticket OCR & parsing system. This system went through the most churn over the week, partially due to expanding scope and partially due to my expanding understanding of the problem space.

My initial thought during the prototyping stage was to target only the “ticket” screenshot from Eki-net you get after purchase. It looks something like this:

Example Eki-net ticket screenshot

In theory, it’d be reasonable to limit the app’s input space to app screenshots from Eki-net (JR-East) and SmartEX (JR-central). Even including web browser screenshots wouldn’t be that much more burden on an OCR-based system. But later on in the project when I’d decided I was happy enough with the prototype that I wanted to target a production release on the App Store, I started thinking about how it would make marketing much harder to say “only works on screenshots” and not physical tickets.

OCR via VisionKit alongside manual parsing has a lot of upsides:

Free for user & developer
Fast
Multilingual
Privacy baked in
No network usage
Relatively mature: less risk of accuracy churn

In theory, multimodal LLMs can handle OCR and understanding more variations of tickets layouts and bad lighting. In fact, while I was writing the parser, I used Opus 4.5 to read the test ticket images in order to create the ground truth test expectation data.

My issue with prototyping with LLMs further was that they had essentially the opposite pros and cons as the VisionKit system:

Unknown, but some ongoing cost (meaning I’d need to come up with a monetization strategy before release)
Unknown which level of model would correctly balance accuracy, cost, and speed over the short term.
Requires network access
Requires sending photo data off device (not hugely private, but still)
Different outputs for the same input

What I didn’t explore directly was using the Apple Foundation model at the parsing layer. The current Apple Foundation model in iOS 26 has no image input API, but I could perhaps prompt it to take the raw output of the VisionKit model and try to make sense of it. My instinct is that this would be a waste of time, but still worth keeping on the table.

Overall strategy

While working on this, I honestly wasn’t thinking strictly in terms of prototype & production. From the first spark of idea I had an understanding of what the overall UX flow of the app would be. It was mostly getting to an answer of “is this feasible to productionize in a couple days?” while still being flexible on the scope of what production-ready meant.

That meant that I started by adding my ticket screenshot to the project, giving Claude my overall strategy, and having Claude create the OCR & parser system that output results directly into the UI.

OCR prototype

The first prototype parser supported just the two screenshots I had of Eki-net tickets (one from the morning of my trip; another from a similar trip a few months ago).

The implementation set up a VNRecognizeTextRequest in Japanese language mode, read out the highest ranking results into several lines of text ([String]), then fed that to Claude’s homegrown parser that pulled the ticket attributes out of that glob of text mostly using regex.

Since the input was from a perfectly legible screenshot, there was no issues with the VisionKit part nor the parser.

OCR for real Shinkansen ticket images

As soon as I tested the system on a real Shinkansen ticket in a photo, the system fell apart.

I searched through my personal Photo library for as many Shinkansen tickets as I could find. I googled for more. I started with 4 images (and later in development I ended up with about 10 images).

At first I was simply trying to naively patch out the parser with Claude. To do this, I set up a unit test system where I’d do the VisionKit request for each image once and write the resulting [String] data structure to disk. Then each unit test would read that data in, run the parser, and compare the expected ticket structure to the test result.

Unintuitively, the standard Swift Testing unit test setup was actually the less efficient way to iterate on this. Claude was having a lot of trouble reading the detailed test failure information after it ran xcodebuild in the command line. Each build & run & test iteration needed to boot up a fresh simulator, install the app, run the test, then tear down the simulator.

Instead, a built a @main App-based test harness that:

temporarily disabled the real UI.
ran the test code on onAppear.
printed the test results via print statements.
called exit(0) when finished.

For quickly prototyping, iterating, and understanding the scope of the problem space, this solved all the issues with the Swift Testing setup. The same booted simulator was reused on each run. DerivedData and build artifacts were reused, so builds were fast. Claude had no trouble reading print statements from the console output.

I let Claude run in its own loop for a while to see what it could and couldn’t improve with the parser based on the limitations of our system.

Claude found several underlying limitations with the VisionKit setup that were unsolvable at the parser level.

For example, concatenating all the text recognition objects into [String] was done somewhat naively by comparing y-coordinates. If the y-coordinates of objects were within a certain range, they were assumed to be on the same line. When the ticket was tilted, this strategy was interleaving text.

Additionally, some numbers and letters were just flat out being interpreted incorrectly. A 37 was read as a 30. CAR was read as CDR.

Off running on its own, Claude was trying to special case as much of these failure cases it could to get the tests passing.

Example test ticket image that produced the Japanese-only VisionKit parsing output shown below

CC制
東
3月18日（
はくたから5り考
¥3,380
新幹線特急券
京
→
8:41発）
軽井
（9:4着）
7号車
3番B席
R001
2025.-3.18東京北乗FN7（2－）
50159-01
沢
0

Comparing some of the fields, you can see the OCR output taken naively is not great:

The departure time is there but the arrival time is 9:4 instead of 9:43.
The arrival station’s 軽井 and 沢 are split up and should be 軽井沢.
The departure station’s 東 and 京 are split up and should be 東京.
The train name and number are a mess: はくたから5り考 instead of はくたか 555号

Using spatial data

The OCR part of the system was trying to abstract away the spatial parts from the parser. Looking at the raw data, my intuition was that the spacial data could be useful within the parsing layer. Instead of passing [String] between layers, I was now passing:

struct TextObservation: Equatable, Sendable {
    let text: String
    let x: Double
    let y: Double
    let width: Double
    let height: Double
}

Each field parser within the parsing system could decide for itself how to use the spacial data. This is especially useful considering how important anchor values are. For example “発” (indicating a departure time) and “→” (indicating the station to the left is the departure station).

This improved things a bit for 5 ticket images, but after adding another 5 and going all in on English ticket support, there were plenty more edge cases to consider.

Transforms

Only after seeing some wonky positioning of the text boxes in my loading animation (see below), I realized that I wasn’t accounting for the image transform properly when converting the input UIImage to CGImage as input to VNRecognizeTextRequest.

extension CGImagePropertyOrientation {
    init(_ uiOrientation: UIImage.Orientation) {
        switch uiOrientation {
        case .up: self = .up
        case .upMirrored: self = .upMirrored
        case .down: self = .down
        case .downMirrored: self = .downMirrored
        case .left: self = .left
        case .leftMirrored: self = .leftMirrored
        case .right: self = .right
        case .rightMirrored: self = .rightMirrored
        @unknown default: self = .up
        }
    }
}

Levenshtein distance

Like our previous example of CDR instead of CAR, VisionKit was reading strings like TOKIO instead of TOKYO. For these parts, I figured calculating Levenshtein distance from known strings was the right strategy. The Shinkansen system is large but not so large that I couldn’t ingest the station name values and the other known strings like JAN, FEB, etc.

One strategy I haven’t tested yet is using VNRecognizeTextRequest.customWords with the full dictionary of station names, etc. to see if that eliminates the need for using Levenshtein distance at all.

English and Japanese mode OCR

For some background, tickets can be printed in an “English” variant that includes a mix of English and Japanese text. If you buy from a ticket vending machine and complete the purchase in English mode, it’ll print an English ticket. Similarly, if you buy from a human ticket vendor at the counter, they will print your ticket in English variant if you speak English to them.

Example of an English variant Shinkansen ticket

At the VisionKit layer, I was first using VNRecognizeTextRequest in Japanese-mode only.

I tried expanding a single instance to include both English and Japanese text. But checking the raw results, a dual language setup severely impaired its abilities.

For a while, I had a dual parsing system that would check for a few English strings, and if any were found, it would assume the ticket was an “English ticket” and run a separate English VNRecognizeTextRequest and return those results. This didn’t work well for a few reasons:

The Japanese VNRecognizeTextRequest was surprisingly bad at reading numbers compared to the English one.
Deciding at the VisionKit layer which text results should come from the English request didn’t make a lot of sense conceptually if I wanted to keep the majority of the logic in the parser layer.

Therefore, I decided to run both the English and Japanese VNRecognizeTextRequests on every input and provide all the results to the parser.

struct BilingualOCRResult: Sendable, Equatable, Codable {
    let jp: [TextObservation]
    let en: [TextObservation]
}

I was also previously discarding the confidence score (0.0...1.0) that VisionKit provides with each observation. This score was actually different in the parallel observations for English and Japanese in some cases, especially number recognition. I added the confidence score to the output so the parser could use it.

struct TextObservation: Equatable, Sendable, Codable {
    let text: String
    let x: Double
    let y: Double
    let width: Double
    let height: Double
    var confidence: Float
}

Unparseable fields fallback UX

At this point in the development of the parser, I was pretty certain reading photos of tickets was never going to be reach 100% accuracy for every field.

I took a break from working on the parser to implement editing for every field in the ticket UI. This meant that users could manually recover from parsing errors and omissions.

Editing screen with editable ticket fields

Not needing to reach 100% accuracy in the parser while still ensuring the user gets value out of the system as a whole opened up a more reasonable strategy for the parser.

Defense-in-depth parsing system

After tweaking more and more of the VisionKit layer, I had to regenerate the VisionKit output test data for each test ticket image, and then essentially rewrite the parser layer.

This time, the strategy was to:

Pass in the full English and Japanese results including relative x and y coordinates to each field.
Create a sub-parser dedicated to each field of the ticket that needed to be parsed.

Within each sub-parser, my strategy was to start with the best case scenario of input data quality, then step-by-step keep loosening the guidelines to account for more unideal cases that had come up in the test data, then finally falling back to returning nil for that field.

OCR testing system

Claude iterated on the field parser implementations for an hour or two, one at a time, checking the test output to ensure there were no regressions along the way.

At a certain point all the tests were passing and as much as I wanted to keep finding test data and tweaking the parser, I knew I had to move on.

Live Activities

During development I had to keep reminding myself that the whole point of this endeavor was to have a slick (read: useful) Live Activity.

Live Activity in Dynamic Island compact, expanded, and lock screen views

My design process was quick and to-the-point:

list out all the Live Activity contexts: lock screen, dynamic island compact leading, compact trailing, minimal, and expanded.
consider all the ticket info I had available from the parser output.
consider all of the above for the “before” and “during” trip phases (if I knew I had more accurate control over Live Activity update timing, I might have divided these phases up even further).

In lock screen and expanded contexts, you can mostly display everything you want. The challenge is in aesthetics and visual hierarchy like in any design.

Lock screen Live Activity showing trip details

When dealing with the compact and minimal contexts, you really do only have the equivalent of about 6 very small characters to work with, and 2 lines if you want to push your luck. If you try to fill the entire available space of the Dynamic Island on either side, you’ll lose the system clock which is no go for my use case.

Compact Live Activity in Dynamic Island

When I thought about it, the most important of all the contexts was the compact leading and trailing in the before trip phase. This is when the information is most needed at a glance.

I stacked in departure time and train number in the compact leading. This information is used to decide when to go to the platform and which platform to go to.

I stacked the car and seat number in the compact trailing. These are used to decide where to line up on the platform and of course where you’ll sit.

View guidelines and tips

Designing for Live Activities is painful. There are significantly more constraints to the SwiftUI View system than in normal app contexts. Most are undocumented. Some quirks can be teased out in the SwiftUI Preview if you’re lucky. Others only appear on the simulator or a real device.

I have a couple guidelines and tips I follow for Live Activities.

Use non-semantic font sizes for compact and minimal

The system ignores Dynamic Type settings in the compact and minimal Dynamic Island contexts. Using point sizes directly gives more flexibility while designing in the very limited space.

In the lock screen and expanded contexts, there’s limited Dynamic Type support (4-levels total), so it’s still worth using semantic fonts as usual (e.g. .headline, .title3).

For dynamically updating times, prepare to spend a lot of time in trial and error

Maybe someday I’ll write a full explainer post on which countdown-style Text fields are supported. In short, if you want dynamically updated fields in any part of your Live Activity, your formatting options are limited and underdocumented.

A couple configurations I used:

// `44 min, 23 sec` or `1 hr, 52 min`
Text(attributes.departureTime, style: .relative)

// Centers `Departing in N min, M, sec` due to `.relative`'s implicit `maxWidth: .infinity`
HStack(spacing: 4) {
    Text(String(localized: "widget.departing-in", comment: "Footer label shown before departure"))
        .frame(maxWidth: .infinity, alignment: .trailing)
    Text(attributes.departureTime, style: .relative)
}

// Linear progress view with no label
ProgressView(timerInterval: attributes.departureTime ... attributes.arrivalTime, countsDown: false)
    .progressViewStyle(.linear)
    .labelsHidden()
   
// `60:00`
Text(
     timerInterval: (Date.now)...(Date(timeIntervalSinceNow: 60*60)),
     pauseTime: Date(timeIntervalSinceNow: 60*60),
     countsDown: true,
     showsHours: false
)

As noted above, any Text using: Text(Date(), style: .relative) will expand to fill its full width.

It’s frustrating just thinking about this again. I basically just banged my head against the wall until I landed on a design I felt embarrassed but comfortable shipping.

Clamped width custom Layout

I use this custom Layout judiciously in the Dynamic Island.

It makes the underlying view’s frame collapse to fit its ideal width, but clamped to a maximum value.

I want the compact or minimal context to be as narrow as possible. I want short input text to result in a very narrow Dynamic Island layout. I want longer input not to expand beyond a certain width; and even more, I want to use the minimumScaleFactor modifier to further shrink the text size once that maximum width is reached.

private struct SingleViewClampedWidthLayout: Layout {
    var maxWidth: CGFloat

    func sizeThatFits(proposal: ProposedViewSize, subviews: Subviews, cache: inout Void) -> CGSize {
        guard let subview = subviews.first else { return .zero }
        let idealWidth = subview.sizeThatFits(.init(width: nil, height: proposal.height)).width
        let width = min(idealWidth, maxWidth)
        return subview.sizeThatFits(.init(width: width, height: proposal.height))
    }

    func placeSubviews(in bounds: CGRect, proposal: ProposedViewSize, subviews: Subviews, cache: inout Void) {
        guard let subview = subviews.first else { return }
        subview.place(at: bounds.origin, proposal: .init(bounds.size))
    }
}

private struct ClampedWidth: ViewModifier {
    let maxWidth: CGFloat

    func body(content: Content) -> some View {
        SingleViewClampedWidthLayout(maxWidth: maxWidth) {
            content
        }
    }
}

extension View {
    @ViewBuilder func clamped(maxWidth: CGFloat) -> some View {
        modifier(ClampedWidth(maxWidth: maxWidth))
    }
}

Updating in background

Live Activities can usually only be updated via remote Push Notification. But if your app gets a chance to wake up and run in the background, it can also issue updates to the Live Activity.

One of the few reliable ways to have your app woken up in the background regularly is to use significant location updates from Core Location. In order to have your app get background time you need to:

Be approved by the user for Always Location Services permission.
Be approved by the user for WhenInUse Location Services permission AND one of the following
- Have an active Live Activity OR
- Start a CLBackgroundActivitySession (that essentially creates a default Live Activity)

In the case of Shinkansen Live, I have two phases for the Live Activity:

before the train departs
after the train departs

The time of that change over is pretty reliably scheduled. But Live Activities have no update schedule like Widgets do (for some reason).

To update the Live Activity after the train departs I could:

Have the user tap a button in the lock screen or expanded Live Activity that triggers an intent to wake up the app in the background and update the Live Activity.
Hope the user opens the app on their own.

Both of these options are unideal, so instead I ask for WhenInUse Location Services permission and start a monitoring for significant locations. One of these will be fired not long after the train departs (within about 1km or 5 minutes). That trigger will open the app in the background, update the Live Activity based on the current time, then go back to sleep.

Persisting the trip

There’s an edge case I wanted to handle with the Live Activity lifetime.

If the system kills the app in the middle of a trip, the Live Activity in theory should continue uninterrupted since it’s an App Extension. But in the case of Shinkansen Live, I’m expecting to update the Live Activity while the app is backgrounded. This means there’s a potential flow where:

The app is in the background with the Live Activity running.
The app is killed by the system.
The Live Activity continues to run.
The system cold launches the app in the background.

At this point, I could decide to query the Live Activities framework to see if there’s a Live Activity running and if so, restore the ticket model layer and UI. However, I prefer not to treat the Live Activity as the source of truth for the model layer.

I added support with the Sharing library to persist the ticket model automatically on changes. On the above flow, I use the persisted ticket model to restore the UI and Live Activity and AlarmKit state, ensuring the ticket data is still valid.

Handling dismissal

One final bit of UX that’s not mission critical but is very user friendly is to respond to user-initiated Live Activity dismissals from the lock screen. If the user swipes from right to left on your Live Activity, the system dismisses it. When your app next runs, it will receive an update from Activity.activityStateUpdates stream (if you’re monitoring it).

If the app detects a user-initiated dismissal, I consider that their trip has ended, clear the ticket, and go back to the app home screen. There’s an argument that it’d be safer to simply toggle the Live Activity, but since my app’s only purpose is to show a Live Activity, I don’t think it makes sense to build in more complexity to keep multiple states.

Animations

The focused nature of this app allowed me a bit more breathing room to experiment with custom screen transitions and multi-stage animations.

Root level transitions

At the bare minimum, I usually try to use a default opacity transition for root level views when they aren’t covered by system transitions like a navigation push or sheet presentation.

For Shinkansen Live, I added a little bit of extra scale effect to the usual opacity transition of the initial screen, both on cold launch and when returning from the trip screen.

For the trip screen, I first animated the card down from the top with some scale and opacity, then fade in the other sections with some scale.

Scanning animation

The most fun was doing the scanning animation. Once the selected image is downloaded and displayable, I animate it in with a bit of 3D effect. Then I use the coordinate results of the text observations to animate those boxes onto the image.

This animation serves a few purposes in my opinion:

Visually expresses to the user what the app is actually doing.
Buys time for the parser to do its job.
Feels fun and playful in a way that is motivating for users to want to go through the trouble of submitting their ticket image.

My final bit of (self) user testing made me realize that even though I’d built in a way to update ticket values that were missing or erroneously parsed, I had no in-app UI for actually doing the field checking. Depending on their input source, the user would have to hold up their physical ticket next to the app’s virtual ticket to double check the fields. Or if they’d used a screenshot, they’d have to flip back and forth between Photos app.

As my last big task, I added support for showing the original image inline with the ticket in a modal overlay.

This setup uses matchedGeometryEffect and was a nightmare to work through. In the end it’s not perfect, but the speed conceals some of the jankiness. matchedGeometryEffect has a lot of undocumented incompatibilities with other modifiers, so it was just hours upon hours of reordering modifiers and building and running to check what had changed. I came out of the experience with little new demonstrably true observations I can share here, unfortunately.

Card dragging

Whenever there’s a card-looking UI on screen, I want it to be interactable even if there’s no real gesture that makes sense.

I created a custom drag gesture with rubberbanding that allows the user to drag the ticket a little bit in any direction. When released, it snaps back with a custom haptic that mirrors the visual.

AlarmKit

AlarmKit is new in iOS 26 and I thought it might be a good fit for Shinkansen Live’s use case.

Alarm setting in trip screen and full screen alarm notification

The integration was mostly straightforward, but it added another layer of complexity to the reducer implementation to ensure that it was added, changed, and removed for all the relevant cases:

Arrival time exists or doesn’t exist.
Arrival time is updated manually by the user.
System time is too close to arrival time to set an alarm.
Journey is ended by the user before arrival time.
Live Activity is dismissed from the lock screen.

And more.

Permission dialog

A last minute annoyance with AlarmKit was testing localization: I found a bug where the localized text for the AlarmKit permissions dialog was not being used on iOS 26.0. But the bug was fixed for iOS 26.1. And no, the bug nor the fix were mentioned in any official SDK release notes.

Localization

One of my favorite usages for coding agents is doing Localization setup. Note I’m specifically not talking about LLMs doing the actual translation, but instead:

doing the initial conversion from inline strings to string keys e.g, loaded.end-journey-button.
adding localizer comments to each string key e.g. Button to end the current journey.
maintaining default values for when string interpolations are required e.g.

Button(
    String(
        localized: "loaded.arrival-alert.status.minutes-before",
        defaultValue: "\(mins) min before",
        comment: "Status showing minutes before arrival (for menu items)"
    )
) { ... }

The initial conversion uses a custom markdown document with some basic rules. It takes about an hour for the first run and then a few more passes with a human in the loop to ensure the xcstrings file is clean.

Dynamic Type

The app still lays out pretty well with most levels of Dynamic Type. I only use semantic font qualifiers. All content is in a scroll view that’s usually fixed.

Landing screen with various Dynamic Type sizes

Trip screen with various Dynamic Type sizes

VisionKit Camera

I’m using VNDocumentCameraViewController as the integrated camera view for scanning. The UX is a little weird because there’s no way to limit the input (output?) to one photo. The result is the user can take a bunch of photos of their ticket before they tap “Done” and the app will only read the first.

Project stats

By the numbers

Development time: ~6 days including App Store materials
Lines of Swift (excluding tests, static data, etc.): 8,388

Daily Devlog

Dec 16, 2025 — Initial layout & OCR foundation
Dec 17, 2025 — Live Activities, AlarmKit, failure states
Dec 18, 2025 — Bilingual OCR, loading animation, app icon, localization
Dec 19, 2025 - Parser rewrite, settings view
Dec 20, 2025 — Polish, persistence, supported formats view, App Store prep
Dec 23, 2025 — Ticket image modal, App Store materials, v1.0 Release

Conclusion

Part of the appeal of this idea is that the scope could only creep so much. I honestly didn’t leave much on the TODO list for a version 1.1 besides endless optimization potential for the parser.

As usual this post was brain-dump style. If there’s any part you connected with and would like me to explore further, feel free to give me a shout.

Shinkansen Live: Scan Your Ticket, Get a Live Activity

Wed, 24 Dec 2025 05:53:39 -0600

Today I’m releasing my latest iOS app: Shinkansen Live or 新幹線ライブ in Japanese.

Shinkansen Live on the App Store

Shinkansen Live app icon

The concept is simple: you scan your Shinkansen ticket or receipt and you can see the details of your trip in a Live Activity on your lock screen and Dynamic Island.

Scan flow in 3 panels: scanning, ticket, lock screen

Here’s a quick screen capture of the main flow:

Motivation

I took a quick Shinkansen trip from Omiya (north Tokyo) to Karuizawa (Nagano) last week to do some co-working with my friends Jens and David. I had pre-purchased a reserved seat with the Eki-net, JR-East’s Shinkansen app (on iOS, it’s a web app wrapper). Similar to when I have a physical ticket (but somehow worse?) I found myself opening the app repeatedly to check my ticket’s listed attributes for:

departure time: in the 30 minutes or so lead up, before I’d entered the gates.
train number: to cross reference and check the platform I should leave from.
car number: when it was time to ascend to the platform and look for where on the platform I should line up.
seat number: when the train pulled up and I was boarding.

Both of my train apps Eki Bright and Eki Live have Live Activities support that I use frequently. I only ride the Shinkansen a few times a year, but while riding up to Karuizawa that day, I wondered, couldn’t I just OCR the Shinkansen ticket info from screenshot and stuff it into a Live Activity?

And so as soon as I arrived at Sawamura Roastery in Karuizawa, I got to work on prototyping a new app. My goal was to have a prototype by the ride home. With some extra polish it ended up taking a few more days of work.

Co-working vibes at Sawamura Roastery in Karuizawa

Features

The structure of the app is essentially a landing screen, a processing screen, and a trip-in-progress screen. The Live Activity requires its own multiple states and layouts of UI. For polish, I needed an error screen, an about screen, and a screen explaining what ticket formats are accepted.

Supported input formats

Landing screen showing input options

My initial scope was just handling screenshots from Eki-net (JR-East’s app) and SmartEX (JR-Central’s app), and in retrospect this probably would have better line to draw in the sand for version 1. However, I added support for scanning physical tickets too since the app seemed like it would be too specialized without physical tickets, probably the majority use-case.

And so, you can scan your physical ticket with the camera, choose a screenshot from the Photo Library, paste an image from the system pasteboard, or create an empty ticket if you want.

OCR and parsing

Scanning a ticket with OCR in progress

As of version 1.0, the app uses on device VisionKit to recognize text in the image and custom algorithm to do error recovery and parse out the relevant attributes from the ticket. I’ll discuss the development aspects of this decision in a future post, but for now, I’ll say that the merits of using OCR over multi-modal LLMs are that OCR is very fast, maintains privacy, and is accurate enough for a V1.

Trip in-progress

Trip in-progress screen showing ticket details

Once the ticket is scanned and parsed, you land on the trip screen for the remainder of your journey.

I recreated a facsimile of the legendary Shinkansen ticket. While doing research into ticket formats, it was surprising to see how different the information layouts are depending on where and by what means they are purchased, but the aesthetic is generally the same.

For the case of physical tickets, parsing is imperfect, so I wanted to ensure users could recover from minor errors like a missing time or train number. Therefore, all fields are user editable by tapping.

Editing screen with editable ticket fields

I also include the input image that a user can reference in an expanded view. This makes it easier to double check values and fix mistakes.

Live Activity

Live Activity in Dynamic Island compact, expanded, and lock screen views

Finally, the whole point of all this is to have a functional Live Activity. The Live Activity has a before and during layout. Before departure we show the departure time, train number, car number, and seat number (for reserved seats). During the trip, we show the arrival station and time.

Due to a technical limitation with Live Activities, I use Location Services to monitor for significant location changes in the background, and use that to wake up the app and update the Live Activity when the departure time has passed. On the technical side, this means I don’t need to run a push notification server or do any other networking from the app.

Arrival alarm

I’m a chronic sufferer of a disease called Scope Creep (this is a joke), so I couldn’t help but add an optional arrival alarm feature. This feature uses the new iOS 26 AlarmKit framework.

Alarm setting in trip screen and full screen alarm notification

Animations and transitions

I spent a unreasonable amount of time working on the animations and transitions for this app. Since there’s comparatively not a lot of screens or unique transitions to handle, it felt like a good opportunity to push the limits and make the upload experience more delightful. After all, there’s not a ton of benefit to cost when you consider needing to download an app, and then screenshot or photo your ticket in order to get that slight benefit of not needing to unlock your phone or take your ticket out of your pocket. Hopefully some fun animations add to the motivations to get over that mental hump.

Scanning transitions and animations

Image popup transition

Outlook

Shinkansen Live is free on v1.0 release. I have no idea whether the App Store listing will get views, whether the listing will convert to downloads, whether the idea will resonate with people to try, and whether any one-time users will keep the app on their devices and remember to use it. I don’t use the Shinkansen enough to estimate this well.

Regardless, I’m glad the app exists now. I hope it saves at least a few people that little extra friction in an otherwise smooth Shinkansen journey.

SwiftUI Group Still(?) Considered Harmful

Fri, 12 Dec 2025 10:37:39 -0600

A number of years ago, I internalized a SwiftUI axiom after getting burned on what appeared at first as a heisenbug.

The axiom:

Never use Group (with only a few exceptions).

TL;DR: I still think this is a useful axiom, although at some point over the last several iOS updates the behavior of Group is less harmful when applied naively. But there are still some reasons why it’s useful to treat it with caution.

`Group` distributes its modifiers amongst its subviews

Group is documented as a wrapper View. From the official docs (emphasis mine):

Use a group to collect multiple views into a single instance, without affecting the layout of those views, like an HStack, VStack, or Section would. After creating a group, any modifier you apply to the group affects all of that group’s members.

Apple specifically calls out what “affects all of that group’s members” means in the next paragraph, with an accompanying code sample:

Group {
    if isLoggedIn {
        WelcomeView()
    } else {
        LoginView()
    }
}
.navigationBarTitle("Start")

The modifier applies to all members of the group — and not to the group itself. For example, if you apply onAppear(perform:) to the above group, it applies to all of the views produced by the if isLoggedIn conditional, and it executes every time isLoggedIn changes.

This burned me in the past because I had a screen pattern like the (very simplified version) below:

struct ContentView: View {
    @State var isLoading = true

    var body: some View {
        Group {
            if isLoading {
                ProgressView()
            } else {
                DataLoadedView()
            }
        }
        .onAppear {
            fetchData()
        }
    }
    
    private func fetchData() {
        // make network request
        // set `isLoading = false` in completion handler
    }
}

With Group’s documented behavior, the onAppear modifier is essentially distributed across the Group’s views:

Group {
    if isLoading {
        ProgressView()
            .onAppear {
                fetchData()
            }
    } else {
        DataLoadedView()
            .onAppear {
                fetchData()
            }
    }
}

The bug was (at the time) that fetchData will be called twice: once when ProgressView appears, and once again when DataLoadedView appears.

Of course, there are many new modifiers and patterns since iOS 13 or 14 or whenever I was bitten by this (probably before the documentation was added). Regardless, after learning about this behavior it sort of makes sense. And so I learned my lesson that Group should not be used in this case.

What’s changed with `Group`

I’d locked this knowledge away and hadn’t considered it in years. However, coding agents seem to love to use Group to write the exact buggy code pattern I just illustrated above. I was curious enough to investigate it again.

Well it turns out that in iOS 26 and maybe even as far back as iOS 15, in most cases Group no longer distributes its onAppear or task calls amongst its subviews. Read on for the caveats.

From my testing, onAppear, onDisappear, task, and maybe other modifiers seem to have been special-cased by Apple to work at the Group-level and not be distributed to subviews like they used to. Note that this means the documentation for Group (that I quoted above) is now incorrect.

`onAppear` and `task`

Consider the following View:

struct ContentView: View {
    @State var isLeft = true
    var body: some View {
        Group {
            if isLeft {
                Rectangle().fill(.red).frame(width: 50)
                    .frame(maxWidth: .infinity, alignment: .leading)
                    .onAppear {
                        print("left")
                    }
            } else {
                Rectangle().fill(.blue).frame(width: 50)
                    .frame(maxWidth: .infinity, alignment: .trailing)
                    .onAppear {
                        print("right")
                    }
            }
        }
        .onAppear {
            print("group")
            Task {
                try? await Task.sleep(for: .seconds(1))
                isLeft.toggle()
            }
        }
    }
}

According to the docs, this code should loop between the left and right rectangles. However, as of iOS 26 (and as far back as I can test, iOS 15), it runs the onAppear modifier once and stays showing the right blue rectangle.

The console:

left
group
right

Screen-level Views

I stumbled on this lonely bug report from June 2020:

Inconsistency in how Group’s onAppear and onDisappear are called - Using Swift - Swift Forums

Something that I noticed today and didn’t expect it is that if a Group is not the root view of the screen, its onAppear is called per each child, while for a Group that is the root, the method is called once, regardless of the number of its children.

struct ContentView: View {
    var body: some View {
        Group { // Group is the root of the screen, onAppear is called once
            Group {  // non-root view, onAppear is called once per each child
                Color(.red)
                Color(.yellow)
            }
            .onAppear { print(".:. onAppear2") }
            .onDisappear { print(".:. onDisappear2") }
            
            Color(.blue)
            Color(.purple)
        }
        .onAppear { print(".:. onAppear1") }
        .onDisappear { print(".:. onDisappear1") }
    }
}
/* Output:
.:. onAppear1
.:. onAppear2
.:. onAppear2
*/

My guess is that this was reported during iOS 13, right before iOS 14 beta.

I tested this code as well, and it turns out the OP’s example no longer prints onAppear2 twice on iOS 15 or iOS 26.

List

The only case I’ve found (so far) where the documented onAppear (and task) distributed-across-subviews behavior still exists is when Group is within a List:

struct ContentView: View {
    @State var isLeft = true
    var body: some View {
        List {
            Group {
                if isLeft {
                    Rectangle().fill(.red).frame(width: 50)
                        .frame(maxWidth: .infinity, alignment: .leading)
                        .onAppear {
                            print("left")
                        }
                } else {
                    Rectangle().fill(.blue).frame(width: 50)
                        .frame(maxWidth: .infinity, alignment: .trailing)
                        .onAppear {
                            print("right")
                        }
                }
            }
            .onAppear {
                print("group")
                Task {
                    try? await Task.sleep(for: .seconds(1))
                    isLeft.toggle()
                }
            }
        }
    }
}

The console:

left
group
right
group
left
group
right
... (repeats forever)

Regular modifiers with Group

As a check that Group still distributes its modifiers in the general case, I created a simple custom modifier that prints on init and applied it to the Group:

struct PrintModifier: ViewModifier {
    init() {
        print("modifier init")
    }

    func body(content: Content) -> some View {
        content
    }
}

struct ContentView: View {
    @State var isLeft = true
    var body: some View {
        Group {
            if isLeft {
                Rectangle().fill(.red).frame(width: 50)
                    .frame(maxWidth: .infinity, alignment: .leading)
                    .onAppear {
                        print("left")
                    }
            } else {
                Rectangle().fill(.blue).frame(width: 50)
                    .frame(maxWidth: .infinity, alignment: .trailing)
                    .onAppear {
                        print("right")
                    }
            }
        }
        .modifier(PrintModifier())
        .onAppear {
            print("group")
            Task {
                try? await Task.sleep(for: .seconds(1))
                isLeft.toggle()
            }
        }
    }
}

The console:

modifier init
group
left
modifier init
right

As we can see, PrintModifier is being applied to each subview independently as is documented. We’ve somewhat proven to ourselves that in the general case, Group still distributes modifiers.

When is Group useful?

When is Group still the right choice?

Honestly, not very often!

Distributing a lot of modifiers across sibling views

If you really really really need to apply one or more modifiers independently to a set of sibling views and you need the sibling views to stay legible to their current parent container, Group is the right choice.

In the below toy example with Form:

Form {
    Text("Title")
    Group {
        Text("Subtitle")
        Text("Description")
    }
    .foregroundStyle(.secondary)
}

I would prefer simply duplicating the modifier manually:

Form {
    Text("Title")
    Text("Subtitle")
        .foregroundStyle(.secondary)
    Text("Description")
        .foregroundStyle(.secondary)
}

In my experience, the conditions that lead to Group being useful for this case are exceedingly rare. In a quick search of my current codebase, I have only a couple examples.

This one just barely makes sense as it applies 4 modifiers to these 2 slightly different conditional subviews. Not quite large enough to justify separating out into named views; not quite small enough to duplicate the modifiers inline.

Group {
    switch routeOption {
    case .departure, .departureOnly:
        HStack(spacing: 3) {
            Image(systemName: "arrow.up.right").imageScale(.small).bold()
            Image(systemName: "train.side.middle.car")
        }
        .padding(.horizontal, 10)
    case .arrival, .arrivalOnly:
        HStack(spacing: 2) {
            Image(systemName: "train.side.middle.car")
            Image(systemName: "arrow.down.right").imageScale(.small).bold()
        }
        .padding(.horizontal, 11)
    }
}
.font(.caption)
.foregroundStyle(.secondary)
.padding(.vertical, 10)
.background(Material.ultraThick, in: RoundedRectangle(cornerRadius: 4))

Overcoming the 10-subview limit (no longer applies)

Before Swift 5.9’s variadic generics, ViewBuilder could only handle 10 non-enumerated subviews. This example is still in the Group docs but no longer applies:

// No longer applies as of Swift 5.9
var body: some View {
    VStack {
        Group {
            Text("1")
            Text("2")
            Text("3")
            Text("4")
            Text("5")
            Text("6")
            Text("7")
            Text("8")
            Text("9")
            Text("10")
        }
        Text("11")
    }
}

Comprehensible Later: A Read-it-later App for Language Learners

Sat, 15 Nov 2025 05:57:31 -0600

This post is a short retrospective on Comprehensible Later, my working-title for a read-it-later iOS app prototype I worked on last week. Although it’s currently in private beta on Test Flight, I want to share the motivation and technical challenges I ran into while working on it.

Share Extension presented from Safari, main app article list, and article detail screens

TLDR: Comprehensible Later is an iOS app for saving articles natively written in a language you’re learning, with automatic translation via LLM to a simpler version of your target language. The goal is to give you more interesting things to read at a level you can understand without needing to pause to look up every other word.

Original level and simple level for a Japanese article

Original level and simple level for an English article (from this blog)

What is Comprehensible Input

Comprehensible Input is part of a language acquisition framework first introduced by Dr. Stephen D. Krashen. The framework states that language is separately acquired and learned. Acquisition happens by ensuring ample input (reading or listening) with the important caveat that the input is comprehensible at the learner’s current level. Learning happens through comprehensive study of rules and vocabulary. From this summary:

When we receive comprehensible input, the conditions are met for our brain to be able to use its natural ability to acquire language, without having to do anything else. There’s no need to study, review vocabulary, or practice anything. Watching and reading itself results in acquisition.

In some senses, this method seems intuitive, not in the least since almost all children acquire language skills before they begin formal teaching. In a second-language context, graded readers – books written for various non-native language levels – have existed for over a century. Wikipedia even has a simple English language variant for many common articles. I’ve occasionally used the modern Satori Reader service for Japanese graded texts.

But I think the important part is recognizing exactly how basic you need to make some input in order for it to be understandable, especially at the absolute-beginner level. In a since-removed introductory YouTube video from the creator, he shows a session of an instructor sitting with a zero-level beginner student, pointing at vivid images in a travel magazine and gesturing heavily while explaining the contents very slowly in the target language as the primary means of bootstrapping.

Language is so multi-dimensional that it’s incredibly time consuming – both as a creator and a consumer of materials – to get the exact level of material that is both comprehensible but challenging enough to increase your overall ability. Then add another dimension of motivation: as a reader, how do you find materials with a subject matter that’s interesting to you and will keep you motivated to push through word-after-word, page-after-page, day-after-day?

This led to a hypothesis:

Graded readers are usually close to the correct difficulty to facilitate learning, but do not have an audience wide enough to support a variety of interesting subject material.
Native materials cover an infinite range of interesting topics, but are infeasible to read until the latest stages of language acquisition.
One of the most commonly accepted use-cases for LLMs is text summary and translation.

What if we used LLMs to translate any native article on-demand to the user’s exact target language level?

The barriers to this being feasible are:

Can an LLM properly translate from e.g. Native Japanese to JLPT N4 level Japanese in a “natural” way – where it is simultaneously challenging, comprehensible, and accurate?
Can the translation happen fast enough to fit within a user’s desired language-learning workflow?
What additional resources are required to facilitate language learning? In-line dictionary lookup? An SRS system? Customized word lists?
What unique points are there to each target language that increase the interface complexity? For example, for Japanese learning, should we include furigana for all potentially unknown kanji?
Does it also make sense to allow translation from e.g. Native English to simple Japanese (if the target language is Japanese)?

Specification for the prototype app

I’ve used 2025 frontier LLMs to write and translate simple versions of text before, so I’m confident it’s either possible now at a minimally acceptable translation quality or will be in the very near future.

What I wasn’t confident about is whether it’s cost prohibitive or time prohibitive to use the highest quality reasoning models to do the translation.

In retrospect, I should have spent at least a little more time doing bench testing on the API versions of various models on a wide array of sample articles. Instead, I took the less (more?) pragmatic route of jumping into the implementation for an app prototype that I could start using ASAP in context, as well as distribute to a few friends.

My initial thought was that my main source of content would be blog posts and news articles I come across from my everyday feed scrolling. But I also felt I should support translating raw text too, like that from social media posts.

I considered a Safari Extension to replace existing text on a webpage with the simplified translation, similar to how the built-in translation function in Safari works. But my gut-feeling was that this would be too limiting for language learning use cases. Even reading a text at a simpler level of a target language still takes enough time that it would be better to ensure the user doesn’t feel obligated to read everything at once. Additionally, this wouldn’t work for native text outside of Safari.

My next thought was a Share Extension. Share Extensions are old iOS technology, but still highly used and useful. In a share extension I could display the translated article content in a dedicated modal and have full control over its presentation and layout.

However, I also wanted to support the read-it-later use case. Personally, I stumble upon articles when doing feed scrolling sessions when I have a few minutes on the train but don’t necessarily have the time to read the whole article, even in English, at that time. I use Instapaper for read-it-later for English articles and I felt this would be a similarly useful use case to model my app after.

With that in mind I got to work on the actual prototype with the following initial spec:

A native app that:
- keeps a list of articles imported from URLs or as raw text.
- has a detail view that shows both the original and translated versions of the text.
A share extension that:
- immediately processes the shared URL or text and displays it in the share modal.
- allows the user to optionally save the translated text in the app for later.

Note that as of iOS 18.4, there’s actually another option for the interface: TranslationUIProviderExtension. iOS users can replace Apple’s Translation app with another translation app, meaning the Translate menu tooltip can open a third-party app. I have mine set to DeepL. Due to the limitations I’ll discuss later (namely, translation processing time), it doesn’t make sense yet to implement Comprehensible Later as a Translation Extension.

Implementation

I worked through several iterations of a detailed implementation spec with Claude and Codex then set them off to work getting the foundations of the app in place. This wasn’t exactly vibe coding because I specified technologies and packages to use up front and guided their output along the way. But I was still aiming to have the agents create the clay that I’d be molding in a distinct second phase of development.

Packages

The key packages that would make this closer to a weekend prototype and not a months-long project were:

swift-readability - wrapper for Firefox’s reader-view parsing library for stripping down a full page HTML to its essential content.
AnyLanguageModel - use any LLM API with Apple’s Foundation Models SDK interface.
swift-markdown-ui - display the full Markdown spec in SwiftUI (note: I later replaced this).
Demark - convert HTML-to-Markdown.
Ink - convert Markdown-to-HTML.
sqlite-data - SQLite wrapper for local article storage and observable data layer for the app.

Data flow

The initial data flow for articles imported via URL was the following:

Input URL
    ↓ URLSession
HTML Data
    ↓ String(data:)
HTML String
    ↓ Readability
Clean HTML
    ↓ Demark
Markdown
    ↓ AnyLanguageModel
Translated MD
    ↓ swift-markdown-ui
Display

The initial flow for raw text:

Input Text
    ↓ AnyLanguageModel
Translated MD
    ↓ swift-markdown-ui
Display

However, due to limitations with the swift-markdown-ui package, the final version of the prototype uses this flow:

Input URL
    ↓ URLSession
HTML Data
    ↓ String(data:)
HTML String
    ↓ Readability
Clean HTML
    ↓ Demark
Markdown
    ↓ AnyLanguageModel
Translated MD
    ↓ Ink
HTML
    ↓ WebView
Display

With the bones of the app architecture and dependencies in place, I began testing and optimizing the data flow.

Readability

I found a small bug in the Readability Swift wrapper where the baseURL parameter was inaccessible to the URL-based initializer Readability().parse(url:options:). This prevented relative image tags from getting properly resolved to a full address. For example, on my website image tags look like /images/example.jpg and are resolved by my browser automatically to be either https://twocentstudios.com/images/example.jpg (the real server) or http://localhost:4000/images/example.jpg (my local machine).

Luckily, the baseURL parameter was accessible in the Readability().parse(html:options:baseURL:) initializer. As a workaround I simply needed to fetch the page data myself with URLSession.

Demark

Demark has two different HTML->Markdown parsing implementations: heavy-and-accurate or fast-and-inaccurate. Since the HTML is getting pre-processed by Readability in advance of being passed to Demark, I’m using the fast-and-inaccurate version that doesn’t load the full page in a headless WKWebView.

AnyLanguageModel

As of iOS 26, Apple’s local Foundation model is slow, not-ubiquitously available on devices, and (arguably) barely functional for most use cases, especially mine. Within a few years I expect it may be useful. Similarly, my impression is that any other MLX-compatible models runnable on an iOS device are not yet accurate or fast enough for my use case.

Therefore, I grabbed both an OpenAI and Gemini API key and wired them up to AnyLanguageModel for testing. I ran a few trials with the top-tier, mini, and nano variants and decided on defaulting to the mini variant as a compromise between speed, cost, and accuracy. Specifically, Gemini Flash 2.5 is the current default, but I suspect I could spend several weeks creating and running benchmarks across the dozens of closed and open models.

AnyLanguageModel made it easy to build a user settings-based model switcher with very little code adjustments required on my side. Technically, Gemini ships an OpenAI-compatible endpoint so I could have kept even more of the same codepath. During debugging, I realized that AnyLanguageModel wasn’t passing through the instructions parameter to OpenAI, so I submitted a quick PR and Mattt had it merged and version bumped by the next day.

In a later mini-sprint, I added a full settings screen that allows switching model provider, model, target language, target difficulty, adding custom translation instructions, and even fully rewriting the system prompt. Of course, I would never include all these settings in a production app, but it’s useful for my trusted beta testers to tinker if they so choose.

The translation settings screen

By default, my (simple) system prompt is:

Faithfully translate the native-level input markdown text into the target language with the target difficulty level.

Be creative in transforming difficult words into simpler phrases that use vocabulary at the target difficulty level. Combine or split sentences when necessary, but try to preserve paragraph integrity.

The output format should be standard Markdown including all supported markdown formatting like image/video tags. Preserve all structure from the input (paragraphs, lists, headings, links, images, videos). DO NOT ADD COMMENTARY.

Target language: \(targetLanguage)
Target difficulty level: \(targetDifficulty)
Additional notes: \(additionalNotes)

I’ll discuss my impressions of the effectiveness of this prompt a little later on.

Something I noticed almost immediately during testing was that requests were taking at minimum 30 seconds and sometimes over 60 seconds to complete. It didn’t really depend on model size either. I found the same performance characteristics for both OpenAI and Gemini APIs direct from first-party servers. I thought it might be the streaming API or perhaps some configuration in AnyLanguageModel I was not in control of, so I switched back to the single-request version. It didn’t help. I also began testing the same prompt and inputs from the API sandbox pages like OpenAI’s playground and Google’s AI Studio and saw basically the same results.

Although the slow translation speed is a pretty substantial blocker, I felt like, at least temporarily, I could work around it in the UX by leaning into the read-it-later nature of the app. I added support for Apple’s Background Tasks API so there was a greater chance that articles added early in the day would be ready to read by the time the user opened the app.

App UI

With the translation flow in place, I began shaping the app UI.

The list of articles was simple enough. I held off on adding lots of important, but not urgent, contextual actions like archiving and deleting from the list view.

I did add both “import from pasteboard” and “import from free text” buttons to the toolbar.

The article list view showing toolbar buttons for importing from pasteboard and free text

The import URL/text screen

I spent more time on the article detail view. Initially, it displayed the title, import state, and translated article. My focus for adding actions was to facilitate debugging primarily for myself and secondarily for my beta testers. This meant buttons for copying the original article text, copying translated article text, deleting an article, opening the original link, and retrying the translation (with different settings).

After some initial usage, I realized I wanted to see the original text and the translated text side-by-side so that I could compare the language usage by sentence and paragraph.

Article detail view with options to display original and translated text

Supported actions for article detail

Debug info viewer

However, the most time-consuming and impactful change was the markdown display system. This was a tough decision, but I think ultimately necessary for the first version.

Originally, I was planning to use swift-markdown-ui to display the translated markdown text in SwiftUI. This implementation was basically plug-and-play, rendered exactly as I wanted, supported images out of the box, and was performant. However, the one fundamental and unsolvable issue is that SwiftUI Text only supports paragraph level copy support and does not support character-level or word-level selection. For language learning, I absolutely need the ability to select a word and use the context menu tooltip action “Look Up” or “Translate” or “Copy” buttons. swift-markdown-ui would not be able to support this and I needed to research other solutions.

I spent nearly a full day researching and experimenting with other Markdown solutions. My second preference was to convert Markdown to AttributedString either natively or with a package, then display the AttributedString in a SwiftUI-wrapped UITextView with selection enabled but editing disabled. However, both the native and package versions of AttributedString initialization failed at properly respecting whitespace, newlines, and supporting images. My estimation was that it’d take significantly more time for me to grok the full Markdown spec, all the underlying packages, and then implement the required patches than I was willing to spend for a prototype.

Therefore, I pivoted to using a browser-based target view instead. iOS 26 was blessed with WebView, a modern SwiftUI-native implementation of the UIWebView and WKWebView UIKit views before it. With a WebView as the new target, I used Ink to convert the LLM output Markdown back to HTML, added a barebones stylesheet, and loaded these contents. I don’t love using a WebView for this use case since it’s comparatively heavy, has plenty of rendering quirks (like occasional white background flashes), and requires a full screen layout. But at the moment it’s the least-worst option.

Unfortunately, the slow translation speed meant some of the complexity of creating a fully-featured Share Extension was in vain; it didn’t make sense for the user to wait 30-60 seconds for the share extension to load a preview of the article content like I’d originally planned.

My initial vision was to load a one-page preview of the translation as quickly as possible. Then, I’d allow the user to tap a button to continue viewing the full translation in line. Or at any time they could tap a button to save the article URL (or raw text) to the main app to read later. I was planning on having an “open in app” button too, but as far as I can tell it’s not supported to open an app directly from a Share Extension.

I kept the full functionality of the share extension intact in case I can solve the translation speed issue in the future. But as another workaround, I added an Action Extension. An Action Extension appears in the bottom section of the system share sheet. Like a Share Extension it can also present custom UI, however since I already have a Share Extension I made my Action Extension have no UI and immediately save the URL to the app.

iOS share sheet showing both the Share Extension and Action Extension for quickly saving articles

Share Extension with translation complete

Import flow

App Extensions can share data on device with the main app using an App Group. When the user indicates they want to add the URL or raw text to the app, the Extension serializes an Article model to a json and writes a new file to the App Group. The main app monitors the shared App Group directory for new files. When it detects a new file, it adds the Article to the app’s SQLite database. If the Article already finished translation, it will include the translated markdown and no further processing is necessary. Otherwise, it will be queued for processing.

I chose not to share the SQLite database between the main app and the extensions because, since the app and extensions are separate processes, there are myriad issues with using SQLite in this way. Since data sharing is one way (from extension to app) there’s no need to introduce that complexity.

Adding articles from the main app instance skips the file encoding/decoding step and simply writes a new Article to the database.

The processing code is admittedly a bit fragile, but in testing has worked well enough that I haven’t felt an immediate need to rewrite it. It uses an enum Status stored alongside each Article in the database in order to manage the translation queue, including failures. SQLiteData supports observation, so both the article list view and the article detail view are always up to date on an Article’s status.

Localization

Localizing a prototype would be something I’d never consider doing before the advent of coding agents. The actual act of translation between a base language and another language is insignificant compared to the amount of additional tooling and operational complexity of introducing localization keys, adding comments, handling interpolation, handling pluralization rules, handling error messages and other strings generated deep in business logic, and handling the indirection involved in looking up the values for the keys. The new xcstrings file’s autogeneration definitely helps ease the burden. But it’s at least an order of magnitude more work in my opinion.

All that said, coding agents can automate enough of this work that I added full localization support for Japanese for one of my beta testers who wanted to try the app for converting English to simple English. I’m still cognizant of the ongoing support complexity full localization adds to a prototype, but for now it’s not a decision I regret.

Various screens with Japanese localization

Impressions so far

What I’ve learned so far is that the prompt needs to be more customized to each target language and should probably go as far as including an allow-list of words to use, especially for the most basic target difficulties.

I’ve found the models have a hard time with native Japanese news articles. Something about the language is just so dense that my first prompt attempt does not push the model to simplify enough.

Similar to what I’ve found with even commercial apps like Instapaper, a large percentage of sites now have enough paywall or otherwise reader-hostile javascript that it’s not enough to fetch a simple URL directly from the source. I’m not ready to handle the endless, unforgiving work of handling all the edge cases of the open web, so URL fetching is going to be best effort for the foreseeable future.

The Readability library itself is not perfect at parsing out text from pages that aren’t obviously written as “articles”. This isn’t all that different from the built-in Safari reader mode which isn’t universally supported across the entire web.

Seeing some of my blog posts in super-simple English was really fun. One of my ongoing goals is to write simpler without giving up my voice, so seeing how an LLM breaks up my sentences and phrases and clauses is enlightening (of course, not at all related to the use case the prototype was built for).

For Japanese, there’s some unpredictability on how the LLM deals with kanji. Usually it includes kanji as is, but sometimes it will add the reading in parentheses directly after for literally every word. For example, “果物（くだもの）を食べる（たべる）”. Native ruby/furigana support would be ideal, and possibly easier using HTML than AttributedString.

What’s next?

Comprehensible Later is on Test Flight in private beta with myself and a few friends. I’m planning on collecting feedback and evaluating the app’s potential for wider release. It could take another generation or two of LLM. It could take as long as waiting for local models to improve. Or the entire concept could be flawed. I’m not sure yet. But that’s what the prototype is for.

Regardless of the result, it was of course a good learning experience to see what it’s like to build a read-it-later service for iOS in 2025.

Let's Write a Train Tracking Algorithm

Mon, 22 Sep 2025 15:00:00 -0500

I delivered a 20-minute presentation on September 20th at iOSDC Japan 2025.

If you prefer video:

Japanese (conference): YouTube
English (post-conference recording): YouTube

Other materials:

GitHub: train-tracker-talk - Open source code and presentation materials
Blog: I Presented At iOSDC 2025 - More about the conference and presentation context
App Store: Eki Live - The app discussed in the presenation

This post is a deconstructed version of the talk with the slide images above and my speaker notes in English below.

Lately I’ve been working on an app called Eki Live.
Today I’m going to talk about a part of that app.
So what do I mean by train tracking algorithm?
Well, when riding a train, it’s useful to know the upcoming station.

On the train, we can see the train information display or listen for announcements.

But would it also be useful to see this information in your Dynamic Island?

In my talk, we’ll first review the data prerequisites we’ll need for the algorithm.
Then, we’ll write each part of the algorithm, improving it step-by-step.

We need two types of data for the train tracking algorithm:
Static data that describes the railway system of greater Tokyo.
And Live GPS data from the iPhone user.

Railways are ordered groups of Stations.
In this example, we can see that the Minatomirai Line is made up of 6 stations.

Trains travel in both Directions on a Railway.
Coordinates make up the path of a Railway’s physical tracks.

This map shows the Railway data we’ll be using.

We collect live GPS data from an iPhone using the Core Location framework.
We store the data in a local SQLite database.

A Location has all data from CLLocation.
Latitude, longitude, speed, course, accuracy, etc.

A Session is an ordered list of Locations.
A Session represents a possible journey.
Green is for fast and red is for stopped.

I created a macOS app to visualize the raw data.
In the left sidebar there is a list of Sessions.
In the bottom panel there is a list of ordered Locations for a Session.
Clicking on a Location shows its position and course on the map.

Our goal is to write an algorithm that determines 3 types of information:
The Railway, the direction of the train, and the next Station.

Here is a brief overview of the system.

The app channels Location values to the algorithm.

The algorithm reads the Location and gathers information from its memory.

The algorithm updates its understanding of the device’s location in the world.

The algorithm calculates a new result set of railway, direction, and station phase.
The result is used to update the app UI and Live Activity.

Let’s start by considering a single Location.
I captured this Location while riding the Tokyu Toyoko Line close to Tsunashima Station.

Can we determine the Railway from just this Location?

We do have coordinates that outline the railway…

First, we find the closest RailwayCoordinate to the Location for each Railway.
Then, we order the Railways by which RailwayCoordinate is nearest.

Here are our results.

The closest RailwayCoordinate is from the Toyoko Line at only 12 meters away.
The next closest RailwayCoordinate is from the Shin-Yokohama Line at 177 meters away.

We did it!
Our algorithm works well for this case.
But…

Let’s consider another Location.
This Location was also captured on the Toyoko Line.

But in this section of the railway track, the Toyoko Line and Meguro Line run parallel.
It’s not possible to determine whether the correct line is Toyoko or Meguro from just this one Location.

The algorithm needs to use all Locations from the journey.
The example journey follows the Toyoko Line for longer than the Meguro Line.

First, we convert the distance between the Location and the nearest RailwayCoordinate to a score.
The score is high if close and exponentially lower when far.
Then, we add the scores over time.

The score from Nakameguro to Hiyoshi is now higher for the Toyoko Line than the Meguro Line.

We did it!
Our algorithm works well for this case.
But…

Let’s consider a third Location.
This Location was captured on the Keihin-Tohoku Line which runs the east corridor of Tokyo.

Several lines run parallel in this corridor.
The Tokaido Line follows the same track as the Keihin-Tohoku Line

But the Tokaido Line skips many stations.

If we only compare railway coordinate proximity scores, the scores will be the same.

Let’s add a small penalty to the score if a station is passed.
If a station is passed, that indicates the iPhone may be on a parallel express railway.
Let’s also add a small penalty to the score if a train stops between stations.
If a train stops between stations, that indicates the iPhone may be on a parallel local railway.

Using this algorithm, the Keihin-Tohoku score is now slightly larger than the Tokaido score.

Let’s consider two example trips to better understand penalties.
For an example trip 1 that starts at Tokyo station…

The train stops at the second Keihin-Tohoku station.
The Tokaido score receives a penalty since the stop occurs between stations.

As we continue…

The Tokaido score receives many penalties.
Therefore, the algorithm determines the trip was on the Keihin-Tohoku Line.

For an example trip 2 that also starts at Tokyo…

The train passes the 2nd Keihin-Tohoku station.
And the Keihin-Tohoku score receives a penalty.

As we continue…

The Keihin-Tohoku score receives many penalties.
Therefore, the algorithm determines the trip was on the Tokaido Line.

We did it!
Our algorithm works well for this case.
There are many more edge cases.
However, let’s continue.

For each potential railway, we will determine which direction the train is moving.

Every railway has 2 directions.
We’re used to seeing separate timetables on the departure board at a non-terminal station.

For example, the Toyoko Line goes inbound towards Shibuya and outbound towards Yokohama.

Let’s consider a Location captured on the Toyoko Line going inbound to Shibuya.

Once we have visited two stations, we can compare the temporal order the station visits.
If the visit order matches the order of the stations in the database, we say that the iPhone is heading in the “ascending” direction.

The iPhone visited Kikuna and then Okurayama.

This ordering does not match the database, so we consider it “descending”.
In the database, “descending” maps to inbound.
Therefore, we know the iPhone is heading inbound to Shibuya.

We did it!
Our algorithm works well for this case.
But…
It could take 5 minutes to determine the train direction.
Can we do better?

Let’s use the Location’s course.
Remember that course is included with some CLLocations by Core Location.
Several points moving at a decent speed are required before Core Location adds course to a CLLocation.
And course itself has its own accuracy value included.

Core Location provides an estimate of the iPhone’s course in degrees.

Note that this is not the iPhone’s orientation using the compass.
The course value should be the same regardless of whether the iPhone is in a pocket or held in a hand facing the rear of the train.

The course for the example Location is 359.6 degrees.
It’s almost directly North.

First, we find the 2 closest stations to the Location

Next, we calculate the vector between the 2 closest stations for the “ascending” direction in our database.
For the Toyoko line, the “ascending” direction is outbound (as mentioned earlier).
Therefore the vector goes from Tsunashima to Okurayama.

We need to take a quick sidebar to talk about the dot product.
Do you remember the dot product from math class?
We can compare the direction of unit vectors with the dot product.
Two vectors facing the same direction have a positive dot product.
Two vectors facing in opposite directions have a negative dot product.

Next, we calculate the dot product between the Location’s course vector and the stations vector.
If the dot product is positive, then the railway direction is “ascending”.
If the dot product is negative, then the railway direction is “descending”.

The dot product is -0.95.
It’s negative.
Negative means “descending”.
And “descending” in our database maps to inbound for the Toyoko Line.
Therefore, the iPhone is heading to Shibuya.

We did it!
Our algorithm works well.
Let’s move on to the last part of the algorithm.

Finally, we can determine the next station.

The next station is shown on the train information display.
We’ll call this the “focus station phase” going forward.
This includes the station name (e.g. Kikuna) and its phase (e.g. Next).

The display cycles through next, soon, and now phases for each station.

On a map, here is where we will show each phase.

We calculate the distance d and direction vector c from the Location to the closest station.
We show the closest station S or the next station in the travel direction S+1 depending on d and c.

When the closest station is in the travel direction, the phase will be “next”.

A Location less than 500m from the station will be “soon”.

A Location less than 200m from the station will be “now”.

Even though the Location is within 500m from the closest station, the station is not in the travel direction.
Therefore, the phase will be “next” for the next station in the travel direction.

We did it!
Our algorithm works well.
But…

GPS data is unreliable.
Especially within big stations.
Especially when not moving.
Here is an example Location stopped inside Kawasaki station that has an abysmal 1 km accuracy.

Let’s create a history of Locations for each station.
For each station, let’s categorize each Location according to its distance and direction.

In this example, “approaching” points are orange, “visiting” points are green, and the departure point is “red”.

Focus station algorithm version 2 has 3 steps.

In step 1, we categorize a Location as “visiting” or “approaching” if it lies within the bounds of a Station.
Our rule is that only 1 Station per Railway will store a unique Location in the visitingLocations or approachingLocations array.
Usually, this is not an issue, but some Stations on the same Railway are within 200m of each other.
To disambiguate, we always choose the closest Station.

If the Location is outside the bounds of any Station that already has visitingLocations or approachingLocations as non-empty, we set the firstDepartureLocation for that Station.
It’s okay for a Location to be set as firstDepartureLocation for Station A while also being in a visitingLocations or approachingLocations array of Station B.
Additionally, there is special handling for the startup case where a railway has no Locations set yet. In this case, we try to find the closest Station opposite the travel direction and set its firstDepartureLocation.
We can then consider that Station the user’s departure station and use it to determine the focus station.

In step 2, we use the station history to calculate the phase for each station.

This is a departure phase for Minami-Senju station.
The StationDirectionalLocationHistory has only a firstDepartureLocation.

This is an approaching phase for Kita-Senju station.
Note: this would still count as an approaching phase even if there were only 1 Location in the approachingLocations array.

This is a visiting phase.
Note: this would still count as a visiting phase even if there were only 1 Location in the visitingLocations array.

This is a visited phase.
You can see the firstDepartureLocation in red.

In step 3, we look through the station phase history for all stations to determine the focus station phase.

In an example, when the latest phase for Kawasaki station is visited, then the focus phase is “Next: Kamata”

In another example, when the latest station phase for Musashi-Kosugi station is visited and Motosumiyoshi station is approaching, then the focus phase is “Soon: Motosumiyoshi”

Using a state machine gives us more stable results.

We did it!
Our algorithm works well…

But can we tell the difference between a visited station and a passed station?
Remember, we need this information to calculate a potential penalty for the railway score.

If the train is stopped within a station’s bounds for more than 20 seconds then we consider it visited.

If the train is moving within a station’s bounds for more than 70 seconds then we also consider it visited.
This case is for stations with bad GPS reception.

Otherwise we consider the station as passed.

Now I’d like to demo the SessionViewer macOS app I created.
I’ll show a journey from Kannai station to Kawasaki station on the Keihin-Tohoku line.
It takes some time for all Locations to be processed by the algorithm (top right).
But while it’s processing, I can start playback to see the journey at 10x speed (top right).
In the inspector (right sidebar), you can see the algorithm’s results updating.
Keihin-Tohoku line has the highest score (top right).
The direction is northbound (top right).
The latest phase for each station is shown (middle right).

When we reach the last Location in the Session, we can see the full Station history (middle right).
We can see the phase history for any station by clicking its current phase.
When I click on a station, I can see on the map the Locations that were used to calculate its phase.

The 5 iOS apps I created to collect this data are open source on GitHub.
The macOS app and algorithm are included as well.

The algorithm is still being improved!

But if you want to try it, Eki Live is on the App Store now.
The app starts up automatically in the background and shows the next station in the Dynamic Island.

Thanks for reading this presentation.
If you have questions or comments, feel free to reach out.

I Presented At iOSDC 2025

Sun, 21 Sep 2025 05:14:02 -0500

I gave a 20-minute presentation at iOSDC 2025 called “Let’s Write a Train Tracking Algorithm”. I’m still gathering up all the presentation materials, but so far:

This is a more behind-the-scenes diary post. I’ll follow up with more details from the actual talk after the conference has ended.

Me looking especially jpg encoded on the live stream

About the conference

iOSDC is a really great conference for both attendees and speakers. This is my first year speaking but I’ve attended the past 3 years. I like that there’s an open proposal system and it draws a wide variety of speakers that are not necessarily on the “circuit”. The rookies lighting talk system is also a fantastic way to create the next generation of great speakers in a supportive environment. I met some cool developers at the speakers’ dinner that got me excited to do my own talk and to see their talks.

After my talk, there were several great questions during Q&A. And I had several more really interesting conversations with developers at the Ask the Speaker table.

This was my first conference talk presenting in Japanese. When I created my proposal, I knew I wanted to do it in Japanese. Although I’m sure the majority of iOSDC attendees could have understood the presentation in English (especially simplified), it felt like the right time to challenge myself.

Writing the talk in Japanese had a significant upside: I had to iterate the spoken lines for each slide several times to their most essential elements. My less expansive vocabulary resulted in a talk that is easier to understand by both developers and non-developers. My inability to improvise made the talk less fluid, but also ensured that my talk was optimized for time and I could learn it down to the word.

The final version of my presentation was 121 slides in 20 minutes. The slides layout was created in Deck Set. I created about 90 images and 5 videos captured from the maps in my custom apps and in Figma. The demo video was lightly edited in Final Cut Pro.

My deck in Deck Set

Thanks to everyone who came to the talk and watched online. Special thanks to my friends who gave early feedback on my drafts.

I’m really happy I had this opportunity. I hope I’ll have something interesting to propose for a talk for next year.

How I developed the presentation materials

At the time of the call for proposals, I’d released Eki Live to the App Store with what I considered “version 2” of the underlying algorithm that determines the user’s railway, the direction, and station and updates it over time.

Eki Live's home screen, showing railway, direction, and next station

Before version 2, there was of course version 1 of the algorithm. Version 1 was the simplest method that worked for the least difficult train journey scenarios. The app would use the iPhone’s distance from station coordinates to create a visit history for the stations on a railway line. This info was enough to provide an estimate of the railway, direction, and station in many cases. But version 1 of the algorithm had several unfixable flaws:

The user had to visit at least 2 stations in order for a result to be produced.
The algorithm was not differentiating between parallel railways that split at some point.
There was no way to differentiate between parallel railways with different station configurations like the Tokaido and Keihin-Tohoku lines.

Although the rest of the app was ready, I didn’t end up releasing version 1 of the tracking algorithm. It didn’t quite feel magical enough.

I rewrote the algorithm to a version 2 before releasing the app. I spent more time collecting data and more time creating visual debug utility apps to assist in development.

However, the biggest change to the algorithm was obtaining and cleaning up railway coordinate data. The dataset I used for Eki Bright only included station (and its coordinates) and railway data. Once I had access to the coordinate data, that opened up a new avenue for being able to estimate a railway instantly and then continue to refine the estimate over time.

In version 2, I doubled down on a scoring system for each aspect of the overall algorithm. Although my head was in the right place, this introduced far too much complexity for too little benefit. It took much longer than I’d hoped, but I eventually got this version to a state I was reasonably satisfied with and released it as Eki Live v1.0 to the App Store.

It would be much easier to develop the tracking algorithm assuming access to an infinite stream of accurate GPS coordinates from an iPhone. Unfortunately, this is not the case. The app needs to work within the bounds of the Core Location APIs for monitoring significant location changes so that device battery life can be preserved. The app also needs to stop tracking a journey when that journey ends. Therefore, there’s two other separate heuristics I needed to iterate on that subtly affect the behavior of the tracking algorithm via the data provided to it.

When I submitted my talk proposal to iOSDC, I was planning on talking about the entire app. After all, there were so many unique and interesting problems I’d run into in the development process.

When my proposal was accepted, I started working on my talk with this in goal in mind: more of a high level overview of the app development. In that early draft of the talk, I set out a map for which data-gathering apps I needed to create. A main theme of this draft would be why you should develop throwaway prototype apps to gather data before developing for production.

Of course, it took a few weeks to develop what became 5 separate prototype apps plus the algorithm viewer app. Along the way, I became less and less satisfied with the prospect of presenting “version 2” of the tracking algorithm to an audience since it had many admitted flaws.

Once I’d finally created all my prototype apps and updated the talk draft, I realized I was already probably 5-10 minutes over time without talking about the tracking algorithm, UI, or Live Activity. I took a step back and realized that the tracking algorithm made for a more easily digestible story for newcomers to this problem than a bunch of minutia about the self-imposed constraints of data collection.

So in terms of the talk draft, I essentially started over. This time I focused purely on the tracking algorithm itself, telling a cleaned up story about how it has improved over time. Although I didn’t show any of the several data collection apps I made, they were still useful in collecting real life data to use in examples and to develop the algorithm itself.

To fit the time constraints, I had to pare down a few examples and gloss over some details. But in spite of this, I was satisfied with the talk being code-free and accessible to even non-developers. It’s a true challenge trying to set up all the background knowledge needed to understand a problem, present the problem, and have the audience understand the solution within the span of 2 or 3 slides. Especially when the solution was something that took me days or weeks to work out.

The tracking algorithm presented in my talk is version 3. But due to spending the last two weeks writing and practicing the talk, I didn’t have time to actually integrate this version into the Eki Live app! The production version is still using tracking algorithm version 2 and in addition has some iOS 26 bugs (iOS 26 was released this week).

I still have a lot of work to do before I can (temporarily) put a bow on this project: integrating tracking algorithm version 3 into the app, improving the manual start/stop tracking UI, and recording and uploading my talk in English.

I wrote two development logs (Devlog 1 Devlog 2) about Eki Live that go into further detail. Check them out and free free to reach out.

3 Swift Concurrency Challenges from the Last 2 Weeks

Tue, 12 Aug 2025 06:38:00 -0500

I started my Apple platforms development journey a year before Grand Central Dispatch was released with iOS 4. I’ve lived through codebase migrations to NSOperation. Then through the slew of FRP frameworks (of which I consider a concurrency solution): ReactiveCocoa, ReactiveSwift, RxSwift, and finally Combine.

My strategy for learning all these paradigms was best described as osmosis while encountering and solving real problems in codebases. Of course, you have to spend time setting breakpoints and patiently stepping through with a debugger to see where all the thread hops are happening. Eventually, I reached the point where I could read code and predict which threads each section would run on. I had 80% of the operators memorized and knew exactly where in the docs to look for the remaining 20%.

Years in, that kind of confidence has eluded me so far with Swift Concurrency. I cannot yet read a snippet of code and predict what the call stack will look like. I haven’t memorized enough of the syntax to formulate solutions in my head and write it fluently. I don’t have a go-to location in the docs to find the primitive at the tip of my tongue.

I ask myself, why is my Swift Concurrency upskilling story so different?

Is the difference that GCD, NSOperation, and the reactive frameworks basically came out of the gate fully baked? Their paradigms may have merits and demerits, but after their first releases, anything new was additive or syntactic sugar. They were born as ugly as they’d always be.

Is it that my confidence was actually unearned and I never really understood what was happening in my code? The kind of bugs that Swift Concurrency aims to solve are often so rare that you can go a whole career without being able to recognize the symptoms.

Is it that Swift Concurrency promises a higher level of abstraction (isolation domains instead of threads), but is so leaky that the programmer now has to understand both the abstraction and the paradigm it’s abstracting?

With that preface, I want to look at a few examples of Swift Concurrency challenges I’ve encountered recently. Let’s see if these shed any light on why I’m finding it so hard to develop intuition.

Unlike most of my posts (I hope), these examples are unsolved problems with likely broken code.

1. UNNotificationCenter

I implemented push notifications in Technicolor. I already had an overall architecture I was happy with, but elegantly massaging push notification support into it took some effort. I finally ended up with an implementation I thought I understood that still properly handled the multiple delegate callback points across UIApplicationDelegate via @UIApplicationDelegateAdaptor and UNNotificationCenter.

Besides the push token registration and user permissions, the key functionality of my push notifications wrapper client is to open the screen of the app that corresponds with the type of push notification the user tapped.

As far as I can tell, UNNotificationCenter has no documented concurrency story. Each developer that sets out to use the framework has to derive what thread each delegate method is called on through trial and error on a real device in production by tapping production push notifications. We haven’t even started talking about Swift Concurrency yet.

To bring User Notifications framework into the concurrency world, each developer needs to start from zero, with zero guarantees and zero support from Apple.

So I started with this (simplified) implementation:

// Version 1
final class PushNotificationDelegateProxy: NSObject, UNUserNotificationCenterDelegate, @unchecked Sendable {
    private var notificationTapHandler: ((PushNotificationPayload) -> Void)?

    func setNotificationTapHandler(_ handler: @escaping (PushNotificationPayload) -> Void) {
        notificationTapHandler = handler
    }

    func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse) async {
        let userInfo = response.notification.request.content.userInfo

        let payload = // Decode payload
        notificationTapHandler?(payload)
    }
}

@MainActor @Observable final class PushNotificationSettingsStore: Identifiable {
    @ObservationIgnored @Dependency(\.pushNotificationClient) var pushNotificationClient

    init() {
        pushNotificationClient.setDelegate()

        conditionallyRegisterForRemoteNotifications()

        // Set the notification tap handler
        pushNotificationClient.setNotificationTapHandler { [weak self] payload in
            Task { @MainActor in
                self?.actions.notificationReceived(payload)
            }
        }
    }
}

In my initial understanding, no matter what thread userNotificationCenter(didReceive:) called the notificationTapHandler closure, inside it the closure notificationReceived(payload) would be called on the main thread and could do any UI operations it needed to.

From what I remember, I confirmed this as working during debug on device. When I was running in production on TestFlight it crashed when I opened a push notification.

Still unsure as to why (was it that @MainActor-isolated self was captured inside a non-isolated closure), I added some code that, although inelegant, would surely fix the problem:

// Version 2
final class PushNotificationDelegateProxy: NSObject, UNUserNotificationCenterDelegate, @unchecked Sendable {
    private var notificationTapHandler: ((PushNotificationPayload) -> Void)?

    func setNotificationTapHandler(_ handler: @escaping (PushNotificationPayload) -> Void) {
        notificationTapHandler = handler
    }

    func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse) async {
        let userInfo = response.notification.request.content.userInfo

        let payload = // Decode payload
        await MainActor.run {
            notificationTapHandler?(payload)
        }
    }
}

Just sprinkle in MainActor.run everywhere, right? Just like back in the DispatchQueue.main days.

This also crashed (every time) in production, with the following stack trace:

Triggered by Thread:  5

Last Exception Backtrace:
0   CoreFoundation                	0x184a5721c __exceptionPreprocess + 164 (NSException.m:249)
1   libobjc.A.dylib               	0x181ef1abc objc_exception_throw + 88 (objc-exception.mm:356)
2   Foundation                    	0x183d55670 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 288 (NSException.m:252)
3   UIKitCore                     	0x1883ad4e8 -[UIApplication _performBlockAfterCATransactionCommitSynchronizes:] + 276 (UIApplication.m:3408)
4   UIKitCore                     	0x1883bdecc -[UIApplication _updateStateRestorationArchiveForBackgroundEvent:saveState:exitIfCouldNotRestoreState:updateSnapshot:windowScene:] + 528 (UIApplication.m:12129)
5   UIKitCore                     	0x1883be278 -[UIApplication _updateSnapshotAndStateRestorationWithAction:windowScene:] + 144 (UIApplication.m:12174)
6   technicolortv                 	0x1027d6670 @objc closure #1 in PushNotificationDelegateProxy.userNotificationCenter(_:didReceive:) + 80 (/<compiler-generated>:0)
// ...
12  libswift_Concurrency.dylib    	0x190521241 completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) + 1 (Task.cpp:537)

State restoration somehow got triggered off the main thread? How? I guess I could understand if the problem were that notificationTapHandler is unsafe to be passed between isolation domains. But that’s not what the crash is saying is it?

Let’s push the isolation annotations even further:

extension UNUserNotificationCenter: @retroactive @unchecked Sendable {}
extension UNNotificationResponse: @retroactive @unchecked Sendable {}

final class PushNotificationDelegateProxy: NSObject, UNUserNotificationCenterDelegate, @unchecked Sendable {
    private var notificationTapHandler: (@MainActor (PushNotificationPayload) -> Void)?

    func setNotificationTapHandler(_ handler: @escaping @MainActor (PushNotificationPayload) -> Void) {
        notificationTapHandler = handler
    }

    @MainActor func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse) async {
        let userInfo = response.notification.request.content.userInfo

        let payload = // Decode payload
        notificationTapHandler?(payload)
    }
}

Let’s try to force userNotificationCenter(didReceive:) to be called on the @MainActor by lying to the compiler in a bunch of places about sendability.

As far as I can tell, this no longer crashes although I haven’t had time to thoroughly test it (since it requires multiple deployments to Test Flight and review cycles to extensively QA).

From this whole weeks-long process, I’ve learned essentially nothing about the proper usage of Swift Concurrency, my mental model is less well-formed than it used to be, and I still have a lingering problem that I cannot practically devote enough time to comprehensively solve right now.

Presumably no one at Apple is working on the User Notifications framework anymore. Nothing is being added to it. No one is giving it concurrency support. No one has (over the past decade) or will document its concurrency story. The best we have is a Stack Overflow post with no unanimous best practice and no solution.

Update 2025/08/18

Mike Apurin solved the underlying issue (the fix is the same). From the stack trace, completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) is the closure automatically created and called by Swift at the end of the async version of the userNotificationCenter(_:didReceive:) function. Although it’s not documented, the crash leads me to believe this closure must be called on the main thread.

We have no direct control over what actor the closure runs on; it’s presumably whatever actor the function is isolated to. This means that we can only:

Add @MainActor to the Delegate class.
Add @MainActor to the function. (the fix we used in the post)
Use the non-async version of the function, userNotificationCenter(_:didReceive:withCompletionHandler:) and wrap the completion handler in a MainActor.run closure.

Update 2025/11/18

Tiziano Coroneo alerted me to a new Swift 6.2 feature that helps address this problem called isolated conformances (further explained by Hacking with Swift.

Tiziano says his declaration now looks like this:

public final class UserNotificationManager: NSObject, @MainActor UNUserNotificationCenterDelegate {

My fix is similar but a little more involved since it involves wrapping my class for use with the Swift Dependencies library. I’ll try to document it fully in a future post.

2. CMMotionActivityManager

Core Motion is another neglected framework that not many iOS devs have the pleasure of integrating. It seems to have mostly been “modernized” around iOS 7 when NSOperation had a small popularity bump. Also when the API best practices for getting permission from the device user were still being tweaked.

For a current project, I’m using CMMotionActivityManager. It’s a slightly higher-level data source for predicting whether the device is held by a user walking, cycling, driving, standing still, etc. It has two primary APIs:

Request historical data for a time range (up to 1 week ago): queryActivityStarting(from:to:to:)
Get live streaming data while the app is in the foreground: startActivityUpdates(to:)

queryActivityStarting returns once to the OperationQueue specified in the parameters. startActivityUpdates keeps returning values until stopActivityUpdates() is called.

I thought each of these would be relatively straightforward to wrap into a withCheckedContinuation and AsyncStream, respectively.

// Version 1: queryActivityStarting
Task { @MainActor [weak self] in
    guard let self else { return }

    // Check motion history for timeout
    let timeout = await withCheckedContinuation { continuation in
        activityManager.queryActivityStarting(
            from: startDate,
            to: endDate,
            to: OperationQueue.main
        ) { activities, error in
            guard let activities, error == nil else {
                continuation.resume(returning: nil)
                return
            }
            
            let value = // do work here to calculate timeout...
            continuation.resume(returning: value)
        }
    }
    
    // ...
}

My strategy recently has been the same as the Swift team’s: isolate everything on the @MainActor unless there’s a reason not to. So that’s what I did in this case. However, is it correct to await with a MainActor-isolated Task while also having the queryActivityStarting function return on OperationQueue.main?

It seemed like it was working fine in debug, but then when I was field testing I was observing what seemed to be deadlocks and the continuation never completing.

So I’m already having trouble reliably reproducing the bug. I’m not even sure whether the bug is related to concurrency or whether queryActivityStarting just doesn’t respond the way I’d expect. After all, there’s a note in the docs that could be interpreted a number of ways:

This method runs asynchronously, returning immediately and delivering the results to the specified handler block. A delay of up to several minutes in reported activities is expected.

In the end, I created this extension that forces @MainActor at the function level and includes a timeout just in case. It seems to work so far.

extension CMMotionActivityManager {
    @MainActor func activities(from start: Date, to end: Date, timeout: TimeInterval) async throws -> [CMMotionActivity]? {
        try await withThrowingTaskGroup(of: [CMMotionActivity]?.self) { group in
            group.addTask {
                try await withCheckedThrowingContinuation { continuation in
                    self.queryActivityStarting(from: start, to: end, to: .main) { activities, error in
                        switch (activities, error) {
                        case let (_, err?):
                            continuation.resume(throwing: err)
                        case let (list?, nil):
                            continuation.resume(returning: list)
                        default:
                            continuation.resume(returning: [])
                        }
                    }
                }
            }

            group.addTask {
                try? await Task.sleep(for: .seconds(timeout))
                return nil
            }

            guard let result = try await group.next() else {
                return nil
            }

            group.cancelAll()
            return result
        }
    }

    @MainActor func activityUpdates() -> AsyncStream<CMMotionActivity> {
        AsyncStream { continuation in
            startActivityUpdates(to: .main) { activity in
                guard let activity else {
                    continuation.finish()
                    return
                }
                continuation.yield(activity)
            }

            continuation.onTermination = { @Sendable _ in
                self.stopActivityUpdates()
            }
        }
    }
}

Again, I feel as if I’ve learned nothing. I have potentially worse intuition than I started with, and I may have to return to this problem again after I’ve released to production.

3. Actor Reentrancy

I’ve been working on a meatier problem within Eki Live for a couple months.

I have the following pipeline:

Core Location produces CLLocation values up to 1 per second.
CLLocations are passed to an actor RailwayTracker which processes them using various info from a database and incorporates that data into the actor long-term state.
RailwayTracker returns a value that is displayed in the UI via SwiftUI and can also update a LiveActivity.

Essentially:

CLLocationManagerDelegate -> RailwayTracker -> ContentView

The actual architecture is a bit more complex due to the relationship between instances of classes doing each part of the work. But the overall pipeline design has always felt precarious due to the underlying assumptions that:

Core Location will never produce CLLocation values faster than RailwayTracker can process them.
RailwayTracker will always process inputs in order, serially.
SwiftUI will receive outputs from RailwayTracker no faster than it can display them.

In practice, I was probably breaking at least the first constraint during testing, since I could simulate CLLocations being produced at 20x real-time speed in a separate macOS app I was using to iterate on the algorithm within RailwayTracker.

Here is a very simplified version of the shape of RailwayTracker:

struct RailwayTrackerResult {
    let location: Location
    let value: Int
}

actor RailwayTracker {
    private let railwayDatabase: (any DatabaseReader)?
    
    private var runningValues: [Location: Int] = [:]

    init(railwayDatabase: (any DatabaseReader)?) {
        self.railwayDatabase = railwayDatabase
    }

    func process(_ input: Location) async -> RailwayTrackerResult {
        // lots of database reads
        // lots of reads from `runningValues`
        // lots of updates to `runningValues`
        return RailwayTrackerResult(location: input, value: someCalculatedValue)
    }

    func reset() {
        runningValues = [:]
    }
}

When originally designing RailwayTracker, an actor seemed like the obvious choice. I wanted a separate isolation for its internal state and I knew it’d be doing enough heavy work that it wasn’t feasible to do on the main actor.

However, I misinterpreted the behavior of actor, thinking that an actor also ensured that an instance’s functions would need to complete before they could be called again. In practice, actors don’t do anything to prevent reentrancy. Meaning that the input locations could be being processed out-of-order by the actor with the database operations being interleaved and there being all kinds of chaos and unspecified behavior.

While doing some new work on the project, I wanted to take another stab at hardening the entire pipeline using Swift Concurrency.

I researched the current state of the Apple officially-sanctioned swift-async-algorithms package. It seemed to be in committee-hell with no real forward progress in the last 2+ years. There’s less than half of the Combine operator API implemented. There’s something called an AsyncChannel. Would that be a good primitive to base my pipeline on?

I decided on my specification:

CLLocations should never be dropped.
CLLocations should always be fully processed one-by-one in the order they arrive.
Pipeline output results should be delivered to one “subscriber” (mixing metaphors here), but I may need multiple in the near future.

I ended up with a generic wrapper abstraction called SerialProcessor:

final class SerialProcessor<Input: Sendable, Output: Sendable> {
    typealias Process = @Sendable (Input) async -> Output

    // Inbound (sync) and outbound (single-consumer) pipes
    private let inPair: (stream: AsyncStream<Input>, continuation: AsyncStream<Input>.Continuation)
    private let outPair: (stream: AsyncStream<Output>, continuation: AsyncStream<Output>.Continuation)

    private let worker: Task<Void, Never>
    private let process: Process

    /// Single-consumer stream of `Output`.
    var results: AsyncStream<Output> { outPair.stream }

    init(
        inputBuffering: AsyncStream<Input>.Continuation.BufferingPolicy = .unbounded,
        outputBuffering: AsyncStream<Output>.Continuation.BufferingPolicy = .unbounded,
        process: @escaping Process
    ) {
        self.process = process

        let inPair = AsyncStream.makeStream(of: Input.self, bufferingPolicy: inputBuffering)
        let outPair = AsyncStream.makeStream(of: Output.self, bufferingPolicy: outputBuffering)
        self.inPair = inPair
        self.outPair = outPair

        worker = Task {
            for await input in inPair.stream {
                let output = await process(input)
                outPair.continuation.yield(output)
            }
            outPair.continuation.finish()
        }
    }

    deinit {
        finish()
    }

    func submit(_ input: Input) {
        inPair.continuation.yield(input)
    }

    func finish() {
        inPair.continuation.finish()
        worker.cancel()
    }
}

I couldn’t use AsyncChannel for the input side because its send function is async and therefore requires an async context. I don’t have that inside the CLLocationManagerDelegate callback function that delivers CLLocations and creating a new Task for each new CLLocation would break the serial ordering guarantee.

In the non-realtime system implementation, usage looks like this:

// Set up the processing
let railwayTracker = RailwayTracker(railwayDatabase: database)
let serialProcessor = SerialProcessor(
    inputBuffering: .unbounded,
    outputBuffering: .unbounded,
    process: { @Sendable input in
        await railwayTracker.process(input)
    }
)

// Queue all locations immediately for processing
// This is in a `Task` so it will run after `serialProcessor.results` is set up
Task {
    for location in locations {
        serialProcessor.submit(location)
    }
}

// Cache the results as they arrive
for await result in serialProcessor.results {
    resultsCache[result.location.id] = result
}

In the realtime system version, I believe the implementation will look something like this (although I’m not at this point in the project yet):

// Set up the processing global
let railwayTracker = RailwayTracker(railwayDatabase: database)
let serialProcessor = SerialProcessor(
    inputBuffering: .unbounded,
    outputBuffering: .bufferingNewest(1), // for UI, drop late values
    process: { @Sendable input in
        await railwayTracker.process(input)
    }
)

// Where we receive new locations from the system
extension LocationManagerDelegate: @preconcurrency CLLocationManagerDelegate {
    func locationManager(_ manager: CLLocationManager, didUpdateLocations locations: [CLLocation]) {
    	// Get the shared `serialProcessor` from somewhere global then...
    	for location in locations {
            serialProcessor.submit(location)
        }
    }
}

// In some View Model...
@MainActor @Observable final class ContentViewModel {
    @ObservationIgnored let serialProcessor: SerialProcessor<CLLocation, RailwayTrackerResult>
    private(set) var latestResult: RailwayTrackerResult?
    
    func task() async {
        for await result in serialProcessor.results {
            latestResult = result
        }
    }
    
    // ...
}

In my testing of the non-realtime setup, the system seemed to work correctly. It buffers inputs as expected, and even drops outputs if I use .bufferingNewest(1).

I’m still actively working on this problem (and should be doing so right now instead of writing this post).

There are other libraries like AsyncExtensions that build on the Swift primitives. However, getting my head around the shape of the problems they can solve and what I can accomplish with the primitives has been tough.

Final thoughts and complaints

Swift Concurrency is just such a bummer. We still need to know about lower-level primitives like locks and mutexes and threads. We still need to have working knowledge of past solutions like Grand Central Dispatch in order to interact with our own and Apple’s legacy APIs. Unlike previous solutions, there are fewer primitives for managing serial vs. concurrent processing. It adds a new abstraction layer and several new concepts (isolation, actors, structured/unstructured) that probably will make sense eventually but don’t right now. It adds a dozen new keywords with more essential ones arriving with each new point update. It’s difficult or impossible to find sanctioned sample code that provides a glimpse into what a “best practice” could be. We’re still not safe from runtime crashes.

I could keep going but I’m tired of complaining. I just want to have the confidence to write my features without having to budget multiple days of field testing and debugging time and reading the Swift forums.

Full-Stack Swift: The Technical Architecture of Technicolor

Mon, 04 Aug 2025 07:04:00 -0500

In my previous post about Technicolor I gave an overview of my hybrid chat app & social network for watching TV shows with friends asynchronously.

Technicolor is a side-project I’ve been iterating on for over a decade (I’ve only used it with small groups of friends). It started its life as a Ruby on Rails app with browser-only support and has now been reborn as a full-stack Swift-on-server web service and native Apple platforms client app.

This post explores the front-end and back-end architectures of Technicolor and some observations about developing its full-stack Swift server and client apps intermittently over several years. The project is not yet open source, but I will share code snippets throughout to illustrate parts of the architecture in context.

TL;DR: Any benefits of full-stack Swift do not meaningfully outweigh its demerits or opportunity costs for any use case I can imagine (happy to change my mind though). But existing Apple platforms developers may enjoy the familiarity of Swift in a new setting and the technical challenge of server-side development in an academic sense.

Technicolor app showing the three main user-facing screens: dashboard for managing active rooms, room interface for timestamped chat during episodes, and profile management screen

Architecture overview
Development experience
Shared API layer
Server-side
Client-side
Lessons learned

Architecture overview

  ┌─────────────────────┐       ┌──────────────────────────┐
  │  Server (tv-vapor)  │       │  Client (Technicolor)    │
  │                     │       │       iOS/macOS          │
  ├─────────────────────┤       ├──────────────────────────┤
  │ • Swift Vapor       │       │ • SwiftUI + Observation  │
  │ • SQLite + Fluent   │       │ • iOS 17+ / macOS 14+    │
  │ • TMDB API Client   │       │ • Mac Catalyst Support   │
  │ • Push Notifications│       │                          │
  │ • Deployed on Fly.io│       │                          │
  └─────────────────────┘       └──────────────────────────┘
                  │                           │
                  └─────────────┬─────────────┘
                                │
                                ▼
                  ┌─────────────────────────────┐
                  │    Shared API Layer         │
                  │       tv-models             │
                  ├─────────────────────────────┤
                  │ • 164 Codable Structures    │
                  │ • Type-Safe Client/Server   │
                  │ • Input/Output DTOs         │
                  └─────────────────────────────┘

Technicolor has a client-server architecture. The server vends json data via HTTP requests to clients authenticated with a bearer token.

The server is written in Swift using the Vapor web framework. The primary database is SQLite via the Fluent sub-framework. It fetches metadata about TV shows and movies from the TMDB API and caches the data in SQLite. It’s deployed to a single Machine on PaaS Fly.io.

The client is written in Swift and SwiftUI and supports iOS and macOS (via Mac Catalyst or Designed for iPad) with one codebase and two targets.

There is a Swift package called tv-models, imported by both the client and server, that contains the Codable models that form the shared API layer. This shared API layer was the primary motivator for using Swift everywhere.

The entire project is contained in a Git mono-repo. The sources for the server side, client side, and shared models live in their own individual directories within the project directory. Most configuration files like the Dockerfile live in the project directory.

I’ll explore the development experience, shared API layer, server-side, and client-side aspects in more detail.

Development experience

The blessing and curse of using full stack Swift is that I can use Xcode for everything. The cons of course are that Xcode can be bloated and buggy. But the pros are that I can explore, develop, and debug the entire codebase in one IDE that’s fine-tuned for Swift.

Xcode workspace and iOS simulator simultaneously running both client and server with live console output from the server

The specific setup within Xcode is a single xcworkspace file that contains:

An XcodeGen generated xcodeproj file for the iOS/macOS clients.
A Swift package for the tv-models shared DTO models.
A Swift package for the server-side Vapor project.

Overall, the xcworkspace-based setup worked okay. There were a few times where Swift Package caching needed manual fixing (several wasted hours I won’t be getting back). I’ve also wrestled with an issue where Swift Packages can force-create schemes in the workspace that cannot be ignored.

I have separate schemes for running the server and client, using separate destinations for running the iOS and macOS clients. Both the server and client are debuggable with all of Xcode’s integrated LLDB support including breakpoints. The console shows logs from server and client.

Server console output showing Vapor web server startup logs and selector to view logs for the client scheme

It’s certainly powerful to be able to set a breakpoint in a server endpoint handler and the client view model and step through the request and response cycle from both sides.

Xcode only pre-builds the active target. For example, if I make a change to the shared tv-models layer while the client target is active, Xcode won’t show me I’ve introduced a compiler error on the server until I switch to the server target and build it. Similarly, if a server source file is visible in the editor window while the client target is active, Xcode will often show a bunch of false-positive errors inline that you need to remember to ignore.

Xcode is confused when the client scheme is selected but server source is visible in the editor

It’s still possible to build and run and test the server from outside Xcode, and I did often during this most recent development cycle with Claude Code. Adding coding agents made the all-in-one Xcode integration experience less impactful than it was a couple years ago.

I use SwiftFormat and a .swiftformat rules file in the project directory to maintain formatting across all Swift source files.

My overall takeaway is that the many of the benefits of server-client development inside Xcode don’t outweigh the demerits of the server-side Swift ecosystem’s relative immaturity. It’s a lonely experience using Xcode as the IDE of choice for server-side development, even if the backend is Swift.

Shared API layer

The tv-model Swift package contains the handful of Codable struct definitions used by each Controller on the server side and the APIClient on the client side. These are often referred to as data transfer objects or DTOs.

This essentially enforces a type-safe client-server API. The caveat of course is that model structs and fields are add-only and higher-level API versioning rules apply.

An example of some shared Comment-related models from tv-models:

public enum Comment {
    struct CreateInput: Equatable, Codable, Sendable {
        public let roomID: UUID
        public let content: String
        public let seconds: Int

        public init(roomID: UUID, content: String, seconds: Int) { */ ... */ }
    }

    struct CreateOutput: Equatable, Codable, Sendable {
        public let roomID: UUID
        public let comment: Full

        public init(roomID: UUID, comment: Full) { */ ... */ }
    }

    struct Full: Equatable, Codable, Sendable {
        public let id: UUID
        public let createdAt: Date
        public let updatedAt: Date
        public let content: String
        public let seconds: Int
        public let user: User.Stub

        public init(id: UUID, createdAt: Date, updatedAt: Date, content: String, seconds: Int, user: User.Stub) { */ ... */ }
    }
}

For the API endpoint envelope models, my convention is using Create, Edit, Delete, Show, etc. prefixes and Input or Output with respect to the server. In other words, the client creates Input models and receives Output models.

I use Full for models that contain the majority of the database model’s data. I use Stub for smaller subsets. It’s generally worked out well to keep the amount of model sub-types low while having the flexibility to create new ones when it makes sense. It simplifies the client side, allowing model types to be passed between sub-systems (unlike GraphQL which has unique model-types per request).

On the client-side, these models are mostly used as-is throughout the codebase, even in the View layer.

On the server-side, Fluent ORM has its own class-based model definitions that are used to interact with the source-of-truth database data. There is a custom encoding/decoding layer for each DTO.

/// Fluent ORM model
final class Comment: Model, @unchecked Sendable {
    static let schema = "comments"

    @ID(key: .id)
    var id: UUID?

    @Timestamp(key: "created_at", on: .create)
    var createdAt: Date?

    @Timestamp(key: "updated_at", on: .update)
    var updatedAt: Date?

    @Field(key: "content")
    var content: String

    @Field(key: "seconds")
    var seconds: Int

    @Parent(key: "user_id")
    var user: User

    @Parent(key: "room_id")
    var room: Room

    init() {}
}

/// Converting a Fluent model to a DTO
extension TV.Comment.Full {
    init(_ comment: Comment) throws {
        guard let createdAt = comment.createdAt,
              let updatedAt = comment.updatedAt
        else {
            throw Abort(.internalServerError)
        }

        try self.init(
            id: comment.requireID(),
            createdAt: createdAt,
            updatedAt: updatedAt,
            content: comment.content,
            seconds: comment.seconds,
            user: TV.User.Stub(comment.user)
        )
    }
}

Although the model definitions are shared and type-safe, the endpoint URLs themselves are duplicated across client and server. There is probably a way I could share these as well, but at the moment I haven’t found this to be enough of a maintenance burden or source of bugs that I need to spend time trying to harmonize it.

My takeaway is that this setup is definitely convenient, but the more popular web frameworks have solved this problem in other ways like using OpenAPI and its code generators. Technicolor has some complexity (164 model structures), but nowhere near that of a large scale SaaS or social network.

Server-side

Technicolor began its life back in 2013 as a Ruby on Rails app with a web client. In 2017 I made a brief foray into rewriting the backend in the Elixir language with the Phoenix web framework, but quickly abandoned that effort when Swift Vapor began gaining some popularity.

Thoughts on the Swift Vapor framework

When I first began development on the Technicolor rewrite, Vapor was still using the EventLoopFuture concurrency primitives from SwiftNIO. EventLoopFuture felt similar to a FRP framework like Combine or RxSwift. But in comparison to async await, it was really painful. Swift’s type inference is awful with the amount of mapping closures required for EventLoopFuture. In these early stages, even simple endpoints took hours to write and test. Compare the EventLoopFuture version of the Comment create func here to the async version in the next section:

func create(req: Request) throws -> EventLoopFuture<TV.EmptyOutput> {
    let userID = try req.auth.require(UserToken.self).user.requireID()
    let input = try req.content.decode(TV.Comment.CreateInput.self)
    let comment = Comment(content: input.content, seconds: input.seconds, userID: userID, roomID: input.roomID)
    return RoomUser.query(on: req.db)
        .filter(\.$room.$id == input.roomID)
        .filter(\.$user.$id == userID)
        .first()
        .unwrap(or: Abort(.unauthorized))
        .flatMap { _ in comment.create(on: req.db) }
        .flatMapThrowing { try comment.requireID() }
        .flatMap { newCommentID in
            Comment.query(on: req.db)
                .with(\.$user)
                .filter(\.$id == newCommentID)
                .first()
                .unwrap(or: Abort(.internalServerError))
        }
        .flatMapThrowing { newComment -> EventLoopFuture<TV.Comment.Room> in
            let fullComment = try TV.Comment.Full(newComment)
            let roomComment = TV.Comment.Room(roomID: input.roomID, comment: fullComment)
            return RoomUser.query(on: req.db)
                .with(\.$user)
                .field(\.$user.$id)
                .filter(\.$room.$id == input.roomID)
                .all()
                .flatMapThrowing { roomUsers -> TV.Comment.Room in
                    let userIDs = roomUsers.compactMap(\.user.id)
                    try req.webSocketClient.send(message: .comment(roomComment), to: userIDs)
                    return roomComment
                }
        }
        .transform(to: TV.EmptyOutput())
}

As Swift Concurrency matured, the Vapor team finished their early work on supporting async/await alongside EventLoopFuture. I spent days tediously rewriting the existing EventLoopFuture signal chains. The async versions looked a lot better, but it was hard won.

Coming back to the project after a couple years, the Vapor team seems to be stalled in finishing up Swift Strict Concurrency support. Fluent models must still be declared @unchecked Sendable. I’m running Swift 6.0 on the deployment, but with Swift 5 language mode.

I don’t want to discount the laudable work done by the Vapor core team and community. But going up against the mature frameworks from JS, Ruby, Python, PHP, etc., dealing with the low prioritization of Swift on the server from Apple, and with the overall churn of the Swift language, my take is that using Vapor is the wrong choice if your aim is pragmatism.

However, like I’ve mentioned so far, there are upsides to using Swift on the server. In theory it’s faster than interpreted languages and the type safety makes refactors safer and obviates the need for massive test suites.

Routes and Controllers

Here is a taste of what it looks like to define the Comment-related routes.

struct CommentController: RouteCollection {
    func boot(routes: RoutesBuilder) throws {
        let comments = routes
            .grouped(UserToken.authenticator())
            .grouped(UserToken.guardMiddleware())
            .grouped("rooms")
            .grouped("comments")

        comments.post("create", use: create)
        comments.post("delete", use: delete)
        comments.post("edit", use: edit)
    }
    
    // ...
}

This generates 3 POST routes that use middleware to parse out a valid auth token and make it available to the request:

/rooms/comments/create
/rooms/comments/delete
/rooms/comments/edit

My unique convention is to use POST for all requests, even read-only CRUD operations like show that would usually be GET. The reasoning behind this is that it allows me to use the same JSON-encoded HTTP body plumbing for all requests instead of having to selectively encode and decode either or both from the URL query and the HTTP body.

The actual request/response implementation on the server side (modern async version) looks something like this:

// CommentController.swift
func create(req: Request) async throws -> TV.Comment.CreateOutput {
    let userID = try req.auth.require(UserToken.self).user.requireID()
    let input = try req.content.decode(TV.Comment.CreateInput.self)

    // Ensure the User is allowed to post Comments in this Room    
    _ = try await RoomUser.query(on: req.db)
        .filter(\.$room.$id == input.roomID)
        .filter(\.$user.$id == userID)
        .first() ?! Abort(.unauthorized)

    // Prefer using timestamp from content string if it exists
    let (finalContent, finalSeconds): (String, Int)
    if let parsed = TimestampParser.parseTimestampFromContent(input.content), let parsedSeconds = parsed.seconds {
        finalContent = parsed.parsedContent
        finalSeconds = parsedSeconds
    } else {
        finalContent = input.content
        finalSeconds = input.seconds
    }

    // Create the comment in the database
    let comment = Comment(content: finalContent, seconds: finalSeconds, userID: userID, roomID: input.roomID)
    try await comment.create(on: req.db)
    
    // Prepare all data needed to populate the DTO
    let newCommentID = try comment.requireID()
    let newComment = try await Comment.query(on: req.db)
        .with(\.$user)
        .filter(\.$id == newCommentID)
        .first() ?! Abort(.internalServerError)

    let fullComment = try TV.Comment.Full(newComment)
    let output = TV.Comment.CreateOutput(roomID: input.roomID, comment: fullComment)
    return output
}

For most CRUD operations, the endpoint implementation is pretty straightforward: parse inputs, specific model authorization, fetch some data, modify and write some data back to the database, convert data to a DTO and return. So much so that coding agents have a pretty easy time interpreting a specification to add or modify existing endpoints.

Authentication

Early on in the project, I decided to roll my own email-identity, bearer token authentication using the primitives provided by Vapor. The reasoning was to keep things simple while getting a better understanding of what goes on behind the scenes as (primarily) a client-side developer. Although I haven’t touched this code in years, I still think I have a better understanding of auth frameworks than I did before giving it a go.

As a quick summary, all requests to Technicolor use bearer token authorization with the exception of:

Create Account: uses no authentication, but checks against an invite token in the request data to ensure the new user is allowed to create an account.
Log In: uses HTTP Basic authentication.

Services

A critical decision I made early on that affects the architecture of the server codebase: I’m deploying to a single server with direct access to the SQLite database. As I discussed in Configuring Swift Vapor on Fly.io with SQLite, there are several tradeoffs to this decision with the primary benefit being simplicity.

This decision works in concert with defining and using services in Controllers (request handlers). State within services can be shared between requests if necessary (using proper locking for thread safety) without the additional complexity of worrying about sharing that data across parallel app servers.

For example, this is how I define and use the client that fetches data from TMDB.

extension Application {
    private struct TMDBClientKey: StorageKey {
        typealias Value = TMDBClient
    }

    var tmdbClient: TMDBClient {
        get {
            storage[TMDBClientKey.self]!
        }
        set {
            storage[TMDBClientKey.self] = newValue
        }
    }
}

public func configure(_ app: Application) async throws {
    // ...
    
    let tmdbApiKey = Environment.get("TMDB_API_KEY") ?? ""
    let tmdbClient = TMDBClient(httpClient: app.http.client.shared, apiKey: tmdbApiKey, logger: app.logger)
    app.tmdbClient = tmdbClient

    try await routes(app)
}

final class TMDBController: RouteCollection {
    // ...
    
    func search(req: Request) async throws -> TV.TMDBPagedResponse<TV.TMDBMultiSearchResult> {
        let input = try req.content.decode(TV.TMDBSearchInput.self)

        guard !input.query.isEmpty else {
            throw Abort(.badRequest, reason: "Query is required")
        }

        let page = input.page ?? 1
        let tmdbClient = req.application.tmdbClient
        let mediaTypeEnum = input.mediaType

        return try await tmdbClient.search(query: input.query, mediaType: mediaTypeEnum, page: page)
    }
}

Testing

I have a db migration that sets up a very minimal set of test users with hard coded token values to make local development easier.

In the early stages of the project, I maintained a Paw file (now RapidAPI) to facilitate manual testing of server endpoints.

In this latest sprint, I’ve developed a full Swift Testing test suite with ~150 tests. Especially when endpoints only depend on the local database, the request/response cycle is a lot more straightforward to write automated tests for than what I’m used to on the iOS side.

The addition of the external TMDB client and caching has made some endpoints a little tricker to test, but overall I feel confident that my test suite is providing value.

As a quick example, all tests follow the below pattern pretty closely. Over time, I’ve built up a lot of helpers to keep the tests focused, consistent, and easy to read and write.

@Suite("CommentController Tests")
struct CommentControllerTests {

    @Suite("Comment Creation")
    struct CommentCreationTests {

        @Test("Create comment with valid input succeeds")
        func createCommentSuccess() async throws {
            try await withApp(configure: configureTestApp) { app in
                try await withAuthenticatedUser(on: app) { user, userToken in
                    // Create a room and add user to it
                    let room = try await createTestRoom(on: app.db, withUsers: [user])

                    let createInput = try TV.Comment.CreateInput(
                        roomID: room.requireID(),
                        content: "This is a test comment",
                        seconds: 120
                    )

                    try await app.testing().authenticatedRequest(.POST, "/rooms/comments/create", token: userToken, beforeRequest: { req in
                        try req.content.encode(createInput)
                    }) { res async throws in
                        #expect(res.status == .ok)

                        try assertDecodable(res, as: TV.Comment.CreateOutput.self) { response in
                            #expect(response.comment.content == "This is a test comment")
                            #expect(response.comment.seconds == 120)
                            #expect(response.comment.user.username == user.username)
                            let roomID = try room.requireID()
                            #expect(response.roomID == roomID)
                        }
                    }
                }
            }
        }
    }
}

All 163 tests run in parallel with a separate copy of the migrated test database and complete on my MacBook Pro in about 12 seconds.

Deployment

I wrote a detailed post about my exact deployment setup, but here I’ll reiterate that I’ve found deployment to be difficult and often an awful time sink. I’d consider setting deployment up from the beginning of Vapor app development and updating periodically as your app grows in complexity.

Although a default Dockerfile is included in the Vapor new project generation, it can be inscrutable to devops novices like myself. I had particular trouble with ensuring the tv-vapor shared models source directory was available as a sibling directory in the development repo, but as a subfolder in the Docker image. It was a lot of slow trial and error.

As great as the Dockerfile is as a generic recipe for deployment, there are always going to be specifics you need to learn about your actual deployment destination. For me, I’ve kept using Fly.io for whatever reason, but I think any PaaS or VPS is going to have the same learning curve. Fly.io had the same kind of configuration churn I experienced with Swift and Vapor for a side project with years-long breaks in the development cycle.

One particular gotcha I ran into a few times is accidentally using Swift APIs that are unavailable on Linux. The open source Foundation has mostly hit feature parity, but there are some sibling frameworks like CryptoKit that require using an open source variant. These were cases that I would unfortunately discover when deploying a new build to Fly.io and it failing with some cryptic error message.

At the moment I’m YOLO-deploying to the production server after developing on localhost. Before exiting beta I’ll also create a development server.

Client-side

Supported platforms

When I first started working on the client side apps several ago, the promise of SwiftUI’s initial pitch of learn-once, apply anywhere was still optimistic for many of us. I built out the initial structure of the apps under the presumption of high SwiftUI compatibility across iPhone, iPad, and macOS. There were lots of SwiftUI modifier shims, and I made sure to aggressively modularize Views so they could be composed uniquely between platforms.

Unfortunately, after the first couple sprints I found myself bogged down in a lot of missing and broken APIs, especially on the macOS side. I eventually gave up on native macOS support and switched over to Mac Catalyst. In my most recent sprints, I realized that Designed for iPad actually looks and functions better than Mac Catalyst, while also requiring nearly zero API conditionals between the two platforms. One of my friends is running Technicolor on a macOS virtual machine without ARM64 support, so I can’t drop Mac Catalyst support yet.

Writing a truly native Mac app and a web client are both on my roadmap, but since this is a side project I’ll probably continue polishing the rough edges of the iOS app during the beta period.

At the moment Technicolor supports iOS 17+ and macOS 14+.

Project setup

I use XcodeGen and a project.yml file for maintaining the xcodeproj file referenced by the xcworkspace file. When using coding agents XcodeGen or similar is essentially mandatory.

I use the tv-models Swift package mentioned earlier.

I use the KeychainAccess package for storing the bearer token securely.

I use several Point-Free libraries: swift-dependencies and swift-navigation. More on these later.

Architecture rules

My client architecture is relatively templated and consistent across features. This helps me, but also helps coding agents write idiomatic code on the first attempt.

I use @Observable Store objects paired with top level SwiftUI Views for each logical screen.

All Stores follow this template:

@MainActor @Observable final class ExampleStore {
    struct State: Equatable {
        // Local feature state
    }
    
    @CasePathable
    enum Destination {
        // Navigation destinations with associated stores
        case detail(DetailStore)
        case settings(SettingsStore)
    }
    
    struct Actions {
        // Callback functions for parent communication
        var completion: (() -> Void) = unimplemented()
    }
    
    var state: State
    var destination: Destination?
    var actions = Actions()
    
    @ObservationIgnored @Dependency(\.service) var service
    private let token: Token  // Immutable data
    
    init(parameters) {
        // Initialize state and private properties
    }
    
    func task() async {
        // Startup or reappear   
    }
    
    func showDetailButtonTapped() async {
        // User actions from the view layer e.g. pushing a new view
        destination = withDependencies(from: self) {
            .detail(DetailStore())
        }
    }
    
    private func helperMethod() { }
}

All Views follow this template:

struct ExampleView: View {
    @Bindable var store: ExampleStore

    var body: some View {
        VStack {
            ...
        }
        .task { await store.task() }
        .navigationDestination(item: $store.destination.detail) { $detailStore in
            DetailView(store: detailStore)
        }
        .sheet(item: $store.destination.settings) { $settingsStore in
            SettingsView(store: settingsStore)
        }
    }
}

Parent-child relationships follow these rules:

Parent Store: Creates optional destination property (var destination: Destination?)
Parent Store: Provides method to create child store (func showChild() { destination = .child(ChildStore()) })
Parent View: Uses sheet(item: $store.destination.child) modifier
Child View: Accepts store as parameter (@Bindable var store: ChildStore or let store: ChildStore)
Child Store: Must conform to Identifiable (class identity-based)
Dependency Injection: Use withDependencies(from: self) only if parent has @Dependency vars

API Client

The API Client is not quite as streamlined as I’d like, but it’s simple to add new endpoints as needed.

I use the swift-dependencies format for the definition.

@DependencyClient
struct AppClient: Sendable {
    var createComment: @Sendable (_ input: Comment.CreateInput, _ token: Token) async -> Result<Comment.CreateOutput, Error> = { _, _ in .failure(AppClient.Error.unknown) }
    // ... one var for each endpoint
}

A notable difference from many projects is that AppClient is stateless – it does not hold onto the user’s bearer token or server environment (i.e. localhost or the server URL). Each Store is responsible for accepting the token as a dependency.

The live client is set up like this, with the static fetch function doing the heavy lifting of making the properly configured request and serializing the response.

extension AppClient {
    static func live() -> Self {
        Self(
            createComment: { input, token async -> Result<Comment.CreateOutput, Error> in
                let request = CreateCommentRequest(input: input, token: token, serverEnvironment: token.serverEnvironment)
                return await fetch(request)
            },
            /// ... all endpoints look similar
        )
    }
}

Most Requests require bearer token authorization and conform to the AuthorizedInputRequest protocol.

protocol InputRequest: FetchRequest {
    associatedtype Input: Encodable
    var authorization: AppClient.Authorization { get }
    var path: String { get }
    var input: Input { get }
    var serverEnvironment: ServerEnvironment { get }
}

protocol AuthorizedInputRequest: InputRequest {
    var token: Token { get }
}

An example request configuration:

struct CreateCommentRequest: AuthorizedInputRequest {
    typealias Output = Comment.CreateOutput
    let path = "/rooms/comments/create"
    let input: Comment.CreateInput
    let token: Token
    let serverEnvironment: ServerEnvironment
}

Again, there’s still a lot of boilerplate, but I’ve made my peace with it.

The specific endpoint can be used as a dependency in the Store:

@MainActor @Observable final class RoomStore {
    // ...
    
    @ObservationIgnored @Dependency(\.appClient.createComment) var createComment

    func createCommentTapped() async {
        // ...
        
        state.createMessageState = .mutating
        let input = Comment.CreateInput(roomID: roomID, content: state.inputText, seconds: state.inputSeconds)
        let result = await createComment(input, token)
        guard state.createMessageState == .mutating else { return }
        
        switch result {
        case let .success(output):
            var newState = state
            newState.comments[output.comment.id] = output.comment
            newState.createMessageState = .idle
            newState.inputText = ""
            state = newState
        case let .failure(error):
            state.createMessageState = .mutationFailed(.init(underlyingError: error))
        }
    }
}

Root architecture

The RootView and RootStore pair handles the boot sequence and handling login and logout.

struct RootView: View {
    @Bindable var store: RootStore

    var body: some View {
        ZStack {
            switch store.state {
            case .initialized:
                ProgressView()
                    .onAppear { store.onAppear() }
            case .restoringState:
                ProgressView()
            case let .signedOut(store):
                AuthenticationView(store: store)
            case let .signedIn(store):
                NavigationStack {
                    RoomsDashboardView(store: store)
                }
            }
        }
    }
}

@MainActor @Observable final class RootStore {
    enum State: Equatable {
        case initialized
        case restoringState
        case signedOut(AuthenticationStore)
        case signedIn(RoomsDashboardStore)
    }

    private(set) var state: State = .initialized

    @ObservationIgnored @Dependency(\.tokenLocalStorageClient) var tokenLocalStorageClient

    func onAppear() {
        restoreState()
    }

    private func restoreState() {
        guard case .initialized = state else { assertionFailure("unexpected state"); return }
        state = .restoringState

        let token = tokenLocalStorageClient.readToken()

        if let token {
            transitionToDashboard(token: token)
        } else {
            transitionToSignIn()
        }
    }
    
    // ...
}

The authenticated vs. unauthenticated domains of the app are fully isolated.

Room View

Room interface showing timestamped comments grouped by video timeline position

The RoomView – the async chat room for a TV show episode or movie – is the most complex screen in the app.

Its usage story is not quite the same as a prototypical chat room. And through my own usage I’m still trying to optimize the micro-decisions in the UX. Should the keyboard dismiss after the user sends a message? When should the main content auto-scroll?

I store the comments in a dictionary and group and sort them live on changes. This makes handling mutations more straightforward (the current user adds new comment, other user adds new comment, a comment is edited, a comment is deleted, etc.).

struct State: Equatable {
    // ...
    var comments: [UUID: Comment.Full]
    // ...
    
    var timestamps: [RoomViewModel.Timestamp] {
        let secondsGrouped = Dictionary(grouping: comments.values, by: { $0.seconds })
        let timestamps = secondsGrouped
            .sorted(by: { $0.key < $1.key })
            .map { group in
                RoomViewModel.Timestamp(
                    title: group.key.timestamp,
                    seconds: group.key,
                    comments: group.value
                        .sorted(by: { $0.createdAt < $1.createdAt })
                        .map { comment in
                            RoomViewModel.Comment(
                                id: comment.id,
                                content: comment.content,
                                username: comment.user.username,
                                relativeCreatedAt: Self.relativeDateFormatter.string(for: comment.updatedAt) ?? "??? ago",
                                color: comment.user.identityColor,
                                belongsToCurrentUser: comment.user.id == meID
                            )
                        }
                )
            }
        return timestamps
    }
}

Read-only data fetching state in the app is handled by the DataState struct. Mutation state is handled by the FallibleMutationState struct. I intentionally do not store the actual data in this struct (but I do store the Error).

enum DataState: Equatable {
    case initialized
    case loading
    case loaded
    case reloading
    case loadingFailed(StoreError)
}

enum FallibleMutationState: Equatable {
    case idle
    case mutating
    case mutationFailed(StoreError)
}

RoomStore.State handles the following read-only data and mutation data states:

struct State: Equatable {
    var dataState: DataState
    var createMessageState: FallibleMutationState
    var editCommentState: FallibleMutationState
    var deleteMessageState: FallibleMutationState
    var updateWatchedState: FallibleMutationState
    var leaveRoomState: FallibleMutationState
    var deleteRoomState: FallibleMutationState
    var isShowingLeaveRoomConfirmation: Bool
    var isShowingDeleteRoomConfirmation: Bool
    
    // ...
}

The available mutations and confirmations start to add up quickly. For this particular View, it might actually be worth it to abstract the current mutation state into its own Enum so that I can more simply enforce only one mutation is happening at a time.

Dashboard

In my initial designs of Technicolor, the dashboard was a simple list of all Rooms ordered by creation date. There was a lot of burden on each user to keep track of which episodes they’ve watched, periodically check which episodes their friends have watched, and periodically check for replies.

In this latest iteration of Technicolor, I used some additional state like “Mark as Watched” and some complex SQL queries to make watchlists easier to manage, especially for power users (like me) who are watching several shows with several groups of friends over potentially multiple weeks.

Dashboard screen organizing active rooms by TV show and member groups

The dashboard now groups sections with the following rules:

TV shows by members: if you’re watching the same show with multiple groups, these Rooms will be grouped separately
All movies by members: if you have a “weekly movie night” with the same set of friends, all those movie Rooms will be grouped into the same section.

By default, Rooms that are unwatched by at least one member will be included on the dashboard. Once all members have finished watching, the Room will appear for the next 7 days to allow time for further discussion.

Each scenario has its own archive screen so you can always access past Rooms. There’s also a comprehensive archive screen that ensures you can even find groups that have been archived.

Various archive screens so all rooms are discoverable

Timestamp control

One of my pet projects within this Technicolor was a custom timestamp adjustment control. It’s still a work in progress but it’s been fun to iterate on.

The goal of the control is to make it easier to adjust the timestamp (e.g. 15:34) of your comment to the timestamp of the running show. That is, easier than typing the timestamp using the iOS software keyboard. On macOS with a hardware keyboard it’s easy enough to type 15:43 this is my comment.

The custom timestamp control works by tapping and dragging up and down. During a drag, moving your touch to the right adjusts the fine-tune to get better accuracy.

Push notification support

One of the reasons I was excited to make native clients for Technicolor was push notification support.

Once a Room member finishes watching an episode, they tap “Mark as Watched”. This not only updates the Room status for the Dashboard, but it also sends a push notification to the other Room members. If the other members have already watched, this push will be a trigger for the user to check out the new comments. If the other members haven’t watched yet, this push is a good reminder they should watch the episode soon.

Various push notifications

There are also quality-of-life pushes for when a user accepts your invite and joins Technicolor, when you receive a new friend request, and when a user accepts your friend request.

There’s still one missing flow: I’d like to add a timeout triggered push notification that sends in the situation that:

User A has already finished watching an episode
User A replies to comments after User B finishes watching
30 minutes have passed since the last comment from User A

This would ensure User B sees the replies from User A, but doesn’t need to get an individual push notification for each comment.

Deployment script

Working on indie projects by myself, I’ve generally found it fast enough to do all my deployment to App Store Connect manually (but using a checklist).

However, for Technicolor I need to build and upload both an iOS and macOS version. This was just enough tedium that I decided to vibe code a deployment bash script that handles the minutiae of deployment.

It took a very long day of debugging certificate and provisioning profile issues, but eventually I got the script working. This has made it marginally easier to ship new builds to my TestFlight beta testers. The 24-48 hour turnaround of TestFlight App Review is still a drag though.

Release automation script output showing the complete deployment workflow with environment validation, version management, build processes, App Store Connect uploads, and GitHub operations

Lessons learned

There’s a lot of essential complexity in social networks and chat apps. As a mobile dev that usually works on the client side, it was great experience learning how to manage things like authentication, schema design for social network relationships, and server deployment.

Standardized architecture

Especially for CRUD apps, it’s incredibly important to find an architecture that makes each feature as templated and boring as possible. Predictable and well-documented features enable coding agents to accelerate development of the features that are necessary but forgettable by users and allow you to focus on the features that make your app unique.

Choice of web framework

It’s hard for me to recommend Swift on the server as a pragmatic choice for a production web service. All of its strengths don’t really make up for how far behind it is in the broader web framework ecosystem. Maybe in another several years if the Swift language has stabilized and the Swift community outside app development grows.

Long-running side projects using volatile technology stacks

I’m glad I finally found 2-3 uninterrupted weeks that I could use to get this project modernized and in a shippable state. As a side project, having a weekend available here and there usually meant that I could only tackle one small feature at a time (I remember spending a coveted 4-day weekend chipping away at getting the first deployment set up). Too much effort was burned rewriting due to language and framework and hosting churn. However, having this project did serve as a useful test bed for experimenting with and immersing myself in new ideas before introducing them into production projects at my day job.

QA as the development bottleneck

In this recent sprint, I found QA to be the bottleneck for feature development. Coding agents have significantly compressed the overall time and effort required for the system design and code writing parts, but to actually verify your feature does what it’s supposed to still means setting up an environment to experience the feature exactly as the user will experience it. That means a feature that only requires booting up the simulator and tapping a button to verify correctness will take 10 or 50 or 100 times less time to validate than installing a build on multiple devices, logging in with different test users, tapping through several screens in a specific order, and verifying the delivery and contents of a push notification. When estimating scope for a feature I want, I now consider the QA burden much more seriously than the one-shot implementation time.

Conclusion

Even after over a decade of intermittent development, Technicolor is still in its early stages. I hope this post gave you some insight into what its been like as a solo dev working on a full-stack Swift project.

If you’re considering or working on something similar, or you want more detail on anything I’ve written about in this post, feel free to reach out on Mastodon or Twitter or email. As of this posting I’m also available for consulting work.

twocentstudios

Print Design Mindset to App Design Mindset

The print design era

The complexity catches up

The mindset shift

Making the shift

Closing the Loop on iOS with Claude Code

Disclaimers before we start

Step 1: Building

Prerequisites

Move DerivedData location to your project folder (optional)

Document project file & scheme

Get simulators

Install xcsift

Building

Clearing DerivedData

Step 2: Installing & Launching

Prerequisites

Document the app binary location for your scheme

Find your bundle identifier

Installing the app on the simulator

Launching the app on the simulator

Navigating by URL (bonus)

Step 3: Reading Console & Log Output

Blocking vs. non-blocking

--terminate-running-process

Launching the app on simulator and reading the output

Blocking console/print direct (when you know output volume is reasonable)

Blocking console/print to file (safer for unknown or expected heavy output)

Non-blocking console/print

Non-blocking Logger/OSLog

Non-blocking console/print & Logger/OSLog

Step 4: Controlling & Viewing the iOS simulator

Prerequisites

AXe

Image Magick (optional)

FFmpeg (optional)

Reading from the simulator

Accessibility info via describe-ui

Screenshots

Video

Manipulating the simulator with taps and swipes

Strategies for increasing tap accuracy

Putting it all together

Step 5: Building, Installing, Launching, Reading Output on a Physical Device

Prerequisites

Get devices

Build for device

Install on device

Launch on device

Blocking console/print capture on device

Non-blocking console/print capture on device

Logger/OSLog capture on device (requires manual sudo)

Final thoughts

How to parameterize names, ids, etc. for these commands

Why not include traditional testing?

Don’t sleep on simctl

What does the future hold?

Going forward

Corrections

Shinkansen Live - Developing the App for iOS

Contents

Overall development strategy

OCR and parsing the ticket image

Why OCR? Why not multi-modal LLMs?

Overall strategy

OCR prototype

OCR for real Shinkansen ticket images

Using spatial data

Transforms

Levenshtein distance

English and Japanese mode OCR

Unparseable fields fallback UX

Defense-in-depth parsing system

OCR testing system

Live Activities

View guidelines and tips

Use non-semantic font sizes for compact and minimal

For dynamically updating times, prepare to spend a lot of time in trial and error

Clamped width custom Layout

`--terminate-running-process`

Accessibility info via `describe-ui`

Don’t sleep on `simctl`

`Group` distributes its modifiers amongst its subviews

What’s changed with `Group`

`onAppear` and `task`