This, as well as playing copious amounts of Need for Speed, Forza, and Gran Turismo, are also the reason I’ve always been drawn to driving sports cars (particularly British, and mid-engine), and of those, those made by vendors that had a significant investment in motorsport. My first roadster was a 2006 Lotus Elise in Solar Yellow. It was so fun to zip around in. It was a super lightweight mid-engine convertible go kart, that also happened to be legal for the road. Built by a company with a legendary presence in Formula 1, it prioritized lightness and handling above all else, especially cost. The steering feel of that car was very direct (no power steering, just raw input from the road), and the interior had basically nothing above the bare essentials (except maybe climate controls, but nothing else, no carpets, not even power windows!).

It was with that, that I experienced some of my first track days racing with other Lotus owners in the Bay Area (many of which were extremely talented and humbled me into knowing just how much I was lacking in the experience department). However, being the Bay Area, it’s riddled with people who don't know how to drive (especially in the rain), and due to the Elise being basically a single fiberglass shell, it was actually really stressful to drive. Wicked fun, but if anyone ever tapped it faster than 2mph, it’d basically be a write off (since replacing the single clamshell cost more than the car itself).

It was for this reason that I eventually shifted to a Lotus Evora 400. The Evora lineup was meant to act as a compliment to the Elise and the Exige. Where the Elise and Exige were primarily focused on being as raw as possible, the Evora was more focused on being a useable road car… that could also put in some wicked lap times. That was actually one of the things that most surprised me when I first drove it, even though the interior was more comfortable, the handling and feel was never lost, in fact it felt just as good as my Elise. A slightly roomier cabin, but still a two seater with an even faster 0-60 (no convertible though, sadly). It was with the Evora that I learned almost everything I know nowadays about spirited driving and track days. Having put tons of miles into that car, driving it every day for ~5 years, I learned to appreciate all it had to offer (and all the extremely strange quirks that went with it). I still got to hang with the Lotus Club, that community really does have some of the nicest people I’ve ever met / learned from; nothing but pure passion and appreciation for motorsport. The amount of confidence you get driving these Lotus cars in corners (due to them being both mid-engine and having a fantastic center of gravity), just makes you want to push it more and more. From steadily improving my times at Thunderhill and Sonoma, to conquering the Corkscrew at Laguna Seca in the rain, I learned a lot about car control, braking, and improving entry / exit speeds.

I'll Have What He's Having

A couple summers ago though, my family ended up renting my dad a few laps in a Ferrari 458 as a gift for his birthday. Growing up he was always fixing cars in the garage (this is also probably a huge reason why I’m so into cars nowadays), so it felt fitting to let him experience something a little faster than the Volkswagens he would drive on the daily. Upon booking the rental for him, I also noticed a few other cars you could rent: a Porsche 911 GT3 RS, a Lamborghini Huracan, and a McLaren 720S.

It was at this point where I had the brilliant idea to also rent myself another mid-engine British race car.

McLaren has always been my number one car manufacturer (with Lotus as a close second), with another legendary presence in motorsport, especially Formula 1. Ever since the McLaren F1 launched at around the time I was born, it was the dream car I would always pick in every single racing game and still to this day is my all-time favourite car. This dream will still sadly probably stay a dream as buying one of the ~106 available globally nowadays will probably set you back at least 20 million dollars (I'm accepting donations if anyone's inclined!).

The McLaren F1 was built with the goal to build the best driving car ever, with no compromises. Turns out, their framework for building it worked so well that they ended up building arguably the best car of all time and held the record for the fastest road car for nearly 13 years straight (reaching a top speed of 240mph). The car was so good, in fact, that they slightly modified it, called it the McLaren F1 GTR, entered 4 of them into the 24 Hours of Le Mans, and then proceeded to win first, third, fourth, and fifth place at their first Le Mans ever. This sadly will probably never happen again as most, if not all, cars in that race nowadays are purpose-built for Le Mans vs. the McLaren F1, a road car being so good it could basically roll up to the start line unchanged. Speaking of wins in motorsport, they're also one of the only manufacturers ever to win the Motorsport Triple Crown (winning Le Mans, the Indianapolis 500, and the Monaco Grand Prix… with Ayrton Senna no less, one of the greatest Formula 1 drivers of all time). The livery of Senna's MP4/6 still lives rent-free in my head and I still cheer them on each year in Formula 1.

Seeing as though I would have an opportunity to drive one of McLaren's other cars and experience what goes into their DNA, I decided to rent the 720S.

Upon stepping into this fighter jet on wheels, I realized that it felt familiar. It gave the same feelings that my Lotus cars gave, a race car that was legal enough to be driven on the road. Off the initial line, I couldn't help but notice that this thing was basically like the NOS scenes you see in Fast in the Furious. You know, where the cars enter hyperspace and get 2x the horsepower when they turn on the NOS tanks, because, damn, it was fast. 0-60mph in like 2.8 seconds (for context, my Evora was almost twice as slow). Not only was it crazy fast, but it was also super stable, as long as you weren't oversteering in the corners and dumping the throttle, never once did I feel like I was losing control. Just like the Elise and Evora, the steering felt super direct and gave tons of information and feeling of the road. A lot of cars nowadays tend to dampen the steering and make it feel way lighter than it really is, but they nailed the steering feel in this car. The car also has this super cool hydraulic suspension system so it's always planted on the road and basically never runs out of grip. If you go over a bump and one wheel on one side of the car is raised, the other wheel on the other side is lowered to compensate and balance the car out. Those feelings sparked so much confidence that only after a couple laps I was flying past other Porsche 911 GT3 RSs as if they were Toyota Priuses. The instructor asked “Have you driven one of these before?”, of course my answer was no, I had only wished I did, but it just felt like a natural progression from the Lotus cars. Raw, direct feedback from steering, mid-engine weight distribution, yet literally 100x faster. It felt like a rocket ship glued to the ground. I remember parking the car and my dad asking "How was it?", I then replied "I need to get one of these".

1 Year Later

For the next year, I could not get the feeling of driving that 720S out of my head. It was one of the most remarkable driving experiences I've ever felt and was probably the closest I'll ever get to experiencing a fraction of what a Formula 1 car is like. I still loved driving my Evora, but this was something else entirely.

Later that year I decided to go visit McLaren of San Francisco "just to see" what they had.

A test drive and a week later, I then said goodbye to my Evora 400 and hello to a McLaren 720S Spider in McLaren Orange (one of the best colours IMO).

Now for context, in case it wasn't obvious, the Evora was my daily driver and as such this took its place. I've never really liked having multiple cars, and I find that the best way to learn everything about a car is to drive it everywhere (or at least wherever possible). This includes everything from getting groceries (the front trunk can actually hold a lot), to sitting in 280 traffic, spirited driving in the Santa Cruz mountains, and even ripping past turn 1 at Sonoma Raceway.

This car is absolutely incredible, and is something I'll be holding onto for a very long time. I'm excited to drive it every time I get to flip the door up vertically, step into the cockpit, press the big red Engine Start button, and hear the twin turbo 3.8L V8 rev up. The interior is sleek and simple, with a buttonless leather and carbon fiber steering wheel (no media controls, no HVAC, just a steering wheel). It's rare to find this nowadays, especially with so many other companies making their steering wheels have more buttons than an Xbox Controller, having terrible touch controls, or being stupid and choosing a yoke instead of a wheel entirely. It's also super comfortable, and has a massive front windshield with lots of visibility. Even the rear quarter panels are transparent so your blind spots are never blocked. All the other buttons are readily accessible to the driver on either sides of the cockpit all the while feeling super low to the ground (107mm off the ground to be specific). You can tell this was built and designed by people who love driving.

The dual clutch gearbox is so smooth and the way the exhaust sounds as it's cut when it upshifts around 5000 RPM sounds like it's straight out of a science fiction movie. At Laguna Seca I remember I had to carry so much speed through turns 5 and 6 with my Lotus cars (since they can lose a lot of speed trying to go uphill), but this car has so much torque it simply does not care. You could be at a standstill at the bottom of the hill, floor it, and in 5 seconds you'd be going 120mph. For all the extra power it has, it's still just as lightweight as my Evora was. I even got the opportunity to learn how to get it sliding sideways a bit through corners at the track to learn how their ESC (stability control) Dynamic setting works.

I have so much more to say about what it's like to drive and how much appreciation I have for all the engineers and designers that worked so hard to make this car exist, but today I want to focus on telling the story of my experience fixing one of it's smaller flaws…

It doesn't have CarPlay.

Editor's Note: huge first world problem, I'm fully aware.

So What Does It Have?

The 720S was introduced at that awkward phase in car stereo technology times where Bluetooth was everywhere and CarPlay was just starting to exist. However, they sadly didn't seem to get around to implementing it, and it was only equipped with USB iPod support. So, I could use my iPhone for music, but that would require either using Bluetooth or USB.

There's two problems with this however:

1. Bluetooth's bitrate maxes out at ~264kbps assuming we're doing AAC (which we are given it's iOS and the car doesn't seem to support anything like aptX or whatever). This is worse than a 320kbps MP3 file, and I could hear the compression artifacts, so this was immediately out of the question.
2. USB does provide the best quality, however that requires constantly plugging in the phone anytime I get in the car and I was already used to having wireless CarPlay in my Evora (which again, another huge first world problem, I'm aware).

Now you might think: "Couldn't you just swap the stereo head unit like other cars?". I could, however for the 720S, McLaren built a fully custom bespoke one-off system that I can't quite swap. The system itself controls loads of proprietary things like tire settings, interior lighting, maps, sensors, track telemetry, even Variable Drift Control (you literally get a slider to adjust how much you want to drift!), and as such, nobody has really spent the time or effort to make an aftermarket swap.

Editor's Note: It turns out the newly released 750S now does have CarPlay, but that also didn't feel like a cost-effective solution.

The whole setup is actually pretty sweet, the gauge cluster can flip down to show you just your shift lights when you're in track mode and you get an audible cue when's the optimal time to shift gears.

So anyways, this custom head unit does a ton of stuff, surely there's something we could do with it, right?

Jailbreaking the Car?

Upon further inspection, the infotainment system seems to have been developed by JVC / Kenwood (which lines up properly with this press coverage denoting JVC did a custom system for McLaren). Upon using it though, I figured that it was running a variant of Android due to the distinct way it handles scrolling at the edges of any given screen (you get that glow effect instead of rubber banding like you would on iOS or Windows Phone 🪦). Now considering that this car was unveiled in 2017 and the electrical systems were developed around then (or earlier, technology moves slow in the car space), it's probably not running Lollipop or Marshmallow (since those were bleeding-edge at the time and this is likely an embedded system). I'd bet probably something like Android 3.0, surely we can just jailbreak it right?

Turns out people have already done this with the McLaren 650S and below, apparently adb is running if you connect your car to the iPod USB port. They sadly patched this for the 720S. In addition, after trying some other things, I realized it wasn't actually Android, but a different embedded Linux system entirely (long story… probably for a future blog post maybe)… so I guess I'll just take the dash apart and try to jailbreak it myself?

Yeah… no. Given how fun of a project that would be to try, I'd really rather not accidentally brick my car. I don't think the dealership would be cool with me bringing it in for service to somehow reflash the infotainment system with a proper image because I managed to get it to not turn on anymore. Loads of Android devices are trivial to hack since they all have fastboot and you can just restore it if you mess something up, but I'm gonna wager that nobody has dumped the software off this system nor has McLaren hosted bootloader images on GitHub. It also seems extremely cumbersome to try and take all of the dash apart, so… yeah, no.

Feeling rather bummed that his avenue was not gonna happen, I started thinking about what I was trying to optimize for: enable wireless playback of my music in my car, with media controls, and at the highest bitrate I can (or at least better than Bluetooth). It was at this point that I remembered that wireless CarPlay operates by first setting up a WiFi network over Bluetooth and then streams CarPlay via AirPlay over WiFi to the car. So maybe I could make my own "CarPlay"? What if somehow I added AirPlay to my car? I saw projects where people ripped up old AirPort Expresses, but I needed to run off a cigarette lighter and not have something nuke the small racecar-optimized battery, so I settled on a Raspberry Pi. They only require 5v at like 5 or 6 watts or so, so the energy usage would be mostly the same as having my iPhone plugged in.

Upon setting up a lite version of Raspbian, I started looking for projects that implemented an open source AirPlay receiver server and I came across shairport-sync. Turns out Mike Brady did all the hard work and already thought of this use case (thank you Mike!). Once setup, you'll have a Raspberry Pi that broadcasts a WiFi network and an AirPlay server that'll output whatever it receives over 3.5mm / AUX. My car did in fact have a 3.5mm input jack, however initial testing resulted in a ton of electrical noise (static, clicking, etc.) coming out the speakers. Turns out the way my car was wired up really didn't like the electrical interference that was created when you hooked a Raspberry Pi up to the cigarette lighter or iPod USB port and then played that same device over 3.5mm back into the car. After spending a couple days trying to debug and troubleshoot with some ground-loop inhibitors and various cables, I still couldn't get a clean signal (maybe it was still interference, maybe just the Pi's DAC, no idea), so I started over.

iPod USB

It turns out the Raspberry Pi's USB-C port supports USB OTG, meaning the Raspberry Pi could dynamically switch that USB port from being a host port (e.g. hosting a keyboard) to a device port (e.g. act as a device that the car could host).

I then started looking through tons of stuff to see how I could somehow "emulate" an iPod via my Raspberry Pi. I looked through RockBox source code, thought about virtual machines, anything really, but I eventually stumbled across ipod-gadget.

This project was perfect:

ipod-gadget simulates an iPod USB device to stream digital audio to iPod compatible devices/docks. It speaks iAP (iPod Accessory Protocol) and starts an audio streaming session.

So assuming this works, I could plug the Raspberry Pi into the car and have it think it's an iPod. I had doubts it would work without issue as it had tons of patches for specific stereo head units, but I set it up just to see what would happen. I configured Shairport Sync to use hw:iPodUSB as this was the ALSA audio device that it exposes, went to the car, plugged it in and…

UNSUPPORTED DEVICE

:(

So either the car didn't recognize the "iPod" or the "iPod" didn't recognize the car. The ipod-gadget project involves two parts:

1. A Linux character device (/dev/iap0`) you send stuff to (to send over USB)
2. A client app (ipod) written in Go that talks to said device.

Upon trying to inspect the logs output by the ipod server binary, it didn't produce anything meaningful. I then did a sanity check with my home stereo to see if it would do anything, and it worked perfectly.

I brought it back to the car and tried again just to see, and no matter what I tried, it would constantly fail to connect. I then re-checked the logs with the ipod server running on the Pi with more verbosity and then I noticed there were a bunch of packets that would fail to parse and would be dropped. So I patched that bit of code to just spit out raw bytes to see if I could get anything.

A few things seemed to stream out and then it would just stop (here, >>> means incoming, as in car -> Raspberry Pi).

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

I took what I saw, and dumped it into the GitHub search box.

🤷‍♂️

I wish it would've been that easy. Trying a few permutations finally lead me somewhere...

This search result was just pure luck, but it was super helpful. Notice how the first result is Rockbox, which is an open source media player implementation that runs on old iPods. As such it also implements the same protocol as the ipod-gadget (iAP). However, it isn't referencing that byte sequence we're looking for, the project below it wiomoc/iap2 is.

Upon reading wiomoc's extremely helpful blog post I now knew what I was dealing with.

My car's stereo didn't implement iAP1… but iAP2 and has "iPod" listed in the car's infotainment as in "iPod Accessory Protocol"… not necessarily supporting "iPod devices". So having read through most of Christoph's (wiomoc) post (thank you for writing this!), I got the idea to just send back the same sequence of bytes the car was sending me. Apparently this is a way for a device to acknowledge that it talks iAP2.

I modified the go ipod project and upon doing this, I had thousands of bytes streaming on the screen to parse through (here, <<< means outgoing, as in Raspberry Pi -> car):

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

<<< 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x5A 0x00 0x1D 0x80 0x74 0x00 0x00 
    0x96 0x01 0x05 0x10 0x00 0x0F 0xA0 0x00 
    0x49 0x1E 0x05 0x0A 0x00 0x01 0x0C 0x01 
    0x01 0x0B 0x02 0x01 0xA8 0x00 0x00 0x00 ...

Neat.

Now what?

Upon reading more of the blog post and iap2 project, I realized I was probably going to have to do a lot of stuff on my own. Apple's iAP2 protocol documentation is only available to members of Apple's MFI program, and to qualify for the program, you basically have to be an accessory maker (e.g. Belkin) that makes or ships thousands of MFI devices. As such, I basically only had open source projects to go off, and thankfully the iap2 project was enough to get me started.

Let's Emulate an iPhone in Swift

The idea, in theory, was what if I could just take my Raspberry Pi and convert it into an "iPhone", but only do enough of the parts that iAP2 cares about to play audio.

To start, I used the ipod-gadget project again in order to reuse that character device /dev/iap0 over which I could send and receive bytes. This project already has the correct USB HID descriptors and whatnot setup to mask as an iPhone, so it saved a lot of time doing the basics.

Next I started a Swift command line utility project that I could compile and run on Linux (since I was running this on a Raspberry Pi).

Sidenote: Swift on Linux is pretty great nowadays. You can just build and run on Linux. You can't use Xcode to debug on Linux, but it's pretty trivial to debug remotely with Visual Studio and the various extensions.

After that, I setup the basis for reading and writing bytes to/from the character device using a DispatchSource both for input and output:

self.io = try! FileDescriptor.open("/dev/iap0", .readWrite, permissions: .ownerReadWriteExecute)
self.buffer = UnsafeMutableRawBufferPointer.allocate(byteCount: 1024, alignment: MemoryLayout<UInt8>.alignment)

self.queue = DispatchQueue(label: "ca.adambell.HIDReader", qos: .userInitiated, autoreleaseFrequency: .workItem)
self.source = DispatchSource.makeReadSource(fileDescriptor: io.rawValue, queue: queue)
source!.setEventHandler { [weak self] in
    guard let self else { return }
    do {
        let bytesRead = try io.read(into: buffer)
        
        ipod.parseBytes(bytesRead)
    } catch {
				// handle errors
    }
}

For output, all I had to do was encode stuff as little-endian bytes and write them to the character device:

self.queue = DispatchQueue(label: "ca.adambell.HIDWriter", qos: .userInitiated, autoreleaseFrequency: .workItem)
self.source = DispatchSource.makeWriteSource(fileDescriptor: io.rawValue, queue: queue)
source.setEventHandler { [weak self] in
    guard let self else { return }

    while let report = reportQueue.popFirst() {
        _write(report)
    }

    if reportQueue.isEmpty {
        source.suspend()
        self.writing = false
    }
}

let bytesToSend = [0xff, 0x55, 0x02, 0x00, 0xee, 0x10, 0x00, 0x00]

queue.async { [weak self] in
    guard let self else { return }

    reportQueue.append(bytesToSend)
		source.resume()
}

Originally I tried using Swift 6 actors / async / await, however I had to switch back to plain old Dispatch as it turns out that Task and DispatchSource really don't play nicely together yet (see: FB14132060) :(

So once I got the project up and running I began map out what exactly I wanted to get working:

1. Music Playback (obviously)
2. Track Metadata (hopefully cover art?)
3. Media Controls (if I can make it work)

The infotainment system in the car supports a bunch of stuff like browsing your Music library and accessing playlists, but I didn't care about much of that since I was just going to be using AirPlay. I could control all that from my iPhone mounted to the dash.

Continuing on, I got back to the same spot I was at with reading bytes with the original ipod client (however, at this point it was all in Swift!):

>>> 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

<<< 0xFF 0x55 0x02 0x00 0xEE 0x10 0x00 0x00

>>> 0xFF 0x5A 0x00 0x1D 0x80 0x74 0x00 0x00 
    0x96 0x01 0x05 0x10 0x00 0x0F 0xA0 0x00 
    0x49 0x1E 0x05 0x0A 0x00 0x01 0x0C 0x01 
    0x01 0x0B 0x02 0x01 0xA8 0x00 0x00 0x00 ...

Referencing wiomoc's blog post once more, I noticed a small detail:

Therefore it first sends a message to the accessory requesting the certificate of the Authentication Coprocessor. The accessory retrieves this certificate from the Coprocessor and send it in a response message. The iPhone checks if this certificate is signed by Apple. Upon successful validation, the iPhone generates a challenge and transmits it to the accessory. [...]

After this authentication procedure is completed, the accessory sends some identification information, like name, serial number, supported transports or offered EA protocols to the iPhone. After an acknowledgement from the iPhone, the iAP2 Connection is fully established.

Crap.

I thought the whole project was probably screwed at this point since I had no way of getting ahold of said iPhone Authentication Coprocessor. Without said coprocessor, I'd never be able to authenticate a session. It's probably baked into the iPhone's system on a chip and I had absolutely no means of taking it out or repurposing it.

However, then I realized, I'm not communicating with an iPhone… I'm spoofing one and talking to the infotainment system.

In order to get an accessory to do anything, you need to have the accessory "identify" itself. Once that starts, it'll then authenticate itself with an iPhone and will use the onboard authentication coprocessor to validate that the MFI accessory is "legit".

In my case, since I'm the super totally legit iPhone™ that's "authenticating" the infotainment system, I can just send a thumbs up to whatever it says.

Infotainment System: "Hey I'm a licensed MFI car stereo: here's a signed challenge showing I'm legit"

Me (a totally legit iPhone):

func checkChallenge() { 
    return true
}

"… Sure! You're good 👍"

I then loosely used wiomoc's code as a point of reference and built out the initial session setup and communication channels necessary to get me to the identification phase. Once I implemented this, gone were the "Unsupported Device" errors that my car was showing me. I still had tons of bytes being thrown at me that I had no idea what they were, but it was connected.

For fun I tried AirPlay-ing what I was listening to at the current time (I remember it vividly, it was "A Walk in the Woods" by Marty O'Donnell from the Halo: Combat Evolved Soundtrack)… and it worked!! It turns out all my car cared about was an initial iAP2 session and then it'd just play whatever was coming over USB by any UAC (USB Audio Class) device!

Editor's Note: QuickTune is fantastic. Great work Mario!

I'd AirPlay my iPhone to the Raspberry Pi, it'd create an iAP2 session and then just dump whatever was coming over AirPlay to the car's infotainment system.

This crazy wild combination of devices ACTUALLY WORKED:

iPhone → [shairport-sync] → [ipod (swift tool)] → McLaren Infotainment System

So here's where we were at:

1. Music Playback (obviously)
2. Track Metadata (hopefully cover art?)
3. Media Controls (if I can make it work)

Now, since I do a lot of track days, I also have McLaren P1 race bucket seats installed in my car. They're fantastic for hugging your body as you take corners super quickly, but absolutely horrible for posture / hacking on things in the car. My neck and back started to hurt after a few of nights of hacking on things in the driver's seat.

In addition, it turns out that my code was originally so bad, I would sometimes send invalid bytes to the car… and the car's infotainment system would just lockup or crash / restart. So I'd have to shut the car off, restart it (in accessory mode, no need to power on the engine), and keep it hooked up to a trickle charger to ensure the battery didn't die as the alternator isn't running without the engine on.

For the sake of my spine, I really needed to figure out an alternative to continue working on this project.

Knowing how software works in most of the tech industry, if a company has functional software, chances are they won't rewrite it for other things (if it works, why rewrite it?). Heck, apparently some companies still rely on WebObjects running on a Mac Mini somewhere under someone's desk. Chances are, if JVC / Kenwood built an infotainment system for McLaren, they probably borrowed a lot of code they already had for other stereos (their implementation of iAP2). In this case, I assumed I could just buy another Kenwood stereo from Best Buy and hack on that instead. If I got things working with that, it should work with my car with little modification. As such, I bought a super cheap single DIN stereo and stuck it on my desk. I had to hard wire some power lines and hook up some speaker cables (it was extremely sketch), but eventually I had a "similar" car stereo on my desk I could mess with.

I plugged in the current WIP I had with the Raspberry Pi and worked out of the box! Shared code!

Sadly, progress on this project swiftly¹ halted as wiomoc's code didn't really cover any sort of media / metadata iAP2 stuff.

That's where I had to get a bit more crafty…

Hanging Out With HID

When reverse engineering APIs for the web, you typically setup some sort of man-in-the-middle attack and read the incoming or outgoing requests. A lot of people default to things like mitmproxy, Proxyman, Wireshark, or Charles Proxy. Now, normally if this were over the web, that's what I'd be pulling out. However, this is all over USB, and I didn't know the format I was working with, so what I sorta needed was a USB Proxy for HID (which is what all the messages are sent over).

Turns out, there's a project that does just that: usb-proxy. It worked great initially, but often times would crash due to the volume of traffic going over the cable.

Eventually I found an alternative: a pretty sweet project called Cynthion that does what usb-proxy does, but in hardware!!

This thing is wicked cool. You put it in the middle of a USB Device (iPhone) and a USB Host (Stereo) and it basically just spits out all the bytes that goes over the USB cable so you can analyze it. It comes with a tool called Packetry that allows you to analyze each HID frame that comes in:

Notice anything familiar? ^_^

From making the other iAP2 messages work (e.g. identification), I knew the general gist for what things looked like (there's a header, some size bytes, a checksum, blah blah, and a payload), so I just needed an easy way to find the payloads I wanted to replicate. Most of the iAP2 messages seemed to start with 0xFF 0x5A so that made it a lot easier to narrow things down.

Media controls were actually trivial to figure out, so I figured those out first. All I did was run my ipod app and have it spit out any incoming frames / messages it didn't understand. In this case, I would just watch the console as I tapped various media buttons on the stereo (Play, Pause, Next, Previous, etc.). I'd spam tap a button (e.g. Play / Pause) and look for similar packets. I'd then diff them to try to build up an enum:

enum MediaControl: UInt8 {
    case playPause = 0x02
    case next = 0x04
    case previous = 0x08
    ...
}

No idea if this is actually the spec, but in my case, I never actually needed to send these to the car, so I didn't care about figuring out the format. I just needed a means to watch for these packets from the car -> Raspberry Pi, get the media button pressed, and then I'd control Shairport Sync myself using the dbus client it exposes:

// Shell is implemented elsewhere, but it just uses Process / Pipe from Foundation
if control == .playPause {
  try Shell.command("dbus-send --system --type=method_call --dest=org.gnome.ShairportSync '/org/gnome/ShairportSync' org.gnome.ShairportSync.RemoteControl.Play")
}

1. Music Playback (obviously)
2. Track Metadata (hopefully cover art?)
3. Media Controls (if I can make it work)

Metadata was a lot more involved. What I was really looking for was a way to display Title, Artist, and Album (and album artwork if possible!) onto the infotainment screen. While this wasn't strictly necessary, I do like being able to see chapter titles from podcasts or songs being streamed on Apple Music. As such, I needed to find the correct offsets / structures to send as USB packets.

I generated a test track and uploaded it to my iPhone with the following:

• Title: TITLE_HERE
• Artist: ARTIS_HERE
• Album: ALBUM_HERE
• Album Art: A simple 512x512 opaque red png.

Note: ARTIS_HERE is an intentional typo as I wanted a consistent length for the strings sent (10 bytes, aka 0xa) to make it easier to find their patterns.

I played this test track on my iPhone when hooked up to the stereo (through Cynthion) and then I found the following bytes in the pcap file:

Notice the highlighted area. Before that you have 0xe as the length, but the length of the strings I was sending was 0xa. It turns out that the length also includes the bytes that specify the size (2 bytes) as well as 2 other bytes that indicate the type of information supplied (0x00, 0x01, aka Title), so removing those 4 bytes, we get 10 bytes: 0xe - 0x04 == 10

Initially I wasn't getting anywhere, but it was far easier to iterate on since I could just keep trying new formatted packets to see if anything would appear on the stereo screen. Still having no luck, I tried replaying the same packets dumped via Packetry but still, nothing would appear on the stereo's display. Eventually I realized that there was consistently another packet sent before the track info (more on that later), I didn't really care what it was, but once I sent it before the metadata packet, it then started to display my strings! Sometimes it would crash due to messing up the sizes sent or other orderings, but eventually I got that working just fine.

Editor's Note: Not shown here were the multiple days of trial and error and sheer frustration that comes out of throwing random stuff at a black box until something happens. There were many times where I nearly gave up because of brittle issues that cascaded and broke other things that weren’t even related. Sometimes things would work, sometimes trying the same thing again would fail and crash the unit.

The only thing that was remaining was figuring out how to send cover art. This actually ended up being the hardest part since I had to figure out how to do the File Transfer Session stuff. It turns out you need to specify an identifier for the image to be sent alongside the track metadata (in that "other" packet mentioned above), and then send the image separately in a different session. I got pretty close a few times trying to figure it out on my own, but then I stumbled upon some "questionable" GitHub repos that gave me a lot more hints and then I eventually got artwork displaying. In short: you setup an identifier, send it along with the track info, send the image bytes in packets, and then tell it you're done on the last packet.

1. Music Playback (obviously)
2. Track Metadata (hopefully cover art?)
3. Media Controls (if I can make it work)

Shairport Sync → Stereo

The next part was finding a way to get the currently played track's metadata to show on the stereo. I needed to get it out of Shairport Sync and pass it to the stereo (i.e. getting metadata out of the AirPlay session). For that, you can build Shairport Sync with metadata and cover art enabled. Once built, it’ll publish metadata info at /tmp/shairport-sync-metadata. So then I wrote a Swift parser to pull out the various metadata items I was looking for and would pass them to the stereo anytime they changed.

Putting it altogether, and skipping ahead in time, I eventually had everything that I wanted!

When I plugged the Raspberry Pi into the car, it'd trick it into thinking it was an iPhone, the car would accept USB audio, and I'd just connect my iPhone over AirPlay to the Raspberry Pi to play my music. AirPlay 1 specifically runs as lossless 44100Hz, meaning I had bit-perfect CD quality audio running wirelessly in my 720S! What's even more wild is stuff like phone calls and Siri just worked perfectly out of the box. The McLaren allows me to disable the media / audio protocols for connected Bluetooth devices and only allows the hands free protocol for sound input (i.e. microphone). I could then simultaneously connect it to AirPlay for sound output! :D

It worked perfectly and even things like chapter titles / artwork from Overcast worked!

Adding a Bit More "CarPlay"

The nice thing about wireless CarPlay is it's very hands-free. You get in the car, drive, and it's already connected. So to complete my "CarPlay", I needed a means to automate its connection.

Shortcuts to the rescue!

Sadly, iOS doesn't have a shortcut to have your iOS device connect to a specific WiFi network (because reasons)… so to have it automatically connect I configured the WiFi network my Raspberry Pi broadcasts to be the highest priority for the iPhone to connect to it. In doing so it would take precedence over my home WiFi and you can set this up via macOS and then it syncs over iCloud back to the iPhone. I then made a Shortcuts automation to automatically run when connected to the car's Bluetooth:

1. Disable WiFi
2. Wait a few seconds (hardcoded to roughly the boot time of the Raspberry Pi)
3. Re-enable WiFi (connecting to the in-car WiFi since it has a higher priority)

I then ran a second Shortcut automation when I connected to the in-car WiFi:

1. Enable the "McLaren" focus mode on my iPhone, which populates my home screen with Albums widgets for my favourites²
2. Wait a few seconds to ensure the AirPlay network is fully ready
3. Set the sound output to my in-car AirPlay network

I'd then do the reverse (and pause media if playing when it ever disconnected from my in-car WiFi) to make it as seamless as possible and to ensure that music wouldn't keep playing after I parked the car.

Oops, I Broke It

One thing that didn't originally occur to me was that anytime I shut off the car… I was just cutting power to the Raspberry Pi without shutting it down safely. At one point the Raspberry Pi stopped booting entirely so I had to re-image it and set it up again.

This time I optimized the boot time to be as fast as possible (getting it down to roughly ~7s). I looked through the startup path using systemd-analyze blame and just disabled anything that wasn't related to audio or networking. I then enabled a read-only filesystem so the filesystem would just be running as a tmpfs in RAM so no more changes could be made to the Raspberry Pi and would prevent further issues.

I also had another issue where some songs would have odd distortion with certain frequencies. This took so much trial and error, but eventually I discovered that if I forced Shairport Sync's format to be S16_LE (signed 16bit little endian), the problem went away.

Lastly, I then cloned the SD Card to disk and saved it in case I ever broke something again (so I could just re-image it easily).

More Broken Stuff!!

Now of course, all this worked perfectly for a few months, and then iOS 17.4 came out and broke my media controls. -_-

It turns out that Apple removed needed parameters to control the AirPlay session (notable DACP-ID and Active-Remote), and without those there wasn't a way for me to do things like play, pause, or skip tracks.

sigh.

Well since my project was already a pile of hacks that magically worked together, I figured I could just hack it up even more. Bluetooth devices can support AVRCP (Audio/Video Remote Control Profile)… so I just made my Raspberry Pi into a fake AVRCP device and had it automatically connect to my iPhone. I'd pair my iPhone via Bluetooth additionally to the Raspberry Pi, and then the Raspberry Pi could then Play / Pause / Skip tracks on my iPhone using AVRCP (acting as a Media Player Remote Control, think Media Keys on a keyboard):

dbus-send --system --print-reply --dest=org.bluez /org/bluez/hci0/dev_00_11_22_33_44_55 org.bluez.MediaControl1.Play

Instead of sending dbus commands to Shairport Sync, I'd send them to a bluez client. I had to do some clowntown Bluetooth configuration in Linux that I don't really remember, but eventually I got things working. It definitely did not include putting a while loop in /etc/rc.local to spam connect to the iPhone over Bluetooth on boot. It was hideous, but it worked.

I then made another backup image of the disk so I never had to debug Bluetooth stuff ever again.

End Notes

Putting all this together has left me with probably one of the most hacked up piles of software that somehow works together advanced audio stacks in any McLaren 720S. I now have a car that'll automatically connect to my phone and play music in its highest available quality (as in available to native implementations on iOS… here's hoping one day AirPlay will support higher quality like 48000Hz). It's not exactly "CarPlay", but it's close enough. I need to give a huge shoutout to Mike Brady, Andrew Onyshchuk, and Christoph Walcher for all the work they did on their respective projects. I wouldn't have been able to do this without them.

This has been an incredible learning opportunity for me, exploring so many things up and down the stack: USB, Linux kernel drivers, HID transports, Bluetooth, reverse-engineering embedded systems and protocols, low level audio formats / sequences, the list goes on and on. This also allowed me to appreciate all the complexities that go into building reliable car infotainment systems. In the end this truly was a fun yet ridiculously over-engineered project to solve a simple problem. That being said, I've now got a pretty sweet computer running in my car all the time, so I've got tons of other ideas for other projects to do in the future (track day logging??). Not only that, but I have Swift powering part of my car's audio stack!

Sometimes the best solutions are those held together with digital duct tape.

¹: 🥁

²: This massive project was actually a catalyst for building Albums. Initially, I wanted a consistent way to get to my favourite music without needing to navigate small tap targets on various streaming apps, but then it ended up becoming a full-fledged app that I'm super proud of and love using all the time! :D

I posted about contemplating on upgrading my blog to use something like Publish so I could spend less time maintaining my blog and more time writing for it. In short, my old blog was a server I wrote myself which would take markdown files (blog posts) and dynamically convert them to HTML and serve them, this was kinda ineffecient, but it was a great learning experience. The previous blog was built atop Vapor, but with every Swift release / update to Vapor, I'd find myself rebuilding it on the server, which took forever on the tiny little server hosting this website. I had looked at using docker to containerize the whole thing, but:

1. I had no idea what I was doing.
2. Cross compiling docker for x86_64 (which is what my server is) on an arm64 laptop (my M1 Max MacBook Pro) was actually slower than building on the server directly, so I ditched that idea.
3. I still don't even know what a kubernetes is (does anybody really?).

I saw loads of people talking very highly of Publish, so I figured I'd give it a shot. Publish does much of the same, it takes markdown files (blog posts) and prints them out to statically served HTML files, which should be a lot less overhead too (than the less efficient route I had before).

Turns out it was pretty trivial:

1. I took the markdown files I had from my previous blog, and copied them over to Publish.
2. I updated some of the markdown files' links to adapt to the new static site directory that Publish uses (i.e. instead of blog.adambell.ca/date-post-title with Publish it's blog.adambell.ca/posts/date-post-title).
3. … yeah that's pretty much it.

Building and running the site is really simple. Just build and run from within Xcode (running is what prints out the website to the Output directory), and then use publish run from within the project directory to startup a server to view it locally at http://localhost:8000.

Once done, it looked a whole lot like SwiftBySundell. 😂

This was still more than enough to get myself started however. Publish uses themes to configure how it looks, and the built-in one is just very basic. I ended up porting all the old css from the old blog to Publish's layout and that worked just fine.

The next part was getting syntax highlighting working. I was originally using pygments.js, but seeing as how much Publish focuses on being a lightweight static site generation platform, I figured I'd try using whatever it had built-in. Publish features Splash, a lightweight syntax highlighter, however when I used that, it would crash due to some recursion issue with how much whitespace I was using, so I'll file an issue for that soon. To get around this, I ended up opting for using highlight.js, but with a Publish plugin so it'll statically run highlight.js at build time and applies styles to code blocks. This means the site will look the exact same with or without javascript support enabled.

Lastly was adapting some of my other pages to work correctly, notably the Talks page and the Archive page.

I have my talks modeled from my previous blog as a struct like this:

public struct Talk: Codable {
    let date: Date

    let title: String
    let venue: String
    let URL: String

    let postDateString: String
    let localizedDateString: String

    init(title: String, venue: String, dateString: String, URL: String) {
        self.title = title
        self.venue = venue
        self.postDateString = dateString
        self.date = postDateFormatter.date(from: dateString)!
        self.localizedDateString = localizedDateFormatter.string(from: date)
        self.URL = URL
    }
}

I couldn't quite figure out how to adapt these entirely to Publish's Item protocol since I'm still learning how this all works, so I ended up just sorta doing my own thing for rendering that section of the site. I then added talks as a section to my site, and created an empty talks folder so that Publish would generate things for that section:

public struct Blog: Website {
    public enum SectionID: String, WebsiteSectionID {
        case posts
        case talks
        case archive
    }

/* ... */

So I created an array of all of my talk recordings:

public let AllTalks: [Talk] = [
    Talk(title: "Crafting Responsive and Playful Interfaces", 
         venue: "NSSpain", 
         dateString: "20220916", 
         URL: "https://vimeo.com/751571352/3330bd3af4"),
    /* ... */
]

And within my implementation of the HTMLFactory protocol for the custom theme, I setup the section like so:

func makeSectionHTML(for section: Section<Site>, context: PublishingContext<Site>) throws -> HTML {
    switch section.id {
        case .talks:
            return HTML(
                .lang(context.site.language),
                .head(for: section, on: context.site),
                .body {
                    SiteHeader(context: context, selectedSelectionID: section.id)
                    Wrapper {
                        H1(section.title)
                        List(AllTalks) { talk in
                            Div {
                                Link("\(talk.title) - \(talk.venue)", url: talk.URL)
                                Text(talk.localizedDateString)
                            }
                            .class("talk-list-item")
                        }
                        .id("talk-list")
                    }
                    SiteFooter()
                }
            )
        /* ... */
    }
}

You'll notice here that I'm iterating over all the talks and creating a List (which effectively maps to a <ul> in HTML, with each item inside being a <div> wrapped within a <li>). It's pretty easy to assign classes or ids to things as needed so you can easily style with CSS. For sections where I didn't want rendered via this method, I just returned an empty HTML(). One example is the home page which is created via makeIndexHTML(for:context:).

Putting it altogether, it worked great!

Note: Publish has its own item management and protocols which make a lot of this more straightforward / less manual, but I didn't quite figure all that out yet, feedback is welcome!

The Archive page way was done in mostly the same way.

The SwiftUI-like syntax is powered by Plot and is really nice. I had some small hiccups trying to understand how to the syntax worked at first, but the raw HTML escape hatches were great for prototyping how things worked when I was stuck. Compared to using Vapor's leaf, I really enjoyed having the type-safety that Plot enforces, it was a lot easier dealing with getting the Swift compiler to be happy vs. debugging why the templating language wouldn't quite do what I wanted.

I also realized that the RSS configuration for my old site was kinda broken, and Publish sorta just fixed all this magically! It generates an RSS feed by default for all the posts you have, which is also another thing I don't need to keep track of.

Deploying it was also really simple. Syncing the Output directory to an nginx server and adding some rewrite rules to map the old blog URLs to the newer format was added to preserve other articles linking back here and avoid link rot.

All in all, it was maybe a few hours of work to port the whole thing over. I even left some room for some more fun to overhaul the navigation and add some springs for fun! :)

This should be the last post here about writing blogs in Swift (hopefully). I have some more stuff in store, especially about animations!

Hopefully now that there's less maintenance needed for this blog, I can write a whole lot more! The lightweight nature of Publish and its simplicity has already made writing this post way more fun! It's also far faster to render (with javascript no longer necessary for syntax highlighting) and acts as a normal statically hosted website (no dynamic swift server code).

Thanks John for making Publish! :)

Introduction

This post follows where my server-side Swift Kitura adventures left off.

Some sad news came about when it was announced that Kitura was winding down, and I realized that, since I built my blog using Kitura, I should probably find an alternative with a more stable future.

I peeked around and Vapor caught my eye.

Vapor

Quoting their GitHub:

"Vapor is a web framework for Swift. It provides a beautifully expressive and easy to use foundation for your next website, API, or cloud project."

It's one of the most popular server-side Swift frameworks nowadays, powered by Swift NIO, and powers some pretty cool apps (like Scribble: an app that let's you share whiteboard space remotely!).

It even has some pretty nice integrations with things like SQLite and has some pretty nice ORM stuff with their Fluent library. For my blog it's pretty basic, and doesn't really use any of this though.

Fixing Routes

The first thing I had to do was setup a new vapor app using their guide.

Note: As of writing this, Vapor 4 is in the RC stages so it may very well be available by the time you're reading this.

The rest of it was relatively painless and just required me to restructure some of my routing.

What once looked like this in Kitura:

app.router.get("/") { request, response, next in
  guard let recentPosts = getAllPosts(atPath: blogFileServer.absoluteRootPath, limit: 6) else { return }

  do {
    try response.render("blog-home.stencil", context: [ "posts": recentPosts.map { $0.dictionaryRepresentation } ])
    response.status(.OK)
  } catch let error {
    response.send(json: ["Error": error.localizedDescription])
  }

  next()
}

Became this in Vapor:

router.get("/") { request -> Future<View> in
  guard let recentPosts = getAllPosts(atPath: rootBlogPath, limit: 6) else { throw Abort(.notFound, reason: "") }

  return try request.view().render("blog-home", ["posts": recentPosts.map { $0.dictionaryRepresentation }])
}

What's nice is the API changes slimmed things down to a significantly more simple way of defining the root page for my blog.

You'll notice that requests return Futures of Views, which will resolve at some point in the future (reminds me a lot of Hack!).

Also, this gets rid of the next() invocations littered about when using Kitura, which I never personally really understood that well.

Stencil to Leaf

The next thing I had to port over were my stencil definitions. Kitura uses Leaf, which is another way of defining HTML templates that Vapor can populate.

They were basically the same, except for a few syntactic changes, namely the use of #(something) instead of {{ something }}.

You can check out the documentation for Leaf here.

Kitura:

<!DOCTYPE html>
<html>
  <body>
    <ul>
      {% for post in posts %}
      <li>
        {{ post.title }}
      </li>
      {% endfor %}
    </ul>
  </body>
</html>

Vapor:

<!DOCTYPE html>
<html>
  <body>
    <ul>
      #for(post in posts) {
      <li>
        #(post.title)
      </li>
      }
    </ul>
  </body>
</html>

Conclusion

For my case, switching over was really simple and felt like a relatively painless API migration. What's kinda neat too is, at least in my case, Vapor is noticeably more stable when running my blog. It hasn't gone down once since porting over (which would happen every month or so with Kitura).

In addition, compile times seem to have been halved, although I'm not sure if this is just tangential to Swift improvements or not.

Vapor seems great to me! I'm sad to see Kitura go, but hopefully this one stays around!

Give it a go if you're interested!

A couple days ago I noticed a really weird bug, images that had been used as templates would occasionally draw darker or lighter (rather than the color originally chosen). I thought it was just some weird duplicate layer ontop of another layer that would lead to the icon being darker / lighter.

I tried using Xcode's View Debugger, but it just wouldn't work at all and would error out with:

Assertion failed: ([layer isKindOfClass:[self class]]), function -[CALayer _initWithReference:], file /BuildRoot/Library/Caches/com.apple.xbs/Sources/QuartzCore/QuartzCore-697.24.4.2/LayerKit/api/CALayer.mm, line 1947.

Somehow we had a CALayer which… wasn't a CALayer? Seeing as though Mojave has been having some super broken CALayer stuff recently, I assumed it might've been a similar issue. Perhaps having too many layers was also corrupting the rendering of images?

It turns out, the original bug in question is actually an AppKit bug, but this bug lead me down a path of understanding how Xcode's View Debugger works.

Debugging a… Debugger

I figured I could just put a breakpoint on -[CALayer initWithReference:], but sadly that breakpoint was never hit. There were potentially a few reasons from this:

1. There's some out of process that handles Xcode's View Debugger.
2. This -[CALayer initWithReference:] is being invoked inside Xcode's process.
3. LLDB breakpoints don't work with Xcode's View Debugger.

Cutting this writeup short, there's no XPC process we need to worry about (thankfully), and manually attaching LLDB to Xcode and trying to break on that function didn't work either. Turns out, LLDB doesn't work when you're trying to set a breakpoint during Xcode's loading of the View Debugger, so we need to get a bit more creative…

After starting the View Debugger, we can use image list to print out all the currently loaded libraries into the app being debugged:

(lldb) image list
...
[  1] F217F7F8-A795-3109-B77F-B1E2277F3E3B 0x0000000103fea000 /usr/lib/dyld
...
[298] BD43F13C-7F88-301D-A860-77EE5F6CBBE4 0x00000001042bc000 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Debugger/libViewDebuggerSupport.dylib  
...

libViewDebuggerSupport.dylib looks interesting.

It turns out, Xcode will dynamically inject that dylib into the running application and call some special methods on it in order to get info on how the application's view hierarchy works. This is very similar to how Reveal works (albeit sadly that app doesn't work for debugging AppKit apps…).

Finding Callsites

Now that we know what binary is being loaded, we need to find out how it's actually starting this view debugging code. We can use static analysis and decompilation to figure this out.

Hopper is a fantastic tool for taking apart binaries, and it's something I always find myself coming back to. The way it works? Simply hand it an executable, and it will disassemble the executable, showing the innerworkings of what it does.

Opening that dylib up into Hopper, and searching for "viewdebugger" will give us lots of results for methods on the class DBGViewDebuggerSupport. Trawling through those results we'll eventually find [DBGViewDebuggerSupport fetchViewHierarchy]. Now this seems to be the main-ish method that's called to kickstart things, so we'll see what it does. We're trying to find where the layers are referenced so we'll see what happens with the sublayer hierarchies. Eventually we'll see something like

var_170 = [r14 _collectSubviewInfoForView:r13 encodeLayers:rcx];

This is shown in the decompiled code, $r14 and $r13 are registers in the CPU, which hold references to objects (kinda). Navigating inside that method, we'll eventually find the magic function that we seem to be breaking: CAEncodeLayerTree(CALayer *layer) which is inside QuartzCore.framework.

Calling Private Functions

Now, whilst it'd be easy to just invoke that function from LLDB, our breakpoint still won't be hit (since you can't break on things being invoked from LLDB), so we need to hardcode a way of invoking it. CAEncodeLayerTree also is private API, and has no header definition, so we'll need to use something else… something like dlopen and dlsym should help! dlopen will open / load a library into the current process and return a pointer to the address where it starts in memory, and dlsym will lookup the pointer to the function itself so we can manually call it:

#import <dlfcn.h>

void *quartzCore = dlopen("/System/Library/Frameworks/QuartzCore.framework/QuartzCore", RTLD_NOW|RTLD_GLOBAL);
void(* CAEncodeLayerTree)(CALayer *) = (void (*)(CALayer *))dlsym(quartzCore, "CAEncodeLayerTree");
CAEncodeLayerTree(view.layer);

I assigned this to a double click, but you can invoke it however you like.

Finally, to figure out the weird layer that's breaking things, we can use LLDB's conditional breakpoints:

(lldb) b -[CALayer _initWithReference:]
Breakpoint 5: where = QuartzCore`-[CALayer _initWithReference:], address = 0x00007fff496ba037
(lldb) breakpoint set --name "-[CALayer _initWithReference:]" --condition '([$arg3 isKindOfClass:[$arg1 class]] == 0)'

Here we're setting a conditional breakpoint on -[CALayer _initWithReference:] emulating the assert that was failing previously. Here, $arg3 is the argument passed to -initWithReference: and $arg1 is the object the method is being invoked on (aka self when you've hit the breakpoint). This is using aliases for registers that'd be used when calling Objective-C methods. For more details on how these "pseudo" registers work, check out @natashatherobot's post, and if you're super interested, this post goes into lots of details.

Finally, invoking this will allow us to finally find our view that's breaking things. Once the breakpoint is hit, we can just invoke po $arg3 to see what object is breaking things.

Closing Notes

In my case, I was incorrectly configuring a CAGradientLayer's delegate incorrectly. Fixing that resolved the issue.

Sometimes the best approach to fixing bugs is to, well, debug a debugger. It's pretty dense, and reverse engineering opaque technologies isn't talked about that much nowadays, so I figured I'd post more broadly about how I go about researching / investigating the internals of various things.

Hope you found this post helpful and If you've got any questions, feel free to message me!

PS: @indragie made an amazing open source project that aims to replicate Xcode's View Debugger / Reveal and it's pretty awesome. Check it out here!

This post follows where the first part left off.

Now that you've written a blog, or whatever you'd like as part of server-side Swift, the next step is to get it hosted somewhere and running all the time. Personally, I use DigitalOcean for my hosting running an Ubuntu VPS. Most of the instructions here are fairly generic, but depending on the distribution of Linux (or Unix) you're running, you may need to switch some things up.

Setting Up Swift

The first thing you'll want to do after getting your VPS is getting Swift 5 (the latest version as of writing this post) setup and running on it.

As a pre-requisite you'll need clang installed.

sudo apt-get install clang

You'll then want to download the latest version from swift.org.

Note: As of this writing, packages are only pre-built for Ubuntu, so if you're using something else, you'll need to compile your own version.

You'll want to unpack the downloaded Swift package, and then add it to your $PATH:

tar xzf swift-<VERSION>-<PLATFORM>.tar.gz
echo 'export PATH=/path/to/swift-<VERSION>-<PLATFORM>/usr/bin:"${PATH}"' >> ~/.bashrc

After that, you should be able to run Swift from the commandline.

> swift
Welcome to Swift version 5.0 (swift-5.0-RELEASE).
Type :help for assistance.
  1> var a = 1 + 1
a: Int = 2

Not Running Swift as Root

A really bad thing™ to do would be to run your Swift server as root… if someone manages to hack your server, they'll be able to wreck havoc as root. You'll want to configure your server to run as an unprivileged user.

sudo adduser swiftserver
sudo usermod -aG sudo swiftserver

This'll create a new user called swiftserver that can run privileged commands with sudo if needed (ps: don't start your server with sudo).

You'll want to logout and then log back in as the swiftserver user.

Serving Server-Side Swift

To serve the server, you can actually just run it directly via swift run and it'll be served (by default) on port 8080. However, this doesn't support SSL or any of the nice things that come with web servers (security, disabling access to folders, etc.). Whilst I'm sure you could do all of this by hand, I wanted to focus mostly on serving blog content and not the intricacies of web servers. To serve the Swift app, I setup NGINX and setup a reverse-proxy to my Swift app running. This way, the Swift app would run on a different port and NGINX would communicate with it (instead of it being the whole webserver).

It looks something like this:

[browser: https://blog.mysite.com/blog/entry] -> [NGINX] -> [swift-server] -> [NGINX] -> [browser]

Setting up NGINX is pretty easy:

sudo apt-get install nginx
systemctl status nginx

# DigitalOcean needs to have the firewall configured via ufw
# This opens both port 80 and 443, but you may want to use just Nginx HTTPS to be more secure after you've setup HTTPS
sudo ufw allow 'Nginx Full' 

Next you'll want to configure NGINX.

Create a file at /etc/nginx/sites-available/swift-blog (replace swift-blog with whatever your binary is called) and fill it out with something like this:

server {
  server_name blog.mysite.com;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;

    proxy_pass          http://127.0.0.1:8888;
    proxy_read_timeout  90;

    proxy_redirect      http://127.0.0.1:8888 http://mysite.com;

    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
  }
}

Lastly, you'll want to activate this configuration by symlinking it:

sudo ln -s /etc/nginx/sites-available/swift-blog /etc/nginx/sites-enabled/swift-blog

# Reload NGINX
sudo systemctl restart nginx

You'll want to set the server_name to use the subdomain of where you're hosting this server, and proxy_pass and proxy_redirect accordingly (this is where you supply the port your server will be running on). Now whenever you visit blog.mysite.com you'll be sending web requests to your Swift app via NGINX.

There's a lot of configuration options available here, and if you'd like to setup a separate webserver alongside this app, you can do so in this file as well. For more information on that check out DigitalOcean's tutorial here.

Securing the Server

The next thing you'll want to do is serve your content securely using SSL. Whilst this is technically optional, I think it's much more future proof and safer to setup for people consuming your content. There's countless ways to set this up, but the easiest way I found was done by setting up Cloudflare (thanks @_inside!). When you setup your domain and verify it through Cloudflare you'll be given a certificate and a private key. You'll want to put those on your server somewhere (I used /etc/cloudflare/). You'll also want to download the Origin Pull certificate here and save that into the same directory as your Cloudflare certificates.

The last thing you'll need is to generate a dhparam.pem and save that into the same directory as well. You can do that via this command:

sudo openssl dhparam -out /etc/cloudflare/dhparam.pem 4096

After you've got all those files, you'll want to edit the top of the server area of your swift-blog config with the following:

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  ssl on;
  
  ssl_certificate               /etc/cloudflare/cloudflare_origin_cert.pem;
  ssl_certificate_key           /etc/cloudflare/cloudflare_origin_privatekey.pem;
  ssl_dhparam                   /etc/cloudflare/dhparam.pem;
  ssl_client_certificate        /etc/cloudflare/origin-pull-ca.pem;
  ssl_verify_client on;

  ...

This change will enable serving content with NGINX using HTTPS / SSL.

After making these changes you'll also want to change your firewall settings and reload NGINX.

sudo ufw allow 'Nginx HTTPS' 
sudo systemctl restart nginx

Keeping Your Server Alive

Sometimes apps crash or servers fail, so you need to ensure that your Swift app can restart itself in case it fails in any way; that way it can still communicate with NGINX and the web. Setting this up with Ubuntu is pretty easy, you just need to create and register a service for it.

Simply create a file at /etc/systemd/system/swift-blog.service with the following contents:

[Unit]
Description=swift-blog
After=network.target
[Service]
Type=simple
Environment="PORT=8888"
WorkingDirectory=/path/to/your/swift-blog-folder
ExecStart=/path/to/your/swift-blog-folder/swift-blog
Restart=always
StandardOutput=file:/var/log/swift-blog.log
[Install]
WantedBy=multi-user.target

You'll then want to activate your server service using the following:

sudo systemctl start swift-blog
sudo systemctl enable swift-blog

After that, you should be able to visit https://blog.mysite.com/ and your Swift server will be running!

Conclusion

Writing my whole blog in Swift and then getting it running as a fully fledged web app was a really neat endeavour. It was fun starting from square one and getting everything running from the ground up; I definitely learned a bunch and I hope in reading all of this you. It's still wild as a concept to me, being able to use all of my Cocoa knowledge I've built up over the years to use for a Linux server box. I'm still learning how to make it better, caching things, etc. so I'm always open to suggestions on how to improve.

Let me know how your own server-side Swift projects go and if you have any questions feel free to ask!

Recently I attended try! Swift in Tokyo, Japan (which was an absolutely fantastic conference), and there I heard about a lot of folks that were using (or beginning to use) Swift on the Server. Now that Swift has reached version 5, and its ABI is stable, I thought it'd be a great time to try it out for myself. I didn't really know what to write, but I saw that @stroughtonsmith had recently rewritten his blog as a static one being served by PHP and I thought that could be a pretty cool thing to make in Swift! My previous blog was hosted on Tumblr, and it ended up being a lot of maintenance to do simple things, which was pretty discouraging from wanting to write things. I wanted to write something that would make publishing a blog with media and code samples trivial without importing thousands of dependencies.

The end result however, well, you're reading it. As of this post, my entire personal website and blog runs on a server I wrote in Swift using IBM's Kitura, which, in-turn, is running on some Linux box in the cloud. I can easily write new posts, upload media and code samples, and the backing for it is all type-safe and native (thanks to Swift). Updating it is as easy as compiling an Xcode project, and there's no more need to trawl through arcane linux directories to find error logs.

This post will go over the basics of how I wrote my blog, and how you can get started writing your own. I wrote it in a way where you can get started with little server-side code experience, so hopefully this post gets you on your feet and helps you get started in the world of Server-Side Swift.

🚨 Disclaimer 🚨

In all honesty, my 15+ years of experience with server-side code has been:

• Publishing jQuery + HTML files with Transmit's upload button
• Copying and pasting some PHP code off stackoverflow
  • … and that's about it

So I went into this completely blind not knowing anything at all of what I'm supposed to be doing. ¯_(ツ)_/¯

I normally write apps, not distributed servers. This ended up being way outside my comfort zone, so I'm still learning, and if there's things I'm missing or could do better, please let me know!
Pro-tip: Don't run your server as root. It's a very bad idea.

SaaS, Swift as a Server

Starting out, I didn't really know what I should be doing or how I should start. How does one get Swift working on Linux? How does the app I write in Swift produce HTML for browsers to read? In the past, if I wanted to write scripts for my website I'd just open a new PHP file and it'd work.

I started searching and was slammed with a wall of information:

• Use Kitura with Kubernetes and Bluemix!
• Use a Vapor Docker image with Apache!
• Use NGINX + SwiftNIO!
• Setup Express and NodeJS… 🤔

All of that was a lot of dense reading and background info on a lot of technolgies I've only heard in passing. Half the time I was thinking to myself wtf is all of this?. I decided to start with the basics and just see what I could get working so I ended up picking up Kitura because their support team were super nice and it seems as though they've been contributing a fair bit to Apple's open source projects too. I have nothing against any competing open source projects (i.e. Vapor), I just personally haven't used any of them.

Anyways, setting up Kitura is pretty simple:

brew install kitura

mkdir blog-test
cd blog-test
kitura init

Note: The name of the folder you run kitura init inside is the name of the target that will be created. Here we're using blog-test.

This'll give you a barebones project setup with the Swift Package Manager. I wanted to write things in Xcode, so I did the following:

swift package generate-xcodeproj

And that created an Xcode project I could build and run with.

Note: This also adds a bunch of other stuff that I didn't end up using but could be helpful (i.e. Health, Analytics, etc.). They all are optional, so if you don't want them you can remove them.

⌘+R and you'll get a web server running at http://127.0.0.1:8080 or http://localhost:8080

Application.swift is the main entry point for things and here is where you'll setup routes. Routes are a way of parsing incoming network requests, doing some logic with them, and then giving back a response.

For example:

router.add("/neat") { (response, something, something) in 
  response.write("neat! 📸")
}

Will output nothing but "neat! 📸" when you navigate to http://localhost:8080/neat. While that's cool and all, that doesn't really allow the dynamism of Swift to show, and it isn't that helpful for building sites easily. Swift + Stencils is something I adopted to make this far more easy.

Stenciling HTML

So Kitura provides out of the box support for Stencil. Stencil is a template language for Swift and is a way of adding preprocessor functionality to HTML. You can take a context defined in Swift and use that to generate or fill-in placeholders in the HTML. I needed to write a listing of all my blog entries, so I could write a stencil file like this:

<!DOCTYPE html>
<html>
  <body>
    <ul>
      {% for post in posts %}
      <li>
        {{ post.title }}
      </li>
      {% endfor %}    
    </ul>
  </body>
</html>

And after using a JSON data structure like this:

{
  "posts": [
    {
      "title": "My first blog post"
    },
    {
      "title": "My second blog post"
    }
  ]
}

I could create something like this:

<!DOCTYPE html>
<html>
  <body>
    <ul>
     <li>
       My first blog post
     </li>
     <li>
       My second blog post
     </li>
   </ul>
  </body>
</html>

Neat huh? It's similar to writing inline PHP code for generating missing parts of HTML, or filling in fields as a page is being loaded. The difference, however, is Swift is supplying a single context object to the HTML stencil, which is what the stencil will be populated with.

Serving Things in Swift

To setup stenciling, we need to opt-in for it. You'll want to edit Package.swift to look something like this:

// swift-tools-version:5.0
import PackageDescription

let package = Package(
  name: "blog-test",
  dependencies: [
    .package(url: "https://github.com/IBM-Swift/Kitura.git", .upToNextMinor(from: "2.6.0")),
    .package(url: "https://github.com/IBM-Swift/HeliumLogger.git", from: "1.7.1"),
    .package(url: "https://github.com/IBM-Swift/CloudEnvironment.git", from: "9.0.0"),
    .package(url: "https://github.com/RuntimeTools/SwiftMetrics.git", from: "2.0.0"),
    .package(url: "https://github.com/IBM-Swift/Health.git", from: "1.0.0"),
    .package(url: "https://github.com/IBM-Swift/Kitura-StencilTemplateEngine.git", from: "1.8.0"),
  ],
  targets: [
    .target(name: "blog-test", dependencies: [ .target(name: "Application"), "Kitura", "HeliumLogger"]),
    .target(name: "Application", dependencies: [ "Kitura", "CloudEnvironment", "SwiftMetrics", "Health", "KituraStencil" ]),

    .testTarget(name: "ApplicationTests" , dependencies: [.target(name: "Application"), "Kitura", "HeliumLogger" ])
  ]
)

Note: At time of writing I'm using Swift 5 for this, so you may need to update .swift-version and ensure your Xcode version is at least 10.2. Also there's a strong chance these packages have updated versions now, so you'll also want to update those accordingly.

Then run the following to regenerate / update your project:

swift package update
swift package build-xcodeproj

Setting up stenciling templates isn't all that difficult. By default, Kitura will render template files provided in Views/. So let's create a folder named Views next to the Sources folder and inside Views/ create a stencil file called blog-posts.stencil.

Fill it with something like this:

<!DOCTYPE html>
<html>
  <body>
    <ul>
      {% for post in posts %}
      <li>
        {{ post.title }}
      </li>
      {% endfor %}    
    </ul>
  </body>
</html>

Next, we need to have our server render the specified template file when requested, so we need to setup a route for anytime someone visits posts to populate a data structure of all the posts for filling in the template.

In Application.swift, import KituraStencil add the following just before Kitura.addHTTPServer(onPort: cloudEnv.port, with: router):

// Enable the stencil engine and add it to the router.
router.add(templateEngine: StencilTemplateEngine())

// Create a data structure to be shown by the stencil.
let posts: [String: Any] = [
  "posts": [
    [
      "title": "My First Post"
    ],
    [
      "title": "My Second Post"
    ],
  ]
]

// Anytime someone navigates to http://mysite.com/posts run this closure.
router.get("/posts") { request, response, next in
  do {
    // If it's successful in rendering, mark the response as OK.
    try response.render("blog-posts.stencil", context: posts)
    response.status(.OK)
  } catch let error {
    // Otherwise, log out an error.
    response.send(json: ["Error": error.localizedDescription])
  }

  // Calling next will allow other parts of your server to respond to similar requests (rather than just halting here).
  next()
}

Build and run, and if you did everything correctly, open Safari and navigate to http://localhost:8080/posts.

You should see something like this:

Next steps here would be to write something to accept an arbitrary URL and use that to serve content dynamically. This would allow for linking back to specific posts via something like this:

http://localhost:8080/myPost

Kitura supports queries and variable parameters too, and it's pretty simple to declare:

router.get("/:postName") { request, response, next in
  // postName would be myPost in this case
  guard let postName = request.parameters["postName"] else {
    response.status(.notFound)
    return
  }

  // TODO: do something with the post name.

  next()
}

Kitura Markdown

One thing that you could do with that postName variable is use it to render a markdown file with a specific filename. Kitura supports it out of the box as well!

Add KituraMarkdown to your Package.swift:

.package(url: "https://github.com/IBM-Swift/Kitura-Markdown", from: "1.1.0"),

And also add it as a dependency to the Application target:

.target(name: "Application", dependencies: [ "Kitura", "CloudEnvironment", "SwiftMetrics", "Health", "KituraStencil", "KituraMarkdown" ]),

Then run and update things:

swift package update
swift package generate-xcodeproj

In Application.swift you'll want to import KituraMarkdown and add the rendering engine for it (just like the StencilTemplateEngine):

let markdownRenderer = KituraMarkdown()
markdownRenderer.setRootPaths(rootPaths: [ "Views/blog" ])
router.add(templateEngine: markdownRenderer)

let blogFileServer = StaticFileServer(path: "Views/blog")
router.get("/", middleware: blogFileServer)

Now anytime the server tries to render a markdown file inside Views/blog it'll convert the markdown file and render it as HTML. You'll also notice that we added a static file server. This is so that anytime a resource is referenced inside the markdown file (i.e. ![myimage](myimage.png)), the server will know to transform that path myimage.png into something like Views/blog/myimage.png.

Create a markdown file called markdown-test.md inside Views/blog/ and fill it with something like this:

This is a markdown file!

# It Supports Big Headers
## Smaller Ones
### And Even Smaller Ones

![cat](meow.jpg)

Note: Don't forget about the image, feel free to swap it with whatever you like, but it's a good way to test. Just put it right next to the markdown-test.md file.

A lot of blog sites allow linking to specific blog articles usually in the form of website.com/your-blog-entry-here so we need to have a way of mapping everything after the / to a particular blog entry to render. To make things super simple, we can just map the filename. In this case, navigating to http://localhost:8080/markdown-test should render our markdown file.

To do this, we need to setup a route to handle that parameter. Kitura, again, makes this pretty easy to setup. Add the following after the blogFileServer setup code to allow the server to render arbitrarily specified markdown files:

router.get("/:postName") { request, response, next in
  // Gets the argument passed after the "/"
  guard let postName = request.parameters["postName"] else {
    response.status(.notFound)
    return
  }

  // Tries to render the markdown file named {postName}.md stored in the blog folder (contained in the Views folder)
  try response.render("blog/\(postName).md", context: [String: Any]())
  response.status(.OK)
  next()
}

Assuming everything worked, you'll have something that looks like this:

Serving / Securing Server-Side Swift

This blog post is already gigantic, so I'll be writing more in Part 2 which will be coming soon! We'll be covering how to take all of this swift server code and putting it on a server. We'll also be talking about how to setup SSL so it's nice and secure for people to consume it.

Closing Notes

Learning all of this and building it over time was a lot of fun. It's inspired me to write a lot more, especially since a lot of the pains of blogging before have been fully removed in my own workflow, written by me. Any minute changes I want I can easily change, and it's built atop a pseudo-familiar platform to me. I hope you found this guide useful, and maybe it'll get you excited about writing Swift on the server too!

If you have any questions, feel free to give me a shout on twitter.

I can't wait to write more, definitely have some things planned to talk about (synths, sounds, etc.) but until then, I'm just praying that this server doesn't get taken down :P

WatchKit apps are handcuffed, as it were, only able to display UI through the elements WatchKit provides. They cannot get the position of touches onscreen, or use swipe gestures, or multitouch.

Seriously, it’s horrible. Being able to build proper apps would be so great, instead of simple ways to display JSON files with Interface Builder (yes, that’s basically what WatchKit currently is). Especially watch faces, I’d love to be able to build my own (even powered by Sprite Kit, I know the stock ones use it!), I’ve got so many ideas for things to build.

Humanizing LLDB

I roped @b3ll into a torturous day & night of testing things, he in a cafe across from WWDC (with @saurik for much-needed moral support!), me across the Atlantic.

10/10, can confirm, working as a remote, human, lldb / gdb session over iMessage was really weird. I’m pretty sure I have 30+ zipped up attempted builds, all of which had more and more aggressive naming schemes as time progressed :P (fku.zip). Sometimes the apps would install, sometimes they wouldn’t. Sometimes they’d insta-crash, sometimes they wouldn’t. Sometimes they’d just insta-crash Carousel.

¯_(ツ)_/¯

PS: Here’s a visual representation of how debugging went.

:/

:(

:D

Specs Pls

After getting a basic UIKit app going, we started trying out various other iOS projects to see how they’d scale (scale… hehe… the watch is tiny… geddit?).

PepperUICore classes are prefixed PUIC and are mostly subclasses of UIKit classes. Your root UIApplication subclass should be PUICApplication.

While messing about with PepperCore, we whipped up an app to report the specs although whenever we tried to query the CPU speed it blew up. It’s got a really weird CPU(s?). Still… that’s a lot of RAM for a tiny watch!

Doomed Betas

Next I tried porting Doom to the Apple Watch, since that’d be well, totally really super awesome. Doom is the most ported game of all time and it runs on calculators, so surely it must work on the Apple Watch, right?

watchOS can be so incredibly fickle when installing binaries. I’ve seen this with non-hacked regular WatchKit apps too, but you can get into a state where the OS refuses to install a binary and gives you random error messages. Sometimes rebooting fixes this, sometimes removing the app from your phone helps. Sometimes the exact same binary won’t install one minute, and works the next.

I actually had it compiling and running at one point, but the fact that the builds were like 30mb made it excruciatingly painful to install and debug (transferring 30mb wirelessly took like 30 minutes). It also didn’t get very far, sadly. It was crashing on some weird and random OpenGL bugs, which I just blamed on the buggy 2.0 beta 1 software. What was even more “fun” was when I’d actually try and use the watch normally by raising my wrist, you know to check the time, and then seeing as it would just deadlock itself.

Yes… yes… I know… “Friends don’t let friends install iOS betas.” -@mxweas

Nano Canabalt

What didn’t help was I had no debugging session to attach to, nor did I have any crashlogs (Steven says they take an hour… um no, more like a day ;P). To make things easier I migrated to a slightly smaller and more manageable codebase, Canabalt! Cramming everything to fit onscreen and make it playable took some messing about, but eventually it worked pretty great. Sound, controls, 60fps, the whole deal (the soundtrack is fantastic btw). Not quite sure how quickly it would drain the battery, but it seemed okay!

Closing Notes

Working on this and keeping it somewhat secret was super fun, and a challenge at times, but now that it’s out in the open, make some cool stuff! If you do make anything cool, feel free to give me or @stroughtonsmith a shout! :D

watchOS is iOS; most of the frameworks you expect to be there are actually on-disk. However, the SDK will not include them, or have complete headers for them.

It’s so true. Everything “just works”, well, within reason. Once you’ve got ramped up and things working, do give it a shot at some point!

I know my Click Wheel Keyboard is just begging for an Apple Watch port… Digital Click Wheel Crown anyone? :D

Ever since Apple first announced Passbook two years ago, I've always liked the idea of using it to store business cards.

However, it'd be far more useful if it could store Pokemon cards... someone should file a radar on that...

Every year at WWDC (and any conference really) it's super fun to play the "trading card game" and collect various business cards from people. Some are really cool, some are so fantastic that the people that gave you their card ... will ask for it back -_-

The first day Passbook was announced was when I made my first card and hopefully this'll stir up some creative Passbook business cards come WWDC 2014!

tl;dr: they're cool, you should make one.

How do I Passbook?

Making a Passbook business card is, well, easier than writing HTML! For the most part you're just filling in blanks in a JSON file. You create a folder, and inside that folder you put the main JSON file, throw in a few images, and BAM Passbook card (waves hands)!

Sharing them is just as easy, especially with iOS 7. With iOS 7 came AirDrop (a somewhat crippled version, fwiw) and with that you can share Passbook cards over AirDrop, or anywhere else from the standard iOS share sheet.

How do I JSON?

Let's look at mine as an example:

{
  "formatVersion" : 1,
  "passTypeIdentifier" : "pass.adambell.businesscard",
  "organizationName" : "diffractive",
  "serialNumber" : "0x1337",
  "teamIdentifier" : "sekrit",
  "description" : "Adam Bell",
  "generic" : {
    "primaryFields" : [
      {
        "key" : "name",
        "label" : "",
        "value" : "Mobile Developer"
      }
    ],
    "secondaryFields" : [
      {
        "key" : "website",
        "label" : "Web",
        "value" : "www.adambell.ca",
        "textAlignment" : "PKTextAlignmentLeft"
      },
      {
        "key" : "twitter",
        "label" : "Twitter",
        "value" : "@b3ll"
      }
    ],
    "auxiliaryFields" : [
      {
        "key" : "email",
        "label" : "Mail",
        "value" : "someemail@gmail.com"
      }
    ]
  },
  "barcode" : {
    "format" : "PKBarcodeFormatPDF417",
    "message" : "https URL here",
    "messageEncoding" : "iso-8859-1"
  },
  "backgroundColor" : "rgb(221, 18, 29)",
  "foregroundColor" : "rgb(255, 255, 255)",
  "labelColor" : "rgb(255, 255, 255)",
  "logoText" : "Adam Bell",
  "suppressStripShine" : true
}

The meat of the pass is under the generic section. Depending on the type of pass you're making, you can specify a multitude of fields (like departure / arrival if you were making a flight ticket), but for our sake we're just re-purposing a generic pass. Your teamIdentifier is used to specify your signing identity if you want to sign your pass but we'll get to that.

Here's how I tailored my pass as a business card:

Description

I don't really know if this used on iOS, but on Windows Phone it shows up in the live tile... more on this later.

Primary Fields

Set the value of name to a title for yourself, I kept Mobile Developer to keep things nice and simple. If you want to be fancy, might I recommend something like "iOS Connoisseur" (complete with air quotes)?

• Side-Note: Each of these fields also has a label which is basically just a way to define what it is. The key can be whatever you want, as long as it's unique ('name' could be 'title', or even 'panda' if you really want to be 'cool').

Secondary Fields

Here you get up to two fields to specify information (after that it looks kinda stupid with text all smushed together). I chose my twitter handle and website, since they're the easiest way to show off what I've done as well as get quickly in contact.

Auxiliary Fields

You get 1-2 more fields for info. I chose my email, since I'd rather people use it as a last resort (for most things) ;)

Barcode

This is the fun part. With iOS 7 came the ability to scan barcodes to download Passbook passes. It uses the same wicked fast iTunes gift card scanning / redeeming tech, but comes with some minor annoyances.

First off, the link specified here must be HTTPS. ...unless of course you break PKAllowHTTP() ;)

You get 3 types to pick from:
- Aztec
- QR
- Standard barcode (which I think looks best)

If you choose Aztec or QR, the rest of the card just ends up being a massive zebra.

Also, you might want to enable suppressStripShine, last thing you want is your pass looking like a shiny zebra (unless you like shiny pokémon!)

Everything Else

• backgroundColor: well... the background colour (in rgb format)
• foregroundColor: you guessed it! the foreground colour (in rgb format)
• logoText: the text that shows at the top of the Pass
• If you drop a logo.png (don't forget @2x assets), Passbook will display your logo in the top left.
• If you drop a thumbnail.png (don't forget the @2x assets), Passbook will display a photo in the middle of the pass.
• On the back side you can provide additional info, but I didn't feel it was necessary (personally).

"Security" Notes

• Update: I was nice and decided to upload a direct link to a version of Apple's signpass utility which skips any .DS_Store files from being added to the pkpass file here.

You can invoke it like so:

pkpass -p <some folder> -o <outputfile.pkpass>

• You need to sign your pass. Apple has some great notes and a sample signing utility here. So get ready for some provisioning profile dancing!
• Read that ^
• If you want to share your pass via barcode:
  • It needs to be hosted on an HTTPS address
  • It needs to be distributed with pkpass as the file extension.
  • It needs to be distributed with application/vnd.apple.pkpass specified as the MIME type.
• Amazon S3 provides all this!
• Read the first point again.

Enter the Windows Phone

What's even more awesome is Microsoft decided to be nice (or lazy :D) and support Apple Passbook cards in Windows Phone 8.1! They work out of the box too! You can simply email a pass to your Windows Phone and add it that way.

What's nice is you literally have to change nothing, it just works.

That's a first for Microsoft. :P

As I mentioned earlier, description is the field that defines what will show on the Windows Phone Start Screen when you add your pass as a live tile (super quick access to contact details!). Best practice for this is to keep it short and simple unless you have an arbitrarily long name that just overflows everything...

• Side-Note: One thing I did notice about loading passes with Windows Phone, if it throws an extremely useful error 0, your JSON is probably malformed. Run it through jsonlint or something equivalent, fix it, and then you should be good to go! :D

Party Time

If you haven't used Passbook yet, this might be a fun way to start. Might put you on the "bleeding edge" of business cards! :D

PS: Pokémon Passbook cards really should be a thing.



I wish I could have a larger strip image, a gold border, and a purple background, but the current Passbook spec doesn't allow it. :(

I'm not a huge vim person... yet. I like IDEs, usually... (I'm looking at you Visual Studio). Therefore I'll only really use a shell for cycript or other random ssh sessions and hax and such.
However, fish is making the shell fun.

What the fish?

So what is fish anyways? While it sounds like the result of preparations of a sushi plate, it's this really awesome shell replacement. Bash, zsh, tcsh (ew), they're all shells that we can use with a terminal to control the operating system or run a bunch of programs without the need for a mouse.

tl;dr: They're what Hollywood uses to make someone look like a "hacker".

Anyways, fish does all that and more. You get ALL the colours with term256 support (it looks nice). You also get auto-completion, which is awesome.

Did I mention it likes emoji? Yeah! Sheep Shell! (or baaaaaaash if you're @stroughtonsmith)!



## git... add? commit? push? --force push? Fish shell does this crazy magical thing, that parses man documents to give you possible auto-completions for whatever you're typing in.

ls -? yep.
ps -? yep.
ssh -? yep.

Look (git):


Simply type in a command and hit tab (as you normally would in most shells to auto-complete directories and such). It also auto-suggests previous commands that you've used in the past so:

history | grep git
// OH RIGHT I REMEMBER
git --force push

becomes:

git --force push <CTRL+F>

It's super simple, and looks great too. To accept a completion, simply hit CTRL+F (which I assume you can change if you really want to, I haven't bothered).

You can add to this as well, such as bpinto's oh my fish (which I highly recommend).

In addition, it auto-completes directories and it tells you if a directory path is valid or not (with colours!):


## script.fish Scripting in fish is actually quite easy. It doesn't like a lot of bash's syntax, but it's super simple to pick-up. Here's a quick primer:

#!/bin/bash

export hey="sup yo"
alias ls="say l s"
whereami () { 
  echo `pwd`
}

#!/usr/local/bin/fish

set -x hey "sup yo"
alias ls "say l s"
function whereami()
  echo (pwd)
end

Basically replace with ( ) and remove some square brackets. It's well documented too, so be sure to check that out. Also, if you still want to use scripts from other shells, just use a hashbang (#!/bin/sh) and it'll work as expected. You can write scripts and save them as functions to call at your leisure, just write the script and stick it in:

~/.config/fish/functions/

Also, functions are super quick to write and save in the shell:

00:21 adam > function whereami
               echo (pwd)
             end
00:21 adam > funcsave whereami
00:21 adam > whereami
/Users/Adam

In addition, you can find your equivalent of a .bashrc / .bash_profile in:

~/.config/fish/config.fish

From there you can edit your greeting and prompt or add other environment variables:

#!/usr/local/bin/fish

function fish_greeting
  echo The time is currently (date).
end

set -x PATH /usr/local/bin $PATH

function fish_prompt
  printf "%s\n%s >" (pwd) (whoami)
end

sushi

Fish is great, and it's fun to change things up!

Do I recommend it? Sure!
Do I not recommend it? Sure!

Only way to know if you'll like it is to try it (if you do launch it from another shell, sometimes it screws up text input). Speaking of trying things, I've also done this (bye nano :( ):

function nano
  vim $argv
end

iOS 4 brought us the wonderful feature to switch between apps quickly, and iOS 7 made this even more interactive through use of live previews of apps.

I noticed an article written by @vpdn shared on twitter about making this switcher a little more lively through use of curated app content for multitasking. In other words, re-purposing the iOS 7 switcher as a sort of dashboard / widget interface for your applications.

I wanted to try and further extend this.

ALL the Apps

The iOS 7 switcher is, as I see it, Exposé (for the Mac) brought to iOS. It gives you an overview of everything that's running (and that has been running) sorted in order of usage. This is displayed through previews of applications, as well as their respective icons and titles. However, it stops there, and doesn't really provide any insight to anything else.

Extensions

A neat example of extending this functionality is re-purposing them as widgets, for example, Facebook or Instagram. Imagine a user switching from the twitter app and noticing a widget like Facebook displaying some photos they were just tagged in. Another usage could be to show coverart for the currently playing as another form of heads up display. These give the switcher a more personalized and unique experience (than simply images which are basically just the app's Default.png).

The Fun Part

In order to conserve battery life, and free up CPU time, iOS throttles down applications that have been dismissed; videos stop playing, NSTimers get paused, applications don't get any updates. When the app enters background state a current snapshot of the application's window is performed and saved to disk to be shown as the preview:

// Depending on orientation it'll append -Portrait, -LandscapeLeft, etc. to the image saved 
[[UIApplication sharedApplication] _saveSnapshotWithName:@"UIApplicationAutomaticSnapshotDefault"];

This is just a static image, and the rendering context for your application is paused when the application enters background state. So, currently, there's no way to alter that without breaking your app (or making Apple's App Review very angry face with you), or so I thought...

See with iOS 7 they introduced some awesome backgrounding features, one of which being Background App Refresh! This allows developers to update their applications in the background (like Facebook, Twitter, etc.) to save users a few "pull to refreshes". When the user returns to an app that's been updated in the background, it'll always feature the latest content.

When a backgrounded app is refreshed, the OS does something like this:

1. Notification for background fetch comes in for application
2. Wake application using -[UIApplication _shouldFakeForegroundTransitionForBackgroundFetch]
3. Application handles network request for update
4. Application refreshes UI to match said request
5. Snapshot the whole window! (-[UIApplication _saveSnapshotWithName:] shown above)
6. Send application back to background
7. Show snapshot in Multitasking.

During this it eventually calls (with proper parameters obviously):

[[UIApplication sharedApplication] _replyToBackgroundFetchRequestWithResult:0
                                                    remoteNotificationToken:@"token"
                                                             sequenceNumber:@(someNumber)
                                                  updateApplicationSnapshot:(YES || NO)]; // ohai

So here's the fun part, this can be horribly abused :D!

Let's re-purpose the app switcher preview!

Whenever we refresh the app in the background, why not just throw a new UIView on top of everything to display ...and hook everything up to a CADisplayLink? >:D

/* UIApplicationDelegateSubclass.m */
- (void)applicationDidEnterBackground:(UIApplication *)application 
{
  _backgroundID = [application beginBackgroundTaskWithExpirationHandler:^{}];
}

- (void)applicationDidBecomeActive:(UIApplication *)application 
{
  [application endBackgroundTask:_backgroundID];
}


/* UIViewControllerSubclass.m */
- (void)viewDidLoad 
{
  [super viewDidLoad];
  [CADisplayLink displayLinkWithTarget:self selector:@selector(updateALLTheThings)];
}

- (void)updateALLTheThings 
{
  [[UIApplication sharedApplication] _replyToBackgroundFetchRequestWithResult:0
                                              remoteNotificationToken:@"no u"
                                                       sequenceNumber:@(1337)
                                            updateApplicationSnapshot:YES];
}

Tada! Dynamic application previews in Multitasking!

1. Don't /ever/ do this. :P
2. If you try this on backgrounded GL applications, it'll explode and crash.
3. This is not App Store-safe and will be an insta-deny.

The Sane Approach (aka App Store-safe-ish)

Since all of this is governed by the background app refreshes, and your UI will automatically update anyways... why not put that to good use?

Take Instagram for example, you're always going to see a timeline of photos in the app, and seeing a tableview as an application preview doesn't really make sense. A more appropriate preview could be a photo slideshow for your timeline (kinda like Facebook Home). To prove my point, I made a demo of it here.

Basically when my demo goes into the background, a background task is created for an indefinite amount of time (10mins most likely, until watchdog or another application kills it), and when it resumes said task is cancelled. By doing this I can keep the application running (swapping photos in the slideshow) and just apply them as an overlay.

However, one could do this without using private API (and no, it's not witchcraft):

- (void)application:(UIApplication *)application performFetchWithCompletionHandler:(void (^)(UIBackgroundFetchResult))completionHandler 
{
  // Update my preview and do cool stuff here
  // Maybe add a UIImageView on top of everything?
  completionHandler(UIBackgroundFetchResultNewData);
}

Obviously you'll need to setup iOS 7 backgrounding by altering your Info.plist and such (you'll find more of that in the iOS 7 documentation).

In addition, you could always try providing a different overlay to summarize information last displayed in your application through use of:
`objectivec

• (void)applicationWillResignActive:(UIApplication *)application `

Give it a go! Might be neat to see a more personal iOS 7 App Switcher :)

On April 12th, 2013, Facebook released Facebook Home, a completely new mobile experience available solely to Android users. With Home came Chat Heads, a wickedly cool way to chat with your friends, in a way that makes that “face-to-face conversation experience” a little more feasible with text messaging. The minute I saw it, I thought “I can make this!”.

Using box2d as a boilerplate, I started working with giving contact photos the same (or similar anyways) physics that powers Chat Heads. I got some progress, and made it system-wide, but it wasn’t even close to being complete. It wasn’t but 2-3 days later that Chat Heads was announced for iOS. However, due to Apple’s sandboxing and policies, it was solely contained in the Facebook app :(

STOP. APP STORE. INSTALL... was my first thought.

STOP. HOLD THE PHONE. HAX... was my second.

Why not “break” Chat Heads out of the iOS Facebook app?

XPC and Watchdog

The Facebook app is pretty complex, loads of view controllers and modules to make it all work together, but thankfully Chat Heads has its own view controller, and with that a view! Well that’s fantastic and all, you can’t just start ripping views out of an app and shoving it somewhere else, or so I thought...

I thought of using XPC to fix this. XPC is a framework developed by Apple that allows you to exchange things between running processes or applications. Those things are somewhat limited:

1. XPC only plays nice with serializable objects (aka Dictionaries, Strings, and stuff)
2. Apple’s doing cooler things with views!

When you open the multitasking bar, by double clicking the home button (or using that awesome slide-up gesture provided by Zephyr) Apple needed a way to move the current app upwards, or transform how it looks, without changing how the actual application renders. They didn’t want to mess with any custom code in the application, so they created another context to share the application’s rendering view (if that makes sense). SBAppContextHostManager handles all this. An application on iOS runs inside a UIWindow, a window that holds a bunch of views. So if we copy all the views inside that UIWindow, and render them inside another view, you can do whatever you want to this new view, without altering the original. Think of it as taking a photo, duplicating it, and messing with the duplicate. You can mess with it all you want, but the original stays the same. When you open the multitasking bar, Apple uses SBAppContextHostManager to create a new view and move it upwards to show the multitasking bar underneath, that way the original application’s layout isn’t touched.

I could do that with the Facebook app; render the Facebook app inside another view, and put it somewhere where it’s on top of everything else. This worked perfectly until you actually closed the Facebook app.

To save power, when an application is closed, the views aren’t rendered anymore (so the GPU can rest), unless you tell it explicitly not to! That's where some private API comes into play!

+[UIWindow setAllWindowsKeepContextInBackground] 

... until a few minutes later when iOS decides to kill the Facebook app. :-|

This was the original problem shown in the Verge article; it freezes. On iOS there’s a daemon called watchdog. This daemon watches over everything running on iOS, and if you’re taking up too much memory or are in the background for too long, it’ll kill your app. It’s fantastic to save power and keep things running quickly, but annoying in this instance.

Some applications, however, can be granted permission to continue to run in the background, for example Reprise. Reprise is a music player, and if you give it backgrounding permissions, it’ll continue to play music after you close the app. Unfortunately Facebook isn’t a music player, and doesn’t have any special backgrounding permissions (other than for VOIP purposes), so I had to tweak things on a lower level than just the Facebook app.

## BackBoardServices

On iOS the main homescreen / lockscreen is called SpringBoard, but in iOS 6, Apple made things more interesting and decided to split up some of SpringBoard into backboardd. The result? SpringBoard handles more springing, and backboardd handles more of the behind-the-scenes application life managing. There’s a neat class in BackBoardServices (a private framework) called BKSProcessAssertion. This class allows you to give a certain application permissions to stay alive and not be killed by watchdog. So then I basically gave the Facebook app infinite backgrounding using the appropriate flags. To do this, I granted SpringBoard the ability to hand out BKSProcessAssertions to running applications (makes things easier tweak-wise). Yay! No more pausing, no more getting killed, and it’s system-wide!

// Replace this function using MobileSubstrate with a dylib into backboardd (or install MessageBox)
static int hax_XPConnectionHasEntitlement(id connection, NSString *entitlement)
{
  //Only grant the required entitlement
  if (xpc_connection_get_pid(connection) == PIDForProcessNamed(@"SpringBoard") && [entitlement isEqualToString:@"com.apple.multitasking.unlimitedassertions"])
  return 1;

  return orig_XPConnectionHasEntitlement(connection, entitlement);
}

Cycript to the Rescue!

Cycript is awesome. It's basically a prompt that can hook into processes and accepts javascript as well as Objective-C syntax to play with memory. It requires jailbreaking your device, but @saurik has also released a version that you can link with App Store applications for debugging :D It's perfect for prototyping, and this was how I created the first prototype that was shown on the Verge. The best part is if anything goes wrong while messing in Cycript, you can just restart the process you're hooking, therefore doing no permanent damage (hopefully! :P).

Hosting Views Where They Probably Shouldn't Be

By creating a SBContextHostView for the Facebook app, one can display the whole Facebook app on top of any running application. You'll also need a BKSProcessAssertion (as mentioned earlier) to keep it running.

SBApplication *app = [[SBApplicationController sharedInstance] applicationWithDisplayIdentifier:@"com.facebook.Facebook"];
UIView *hostView = [app1 contextHostViewForRequester:@"epichax1" enableAndOrderFront:YES];
[[UIApplication keyWindow] addSubview:view1];
view1.transform = CGAffineTransformMakeScale(0.4, 0.4);
view1.frame = CGRectMake(0, 0, 320, 568);

BKSProcessAssertion *keepAlive = [[BKSProcessAssertion alloc] initWithPID:[app pid]
                                                                    flags:0xF
                                                                   reason:7
                                                                     name:@"epichax1"
                                                              withHandler:nil];

Tada! Mini Facebook app! :D

For JailbreakCon 2013 I wrote a cycript demo that does all this and displays 4 applications running simultaneously on top of SpringBoard. It's hilarious to see!

Post-Cycript

Of course there was much more work than this to actually make it usable and pleasant, such as transitions when quitting the Facebook app, and handling rotation even when the Facebook app is closed. Originally I was refreshing the app every 15-60 seconds to see if there were any new chat messages, however this shoved battery life into the ground and stomped on it. To save battery life, I hooked Facebook Messenger’s push notifications and redirected them to the Facebook app so Chat Heads update their conversations accordingly.

When all was said and done I released this hack, for free, under the name MessageBox. Sadly this no longer works on iOS 7 (nor the current version of the Facebook app), but I'll be sure to bring it back... someday!

Facebook Chat Heads are a wonderful way to chat. The ability to be able to chat with anyone, from inside any application, without the need to switch apps is something that isn’t new to mobile platforms (Android for example), however Chat Heads takes things to the next level. An experience that still, to this day, makes me smile when a chat head springs off one side and docks on the other.

Personally, I see the Native SDK as the most interesting, as it allows one to use the device to its full potential alongside its realtime operating system (RTOS), so I started to do some digging. I poked around with the simulator for a bit, when @stroughtonsmith suggested I download QNX's desktop environment, otherwise named the Neutrino SDP (software development platform). What's interesting here, is the fact that it's equipped with an arm cross-compiler. What this means, is that one can already create native applications for the PlayBook, which end up being very similar to those produced with the native SDK, as well as being sandboxed.

So after installing QNX from the DVD (can be to a virtual machine, or an actual desktop), boot it up, and login as root (no password is needed). Afterwards, simply write up an application in C/C++, and, when ready to compile, invoke the cross-compiler like so (help info here):

qcc -Vgcc_ntoarmle -o nativeapp nativeapp.c

To actually get this compiled application on the device, however, does require a little extra legwork... With the AIR SDK, we're given the ability to SSH into the device as "devuser", with Developer Mode enabled. First off I copied the compiled binary to the downloads folder (plug in the PlayBook, and access its shared folders). Now, even though this is on the PlayBook, it's unusable, since it doesn't have the proper permissions. Since devuser is sandboxed and limited, it can't set permissions. However, you can get around this by first setting the binary as executable, and then zipping it up:

chmod +x nativeapp
zip nativeapp.zip nativeapp

Once you've copied that zip over to the downloads folder, copy it to somewhere on the device and unzip it (I used /tmp). From there you can run the binary!

I do recommend that you guys check this out, give it a go, and at the same time, get a little practice in before the Native SDK's release. Let me know how it goes! QNX is looking to be an awesome system to develop with!

Download the Source here. Download the Binary (zipped) here.

/Library/LaunchAgents/org.thebends.iphonedisk.mobile_fs_util.plist

After changing the values to your likings, simply reboot, and plug in your iDevice. In doing so (as well as having the afc2 protocol installed via Cydia), you are presented with a free and seamless way of transferring/editing files without other applications or ssh-access. Note: All symbolic links do not currently work, and clearly Finder does not exist in Windows, so for the time being this is not supported in windows...

Site and installer are here