DEV Community: Warren Burton

Updating your Core Data model. Oh no.

Warren Burton — Tue, 08 Sep 2020 13:19:01 +0000

If you have spent any time with CoreData you will have opinions.

I personally love it. It's a simple persistence layer with graphical modelling and relationship resolution. The current NSPersistentContainer API can be spun up in just a few lines of code. The team who make it clearly like it too and every year brings a fresh crop of improvements.

You do need to realise that CoreData is not an SQLite database. While the default mode is to store your data in a SQLite file that's just an implementation detail. CoreData is also not a relational database so don't treat it like one.

One activity that can strike fear into even the most veteran developer is upgrading your xcdatamodel. Simple changes like addition of properties are very easy but more complex changes like modifying relationships can be difficult particularly when you have multiple model versions to deal with. No-one wants to lose client data.

In reality using the automated model upgrade system by using the upgrade toggle in your persistent store when you is mostly easy and transparent.

let container = NSPersistentContainer(name: "Unicorn")
let url = //url to store file
let store = NSPersistentStoreDescription(url: url)
store.shouldMigrateStoreAutomatically = true
store.shouldInferMappingModelAutomatically = true
container.persistentStoreDescriptions = [store]
container.loadPersistentStores { (storedesc, error) in
     //do data dependant setup
}

You can set shouldMigrateStoreAutomatically & shouldInferMappingModelAutomatically to true and move on with your day.

How do you deal with updating your model without introducing an new xcdatamodel version? If you prepare in advance you can add a little bit of wriggle room to your entities to avoid a dreaded model upgrade.

You use the Bucket of Goo™ technique. You add a place in your entity to store a generic Dictionary and store synthetic properties in that dictionary. This is not quite an anti-pattern, but it's not very scalable as there are costs both direct and in the form of technical debt. Still it might get you out of a temporary hole.

The first thing to do is add an metadata attribute to your Entity of type Data. You'll also see why you don't want to use String as the type.

Either the autogenerated class files or one that you create for yourself will look something like this

class Thing: NSManagedObject {

    @NSManaged var name: String?
    @NSManaged var index: Int16
    @NSManaged var metadata1: String?
    @NSManaged var metadata2: Data?
}

All the properties are expressed as @NSManaged to bind them into Objective-C runtime and KVO mechanics.

Next you want to provide simple access to your synthetic dictionary.

var metadataDictionaryString: SwiftDict {
    get {
        let string = metadata1
        if let dict = string?.propertyList() as? SwiftDict {
            return dict
        }
        return [:]
    }
    set {
        let string = (newValue as NSDictionary).description
        metadata1 = string
    }
}

Don't use this snippet. It's included to show you what an idiot past me was. It relies on the fact that description used to generate a valid plist for NSString,NSDictionary and NSArray. Thats no longer the case and types such as Data will crash at runtime.

The way to implement synthetic properties is to use Data for your bucket type and PropertyListSerialization to shuttle in and out.

var metadataDictionaryData: SwiftDict {
    get {
        if let data = metadata2 {
            do {
                let dict = try PropertyListSerialization.propertyList(from: data, options: [], format: nil) as? SwiftDict
                return dict ?? [:]
            }
            catch {
                print("plist decode err - ",error)
            }
        }
        return [:]
    }
    set {
        do {
            let data = try PropertyListSerialization.data(fromPropertyList: newValue, format: .binary, options: 0)
            metadata2 = data
        }
        catch {
            print("plist encode err - ",error)
        }
    }
}

The next step is to synthesise your new unexpected attribute. Lets call it extendedName.

extension Thing {

    @objc var extendedName: String? {
        get {
            return metadataDictionaryData["extendedName"] as? String
        }
        set {
            metadataDictionaryData["extendedName"] = newValue
        }
    }

}

You now have a new attribute in your model without needing to burn an xcdatamodel generation.

What's the cost?

The cost is:

Time to use. Each request needs to shuffle in or out of a Data blob.
CPU Cycles == Power consumption.
Not accessible via NSFetchRequest, i.e not searchable.
Possible tech debt from not doing it correctly first.

Taking a short test on macOS you can see

To access and update name which is a property on the entity Thing.

0.00485 s to add 1000 items
0.00513 s to update 1000 items
0.00657 s to read 1000 items

To access and update extendedName which is the synthetic property backed by metadata2

0.02022 s to add extended 1000 items
0.03116 s to update extended 1000 items
0.01254 s to read extended 1000 items

Add operations are ~5x more expensive.
Update is ~6x more expensive.
Read is ~2x more expensive.

Can you mitigate this?

In write-few/read-many situations you can use a cache. This does however increase your memory footprint.

fileprivate var metadata2Cache: SwiftDict?

var metadataDictionaryData: SwiftDict {
    get {
        if let cache = metadata2Cache {
            return cache
        }
        if let data = metadata2 {
            ...
        }
        return [:]

    }
    set {
        do {
            let data = ...
            metadata2 = data
            metadata2Cache = nil
        }
        catch {
            print("plist encode err - ",error)
        }
    }
}

Load the cache on first get and burn it down on any set.

Take a look at the difference.

**0.02022 s to add extended 1000 items
**0.03116 s to update extended 1000 items
**0.01254 s to read extended 1000 items
**0.00972 s to read extended 1000 items
**0.00840 s to read extended 1000 items

First read needs to decode the Data blob but subsequent reads come out of the cache and are about ~1.25x more expensive than a native attribute.

Conclusions

You can avoid a model upgrade by dumping attributes into synthetic version but there are large disadvantages in terms of processing cost. This technique would not be suitable for a performance sensitive loop.

You might be able to dig yourself out of a hole with this technique but don't rely on it as a long term expansion strategy. Create new model generations.

Refactoring for testability

Warren Burton — Tue, 07 Jul 2020 11:21:42 +0000

A lot of the time I will build a component and in the rush to make the component work I don't write tests. One thing that writing tests exposes is how badly your object is made for testing. To paraphrase a famous sailor "If it's difficult to test, you haven't written it right".

The case in hand is a file I wrote to request App Store reviews.

import Foundation
import StoreKit

class ReviewRequester {
    private let requestRateKey = "lastReviewRequestDate"

    @discardableResult public func requestReviewIfCriteriaMatching() -> Bool {
        if reviewCriteriaSatisfied() {
            markDate()
            SKStoreReviewController.requestReview()
            return true
        }
        return false
    }

    private func reviewCriteriaSatisfied() -> Bool {
        guard dateCriteriaPassed() else {
            return false
        }

        if someOtherBizCriteria {
            return true
        }
        return false
    }

    private func dateCriteriaPassed() -> Bool {
        let date = UserDefaults.standard.double(forKey: requestRateKey)
        let now = Date().timeIntervalSince1970
        let sevenDays = 3600.0 * 24 * 7
        return (now - date) > sevenDays
    }

    private func markDate() {
        let now = Date().timeIntervalSince1970
        UserDefaults.standard.setValue(now, forKey: requestRateKey)
    }

    func clearMarkers() {
        UserDefaults.standard.removeObject(forKey: requestRateKey)
        //...
    }

}

The object checks to see if it has been seven days since the last request and if other business criteria have been matched then allows a call to SKStoreReviewController.requestReview(). SKStoreReviewController has its own internal rules for presentation frequency but that's out of my hands. What I want to check is that my logic is correct.

The first thing to notice is that the elapsed time in dateCriteriaPassed() is dependant on Date which we have no control over. This is impossible to test without manipulating the system clock right?

Making a Time Machine.

You need a Time Machine. The testing requirement is that you can manipulate the date to perform the following tests.

If it has been less than 7 days since last request then no review request is made.
If it has been more than 7 days since last request then a review request is made.

To build a time machine im going to use a design pattern known as Strategy. This pattern allows you replace an algorithim with one of your choice.

protocol TimeStrategy {
    var now: Date { get }
}

class RealTime: TimeStrategy {
    var now: Date {
        return Date()
    }
}

I now have a date generator that conforms to TimeStrategy.

I add an init to optionally supply any TimeStrategy at instantiation.

var timeStrategy: TimeStrategy

init(timeStrategy: TimeStrategy = RealTime()) {
    self.timeStrategy = timeStrategy
}

I can replace all instances of Date() with timeStrategy.now.

private func dateCriteriaPassed() -> Bool {
    let date = UserDefaults.standard.double(forKey: requestRateKey)
    let now = timeStrategy.now.timeIntervalSince1970
    let sevenDays = 3600.0 * 24 * 7
    return (now - date) > sevenDays
}

private func markDate() {
    let now = timeStrategy.now.timeIntervalSince1970
    UserDefaults.standard.setValue(now, forKey: requestRateKey)
}

Manipulating Time but not Space

Now im ready to write those tests. First step is to create a TimeStrategy that can be manipulated.

class TestTime: TimeStrategy {
    private var relativeDate = Date().timeIntervalSince1970
    var now: Date {
        return Date(timeIntervalSince1970: relativeDate)
    }

    func advance(_ interval: TimeInterval) {
        relativeDate += interval
    }
}

Then I can use that time machine in my tests. In Quick/Nimble syntax:

import Quick
import Nimble

class ReviewRequesterTests: QuickSpec {

    override func spec() {

        it("less than 7 days since last request then no review request is made") {
            let timeMachine = TestTime()
            let requestor = ReviewRequester(timeStrategy: timeMachine)
            requestor.clearMarkers()

            expect(requestor.requestReviewIfCriteriaMatching()).to(beTrue())
            expect(requestor.requestReviewIfCriteriaMatching()).to(beFalse())
            timeMachine.advance(3600.0 * 24 * 7 - 1) //7 days - 1 sec
            expect(requestor.requestReviewIfCriteriaMatching()).to(beFalse())
        }

        it("more than 7 days since last request then a review request is made") {
            let timeMachine = TestTime()
            let requestor = ReviewRequester(timeStrategy: timeMachine)
            requestor.clearMarkers()

            expect(requestor.requestReviewIfCriteriaMatching()).to(beTrue())
            timeMachine.advance(3600.0 * 24 * 7 + 1) //7 days + 1 sec
            expect(requestor.requestReviewIfCriteriaMatching()).to(beTrue())
        }
    }
}

The negative test advances the date by not enough while the positive test does the opposite proving to me that the component works under changing conditions.

Summary

ReviewRequester was modified from an untestable component to an object that allows you to inject an abstracted time function. Production calls are unaffected at the call site but at a test level you know have power of the decisions that can be made.

Identify your indeterminate API.
Replace that API with a shim layer under your control.
Validate your assumptions about the object.

Always add a test

Warren Burton — Sun, 28 Jun 2020 22:18:23 +0000

Im writing about the flow for fixing a bug and committing that fix to the test suite.

My Crashlytics log for Trout OSX was showing this crash:

For some reason when a person went to delete a Link Type entry a crash was happening.

What was going on? A customer selects a link type and presses the delete button to remove it.

Its an arbitrary biz rule to not allow the default type to be deleted.

I had forgotten to bind the enabled property of the delete button to isDefault on the model object. A simple enough fix at UI level in the XIB.

But what about the future? Maybe I redo the app in SwiftUI ? The next task is to fix this in the logic.

The logic to delete looked like this:

@objc func deleteLinkType(_ type: VMLinkType) {

    if let links = type.links {
        let dtype = defaultLinkType()
        for link in links {
            link.type = dtype
        }
    }
    context.delete(type)
    normaliseTypeIndexes()
}

So I inhibit the default type from being deleted, again simple enough.

@objc func deleteLinkType(_ type: VMLinkType) {

    guard type.isDefaultValue == false else { return }

    //...
}

But now how do I capture this decision. I add a test for this condition.

I like using Quick & Nimble as the test language is more natural than base XCTest. So I drag the relevant file in DevSketch and write two tests.

delete link type cannot delete default link
delete link type can delete non default link

DevSketch generates the boilerplate for me as I can never remember the Quick syntax off the top of my head.

I can drag that boiler plate straight to Xcode and construct the tests.

import Quick
import Nimble
@testable import Trout

class TroutMetadataSpaceTests: QuickSpec {

    override func spec() {

        it("delete link type cannot delete default type") {
            let stack = TestCoreDataStack()
            let metadata = TroutMetadataSpace(context: stack.managedObjectContext)
            guard let defaulttype = metadata.defaultLinkType() else {
                XCTFail()
                return
            }

            metadata.deleteLinkType(defaulttype)
            expect(metadata.defaultLinkType()).toNot(beNil())
        }

        it("delete link type can delete non default type") {
            let stack = TestCoreDataStack()
            let metadata = TroutMetadataSpace(context: stack.managedObjectContext)

            let types = metadata.allLinkTypes
            guard let deletable = types.last else {
                XCTFail()
                return
            }

            metadata.deleteLinkType(deletable)
            let postdeletetypes = metadata.allLinkTypes
            expect(postdeletetypes.contains(deletable)).to(beFalse())
        }
    }
}

Two atomic tests to express the biz logic. Even if I end up refactoring the whole system in the future I have captured the decision.

In summary

Fix the UI bug. Technically I could stop here as the customer won't care.
Fix the underlying logic so that it can't happen again.
Add tests so the decision is captured for the long term good of the project.

Writing the tests might take a few minutes more than just putting a sticking plaster on the UI but now I can be confident it won't get borked in the future.