Invalid glyph index when setting viewController's layoutManager for NSTextStorage subclass

Resolving 'Invalid Glyph Index' in NSTextStorage Subclasses with Custom Layout Managers

Understand and fix the 'Invalid glyph index' error when integrating a custom NSLayoutManager with an NSTextStorage subclass in iOS and macOS applications using TextKit.

When working with TextKit in iOS or macOS, particularly when subclassing NSTextStorage and providing a custom NSLayoutManager, you might encounter a cryptic 'Invalid glyph index' error. This error typically arises when the NSLayoutManager attempts to access glyphs that haven't been properly generated or are out of sync with the underlying text storage. This article delves into the common causes of this issue and provides a robust solution to ensure your custom text components work harmoniously.

Understanding the TextKit Architecture

Before diving into the solution, it's crucial to understand how TextKit components interact. NSTextStorage holds the text and its attributes, NSLayoutManager performs the actual layout of glyphs, and NSTextContainer defines the area where text is laid out. These three classes form a tightly coupled system. When NSTextStorage changes, it notifies its NSLayoutManagers to invalidate their layout caches and regenerate glyphs. If this notification or the subsequent glyph generation process is mishandled, an 'Invalid glyph index' error can occur.

flowchart TD
    A[NSTextStorage] --> B{Text Changes}
    B --> C[NSTextStorage notifies NSLayoutManager]
    C --> D[NSLayoutManager invalidates layout]
    D --> E[NSLayoutManager generates glyphs]
    E --> F[NSTextContainer defines layout area]
    F --> G[Text displayed in UIView/NSView]
    C -- X Invalid Glyph Index --> H[Error: Glyph index out of bounds]
    style H fill:#f9f,stroke:#333,stroke-width:2px

TextKit Component Interaction Flow and Potential Error Point

The Root Cause: Layout Manager Synchronization

The 'Invalid glyph index' error often stems from a synchronization problem between your NSTextStorage subclass and its associated NSLayoutManager. Specifically, if your NSTextStorage subclass overrides methods like replaceCharactersInRange:withString: or setAttributes:range:, it's responsible for notifying its layout managers about these changes. Failing to do so, or doing so incorrectly, can leave the layout manager with an outdated view of the text, leading to attempts to access non-existent glyphs.

The Solution: Proper edited:range:changeInLength: Invocation

The key to resolving this issue lies in correctly invoking the edited:range:changeInLength: method within your NSTextStorage subclass after any text or attribute modifications. This method is the primary way NSTextStorage informs its attached NSLayoutManagers about changes, prompting them to invalidate their caches and re-layout the text. The edited: parameter should specify the type of edit (e.g., NSTextStorageEditedCharacters for text changes, NSTextStorageEditedAttributes for attribute changes, or both). The range: parameter indicates the range of the change, and changeInLength: specifies how much the text length has changed.

// MyTextStorage.h
#import <UIKit/UIKit.h>

@interface MyTextStorage : NSTextStorage

@end

// MyTextStorage.m
#import "MyTextStorage.h"

@interface MyTextStorage ()
@property (nonatomic, strong) NSMutableAttributedString *backingStore;
@end

@implementation MyTextStorage

- (instancetype)init {
    self = [super init];
    if (self) {
        _backingStore = [[NSMutableAttributedString alloc] init];
    }
    return self;
}

- (NSString *)string {
    return self.backingStore.string;
}

- (NSDictionary<NSAttributedStringKey,id> *)attributesAtIndex:(NSUInteger)location effectiveRange:(NSRangePointer)range {
    return [self.backingStore attributesAtIndex:location effectiveRange:range];
}

- (void)replaceCharactersInRange:(NSRange)range withString:(NSString *)str {
    [self beginEditing];
    [self.backingStore replaceCharactersInRange:range withString:str];
    [self edited:NSTextStorageEditedCharacters range:range changeInLength:(NSInteger)str.length - (NSInteger)range.length];
    [self endEditing];
}

- (void)setAttributes:(NSDictionary<NSAttributedStringKey,id> *)attrs range:(NSRange)range {
    [self beginEditing];
    [self.backingStore setAttributes:attrs range:range];
    [self edited:NSTextStorageEditedAttributes range:range changeInLength:0];
    [self endEditing];
}

@end

Correct implementation of NSTextStorage subclass methods to notify layout managers.

In the example above, MyTextStorage correctly wraps its modifications to the backingStore (an NSMutableAttributedString) with beginEditing and endEditing. Crucially, it calls edited:range:changeInLength: immediately after the modification, providing the necessary information for the layout manager to update itself. This ensures that the NSLayoutManager is always aware of the latest state of the text storage, preventing 'Invalid glyph index' errors.

1. Subclass NSTextStorage

Create a custom subclass of NSTextStorage and provide an internal NSMutableAttributedString as a backing store for the text content.

2. Override Essential Methods

Override string, attributesAtIndex:effectiveRange:, replaceCharactersInRange:withString:, and setAttributes:range: to delegate to your backing store.

3. Implement beginEditing/endEditing and edited:range:changeInLength:

Crucially, wrap all modifications to the backing store within beginEditing and endEditing calls. After each modification, call edited:range:changeInLength: with the correct parameters to notify attached layout managers.

4. Integrate with NSLayoutManager and NSTextContainer

When setting up your NSTextView or UITextView, ensure you initialize it with your custom NSTextStorage and NSLayoutManager instances. The NSTextView will automatically create an NSTextContainer.